[{"id": "rec_Artificial_intelligence_0000", "text": "Artificial intelligence (AI) is the capability of computational systems to perform tasks typically associated with human intelligence, such as learning, reasoning, problem-solving, perception, and decision-making. It is a field of research in engineering, mathematics and computer science that develops and studies methods and software that enable machines to perceive their environment and use learning and intelligence to take actions that maximize their chances of achieving defined goals.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0001", "text": "High-profile applications of AI include advanced web search engines, chatbots, virtual assistants, autonomous vehicles, and play and analysis in strategy games (e.g., chess and Go). Since the 2020s, generative AI has become widely availabe to generate image, audio, and videos from text prompts.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0002", "text": "The traditional goals of AI research include learning, reasoning, knowledge representation, planning, natural language processing, and perception, as well as support for robotics. To reach these goals, AI researchers have used techniques including state space search and mathematical optimization, formal logic, artificial neural networks, and methods based on statistics, operations research, and economics. AI also draws upon psychology, linguistics, philosophy, neuroscience, and other fields", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0003", "text": ". Some companies, such as OpenAI, Google DeepMind and Meta, aim to create artificial general intelligence (AGI) \u2013 AI that can complete virtually any cognitive task at least as well as a human.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0004", "text": "Artificial intelligence was founded as an academic discipline in 1956, and the field went through multiple cycles of optimism throughout its history, followed by periods of disappointment and loss of funding, known as AI winters. Funding and interest increased substantially after 2012, when graphics processing units began being used to accelerate neural networks, and deep learning outperformed previous AI techniques. This growth accelerated further after 2017 with the transformer architecture", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0005", "text": ". In the 2020s, an AI boom has coincided with advances in generative AI, which allowed for the creation and modification of media. In addition to AI safety and unintended consequences and harms from the use of AI, ethical concerns, AI's long-term effects, and potential existential risks have prompted discussions of AI regulation.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0006", "text": "Goals\nThe general problem of simulating (or creating) intelligence has been broken into subproblems. These consist of particular traits or capabilities that researchers expect an intelligent system to display. The traits described below have received the most attention and cover the scope of AI research.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0007", "text": "Reasoning and problem-solving\nEarly researchers developed algorithms that imitated step-by-step reasoning that humans use when they solve puzzles or make logical deductions. By the late 1980s and 1990s, methods were developed for dealing with uncertain or incomplete information, employing concepts from probability and economics.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0008", "text": "Many of these algorithms are insufficient for solving large reasoning problems because they experience a \"combinatorial explosion\": They become exponentially slower as the problems grow. Even humans rarely use the step-by-step deduction that early AI research could model. They solve most of their problems using fast, intuitive judgments. Accurate and efficient reasoning is an unsolved problem.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0009", "text": "Knowledge representation\nKnowledge representation and knowledge engineering allow AI programs to answer questions intelligently and make deductions about real-world facts. Formal knowledge representations are used in content-based indexing and retrieval, scene interpretation, clinical decision support, knowledge discovery (mining \"interesting\" and actionable inferences from large databases), and other areas.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0010", "text": "A knowledge base is a body of knowledge represented in a form that can be used by a program. An ontology is the set of objects, relations, concepts, and properties used by a particular domain of knowledge", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0011", "text": ". Knowledge bases need to represent things such as objects, properties, categories, and relations between objects; situations, events, states, and time; causes and effects; knowledge about knowledge (what we know about what other people know); default reasoning (things that humans assume are true until they are told differently and will remain true even when other facts are changing); and many other aspects and domains of knowledge.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0012", "text": "Among the most difficult problems in knowledge representation are the breadth of commonsense knowledge (the set of atomic facts that the average person knows is enormous); and the sub-symbolic form of most commonsense knowledge (much of what people know is not represented as \"facts\" or \"statements\" that they could express verbally). There is also the difficulty of knowledge acquisition, the problem of obtaining knowledge for AI applications.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0013", "text": "Planning and decision-making", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0014", "text": "An \"agent\" is any entity (artificial or not) that perceives and takes actions in the world. A rational agent has goals or preferences and takes actions to make them happen. In automated planning, the agent has a specific goal. In automated decision-making, the agent has preferences\u2014there are some situations it would prefer to be in, and some situations it is trying to avoid", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0015", "text": ". The decision-making agent assigns a number to each situation (called the \"utility\") that measures how much the agent prefers it. For each possible action, it can calculate the \"expected utility\": the utility of all possible outcomes of the action, weighted by the probability that the outcome will occur. It can then choose the action with the maximum expected utility.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0016", "text": "In classical planning, the agent knows exactly what the effect of any action will be. In most real-world problems, however, the agent may not be certain about the situation they are in (it is \"unknown\" or \"unobservable\") and it may not know for certain what will happen after each possible action (it is not \"deterministic\"). It must choose an action by making a probabilistic guess and then reassess the situation to see if the action worked.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0017", "text": "Alongside thorough testing and improvement based on previous decisions, having an explanation for why the agent took certain decisions is a way to build trust, especially when the decisions have to be relied upon.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0018", "text": "In some problems, the agent's preferences may be uncertain, especially if there are other agents or humans involved. These can be learned (e.g., with inverse reinforcement learning), or the agent can seek information to improve its preferences. Information value theory can be used to weigh the value of exploratory or experimental actions", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0019", "text": ". The space of possible future actions and situations is typically intractably large, so the agents must take actions and evaluate situations while being uncertain of what the outcome will be.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0020", "text": "A Markov decision process has a transition model that describes the probability that a particular action will change the state in a particular way and a reward function that supplies the utility of each state and the cost of each action. A policy associates a decision with each possible state. The policy could be calculated (e.g., by iteration), be heuristic, or it can be learned.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0021", "text": "Game theory describes the rational behavior of multiple interacting agents and is used in AI programs that make decisions that involve other agents.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0022", "text": "Learning\nMachine learning is the study of programs that can improve their performance on a given task automatically. It has been a part of AI from the beginning.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0023", "text": "There are several kinds of machine learning. Unsupervised learning analyzes a stream of data and finds patterns and makes predictions without any other guidance. Supervised learning requires labeling the training data with the expected answers, and comes in two main varieties: classification (where the program must learn to predict what category the input belongs in) and regression (where the program must deduce a numeric function based on numeric input).", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0024", "text": "In reinforcement learning, the agent is rewarded for good responses and punished for bad ones. The agent learns to choose responses that are classified as \"good\". Transfer learning is when the knowledge gained from one problem is applied to a new problem. Deep learning is a type of machine learning that runs inputs through biologically inspired artificial neural networks for all of these types of learning.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0025", "text": "Computational learning theory can assess learners by computational complexity, by sample complexity (how much data is required), or by other notions of optimization.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0026", "text": "Natural language processing\nNatural language processing (NLP) allows programs to read, write and communicate in human languages. Specific problems include speech recognition, speech synthesis, machine translation, information extraction, information retrieval and question answering.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0027", "text": "Early work, based on Noam Chomsky's generative grammar and semantic networks, had difficulty with word-sense disambiguation unless restricted to small domains called \"micro-worlds\" (due to the common sense knowledge problem). Margaret Masterman believed that it was meaning and not grammar that was the key to understanding languages, and that thesauri and not dictionaries should be the basis of computational language structure.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0028", "text": "Modern deep learning techniques for NLP include word embedding (representing words, typically as vectors encoding their meaning), transformers (a deep learning architecture using an attention mechanism), and others. In 2019, generative pre-trained transformer (or \"GPT\") language models began to generate coherent text, and by 2023, these models were able to get human-level scores on the bar exam, SAT test, GRE test, and many other real-world applications.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0029", "text": "Perception\nMachine perception is the ability to use input from sensors (such as cameras, microphones, wireless signals, active lidar, sonar, radar, and tactile sensors) to deduce aspects of the world. Computer vision is the ability to analyze visual input.\nThe field includes speech recognition, image classification, facial recognition, object recognition, object tracking, and robotic perception.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0030", "text": "Social intelligence\nAffective computing is a field that comprises systems that recognize, interpret, process, or simulate human feeling, emotion, and mood. For example, some virtual assistants are programmed to speak conversationally or even to banter humorously; it makes them appear more sensitive to the emotional dynamics of human interaction, or to otherwise facilitate human\u2013computer interaction.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0031", "text": "However, this tends to give na\u00efve users an unrealistic conception of the intelligence of existing computer agents. Moderate successes related to affective computing include textual sentiment analysis and, more recently, multimodal sentiment analysis, wherein AI classifies the effects displayed by a videotaped subject.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0032", "text": "General intelligence\nA machine with artificial general intelligence would be able to solve a wide variety of problems with breadth and versatility similar to human intelligence.\n\nTechniques\nAI research uses a wide variety of techniques to accomplish the goals above.\n\nSearch and optimization\nThere are two different kinds of search used in AI: state space search and local search:", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0033", "text": "State space search\nState space search searches through a tree of possible states to try to find a goal state. For example, planning algorithms search through trees of goals and subgoals, attempting to find a path to a target goal, a process called means-ends analysis.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0034", "text": "Simple exhaustive searches are rarely sufficient for most real-world problems: the search space (the number of places to search) quickly grows to astronomical numbers. The result is a search that is too slow or never completes. \"Heuristics\" or \"rules of thumb\" can help prioritize choices that are more likely to reach a goal.\nAdversarial search is used for game-playing programs, such as chess or Go. It searches through a tree of possible moves and countermoves, looking for a winning position.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0035", "text": "Local search\nLocal search uses mathematical optimization to find a solution to a problem. It begins with some form of guess and refines it incrementally.\nGradient descent is a type of local search that optimizes a set of numerical parameters by incrementally adjusting them to minimize a loss function. Variants of gradient descent are commonly used to train neural networks, through the backpropagation algorithm.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0036", "text": "Another type of local search is evolutionary computation, which aims to iteratively improve a set of candidate solutions by \"mutating\" and \"recombining\" them, selecting only the fittest to survive each generation.\nDistributed search processes can coordinate via swarm intelligence algorithms. Two popular swarm algorithms used in search are particle swarm optimization (inspired by bird flocking) and ant colony optimization (inspired by ant trails).", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0037", "text": "Logic\nFormal logic is used for reasoning and knowledge representation.\nFormal logic comes in two main forms: propositional logic (which operates on statements that are true or false and uses logical connectives such as \"and\", \"or\", \"not\" and \"implies\") and predicate logic (which also operates on objects, predicates and relations and uses quantifiers such as \"Every X is a Y\" and \"There are some Xs that are Ys\").", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0038", "text": "Deductive reasoning in logic is the process of proving a new statement (conclusion) from other statements that are given and assumed to be true (the premises). Proofs can be structured as proof trees, in which nodes are labelled by sentences, and children nodes are connected to parent nodes by inference rules.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0039", "text": "Given a problem and a set of premises, problem-solving reduces to searching for a proof tree whose root node is labelled by a solution of the problem and whose leaf nodes are labelled by premises or axioms. In the case of Horn clauses, problem-solving search can be performed by reasoning forwards from the premises or backwards from the problem", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0040", "text": ". In the more general case of the clausal form of first-order logic, resolution is a single, axiom-free rule of inference, in which a problem is solved by proving a contradiction from premises that include the negation of the problem to be solved.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0041", "text": "Inference in both Horn clause logic and first-order logic is undecidable, and therefore intractable. However, backward reasoning with Horn clauses, which underpins computation in the logic programming language Prolog, is Turing complete. Moreover, its efficiency is competitive with computation in other symbolic programming languages.\nFuzzy logic assigns a \"degree of truth\" between 0 and 1. It can therefore handle propositions that are vague and partially true.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0042", "text": "Non-monotonic logics, including logic programming with negation as failure, are designed to handle default reasoning. Other specialized versions of logic have been developed to describe many complex domains.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0043", "text": "Probabilistic methods for uncertain reasoning", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0044", "text": "Many problems in AI (including reasoning, planning, learning, perception, and robotics) require the agent to operate with incomplete or uncertain information. AI researchers have devised a number of tools to solve these problems using methods from probability theory and economics. Precise mathematical tools have been developed that analyze how an agent can make choices and plan, using decision theory, decision analysis, and information value theory", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0045", "text": ". These tools include models such as Markov decision processes, dynamic decision networks, game theory and mechanism design.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0046", "text": "Bayesian networks are a tool that can be used for reasoning (using the Bayesian inference algorithm), learning (using the expectation\u2013maximization algorithm), planning (using decision networks) and perception (using dynamic Bayesian networks).\nProbabilistic algorithms can also be used for filtering, prediction, smoothing, and finding explanations for streams of data, thus helping perception systems analyze processes that occur over time (e.g., hidden Markov models or Kalman filters).", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0047", "text": "Classifiers and statistical learning methods", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0048", "text": "The simplest AI applications can be divided into two types: classifiers (e.g., \"if shiny then diamond\"), on one hand, and controllers (e.g., \"if diamond then pick up\"), on the other hand. Classifiers are functions that use pattern matching to determine the closest match. They can be fine-tuned based on chosen examples using supervised learning. Each pattern (also called an \"observation\") is labeled with a certain predefined class", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0049", "text": ". All the observations combined with their class labels are known as a data set. When a new observation is received, that observation is classified based on previous experience.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0050", "text": "There are many kinds of classifiers in use. The decision tree is the simplest and most widely used symbolic machine learning algorithm. K-nearest neighbor algorithm was the most widely used analogical AI until the mid-1990s, and Kernel methods such as the support vector machine (SVM) displaced k-nearest neighbor in the 1990s.\nThe naive Bayes classifier is reportedly the \"most widely used learner\" at Google, due in part to its scalability.\nNeural networks are also used as classifiers.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0051", "text": "Artificial neural networks", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0052", "text": "An artificial neural network is based on a collection of nodes also known as artificial neurons, which loosely model the neurons in a biological brain. It is trained to recognise patterns; once trained, it can recognise those patterns in fresh data. There is an input, at least one hidden layer of nodes and an output. Each node applies a function and once the weight crosses its specified threshold, the data is transmitted to the next layer", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0053", "text": ". A network is typically called a deep neural network if it has at least 2 hidden layers.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0054", "text": "Learning algorithms for neural networks use local search to choose the weights that will get the right output for each input during training. The most common training technique is the backpropagation algorithm. Neural networks learn to model complex relationships between inputs and outputs and find patterns in data. In theory, a neural network can learn any function.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0055", "text": "In feedforward neural networks the signal passes in only one direction. The term perceptron typically refers to a single-layer neural network. In contrast, deep learning uses many layers. Recurrent neural networks (RNNs) feed the output signal back into the input, which allows short-term memories of previous input events. Long short-term memory networks (LSTMs) are recurrent neural networks that better preserve longterm dependencies and are less sensitive to the vanishing gradient problem", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0056", "text": ". Convolutional neural networks (CNNs) use layers of kernels to more efficiently process local patterns. This local processing is especially important in image processing, where the early CNN layers typically identify simple local patterns such as edges and curves, with subsequent layers detecting more complex patterns like textures, and eventually whole objects.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0057", "text": "Deep learning\nDeep learning uses several layers of neurons between the network's inputs and outputs. The multiple layers can progressively extract higher-level features from the raw input. For example, in image processing, lower layers may identify edges, while higher layers may identify the concepts relevant to a human such as digits, letters, or faces.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0058", "text": "Deep learning has profoundly improved the performance of programs in many important subfields of artificial intelligence, including computer vision, speech recognition, natural language processing, image classification, and others. The reason that deep learning performs so well in so many applications is not known as of 2021", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0059", "text": ". The sudden success of deep learning in 2012\u20132015 did not occur because of some new discovery or theoretical breakthrough (deep neural networks and backpropagation had been described by many people, as far back as the 1950s) but because of two factors: the incredible increase in computer power (including the hundred-fold increase in speed by switching to GPUs) and the availability of vast amounts of training data, especially the giant curated datasets used for benchmark testing, such as", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0060", "text": "datasets used for benchmark testing, such as ImageNet.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0061", "text": "GPT", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0062", "text": "Generative pre-trained transformers (GPT) are large language models (LLMs) that generate text based on the semantic relationships between words in sentences. Text-based GPT models are pre-trained on a large corpus of text that can be from the Internet. The pretraining consists of predicting the next token (a token being usually a word, subword, or punctuation)", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0063", "text": ". Throughout this pretraining, GPT models accumulate knowledge about the world and can then generate human-like text by repeatedly predicting the next token. Typically, a subsequent training phase makes the model more truthful, useful, and harmless, usually with a technique called reinforcement learning from human feedback (RLHF). Current GPT models are prone to generating falsehoods called \"hallucinations\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0064", "text": ". These can be reduced with RLHF and quality data, but the problem has been getting worse for reasoning systems. Such systems are used in chatbots, which allow people to ask a question or request a task in simple text.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0065", "text": "Current models and services include ChatGPT, Claude, Gemini, Copilot, and Meta AI. Multimodal GPT models can process different types of data (modalities) such as images, videos, sound, and text.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0066", "text": "Hardware and software", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0067", "text": "In the late 2010s, graphics processing units (GPUs) that were increasingly designed with AI-specific enhancements and used with specialized TensorFlow software had replaced previously used central processing unit (CPUs) as the dominant means for large-scale (commercial and academic) machine learning models' training. Specialized programming languages such as Prolog were used in early AI research, but general-purpose programming languages like Python have become predominant.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0068", "text": "The transistor density in integrated circuits has been observed to roughly double every 18 months\u2014a trend known as Moore's law, named after the Intel co-founder Gordon Moore, who first identified it. Improvements in GPUs have been even faster, a trend sometimes called Huang's law, named after Nvidia co-founder and CEO Jensen Huang.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0069", "text": "Applications\nAI and machine learning technology is used in most of the essential applications of the 2020s, including:", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0070", "text": "search engines (such as Google Search)\ntargeting online advertisements\nrecommendation systems (offered by Netflix, YouTube or Amazon) driving internet traffic\ntargeted advertising (AdSense, Facebook)\nvirtual assistants (such as Siri or Alexa)\nautonomous vehicles (including drones, ADAS and self-driving cars)\nautomatic language translation (Microsoft Translator, Google Translate)\nfacial recognition (Apple's FaceID or Microsoft's DeepFace and Google's FaceNet)", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0071", "text": "image labeling (used by Facebook, Apple's Photos and TikTok).\nThe deployment of AI may be overseen by a chief automation officer (CAO).", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0072", "text": "Health and medicine\nIt has been suggested that AI can overcome discrepancies in funding allocated to different fields of research.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0073", "text": "AlphaFold 2 (2021) demonstrated the ability to approximate, in hours rather than months, the 3D structure of a protein. In 2023, it was reported that AI-guided drug discovery helped find a class of antibiotics capable of killing two different types of drug-resistant bacteria. In 2024, researchers used machine learning to accelerate the search for Parkinson's disease drug treatments", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0074", "text": ". Their aim was to identify compounds that block the clumping, or aggregation, of alpha-synuclein (the protein that characterises Parkinson's disease). They were able to speed up the initial screening process ten-fold and reduce the cost by a thousand-fold.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0075", "text": "Gaming", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0076", "text": "Game playing programs have been used since the 1950s to demonstrate and test AI's most advanced techniques. Deep Blue became the first computer chess-playing system to beat a reigning world chess champion, Garry Kasparov, on 11 May 1997. In 2011, in a Jeopardy! quiz show exhibition match, IBM's question answering system, Watson, defeated the two greatest Jeopardy! champions, Brad Rutter and Ken Jennings, by a significant margin", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0077", "text": ". In March 2016, AlphaGo won 4 out of 5 games of Go in a match with Go champion Lee Sedol, becoming the first computer Go-playing system to beat a professional Go player without handicaps. Then, in 2017, it defeated Ke Jie, who was the best Go player in the world. Other programs handle imperfect-information games, such as the poker-playing program Pluribus", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0078", "text": ". DeepMind developed increasingly generalistic reinforcement learning models, such as with MuZero, which could be trained to play chess, Go, or Atari games. In 2019, DeepMind's AlphaStar achieved grandmaster level in StarCraft II, a particularly challenging real-time strategy game that involves incomplete knowledge of what happens on the map", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0079", "text": ". In 2021, an AI agent competed in a PlayStation Gran Turismo competition, winning against four of the world's best Gran Turismo drivers using deep reinforcement learning. In 2024, Google DeepMind introduced SIMA, a type of AI capable of autonomously playing nine previously unseen open-world video games by observing screen output, as well as executing short, specific tasks in response to natural language instructions.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0080", "text": "Mathematics", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0081", "text": "In mathematics, probabilistic large language models are versatile, but can also produce wrong answers in the form of hallucinations. The Alibaba Group developed a version of its Qwen models called Qwen2-Math, that achieved state-of-the-art performance on several mathematical benchmarks, including 84% accuracy on the MATH dataset of competition mathematics problems", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0082", "text": ". In January 2025, Microsoft proposed the technique rStar-Math that leverages Monte Carlo tree search and step-by-step reasoning, enabling a relatively small language model like Qwen-7B to solve 53% of the AIME 2024 and 90% of the MATH benchmark problems. Google DeepMind has developed models for solving mathematical problems: AlphaTensor, AlphaGeometry, AlphaProof and AlphaEvolve.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0083", "text": "When natural language is used to describe mathematical problems, converters can transform such prompts into a formal language such as Lean to define mathematical tasks. The experimental model Gemini Deep Think accepts natural language prompts directly and achieved gold medal results in the International Math Olympiad of 2025.\nTopological deep learning integrates various topological approaches.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0084", "text": "Finance\nAccording to Nicolas Firzli, director of the World Pensions & Investments Forum, it may be too early to see the emergence of highly innovative AI-informed financial products and services. He argues that \"the deployment of AI tools will simply further automatise things: destroying tens of thousands of jobs in banking, financial planning, and pension advice in the process, but I'm not sure it will unleash a new wave of [e.g., sophisticated] pension innovation.\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0085", "text": "Military", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0086", "text": "Various countries are deploying AI military applications. The main applications enhance command and control, communications, sensors, integration and interoperability. Research is targeting intelligence collection and analysis, logistics, cyber operations, information operations, and semiautonomous and autonomous vehicles", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0087", "text": ". AI technologies enable coordination of sensors and effectors, threat detection and identification, marking of enemy positions, target acquisition, coordination and deconfliction of distributed Joint Fires between networked combat vehicles, both human-operated and autonomous.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0088", "text": "AI has been used in military operations in Iraq, Syria, Israel and Ukraine.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0089", "text": "Generative AI\nAgents", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0090", "text": "AI agents are software entities designed to perceive their environment, make decisions, and take actions autonomously to achieve specific goals. These agents can interact with users, their environment, or other agents. AI agents are used in various applications, including virtual assistants, chatbots, autonomous vehicles, game-playing systems, and industrial robotics. AI agents operate within the constraints of their programming, available computational resources, and hardware limitations", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0091", "text": ". This means they are restricted to performing tasks within their defined scope and have finite memory and processing capabilities. In real-world applications, AI agents often face time constraints for decision-making and action execution. Many AI agents incorporate learning algorithms, enabling them to improve their performance over time through experience or training. Using machine learning, AI agents can adapt to new situations and optimise their behaviour for their designated tasks.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0092", "text": "Web search\nMicrosoft introduced Copilot Search in February 2023 under the name Bing Chat. Copilot Search provides AI-generated summaries.\nGoogle introduced an AI Mode at its Google I/O event on 20 May 2025.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0093", "text": "Sexuality\nApplications of AI in this domain include AI-enabled menstruation and fertility trackers that analyze user data to offer predictions, AI-integrated sex toys (e.g., teledildonics), AI-generated sexual education content, and AI agents that simulate sexual and romantic partners (e.g., Replika). AI is also used for the production of non-consensual deepfake pornography, raising significant ethical and legal concerns.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0094", "text": "AI technologies have also been used to attempt to identify online gender-based violence and online sexual grooming of minors.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0095", "text": "Other industry-specific tasks\nIn a 2017 survey, one in five companies reported having incorporated \"AI\" in some offerings or processes. \nIn the field of evacuation and disaster managemen, AI has been used to investigate patterns in large-scale and small-scale evacuations using historical data from GPS, videos or social media.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0096", "text": "During the 2024 Indian elections, US$50 million was spent on authorized AI-generated content, notably by creating deepfakes of allied (including sometimes deceased) politicians to better engage with voters, and by translating speeches to various local languages.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0097", "text": "Ethics", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0098", "text": "AI has potential benefits and potential risks. AI may be able to advance science and find solutions for serious problems: Demis Hassabis of DeepMind hopes to \"solve intelligence, and then use that to solve everything else\". However, as the use of AI has become widespread, several unintended consequences and risks have been identified", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0099", "text": ". In-production systems can sometimes not factor ethics and bias into their AI training processes, especially when the AI algorithms are inherently unexplainable in deep learning.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0100", "text": "Risks and harm\nPrivacy and copyright\nMachine learning algorithms require large amounts of data. The techniques used to acquire this data have raised concerns about privacy, surveillance and copyright.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0101", "text": "AI-powered devices and services, such as virtual assistants and IoT products, continuously collect personal information, raising concerns about intrusive data gathering and unauthorized access by third parties. The loss of privacy is further exacerbated by AI's ability to process and combine vast amounts of data, potentially leading to a surveillance society where individual activities are constantly monitored and analyzed without adequate safeguards or transparency.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0102", "text": "Sensitive user data collected may include online activity records, geolocation data, video, or audio. For example, in order to build speech recognition algorithms, Amazon has recorded millions of private conversations and allowed temporary workers to listen to and transcribe some of them. Opinions about this widespread surveillance range from those who see it as a necessary evil to those for whom it is clearly unethical and a violation of the right to privacy.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0103", "text": "AI developers argue that this is the only way to deliver valuable applications and have developed several techniques that attempt to preserve privacy while still obtaining the data, such as data aggregation, de-identification and differential privacy. Since 2016, some privacy experts, such as Cynthia Dwork, have begun to view privacy in terms of fairness. Brian Christian wrote that experts have pivoted \"from the question of 'what they know' to the question of 'what they're doing with it'.\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0104", "text": "Generative AI is often trained on unlicensed copyrighted works, including in domains such as images or computer code; the output is then used under the rationale of \"fair use\". Experts disagree about how well and under what circumstances this rationale will hold up in courts of law; relevant factors may include \"the purpose and character of the use of the copyrighted work\" and \"the effect upon the potential market for the copyrighted work\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0105", "text": ". Website owners can indicate that they do not want their content scraped via a \"robots.txt\" file. However, some companies will scrape content regardless because the robots.txt file has no real authority. In 2023, leading authors (including John Grisham and Jonathan Franzen) sued AI companies for using their work to train generative AI", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0106", "text": ". Another discussed approach is to envision a separate sui generis system of protection for creations generated by AI to ensure fair attribution and compensation for human authors.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0107", "text": "Dominance by tech giants\nThe commercial AI scene is dominated by Big Tech companies such as Alphabet Inc., Amazon, Apple Inc., Meta Platforms, and Microsoft. Some of these players already own the vast majority of existing cloud infrastructure and computing power from data centers, allowing them to entrench further in the marketplace.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0108", "text": "Power needs and environmental impacts\nTechnology companies have built electricity and artificial intelligence infrastructure to facilitate the AI boom of the 2020s. A 2025 report from the consulting firm McKinsey & Company estimated that by 2030, $2.7 trillion would be invested into AI infrastructure and data centers in the US, surpassing World War II's Manhattan Project every month.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0109", "text": "In January 2024, the International Energy Agency (IEA) released Electricity 2024, Analysis and Forecast to 2026. This is the first IEA report to make projections for data centers and power consumption by AI and cryptocurrency. The report states that power demand for these uses might double by 2026, with the additional power consumption equaling that of Japan.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0110", "text": "Power consumption by AI is responsible for an increase in fossil fuel use, and has delayed closings of obsolete, carbon-emitting coal energy facilities. A ChatGPT search involves the use of 10 times the electrical energy as a Google search.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0111", "text": "A 2024 Goldman Sachs Research Paper, AI Data Centers and the Coming US Power Demand Surge, found \"US power demand (is) likely to experience growth not seen in a generation....\" and forecasts that, by 2030, US data centers will consume 8% of US power, as opposed to 3% in 2022, presaging growth for the electrical power generation industry by a variety of means. Data centers' need for more and more electrical power is such that they might max out the electrical grid", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0112", "text": ". The Big Tech companies counter that AI can be used to maximize the utilization of the grid by all.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0113", "text": "In 2024, the Wall Street Journal reported that big AI companies have begun negotiations with the US nuclear power providers to provide electricity to the data centers. In March 2024 Amazon purchased a Pennsylvania nuclear-powered data center for US$650 million.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0114", "text": "In September 2024, Microsoft announced an agreement with Constellation Energy to re-open the Three Mile Island nuclear power plant to provide Microsoft with 100% of all electric power produced by the plant for 20 years. Reopening the plant, which suffered a partial nuclear meltdown of its Unit 2 reactor in 1979, will require Constellation to get through strict regulatory processes which will include extensive safety scrutiny from the US Nuclear Regulatory Commission", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0115", "text": ". If approved (this will be the first ever US re-commissioning of a nuclear plant), over 835 megawatts of power \u2013 enough for 800,000 homes \u2013 of energy will be produced. The cost for re-opening and upgrading is estimated at US$1.6 billion and is dependent on tax breaks for nuclear power contained in the 2022 US Inflation Reduction Act. As of 2024, the US government and the state of Michigan have been investing almost US$2 billion to reopen the Palisades Nuclear reactor on Lake Michigan", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0116", "text": ". Closed since 2022, the plant was planned to be reopened in October 2025.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0117", "text": "After the last approval in September 2023, Taiwan suspended the approval of data centers north of Taoyuan with a capacity of more than 5 MW in 2024, due to power supply shortages. Taiwan aims to phase out nuclear power by 2025. \nSingapore imposed a ban on the opening of data centers in 2019 due to electric power, but in 2022, lifted this ban.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0118", "text": "Although most nuclear plants in Japan have been shut down after the 2011 Fukushima nuclear accident, according to an October 2024 Bloomberg article in Japanese, cloud gaming services company Ubitus, in which Nvidia has a stake, is looking for land in Japan near a nuclear power plant for a new data center for generative AI.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0119", "text": "On 1 November 2024, the Federal Energy Regulatory Commission (FERC) rejected an application submitted by Talen Energy for approval to supply some electricity from the nuclear power station Susquehanna to Amazon's data center.\nAccording to the Commission Chairman Willie L. Phillips, it is a burden on the electricity grid as well as a significant cost shifting concern to households and other business sectors.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0120", "text": "In 2025, a report prepared by the IEA estimated the greenhouse gas emissions from the energy consumption of AI at 180 million tons. By 2035, these emissions could rise to 300\u2013500 million tonnes depending on what measures will be taken. This is below 1.5% of the energy sector emissions. The emissions reduction potential of AI was estimated at 5% of the energy sector emissions, but rebound effects (for example if people switch from public transport to autonomous cars) can reduce it.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0121", "text": "Misinformation", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0122", "text": "YouTube, Facebook and others use recommender systems to guide users to more content. These AI programs were given the goal of maximizing user engagement (that is, the only goal was to keep people watching). The AI learned that users tended to choose misinformation, conspiracy theories, and extreme partisan content, and, to keep them watching, the AI recommended more of it", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0123", "text": ". Users also tended to watch more content on the same subject, so the AI led people into filter bubbles where they received multiple versions of the same misinformation. This convinced many users that the misinformation was true, and ultimately undermined trust in institutions, the media and the government. The AI program had correctly learned to maximize its goal, but the result was harmful to society. After the U.S", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0124", "text": ". After the U.S. election in 2016, major technology companies took some steps to mitigate the problem.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0125", "text": "In the early 2020s, generative AI began to create images, audio, and texts that are virtually indistinguishable from real photographs, recordings, or human writing, while realistic AI-generated videos became feasible in the mid-2020s. It is possible for bad actors to use this technology to create massive amounts of misinformation or propaganda; one such potential malicious use is deepfakes for computational propaganda", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0126", "text": ". AI pioneer and Nobel Prize-winning computer scientist Geoffrey Hinton expressed concern about AI enabling \"authoritarian leaders to manipulate their electorates\" on a large scale, among other risks. The ability to influence electorates has been proved in at least one study. This same study shows more inaccurate statements from the models when they advocate for candidates of the political right.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0127", "text": "AI researchers at Microsoft, OpenAI, universities and other organisations have suggested using \"personhood credentials\" as a way to overcome online deception enabled by AI models.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0128", "text": "Algorithmic bias and fairness", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0129", "text": "Machine learning applications can be biased if they learn from biased data. The developers may not be aware that the bias exists. Discriminatory behavior by some LLMs can be observed in their output. Bias can be introduced by the way training data is selected and by the way a model is deployed. If a biased algorithm is used to make decisions that can seriously harm people (as it can in medicine, finance, recruitment, housing or policing) then the algorithm may cause discrimination", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0130", "text": ". The field of fairness studies how to prevent harms from algorithmic biases.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0131", "text": "On 28 June 2015, Google Photos's new image labeling feature mistakenly identified Jacky Alcine and a friend as \"gorillas\" because they were black. The system was trained on a dataset that contained very few images of black people, a problem called \"sample size disparity\". Google \"fixed\" this problem by preventing the system from labelling anything as a \"gorilla\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0132", "text": ". Eight years later, in 2023, Google Photos still could not identify a gorilla, and neither could similar products from Apple, Facebook, Microsoft and Amazon.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0133", "text": "COMPAS is a commercial program widely used by U.S. courts to assess the likelihood of a defendant becoming a recidivist. In 2016, Julia Angwin at ProPublica discovered that COMPAS exhibited racial bias, despite the fact that the program was not told the races of the defendants", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0134", "text": ". Although the error rate for both whites and blacks was calibrated equal at exactly 61%, the errors for each race were different\u2014the system consistently overestimated the chance that a black person would re-offend and would underestimate the chance that a white person would not re-offend", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0135", "text": ". In 2017, several researchers showed that it was mathematically impossible for COMPAS to accommodate all possible measures of fairness when the base rates of re-offense were different for whites and blacks in the data.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0136", "text": "A program can make biased decisions even if the data does not explicitly mention a problematic feature (such as \"race\" or \"gender\"). The feature will correlate with other features (like \"address\", \"shopping history\" or \"first name\"), and the program will make the same decisions based on these features as it would on \"race\" or \"gender\". Moritz Hardt said \"the most robust fact in this research area is that fairness through blindness doesn't work.\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0137", "text": "Criticism of COMPAS highlighted that machine learning models are designed to make \"predictions\" that are only valid if we assume that the future will resemble the past. If they are trained on data that includes the results of racist decisions in the past, machine learning models must predict that racist decisions will be made in the future. If an application then uses these predictions as recommendations, some of these \"recommendations\" will likely be racist", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0138", "text": ". Thus, machine learning is not well suited to help make decisions in areas where there is hope that the future will be better than the past. It is descriptive rather than prescriptive.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0139", "text": "Bias and unfairness may go undetected because the developers are overwhelmingly white and male: among AI engineers, about 4% are black and 20% are women.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0140", "text": "There are various conflicting definitions and mathematical models of fairness. These notions depend on ethical assumptions, and are influenced by beliefs about society. One broad category is distributive fairness, which focuses on the outcomes, often identifying groups and seeking to compensate for statistical disparities. Representational fairness tries to ensure that AI systems do not reinforce negative stereotypes or render certain groups invisible", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0141", "text": ". Procedural fairness focuses on the decision process rather than the outcome. The most relevant notions of fairness may depend on the context, notably the type of AI application and the stakeholders. The subjectivity in the notions of bias and fairness makes it difficult for companies to operationalize them", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0142", "text": ". Having access to sensitive attributes such as race or gender is also considered by many AI ethicists to be necessary in order to compensate for biases, but it may conflict with anti-discrimination laws.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0143", "text": "At the 2022 ACM Conference on Fairness, Accountability, and Transparency a paper reported that a CLIP\u2011based (Contrastive Language-Image Pre-training) robotic system reproduced harmful gender\u2011 and race\u2011linked stereotypes in a simulated manipulation task. The authors recommended robot\u2011learning methods which physically manifest such harms be \"paused, reworked, or even wound down when appropriate, until outcomes can be proven safe, effective, and just.\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0144", "text": "Lack of transparency\nMany AI systems are so complex that their designers cannot explain how they reach their decisions. Particularly with deep neural networks, in which there are many non-linear relationships between inputs and outputs. But some popular explainability techniques exist.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0145", "text": "It is impossible to be certain that a program is operating correctly if no one knows how exactly it works. There have been many cases where a machine learning program passed rigorous tests, but nevertheless learned something different than what the programmers intended", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0146", "text": ". For example, a system that could identify skin diseases better than medical professionals was found to actually have a strong tendency to classify images with a ruler as \"cancerous\", because pictures of malignancies typically include a ruler to show the scale. Another machine learning system designed to help effectively allocate medical resources was found to classify patients with asthma as being at \"low risk\" of dying from pneumonia", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0147", "text": ". Having asthma is actually a severe risk factor, but since the patients having asthma would usually get much more medical care, they were relatively unlikely to die according to the training data. The correlation between asthma and low risk of dying from pneumonia was real, but misleading.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0148", "text": "People who have been harmed by an algorithm's decision have a right to an explanation. Doctors, for example, are expected to clearly and completely explain to their colleagues the reasoning behind any decision they make. Early drafts of the European Union's General Data Protection Regulation in 2016 included an explicit statement that this right exists. Industry experts noted that this is an unsolved problem with no solution in sight", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0149", "text": ". Regulators argued that nevertheless the harm is real: if the problem has no solution, the tools should not be used.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0150", "text": "DARPA established the XAI (\"Explainable Artificial Intelligence\") program in 2014 to try to solve these problems.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0151", "text": "Several approaches aim to address the transparency problem. SHAP enables to visualise the contribution of each feature to the output. LIME can locally approximate a model's outputs with a simpler, interpretable model. Multitask learning provides a large number of outputs in addition to the target classification. These other outputs can help developers deduce what the network has learned", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0152", "text": ". Deconvolution, DeepDream and other generative methods can allow developers to see what different layers of a deep network for computer vision have learned, and produce output that can suggest what the network is learning. For generative pre-trained transformers, Anthropic developed a technique based on dictionary learning that associates patterns of neuron activations with human-understandable concepts.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0153", "text": "Bad actors and weaponized AI\nArtificial intelligence provides a number of tools that are useful to bad actors, such as authoritarian governments, terrorists, criminals or rogue states.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0154", "text": "A lethal autonomous weapon is a machine that locates, selects and engages human targets without human supervision. Widely available AI tools can be used by bad actors to develop inexpensive autonomous weapons and, if produced at scale, they are potentially weapons of mass destruction. Even when used in conventional warfare, they currently cannot reliably choose targets and could potentially kill an innocent person", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0155", "text": ". In 2014, 30 nations (including China) supported a ban on autonomous weapons under the United Nations' Convention on Certain Conventional Weapons, however the United States and others disagreed. By 2015, over fifty countries were reported to be researching battlefield robots.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0156", "text": "AI tools make it easier for authoritarian governments to efficiently control their citizens in several ways. Face and voice recognition allow widespread surveillance. Machine learning, operating this data, can classify potential enemies of the state and prevent them from hiding. Recommendation systems can precisely target propaganda and misinformation for maximum effect. Deepfakes and generative AI aid in producing misinformation", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0157", "text": ". Advanced AI can make authoritarian centralized decision-making more competitive than liberal and decentralized systems such as markets. It lowers the cost and difficulty of digital warfare and advanced spyware. All these technologies have been available since 2020 or earlier\u2014AI facial recognition systems are already being used for mass surveillance in China.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0158", "text": "There are many other ways in which AI is expected to help bad actors, some of which can not be foreseen. For example, machine-learning AI is able to design tens of thousands of toxic molecules in a matter of hours.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0159", "text": "Technological unemployment\nEconomists have frequently highlighted the risks of redundancies from AI, and speculated about unemployment if there is no adequate social policy for full employment.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0160", "text": "In the past, technology has tended to increase rather than reduce total employment, but economists acknowledge that \"we're in uncharted territory\" with AI. A survey of economists showed disagreement about whether the increasing use of robots and AI will cause a substantial increase in long-term unemployment, but they generally agree that it could be a net benefit if productivity gains are redistributed", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0161", "text": ". Risk estimates vary; for example, in the 2010s, Michael Osborne and Carl Benedikt Frey estimated 47% of U.S. jobs are at \"high risk\" of potential automation, while an OECD report classified only 9% of U.S. jobs as \"high risk\". The methodology of speculating about future employment levels has been criticised as lacking evidential foundation, and for implying that technology, rather than social policy, creates unemployment, as opposed to redundancies", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0162", "text": ". In April 2023, it was reported that 70% of the jobs for Chinese video game illustrators had been eliminated by generative artificial intelligence. Early-career workers showed decreasing employment rates in some AI-exposed occupations.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0163", "text": "Unlike previous waves of automation, many middle-class jobs may be eliminated by artificial intelligence; The Economist stated in 2015 that \"the worry that AI could do to white-collar jobs what steam power did to blue-collar ones during the Industrial Revolution\" is \"worth taking seriously\". Jobs at extreme risk range from paralegals to fast food cooks, while job demand is likely to increase for care-related professions ranging from personal healthcare to the clergy", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0164", "text": ". In July 2025, Ford CEO Jim Farley predicted that \"artificial intelligence is going to replace literally half of all white-collar workers in the U.S.\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0165", "text": "From the early days of the development of artificial intelligence, there have been arguments, for example, those put forward by Joseph Weizenbaum, about whether tasks that can be done by computers actually should be done by them, given the difference between computers and humans, and between quantitative calculation and qualitative, value-based judgement.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0166", "text": "Substitution for human\u2013human interaction", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0167", "text": "With the increase of loneliness in the early 21st century, AI is sometimes identified as a potential source of relief to this problem. It would be possible, via human-like qualities built into AI products, for individuals to assume that this need can be met by artificial means. In some cases, people approach artificial intelligence for companionship when they believe that they would not find acceptance due to feeling outcast", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0168", "text": ". Examples of harm coming to humans from advanced chatbots have been reported in courts in the United States, with AI companies accused of creating products that endanger humans through emotional confusion or deception.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0169", "text": "Existential risk", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0170", "text": "Recent public debates in artificial intelligence have increasingly focused on its broader societal and ethical implications. It has been argued AI will become so powerful that humanity may irreversibly lose control of it. This could, as physicist Stephen Hawking stated, \"spell the end of the human race\". This scenario has been common in science fiction, when a computer or robot suddenly develops a human-like \"self-awareness\" (or \"sentience\" or \"consciousness\") and becomes a malevolent character", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0171", "text": ". These sci-fi scenarios are misleading in several ways.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0172", "text": "First, AI does not require human-like sentience to be an existential risk. Modern AI programs are given specific goals and use learning and intelligence to achieve them. Philosopher Nick Bostrom argued that if one gives almost any goal to a sufficiently powerful AI, it may choose to destroy humanity to achieve it (he used the example of an automated paperclip factory that destroys the world to get more iron for paperclips)", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0173", "text": ". Stuart Russell gives the example of household robot that tries to find a way to kill its owner to prevent it from being unplugged, reasoning that \"you can't fetch the coffee if you're dead.\" In order to be safe for humanity, a superintelligence would have to be genuinely aligned with humanity's morality and values so that it is \"fundamentally on our side\".", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0174", "text": "Second, Yuval Noah Harari argues that AI does not require a robot body or physical control to pose an existential risk. The essential parts of civilization are not physical. Things like ideologies, law, government, money and the economy are built on language; they exist because there are stories that billions of people believe. The current prevalence of misinformation suggests that an AI could use language to convince people to believe anything, even to take actions that are destructive", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0175", "text": ". Geoffrey Hinton said in 2025 that modern AI is particularly \"good at persuasion\" and getting better all the time. He asks \"Suppose you wanted to invade the capital of the US. Do you have to go there and do it yourself? No. You just have to be good at persuasion.\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0176", "text": "The opinions amongst experts and industry insiders are mixed, with sizable fractions both concerned and unconcerned by risk from eventual superintelligent AI. Personalities such as Stephen Hawking, Bill Gates, and Elon Musk, as well as AI pioneers such as Geoffrey Hinton, Yoshua Bengio, Stuart Russell, Demis Hassabis, and Sam Altman, have expressed concerns about existential risk from AI.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0177", "text": "In May 2023, Geoffrey Hinton announced his resignation from Google in order to be able to \"freely speak out about the risks of AI\" without \"considering how this impacts Google\". He notably mentioned risks of an AI takeover, and stressed that in order to avoid the worst outcomes, establishing safety guidelines will require cooperation among those competing in use of AI.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0178", "text": "In 2023, many leading AI experts endorsed the joint statement that \"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war\".", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0179", "text": "Some other researchers were more optimistic", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0180", "text": ". AI pioneer J\u00fcrgen Schmidhuber did not sign the joint statement, emphasising that in 95% of all cases, AI research is about making \"human lives longer and healthier and easier.\" While the tools that are now being used to improve lives can also be used by bad actors, \"they can also be used against the bad actors.\" Andrew Ng also argued that \"it's a mistake to fall for the doomsday hype on AI\u2014and that regulators who do will only benefit vested interests.\" Yann LeCun, a Turing Award winner,", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0181", "text": "interests.\" Yann LeCun, a Turing Award winner, disagreed with the idea that AI will subordinate humans \"simply because they are smarter, let alone destroy [us]\", \"scoff[ing] at his peers' dystopian scenarios of supercharged misinformation and even, eventually, human extinction.\" In contrast, he claimed that \"intelligent machines will usher in a new renaissance for humanity, a new era of enlightenment.\" In the early 2010s, experts argued that the risks are too distant in the future to warrant", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0182", "text": "risks are too distant in the future to warrant research or that humans will be valuable from the perspective of a superintelligent machine", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0183", "text": ". However, after 2016, the study of current and future risks and possible solutions became a serious area of research.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0184", "text": "Ethical machines and alignment\nFriendly AI are machines that have been designed from the beginning to minimize risks and to make choices that benefit humans. Eliezer Yudkowsky, who coined the term, argues that developing friendly AI should be a higher research priority: it may require a large investment and it must be completed before AI becomes an existential risk.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0185", "text": "Machines with intelligence have the potential to use their intelligence to make ethical decisions. The field of machine ethics provides machines with ethical principles and procedures for resolving ethical dilemmas.\nThe field of machine ethics is also called computational morality,\nand was founded at an AAAI symposium in 2005.\nOther approaches include Wendell Wallach's \"artificial moral agents\" and Stuart J. Russell's three principles for developing provably beneficial machines.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0186", "text": "Open source", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0187", "text": "Active organizations in the AI open-source community include Hugging Face, Google, EleutherAI and Meta. Various AI models, such as Llama 2, Mistral or Stable Diffusion, have been made open-weight, meaning that their architecture and trained parameters (the \"weights\") are publicly available. Open-weight models can be freely fine-tuned, which allows companies to specialize them with their own data and for their own use-case", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0188", "text": ". Open-weight models are useful for research and innovation but can also be misused. Since they can be fine-tuned, any built-in security measure, such as objecting to harmful requests, can be trained away until it becomes ineffective. Some researchers warn that future AI models may develop dangerous capabilities (such as the potential to drastically facilitate bioterrorism) and that once released on the Internet, they cannot be deleted everywhere if needed", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0189", "text": ". They recommend pre-release audits and cost-benefit analyses.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0190", "text": "Frameworks\nArtificial intelligence projects can be guided by ethical considerations during the design, development, and implementation of an AI system. An AI framework such as the Care and Act Framework, developed by the Alan Turing Institute and based on the SUM values, outlines four main ethical dimensions, defined as follows:", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0191", "text": "Respect the dignity of individual people\nConnect with other people sincerely, openly, and inclusively\nCare for the wellbeing of everyone\nProtect social values, justice, and the public interest", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0192", "text": "Other developments in ethical frameworks include those decided upon during the Asilomar Conference, the Montreal Declaration for Responsible AI, and the IEEE's Ethics of Autonomous Systems initiative, among others; however, these principles are not without criticism, especially regarding the people chosen to contribute to these frameworks.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0193", "text": "Promotion of the wellbeing of the people and communities that these technologies affect requires consideration of the social and ethical implications at all stages of AI system design, development and implementation, and collaboration between job roles such as data scientists, product managers, data engineers, domain experts, and delivery managers.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0194", "text": "The UK AI Safety Institute released in 2024 a testing toolset called 'Inspect' for AI safety evaluations available under an MIT open-source licence which is freely available on GitHub and can be improved with third-party packages. It can be used to evaluate AI models in a range of areas including core knowledge, ability to reason, and autonomous capabilities.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0195", "text": "Regulation", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0196", "text": "The regulation of artificial intelligence is the development of public sector policies and laws for promoting and regulating AI; it is therefore related to the broader regulation of algorithms. The regulatory and policy landscape for AI is an emerging issue in jurisdictions globally. According to AI Index at Stanford, the annual number of AI-related laws passed in the 127 survey countries jumped from one passed in 2016 to 37 passed in 2022 alone", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0197", "text": ". Between 2016 and 2020, more than 30 countries adopted dedicated strategies for AI. Most EU member states had released national AI strategies, as had Canada, China, India, Japan, Mauritius, the Russian Federation, Saudi Arabia, United Arab Emirates, U.S., and Vietnam. Others were in the process of elaborating their own AI strategy, including Bangladesh, Malaysia and Tunisia", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0198", "text": ". The Global Partnership on Artificial Intelligence was launched in June 2020, stating a need for AI to be developed in accordance with human rights and democratic values, to ensure public confidence and trust in the technology. Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher published a joint statement in November 2021 calling for a government commission to regulate AI", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0199", "text": ". In 2023, OpenAI leaders published recommendations for the governance of superintelligence, which they believe may happen in less than 10 years. In 2023, the United Nations also launched an advisory body to provide recommendations on AI governance; the body comprises technology company executives, government officials and academics. On 1 August 2024, the EU Artificial Intelligence Act entered into force, establishing the first comprehensive EU-wide AI regulation", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0200", "text": ". In 2024, the Council of Europe created the first international legally binding treaty on AI, called the \"Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law\". It was adopted by the European Union, the United States, the United Kingdom, and other signatories.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0201", "text": "In a 2022 Ipsos survey, attitudes towards AI varied greatly by country; 78% of Chinese citizens, but only 35% of Americans, agreed that \"products and services using AI have more benefits than drawbacks\". A 2023 Reuters/Ipsos poll found that 61% of Americans agree, and 22% disagree, that AI poses risks to humanity", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0202", "text": ". In a 2023 Fox News poll, 35% of Americans thought it \"very important\", and an additional 41% thought it \"somewhat important\", for the federal government to regulate AI, versus 13% responding \"not very important\" and 8% responding \"not at all important\".", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0203", "text": "In November 2023, the first global AI Safety Summit was held in Bletchley Park in the UK to discuss the near and far term risks of AI and the possibility of mandatory and voluntary regulatory frameworks. 28 countries including the United States, China, and the European Union issued a declaration at the start of the summit, calling for international co-operation to manage the challenges and risks of artificial intelligence", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0204", "text": ". In May 2024 at the AI Seoul Summit, 16 global AI tech companies agreed to safety commitments on the development of AI.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0205", "text": "In March 2026, the United Nations convened the inaugural meeting of the Independent International Scientific Panel on AI, a 40-member expert body established under the Global Digital Compact to produce annual evidence-based reports on AI's societal impacts.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0206", "text": "History", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0207", "text": "The study of mechanical or \"formal\" reasoning began with philosophers and mathematicians in antiquity. The study of logic led directly to Alan Turing's theory of computation, which suggested that a machine, by shuffling symbols as simple as \"0\" and \"1\", could simulate any conceivable form of mathematical reasoning. This, along with concurrent discoveries in cybernetics, information theory and neurobiology, led researchers to consider the possibility of building an \"electronic brain\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0208", "text": ". They developed several areas of research that would become part of AI, such as McCulloch and Pitts design for \"artificial neurons\" in 1943, and Turing's influential 1950 paper 'Computing Machinery and Intelligence', which introduced the Turing test and showed that \"machine intelligence\" was plausible.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0209", "text": "The field of AI research was founded at a workshop at Dartmouth College in 1956. The attendees became the leaders of AI research in the 1960s. They and their students produced programs that the press described as \"astonishing\": computers were learning checkers strategies, solving word problems in algebra, proving logical theorems and speaking English. Artificial intelligence laboratories were set up at a number of British and U.S. universities in the latter 1950s and early 1960s.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0210", "text": "Researchers in the 1960s and the 1970s were convinced that their methods would eventually succeed in creating a machine with general intelligence and considered this the goal of their field. In 1965 Herbert Simon predicted, \"machines will be capable, within twenty years, of doing any work a man can do\". In 1967 Marvin Minsky agreed, writing that \"within a generation ... the problem of creating 'artificial intelligence' will substantially be solved\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0211", "text": ". They had, however, underestimated the difficulty of the problem. In 1974, both the U.S. and British governments cut off exploratory research in response to the criticism of Sir James Lighthill and ongoing pressure from the U.S. Congress to fund more productive projects. Minsky and Papert's book Perceptrons was understood as proving that artificial neural networks would never be useful for solving real-world tasks, thus discrediting the approach altogether", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0212", "text": ". The \"AI winter\", a period when obtaining funding for AI projects was difficult, followed.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0213", "text": "In the early 1980s, AI research was revived by the commercial success of expert systems, a form of AI program that simulated the knowledge and analytical skills of human experts. By 1985, the market for AI had reached over a billion dollars. At the same time, Japan's fifth generation computer project inspired the U.S. and British governments to restore funding for academic research", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0214", "text": ". However, beginning with the collapse of the Lisp Machine market in 1987, AI once again fell into disrepute, and a second, longer-lasting winter began.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0215", "text": "Up to this point, most of AI's funding had gone to projects that used high-level symbols to represent mental objects like plans, goals, beliefs, and known facts. In the 1980s, some researchers began to doubt that this approach would be able to imitate all the processes of human cognition, especially perception, robotics, learning and pattern recognition, and began to look into \"sub-symbolic\" approaches", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0216", "text": ". Rodney Brooks rejected \"representation\" in general and focussed directly on engineering machines that move and survive. Judea Pearl, Lotfi Zadeh, and others developed methods that handled incomplete and uncertain information by making reasonable guesses rather than precise logic. But the most important development was the revival of \"connectionism\", including neural network research, by Geoffrey Hinton and others", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0217", "text": ". In 1990, Yann LeCun successfully showed that convolutional neural networks can recognize handwritten digits, the first of many successful applications of neural networks.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0218", "text": "AI gradually restored its reputation in the late 1990s and early 21st century by exploiting formal mathematical methods and by finding specific solutions to specific problems. This \"narrow\" and \"formal\" focus allowed researchers to produce verifiable results and collaborate with other fields (such as statistics, economics and mathematics)", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0219", "text": ". By 2000, solutions developed by AI researchers were being widely used, although in the 1990s they were rarely described as \"artificial intelligence\" (a tendency known as the AI effect).", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0220", "text": "However, several academic researchers became concerned that AI was no longer pursuing its original goal of creating versatile, fully intelligent machines. Beginning around 2002, they founded the subfield of artificial general intelligence (or \"AGI\"), which had several well-funded institutions by the 2010s.\nDeep learning began to dominate industry benchmarks in 2012 and was adopted throughout the field.\nFor many specific tasks, other methods were abandoned.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0221", "text": "Deep learning's success was based on both hardware improvements (faster computers, graphics processing units, cloud computing) and access to large amounts of data (including curated datasets, such as ImageNet). Deep learning's success led to an enormous increase in interest and funding in AI. The amount of machine learning research (measured by total publications) increased by 50% in the years 2015\u20132019.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0222", "text": "In 2016, issues of fairness and the misuse of technology were catapulted into center stage at machine learning conferences, publications vastly increased, funding became available, and many researchers re-focussed their careers on these issues. The alignment problem became a serious field of academic study.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0223", "text": "In the late 2010s and early 2020s, AGI companies began to deliver programs that created enormous interest. In 2015, AlphaGo, developed by DeepMind, beat the world champion Go player. The program taught only the game's rules and developed a strategy by itself. GPT-3 is a large language model that was released in 2020 by OpenAI and is capable of generating high-quality human-like text", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0224", "text": ". ChatGPT, launched on 30 November 2022, became the fastest-growing consumer software application in history, gaining over 100 million users in two months. It marked what is widely regarded as AI's breakout year, bringing it into the public consciousness. These programs, and others, inspired an aggressive AI boom, where large companies began investing billions of dollars in AI research. According to AI Impacts, about US$50 billion annually was invested in \"AI\" around 2022 in the U.S", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0225", "text": ". alone and about 20% of the new U.S. Computer Science PhD graduates have specialized in \"AI\". About 800,000 \"AI\"-related U.S. job openings existed in 2022. According to PitchBook research, 22% of newly funded startups in 2024 claimed to be AI companies.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0226", "text": "Philosophy\nPhilosophical debates have historically sought to determine the nature of intelligence and how to make intelligent machines. Another major focus has been whether machines can be conscious, and the associated ethical implications. Many other topics in philosophy are relevant to AI, such as epistemology and free will. Rapid advancements have intensified public discussions on the philosophy and ethics of AI.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0227", "text": "Defining artificial intelligence", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0228", "text": "Alan Turing wrote in 1950 \"I propose to consider the question 'can machines think'?\" He advised changing the question from whether a machine \"thinks\", to \"whether or not it is possible for machinery to show intelligent behaviour\". He devised the Turing test, which measures the ability of a machine to simulate human conversation. Since we can only observe the behavior of the machine, it does not matter if it is \"actually\" thinking or literally has a \"mind\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0229", "text": ". Turing notes that we can not determine these things about other people but \"it is usual to have a polite convention that everyone thinks.\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0230", "text": "Russell and Norvig agree with Turing that intelligence must be defined in terms of external behavior, not internal structure. However, they are critical that the test requires the machine to imitate humans", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0231", "text": ". \"Aeronautical engineering texts\", they wrote, \"do not define the goal of their field as making 'machines that fly so exactly like pigeons that they can fool other pigeons.'\" AI founder John McCarthy agreed, writing that \"Artificial intelligence is not, by definition, simulation of human intelligence\".", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0232", "text": "McCarthy defines intelligence as \"the computational part of the ability to achieve goals in the world\". Another AI founder, Marvin Minsky, similarly describes it as \"the ability to solve hard problems\". Artificial Intelligence: A Modern Approach defines it as the study of agents that perceive their environment and take actions that maximize their chances of achieving defined goals.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0233", "text": "The many differing definitions of AI have been critically analyzed. During the 2020s AI boom, the term has been used as a marketing buzzword to promote products and services which do not use AI.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0234", "text": "Legal definitions", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0235", "text": "The International Organization for Standardization describes an AI system as a \"an engineered system that generates outputs such as content, forecasts, recommendations, or decisions for a given set of human\u2011defined objectives, and can operate with varying levels of automation\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0236", "text": ". The EU AI Act defines an AI system as \"a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0237", "text": ". In the United States, influential but non\u2011binding guidance such as the National Institute of Standards and Technology's AI Risk Management Framework describes an AI system as \"an engineered or machine-based system that can, for a given set of objectives, generate outputs such as predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy\".", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0238", "text": "Evaluating approaches to AI", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0239", "text": "No established unifying theory or paradigm has guided AI research for most of its history. The unprecedented success of statistical machine learning in the 2010s eclipsed all other approaches (so much so that some sources, especially in the business world, use the term \"artificial intelligence\" to mean \"machine learning with neural networks\"). This approach is mostly sub-symbolic, soft and narrow", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0240", "text": ". Critics argue that these questions may have to be revisited by future generations of AI researchers.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0241", "text": "Symbolic AI and its limits\nSymbolic AI (or \"GOFAI\") simulated the high-level conscious reasoning that people use when they solve puzzles, express legal reasoning and do mathematics. They were highly successful at \"intelligent\" tasks such as algebra or IQ tests. In the 1960s, Newell and Simon proposed the physical symbol systems hypothesis: \"A physical symbol system has the necessary and sufficient means of general intelligent action.\"", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0242", "text": "However, the symbolic approach failed on many tasks that humans solve easily, such as learning, recognizing an object or commonsense reasoning. Moravec's paradox is the discovery that high-level \"intelligent\" tasks were easy for AI, but low level \"instinctive\" tasks were extremely difficult", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0243", "text": ". Philosopher Hubert Dreyfus had argued since the 1960s that human expertise depends on unconscious instinct rather than conscious symbol manipulation, and on having a \"feel\" for the situation, rather than explicit symbolic knowledge. Although his arguments had been ridiculed and ignored when they were first presented, eventually, AI research came to agree with him.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0244", "text": "The issue is not resolved: sub-symbolic reasoning can make many of the same inscrutable mistakes that human intuition does, such as algorithmic bias. Critics such as Noam Chomsky argue continuing research into symbolic AI will still be necessary to attain general intelligence, in part because sub-symbolic AI is a move away from explainable AI: it can be difficult or impossible to understand why a modern statistical AI program made a particular decision", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0245", "text": ". The emerging field of neuro-symbolic artificial intelligence attempts to bridge the two approaches.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0246", "text": "Neat vs. scruffy\n\"Neats\" hope that intelligent behavior is described using simple, elegant principles (such as logic, optimization, or neural networks). \"Scruffies\" expect that it necessarily requires solving a large number of unrelated problems. Neats defend their programs with theoretical rigor, scruffies rely mainly on incremental testing to see if they work. This issue was actively discussed in the 1970s and 1980s, but eventually was seen as irrelevant. Modern AI has elements of both.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0247", "text": "Soft vs. hard computing\nFinding a provably correct or optimal solution is intractable for many important problems. Soft computing is a set of techniques, including genetic algorithms, fuzzy logic and neural networks, that are tolerant of imprecision, uncertainty, partial truth and approximation. Soft computing was introduced in the late 1980s and most successful AI programs in the 21st century are examples of soft computing with neural networks.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0248", "text": "Narrow vs. general AI", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0249", "text": "AI researchers are divided as to whether to pursue the goals of artificial general intelligence and superintelligence directly or to solve as many specific problems as possible (narrow AI) in hopes these solutions will lead indirectly to the field's long-term goals. General intelligence is difficult to define and difficult to measure, and modern AI has had more verifiable successes by focusing on specific problems with specific solutions", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0250", "text": ". The sub-field of artificial general intelligence studies this area exclusively.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0251", "text": "Machine consciousness, sentience, and mind", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0252", "text": "There is no settled consensus in philosophy of mind on whether a machine can have a mind, consciousness and mental states in the same sense that human beings do. This issue considers the internal experiences of the machine, rather than its external behavior. Mainstream AI research considers this issue irrelevant because it does not affect the goals of the field: to build machines that can solve problems using intelligence", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0253", "text": ". Russell and Norvig add that \"[t]he additional project of making a machine conscious in exactly the way humans are is not one that we are equipped to take on.\" However, the question has become central to the philosophy of mind. It is also typically the central question at issue in artificial intelligence in fiction.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0254", "text": "Consciousness", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0255", "text": "David Chalmers identified two problems in understanding the mind, which he named the \"hard\" and \"easy\" problems of consciousness. The easy problem is understanding how the brain processes signals, makes plans and controls behavior. The hard problem is explaining how this feels or why it should feel like anything at all, assuming we are right in thinking that it truly does feel like something (Dennett's consciousness illusionism says this is an illusion)", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0256", "text": ". While human information processing is easy to explain, human subjective experience is difficult to explain. For example, it is easy to imagine a color-blind person who has learned to identify which objects in their field of view are red, but it is not clear what would be required for the person to know what red looks like.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0257", "text": "Computationalism and functionalism", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0258", "text": "Computationalism is the position in the philosophy of mind that the human mind is an information processing system and that thinking is a form of computing. Computationalism argues that the relationship between mind and body is similar or identical to the relationship between software and hardware and thus may be a solution to the mind\u2013body problem", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0259", "text": ". This philosophical position was inspired by the work of AI researchers and cognitive scientists in the 1960s and was originally proposed by philosophers Jerry Fodor and Hilary Putnam.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0260", "text": "Philosopher John Searle characterized this position as \"strong AI\": \"The appropriately programmed computer with the right inputs and outputs would thereby have a mind in exactly the same sense human beings have minds.\" Searle challenges this claim with his Chinese room argument, which attempts to show that even a computer capable of perfectly simulating human behavior would not have a mind.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0261", "text": "AI welfare and rights", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0262", "text": "It is difficult or impossible to reliably evaluate whether an advanced AI is sentient (has the ability to feel), and if so, to what degree. But if there is a significant chance that a given machine can feel and suffer, then it may be entitled to certain rights or welfare protection measures, similarly to animals. Sapience (a set of capacities related to high intelligence, such as discernment or self-awareness) may provide another moral basis for AI rights", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0263", "text": ". Robot rights are also sometimes proposed as a practical way to integrate autonomous agents into society.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0264", "text": "In 2017, the European Union considered granting \"electronic personhood\" to some of the most capable AI systems. Similarly to the legal status of companies, it would have conferred rights but also responsibilities. Critics argued in 2018 that granting rights to AI systems would downplay the importance of human rights, and that legislation should focus on user needs rather than speculative futuristic scenarios. They also noted that robots lacked the autonomy to take part in society on their own.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0265", "text": "Progress in AI increased interest in the topic. Proponents of AI welfare and rights often argue that AI sentience, if it emerges, would be particularly easy to deny. They warn that this may be a moral blind spot analogous to slavery or factory farming, which could lead to large-scale suffering if sentient AI is created and carelessly exploited.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0266", "text": "Future\nSuperintelligence and the singularity\nA superintelligence is a hypothetical agent that would possess intelligence far surpassing that of the brightest and most gifted human mind. If research into artificial general intelligence produced sufficiently intelligent software, it might be able to reprogram and improve itself. The improved software would be even better at improving itself, leading to what I. J. Good called an \"intelligence explosion\" and Vernor Vinge called a \"singularity\".", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0267", "text": "However, technologies cannot improve exponentially indefinitely, and typically follow an S-shaped curve, slowing when they reach the physical limits of what the technology can do.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0268", "text": "Transhumanism\nRobot designer Hans Moravec, cyberneticist Kevin Warwick and inventor Ray Kurzweil have predicted that humans and machines may merge in the future into cyborgs that are more capable and powerful than either. This idea, called transhumanism, has roots in the writings of Aldous Huxley and Robert Ettinger.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0269", "text": "Edward Fredkin argues that \"artificial intelligence is the next step in evolution\", an idea first proposed by Samuel Butler's \"Darwin among the Machines\" as far back as 1863, and expanded upon by George Dyson in his 1998 book Darwin Among the Machines: The Evolution of Global Intelligence.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0270", "text": "In fiction\nThought-capable artificial beings have appeared as storytelling devices since antiquity, and have been a persistent theme in science fiction.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0271", "text": "A common trope in these works began with Mary Shelley's Frankenstein, where a human creation becomes a threat to its masters. This includes such works as Arthur C. Clarke's and Stanley Kubrick's 2001: A Space Odyssey (both 1968), with HAL 9000, the murderous computer in charge of the Discovery One spaceship, as well as Blade Runner (1982), The Terminator (1984) and The Matrix (1999)", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0272", "text": ". In contrast, the rare loyal robots such as Gort from The Day the Earth Stood Still (1951) and Bishop from Aliens (1986) are less prominent in popular culture.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0273", "text": "Isaac Asimov introduced the Three Laws of Robotics in many stories, most notably with the \"Multivac\" super-intelligent computer. Asimov's laws are often brought up during lay discussions of machine ethics; while almost all artificial intelligence researchers are familiar with Asimov's laws through popular culture, they generally consider the laws useless for many reasons, one of which is their ambiguity.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0274", "text": "Several works use AI to force us to confront the fundamental question of what makes us human, showing us artificial beings that have the ability to feel, and thus to suffer. This appears in Karel \u010capek's R.U.R., the films A.I. Artificial Intelligence and Ex Machina, as well as the novel Do Androids Dream of Electric Sheep?, by Philip K. Dick. Dick considers the idea that our understanding of human subjectivity is altered by technology created with artificial intelligence.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Artificial_intelligence_0275", "text": "See also\nExplanatory notes\nReferences\nTextbooks\nHistory of AI\nOther sources\nExternal links\n\nHauser, Larry. \"Artificial Intelligence\". In Fieser, James; Dowden, Bradley (eds.). Internet Encyclopedia of Philosophy. ISSN 2161-0002. OCLC 37741658.", "title": "Artificial intelligence", "url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}, {"id": "rec_Machine_learning_0000", "text": "Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of statistical algorithms that can learn from data and generalize to unseen data, and thus perform tasks without explicit programming language instructions. Within a subdiscipline of machine learning, advances in the field of deep learning have allowed neural networks, a class of statistical algorithms, to surpass many previous machine learning approaches in performance.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0001", "text": "Statistics and mathematical optimisation (mathematical programming) methods compose the foundations of machine learning. Data mining is a related field of study, focusing on exploratory data analysis (EDA) through unsupervised learning.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0002", "text": "From a theoretical viewpoint, probably approximately correct learning provides a mathematical and statistical framework for describing machine learning. Most traditional machine learning and deep learning algorithms can be described as empirical risk minimisation under this framework.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0003", "text": "History\nThe term machine learning was coined in 1959 by Arthur Samuel, an IBM employee and pioneer in the field of computer gaming and artificial intelligence. The synonym self-teaching computers was also used during this time period.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0004", "text": "The earliest machine learning program was introduced in the 1950s when Arthur Samuel invented a computer program that calculated the winning chance in checkers for each side, but the history of machine learning roots back to decades of human desire and effort to study human cognitive processes. In 1949, Canadian psychologist Donald Hebb published the book The Organization of Behavior, in which he introduced a theoretical neural structure formed by certain interactions among nerve cells", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0005", "text": ". Hebb's model of neurons interacting with one another set a groundwork for how AIs and machine learning algorithms work under nodes, or artificial neurons used by computers to communicate data. Other researchers who have studied human cognitive systems contributed to the modern machine learning technologies as well, including logician Walter Pitts and Warren McCulloch, who proposed the early mathematical models of neural networks to come up with algorithms that mirror human thought processes.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0006", "text": "By the early 1960s, an experimental \"learning machine\" with punched tape memory, called Cybertron, had been developed by Raytheon Company to analyse sonar signals, electrocardiograms, and speech patterns using rudimentary reinforcement learning. It was repetitively \"trained\" by a human operator/teacher to recognise patterns and equipped with a \"goof\" button to cause it to reevaluate incorrect decisions", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0007", "text": ". A representative book on research into machine learning during the 1960s was Nils Nilsson's book on Learning Machines, dealing mostly with machine learning for pattern classification. Interest related to pattern recognition continued into the 1970s, as described by Duda and Hart in 1973. In 1981, a report was given on using teaching strategies so that an artificial neural network learns to recognise 40 characters (26 letters, 10 digits, and 4 special symbols) from a computer terminal.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0008", "text": "Tom M. Mitchell provided a widely quoted, more formal definition of the algorithms studied in the machine learning field: \"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P,  improves with experience E.\" This definition of the tasks in which machine learning is concerned offers a fundamentally operational definition rather than defining the field in cognitive terms", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0009", "text": ". This follows Alan Turing's proposal in his paper \"Computing Machinery and Intelligence\", in which the question, \"Can machines think?\", is replaced with the question, \"Can machines do what we (as thinking entities) can do?\".", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0010", "text": "Modern-day Machine Learning algorithms are broken into 3 algorithm types: Supervised Learning Algorithms, Unsupervised Learning Algorithms, and Reinforcement Learning Algorithms.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0011", "text": "Current Supervised Learning Algorithms have objectives of classification and regression.\nCurrent Unsupervised Learning Algorithms have objectives of clustering, dimensionality reduction, and association rule.\nCurrent Reinforcement Learning Algorithms focus on decisions that must be made with respect to some previous, unknown time and are broken down to either be studies of model-based methods or model-free methods.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0012", "text": "In 2014 Ian Goodfellow and others introduced generative adversarial networks (GANs) with realistic data synthesis. By 2016 AlphaGo obtained victory against top human players using reinforcement learning techniques.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0013", "text": "Relationships to other fields\nArtificial intelligence", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0014", "text": "As a scientific endeavour, machine learning grew out of the quest for artificial intelligence (AI). In the early days of AI as an academic discipline, some researchers were interested in having machines learn from data. They attempted to approach the problem with various symbolic methods, as well as what were then termed \"neural networks\"; these were mostly perceptrons and other models that were later found to be reinventions of the generalised linear models of statistics", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0015", "text": ". Probabilistic reasoning was also employed, especially in automated medical diagnosis.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0016", "text": "However, an increasing emphasis on the logical, knowledge-based approach caused a rift between AI and machine learning. Probabilistic systems were plagued by theoretical and practical problems of data acquisition and representation. By 1980, expert systems had come to dominate AI, and statistics was out of favour", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0017", "text": ". Work on symbolic/knowledge-based learning did continue within AI, leading to inductive logic programming(ILP), but the more statistical line of research was now outside the field of AI proper, in pattern recognition and information retrieval. Neural networks research had been abandoned by AI and computer science around the same time", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0018", "text": ". This line, too, was continued outside the AI/CS field, as \"connectionism\", by researchers from other disciplines, including John Hopfield, David Rumelhart, and Geoffrey Hinton. Their main success came in the mid-1980s with the reinvention of backpropagation.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0019", "text": "Machine learning (ML), reorganised and recognised as its own field, started to flourish in the 1990s. The field changed its goal from achieving artificial intelligence to tackling solvable problems of a practical nature. It shifted focus away from the symbolic approaches it had inherited from AI, and toward methods and models borrowed from statistics, fuzzy logic, and probability theory.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0020", "text": "Data compression\nData mining", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0021", "text": "Machine learning and data mining often employ the same methods and overlap significantly, but while machine learning focuses on prediction, based on known properties learned from the training data, data mining focuses on the discovery of (previously) unknown properties in the data (this is the analysis step of knowledge discovery in databases)", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0022", "text": ". Data mining uses many machine learning methods, but with different goals; on the other hand, machine learning also employs data mining methods as \"unsupervised learning\" or as a preprocessing step to improve learner accuracy", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0023", "text": ". Much of the confusion between these two research communities (which do often have separate conferences and separate journals, ECML PKDD being a major exception) comes from the basic assumptions they work with: in machine learning, performance is usually evaluated with respect to the ability to reproduce known knowledge, while in knowledge discovery and data mining (KDD) the key task is the discovery of previously unknown knowledge", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0024", "text": ". Evaluated with respect to known knowledge, an uninformed (unsupervised) method will easily be outperformed by other supervised methods, while in a typical KDD task, supervised methods cannot be used due to the unavailability of training data.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0025", "text": "Machine learning also has intimate ties to optimisation: Many learning problems are formulated as minimisation of some loss function on a training set of examples. Loss functions express the discrepancy between the predictions of the model being trained and the actual problem instances (for example, in classification, one wants to assign a label to instances, and models are trained to correctly predict the preassigned labels of a set of examples).", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0026", "text": "Generalization\nCharacterizing the generalisation of various learning algorithms is an active topic of current research, especially for deep learning algorithms.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0027", "text": "Statistics\nMachine learning and statistics are closely related fields in terms of methods, but distinct in their principal goal: statistics draws population inferences from a sample, while machine learning finds generalisable predictive patterns.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0028", "text": "Conventional statistical analyses require the a priori selection of a model most suitable for the study data set. In addition, only significant or theoretically relevant variables based on previous experience are included for analysis. In contrast, machine learning is not built on a pre-structured model; rather, the data shape the model by detecting underlying patterns. The more variables (input) used to train the model, the more accurate the ultimate model will be.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0029", "text": "Leo Breiman distinguished two statistical modelling paradigms: data model and algorithmic model, wherein \"algorithmic model\" means more or less the machine learning algorithms like Random Forest.\nSome statisticians have adopted methods from machine learning, leading to a combined field that they call statistical learning.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0030", "text": "Statistical physics\nAnalytical and computational techniques derived from deep-rooted physics of disordered systems can be extended to large-scale problems, including machine learning, e.g., to analyse the weight space of deep neural networks. Statistical physics is thus finding applications in the area of medical diagnostics.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0031", "text": "Theory", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0032", "text": "A core objective of a learner is to generalise from its experience. Generalisation in this context is the ability of a learning machine to perform accurately on new, unseen examples/tasks after having experienced a learning data set", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0033", "text": ". The training examples come from some generally unknown probability distribution (considered representative of the space of occurrences) and the learner has to build a general model about this space that enables it to produce sufficiently accurate predictions in new cases.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0034", "text": "The computational analysis of machine learning algorithms and their performance is a branch of theoretical computer science known as computational learning theory via the probably approximately correct learning  model. Because training sets are finite and the future is uncertain, learning theory usually does not yield guarantees of the performance of algorithms. Instead, probabilistic bounds on the performance are quite common", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0035", "text": ". The bias\u2013variance decomposition is one way to quantify generalisation error.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0036", "text": "For the best performance in the context of generalisation, the complexity of the hypothesis should match the complexity of the function underlying the data. If the hypothesis is less complex than the function, then the model has underfitted the data. If the complexity of the model is increased in response, then the training error decreases. But if the hypothesis is too complex, then the model is subject to overfitting and generalisation will be poorer.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0037", "text": "In addition to performance bounds, learning theorists study the time complexity and feasibility of learning. In computational learning theory, a computation is considered feasible if it can be done in polynomial time. There are two kinds of time complexity results: Positive results show that a certain class of functions can be learned in polynomial time. Negative results show that certain classes cannot be learned in polynomial time.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0038", "text": "Approaches\nMachine learning approaches are traditionally divided into three broad categories, which correspond to learning paradigms, depending on the nature of the \"signal\" or \"feedback\" available to the learning system:", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0039", "text": "Supervised learning: The computer is presented with example inputs and their desired outputs, given by a \"teacher\", and the goal is to learn a general rule that maps inputs to outputs.\nUnsupervised learning: No labels are given to the learning algorithm, leaving it on its own to find structure in its input. Unsupervised learning can be a goal in itself (discovering hidden patterns in data) or a means towards an end (feature learning).", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0040", "text": "Reinforcement learning: A computer program interacts with a dynamic environment in which it must perform a certain goal (such as driving a vehicle or playing a game against an opponent). As it navigates its problem space, the program is provided feedback that's analogous to rewards, which it tries to maximise.\nAlthough each algorithm has advantages and limitations, no single algorithm works for all problems.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0041", "text": "Supervised learning", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0042", "text": "Supervised learning algorithms build a mathematical model of a set of data that contains both the inputs and the desired outputs. The data, known as training data, consists of a set of training examples. Each training example has one or more inputs and the desired output, also known as a supervisory signal. In the mathematical model, each training example is represented by an array or vector, sometimes called a feature vector, and the training data is represented by a matrix", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0043", "text": ". Through iterative optimisation of an objective function, supervised learning algorithms learn a function that can be used to predict the output associated with new inputs. An optimal function allows the algorithm to correctly determine the output for inputs that were not a part of the training data. An algorithm that improves the accuracy of its outputs or predictions over time is said to have learned to perform that task.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0044", "text": "Types of supervised-learning algorithms include active learning, classification and regression. Classification algorithms are used when the outputs are restricted to a limited set of values, while regression algorithms are used when the outputs can take any numerical value within a range. For example, in a classification algorithm that filters emails, the input is an incoming email, and the output is the folder in which to file the email", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0045", "text": ". In contrast, regression is used for tasks such as predicting a person's height based on factors like age and genetics or forecasting future temperatures based on historical data.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0046", "text": "Similarity learning is an area of supervised machine learning closely related to regression and classification, but the goal is to learn from examples using a similarity function that measures how similar or related two objects are. It has applications in ranking, recommendation systems, visual identity tracking, face verification, and speaker verification.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0047", "text": "Unsupervised learning\nUnsupervised learning algorithms find structures in data that has not been labelled, classified or categorised. Instead of responding to feedback, unsupervised learning algorithms identify commonalities in the data and react based on the presence or absence of such commonalities in each new piece of data. Central applications of unsupervised machine learning include clustering, dimensionality reduction, and density estimation.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0048", "text": "Cluster analysis is the assignment of a set of observations into subsets (called clusters) so that observations within the same cluster are similar according to one or more predesignated criteria, while observations drawn from different clusters are dissimilar", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0049", "text": ". Different clustering techniques make different assumptions on the structure of the data, often defined by some similarity metric and evaluated, for example, by internal compactness, or the similarity between members of the same cluster, and separation, the difference between clusters. Other methods are based on estimated density and graph connectivity.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0050", "text": "A special type of unsupervised learning called self-supervised learning involves training a model by generating the supervisory signal from the data itself.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0051", "text": "Dimensionality reduction", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0052", "text": "Dimensionality reduction is a process of reducing the number of random variables under consideration by obtaining a set of principal variables. In other words, it is a process of reducing the dimension of the feature set, also called the \"number of features\". Most of the dimensionality reduction techniques can be considered as either feature elimination or extraction. One of the popular methods of dimensionality reduction is principal component analysis (PCA)", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0053", "text": ". PCA involves changing higher-dimensional data (e.g., 3D) to a smaller space (e.g., 2D).", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0054", "text": "The manifold hypothesis proposes that high-dimensional data sets lie along low-dimensional manifolds, and many dimensionality reduction techniques make this assumption, leading to the areas of manifold learning and manifold regularisation.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0055", "text": "Semi-supervised learning\nSemi-supervised learning falls between unsupervised learning (without any labelled training data) and supervised learning (with completely labelled training data). Some of the training examples are missing training labels, yet many machine-learning researchers have found that unlabelled data, when used in conjunction with a small amount of labelled data, can produce a considerable improvement in learning accuracy.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0056", "text": "In weakly supervised learning, the training labels are noisy, limited, or imprecise; however, these labels are often cheaper to obtain, resulting in larger effective training sets.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0057", "text": "Reinforcement learning", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0058", "text": "Reinforcement learning is an area of machine learning concerned with how software agents ought to take actions in an environment to maximise some notion of cumulative reward. Due to its generality, the field is studied in many other disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimisation, multi-agent systems, swarm intelligence, statistics and genetic algorithms", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0059", "text": ". In reinforcement learning, the environment is typically represented as a Markov decision process (MDP). Many reinforcement learning algorithms use dynamic programming techniques. Reinforcement learning algorithms do not assume knowledge of an exact mathematical model of the MDP and are used when exact models are infeasible. Reinforcement learning algorithms are used in autonomous vehicles or in learning to play a game against a human opponent.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0060", "text": "Other types\nOther approaches have been developed which do not fit neatly into this three-fold categorisation, and sometimes more than one is used by the same machine learning system. For example, topic modelling, meta-learning.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0061", "text": "Self-learning", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0062", "text": "Self-learning, as a machine learning paradigm, was introduced in 1982 along with a neural network capable of self-learning, named crossbar adaptive array (CAA). It gives a solution to the problem learning without any external reward, by introducing emotion as an internal reward. Emotion is used as a state evaluation of a self-learning agent. The CAA self-learning algorithm computes, in a crossbar fashion, both decisions about actions and emotions (feelings) about consequence situations", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0063", "text": ". The system is driven by the interaction between cognition and emotion.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0064", "text": "The self-learning algorithm updates a memory matrix W =||w(a,s)|| such that in each iteration executes the following machine learning routine:", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0065", "text": "in situation s act a\nreceive a consequence situation s'\ncompute emotion of being in the consequence situation v(s')\nupdate crossbar memory  w'(a,s) = w(a,s) + v(s')", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0066", "text": "It is a system with only one input, situation, and only one output, action (or behaviour) a. There is neither a separate reinforcement input nor an advice input from the environment. The backpropagated value (secondary reinforcement) is the emotion toward the consequence situation", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0067", "text": ". The CAA exists in two environments, one is the behavioural environment where it behaves, and the other is the genetic environment, wherefrom it initially and only once receives initial emotions about situations to be encountered in the behavioural environment. After receiving the genome (species) vector from the genetic environment, the CAA learns a goal-seeking behaviour in an environment that contains both desirable and undesirable situations.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0068", "text": "Feature learning", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0069", "text": "Several learning algorithms aim at discovering better representations of the inputs provided during training. Classic examples include principal component analysis and cluster analysis. Feature learning algorithms, also called representation learning algorithms, often attempt to preserve the information in their input but also transform it in a way that makes it useful, often as a pre-processing step before performing classification or predictions", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0070", "text": ". This technique allows reconstruction of the inputs coming from the unknown data-generating distribution, while not being necessarily faithful to configurations that are implausible under that distribution. This replaces manual feature engineering, and allows a machine to both learn the features and use them to perform a specific task.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0071", "text": "Feature learning can be either supervised or unsupervised. In supervised feature learning, features are learned using labelled input data. Examples include artificial neural networks, multilayer perceptrons, and supervised dictionary learning. In unsupervised feature learning, features are learned with unlabelled input data.  Examples include dictionary learning, independent component analysis, autoencoders, matrix factorisation and various forms of clustering.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0072", "text": "Manifold learning algorithms attempt to do so under the constraint that the learned representation is low-dimensional. Sparse coding algorithms attempt to do so under the constraint that the learned representation is sparse, meaning that the mathematical model has many zeros. Multilinear subspace learning algorithms aim to learn low-dimensional representations directly from tensor representations for multidimensional data, without reshaping them into higher-dimensional vectors", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0073", "text": ". Deep learning algorithms discover multiple levels of representation, or a hierarchy of features, with higher-level, more abstract features defined in terms of (or generating) lower-level features. It has been argued that an intelligent machine learns a representation that disentangles the underlying factors of variation that explain the observed data.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0074", "text": "Feature learning is motivated by the fact that machine learning tasks such as classification often require input that is mathematically and computationally convenient to process. However, real-world data such as images, video, and sensory data have not yielded attempts to algorithmically define specific features. An alternative is to discover such features or representations through examination, without relying on explicit algorithms.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0075", "text": "Sparse dictionary learning", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0076", "text": "Sparse dictionary learning is a feature learning method where a training example is represented as a linear combination of basis functions and assumed to be a sparse matrix. The method is strongly NP-hard and difficult to solve approximately. A popular heuristic method for sparse dictionary learning is the k-SVD algorithm. Sparse dictionary learning has been applied in several contexts", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0077", "text": ". In classification, the problem is to determine the class to which a previously unseen training example belongs. For a dictionary where each class has already been built, a new training example is associated with the class that is best sparsely represented by the corresponding dictionary. Sparse dictionary learning has also been applied in image denoising. The key idea is that a clean image patch can be sparsely represented by an image dictionary, but the noise cannot.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0078", "text": "Anomaly detection\nIn data mining, anomaly detection, also known as outlier detection, is the identification of rare items, events or observations that raise suspicions by differing significantly from the majority of the data. Typically, the anomalous items represent an issue such as bank fraud, a structural defect, medical problems or errors in a text. Anomalies are referred to as outliers, novelties, noise, deviations and exceptions.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0079", "text": "In particular, in the context of abuse and network intrusion detection, the interesting objects are often not rare, but unexpected bursts of inactivity. This pattern does not adhere to the common statistical definition of an outlier as a rare object. Many outlier detection methods (in particular, unsupervised algorithms) will fail on such data unless aggregated appropriately. Instead, a cluster analysis algorithm may be able to detect the micro-clusters formed by these patterns.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0080", "text": "Three broad categories of anomaly detection techniques exist. Unsupervised anomaly detection techniques detect anomalies in an unlabelled test data set under the assumption that the majority of the instances in the data set are normal, by looking for instances that seem to fit the least to the remainder of the data set", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0081", "text": ". Supervised anomaly detection techniques require a data set that has been labelled as \"normal\" and \"abnormal\" and involves training a classifier (the key difference from many other statistical classification problems is the inherently unbalanced nature of outlier detection). Semi-supervised anomaly detection techniques construct a model representing normal behaviour from a given normal training data set and then test the likelihood of a test instance being generated by the model.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0082", "text": "Robot learning\nRobot learning is inspired by a multitude of machine learning methods, starting from supervised learning, reinforcement learning, and finally meta-learning (e.g. MAML).", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0083", "text": "Association rules\nAssociation rule learning is a rule-based machine learning method for discovering relationships between variables in large databases. It is intended to identify strong rules discovered in databases using some measure of \"interestingness\".", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0084", "text": "Rule-based machine learning is a general term for any machine learning method that identifies, learns, or evolves \"rules\" to store, manipulate or apply knowledge. The defining characteristic of a rule-based machine learning algorithm is the identification and utilisation of a set of relational rules that collectively represent the knowledge captured by the system", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0085", "text": ". This is in contrast to other machine learning algorithms that commonly identify a singular model that can be universally applied to any instance in order to make a prediction. Rule-based machine learning approaches include learning classifier systems, association rule learning, and artificial immune systems.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0086", "text": "Based on the concept of strong rules, Rakesh Agrawal, Tomasz Imieli\u0144ski and Arun Swami introduced association rules for discovering regularities between products in large-scale transaction data recorded by point-of-sale (POS) systems in supermarkets. For example, the rule \n  \n    \n      \n        {\n        \n          o\n          n\n          i\n          o\n          n\n          s\n          ,\n          p\n          o\n          t\n          a\n          t\n          o\n          e\n          s", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0087", "text": "o\n          e\n          s\n        \n        }\n        \u21d2\n        {\n        \n          b\n          u\n          r\n          g\n          e\n          r\n        \n        }\n      \n    \n    {\\displaystyle \\{\\mathrm {onions,potatoes} \\}\\Rightarrow \\{\\mathrm {burger} \\}}", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0088", "text": "found in the sales data of a supermarket would indicate that if a customer buys onions and potatoes together, they are likely to also buy hamburger meat. Such information can be used as the basis for decisions about marketing activities such as promotional pricing or product placements. In addition to market basket analysis, association rules are employed today in application areas including Web usage mining, intrusion detection, continuous production, and bioinformatics", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0089", "text": ". In contrast with sequence mining, association rule learning typically does not consider the order of items either within a transaction or across transactions.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0090", "text": "Learning classifier systems (LCS) are a family of rule-based machine learning algorithms that combine a discovery component, typically a genetic algorithm, with a learning component, performing either supervised learning, reinforcement learning, or unsupervised learning. They seek to identify a set of context-dependent rules that collectively store and apply knowledge in a piecewise manner to make predictions.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0091", "text": "Inductive logic programming (ILP) is an approach to rule learning using logic programming as a uniform representation for input examples, background knowledge, and hypotheses. Given an encoding of the known background knowledge and a set of examples represented as a logical database of facts, an ILP system will derive a hypothesized logic program that entails all positive and no negative examples", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0092", "text": ". Inductive programming is a related field that considers any kind of programming language for representing hypotheses (and not only logic programming), such as functional programs.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0093", "text": "Inductive logic programming is particularly useful in bioinformatics and natural language processing. Gordon Plotkin and Ehud Shapiro laid the initial theoretical foundation for inductive machine learning in a logical setting. Shapiro built their first implementation (Model Inference System) in 1981: a Prolog program that inductively inferred logic programs from positive and negative examples", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0094", "text": ". The term inductive here refers to philosophical induction, suggesting a theory to explain observed facts, rather than mathematical induction, proving a property for all members of a well-ordered set.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0095", "text": "Models", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0096", "text": "A machine learning model is a type of mathematical model that, once \"trained\" on a given dataset, can be used to make predictions or classifications on new data. During training, a learning algorithm iteratively adjusts the model's internal parameters to minimise errors in its predictions", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0097", "text": ". By extension, the term \"model\" can refer to several levels of specificity, from a general class of models and their associated learning algorithms to a fully trained model with all its internal parameters tuned.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0098", "text": "Various types of models have been used and researched for machine learning systems, picking the best model for a task is called model selection.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0099", "text": "Artificial neural networks\nArtificial neural networks (ANNs), or connectionist systems, are computing systems vaguely inspired by the biological neural networks that constitute animal brains. Such systems \"learn\" to perform tasks by considering examples, generally without being programmed with any task-specific rules.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0100", "text": "An ANN is a model based on a collection of connected units or nodes called \"artificial neurons\", which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, can transmit information, a \"signal\", from one artificial neuron to another. An artificial neuron that receives a signal can process it and then signal additional artificial neurons connected to it", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0101", "text": ". In common ANN implementations, the signal at a connection between artificial neurons is a real number, and the output of each artificial neuron is computed by some non-linear function of the sum of its inputs. The connections between artificial neurons are called \"edges\". Artificial neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0102", "text": ". Artificial neurons may have a threshold such that the signal is only sent if the aggregate signal crosses that threshold. Typically, artificial neurons are aggregated into layers. Different layers may perform different kinds of transformations on their inputs. Signals travel from the first layer (the input layer) to the last layer (the output layer), possibly after traversing the layers multiple times.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0103", "text": "The original goal of the ANN approach was to solve problems in the same way that a human brain would. However, over time, attention moved to performing specific tasks, leading to deviations from biology. Artificial neural networks have been used on a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games and medical diagnosis.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0104", "text": "Deep learning consists of multiple hidden layers in an artificial neural network. This approach tries to model the way the human brain processes light and sound into vision and hearing. Some successful applications of deep learning are computer vision and speech recognition.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0105", "text": "Decision trees", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0106", "text": "Decision tree learning uses a decision tree as a predictive model to go from observations about an item (represented in the branches) to conclusions about the item's target value (represented in the leaves). It is one of the predictive modelling approaches used in statistics, data mining, and machine learning", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0107", "text": ". Tree models where the target variable can take a discrete set of values are called classification trees; in these tree structures, leaves represent class labels, and branches represent conjunctions of features that lead to those class labels. Decision trees where the target variable can take continuous values (typically real numbers) are called regression trees. In decision analysis, a decision tree can be used to visually and explicitly represent decisions and decision making", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0108", "text": ". In data mining, a decision tree describes data, but the resulting classification tree can be an input for decision-making.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0109", "text": "Random forest regression", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0110", "text": "Random forest regression (RFR) falls under the umbrella of decision tree-based models. RFR is an ensemble learning method that builds multiple decision trees and averages their predictions to improve accuracy and to avoid overfitting. To build decision trees, RFR uses bootstrapped sampling; for instance, each decision tree is trained on random data from the training set", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0111", "text": ". This random selection of RFR for training enables the model to reduce biased predictions and achieve a higher degree of accuracy. RFR generates independent decision trees, and it can work on single-output data as well as multiple regressor tasks. This makes RFR compatible to be use in various applications.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0112", "text": "Support-vector machines", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0113", "text": "Support-vector machines (SVMs), also known as support-vector networks, are a set of related supervised learning methods used for classification and regression. Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that predicts whether a new example falls into one category", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0114", "text": ". An SVM training algorithm is a non-probabilistic, binary, linear classifier, although methods such as Platt scaling exist to use SVM in a probabilistic classification setting. In addition to performing linear classification, SVMs can efficiently perform a non-linear classification using what is called the kernel trick, implicitly mapping their inputs into high-dimensional feature spaces.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0115", "text": "Regression analysis", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0116", "text": "Regression analysis encompasses a large variety of statistical methods to estimate the relationship between input variables and their associated features. Its most common form is linear regression, where a single line is drawn to best fit the given data according to a mathematical criterion such as ordinary least squares. The latter is often extended by regularisation methods to mitigate overfitting and bias, as in ridge regression", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0117", "text": ". When dealing with non-linear problems, go-to models include polynomial regression (for example, used for trendline fitting in Microsoft Excel), logistic regression (often used in statistical classification) or even kernel regression, which introduces non-linearity by taking advantage of the kernel trick to implicitly map input variables to higher-dimensional space.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0118", "text": "Multivariate linear regression extends the concept of linear regression to handle multiple dependent variables simultaneously. This approach estimates the relationships between a set of input variables and several output variables by fitting a multidimensional linear model. It is particularly useful in scenarios where outputs are interdependent or share underlying patterns, such as predicting multiple economic indicators or reconstructing images, which are inherently multi-dimensional.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0119", "text": "Bayesian networks", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0120", "text": "A Bayesian network, belief network, or directed acyclic graphical model is a probabilistic graphical model that represents a set of random variables and their conditional independence with a directed acyclic graph (DAG). For example, a Bayesian network could represent the probabilistic relationships between diseases and symptoms. Given symptoms, the network can be used to compute the probabilities of the presence of various diseases", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0121", "text": ". Efficient algorithms exist that perform inference and learning. Bayesian networks that model sequences of variables, like speech signals or protein sequences, are called dynamic Bayesian networks. Generalisations of Bayesian networks that can represent and solve decision problems under uncertainty are called influence diagrams.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0122", "text": "Gaussian processes\nA Gaussian process is a stochastic process in which every finite collection of the random variables in the process has a multivariate normal distribution, and it relies on a pre-defined covariance function, or kernel, that models how pairs of points relate to each other depending on their locations.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0123", "text": "Given a set of observed points, or input\u2013output examples, the distribution of the (unobserved) output of a new point as a function of its input data can be directly computed by looking at the observed points and the covariances between those points and the new, unobserved point.\nGaussian processes are popular surrogate models in Bayesian optimisation used to do hyperparameter optimisation.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0124", "text": "Genetic algorithms\nA genetic algorithm (GA) is a search algorithm and heuristic technique that mimics the process of natural selection, using methods such as mutation and crossover to generate new genotypes in the hope of finding good solutions to a given problem. In machine learning, genetic algorithms were used in the 1980s and 1990s. Conversely, machine learning techniques have been used to improve the performance of genetic and evolutionary algorithms.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0125", "text": "Belief functions", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0126", "text": "The theory of belief functions, also referred to as evidence theory or Dempster\u2013Shafer theory, is a general framework for reasoning with uncertainty, with understood connections to other frameworks such as probability, possibility and  imprecise probability theories", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0127", "text": ". These theoretical frameworks can be thought of as a kind of learner and have some analogous properties of how evidence is combined (e.g.,  Dempster's rule of combination), just like how in a pmf-based Bayesian approach would combine probabilities. However, there are many caveats to these beliefs functions when compared to Bayesian approaches to incorporate ignorance and uncertainty quantification", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0128", "text": ". These belief function approaches that are implemented within the machine learning domain typically leverage a fusion approach of various ensemble methods to better handle the learner's decision boundary, low samples, and ambiguous class issues that standard machine learning approach tend to have difficulty resolving", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0129", "text": ". However, the computational complexity of these algorithms is dependent on the number of propositions (classes), and can lead to a much higher computation time when compared to other machine learning approaches.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0130", "text": "Rule-based models\nRule-based machine learning (RBML) is a branch of machine learning that automatically discovers and learns 'rules' from data. It provides interpretable models, making it useful for decision-making in fields like healthcare, fraud detection, and cybersecurity. Key RBML techniques includes learning classifier systems, association rule learning, artificial immune systems, and other similar models. These methods extract patterns from data and evolve rules over time.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0131", "text": "Training models", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0132", "text": "Typically, machine learning models require a high quantity of reliable data to perform accurate predictions. When training a machine learning model, machine learning engineers need to target and collect a large and representative sample of data. Data from the training set can be as varied as a corpus of text, a collection of images, sensor data, and data collected from individual users of a service. Overfitting is something to watch out for when training a machine learning model", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0133", "text": ". Trained models derived from biased or non-evaluated data can result in skewed or undesired predictions. Biased models may result in detrimental outcomes, thereby furthering the negative impacts on society or objectives. Algorithmic bias is a potential result of data not being fully prepared for training. Machine learning ethics is becoming a field of study and, notably, becoming integrated within machine learning engineering teams.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0134", "text": "Federated learning", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0135", "text": "Federated learning is an adapted form of distributed artificial intelligence to train machine learning models that decentralises the training process, allowing for users' privacy to be maintained by not needing to send their data to a centralised server. This also increases efficiency by decentralising the training process to many devices", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0136", "text": ". For example, Gboard uses federated machine learning to train search query prediction models on users' mobile phones without having to send individual searches back to Google.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0137", "text": "Applications\nThere are many applications for machine learning, including:", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0138", "text": "In 2006, the media-services provider Netflix held the first \"Netflix Prize\" competition to find a program to better predict user preferences and improve the accuracy of its existing Cinematch movie recommendation algorithm by at least 10%. A joint team made up of researchers from AT&T Labs-Research in collaboration with the teams Big Chaos and Pragmatic Theory built an ensemble model to win the Grand Prize in 2009 for $1 million", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0139", "text": ". Shortly after the prize was awarded, Netflix realised that viewers' ratings were not the best indicators of their viewing patterns (\"everything is a recommendation\") and they changed their recommendation engine accordingly. In 2010, an article in The Wall Street Journal noted the use of machine learning by Rebellion Research to predict the 2008 financial crisis", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0140", "text": ". In 2012, co-founder of Sun Microsystems, Vinod Khosla, predicted that 80% of medical doctors jobs would be lost in the next two decades to automated machine learning medical diagnostic software. In 2014, it was reported that a machine learning algorithm had been applied in the field of art history to study fine art paintings and that it may have revealed previously unrecognised influences among artists. In 2019 Springer Nature published the first research book created using machine learning", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0141", "text": ". In 2020, machine learning technology was used to help make diagnoses and aid researchers in developing a cure for COVID-19. Machine learning was recently applied to predict the pro-environmental behaviour of travellers. Recently, machine learning technology was also applied to optimise smartphone's performance and thermal behaviour based on the user's interaction with the phone", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0142", "text": ". When applied correctly, machine learning algorithms (MLAs) can utilise a wide range of company characteristics to predict stock returns without overfitting. By employing effective feature engineering and combining forecasts, MLAs can generate results that far surpass those obtained from basic linear techniques like OLS.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0143", "text": "Recent advancements in machine learning have extended into the field of quantum chemistry, where novel algorithms now enable the prediction of solvent effects on chemical reactions, thereby offering new tools for chemists to tailor experimental conditions for optimal outcomes.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0144", "text": "Machine Learning is becoming a useful tool to investigate and predict evacuation decision-making in large-scale and small-scale disasters. Different solutions have been tested to predict if and when householders decide to evacuate during wildfires and hurricanes. Other applications have been focusing on pre evacuation decisions in building fires.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0145", "text": "Limitations\nAlthough machine learning has been transformative in some fields, machine-learning programs often fail to deliver expected results. Reasons for this are numerous: lack of (suitable) data, lack of access to the data, data bias, privacy problems, badly chosen tasks and algorithms, wrong tools and people, lack of resources, and evaluation problems.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0146", "text": "The \"black box theory\" poses another yet significant challenge. Black box refers to a situation where the algorithm or the process of producing an output is entirely opaque, meaning that even the coders of the algorithm cannot audit the pattern that the machine extracted from the data", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0147", "text": ". The House of Lords Select Committee, which claimed that such an \"intelligence system\" that could have a \"substantial impact on an individual's life\" would not be considered acceptable unless it provided \"a full and satisfactory explanation for the decisions\" it makes.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0148", "text": "In 2018, a self-driving car from Uber failed to detect a pedestrian, who was killed after a collision. Attempts to use machine learning in healthcare with the IBM Watson system failed to deliver even after years of time and billions of dollars invested. Microsoft's Bing Chat chatbot has been reported to produce hostile and offensive response against its users.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0149", "text": "Machine learning has been used as a strategy to update the evidence related to a systematic review and increased reviewer burden related to the growth of biomedical literature. While it has improved with training sets, it has not yet developed sufficiently to reduce the workload burden without limiting the necessary sensitivity for the findings research itself.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0150", "text": "Explainability", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0151", "text": "Explainable AI (XAI), or Interpretable AI, or Explainable Machine Learning (XML), is artificial intelligence (AI) in which humans can understand the decisions or predictions made by the AI. It contrasts with the \"black box\" concept in machine learning where even its designers cannot explain why an AI arrived at a specific decision. By refining the mental models of users of AI-powered systems and dismantling their misconceptions, XAI promises to help users perform more effectively", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0152", "text": ". XAI may be an implementation of the social right to explanation.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0153", "text": "Overfitting\nSettling on a bad, overly complex theory gerrymandered to fit all the past training data is known as overfitting. Many systems attempt to reduce overfitting by rewarding a theory in accordance with how well it fits the data but penalising the theory in accordance with how complex the theory is.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0154", "text": "Other limitations and vulnerabilities", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0155", "text": "Learners can also be disappointed by \"learning the wrong lesson\". A toy example is that an image classifier trained only on pictures of brown horses and black cats might conclude that all brown patches are likely to be horses", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0156", "text": ". A real-world example is that, unlike humans, current image classifiers often do not primarily make judgments from the spatial relationship between components of the picture, and they learn relationships between pixels that humans are oblivious to, but that still correlate with images of certain types of real objects. Modifying these patterns on a legitimate image can result in \"adversarial\" images that the system misclassifies.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0157", "text": "Adversarial vulnerabilities can also result in nonlinear systems or from non-pattern perturbations. For some systems, it is possible to change the output by only changing a single adversarially chosen pixel. Machine learning models are often vulnerable to manipulation or evasion via adversarial machine learning.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0158", "text": "Researchers have demonstrated how backdoors can be placed undetectably into classifying (e.g., for categories \"spam\" and \"not spam\" of posts) machine learning models that are often developed or trained by third parties. Parties can change the classification of any input, including in cases for which a type of data/software transparency is provided, possibly including white-box access.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0159", "text": "Model assessments", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0160", "text": "Classification of machine learning models can be validated by accuracy estimation techniques like the holdout method, which splits the data into a training and test set (conventionally 2/3 training set and 1/3 test set designation) and evaluates the performance of the training model on the test set", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0161", "text": ". In comparison, the K-fold-cross-validation method randomly partitions the data into K subsets and then K experiments are performed each considering 1 subset for evaluation and the remaining K-1 subsets for training the model. In addition to the holdout and cross-validation methods, bootstrap, which samples n instances with replacement from the dataset, can be used to assess model accuracy.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0162", "text": "In addition to overall accuracy, investigators frequently report sensitivity and specificity, meaning true positive rate (TPR) and true negative rate (TNR), respectively. Similarly, investigators sometimes report the false positive rate (FPR) as well as the false negative rate (FNR). However, these rates are ratios that fail to reveal their numerators and denominators", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0163", "text": ". Receiver operating characteristic (ROC), along with the accompanying Area Under the ROC Curve (AUC), offer additional tools for classification model assessment. Higher AUC is associated with a better performing model.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0164", "text": "Ethics\nBias\nDifferent machine learning approaches can suffer from different data biases. A machine learning system trained specifically on current customers may not be able to predict the needs of new customer groups that are not represented in the training data. When trained on human-made data, machine learning is likely to pick up the constitutional and unconscious biases already present in society.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0165", "text": "Systems that are trained on datasets collected with biases may exhibit these biases upon use (algorithmic bias), thus digitising cultural prejudices. For example, in 1988, the UK's Commission for Racial Equality found that St. George's Medical School had been using a computer program trained from data of previous admissions staff and this program had denied nearly 60 candidates who were found to either be women or have non-European-sounding names", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0166", "text": ". Using job hiring data from a firm with racist hiring policies may lead to a machine learning system duplicating the bias by scoring job applicants by similarity to previous successful applicants. Another example includes predictive policing company Geolitica's predictive algorithm that resulted in \"disproportionately high levels of over-policing in low-income and minority communities\" after being trained with historical crime data.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0167", "text": "While responsible collection of data and documentation of algorithmic rules used by a system is considered a critical part of machine learning, some researchers blame the lack of participation and representation of minority populations in the field of AI for machine learning's vulnerability to biases", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0168", "text": ". In fact, according to research carried out by the Computing Research Association in 2021, \"female faculty make up just 16.1%\" of all faculty members who focus on AI among several universities around the world. Furthermore, among the group of \"new U.S. resident AI PhD graduates,\" 45% identified as white, 22.4% as Asian, 3.2% as Hispanic, and 2.4% as African American, which further demonstrates a lack of diversity in the field of AI.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0169", "text": "Language models learned from data have been shown to contain human-like biases. Because human languages contain biases, machines trained on language corpora will necessarily also learn these biases. In 2016, Microsoft tested Tay, a chatbot that learned from Twitter, and it quickly picked up racist and sexist language.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0170", "text": "In an experiment carried out by ProPublica, an investigative journalism organisation, a machine learning algorithm's insight into the recidivism rates among prisoners falsely flagged \"black defendants high risk twice as often as white defendants\". In 2015, Google Photos once tagged a couple of black people as gorillas, which caused controversy. The gorilla label was subsequently removed, and in 2023, it still cannot recognise gorillas", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0171", "text": ". Similar issues with recognising non-white people have been found in many other systems.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0172", "text": "Because of such challenges, the effective use of machine learning may take longer to be adopted in other domains. Concern for fairness in machine learning, that is, reducing bias in machine learning and propelling its use for human good, is increasingly expressed by artificial intelligence scientists, including Fei-Fei Li, who said that \"[t]here's nothing artificial about AI. It's inspired by people, it's created by people, and\u2014most importantly\u2014it impacts people", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0173", "text": ". It is a powerful tool we are only just beginning to understand, and that is a profound responsibility.\"", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0174", "text": "Financial incentives", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0175", "text": "There are concerns among health care professionals that these systems might not be designed in the public's interest but as income-generating machines. This is especially true in the United States, where there is a long-standing ethical dilemma of improving health care, but also increasing profits. For example, the algorithms could be designed to provide patients with unnecessary tests or medication in which the algorithm's proprietary owners hold stakes", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0176", "text": ". There is potential for machine learning in health care to provide professionals with an additional tool to diagnose, medicate, and plan recovery paths for patients, but this requires these biases to be mitigated.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0177", "text": "Hardware", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0178", "text": "Since the 2010s, advances in both machine learning algorithms and computer hardware have led to more efficient methods for training deep neural networks (a particular narrow subdomain of machine learning) that contain many layers of nonlinear hidden units. By 2019, graphics processing units (GPUs), often with AI-specific enhancements, had displaced CPUs as the dominant method of training large-scale commercial cloud AI", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0179", "text": ". OpenAI estimated the hardware compute used in the largest deep learning projects from AlexNet (2012) to AlphaZero (2017), and found a 300,000-fold increase in the amount of compute required, with a doubling-time trendline of 3.4 months.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0180", "text": "Tensor Processing Units (TPUs)", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0181", "text": "Tensor Processing Units (TPUs) are specialised hardware accelerators developed by Google specifically for machine learning workloads. Unlike general-purpose GPUs and FPGAs, TPUs are optimised for tensor computations, making them particularly efficient for deep learning tasks such as training and inference. They are widely used in Google Cloud AI services and large-scale machine learning models like Google's DeepMind AlphaFold and large language models", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0182", "text": ". TPUs leverage matrix multiplication units and high-bandwidth memory to accelerate computations while maintaining energy efficiency. Since their introduction in 2016, TPUs have become a key component of AI infrastructure, especially in cloud-based environments.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0183", "text": "Neuromorphic computing\nNeuromorphic computing refers to a class of computing systems designed to emulate the structure and functionality of biological neural networks. These systems may be implemented through software-based simulations on conventional hardware or through specialised hardware architectures.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0184", "text": "Physical neural networks\nA physical neural network is a specific type of neuromorphic hardware that relies on electrically adjustable materials, such as memristors, to emulate the function of neural synapses. The term \"physical neural network\" highlights the use of physical hardware for computation, as opposed to software-based implementations. It broadly refers to artificial neural networks that use materials with adjustable resistance to replicate neural synapses.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0185", "text": "Embedded machine learning", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0186", "text": "Embedded machine learning is a sub-field of machine learning where models are deployed on embedded systems with limited computing resources, such as wearable computers, edge devices and microcontrollers. Running models directly on these devices eliminates the need to transfer and store data on cloud servers for further processing, thereby reducing the risk of data breaches, privacy leaks and theft of intellectual property, personal data and business secrets", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0187", "text": ". Embedded machine learning can be achieved through various techniques, such as hardware acceleration, approximate computing, and model optimisation. Common optimisation techniques include pruning, quantisation, knowledge distillation, low-rank factorisation, network architecture search, and parameter sharing.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0188", "text": "Software\nSoftware suites containing a variety of machine learning algorithms include the following:\n\nFree and open-source software\nProprietary software with free and open-source editions\nKNIME\nRapidMiner\n\nProprietary software\nJournals\nJournal of Machine Learning Research\nMachine Learning\nNature Machine Intelligence\nNeural Computation\nIEEE Transactions on Pattern Analysis and Machine Intelligence", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0189", "text": "Conferences\nAAAI Conference on Artificial Intelligence\nAssociation for Computational Linguistics (ACL)\nEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD)\nInternational Conference on Computational Intelligence Methods for Bioinformatics and Biostatistics (CIBB)\nInternational Conference on Machine Learning (ICML)\nInternational Conference on Learning Representations (ICLR)", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0190", "text": "International Conference on Intelligent Robots and Systems (IROS)\nConference on Knowledge Discovery and Data Mining (KDD)\nConference on Neural Information Processing Systems (NeurIPS)", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0191", "text": "See also\nAutomated machine learning \u2013 Process of automating the application of machine learning\nBig data \u2013 Extremely large or complex datasets\nDeep learning \u2014 branch of ML concerned with artificial neural networks\nDifferentiable programming \u2013 Programming paradigm\nList of datasets for machine-learning research\nList of machine learning algorithms and List of algorithms for machine learning and statistical classification\nM-theory (learning framework) \u2013 Framework in machine learning", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0192", "text": "Machine unlearning \u2013 Field of study in artificial intelligence\nOutline of machine learning\nSolomonoff's theory of inductive inference \u2013 Mathematical theory", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0193", "text": "References\nSources\nDomingos, Pedro (22 September 2015). The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World. Basic Books. ISBN 978-0-465-06570-7.\nNilsson, Nils (1998). Artificial Intelligence: A New Synthesis. Morgan Kaufmann. ISBN 978-1-55860-467-4. Archived from the original on 26 July 2020. Retrieved 18 November 2019.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0194", "text": "Poole, David; Mackworth, Alan; Goebel, Randy (1998). Computational Intelligence: A Logical Approach. New York: Oxford University Press. ISBN 978-0-19-510270-3. Archived from the original on 26 July 2020. Retrieved 22 August 2020.\nRussell, Stuart J.; Norvig, Peter (2003), Artificial Intelligence: A Modern Approach (2nd ed.), Upper Saddle River, New Jersey: Prentice Hall, ISBN 0-13-790395-2.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Machine_learning_0195", "text": "Further reading\nExternal links\nInternational Machine Learning Society\nmloss is an academic database of open-source machine learning software.", "title": "Machine learning", "url": "https://en.wikipedia.org/wiki/Machine_learning"}, {"id": "rec_Deep_learning_0000", "text": "In machine learning, deep learning (DL) focuses on utilizing multilayered neural networks to perform tasks such as classification, regression, and representation learning. The field takes inspiration from biological neuroscience and revolves around stacking artificial neurons into layers and \"training\" them to process data. The adjective \"deep\" refers to the use of multiple layers (ranging from three to several hundred or thousands) in the network", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0001", "text": ". Methods used can be supervised, semi-supervised or unsupervised.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0002", "text": "Some common deep learning network architectures include fully connected networks, deep belief networks, recurrent neural networks, convolutional neural networks, generative adversarial networks, transformers, and neural radiance fields", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0003", "text": ". These architectures have been applied to fields including computer vision, speech recognition, natural language processing, machine translation, bioinformatics, drug design, medical image analysis, climate science, material inspection and board game programs, where they have produced results comparable to and in some cases surpassing human expert performance.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0004", "text": "Early forms of neural networks were inspired by information processing and distributed communication nodes in biological systems, particularly the human brain. However, current neural networks do not intend to model the brain function of organisms, and are generally seen as low-quality models for that purpose.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0005", "text": "Overview\nMost modern deep learning models are based on multi-layered neural networks such as convolutional neural networks and transformers, although they can also include propositional formulas or latent variables organized layer-wise in deep generative models such as the nodes in deep belief networks and deep Boltzmann machines.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0006", "text": "Fundamentally, deep learning refers to a class of machine learning algorithms in which a hierarchy of layers is used to transform input data into a progressively more abstract and composite representation. For example, in an image recognition model, the raw input may be an image (represented as a tensor of pixels)", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0007", "text": ". The first representational layer may attempt to identify basic shapes such as lines and circles, the second layer may compose and encode arrangements of edges, the third layer may encode a nose and eyes, and the fourth layer may recognize that the image contains a face.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0008", "text": "Importantly, a deep learning process can learn which features to optimally place at which level on its own. Prior to deep learning, machine learning techniques often involved hand-crafted feature engineering to transform the data into a more suitable representation for a classification algorithm to operate on. In the deep learning approach, features are not hand-crafted and the model discovers useful feature representations from the data automatically", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0009", "text": ". This does not eliminate the need for hand-tuning; for example, varying numbers of layers and layer sizes can provide different degrees of abstraction.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0010", "text": "The word \"deep\" in \"deep learning\" refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a substantial credit assignment path (CAP) depth. The CAP is the chain of transformations from input to output. CAPs describe potentially causal connections between input and output. For a feedforward neural network, the depth of the CAPs is that of the network and is the number of hidden layers plus one (as the output layer is also parameterized)", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0011", "text": ". For recurrent neural networks, in which a signal may propagate through a layer more than once, the CAP depth is potentially unlimited. No universally agreed-upon threshold of depth divides shallow learning from deep learning, but most researchers agree that deep learning involves CAP depth higher than two. CAP of depth two has been shown to be a universal approximator in the sense that it can emulate any function", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0012", "text": ". Beyond that, more layers do not add to the function approximator ability of the network. Deep models (CAP > two) are able to extract better features than shallow models and hence, extra layers help in learning the features effectively.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0013", "text": "Deep learning architectures can be constructed with a greedy layer-by-layer method. Deep learning helps to disentangle these abstractions and pick out which features improve performance.\nDeep learning algorithms can be applied to unsupervised learning tasks. This is an important benefit because unlabeled data is more abundant than the labeled data. Examples of deep structures that can be trained in an unsupervised manner are deep belief networks.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0014", "text": "The term deep learning was introduced to the machine learning community by Rina Dechter in 1986, and to artificial neural networks by Igor Aizenberg and colleagues in 2000, in the context of Boolean threshold neurons. The etymology of the term is more complicated.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0015", "text": "Interpretations\nDeep neural networks are generally interpreted in terms of the universal approximation theorem or probabilistic inference.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0016", "text": "The classic universal approximation theorem concerns the capacity of feedforward neural networks with a single hidden layer of finite size to approximate continuous functions. In 1989, the first proof was published by George Cybenko for sigmoid activation functions and was generalised to feed-forward multi-layer architectures in 1991 by Kurt Hornik", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0017", "text": ". Recent work also showed that universal approximation also holds for non-bounded activation functions such as Kunihiko Fukushima's rectified linear unit.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0018", "text": "The universal approximation theorem for deep neural networks concerns the capacity of networks with bounded width but the depth is allowed to grow. Lu et al. proved that if the width of a deep neural network with ReLU activation is strictly larger than the input dimension, then the network can approximate any Lebesgue integrable function; if the width is smaller or equal to the input dimension, then a deep neural network is not a universal approximator.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0019", "text": "The probabilistic interpretation derives from the field of machine learning. It features inference, as well as the optimization concepts of training and testing, related to fitting and generalization, respectively. More specifically, the probabilistic interpretation considers the activation nonlinearity as a cumulative distribution function. The probabilistic interpretation led to the introduction of dropout as regularizer in neural networks", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0020", "text": ". The probabilistic interpretation was introduced by researchers including Hopfield, Widrow and Narendra and popularized in surveys such as the one by Bishop.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0021", "text": "History\nBefore 1980", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0022", "text": "There are two types of artificial neural network (ANN): feedforward neural network (FNN) or multilayer perceptron (MLP) and recurrent neural networks (RNN). RNNs have cycles in their connectivity structure, FNNs don't. In the 1920s, Wilhelm Lenz and Ernst Ising created the Ising model which is essentially a non-learning RNN architecture consisting of neuron-like threshold elements. In 1972, Shun'ichi Amari made this architecture adaptive", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0023", "text": ". His learning RNN was republished by John Hopfield in 1982. Other early recurrent neural networks were published by Kaoru Nakano in 1971. Already in 1948, Alan Turing produced work on \"Intelligent Machinery\"  that was not published in his lifetime, containing \"ideas related to artificial evolution and learning RNNs\".", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0024", "text": "Frank Rosenblatt (1958) proposed the perceptron, an MLP with 3 layers: an input layer, a hidden layer with randomized weights that did not learn, and an output layer. He later published a 1962 book that also introduced variants and computer experiments, including a version with four-layer perceptrons \"with adaptive preterminal networks\" where the last two layers have learned weights (here he credits H. D. Block and B. W. Knight). The book cites an earlier network by R. D", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0025", "text": ". The book cites an earlier network by R. D. Joseph (1960) \"functionally equivalent to a variation of\" this four-layer system (the book mentions Joseph over 30 times). Should Joseph therefore be considered the originator of proper adaptive multilayer perceptrons with learning hidden units? Unfortunately, the learning algorithm was not a functional one, and fell into oblivion.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0026", "text": "The first working deep learning algorithm was the Group method of data handling, a method to train arbitrarily deep neural networks, published by Alexey Ivakhnenko and Lapa in 1965. They regarded it as a form of polynomial regression, or a generalization of Rosenblatt's perceptron to handle more complex, nonlinear, and hierarchical relationships. A 1971 paper described a deep network with eight layers trained by this method, which is based on layer by layer training through regression analysis", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0027", "text": ". Superfluous hidden units are pruned using a separate validation set. Since the activation functions of the nodes are Kolmogorov-Gabor polynomials, these were also the first deep networks with multiplicative units or \"gates\".", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0028", "text": "The first deep learning multilayer perceptron trained by stochastic gradient descent was published in 1967 by Shun'ichi Amari. In computer experiments conducted by Amari's student Saito, a five layer MLP with two modifiable layers learned  internal representations to classify non-linearily separable pattern classes. Subsequent developments in hardware and hyperparameter tunings have made end-to-end stochastic gradient descent the currently dominant training technique.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0029", "text": "In 1969, Kunihiko Fukushima introduced the ReLU (rectified linear unit) activation function. The rectifier has become the most popular activation function for deep learning.\nDeep learning architectures for convolutional neural networks (CNNs) with convolutional layers and downsampling layers began with the Neocognitron introduced by Kunihiko Fukushima in 1979, though not trained by backpropagation.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0030", "text": "Backpropagation is an efficient application of the chain rule derived by Gottfried Wilhelm Leibniz in 1673 to networks of differentiable nodes. The terminology \"back-propagating errors\" was actually introduced in 1962 by Rosenblatt, but he did not know how to implement this, although Henry J. Kelley had a continuous precursor of backpropagation in 1960 in the context of control theory. The modern form of backpropagation was first published in Seppo Linnainmaa's master thesis (1970). G.M", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0031", "text": ". G.M. Ostrovski et al. republished it in 1971. Paul Werbos applied backpropagation to neural networks in 1982 (his 1974 PhD thesis, reprinted in a 1994 book, did not yet describe the algorithm). In 1986, David E. Rumelhart et al. popularised backpropagation but did not cite the original work.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0032", "text": "1980s-2000s\nThe time delay neural network (TDNN) was introduced in 1987 by Alex Waibel to apply CNN to phoneme recognition. It used convolutions, weight sharing, and backpropagation.  In 1988, Wei Zhang applied a backpropagation-trained CNN to alphabet recognition.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0033", "text": "In 1989, Yann LeCun et al. created a CNN called LeNet for recognizing handwritten ZIP codes on mail. Training required 3 days. In 1990, Wei Zhang implemented a CNN on optical computing hardware. In 1991, a CNN was applied to medical image object segmentation and breast cancer detection in mammograms. LeNet-5 (1998), a 7-level CNN by Yann LeCun et al., that classifies digits, was applied by several banks to recognize hand-written numbers on checks  digitized in 32x32 pixel images.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0034", "text": "Recurrent neural networks (RNN) were further developed in the 1980s. Recurrence is used for sequence processing, and when a recurrent network is unrolled, it mathematically resembles a deep feedforward layer. Consequently, they have similar properties and issues, and their developments had mutual influences. In RNN, two early influential works were the Jordan network (1986) and the Elman network (1990), which applied RNN to study problems in cognitive psychology.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0035", "text": "In the 1980s, backpropagation did not work well for deep learning with long credit assignment paths. To overcome this problem, in 1991, J\u00fcrgen Schmidhuber proposed a hierarchy of RNNs pre-trained one level at a time by self-supervised learning where each RNN tries to predict its own next input, which is the next unexpected input of the RNN below. This \"neural history compressor\" uses predictive coding  to learn internal representations at multiple self-organizing time scales", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0036", "text": ". This can substantially facilitate downstream deep learning. The RNN hierarchy can be collapsed into a single RNN, by  distilling a higher level chunker network into a lower level automatizer network. In 1993, a neural history compressor solved a \"Very Deep Learning\" task that required more than 1000 subsequent layers in an RNN unfolded in time. The \"P\" in ChatGPT refers to such pre-training.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0037", "text": "Sepp Hochreiter's diploma thesis (1991) implemented the neural history compressor, and identified and analyzed the vanishing gradient problem.  Hochreiter proposed recurrent residual connections to solve the vanishing gradient problem. This led to the long short-term memory (LSTM), published in 1995. LSTM can learn \"very deep learning\" tasks with long credit assignment paths that require memories of events that happened thousands of discrete time steps before", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0038", "text": ". That LSTM was not yet the modern architecture, which required a \"forget gate\", introduced in 1999, which became the standard RNN architecture.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0039", "text": "In 1991, J\u00fcrgen Schmidhuber also published adversarial neural networks that contest with each other in the form of a zero-sum game, where one network's gain is the other network's loss. The first network is a generative model that models a probability distribution over output patterns. The second network learns by gradient descent to predict the reactions of the environment to these patterns. This was called \"artificial curiosity\"", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0040", "text": ". This was called \"artificial curiosity\". In 2014, this principle was used in generative adversarial networks (GANs).", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0041", "text": "During 1985\u20131995, inspired by statistical mechanics, several architectures and methods were developed by Terry Sejnowski, Peter Dayan, Geoffrey Hinton, etc., including the Boltzmann machine, restricted Boltzmann machine, Helmholtz machine, and the wake-sleep algorithm. These were designed for unsupervised learning of deep generative models. However, those were more computationally expensive compared to backpropagation", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0042", "text": ". Boltzmann machine learning algorithm, published in 1985, was briefly popular before being eclipsed by the backpropagation algorithm in 1986. (p. 112 ). A 1988 network became state of the art in protein structure prediction, an early application of deep learning to bioinformatics.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0043", "text": "Both shallow and deep learning (e.g., recurrent nets) of ANNs for speech recognition have been explored for many years. These methods never outperformed non-uniform internal-handcrafting Gaussian mixture model/Hidden Markov model (GMM-HMM) technology based on generative models of speech trained discriminatively. Key difficulties have been analyzed, including gradient diminishing and weak temporal correlation structure in neural predictive models", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0044", "text": ". Additional difficulties were the lack of training data and limited computing power.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0045", "text": "Most speech recognition researchers moved away from neural nets to pursue generative modeling. An exception was at SRI International in the late 1990s. Funded by the US government's NSA and DARPA, SRI researched in speech and speaker recognition. The speaker recognition team led by Larry Heck reported significant success with deep neural networks in speech processing in the 1998 NIST Speaker Recognition benchmark", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0046", "text": ". It was deployed in the Nuance Verifier, representing the first major industrial application of deep learning.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0047", "text": "The principle of elevating \"raw\" features over hand-crafted optimization was first explored successfully in the architecture of deep autoencoder on the \"raw\" spectrogram or linear filter-bank features in the late 1990s, showing its superiority over the Mel-Cepstral features that contain stages of fixed transformation from spectrograms. The raw features of speech, waveforms, later produced excellent larger-scale results.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0048", "text": "2000s\nNeural networks entered a lull, and simpler models that use task-specific handcrafted features such as Gabor filters and support vector machines (SVMs) became the preferred choices in the 1990s and 2000s, because of artificial neural networks' computational cost and a lack of understanding of how the brain wires its biological networks.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0049", "text": "In 2003, LSTM became competitive with traditional speech recognizers on certain tasks. In 2006, Alex Graves, Santiago Fern\u00e1ndez, Faustino Gomez, and Schmidhuber combined it with connectionist temporal classification (CTC) in stacks of LSTMs. In 2009, it became the first RNN to win a pattern recognition contest, in connected handwriting recognition.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0050", "text": "In 2006, publications by Geoff Hinton, Ruslan Salakhutdinov, Osindero and Teh deep belief networks were developed for generative modeling. They are trained by training one restricted Boltzmann machine, then freezing it and training another one on top of the first one, and so on, then optionally fine-tuned using supervised backpropagation. They could model high-dimensional probability distributions, such as the distribution of MNIST images, but convergence was slow.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0051", "text": "The impact of deep learning in industry began in the early 2000s, when CNNs already processed an estimated 10% to 20% of all the checks written in the US, according to Yann LeCun. Industrial applications of deep learning to large-scale speech recognition started around 2010.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0052", "text": "The 2009 NIPS Workshop on Deep Learning for Speech Recognition was motivated by the limitations of deep generative models of speech, and the possibility that given more capable hardware and large-scale data sets that deep neural nets might become practical. It was believed that pre-training DNNs using generative models of deep belief nets (DBN) would overcome the main difficulties of neural nets", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0053", "text": ". However, it was discovered that replacing pre-training with large amounts of training data for straightforward backpropagation when using DNNs with large, context-dependent output layers produced error rates dramatically lower than then-state-of-the-art Gaussian mixture model (GMM)/Hidden Markov Model (HMM) and also than more-advanced generative model-based systems", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0054", "text": ". The nature of the recognition errors produced by the two types of systems was characteristically different, offering technical insights into how to integrate deep learning into the existing highly efficient, run-time speech decoding system deployed by all major speech recognition systems. Analysis around 2009\u20132010, contrasting the GMM (and other generative speech models) vs. DNN models, stimulated early industrial investment in deep learning for speech recognition", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0055", "text": ".  That analysis was done with comparable performance (less than 1.5% in error rate) between discriminative DNNs and generative models.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0056", "text": "In 2010, researchers extended deep learning from TIMIT to large vocabulary speech recognition, by adopting large output layers of the DNN based on context-dependent HMM states constructed by decision trees.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0057", "text": "Deep learning revolution\nThe deep learning revolution started around CNN- and GPU-based computer vision.\nAlthough CNNs trained by backpropagation had been around for decades and GPU implementations of NNs for years, including CNNs, faster implementations of CNNs on GPUs were needed to progress on computer vision. Later, as deep learning becomes widespread, specialized hardware and algorithm optimizations were developed specifically for deep learning.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0058", "text": "A key advance for the deep learning revolution was hardware advances, especially GPU. Some early work dated back to 2004. In 2009, Raina, Madhavan, and Andrew Ng reported a 100M deep belief network trained on 30 Nvidia GeForce GTX 280 GPUs, an early demonstration of GPU-based deep learning. They reported up to 70 times faster training.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0059", "text": "In 2011, a CNN named DanNet by Dan Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, and J\u00fcrgen Schmidhuber achieved for the first time superhuman performance in a visual pattern recognition contest, outperforming traditional methods by a factor of 3. It then won more contests. They also showed how max-pooling CNNs on GPU improved performance significantly.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0060", "text": "In 2012, Andrew Ng and Jeff Dean created an FNN that learned to recognize higher-level concepts, such as cats, only from watching unlabeled images taken from YouTube videos.\nIn October 2012, AlexNet by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton won the large-scale ImageNet competition by a significant margin over shallow machine learning methods. Further incremental improvements included the VGG-16 network by Karen Simonyan and Andrew Zisserman and Google's Inceptionv3.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0061", "text": "The success in image classification was then extended to the more challenging task of generating descriptions (captions) for images, often as a combination of CNNs and LSTMs.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0062", "text": "In 2014, the state of the art was training \"very deep neural network\" with 20 to 30 layers. Stacking too many layers led to a steep reduction in training accuracy, known as the \"degradation\" problem. In 2015, two techniques were developed to train very deep networks: the highway network was published in May 2015, and the residual neural network (ResNet) in Dec 2015. ResNet behaves like an open-gated Highway Net.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0063", "text": "Around the same time, deep learning started impacting the field of art. Early examples included Google DeepDream (2015), and neural style transfer (2015), both of which were based on pretrained image classification neural networks, such as VGG-19.\nGenerative adversarial network (GAN) by (Ian Goodfellow et al., 2014) (based on  J\u00fcrgen Schmidhuber's principle of artificial curiosity)", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0064", "text": "became state of the art in generative modeling during 2014-2018 period. Excellent image quality is achieved by Nvidia's StyleGAN (2018) based on the Progressive GAN by Tero Karras et al. Here the GAN generator is grown from small to large scale in a pyramidal fashion. Image generation by GAN reached popular success, and provoked discussions concerning deepfakes", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0065", "text": ".  Diffusion models (2015) eclipsed GANs in generative modeling since then, with systems such as DALL\u00b7E 2 (2022) and Stable Diffusion (2022).", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0066", "text": "In 2015, Google's speech recognition improved by 49% by an LSTM-based model, which they made available through Google Voice Search on smartphone.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0067", "text": "Deep learning is part of state-of-the-art systems in various disciplines, particularly computer vision and automatic speech recognition (ASR). Results on commonly used evaluation sets such as TIMIT (ASR) and MNIST (image classification), as well as a range of large-vocabulary speech recognition tasks have steadily improved. Convolutional neural networks were superseded for ASR by LSTM. but are more successful in computer vision.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0068", "text": "Yoshua Bengio, Geoffrey Hinton and Yann LeCun were awarded the 2018 Turing Award for \"conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing\".", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0069", "text": "Neural networks", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0070", "text": "Artificial neural networks (ANNs) or connectionist systems are computing systems inspired by the biological neural networks that constitute animal brains. Such systems learn (progressively improve their ability) to do tasks by considering examples, generally without task-specific programming", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0071", "text": ". For example, in image recognition, they might learn to identify images that contain cats by analyzing example images that have been manually labeled as \"cat\" or \"no cat\" and using the analytic results to identify cats in other images. They have found most use in applications difficult to express with a traditional computer algorithm using rule-based programming.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0072", "text": "An ANN is based on a collection of connected units called artificial neurons, (analogous to biological neurons in a biological brain). Each connection (synapse) between neurons can transmit a signal to another neuron. The receiving (postsynaptic) neuron can process the signal(s) and then signal downstream neurons connected to it. Neurons may have state, generally represented by real numbers, typically between 0 and 1", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0073", "text": ". Neurons and synapses may also have a weight that varies as learning proceeds, which can increase or decrease the strength of the signal that it sends downstream.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0074", "text": "Typically, neurons are organized in layers. Different layers may perform different kinds of transformations on their inputs. Signals travel from the first (input), to the last (output) layer, possibly after traversing the layers multiple times.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0075", "text": "The original goal of the neural network approach was to solve problems in the same way that a human brain would. Over time, attention focused on matching specific mental abilities, leading to deviations from biology such as backpropagation, or passing information in the reverse direction and adjusting the network to reflect that information.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0076", "text": "Neural networks have been used on a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games and medical diagnosis.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0077", "text": "As of 2017, neural networks typically have a few thousand to a few million units and millions of connections. Despite this number being several order of magnitude less than the number of neurons on a human brain, these networks can perform many tasks at a level beyond that of humans (e.g., recognizing faces, or playing \"Go\").", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0078", "text": "Deep neural networks\nA deep neural network (DNN) is an artificial neural network with multiple layers between the input and output layers. There are different types of neural networks but they always consist of the same components: neurons, synapses, weights, biases, and functions. These components as a whole function in a way that mimics functions of the human brain, and can be trained like any other ML algorithm.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0079", "text": "For example, a DNN that is trained to recognize dog breeds will go over the given image and calculate the probability that the dog in the image is a certain breed. The user can review the results and select which probabilities the network should display (above a certain threshold, etc.) and return the proposed label. Each mathematical manipulation as such is considered a layer, and complex DNN have many layers, hence the name \"deep\" networks.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0080", "text": "DNNs can model complex non-linear relationships. DNN architectures generate compositional models where the object is expressed as a layered composition of primitives. The extra layers enable composition of features from lower layers, potentially modeling complex data with fewer units than a similarly performing shallow network. For instance, it was proved that sparse multivariate polynomials are exponentially easier to approximate with DNNs than with shallow networks.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0081", "text": "Deep architectures include many variants of a few basic approaches. Each architecture has found success in specific domains. It is not always possible to compare the performance of multiple architectures, unless they have been evaluated on the same data sets.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0082", "text": "DNNs are typically feedforward networks in which data flows from the input layer to the output layer without looping back. At first, the DNN creates a map of virtual neurons and assigns random numerical values, or \"weights\", to connections between them. The weights and inputs are multiplied and return an output between 0 and 1. If the network did not accurately recognize a particular pattern, an algorithm would adjust the weights", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0083", "text": ". That way the algorithm can make certain parameters more influential, until it determines the correct mathematical manipulation to fully process the data.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0084", "text": "Recurrent neural networks, in which data can flow in any direction, are used for applications such as language modeling. Long short-term memory is particularly effective for this use.\nConvolutional neural networks (CNNs) are used in computer vision. CNNs also have been applied to acoustic modeling for automatic speech recognition (ASR).", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0085", "text": "Challenges\nAs with ANNs, many issues can arise with naively trained DNNs. Two common issues are overfitting and computation time.\nDNNs are prone to overfitting because of the added layers of abstraction, which allow them to model rare dependencies in the training data. Regularization methods such as Ivakhnenko's unit pruning or weight decay (\n  \n    \n      \n        \n          \u2113\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\ell _{2}}", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0086", "text": "{\\displaystyle \\ell _{2}}\n  \n-regularization) or sparsity (\n  \n    \n      \n        \n          \u2113\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\ell _{1}}", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0087", "text": "-regularization) can be applied during training to combat overfitting. Alternatively dropout regularization randomly omits units from the hidden layers during training. This helps to exclude rare dependencies. Another interesting recent development is research into models of just enough complexity through an estimation of the intrinsic complexity of the task being modelled. This approach has been successfully applied for multivariate time series prediction tasks such as traffic prediction", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0088", "text": ". Finally, data can be augmented via methods such as cropping and rotating such that smaller training sets can be increased in size to reduce the chances of overfitting.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0089", "text": "DNNs must consider many training parameters, such as the size (number of layers and number of units per layer), the learning rate, and initial weights. Sweeping through the parameter space for optimal parameters may not be feasible due to the cost in time and computational resources. Various tricks, such as batching (computing the gradient on several training examples at once rather than individual examples) speed up computation", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0090", "text": ". Large processing capabilities of many-core architectures (such as GPUs or the Intel Xeon Phi) have produced significant speedups in training, because of the suitability of such processing architectures for the matrix and vector computations.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0091", "text": "Alternatively, engineers may look for other types of neural networks with more straightforward and convergent training algorithms. CMAC (cerebellar model articulation controller) is one such kind of neural network. It doesn't require learning rates or randomized initial weights. The training process can be guaranteed to converge in one step with a new batch of data, and the computational complexity of the training algorithm is linear with respect to the number of neurons involved.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0092", "text": "Hardware", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0093", "text": "Since the 2010s, advances in both machine learning algorithms and computer hardware have led to more efficient methods for training deep neural networks that contain many layers of non-linear hidden units and a very large output layer. By 2019, graphics processing units (GPUs), often with AI-specific enhancements, had displaced CPUs as the dominant method for training large-scale commercial cloud AI", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0094", "text": ". OpenAI estimated the hardware computation used in the largest deep learning projects from AlexNet (2012) to AlphaZero (2017) and found a 300,000-fold increase in the amount of computation required, with a doubling-time trendline of 3.4 months.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0095", "text": "Special electronic circuits called deep learning processors were designed to speed up deep learning algorithms. Deep learning processors include neural processing units (NPUs) in Huawei cellphones and cloud computing servers such as tensor processing units (TPU) in the Google Cloud Platform. Cerebras Systems has also built a dedicated system to handle large deep learning models, the CS-2, based on the largest processor in the industry, the second-generation Wafer Scale Engine (WSE-2).", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0096", "text": "Atomically thin semiconductors are considered promising for energy-efficient deep learning hardware where the same basic device structure is used for both logic operations and data storage.\nIn 2020, Marega et al. published experiments with a large-area active channel material for developing logic-in-memory devices and circuits based on floating-gate field-effect transistors (FGFETs).", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0097", "text": "In 2021, J. Feldmann et al. proposed an integrated photonic hardware accelerator for parallel convolutional processing. The authors identify two key advantages of integrated photonics over its electronic counterparts: (1) massively parallel data transfer through wavelength division multiplexing in conjunction with frequency combs, and (2) extremely high data modulation speeds", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0098", "text": ". Their system can execute trillions of multiply-accumulate operations per second, indicating the potential of integrated photonics in data-heavy AI applications.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0099", "text": "Applications\nAutomatic speech recognition\nLarge-scale automatic speech recognition is the first and most convincing successful case of deep learning. LSTM RNNs can learn \"Very Deep Learning\" tasks that involve multi-second intervals containing speech events separated by thousands of discrete time steps, where one time step corresponds to about 10 ms. LSTM with forget gates is competitive with traditional speech recognizers on certain tasks.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0100", "text": "The initial success in speech recognition was based on small-scale recognition tasks based on TIMIT. The data set contains 630 speakers from eight major dialects of American English, where each speaker reads 10 sentences. Its small size lets many configurations be tried. More importantly, the TIMIT task concerns phone-sequence recognition, which, unlike word-sequence recognition, allows weak phone bigram language models", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0101", "text": ". This lets the strength of the acoustic modeling aspects of speech recognition be more easily analyzed. The error rates listed below, including these early results and measured as percent phone error rates (PER), have been summarized since 1991.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0102", "text": "The debut of DNNs for speaker recognition in the late 1990s and speech recognition around 2009-2011 and of LSTM around 2003\u20132007, accelerated progress in eight major areas:", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0103", "text": "Scale-up/out and accelerated DNN training and decoding\nSequence discriminative training\nFeature processing by deep models with solid understanding of the underlying mechanisms\nAdaptation of DNNs and related deep models\nMulti-task and transfer learning by DNNs and related deep models\nCNNs and how to design them to best exploit domain knowledge of speech\nRNN and its rich LSTM variants\nOther types of deep models including tensor-based models and integrated deep generative/discriminative models.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0104", "text": "More recent speech recognition models use Transformers or Temporal Convolution Networks with significant success and widespread applications. All major commercial speech recognition systems (e.g., Microsoft Cortana, Xbox, Skype Translator, Amazon Alexa, Google Now, Apple Siri, Baidu and iFlyTek voice search, and a range of Nuance speech products, etc.) are based on deep learning.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0105", "text": "Image recognition\nA common evaluation set for image classification is the MNIST database data set. MNIST is composed of handwritten digits and includes 60,000 training examples and 10,000 test examples. As with TIMIT, its small size lets users test multiple configurations. A comprehensive list of results on this set is available.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0106", "text": "Deep learning-based image recognition has become \"superhuman\", producing more accurate results than human contestants. This first occurred in 2011 in recognition of traffic signs, and in 2014, with recognition of human faces.\nDeep learning-trained vehicles now interpret 360\u00b0 camera views. Another example is Facial Dysmorphology Novel Analysis (FDNA) used to analyze cases of human malformation connected to a large database of genetic syndromes.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0107", "text": "Visual art processing\nClosely related to the progress that has been made in image recognition is the increasing application of deep learning techniques to various visual art tasks. DNNs have proven themselves capable, for example, of\n\nidentifying the style period of a given painting\nNeural Style Transfer \u2013  capturing the style of a given artwork and applying it in a visually pleasing manner to an arbitrary photograph or video\ngenerating striking imagery based on random visual input fields.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0108", "text": "Natural language processing\nNeural networks have been used for implementing language models since the early 2000s. LSTM helped to improve machine translation and language modeling.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0109", "text": "Other key techniques in this field are negative sampling and word embedding. Word embedding, such as word2vec, can be thought of as a representational layer in a deep learning architecture that transforms an atomic word into a positional representation of the word relative to other words in the dataset; the position is represented as a point in a vector space", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0110", "text": ". Using word embedding as an RNN input layer allows the network to parse sentences and phrases using an effective compositional vector grammar. A compositional vector grammar can be thought of as probabilistic context free grammar (PCFG) implemented by an RNN. Recursive auto-encoders built atop word embeddings can assess sentence similarity and detect paraphrasing", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0111", "text": ". Deep neural architectures provide the best results for constituency parsing, sentiment analysis, information retrieval, spoken language understanding, machine translation, contextual entity linking, writing style recognition, named-entity recognition (token classification), text classification, and others.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0112", "text": "Recent developments generalize word embedding to sentence embedding.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0113", "text": "Google Translate (GT) uses a large end-to-end long short-term memory (LSTM) network. Google Neural Machine Translation (GNMT) uses an example-based machine translation method in which the system \"learns from millions of examples\". It translates \"whole sentences at a time, rather than pieces\". Google Translate supports over one hundred languages. The network encodes the \"semantics of the sentence rather than simply memorizing phrase-to-phrase translations\"", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0114", "text": ". GT uses English as an intermediate between most language pairs.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0115", "text": "Drug discovery and toxicology\nA large percentage of candidate drugs fail to win regulatory approval. These failures are caused by insufficient efficacy (on-target effect), undesired interactions (off-target effects), or unanticipated toxic effects. Research has explored use of deep learning to predict the biomolecular targets, off-targets, and toxic effects of environmental chemicals in nutrients, household products and drugs.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0116", "text": "AtomNet is a deep learning system for structure-based rational drug design. AtomNet was used to predict novel candidate biomolecules for disease targets such as the Ebola virus and multiple sclerosis.\nIn 2017 graph neural networks were used for the first time to predict various properties of molecules in a large toxicology data set. In 2019, generative neural networks were used to produce molecules that were validated experimentally all the way into mice.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0117", "text": "Recommendation systems\nRecommendation systems have used deep learning to extract meaningful features for a latent factor model for content-based music and journal recommendations. Multi-view deep learning has been applied for learning user preferences from multiple domains. The model uses a hybrid collaborative and content-based approach and enhances recommendations in multiple tasks.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0118", "text": "Bioinformatics\nAn autoencoder ANN was used in bioinformatics, to predict gene ontology annotations and gene-function relationships.\nIn medical informatics, deep learning was used to predict sleep quality based on data from wearables and predictions of health complications from electronic health record data.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0119", "text": "Deep neural networks have shown unparalleled performance in predicting protein structure, according to the sequence of the amino acids that make it up. In 2020, AlphaFold, a deep-learning based system, achieved a level of accuracy significantly higher than all previous computational methods.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0120", "text": "Deep Neural Network Estimations", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0121", "text": "Deep neural networks can be used to estimate the entropy of a stochastic process through an arrangement called a Neural Joint Entropy Estimator (NJEE). Such an estimation provides insights on the effects of input random variables on an independent random variable. Practically, the DNN is trained as a classifier that maps an input vector or matrix X to an output probability distribution over the possible classes of random variable Y, given input X", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0122", "text": ". For example, in image classification tasks, the NJEE maps a vector of pixels' color values to probabilities over possible image classes. In practice, the probability distribution of Y is obtained by a Softmax layer with number of nodes that is equal to the alphabet size of Y. NJEE uses continuously differentiable activation functions, such that the conditions for the universal approximation theorem holds", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0123", "text": ". It is shown that this method provides a strongly consistent estimator and outperforms other methods in cases of large alphabet sizes.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0124", "text": "Medical image analysis\nDeep learning has been shown to produce competitive results in medical applications such as cancer cell classification, lesion detection, organ segmentation and image enhancement. Modern deep learning tools demonstrate the high accuracy of detecting various diseases and the helpfulness of their use by specialists to improve the diagnosis efficiency.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0125", "text": "Mobile advertising\nFinding the appropriate mobile audience for mobile advertising is always challenging, since many data points must be considered and analyzed before a target segment can be created and used in ad serving by any ad server. Deep learning has been used to interpret large, many-dimensioned advertising datasets. Many data points are collected during the request/serve/click internet advertising cycle. This information can form the basis of machine learning to improve ad selection.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0126", "text": "Image restoration\nDeep learning has been successfully applied to inverse problems such as denoising, super-resolution, inpainting, and film colorization. These applications include learning methods such as \"Shrinkage Fields for Effective Image Restoration\" which trains on an image dataset, and Deep Image Prior, which trains on the image that needs restoration.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0127", "text": "Financial fraud detection\nDeep learning is being applied to financial fraud detection, tax evasion detection, and anti-money laundering.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0128", "text": "Materials science", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0129", "text": "In November 2023, researchers at Google DeepMind and Lawrence Berkeley National Laboratory announced that they had developed an AI system known as GNoME. This system has contributed to materials science by discovering over 2 million new materials within a relatively short timeframe. GNoME employs deep learning techniques to efficiently explore potential material structures, achieving a significant increase in the identification of stable inorganic crystal structures", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0130", "text": ". The system's predictions were validated through autonomous robotic experiments, demonstrating a noteworthy success rate of 71%. The data of newly discovered materials is publicly available through the Materials Project database, offering researchers the opportunity to identify materials with desired properties for various applications", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0131", "text": ". This development has implications for the future of scientific discovery and the integration of AI in material science research, potentially expediting material innovation and reducing costs in product development. The use of AI and deep learning suggests the possibility of minimizing or eliminating manual lab experiments and allowing scientists to focus more on the design and analysis of unique compounds.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0132", "text": "Military\nThe United States Department of Defense applied deep learning to train robots in new tasks through observation.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0133", "text": "Partial differential equations", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0134", "text": "Physics informed neural networks have been used to solve partial differential equations in both forward and inverse problems in a data driven manner. One example is the reconstructing fluid flow governed by the Navier-Stokes equations. Using physics informed neural networks does not require the often expensive mesh generation that conventional CFD methods rely on", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0135", "text": ". It is evident that geometric and physical constraints have a synergistic effect on neural PDE surrogates, thereby enhancing their efficacy in predicting stable and super long rollouts.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0136", "text": "Deep backward stochastic differential equation method", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0137", "text": "Deep backward stochastic differential equation method is a numerical method that combines deep learning with Backward stochastic differential equation (BSDE). This method is particularly useful for solving high-dimensional problems in financial mathematics. By leveraging the powerful function approximation capabilities of deep neural networks, deep BSDE addresses the computational challenges faced by traditional numerical methods in high-dimensional settings", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0138", "text": ". Specifically, traditional methods like finite difference methods or Monte Carlo simulations often struggle with the curse of dimensionality, where computational cost increases exponentially with the number of dimensions. Deep BSDE methods, however, employ deep neural networks to approximate solutions of high-dimensional partial differential equations (PDEs), effectively reducing the computational burden.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0139", "text": "In addition, the integration of Physics-informed neural networks (PINNs) into the deep BSDE framework enhances its capability by embedding the underlying physical laws directly into the neural network architecture. This ensures that the solutions not only fit the data but also adhere to the governing stochastic differential equations", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0140", "text": ". PINNs leverage the power of deep learning while respecting the constraints imposed by the physical models, resulting in more accurate and reliable solutions for financial mathematics problems.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0141", "text": "Image reconstruction\nImage reconstruction is the reconstruction of the underlying images from the image-related measurements. Several works showed the better and superior performance of the deep learning methods compared to analytical methods for various applications, e.g., spectral imaging  and ultrasound imaging.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0142", "text": "Weather prediction\nTraditional weather prediction systems solve a very complex system of partial differential equations. GraphCast is a deep learning based model, trained on a long history of weather data to predict how weather patterns change over time. It is able to  predict weather conditions for up to 10 days globally, at a very detailed level, and in under a minute, with precision similar to state of the art systems.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0143", "text": "Epigenetic clock", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0144", "text": "An epigenetic clock is a biochemical test that can be used to measure age. Galkin et al. used deep neural networks to train an epigenetic aging clock of unprecedented accuracy using >6,000 blood samples. The clock uses information from 1000 CpG sites and predicts people with certain conditions older than healthy controls: IBD, frontotemporal dementia, ovarian cancer, obesity", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0145", "text": ". The aging clock was planned to be released for public use in 2021 by an Insilico Medicine spinoff company Deep Longevity.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0146", "text": "Relation to human cognitive and brain development", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0147", "text": "Deep learning is closely related to a class of theories of brain development (specifically, neocortical development) proposed by cognitive neuroscientists in the early 1990s. These developmental theories were instantiated in computational models, making them predecessors of deep learning systems", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0148", "text": ". These developmental models share the property that various proposed learning dynamics in the brain (e.g., a wave of nerve growth factor) support the self-organization somewhat analogous to the neural networks utilized in deep learning models. Like the neocortex, neural networks employ a hierarchy of layered filters in which each layer considers information from a prior layer (or the operating environment), and then passes its output (and possibly the original input), to other layers", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0149", "text": ". This process yields a self-organizing stack of transducers, well-tuned to their operating environment. A 1995 description stated, \"...the infant's brain seems to organize itself under the influence of waves of so-called trophic-factors ... different regions of the brain become connected sequentially, with one layer of tissue maturing before another and so on until the whole brain is mature\".", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0150", "text": "A variety of approaches have been used to investigate the plausibility of deep learning models from a neurobiological perspective. On the one hand, several variants of the backpropagation algorithm have been proposed in order to increase its processing realism. Other researchers have argued that unsupervised forms of deep learning, such as those based on hierarchical generative models and deep belief networks, may be closer to biological reality", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0151", "text": ". In this respect, generative neural network models have been related to neurobiological evidence about sampling-based processing in the cerebral cortex.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0152", "text": "Although a systematic comparison between the human brain organization and the neuronal encoding in deep networks has not yet been established, several analogies have been reported. For example, the computations performed by deep learning units could be similar to those of actual neurons and neural populations. Similarly, the representations developed by deep learning models are similar to those measured in the primate visual system both at the single-unit and at the population levels.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0153", "text": "Commercial activity\nFacebook's AI lab performs tasks such as automatically tagging uploaded pictures with the names of the people in them.\nGoogle's DeepMind Technologies developed a system capable of learning how to play Atari video games using only pixels as data input. In 2015 they demonstrated their AlphaGo system, which learned the game of Go well enough to beat a professional Go player. Google Translate uses a neural network to translate between more than 100 languages.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0154", "text": "In 2017, Covariant.ai was launched, which focuses on integrating deep learning into factories.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0155", "text": "As of 2008, researchers at The University of Texas at Austin (UT) developed a machine learning framework called Training an Agent Manually via Evaluative Reinforcement, or TAMER, which proposed new methods for robots or computer programs to learn how to perform tasks by interacting with a human instructor. First developed as TAMER, a new algorithm called Deep TAMER was later introduced in 2018 during a collaboration between U.S. Army Research Laboratory (ARL) and UT researchers", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0156", "text": ". Deep TAMER used deep learning to provide a robot with the ability to learn new tasks through observation. Using Deep TAMER, a robot learned a task with a human trainer, watching video streams or observing a human perform a task in-person. The robot later practiced the task with the help of some coaching from the trainer, who provided feedback such as \"good job\" and \"bad job\".", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0157", "text": "Criticism and comment\nDeep learning has attracted both criticism and comment, in some cases from outside the field of computer science.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0158", "text": "Theory\nA main criticism concerns the lack of theory surrounding some methods. Learning in the most common deep architectures is implemented using well-understood gradient descent. However, the theory surrounding other algorithms, such as contrastive divergence is less clear. (e.g., Does it converge? If so, how fast? What is it approximating?) Deep learning methods are often looked at as a black box, with most confirmations done empirically, rather than theoretically.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0159", "text": "In further reference to the idea that artistic sensitivity might be inherent in relatively low levels of the cognitive hierarchy, a published series of graphic representations of the internal states of deep (20-30 layers) neural networks attempting to discern within essentially random data the images on which they were trained demonstrate a visual appeal: the original research notice received well over 1,000 comments, and was the subject of what was for a time the most frequently accessed", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0160", "text": "what was for a time the most frequently accessed article on The Guardian's website.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0161", "text": "With the support of Innovation Diffusion Theory (IDT), a study analyzed the diffusion of Deep Learning in BRICS and OECD countries using data from Google Trends.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0162", "text": "Errors", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0163", "text": "Some deep learning architectures display problematic behaviors, such as confidently classifying unrecognizable images as belonging to a familiar category of ordinary images (2014) and misclassifying minuscule perturbations of correctly classified images (2013)", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0164", "text": ". Goertzel hypothesized that these behaviors are due to limitations in their internal representations and that these limitations would inhibit integration into heterogeneous multi-component artificial general intelligence (AGI) architectures. These issues may possibly be addressed by deep learning architectures that internally form states homologous to image-grammar decompositions of observed entities and events", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0165", "text": ". Learning a grammar (visual or linguistic) from training data would be equivalent to restricting the system to commonsense reasoning that operates on concepts in terms of grammatical production rules and is a basic goal of both human language acquisition and artificial intelligence (AI).", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0166", "text": "Cyber threat", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0167", "text": "As deep learning moves from the lab into the world, research and experience show that artificial neural networks are vulnerable to hacks and deception. By identifying patterns that these systems use to function, attackers can modify inputs to ANNs in such a way that the ANN finds a match that human observers would not recognize. For example, an attacker can make subtle changes to an image such that the ANN finds a match even though the image looks to a human nothing like the search target", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0168", "text": ". Such manipulation is termed an \"adversarial attack\".", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0169", "text": "In 2016 researchers used one ANN to doctor images in trial and error fashion, identify another's focal points, and thereby generate images that deceived it. The modified images looked no different to human eyes. Another group showed that printouts of doctored images then photographed successfully tricked an image classification system. One defense is reverse image search, in which a possible fake image is submitted to a site such as TinEye that can then find other instances of it", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0170", "text": ". A refinement is to search using only parts of the image, to identify images from which that piece may have been taken.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0171", "text": "Another group showed that certain psychedelic spectacles could fool a facial recognition system into thinking ordinary people were celebrities, potentially allowing one person to impersonate another. In 2017 researchers added stickers to stop signs and caused an ANN to misclassify them.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0172", "text": "ANNs can however be further trained to detect attempts at deception, potentially leading attackers and defenders into an arms race similar to the kind that already defines the malware defense industry. ANNs have been trained to defeat ANN-based anti-malware software by repeatedly attacking a defense with malware that was continually altered by a genetic algorithm until it tricked the anti-malware while retaining its ability to damage the target.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0173", "text": "In 2016, another group demonstrated that certain sounds could make the Google Now voice command system open a particular web address, and hypothesized that this could \"serve as a stepping stone for further attacks (e.g., opening a web page hosting drive-by malware)\".\nIn \"data poisoning\", false data is continually smuggled into a machine learning system's training set to prevent it from achieving mastery.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0174", "text": "Data collection ethics", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0175", "text": "The deep learning systems that are trained using supervised learning often rely on data that is created or annotated by humans, or both. It has been argued that not only low-paid clickwork (such as on Amazon Mechanical Turk) is regularly deployed for this purpose, but also implicit forms of human microwork that are often not recognized as such", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0176", "text": ". The philosopher Rainer M\u00fchlhoff distinguishes five types of \"machinic capture\" of human microwork to generate training data: (1) gamification (the embedding of annotation or computation tasks in the flow of a game), (2) \"trapping and tracking\" (e.g. CAPTCHAs for image recognition or click-tracking on Google search results pages), (3) exploitation of social motivations (e.g. tagging faces on Facebook to obtain labeled facial images), (4) information mining (e.g", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0177", "text": ". by leveraging quantified-self devices such as activity trackers) and (5) clickwork.", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Deep_learning_0178", "text": "See also\nApplications of artificial intelligence\nComparison of deep learning software\nCompressed sensing\nDifferentiable programming\nEcho state network\nList of artificial intelligence projects\nLiquid state machine\nList of datasets for machine-learning research\nReservoir computing\nScale space and deep learning\nSparse coding\nStochastic parrot\nTopological deep learning\n\nReferences\n\n\n== Further reading ==", "title": "Deep learning", "url": "https://en.wikipedia.org/wiki/Deep_learning"}, {"id": "rec_Neural_network_machine_learning_0000", "text": "In machine learning, a neural network (NN) or neural net, also known as an artificial neural network (ANN), is a computational model inspired by the structure and functions of biological neural networks.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0001", "text": "A neural network consists of connected units or nodes called artificial neurons, which loosely model the neurons in the brain. Artificial neuron models that mimic biological neurons more closely have also been recently investigated and shown to significantly improve performance. These are connected by edges, which model the synapses in the brain. Each artificial neuron receives signals from connected neurons, then processes them and sends a signal to other connected neurons", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0002", "text": ". The \"signal\" is a real number, and the output of each neuron is computed by some non-linear function of the totality of its inputs, called the activation function. The strength of the signal at each connection is determined by a weight, which adjusts during the learning process.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0003", "text": "Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. Signals travel from the first layer (the input layer) to the last layer (the output layer), possibly passing through multiple intermediate layers (hidden layers). A network is typically called a deep neural network if it has at least two hidden layers.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0004", "text": "Advances in computing power, particularly the use of graphics processing units (GPUs), and the availability of large datasets further accelerated neural network research in the early 21st century. These developments enabled the training of deep neural networks capable of learning hierarchical representations from complex data.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0005", "text": "Architectural innovations such as convolutional neural networks (CNNs) significantly improved performance in computer vision tasks, while recurrent neural networks (RNNs) enabled modeling of sequential data such as speech and time-series information. More recently, transformer architectures introduced attention mechanisms that allow neural networks to model long-range dependencies in data and have become foundational for modern large language models.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0006", "text": "Today, artificial neural networks are used for various tasks, including predictive modeling, adaptive control, and solving problems in artificial intelligence.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0007", "text": "History\nMathematical foundations", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0008", "text": "Today's deep neural networks are based on early work in statistics over 200 years ago. The simplest kind of feedforward neural network (FNN) is a linear network, which consists of a single layer of output nodes with linear activation functions; the inputs are fed directly to the outputs via a series of weights. The sum of the products of the weights and the inputs is calculated at each node", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0009", "text": ". The mean squared errors between these calculated outputs and the given target values are minimized by creating an adjustment to the weights. This technique has been known for over two centuries as the method of least squares or linear regression. It was used as a means of finding a good rough linear fit to a set of points by Legendre (1805) and Gauss (1795) for the prediction of planetary movement.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0010", "text": "Perceptrons\nHistorically, digital computers such as the von Neumann model operate via the execution of explicit instructions with access to memory by a number of processors. Some neural networks, on the other hand, originated from efforts to model information processing in biological systems through the framework of connectionism. Unlike the von Neumann model, connectionist computing does not separate memory and processing.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0011", "text": "Warren McCulloch and Walter Pitts (1943) considered a non-learning computational model for neural networks. This model paved the way for research to split into two approaches; one approach focused on biological processes while the other focused on the application of neural networks to artificial intelligence. McCulloch and Pitts also developed mathematical models of artificial neurons capable of representing logical functions.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0012", "text": "In the late 1940s, D. O. Hebb proposed a learning hypothesis based on the mechanism of neural plasticity that became known as Hebbian learning. It was used in many early neural networks, such as Rosenblatt's perceptron and the Hopfield network. Farley and Clark (1954) used computational machines to simulate a Hebbian network. Other neural network computational machines were created by Rochester, Holland, Habit and Duda (1956).", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0013", "text": "In 1958, psychologist Frank Rosenblatt described the perceptron, one of the first implemented artificial neural networks, funded by the United States Office of Naval Research. R. D. Joseph (1960) mentions an even earlier perceptron-like device by B. G. Farley and W. A. Clark of the MIT Lincoln Laboratory; however, according to Joseph, \"they dropped the subject.\"", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0014", "text": "The first perceptrons did not have adaptive hidden units. However, Joseph (1960) also discussed multilayer perceptrons with an adaptive hidden layer. Rosenblatt (1962) cited and adopted these ideas, also crediting work by H. D. Block and B. W. Knight. Unfortunately, these early efforts did not lead to a working learning algorithm for hidden units, i.e., deep learning.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0015", "text": "The perceptron raised public excitement for research in artificial neural networks, causing the US government to drastically increase funding. This contributed to \"the Golden Age of AI\", fueled by the optimistic claims made by computer scientists regarding the ability of perceptrons to emulate human intelligence.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0016", "text": "Historical foundations and the Dartmouth proposal\nArtificial neural networks were identified as a promising direction for artificial intelligence research in the 1955 proposal for the Dartmouth Summer Research Project on Artificial Intelligence. In the proposal, researchers suggested that simplified computational models of biological neurons, described as \"neuron nets,\" might enable machines to learn, form concepts, and improve performance through experience.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0017", "text": "Despite this early promise, neural network models faced major limitations during the early decades of artificial intelligence research. Hardware constraints limited network size and training efficiency, while theoretical understanding of learning algorithms remained incomplete. Many early models relied on single-layer perceptrons, which were restricted to solving linearly separable problems", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0018", "text": ". These limitations were highlighted in the book Perceptrons by Marvin Minsky and Seymour Papert, which contributed to reduced interest in neural network research during the late 1960s and 1970s.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0019", "text": "Deep learning breakthroughs in the 1960s and 1970s", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0020", "text": "Fundamental research was conducted on ANNs in the 1960s and 1970s. The first working deep learning algorithm was the Group method of data handling, a method to train arbitrarily deep neural networks, published by Alexey Ivakhnenko and Valentin Lapa in the Soviet Union (1965). They regarded it as a form of polynomial regression, or a generalization of Rosenblatt's perceptron", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0021", "text": ". A 1971 paper described a deep network with eight layers trained by this method, which is based on layer by layer training through regression analysis. Superfluous hidden units are pruned using a separate validation set. Since the activation functions of the nodes are Kolmogorov-Gabor polynomials, these were also the first deep networks with multiplicative units or \"gates.\"", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0022", "text": "The first deep learning multilayer perceptron (MLP) trained by stochastic gradient descent was published in 1967 by Shun'ichi Amari. In computer experiments conducted by Amari's student S. Saito, a five layer MLP with two modifiable layers learned internal representations to classify non-linearily separable pattern classes. Subsequent developments in hardware and hyperparameter tuning have made end-to-end stochastic gradient descent the currently dominant training technique.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0023", "text": "In 1969, Kunihiko Fukushima introduced the ReLU (rectified linear unit) activation function. The rectifier has become the most popular activation function for deep learning.\nNevertheless, research stagnated in the United States following the work of Minsky and Papert (1969), who emphasized that basic perceptrons were incapable of processing the exclusive-or circuit. This insight was irrelevant for the deep networks of Ivakhnenko (1965) and Amari (1967).", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0024", "text": "In 1976, transfer learning was introduced.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0025", "text": "Backpropagation", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0026", "text": "Interest in neural networks revived during the 1980s with the development of the backpropagation algorithm, which allowed multi-layer neural networks to be trained efficiently by propagating error gradients backward through network layers. Backpropagation is an efficient application of the chain rule derived by Gottfried Wilhelm Leibniz in 1673 to networks of differentiable nodes", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0027", "text": ". The terminology \"back-propagating errors\" was actually introduced in 1962 by Rosenblatt, but he did not know how to implement this, although Henry J. Kelley had a continuous precursor of backpropagation in 1960 in the context of control theory. In 1970, Seppo Linnainmaa published the modern form of backpropagation in his master's thesis (1970). G.M. Ostrovski et al. republished it in 1971", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0028", "text": ". G.M. Ostrovski et al. republished it in 1971. Paul Werbos applied backpropagation to neural networks in 1982 (his 1974 PhD thesis, reprinted in a 1994 book, did not yet describe the algorithm). In 1986, David E. Rumelhart et al. popularised backpropagation but did not cite the original work.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0029", "text": "Convolutional neural networks\nDeep learning architectures for convolutional neural networks (CNNs) with convolutional layers and downsampling layers and weight replication began with the neocognitron introduced by Kunihiko Fukushima in 1979, though not trained by backpropagation. Fukushima's CNN architecture also introduced max pooling, a popular downsampling procedure for CNNs. CNNs have become an essential tool for computer vision.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0030", "text": "The time delay neural network (TDNN) was introduced in 1987 by Alex Waibel to apply CNNs to phoneme recognition. It used convolutions, weight sharing, and backpropagation. In 1988, Wei Zhang applied a backpropagation-trained CNN to alphabet recognition.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0031", "text": "In 1989, Yann LeCun et al. created a CNN called LeNet for recognizing handwritten ZIP codes on mail. Training required 3 days. In 1990, Wei Zhang implemented a CNN on optical computing hardware. In 1991, a CNN was applied to medical image object segmentation and breast cancer detection in mammograms. LeNet-5 (1998), a 7-level CNN by Yann LeCun et al. that classifies digits, was applied by several banks to recognize hand-written numbers on checks digitized in 32\u00d732 pixel images.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0032", "text": "From 1988 onward, the use of neural networks transformed the field of protein structure prediction, in particular when the first cascading networks were trained on profiles (matrices) produced by multiple sequence alignments.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0033", "text": "Recurrent neural networks", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0034", "text": "One origin of RNN was statistical mechanics. In 1972, Shun'ichi Amari proposed to modify the weights of an Ising model by Hebbian learning rule as a model of associative memory, adding in the component of learning. This was popularized as the Hopfield network by John Hopfield (1982). Another origin of RNN was neuroscience. The word \"recurrent\" is used to describe loop-like structures in anatomy. In 1901, Cajal observed \"recurrent semicircles\" in the cerebellar cortex", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0035", "text": ". Hebb considered \"reverberating circuit\" as an explanation for short-term memory. The McCulloch and Pitts paper (1943) considered neural networks that contain cycles, and noted that the current activity of such networks can be affected by activity indefinitely far in the past.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0036", "text": "In 1982 a recurrent neural network with an array architecture (rather than a multilayer perceptron architecture), namely a Crossbar Adaptive Array, used direct recurrent connections from the output to the supervisor (teaching) inputs. In addition of computing actions (decisions), it computed internal state evaluations (emotions) of the consequence situations. Eliminating the external supervisor, it introduced the self-learning method in neural networks.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0037", "text": "In cognitive psychology, the journal American Psychologist in early 1980s carried out a debate on the relation between cognition and emotion. Social psychologist Robert Zajonc in 1980 stated that emotion is computed first and is independent from cognition, while Richard Lazarus in 1982 stated that cognition is computed first and is inseparable from emotion. In 1982 the Crossbar Adaptive Array gave a neural network model of cognition-emotion relation", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0038", "text": ". It was an example of a debate where an AI system, a recurrent neural network, contributed to an issue in the same time addressed by cognitive psychology.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0039", "text": "Two early influential works were the Jordan network (1986) and the Elman network (1990), which applied RNN to study cognitive psychology.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0040", "text": "In the 1980s, backpropagation did not work well for deep RNNs. To overcome this problem, in 1991, J\u00fcrgen Schmidhuber proposed the \"neural sequence chunker\" or \"neural history compressor\" which introduced the important concepts of self-supervised pre-training (the \"P\" in ChatGPT) and neural knowledge distillation. In 1993, a neural history compressor system solved a \"Very Deep Learning\" task that required more than 1000 subsequent layers in an RNN unfolded in time.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0041", "text": "In 1991, Sepp Hochreiter's diploma thesis identified and analyzed the vanishing gradient problem and proposed recurrent residual connections to solve it. He and Schmidhuber introduced long short-term memory (LSTM), which set accuracy records in multiple applications domains. This was not yet the modern version of LSTM, which required the forget gate, which was introduced in 1999. It became the default choice for RNN architecture.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0042", "text": "During 1985\u20131995, inspired by statistical mechanics, several architectures and methods were developed by Terry Sejnowski, Peter Dayan, Geoffrey Hinton, and others, including the Boltzmann machine, restricted Boltzmann machine, Helmholtz machine, and the wake-sleep algorithm. These were designed for unsupervised learning of deep generative models.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0043", "text": "Modern deep learning", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0044", "text": "Between 2009 and 2012, ANNs began winning prizes in image recognition contests, approaching human level performance on various tasks, initially in pattern recognition and handwriting recognition. In 2011, a CNN named DanNet by Dan Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, and J\u00fcrgen Schmidhuber achieved for the first time superhuman performance in a visual pattern recognition contest, outperforming traditional methods by a factor of 3. It then won more contests", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0045", "text": ". It then won more contests. They also showed how max-pooling CNNs on GPU improved performance significantly.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0046", "text": "In October 2012, AlexNet by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton won the large-scale ImageNet competition by a significant margin over shallow machine learning methods. Further incremental improvements included the VGG-16 network by Karen Simonyan and Andrew Zisserman and Google's Inceptionv3.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0047", "text": "In 2012, Ng and Dean created a network that learned to recognize higher-level concepts, such as cats, only from watching unlabeled images. Unsupervised pre-training and increased computing power from GPUs and distributed computing allowed the use of larger networks, particularly in image and visual recognition problems, which became known as \"deep learning\".", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0048", "text": "Radial basis function and wavelet networks were introduced in 2013. These can be shown to offer best approximation properties and have been applied in nonlinear system identification and classification applications.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0049", "text": "Generative adversarial networks (GANs) (Ian Goodfellow et al., 2014) became state of the art in generative modeling in 2014\u20132018. The GAN principle was originally published in 1991 by J\u00fcrgen Schmidhuber, who called it \"artificial curiosity\": two neural networks contest with each other in the form of a zero-sum game, where one network's gain is the other network's loss. The first network is a generative model that models a probability distribution over output patterns", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0050", "text": ". The second network learns by gradient descent to predict the reactions of the environment to these patterns. Excellent image quality is achieved by Nvidia's StyleGAN (2018) based on the Progressive GAN by Tero Karras et al. Here, the GAN generator is grown from small to large scale in a pyramidal fashion. Image generation by GAN reached popular success, and provoked discussions concerning deepfakes", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0051", "text": ". Diffusion models (2015) eclipsed GANs in generative modeling since then, with systems such as DALL\u00b7E 2 (2022) and Stable Diffusion (2022).", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0052", "text": "In 2014, the state of the art was training \"[a] very deep neural network\" with 20 to 30 layers. Stacking too many layers led to a steep reduction in training accuracy, known as the \"degradation\" problem. In 2015, two techniques were developed to train very deep networks: the highway network was published in May 2015, and the residual neural network (ResNet) in December 2015. ResNet behaves like an open-gated Highway Net.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0053", "text": "Transformers\nDuring the 2010s, the seq2seq model was developed, and attention mechanisms were added. It led to the modern transformer architecture in 2017 in Attention Is All You Need.\nIt requires computation time that is quadratic in the size of the context window. J\u00fcrgen Schmidhuber's fast weight controller (1992) scales linearly and was later shown to be equivalent to the unnormalized linear transformer.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0054", "text": "Transformers have increasingly become the model of choice for natural language processing. Many modern large language models such as ChatGPT, GPT-4, and BERT use this architecture.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0055", "text": "Models", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0056", "text": "ANNs began as an attempt to exploit the architecture of the human brain to perform tasks that conventional algorithms had little success with. They soon reoriented towards improving empirical results, abandoning attempts to remain true to their biological precursors. ANNs have the ability to learn and model complex, non-linear relationships. This is achieved by neurons being connected in various patterns, allowing the output of some neurons to become the input of others", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0057", "text": ". The network forms a directed, weighted graph.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0058", "text": "An artificial neural network consists of simulated neurons. Each neuron is connected to other nodes via links like a biological axon-synapse-dendrite connection. All the nodes connected by links take in some data and use it to perform specific operations and tasks on the data. Each link has a weight, determining the strength of one node's influence on another.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0059", "text": "Artificial neurons", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0060", "text": "ANNs are composed of artificial neurons, conceptually derived from biological neurons. Each artificial neuron has one or more numerical inputs and produces a single numerical output. To find the output of a neuron, we take the (weighted) sum of all the neuron's inputs, weighted by the weights of the connections from each of those inputs. We then add a bias term to this sum. This weighted sum (plus the bias) is then passed through a nonlinear activation function to produce the neuron's output.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0061", "text": "To create a neural network, multiple artificial neurons are connected together, with the outputs of some neurons being fed into the input of others. The inputs to the initial input neurons are external data, such as images and text (encoded as a series of numbers). The outputs of the final output neurons correspond to the task the network was created to do, such as recognizing an object in an image.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0062", "text": "Organization\nThe neurons are typically organized into multiple layers, especially in deep learning. The layer that receives external data is the input layer, and the layer that produces the ultimate result is the output layer. In between them are zero or more hidden layers. Single layer and unlayered networks are also used.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0063", "text": "Between two layers, multiple connection patterns are possible. Traditionally, they are fully connected, with every neuron in one layer connecting to every neuron in the next layer. However, in convolutional neural networks, some layers are convolutional, meaning each neuron in the next layer is connected to a set of spatially grouped neurons in the previous layer.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0064", "text": "In most neural networks, the outputs of the neurons in one layer are only connected to the inputs of neurons in the immediately  following layer (forming a directed acyclic graph), meaning information can only flow \"forward\" from one layer to the next. These are known as feedforward networks. In contrast, networks that allow connections between neurons in the same or previous layers are known as recurrent networks.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0065", "text": "Learning\nThe function of a neural network is defined by its architecture \u2013 how the different neurons are organized and connected to one another \u2013 and the values, or weights, associated with each of the neurons, determining how it calculates its output based on its input connections. The goal of learning involves adjusting the weights (and optional thresholds) of the network to improve the accuracy of the result. This is done by minimizing the observed errors among sample observations.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0066", "text": "Neural networks are typically trained through empirical risk minimization, which is based on the idea of optimizing the network's parameters to minimize the difference, or empirical risk, between the predicted output and the actual target values in a given dataset. Gradient-based methods such as backpropagation are usually used to estimate the parameters of the network", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0067", "text": ". During the training phase, ANNs learn from labeled training data by iteratively updating their parameters to minimize a defined loss function. This method allows the network to generalize to unseen data.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0068", "text": "Practically this is done by defining a loss function that is evaluated periodically during learning. As long as its output continues to decline, learning continues. The cost is frequently defined as a statistic whose value can only be approximated. The outputs are actually numbers, so when the error is low, the difference between the output (almost certainly a cat) and the correct answer (cat) is small. Learning attempts to reduce the total of the differences across the observations", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0069", "text": ". Most learning models can be viewed as a straightforward application of optimization theory and statistical estimation.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0070", "text": "Learning is typically considered complete when examining additional observations does not usefully reduce the error rate. Even after learning, the error rate typically does not reach 0. If after learning, the error rate is too high, the network typically must be redesigned.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0071", "text": "Loss function\nWhile it is possible to define a loss function ad hoc, frequently the choice is determined by the function's desirable properties (such as convexity, differentiability, and robustness)  because it arises from the model (e.g. in a probabilistic model, the model's posterior probability can be used as an inverse cost).", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0072", "text": "Backpropagation", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0073", "text": "Backpropagation is a method used to adjust the connection weights to compensate for each error found during learning. The error amount is effectively divided among the connections. Technically, backpropagation calculates the gradient (the derivative) of the loss function associated with a given state with respect to the weights", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0074", "text": ". The weight updates can be done via stochastic gradient descent or other methods, such as extreme learning machines, \"no-prop\" networks, training without backtracking, \"weightless\" networks, and non-connectionist neural networks.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0075", "text": "Hyperparameters\nA hyperparameter is a parameter defining any configurable part of the learning process, whose value is set prior to training. Examples of hyperparameters include learning rate, batch size and regularization parameters.. The performance of a neural network is strongly influenced by the choice of hyperparameter values, and thus the hyperparameters are often optimized as part of the training process, a process called hyperparameter tuning or hyperparameter optimization.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0076", "text": "Learning rate", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0077", "text": "The learning rate defines the size of the corrective steps that the model takes to adjust for errors in each observation. A high learning rate shortens the training time, but with lower ultimate accuracy, while a lower learning rate takes longer, but with the potential for greater accuracy. Optimizations such as Quickprop are primarily aimed at speeding up error minimization, while other improvements mainly try to increase reliability", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0078", "text": ". In order to avoid oscillation inside the network such as alternating connection weights, and to improve the rate of convergence, refinements use an adaptive learning rate that increases or decreases as appropriate. The concept of momentum allows the balance between the gradient and the previous change to be weighted such that the weight adjustment depends to some degree on the previous change. A momentum close to 0 emphasizes the gradient, while a value close to 1 emphasizes the last change.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0079", "text": "Learning paradigms\nMachine learning is commonly separated into three main learning paradigms, supervised learning, unsupervised learning and reinforcement learning. Each corresponds to a particular learning task.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0080", "text": "Supervised learning", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0081", "text": "Supervised learning uses a set of paired inputs and desired outputs. The learning task is to produce the desired output for each input. In this case, the cost function is related to eliminating incorrect deductions. A commonly used cost is the mean-squared error, which tries to minimize the average squared error between the network's output and the desired output", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0082", "text": ". Tasks suited for supervised learning are pattern recognition (also known as classification) and regression (also known as function approximation). Supervised learning is also applicable to sequential data (e.g., for handwriting, speech and gesture recognition). This can be thought of as learning with a \"teacher\", in the form of a function that provides continuous feedback on the quality of solutions obtained thus far.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0083", "text": "Unsupervised learning\nIn unsupervised learning, input data is given along with the cost function, some function of the data \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\textstyle x}\n  \n and the network's output. The cost function is dependent on the task (the model domain) and any a priori assumptions (the implicit properties of the model, its parameters and the observed variables). As a trivial example, consider the model \n  \n    \n      \n        \n          f", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0084", "text": "f\n          (\n          x\n          )\n          =\n          a\n        \n      \n    \n    {\\displaystyle \\textstyle f(x)=a}\n  \n where \n  \n    \n      \n        \n          a\n        \n      \n    \n    {\\displaystyle \\textstyle a}\n  \n is a constant and the cost \n  \n    \n      \n        \n          C\n          =\n          E\n          [\n          (\n          x\n          \u2212\n          f\n          (\n          x\n          )\n          \n            )\n            \n              2", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0085", "text": ")\n            \n              2\n            \n          \n          ]\n        \n      \n    \n    {\\displaystyle \\textstyle C=E[(x-f(x))^{2}]}\n  \n. Minimizing this cost produces a value of \n  \n    \n      \n        \n          a\n        \n      \n    \n    {\\displaystyle \\textstyle a}\n  \n that is equal to the mean of the data. The cost function can be much more complicated. Its form depends on the application: for example, in compression it could be related to the mutual information between", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0086", "text": "x\n        \n      \n    \n    {\\displaystyle \\textstyle x}\n  \n and \n  \n    \n      \n        \n          f\n          (\n          x\n          )\n        \n      \n    \n    {\\displaystyle \\textstyle f(x)}", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0087", "text": "{\\displaystyle \\textstyle f(x)}\n  \n, whereas in statistical modeling, it could be related to the posterior probability of the model given the data (note that in both of those examples, those quantities would be maximized rather than minimized). Tasks that fall within the paradigm of unsupervised learning are in general estimation problems; the applications include clustering, the estimation of statistical distributions, compression and filtering.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0088", "text": "Reinforcement learning", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0089", "text": "In applications such as playing video games, an actor takes a string of actions, receiving a generally unpredictable response from the environment after each one. The goal is to win the game, i.e., generate the most positive (lowest cost) responses. In reinforcement learning, the aim is to weight the network (devise a policy) to perform actions that minimize long-term (expected cumulative) cost", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0090", "text": ". At each point in time the agent performs an action and the environment generates an observation and an instantaneous cost, according to some (usually unknown) rules. The rules and the long-term cost usually only can be estimated. At any juncture, the agent decides whether to explore new actions to uncover their costs or to exploit prior learning to proceed more quickly.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0091", "text": "Formally, the environment is modeled as a Markov decision process (MDP) with states \n  \n    \n      \n        \n          \n            \n              s\n              \n                1\n              \n            \n            ,\n            .\n            .\n            .\n            ,\n            \n              s\n              \n                n\n              \n            \n          \n          \u2208\n          S\n        \n      \n    \n    {\\displaystyle \\textstyle {s_{1},...,s_{n}}\\in S}\n  \n and actions", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0092", "text": "and actions \n  \n    \n      \n        \n          \n            \n              a\n              \n                1\n              \n            \n            ,\n            .\n            .\n            .\n            ,\n            \n              a\n              \n                m\n              \n            \n          \n          \u2208\n          A\n        \n      \n    \n    {\\displaystyle \\textstyle {a_{1},...,a_{m}}\\in A}", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0093", "text": ". Because the state transitions are not known, probability distributions are used instead: the instantaneous cost distribution \n  \n    \n      \n        \n          P\n          (\n          \n            c\n            \n              t\n            \n          \n          \n            |\n          \n          \n            s\n            \n              t\n            \n          \n          )\n        \n      \n    \n    {\\displaystyle \\textstyle P(c_{t}|s_{t})}\n  \n, the observation distribution", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0094", "text": ", the observation distribution \n  \n    \n      \n        \n          P\n          (\n          \n            x\n            \n              t\n            \n          \n          \n            |\n          \n          \n            s\n            \n              t\n            \n          \n          )\n        \n      \n    \n    {\\displaystyle \\textstyle P(x_{t}|s_{t})}\n  \n and the transition distribution \n  \n    \n      \n        \n          P\n          (\n          \n            s\n            \n              t", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0095", "text": "s\n            \n              t\n              +\n              1\n            \n          \n          \n            |\n          \n          \n            s\n            \n              t\n            \n          \n          ,\n          \n            a\n            \n              t\n            \n          \n          )\n        \n      \n    \n    {\\displaystyle \\textstyle P(s_{t+1}|s_{t},a_{t})}", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0096", "text": ", while a policy is defined as the conditional distribution over actions given the observations. Taken together, the two define a Markov chain (MC). The aim is to discover the lowest-cost MC.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0097", "text": "ANNs serve as the learning component in such applications. Dynamic programming coupled with ANNs (giving neurodynamic programming) has been applied to problems such as those involved in vehicle routing, video games, natural resource management and medicine because of ANNs ability to mitigate losses of accuracy even when reducing the discretization grid density for numerically approximating the solution of control problems", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0098", "text": ". Tasks that fall within the paradigm of reinforcement learning are control problems, games and other sequential decision making tasks.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0099", "text": "Self-learning", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0100", "text": "Self-learning in neural networks was introduced in 1982 along with a neural network capable of self-learning named crossbar adaptive array (CAA). It is a system with only one input, situation s, and only one output, action (or behavior) a. It has neither external advice input nor external reinforcement input from the environment. The CAA computes, in a crossbar fashion, both decisions about actions and emotions (feelings) about encountered situations", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0101", "text": ". The system is driven by the interaction between cognition and emotion. Given the memory matrix, W =||w(a,s)||, the crossbar self-learning algorithm in each iteration performs the following computation:", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0102", "text": "In situation s perform action a;\n Receive consequence situation s';\n Compute emotion of being in consequence situation v(s');\n Update crossbar memory w'(a,s) = w(a,s) + v(s').", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0103", "text": "The backpropagated value (secondary reinforcement) is the emotion toward the consequence situation. The CAA exists in two environments, one is behavioral environment where it behaves, and the other is genetic environment, where from it receives initial emotions (only once) about to be encountered situations in the behavioral environment", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0104", "text": ". Having received the genome vector (species vector) from the genetic environment, the CAA will learn a goal-seeking behavior, in the behavioral environment that contains both desirable and undesirable situations.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0105", "text": "Neuroevolution\nNeuroevolution can create neural network topologies and weights using evolutionary computation. It is competitive with sophisticated gradient descent approaches. One advantage of neuroevolution is that it may be less prone to get caught in \"dead ends\".", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0106", "text": "Stochastic neural network", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0107", "text": "Stochastic neural networks originating from Sherrington\u2013Kirkpatrick models are a type of artificial neural network built by introducing random variations into the network, either by giving the network's artificial neurons stochastic transfer functions, or by giving them stochastic weights. This makes them useful tools for optimization problems, since the random fluctuations help the network escape from local minima", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0108", "text": ". Stochastic neural networks trained using a Bayesian approach are known as Bayesian neural networks.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0109", "text": "Topological deep learning", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0110", "text": "Topological deep learning, first introduced in 2017, is an emerging approach in machine learning that integrates topology with deep neural networks to address highly intricate and high-order data. Initially rooted in algebraic topology, TDL has since evolved into a versatile framework incorporating tools from other mathematical disciplines, such as differential topology and geometric topology", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0111", "text": ". As a successful example of mathematical deep learning, TDL continues to inspire advancements in mathematical artificial intelligence, fostering a mutually beneficial relationship between AI and mathematics.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0112", "text": "Other\nIn a Bayesian framework, a distribution over the set of allowed models is chosen to minimize the cost. Evolutionary methods, gene expression programming, simulated annealing, expectation\u2013maximization, non-parametric methods and particle swarm optimization are other learning algorithms. Convergent recursion is a learning algorithm for cerebellar model articulation controller (CMAC) neural networks.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0113", "text": "Modes", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0114", "text": "Two modes of learning are available: stochastic and batch. In stochastic learning, each input creates a weight adjustment. In batch learning, weights are adjusted based on a batch of inputs, accumulating errors over the batch. Stochastic learning introduces \"noise\" into the process, using the local gradient calculated from one data point; this reduces the chance of the network getting stuck in local minima", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0115", "text": ". However, batch learning typically yields a faster, more stable descent to a local minimum, since each update is performed in the direction of the batch's average error. A common compromise is to use \"mini-batches\", small batches with samples in each batch selected stochastically from the entire data set.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0116", "text": "Types", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0117", "text": "ANNs have evolved into a broad family of techniques that have advanced the state of the art across multiple domains. The simplest types have one or more static components, including number of units, number of layers, unit weights and topology. Dynamic types allow one or more of these to evolve via learning. The latter is much more complicated but can shorten learning periods and produce better results", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0118", "text": ". Some types allow/require learning to be \"supervised\" by the operator, while others operate independently. Some types operate purely in hardware, while others are purely software and run on general purpose computers.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0119", "text": "Some of the main breakthroughs include:", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0120", "text": "Convolutional neural networks (CNN) that have proven particularly successful in processing visual and other two-dimensional data; where long short-term memory avoids the vanishing gradient problem and can handle signals that have a mix of low and high frequency components aiding large-vocabulary speech recognition, text-to-speech synthesis, and photo-real talking heads. Convolutional Neural Networks have also been applied to fraud detection.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0121", "text": "Competitive networks such as generative adversarial networks in which multiple networks (of varying structure) compete with each other, on tasks such as winning a game or on deceiving the opponent about the authenticity of an input.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0122", "text": "Network design\nUsing artificial neural networks requires an understanding of their characteristics.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0123", "text": "Choice of model: This depends on the data representation and the application. Model parameters include the number, type, and connectedness of network layers, as well as the size of each and the connection type (full, pooling, etc.). Overly complex models learn slowly.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0124", "text": "Learning algorithm: Numerous trade-offs exist between learning algorithms. Almost any algorithm will work well with the correct hyperparameters for training on a particular data set. However, selecting and tuning an algorithm for training on unseen data requires significant experimentation.\nRobustness: If the model, cost function and learning algorithm are selected appropriately, the resulting ANN can become robust.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0125", "text": "Neural architecture search (NAS) uses machine learning to automate ANN design. Various approaches to NAS have designed networks that compare well with hand-designed systems. The basic search algorithm is to propose a candidate model, evaluate it against a dataset, and use the results as feedback to teach the NAS network. Available systems include AutoML and AutoKeras. scikit-learn library provides functions to help with building a deep network from scratch", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0126", "text": ". We can then implement a deep network with TensorFlow or Keras.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0127", "text": "Hyperparameters must also be defined as part of the design (they are not learned), governing matters such as how many neurons are in each layer, learning rate, step, stride, depth, receptive field and padding (for CNNs), etc. The Python code snippet provides an overview of the training function, which uses the training dataset, number of hidden layer units, learning rate, and number of iterations as parameters:", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0128", "text": "Theoretical properties\nComputational power\nThe multilayer perceptron is a universal function approximator, as proven by the universal approximation theorem. However, the proof is not constructive regarding the number of neurons required, the network topology, the weights and the learning parameters.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0129", "text": "A specific recurrent architecture with rational-valued weights (as opposed to full precision real number-valued weights) has the power of a universal Turing machine, using a finite number of neurons and standard linear connections. Further, the use of irrational values for weights results in a machine with super-Turing power.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0130", "text": "Capacity\nA model's \"capacity\" property corresponds to its ability to model any given function. It is related to the amount of information that can be stored in the network and to the notion of complexity.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0131", "text": "Two notions of capacity are known by the community. The information capacity and the VC Dimension. The information capacity of a perceptron is intensively discussed in Sir David MacKay's book which summarizes work by Thomas Cover. The capacity of a network of standard neurons (not convolutional) can be derived by four rules that derive from understanding a neuron as an electrical element. The information capacity captures the functions modelable by the network given any data as input", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0132", "text": ". The second notion, is the VC dimension. VC Dimension uses the principles of measure theory and finds the maximum capacity under the best possible circumstances. This is, given input data in a specific form. As noted in, the VC Dimension for arbitrary inputs is half the information capacity of a perceptron. The VC Dimension for arbitrary points is sometimes referred to as Memory Capacity.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0133", "text": "Convergence\nModels may not consistently converge on a single solution, firstly because local minima may exist, depending on the cost function and the model. Secondly, the optimization method used might not guarantee to converge when it begins far from any local minimum. Thirdly, for sufficiently large data or parameters, some methods become impractical.\nAnother issue worthy to mention is that training may cross some saddle point which may lead the convergence to the wrong direction.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0134", "text": "The convergence behavior of certain types of ANN architectures are more understood than others. When the width of network approaches to infinity, the ANN is well described by its first order Taylor expansion throughout training, and so inherits the convergence behavior of affine models. Another example is when parameters are small, it is observed that ANNs often fit target functions from low to high frequencies", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0135", "text": ". This behavior is referred to as the spectral bias, or frequency principle, of neural networks. This phenomenon is the opposite to the behavior of some well studied iterative numerical schemes such as Jacobi method. Deeper neural networks have been observed to be more biased towards low frequency functions.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0136", "text": "Generalization and statistics\nApplications whose goal is to create a system that generalizes well to unseen examples, face the possibility of over-training. This arises in convoluted or over-specified systems when the network capacity significantly exceeds the needed free parameters.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0137", "text": "Two approaches address over-training. The first is to use cross-validation and similar techniques to check for the presence of over-training and to select hyperparameters to minimize the generalization error. The second is to use some form of regularization", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0138", "text": ". The second is to use some form of regularization. This concept emerges in a probabilistic (Bayesian) framework, where regularization can be performed by selecting a larger prior probability over simpler models; but also in statistical learning theory, where the goal is to minimize over two quantities: the 'empirical risk' and the 'structural risk', which roughly corresponds to the error over the training set and the predicted error in unseen data due to overfitting.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0139", "text": "Supervised neural networks that use a mean squared error (MSE) cost function can use formal statistical methods to determine the confidence of the trained model. The MSE on a validation set can be used as an estimate for variance. This value can then be used to calculate the confidence interval of network output, assuming a normal distribution", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0140", "text": ". A confidence analysis made this way is statistically valid as long as the output probability distribution stays the same and the network is not modified.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0141", "text": "By assigning a softmax activation function, a generalization of the logistic function, on the output layer of the neural network (or a softmax component in a component-based network) for categorical target variables, the outputs can be interpreted as posterior probabilities. This is useful in classification as it gives a certainty measure on classifications.\nThe softmax activation function is:", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0142", "text": "y\n          \n            i\n          \n        \n        =\n        \n          \n            \n              e\n              \n                \n                  x\n                  \n                    i\n                  \n                \n              \n            \n            \n              \n                \u2211\n                \n                  j\n                  =\n                  1\n                \n                \n                  c", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0143", "text": "c\n                \n              \n              \n                e\n                \n                  \n                    x\n                    \n                      j\n                    \n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle y_{i}={\\frac {e^{x_{i}}}{\\sum _{j=1}^{c}e^{x_{j}}}}}", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0144", "text": "Applications", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0145", "text": "Artificial neural networks (ANNs) have undergone significant advancements, particularly in their ability to model complex systems, handle large data sets, and adapt to various types of applications. Their evolution over the past few decades has been marked by a broad range of applications in fields such as image processing, speech recognition, natural language processing, finance, and medicine", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0146", "text": ". Artificial neural networks now form the core of many artificial intelligence systems, including natural language processing, computer vision, speech recognition, and autonomous systems.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0147", "text": "Because of their ability to model and reproduce nonlinear processes, artificial neural networks have found applications in many disciplines. These include:", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0148", "text": "Function approximation, or regression analysis, (including time series prediction, fitness approximation, and modeling)\nData processing (including filtering, clustering, blind source separation, and compression)\nNonlinear system identification and control (including vehicle control, trajectory prediction, adaptive control, process control, and natural resource management)", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0149", "text": "Pattern recognition (including radar systems, face identification, signal classification, novelty detection, 3D reconstruction, object recognition, and sequential decision making)\nSequence recognition (including gesture, speech, and handwritten and printed text recognition)\nSensor data analysis (including image analysis)\nRobotics (including directing manipulators and prostheses)\nData mining (including knowledge discovery in databases)", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0150", "text": "Finance (such as ex-ante models for specific financial long-run forecasts and artificial financial markets)\nQuantum chemistry\nGeneral game playing\nGenerative AI\nData visualization\nMachine translation\nSocial network filtering\nE-mail spam filtering\nMedical diagnosis\nANNs have been used to diagnose several types of cancers and to distinguish highly invasive cancer cell lines from less invasive lines using only cell shape information.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0151", "text": "ANNs have been used to accelerate reliability analysis of infrastructures subject to natural disasters and to predict foundation settlements. It can also be useful to mitigate flood by the use of ANNs for modelling rainfall-runoff. ANNs have also been used for building black-box models in geoscience: hydrology, ocean modelling and coastal engineering, and geomorphology. ANNs have been employed in cybersecurity, with the objective to discriminate between legitimate activities and malicious ones", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0152", "text": ". For example, machine learning has been used for classifying Android malware, for identifying domains belonging to threat actors and for detecting URLs posing a security risk. Research is underway on ANN systems designed for penetration testing, for detecting botnets, credit cards frauds and network intrusions.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0153", "text": "ANNs have been proposed as a tool to solve partial differential equations in physics and simulate the properties of many-body open quantum systems. In brain research ANNs have studied short-term behavior of individual neurons, the dynamics of neural circuitry arise from interactions between individual neurons and how behavior can arise from abstract neural modules that represent complete subsystems", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0154", "text": ". Studies considered long-and short-term plasticity of neural systems and their relation to learning and memory from the individual neuron to the system level.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0155", "text": "It is possible to create a profile of a user's interests from pictures, using artificial neural networks trained for object recognition.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0156", "text": "Beyond their traditional applications, artificial neural networks are increasingly being utilized in interdisciplinary research, such as materials science. For instance, graph neural networks (GNNs) have demonstrated their capability in scaling deep learning for the discovery of new stable materials by efficiently predicting the total energy of crystals", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0157", "text": ". This application underscores the adaptability and potential of ANNs in tackling complex problems beyond the realms of predictive modeling and artificial intelligence, opening new pathways for scientific discovery and innovation.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0158", "text": "Image processing", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0159", "text": "In the realm of image processing, ANNs are employed in computer vision tasks such as image classification, object recognition, and image segmentation. For instance, deep convolutional neural networks (CNNs) have been important in handwritten digit recognition, achieving state-of-the-art performance. This demonstrates the ability of ANNs to effectively process and interpret complex visual information, leading to advancements in fields ranging from automated surveillance to medical imaging.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0160", "text": "Speech recognition\nBy modeling speech signals, ANNs are used for tasks like speaker identification and speech-to-text conversion. Deep neural network architectures have introduced significant improvements in large vocabulary continuous speech recognition, outperforming traditional techniques. These advancements have enabled the development of more accurate and efficient voice-activated systems, enhancing user interfaces in technology products.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0161", "text": "Natural language processing\nIn natural language processing, ANNs are used for tasks such as text classification, sentiment analysis, and machine translation. They have enabled the development of models that can accurately translate between languages, understand the context and sentiment in textual data, and categorize text based on content. This has implications for automated customer service, content moderation, and language understanding technologies.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0162", "text": "Control systems\nIn the domain of control systems, ANNs are used to model dynamic systems for tasks such as system identification, control design, and optimization. For instance, deep feedforward neural networks are important in system identification and control applications.\n\nFinance\nANNs are used for stock market prediction and credit scoring:", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0163", "text": "In investing, ANNs can process vast amounts of financial data, recognize complex patterns, and forecast stock market trends, aiding investors and risk managers in making informed decisions.\nIn credit scoring, ANNs offer data-driven, personalized assessments of creditworthiness, improving the accuracy of default predictions and automating the lending process.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0164", "text": "ANNs require high-quality data and careful tuning, and their \"black-box\" nature can pose challenges in interpretation. Nevertheless, ANNs continue to play a role in finance, offering insights and enhancing risk management strategies.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0165", "text": "Medicine", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0166", "text": "ANNs are able to process and analyze vast medical datasets. They enhance diagnostic accuracy, especially by interpreting complex medical imaging for early disease detection, and by predicting patient outcomes for personalized treatment planning. In drug discovery, ANNs speed up the identification of potential drug candidates and predict their efficacy and safety, significantly reducing development time and costs", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0167", "text": ". Additionally, their application in personalized medicine and healthcare data analysis allows tailored therapies and efficient patient care management. Ongoing research is aimed at addressing remaining challenges such as data privacy and model interpretability, as well as expanding the scope of ANN applications in medicine.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0168", "text": "Cybersecurity\nNeural networks are widely applied in cybersecurity for tasks such as anomaly detection, malware classification, and intrusion detection. By learning patterns of normal system or network behavior, neural models can identify deviations that may indicate malicious activity.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0169", "text": "Content creation", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0170", "text": "ANNs such as generative adversarial networks and transformers are used for content creation across numerous industries. Deep learning models are able to learn the style of an artist or musician from huge datasets and generate completely new artworks and music compositions. For instance, DALL-E is a deep neural network trained on 650 million pairs of images and texts across the internet that can create artworks based on text entered by the user", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0171", "text": ". In the field of music, transformers have been used to create original music for commercials and documentaries through companies such as AIVA and Jukedeck. In the marketing industry, generative models have been used to create personalized advertisements for consumers. Additionally, major film companies are partnering with technology companies to analyze the financial success of a film, such as the partnership between Warner Bros and technology company Cinelytic established in 2020", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0172", "text": ". Furthermore, neural networks have found uses in video game creation, where non-player characters can make decisions based on all the characters currently in the game.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0173", "text": "Criticism\nTraining\nA common criticism of neural networks, particularly in robotics, is that they require too many training samples for real-world operation.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0174", "text": "Any learning machine needs sufficient representative examples in order to capture the underlying structure that allows it to generalize to new cases. Potential solutions include randomly shuffling training examples, by using a numerical optimization algorithm that does not take too large steps when changing the network connections following an example, grouping examples in so-called mini-batches and/or introducing a recursive least squares algorithm for CMAC.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0175", "text": "Dean Pomerleau uses a neural network to train a robotic vehicle to drive on multiple types of roads (single lane, multi-lane, dirt, etc.), and a large amount of his research is devoted to extrapolating multiple training scenarios from a single training experience, and preserving past training diversity so that the system does not become overtrained (if, for example, it is presented with a series of right turns\u2014it should not learn to always turn right).", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0176", "text": "Theory\nA central claim of ANNs is that they embody new and powerful general principles for processing information. These principles are ill-defined. This allows simple statistical association (the basic function of artificial neural networks) to be described as learning or recognition. In 1997, Alexander Dewdney, a former Scientific American columnist, commented that as a result, artificial neural networks have a", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0177", "text": "something-for-nothing quality, one that imparts a peculiar aura of laziness and a distinct lack of curiosity about just how good these computing systems are. No human hand (or mind) intervenes; solutions are found as if by magic; and no one, it seems, has learned anything. One response to Dewdney is that neural networks have been successfully used to handle many complex and diverse tasks, ranging from autonomously flying aircraft to detecting credit card fraud to mastering the game of Go.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0178", "text": "Technology writer Roger Bridgman commented:", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0179", "text": "Neural networks, for instance, are in the dock not only because they have been hyped to high heaven, (what hasn't?) but also because you could create a successful net without understanding how it worked: the bunch of numbers that captures its behaviour would in all probability be \"an opaque, unreadable table...valueless as a scientific resource\".", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0180", "text": "In spite of his emphatic declaration that science is not technology, Dewdney seems here to pillory neural nets as bad science when most of those devising them are just trying to be good engineers. An unreadable table that a useful machine could read would still be well worth having.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0181", "text": "Although it is true that analyzing what has been learned by an artificial neural network is difficult, it is much easier to do so than to analyze what has been learned by a biological neural network. Moreover, recent emphasis on the explainability of AI has contributed towards the development of methods, notably those based on attention mechanisms, for visualizing and explaining learned neural networks and decomposing how circuits chain to reach a given goal", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0182", "text": ". Furthermore, researchers involved in exploring learning algorithms for neural networks are gradually uncovering generic principles that allow a learning machine to be successful. For example, Bengio and LeCun (2007) wrote an article regarding local vs non-local learning, as well as shallow vs deep architecture.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0183", "text": "Biological brains use both shallow and deep circuits as reported by brain anatomy, displaying a wide variety of invariance. Weng argued that the brain self-wires largely according to signal statistics and therefore, a serial cascade cannot catch all major statistical dependencies.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0184", "text": "Hardware\nLarge and effective neural networks require considerable computing resources. While the brain has hardware tailored to the task of processing signals through a graph of neurons, simulating even a simplified neuron on von Neumann architecture may consume vast amounts of memory and storage. Furthermore, the designer often needs to transmit signals through many of these connections and their associated neurons \u2013 which require enormous CPU power and time.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0185", "text": "Some argue that the resurgence of neural networks in the twenty-first century is largely attributable to advances in hardware: from 1991 to 2015, computing power, especially as delivered by GPGPUs (on GPUs), has increased around a million-fold, making the standard backpropagation algorithm feasible for training networks that are several layers deeper than before. The use of accelerators such as FPGAs and GPUs can reduce training times from months to days.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0186", "text": "Neuromorphic engineering or a physical neural network addresses the hardware difficulty directly, by constructing non-von-Neumann chips to directly implement neural networks in circuitry. Another type of chip optimized for neural network processing is called a Tensor Processing Unit, or TPU.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0187", "text": "Concept drift\nWhen neural networks are deployed in real-world applications, the statistical properties of the input data may change over time, a phenomenon known as concept drift or non-stationarity. Drift can reduce predictive accuracy and lead to unreliable or biased decisions if it is not detected and corrected. In practice, this means that the model's accuracy in deployment may differ substantially from the levels observed during training or cross-validation.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0188", "text": "Several strategies have been developed to monitor neural networks for drift and degradation:", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0189", "text": "Error-based monitoring: comparing current predictions against ground-truth labels when they become available. This approach directly quantifies predictive performance but may be impractical when labels are delayed or costly to obtain.\nData distribution monitoring: detecting changes in the input data distribution using statistical tests, divergence measures, or density-ratio estimation.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0190", "text": "Representation monitoring: tracking the distribution of internal embeddings or hidden-layer features. Shifts in the latent representation can indicate nonstationarity even when labels are unavailable. Statistical methods such as statistical process control charts have been adapted for this purpose.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0191", "text": "Dataset bias", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0192", "text": "Neural networks are dependent on the quality of the data they are trained on, thus low quality data with imbalanced representativeness can lead to the model learning and perpetuating societal biases. These inherited biases become especially critical when the ANNs are integrated into real-world scenarios where the training data may be imbalanced due to the scarcity of data for a specific race, gender or other attribute", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0193", "text": ". This imbalance can result in the model having inadequate representation and understanding of underrepresented groups, leading to discriminatory outcomes that exacerbate societal inequalities, especially in applications like facial recognition, hiring processes, and law enforcement. For example, in 2018, Amazon had to scrap a recruiting tool because the model favored men over women for jobs in software engineering due to the higher number of male workers in the field", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0194", "text": ". The program would penalize any resume with the word \"woman\" or the name of any women's college. However, the use of synthetic data can help reduce dataset bias and increase representation in datasets.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0195", "text": "Lack of interpretability\nMany deep learning models operate as opaque \"black box\" systems, making it difficult to interpret their internal decision-making processes. Neural networks can also be vulnerable to adversarial examples, where small perturbations to input data can cause incorrect predictions.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0196", "text": "These concerns have led to increased research in explainable artificial intelligence (XAI), robust machine learning, and hybrid AI approaches that combine neural learning with symbolic reasoning.  Advocates of hybrid models also say that such a mixture can better capture the mechanisms of the human mind.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0197", "text": "Practical counterexamples\nAnalyzing what has been learned by an ANN is much easier than analyzing what has been learned by a biological neural network. Furthermore, researchers involved in exploring learning algorithms for neural networks are gradually uncovering general principles that allow a learning machine to be successful \u2013 for example, local vs. non-local learning and shallow vs. deep architecture.\n\nGallery\nSee also\nNotes\nReferences\nBibliography\nExternal links", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0198", "text": "A Brief Introduction to Neural Networks (D. Kriesel) \u2013 Illustrated, bilingual manuscript about artificial neural networks; Topics so far: Perceptrons, Backpropagation, Radial Basis Functions, Recurrent Neural Networks, Self Organizing Maps, Hopfield Networks.\nReview of Neural Networks in Materials Science Archived 7 June 2015 at the Wayback Machine\nArtificial Neural Networks Tutorial in three languages (Univ. Polit\u00e9cnica de Madrid)\nAnother introduction to ANN", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0199", "text": "Another introduction to ANN\nNext Generation of Neural Networks Archived 24 January 2011 at the Wayback Machine \u2013 Google Tech Talks\nPerformance of Neural Networks\nNeural Networks and Information Archived 9 July 2009 at the Wayback Machine\nSanderson G (5 October 2017). \"But what is a Neural Network?\". 3Blue1Brown. Archived from the original on 7 November 2021 \u2013 via YouTube.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Neural_network_machine_learning_0200", "text": "Hakim MA, Alam MI (2025). \"Biologically inspired neural network layer with homeostatic regulation and adaptive repair mechanisms\". Scientific Reports. 15 (1). Nature Publishing Group: 33903. Bibcode:2025NatSR..1533903H. doi:10.1038/s41598-025-09114-8. ISSN 2045-2322. PMC 12484884. PMID 41028030.", "title": "Neural network (machine learning)", "url": "https://en.wikipedia.org/wiki/Neural_network_(machine_learning)"}, {"id": "rec_Perceptron_0000", "text": "In machine learning, the perceptron is an algorithm for supervised learning of binary classifiers.  A binary classifier is a function that can decide whether or not an input, represented by a vector of numbers, belongs to some specific class.  It is a type of linear classifier, i.e. a classification algorithm that makes its predictions based on a linear predictor function combining a set of weights with the feature vector.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0001", "text": "History\nThe artificial neuron network was invented in 1943 by Warren McCulloch and Walter Pitts in A logical calculus of the ideas immanent in nervous activity.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0002", "text": "In 1957, Frank Rosenblatt was at the Cornell Aeronautical Laboratory. He simulated the perceptron on an IBM 704. Later, he obtained funding by the Information Systems Branch of the United States Office of Naval Research and the Rome Air Development Center, to build a custom-made computer, the Mark I Perceptron. It was first publicly demonstrated on 23 June 1960", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0003", "text": ". The machine was \"part of a previously secret four-year NPIC [the US' National Photographic Interpretation Center] effort from 1963 through 1966 to develop this algorithm into a useful tool for photo-interpreters\".", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0004", "text": "Rosenblatt described the details of the perceptron in a 1958 paper. His organization of a perceptron is constructed of three kinds of cells (\"units\"): AI, AII, R, which stand for \"projection\", \"association\" and \"response\". He presented at the first international symposium on AI, Mechanisation of Thought Processes, which took place in 1958 November.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0005", "text": "Rosenblatt's project was funded under Contract Nonr-401(40) \"Cognitive Systems Research Program\", which lasted from 1959 to 1970, and Contract Nonr-2381(00) \"Project PARA\" (\"PARA\" means \"Perceiving and Recognition Automata\"), which lasted from 1957 to 1963.\nIn 1959, the Institute for Defense Analysis awarded his group a $10,000 contract. By September 1961, the ONR awarded further $153,000 worth of contracts, with $108,000 committed for 1962.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0006", "text": "The ONR research manager, Marvin Denicoff, stated that ONR, instead of ARPA, funded the Perceptron project, because the project was unlikely to produce technological results in the near or medium term. Funding from ARPA go up to the order of millions dollars, while from ONR are on the order of 10,000 dollars. Meanwhile, the head of IPTO at ARPA, J.C.R", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0007", "text": ". Meanwhile, the head of IPTO at ARPA, J.C.R. Licklider, was interested in 'self-organizing', 'adaptive' and other biologically-inspired methods in the 1950s; but by the mid-1960s he was openly critical of these, including the perceptron. Instead he strongly favored the logical AI approach of Simon and Newell.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0008", "text": "Mark I Perceptron machine\nThe perceptron was intended to be a machine, rather than a program, and while its first implementation was in software for the IBM 704, it was subsequently implemented in custom-built hardware as the Mark I Perceptron with the project name \"Project PARA\", designed for image recognition. The machine is currently in Smithsonian National Museum of American History.\nThe Mark I Perceptron had three layers. One version was implemented as follows:", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0009", "text": "An array of 400 photocells arranged in a 20x20 grid, named \"sensory units\" (S-units), or \"input retina\". Each S-unit can connect to up to 40 A-units.\nA hidden layer of 512 perceptrons, named \"association units\" (A-units).\nAn output layer of eight perceptrons, named \"response units\" (R-units).\nRosenblatt called this three-layered perceptron network the alpha-perceptron, to distinguish it from other perceptron models he experimented with.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0010", "text": "The S-units are connected to the A-units randomly (according to a table of random numbers) via a plugboard (see photo), to \"eliminate any particular intentional bias in the perceptron\". The connection weights are fixed, not learned. Rosenblatt was adamant about the random connections, as he believed the retina was randomly connected to the visual cortex, and he wanted his perceptron machine to resemble human visual perception.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0011", "text": "The A-units are connected to the R-units, with adjustable weights encoded in potentiometers, and weight updates during learning were performed by electric motors.The hardware details are in an operators' manual.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0012", "text": "In a 1958 press conference organized by the US Navy, Rosenblatt made statements about the perceptron that caused a heated controversy among the fledgling AI community; based on Rosenblatt's statements, The New York Times reported the perceptron to be \"the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.\"", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0013", "text": "The Photo Division of Central Intelligence Agency, from 1960 to 1964, studied the use of Mark I Perceptron machine for recognizing militarily interesting silhouetted targets (such as planes and ships) in aerial photos.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0014", "text": "Principles of Neurodynamics (1962)\nRosenblatt described his experiments with many variants of the Perceptron machine in a book Principles of Neurodynamics (1962). The book is a published version of the 1961 report.\nAmong the variants are:", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0015", "text": "\"cross-coupling\" (connections between units within the same layer) with possibly closed loops,\n\"back-coupling\" (connections from units in a later layer to units in a previous layer),\nfour-layer perceptrons where the last two layers have adjustable weights (and thus a proper multilayer perceptron),\nincorporating time-delays to perceptron units, to allow for processing sequential data,\nanalyzing audio (instead of images).", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0016", "text": "analyzing audio (instead of images).\nThe machine was shipped from Cornell to Smithsonian in 1967, under a government transfer administered by the Office of Naval Research.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0017", "text": "Perceptrons (1969)\nAlthough the perceptron initially seemed promising, it was quickly proved that perceptrons could not be trained to recognise many classes of patterns. This caused the field of neural network research to stagnate for many years, before it was recognised that a feedforward neural network with two or more layers (also called a multilayer perceptron) had greater processing power than perceptrons with one layer (also called a single-layer perceptron).", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0018", "text": "Single-layer perceptrons are only capable of learning linearly separable patterns. For a classification task with some step activation function, a single node will have a single line dividing the data points forming the patterns. More nodes can create more dividing lines, but those lines must somehow be combined to form more complex classifications. A second layer of perceptrons, or even linear nodes, are sufficient to solve many otherwise non-separable problems.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0019", "text": "In 1969, a famous book entitled Perceptrons by Marvin Minsky and Seymour Papert showed that it was impossible for these classes of network to learn an XOR function. It is often incorrectly believed that they also conjectured that a similar result would hold for a multi-layer perceptron network. However, this is not true, as both Minsky and Papert already knew that multi-layer perceptrons were capable of producing an XOR function", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0020", "text": ". (See the page on Perceptrons (book) for more information.) Nevertheless, the often-miscited Minsky and Papert text caused a significant decline in interest and funding of neural network research. It took ten more years until neural network research experienced a resurgence in the 1980s. This text was reprinted in 1987 as \"Perceptrons - Expanded Edition\" where some errors in the original text are shown and corrected.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0021", "text": "Subsequent work\nRosenblatt continued working on perceptrons despite diminishing funding. The last attempt was Tobermory, built between 1961 and 1967, built for speech recognition. It occupied an entire room. It had 4 layers with 12,000 weights implemented by toroidal magnetic cores. By the time of its completion, simulation on digital computers had become faster than purpose-built perceptron machines. He died in a boating accident in 1971.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0022", "text": "A simulation program for neural networks was written for IBM 7090/7094, and was used to study various pattern recognition applications, such as character recognition, particle tracks in bubble-chamber photographs; phoneme, isolated word, and continuous speech recognition; speaker verification; and center-of-attention mechanisms for image processing.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0023", "text": "The kernel perceptron algorithm was already introduced in 1964 by Aizerman et al. Margin bounds guarantees were given for the Perceptron algorithm in the general non-separable case first by Freund and Schapire (1998), and more recently by Mohri and Rostamizadeh (2013) who extend previous results and give new and more favorable L1 bounds.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0024", "text": "The perceptron is a simplified model of a biological neuron. While the complexity of biological neuron models is often required to fully understand neural behavior, research suggests a perceptron-like linear model can produce some behavior seen in real neurons.\nThe solution spaces of decision boundaries for all binary functions and learning behaviors are studied in.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0025", "text": "Definition\nIn the modern sense, the perceptron is an algorithm for learning a binary classifier called a threshold function: a function that maps its input \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n (a real-valued vector) to an output value \n  \n    \n      \n        f\n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle f(\\mathbf {x} )}\n  \n (a single binary value):", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0026", "text": "f\n        (\n        \n          x\n        \n        )\n        =\n        h\n        (\n        \n          w\n        \n        \u22c5\n        \n          x\n        \n        +\n        b\n        )\n      \n    \n    {\\displaystyle f(\\mathbf {x} )=h(\\mathbf {w} \\cdot \\mathbf {x} +b)}", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0027", "text": "where \n  \n    \n      \n        h\n      \n    \n    {\\displaystyle h}\n  \n is the Heaviside step-function (where an input of \n  \n    \n      \n        >\n        0\n      \n    \n    {\\textstyle >0}\n  \n outputs 1; otherwise 0 is the output ), \n  \n    \n      \n        \n          w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n is a vector of real-valued weights, \n  \n    \n      \n        \n          w\n        \n        \u22c5\n        \n          x", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0028", "text": "x\n        \n      \n    \n    {\\displaystyle \\mathbf {w} \\cdot \\mathbf {x} }\n  \n is the dot product \n  \n    \n      \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            m\n          \n        \n        \n          w\n          \n            i\n          \n        \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\textstyle \\sum _{i=1}^{m}w_{i}x_{i}}", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0029", "text": "{\\textstyle \\sum _{i=1}^{m}w_{i}x_{i}}\n  \n, where m is the number of inputs to the perceptron, and b is the bias. The bias shifts the decision boundary away from the origin and does not depend on any input value.\nEquivalently, since \n  \n    \n      \n        \n          w\n        \n        \u22c5\n        \n          x\n        \n        +\n        b\n        =\n        (\n        \n          w\n        \n        ,\n        b\n        )\n        \u22c5\n        (\n        \n          x\n        \n        ,\n        1", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0030", "text": "x\n        \n        ,\n        1\n        )\n      \n    \n    {\\displaystyle \\mathbf {w} \\cdot \\mathbf {x} +b=(\\mathbf {w} ,b)\\cdot (\\mathbf {x} ,1)}\n  \n, we can add the bias term \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}\n  \n as another weight \n  \n    \n      \n        \n          \n            w\n          \n          \n            m\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {w} _{m+1}}\n  \n and add a coordinate", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0031", "text": "and add a coordinate \n  \n    \n      \n        1\n      \n    \n    {\\displaystyle 1}\n  \n to each input \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n, and then write it as a linear classifier that passes the origin:\n  \n    \n      \n        f\n        (\n        \n          x\n        \n        )\n        =\n        h\n        (\n        \n          w\n        \n        \u22c5\n        \n          x\n        \n        )", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0032", "text": "x\n        \n        )\n      \n    \n    {\\displaystyle f(\\mathbf {x} )=h(\\mathbf {w} \\cdot \\mathbf {x} )}", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0033", "text": "The binary value of \n  \n    \n      \n        f\n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle f(\\mathbf {x} )}\n  \n (0 or 1) is used to perform binary classification on \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n as either a positive or a negative instance. Spatially, the bias shifts the position (though not the orientation) of the planar decision boundary.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0034", "text": "In the context of neural networks, a perceptron is an artificial neuron using the Heaviside step function as the activation function. The perceptron algorithm is also termed the single-layer perceptron, to distinguish it from a multilayer perceptron, which is a misnomer for a more complicated neural network.  As a linear classifier, the single-layer perceptron is the simplest feedforward neural network.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0035", "text": "Power of representation\nInformation theory\nFrom an information theory point of view, a single perceptron with K inputs has a capacity of 2K bits of information. This result is due to Thomas Cover.\nSpecifically let \n  \n    \n      \n        T\n        (\n        N\n        ,\n        K\n        )\n      \n    \n    {\\displaystyle T(N,K)}\n  \n be the number of ways to linearly separate N points in K dimensions, then\n  \n    \n      \n        T\n        (\n        N\n        ,\n        K\n        )\n        =", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0036", "text": "N\n        ,\n        K\n        )\n        =\n        \n          {\n          \n            \n              \n                \n                  \n                    2\n                    \n                      N\n                    \n                  \n                \n                \n                  K\n                  \u2265\n                  N\n                \n              \n              \n                \n                  2\n                  \n                    \u2211", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0037", "text": "\u2211\n                    \n                      k\n                      =\n                      0\n                    \n                    \n                      K\n                      \u2212\n                      1\n                    \n                  \n                  \n                    (\n                    \n                      \n                        \n                          \n                            N\n                            \u2212\n                            1", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0038", "text": "1\n                          \n                        \n                        \n                          \n                            k\n                          \n                        \n                      \n                    \n                    )\n                  \n                \n                \n                  K\n                  <\n                  N", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0039", "text": "{\\displaystyle T(N,K)=\\left\\{{\\begin{array}{cc}2^{N}&K\\geq N\\\\2\\sum _{k=0}^{K-1}\\left({\\begin{array}{c}N-1\\\\k\\end{array}}\\right)&K<N\\end{array}}\\right.}\n  \nWhen K is large, \n  \n    \n      \n        T\n        (\n        N\n        ,\n        K\n        )\n        \n          /\n        \n        \n          2\n          \n            N\n          \n        \n      \n    \n    {\\displaystyle T(N,K)/2^{N}}\n  \n is very close to one when \n  \n    \n      \n        N", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0040", "text": "N\n        \u2264\n        2\n        K\n      \n    \n    {\\displaystyle N\\leq 2K}\n  \n, but very close to zero when \n  \n    \n      \n        N\n        >\n        2\n        K\n      \n    \n    {\\displaystyle N>2K}\n  \n. In words, one perceptron unit can almost certainly memorize a random assignment of binary labels on N points when \n  \n    \n      \n        N\n        \u2264\n        2\n        K\n      \n    \n    {\\displaystyle N\\leq 2K}\n  \n, but almost certainly not when \n  \n    \n      \n        N", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0041", "text": "N\n        >\n        2\n        K\n      \n    \n    {\\displaystyle N>2K}\n  \n.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0042", "text": "Boolean function\nWhen operating on only binary inputs, a perceptron is called a linearly separable Boolean function, or threshold Boolean function. The sequence of numbers of threshold Boolean functions on n inputs is OEIS A000609. The value is only known exactly up to \n  \n    \n      \n        n\n        =\n        9\n      \n    \n    {\\displaystyle n=9}\n  \n case, but the order of magnitude is known quite exactly: it has upper bound \n  \n    \n      \n        \n          2", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0043", "text": "2\n          \n            \n              n\n              \n                2\n              \n            \n            \u2212\n            n\n            \n              log\n              \n                2\n              \n            \n            \u2061\n            n\n            +\n            O\n            (\n            n\n            )\n          \n        \n      \n    \n    {\\displaystyle 2^{n^{2}-n\\log _{2}n+O(n)}}\n  \n and lower bound \n  \n    \n      \n        \n          2", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0044", "text": "2\n          \n            \n              n\n              \n                2\n              \n            \n            \u2212\n            n\n            \n              log\n              \n                2\n              \n            \n            \u2061\n            n\n            \u2212\n            O\n            (\n            n\n            )\n          \n        \n      \n    \n    {\\displaystyle 2^{n^{2}-n\\log _{2}n-O(n)}}\n  \n.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0045", "text": ".\nAny Boolean linear threshold function can be implemented with only integer weights. Furthermore, the number of bits necessary and sufficient for representing a single integer weight parameter is \n  \n    \n      \n        \u0398\n        (\n        n\n        ln\n        \u2061\n        n\n        )\n      \n    \n    {\\displaystyle \\Theta (n\\ln n)}\n  \n.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0046", "text": "Universal approximation theorem\nA single perceptron can learn to classify any half-space. It cannot solve any linearly nonseparable vectors, such as the Boolean exclusive-or problem (the famous \"XOR problem\").", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0047", "text": "A perceptron network with one hidden layer can learn to classify any compact subset arbitrarily closely. Similarly, it can also approximate any compactly-supported continuous function arbitrarily closely. This is essentially a special case of the theorems by George Cybenko and Kurt Hornik.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0048", "text": "Conjunctively local perceptron\nPerceptrons (Minsky and Papert, 1969) studied the kind of perceptron networks necessary to learn various Boolean functions.\nConsider a perceptron network with \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n input units, one hidden layer, and one output, similar to the Mark I Perceptron machine. It computes a Boolean function of type \n  \n    \n      \n        f\n        :\n        \n          2\n          \n            n\n          \n        \n        \u2192", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0049", "text": "n\n          \n        \n        \u2192\n        2\n      \n    \n    {\\displaystyle f:2^{n}\\to 2}\n  \n. They call a function conjunctively local of order \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n, iff there exists a perceptron network such that each unit in the hidden layer connects to at most \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n input units.\nTheorem. (Theorem 3.1.1): The parity function is conjunctively local of order \n  \n    \n      \n        n", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0050", "text": "n\n      \n    \n    {\\displaystyle n}\n  \n.\nTheorem. (Section 5.5): The connectedness function is conjunctively local of order \n  \n    \n      \n        \u03a9\n        (\n        \n          n\n          \n            1\n            \n              /\n            \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle \\Omega (n^{1/2})}\n  \n.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0051", "text": "Learning algorithm for a single-layer perceptron\nBelow is an example of a learning algorithm for a single-layer perceptron with a single output unit. For a single-layer perceptron with multiple output units, since the weights of one output unit are completely separate from all the others', the same algorithm can be run for each output unit.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0052", "text": "For multilayer perceptrons, where a hidden layer exists, more sophisticated algorithms such as backpropagation must be used. If the activation function or the underlying process being modeled by the perceptron is nonlinear, alternative learning algorithms such as the delta rule can be used as long as the activation function is differentiable. Nonetheless, the learning algorithm described in the steps below will often work, even for multilayer perceptrons with nonlinear activation functions.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0053", "text": "When multiple perceptrons are combined in an artificial neural network, each output neuron operates independently of all the others; thus, learning each output can be considered in isolation.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0054", "text": "Definitions\nWe first define some variables:\n\n  \n    \n      \n        r\n      \n    \n    {\\displaystyle r}\n  \n is the learning rate of the perceptron. Learning rate is a positive number usually chosen to be less than 1. The larger the value, the greater the chance for volatility in the weight changes.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0055", "text": "D\n        =\n        {\n        (\n        \n          \n            x\n          \n          \n            1\n          \n        \n        ,\n        \n          d\n          \n            1\n          \n        \n        )\n        ,\n        \u2026\n        ,\n        (\n        \n          \n            x\n          \n          \n            s\n          \n        \n        ,\n        \n          d\n          \n            s\n          \n        \n        )\n        }", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0056", "text": ")\n        }\n      \n    \n    {\\displaystyle D=\\{(\\mathbf {x} _{1},d_{1}),\\dots ,(\\mathbf {x} _{s},d_{s})\\}}\n  \n is the training set of \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n samples, where:", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0057", "text": "x\n          \n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{j}}\n  \n is the \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \nth input vector from the \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n-dimensional Euclidean space \n  \n    \n      \n        \n          \n            R\n          \n          \n            n", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0058", "text": "n\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {R} ^{n}}\n  \n.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0059", "text": "d\n          \n            j\n          \n        \n        \u2208\n        {\n        0\n        ,\n        1\n        }\n      \n    \n    {\\displaystyle d_{j}\\in \\{0,1\\}}\n  \n is the desired output value of the perceptron for that input.\nWe show the values of the features as follows:", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0060", "text": "x\n          \n            j\n            ,\n            i\n          \n        \n      \n    \n    {\\displaystyle x_{j,i}}\n  \n is the value of the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \nth feature of the \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \nth training input vector.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0061", "text": "x\n          \n            j\n            ,\n            0\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle x_{j,0}=1}\n  \n.\nTo represent the weights:", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0062", "text": "w\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle w_{i}}\n  \n is the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \nth value in the weight vector, to be multiplied by the value of the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \nth input feature.\nBecause \n  \n    \n      \n        \n          x\n          \n            j\n            ,\n            0\n          \n        \n        =\n        1", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0063", "text": "=\n        1\n      \n    \n    {\\displaystyle x_{j,0}=1}\n  \n, the \n  \n    \n      \n        \n          w\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle w_{0}}\n  \n is effectively a bias that we use instead of the bias constant \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}\n  \n.\nTo show the time-dependence of \n  \n    \n      \n        \n          w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n, we use:", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0064", "text": "w\n          \n            i\n          \n        \n        (\n        t\n        )\n      \n    \n    {\\displaystyle w_{i}(t)}\n  \n is the weight \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n at time \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0065", "text": "Steps\nFor offline learning, the second step may be repeated until the iteration error \n  \n    \n      \n        \n          \n            1\n            s\n          \n        \n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            s\n          \n        \n        \n          |\n        \n        \n          d\n          \n            j\n          \n        \n        \u2212\n        \n          y\n          \n            j\n          \n        \n        (\n        t", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0066", "text": "(\n        t\n        )\n        \n          |\n        \n      \n    \n    {\\displaystyle {\\frac {1}{s}}\\sum _{j=1}^{s}|d_{j}-y_{j}(t)|}\n  \n is less than a user-specified error threshold \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n, or a predetermined number of iterations have been completed, where s is again the size of the sample set.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0067", "text": "The algorithm updates the weights after every training sample in step 2b, though one may note that the weights remain unchanged whenever \n  \n    \n      \n        \n          d\n          \n            j\n          \n        \n        =\n        \n          y\n          \n            j\n          \n        \n        (\n        t\n        )\n      \n    \n    {\\displaystyle d_{j}=y_{j}(t)}\n  \n.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0068", "text": "Convergence of one perceptron on a linearly separable dataset", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0069", "text": "A single perceptron is a linear classifier. It can only reach a stable state if all input vectors are classified correctly. In case the training set D is not linearly separable, i.e. if the positive examples cannot be separated from the negative examples by a hyperplane, then the algorithm would not converge since there is no solution. Hence, if linear separability of the training set is not known a priori, one of the training variants below should be used", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0070", "text": ". Detailed analysis and extensions to the convergence theorem are in Chapter 11 of Perceptrons (1969).", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0071", "text": "Linear separability is testable in time \n  \n    \n      \n        min\n        (\n        O\n        (\n        \n          n\n          \n            d\n            \n              /\n            \n            2\n          \n        \n        )\n        ,\n        O\n        (\n        \n          d\n          \n            2\n            n\n          \n        \n        )\n        ,\n        O\n        (\n        \n          n\n          \n            d\n            \u2212\n            1\n          \n        \n        ln\n        \u2061", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0072", "text": "ln\n        \u2061\n        n\n        )\n        )\n      \n    \n    {\\displaystyle \\min(O(n^{d/2}),O(d^{2n}),O(n^{d-1}\\ln n))}\n  \n, where \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n is the number of data points, and \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n is the dimension of each point.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0073", "text": "is the dimension of each point.\nIf the training set is linearly separable, then the perceptron is guaranteed to converge after making finitely many mistakes. The theorem is proved by Rosenblatt et al.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0074", "text": "The following simple proof is due to Novikoff (1962). The idea of the proof is that the weight vector is always adjusted by a bounded amount in a direction with which it has a negative dot product, and thus can be bounded above by O(\u221at), where t is the number of changes to the weight vector", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0075", "text": ". However, it can also be bounded below by O(t) because if there exists an (unknown) satisfactory weight vector, then every change makes progress in this (unknown) direction by a positive amount that depends only on the input vector.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0076", "text": "While the perceptron algorithm is guaranteed to converge on some solution in the case of a linearly separable training set, it may still pick any solution and problems may admit many solutions of varying quality. The perceptron of optimal stability, nowadays better known as the linear support-vector machine, was designed to solve this problem (Krauth and Mezard, 1987).", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0077", "text": "Perceptron cycling theorem\nWhen the dataset is not linearly separable, then there is no way for a single perceptron to converge. However, we still have\n\nThis is proved first by Bradley Efron.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0078", "text": "Learning a Boolean function\nConsider a dataset where the \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n are from \n  \n    \n      \n        {\n        \u2212\n        1\n        ,\n        +\n        1\n        \n          }\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\{-1,+1\\}^{n}}\n  \n, that is, the vertices of an n-dimensional hypercube centered at origin, and \n  \n    \n      \n        y\n        =\n        \u03b8\n        (\n        \n          x", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0079", "text": "(\n        \n          x\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle y=\\theta (x_{i})}\n  \n. That is, all data points with positive \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n have \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n, and vice versa. By the perceptron convergence theorem, a perceptron would converge after making at most", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0080", "text": "n\n      \n    \n    {\\displaystyle n}\n  \n mistakes.\nIf we were to write a logical program to perform the same task, each positive example shows that one of the coordinates is the right one, and each negative example shows that its complement is a positive example. By collecting all the known positive examples, we eventually eliminate all but one coordinate, at which point the dataset is learned.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0081", "text": "This bound is asymptotically tight in terms of the worst-case. In the worst-case, the first presented example is entirely new, and gives \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n bits of information, but each subsequent example would differ minimally from previous examples, and gives 1 bit each. After \n  \n    \n      \n        n\n        +\n        1\n      \n    \n    {\\displaystyle n+1}\n  \n examples, there are \n  \n    \n      \n        2\n        n", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0082", "text": "2\n        n\n      \n    \n    {\\displaystyle 2n}\n  \n bits of information, which is sufficient for the perceptron (with \n  \n    \n      \n        2\n        n\n      \n    \n    {\\displaystyle 2n}\n  \n bits of information).\nHowever, it is not tight in terms of expectation if the examples are presented uniformly at random, since the first would give \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n bits, the second \n  \n    \n      \n        n\n        \n          /", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0083", "text": "n\n        \n          /\n        \n        2\n      \n    \n    {\\displaystyle n/2}\n  \n bits, and so on, taking \n  \n    \n      \n        O\n        (\n        ln\n        \u2061\n        n\n        )\n      \n    \n    {\\displaystyle O(\\ln n)}\n  \n examples in total.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0084", "text": "Variants", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0085", "text": "The pocket algorithm with ratchet (Gallant, 1990) solves the stability problem of perceptron learning by keeping the best solution seen so far \"in its pocket\". The pocket algorithm then returns the solution in the pocket, rather than the last solution. It can be used also for non-separable data sets, where the aim is to find a perceptron with a small number of misclassifications", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0086", "text": ". However, these solutions appear purely stochastically and hence the pocket algorithm neither approaches them gradually in the course of learning, nor are they guaranteed to show up within a given number of learning steps.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0087", "text": "The Maxover algorithm (Wendemuth, 1995) is \"robust\" in the sense that it will converge regardless of (prior) knowledge of linear separability of the data set. In the linearly separable case, it will solve the training problem \u2013 if desired, even with optimal stability (maximum margin between the classes). For non-separable data sets, it will return a solution with a computable small number of misclassifications", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0088", "text": ". In all cases, the algorithm gradually approaches the solution in the course of learning, without memorizing previous states and without stochastic jumps. Convergence is to global optimality for separable data sets and to local optimality for non-separable data sets.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0089", "text": "The Voted Perceptron (Freund and Schapire, 1999), is a variant using multiple weighted perceptrons. The algorithm starts a new perceptron every time an example is wrongly classified, initializing the weights vector with the final weights of the last perceptron. Each perceptron will also be given another weight corresponding to how many examples do they correctly classify before wrongly classifying one, and at the end the output will be a weighted vote on all perceptrons.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0090", "text": "In separable problems, perceptron training can also aim at finding the largest separating margin between the classes. The so-called perceptron of optimal stability can be determined by means of iterative training and optimization schemes, such as the Min-Over algorithm (Krauth and Mezard, 1987)  or the AdaTron (Anlauf and Biehl, 1989)). AdaTron uses the fact that the corresponding quadratic optimization problem is convex", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0091", "text": ". The perceptron of optimal stability, together with the kernel trick, are the conceptual foundations of the support-vector machine.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0092", "text": "The \n  \n    \n      \n        \u03b1\n      \n    \n    {\\displaystyle \\alpha }\n  \n-perceptron further used a pre-processing layer of fixed random weights, with thresholded output units. This enabled the perceptron to classify analogue patterns, by projecting them into a binary space. In fact, for a projection space of sufficiently high dimension, patterns can become linearly separable.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0093", "text": "Another way to solve nonlinear problems without using multiple layers is to use higher-order networks (sigma-pi unit). In this type of network, each element in the input vector is extended with each pairwise combination of multiplied inputs (second order). This can be extended to an n-order network.\nA generalization of the perceptron model is the Receptron that incorporates non-linear interactions between inputs. A single receptron is able to classify non-linear Boolean functions.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0094", "text": "It should be kept in mind, however, that the best classifier is not necessarily that which classifies all the training data perfectly. Indeed, if we had the prior constraint that the data come from equi-variant Gaussian distributions, the linear separation in the input space is optimal, and the nonlinear solution is overfitted.\nOther linear classification algorithms include Winnow, support-vector machine, and logistic regression.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0095", "text": "Multiclass perceptron\nLike most other techniques for training linear classifiers, the perceptron generalizes naturally to multiclass classification.  Here, the input \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n and the output \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n are drawn from arbitrary sets. A feature representation function \n  \n    \n      \n        f\n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle f(x,y)}", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0096", "text": "{\\displaystyle f(x,y)}\n  \n maps each possible input/output pair to a finite-dimensional real-valued feature vector.  As before, the feature vector is multiplied by a weight vector \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n, but now the resulting score is used to choose among many possible outputs:", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0097", "text": "y\n              ^\n            \n          \n        \n        =\n        \n          argmax\n          \n            y\n          \n        \n        \u2061\n        f\n        (\n        x\n        ,\n        y\n        )\n        \u22c5\n        w\n        .\n      \n    \n    {\\displaystyle {\\hat {y}}=\\operatorname {argmax} _{y}f(x,y)\\cdot w.}", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0098", "text": "Learning again iterates over the examples, predicting an output for each, leaving the weights unchanged when the predicted output matches the target, and changing them when it does not.  The update becomes:", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0099", "text": "w\n          \n            t\n            +\n            1\n          \n        \n        =\n        \n          w\n          \n            t\n          \n        \n        +\n        f\n        (\n        x\n        ,\n        y\n        )\n        \u2212\n        f\n        (\n        x\n        ,\n        \n          \n            \n              y\n              ^\n            \n          \n        \n        )\n        .\n      \n    \n    {\\displaystyle w_{t+1}=w_{t}+f(x,y)-f(x,{\\hat {y}}).}", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0100", "text": "This multiclass feedback formulation reduces to the original perceptron when \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n is a real-valued vector, \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n is chosen from \n  \n    \n      \n        {\n        0\n        ,\n        1\n        }\n      \n    \n    {\\displaystyle \\{0,1\\}}\n  \n, and \n  \n    \n      \n        f\n        (\n        x\n        ,\n        y\n        )\n        =\n        y\n        x", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0101", "text": "=\n        y\n        x\n      \n    \n    {\\displaystyle f(x,y)=yx}\n  \n.\nFor certain problems, input/output representations and features can be chosen so that \n  \n    \n      \n        \n          \n            a\n            r\n            g\n            m\n            a\n            x\n          \n          \n            y\n          \n        \n        f\n        (\n        x\n        ,\n        y\n        )\n        \u22c5\n        w\n      \n    \n    {\\displaystyle \\mathrm {argmax} _{y}f(x,y)\\cdot w}", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0102", "text": "can be found efficiently even though \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n is chosen from a very large or even infinite set.\nSince 2002, perceptron training has become popular in the field of natural language processing for such tasks as part-of-speech tagging and syntactic parsing (Collins, 2002). It has also been applied to large-scale machine learning problems in a distributed computing setting.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0103", "text": "References\nFurther reading\nAizerman, M. A. and Braverman, E. M. and Lev I. Rozonoer. Theoretical foundations of the potential function method in pattern recognition learning. Automation and Remote Control, 25:821\u2013837, 1964.\nRosenblatt, Frank (1958), The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain, Cornell Aeronautical Laboratory, Psychological Review, v65, No. 6, pp. 386\u2013408. doi:10.1037/h0042519.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0104", "text": "Rosenblatt, Frank (1962), Principles of Neurodynamics. Washington, DC: Spartan Books.\nMinsky, M. L. and Papert, S. A. 1969. Perceptrons. Cambridge, MA: MIT Press.\nGallant, S. I. (1990). Perceptron-based learning algorithms. IEEE Transactions on Neural Networks, vol. 1, no. 2, pp. 179\u2013191.\nOlazaran Rodriguez, Jose Miguel. A historical sociology of neural network research. PhD Dissertation. University of Edinburgh, 1991.", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0105", "text": "Mohri, Mehryar and Rostamizadeh, Afshin (2013). Perceptron Mistake Bounds arXiv:1305.0208, 2013.\nNovikoff, A. B. (1962). On convergence proofs on perceptrons. Symposium on the Mathematical Theory of Automata, 12, 615\u2013622. Polytechnic Institute of Brooklyn.\nWidrow, B., Lehr, M.A., \"30 years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation,\" Proc. IEEE, vol 78, no 9, pp. 1415\u20131442, (1990).", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0106", "text": "Collins, M. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with the perceptron algorithm in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP '02).\nYin, Hongfeng (1996), Perceptron-Based Algorithms and Analysis, Spectrum Library, Concordia University, Canada", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Perceptron_0107", "text": "External links\n\nA Perceptron implemented in MATLAB to learn binary NAND function Archived 2012-11-06 at the Wayback Machine\nChapter 3 Weighted networks - the perceptron and chapter 4 Perceptron learning of Neural Networks - A Systematic Introduction by Ra\u00fal Rojas (ISBN 978-3-540-60505-8)\nHistory of perceptrons\nMathematics of multilayer perceptrons\nApplying a perceptron model using scikit-learn - https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Perceptron.html", "title": "Perceptron", "url": "https://en.wikipedia.org/wiki/Perceptron"}, {"id": "rec_Multilayer_perceptron_0000", "text": "In deep learning, a multilayer perceptron (MLP) is a kind of modern feedforward neural network consisting of fully connected neurons with nonlinear activation functions, organized in layers, notable for being able to distinguish data that is not linearly separable.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0001", "text": "Modern neural networks are trained using backpropagation and are colloquially referred to as \"vanilla\" networks. MLPs grew out of an effort to improve on single-layer perceptrons, which could only be applied to linearly separable data. A perceptron traditionally used a Heaviside step function as its nonlinear activation function. However, the backpropagation algorithm requires that modern MLPs use continuous activation functions such as sigmoid or ReLU.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0002", "text": "Multilayer perceptrons form the basis of deep learning, and are applicable across a vast set of diverse domains.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0003", "text": "Timeline\nIn 1943, Warren McCulloch and Walter Pitts proposed the binary artificial neuron as a logical model of biological neural networks.\nIn 1958, Frank Rosenblatt proposed the multilayered perceptron model, consisting of an input layer, a hidden layer with randomized weights that did not learn, and an output layer with learnable connections.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0004", "text": "In 1962, Rosenblatt published many variants and experiments on perceptrons in his book Principles of Neurodynamics, including up to 2 trainable layers by \"back-propagating errors\". However, it was not the backpropagation algorithm, and he did not have a general method for training multiple layers.\nIn 1965, Alexey Grigorevich Ivakhnenko and Valentin Lapa published Group Method of Data Handling. It was one of the first deep learning methods, used to train an eight-layer neural net in 1971.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0005", "text": "In 1967, Shun'ichi Amari reported  the first multilayered neural network trained by stochastic gradient descent, was able to classify non-linearily separable pattern classes. Amari's student Saito conducted the computer experiments, using a five-layered feedforward network with two learning layers.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0006", "text": "Backpropagation was independently developed multiple times in early 1970s. The earliest published instance was Seppo Linnainmaa's master thesis (1970). Paul Werbos developed it independently in 1971, but had difficulty publishing it until 1982.\nIn 1986, David E. Rumelhart et al. popularized backpropagation.\nIn 2003, interest in backpropagation networks returned due to the successes of deep learning being applied to language modelling by Yoshua Bengio with co-authors.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0007", "text": "In 2021, a very simple NN architecture combining two deep MLPs with skip connections and layer normalizations was designed and called MLP-Mixer; its realizations featuring 19 to 431 millions of parameters were shown to be comparable to vision transformers of similar size on ImageNet and similar image classification tasks.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0008", "text": "Mathematical foundations\nActivation function\nIf a multilayer perceptron has a linear activation function in all neurons, that is, a linear function that maps the weighted inputs to the output of each neuron, then linear algebra shows that any number of layers can be reduced to a two-layer input-output model. In MLPs some neurons use a nonlinear activation function that was developed to model the frequency of action potentials, or firing, of biological neurons.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0009", "text": "The two historically common activation functions are both sigmoids, and are described by", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0010", "text": "y\n        (\n        \n          v\n          \n            i\n          \n        \n        )\n        =\n        tanh\n        \u2061\n        (\n        \n          v\n          \n            i\n          \n        \n        )\n         \n         \n        \n          \n            and\n          \n        \n         \n         \n        y\n        (\n        \n          v\n          \n            i\n          \n        \n        )\n        =\n        (\n        1\n        +\n        \n          e", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0011", "text": "+\n        \n          e\n          \n            \u2212\n            \n              v\n              \n                i\n              \n            \n          \n        \n        \n          )\n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle y(v_{i})=\\tanh(v_{i})~~{\\textrm {and}}~~y(v_{i})=(1+e^{-v_{i}})^{-1}}\n  \n.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0012", "text": ".\nThe first is a hyperbolic tangent that ranges from \u22121 to 1, while the other is the logistic function, which is similar in shape but ranges from 0 to 1. Here \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}\n  \n is the output of the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \nth node (neuron) and \n  \n    \n      \n        \n          v\n          \n            i", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0013", "text": "i\n          \n        \n      \n    \n    {\\displaystyle v_{i}}\n  \n is the weighted sum of the input connections. Alternative activation functions have been proposed, including the rectifier and softplus functions. More specialized activation functions include radial basis functions (used in radial basis networks, another class of supervised neural network models).", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0014", "text": "In recent developments of deep learning the rectified linear unit (ReLU) is more frequently used as one of the possible ways to overcome the numerical problems related to the sigmoids.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0015", "text": "Layers\nThe MLP consists of three or more layers (an input and an output layer with one or more hidden layers) of nonlinearly-activating nodes. Since MLPs are fully connected, each node in one layer connects with a certain weight \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n to every node in the following layer.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0016", "text": "Learning\nLearning occurs in the perceptron by changing connection weights after each piece of data is processed, based on the amount of error in the output compared to the expected result. This is an example of supervised learning, and is carried out through backpropagation, a generalization of the least mean squares algorithm in the linear perceptron.\nWe can represent the degree of error in an output node \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n in the", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0017", "text": "{\\displaystyle j}\n  \n in the \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \nth data point (training example) by \n  \n    \n      \n        \n          e\n          \n            j\n          \n        \n        (\n        n\n        )\n        =\n        \n          d\n          \n            j\n          \n        \n        (\n        n\n        )\n        \u2212\n        \n          y\n          \n            j\n          \n        \n        (\n        n\n        )", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0018", "text": "(\n        n\n        )\n      \n    \n    {\\displaystyle e_{j}(n)=d_{j}(n)-y_{j}(n)}\n  \n, where \n  \n    \n      \n        \n          d\n          \n            j\n          \n        \n        (\n        n\n        )\n      \n    \n    {\\displaystyle d_{j}(n)}\n  \n is the desired target value for \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \nth data point at node \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n, and \n  \n    \n      \n        \n          y", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0019", "text": "y\n          \n            j\n          \n        \n        (\n        n\n        )\n      \n    \n    {\\displaystyle y_{j}(n)}\n  \n is the value produced by the perceptron at node \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n when the \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \nth data point is given as an input.\nThe node weights can then be adjusted based on corrections that minimize the error in the entire output for the", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0020", "text": "n\n      \n    \n    {\\displaystyle n}\n  \nth data point, given by", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0021", "text": "E\n          \n        \n        (\n        n\n        )\n        =\n        \n          \n            1\n            2\n          \n        \n        \n          \u2211\n          \n            \n              output node \n            \n            j\n          \n        \n        \n          e\n          \n            j\n          \n          \n            2\n          \n        \n        (\n        n\n        )", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0022", "text": "(\n        n\n        )\n      \n    \n    {\\displaystyle {\\mathcal {E}}(n)={\\frac {1}{2}}\\sum _{{\\text{output node }}j}e_{j}^{2}(n)}\n  \n.\nUsing gradient descent, the change in each weight \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n is", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0023", "text": "\u0394\n        \n          w\n          \n            j\n            i\n          \n        \n        (\n        n\n        )\n        =\n        \u2212\n        \u03b7\n        \n          \n            \n              \u2202\n              \n                \n                  E\n                \n              \n              (\n              n\n              )\n            \n            \n              \u2202\n              \n                v\n                \n                  j", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0024", "text": "(\n              n\n              )\n            \n          \n        \n        \n          y\n          \n            i\n          \n        \n        (\n        n\n        )\n      \n    \n    {\\displaystyle \\Delta w_{ji}(n)=-\\eta {\\frac {\\partial {\\mathcal {E}}(n)}{\\partial v_{j}(n)}}y_{i}(n)}", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0025", "text": "where \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n        (\n        n\n        )\n      \n    \n    {\\displaystyle y_{i}(n)}\n  \n is the output of the previous neuron \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n, and \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n is the learning rate, which is selected to ensure that the weights quickly converge to a response, without oscillations. In the previous expression,", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0026", "text": "\u2202\n              \n                \n                  E\n                \n              \n              (\n              n\n              )\n            \n            \n              \u2202\n              \n                v\n                \n                  j\n                \n              \n              (\n              n\n              )", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0027", "text": "{\\displaystyle {\\frac {\\partial {\\mathcal {E}}(n)}{\\partial v_{j}(n)}}}\n  \n denotes the partial derivate of the error \n  \n    \n      \n        \n          \n            E\n          \n        \n        (\n        n\n        )\n      \n    \n    {\\displaystyle {\\mathcal {E}}(n)}\n  \n according to the weighted sum \n  \n    \n      \n        \n          v\n          \n            j\n          \n        \n        (\n        n\n        )", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0028", "text": "(\n        n\n        )\n      \n    \n    {\\displaystyle v_{j}(n)}\n  \n of the input connections of neuron \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n.\nThe derivative to be calculated depends on the induced local field \n  \n    \n      \n        \n          v\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle v_{j}}\n  \n, which itself varies. It is easy to prove that for an output node this derivative can be simplified to", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0029", "text": "\u2212\n        \n          \n            \n              \u2202\n              \n                \n                  E\n                \n              \n              (\n              n\n              )\n            \n            \n              \u2202\n              \n                v\n                \n                  j\n                \n              \n              (\n              n\n              )\n            \n          \n        \n        =\n        \n          e\n          \n            j", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0030", "text": "e\n          \n            j\n          \n        \n        (\n        n\n        )\n        \n          \u03d5\n          \n            \u2032\n          \n        \n        (\n        \n          v\n          \n            j\n          \n        \n        (\n        n\n        )\n        )\n      \n    \n    {\\displaystyle -{\\frac {\\partial {\\mathcal {E}}(n)}{\\partial v_{j}(n)}}=e_{j}(n)\\phi ^{\\prime }(v_{j}(n))}", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0031", "text": "where \n  \n    \n      \n        \n          \u03d5\n          \n            \u2032\n          \n        \n      \n    \n    {\\displaystyle \\phi ^{\\prime }}\n  \n is the derivative of the activation function described above, which itself does not vary. The analysis is more difficult for the change in weights to a hidden node, but it can be shown that the relevant derivative is", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0032", "text": "\u2212\n        \n          \n            \n              \u2202\n              \n                \n                  E\n                \n              \n              (\n              n\n              )\n            \n            \n              \u2202\n              \n                v\n                \n                  j\n                \n              \n              (\n              n\n              )\n            \n          \n        \n        =\n        \n          \u03d5\n          \n            \u2032", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0033", "text": "\u03d5\n          \n            \u2032\n          \n        \n        (\n        \n          v\n          \n            j\n          \n        \n        (\n        n\n        )\n        )\n        \n          \u2211\n          \n            k\n          \n        \n        \u2212\n        \n          \n            \n              \u2202\n              \n                \n                  E\n                \n              \n              (\n              n\n              )\n            \n            \n              \u2202", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0034", "text": "\u2202\n              \n                v\n                \n                  k\n                \n              \n              (\n              n\n              )\n            \n          \n        \n        \n          w\n          \n            k\n            j\n          \n        \n        (\n        n\n        )", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0035", "text": "(\n        n\n        )\n      \n    \n    {\\displaystyle -{\\frac {\\partial {\\mathcal {E}}(n)}{\\partial v_{j}(n)}}=\\phi ^{\\prime }(v_{j}(n))\\sum _{k}-{\\frac {\\partial {\\mathcal {E}}(n)}{\\partial v_{k}(n)}}w_{kj}(n)}\n  \n.\nThis depends on the change in weights of the \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0036", "text": "k\n      \n    \n    {\\displaystyle k}\n  \nth nodes, which represent the output layer. So to change the hidden layer weights, the output layer weights change according to the derivative of the activation function, and so this algorithm represents a backpropagation of the activation function.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Multilayer_perceptron_0037", "text": "References\nExternal links\nWeka: Open source data mining software with multilayer perceptron implementation.\nNeuroph Studio documentation, implements this algorithm and a few others.", "title": "Multilayer perceptron", "url": "https://en.wikipedia.org/wiki/Multilayer_perceptron"}, {"id": "rec_Convolutional_neural_network_0000", "text": "A convolutional neural network (CNN) is a type of feedforward neural network that learns features via filter (or kernel) optimization. This type of deep learning network has been applied to process and make predictions from many different types of data including text, images and audio. CNNs are the de-facto standard in deep learning-based approaches to computer vision and image processing, and have only recently been replaced\u2014in some cases\u2014by newer architectures such as the transformer.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0001", "text": "Vanishing gradients and exploding gradients, seen during backpropagation in earlier neural networks, are prevented by the regularization that comes from using shared weights over fewer connections. For example, for each neuron in the fully-connected layer, 10,000 weights would be required for processing an image sized 100 \u00d7 100 pixels. However, applying cascaded convolution (or cross-correlation) kernels, only 25 weights for each convolutional layer are required to process 5x5-sized tiles", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0002", "text": ". Higher-layer features are extracted from wider context windows, compared to lower-layer features.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0003", "text": "Some applications of CNNs include:", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0004", "text": "image and video recognition,\nrecommender systems,\nimage classification,\nimage segmentation,\nmedical image analysis,\nnatural language processing,\nbrain\u2013computer interfaces, and\nfinancial time series.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0005", "text": "financial time series.\nCNNs are also known as shift invariant or space invariant artificial neural networks, based on the shared-weight architecture of the convolution kernels or filters that slide along input features and provide translation-equivariant responses known as feature maps. Counter-intuitively, most convolutional neural networks are not invariant to translation, due to the downsampling operation they apply to the input.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0006", "text": "Feedforward neural networks are usually fully connected networks, that is, each neuron in one layer is connected to all neurons in the next layer. The \"full connectivity\" of these networks makes them prone to overfitting data", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0007", "text": ". Typical ways of regularization, or preventing overfitting, include: penalizing parameters during training (such as weight decay) or trimming connectivity (skipped connections, dropout, etc.) Robust datasets also increase the probability that CNNs will learn the generalized principles that characterize a given dataset rather than the biases of a poorly-populated set.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0008", "text": "Convolutional networks were inspired by biological processes in that the connectivity pattern between neurons resembles the organization of the animal visual cortex. Individual cortical neurons respond to stimuli only in a restricted region of the visual field known as the receptive field. The receptive fields of different neurons partially overlap such that they cover the entire visual field.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0009", "text": "CNNs use relatively little pre-processing compared to other image classification algorithms. This means that the network learns to optimize the filters (or kernels) through automated learning, whereas in traditional algorithms these filters are hand-engineered. This simplifies and automates the process, enhancing efficiency and scalability overcoming human-intervention bottlenecks.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0010", "text": "Architecture", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0011", "text": "A convolutional neural network consists of an input layer, hidden layers and an output layer. In a convolutional neural network, the hidden layers include one or more layers that perform convolutions. Typically this includes a layer that performs a dot product of the convolution kernel with the layer's input matrix. This product is usually the Frobenius inner product, and its activation function is commonly ReLU", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0012", "text": ". As the convolution kernel slides along the input matrix for the layer, the convolution operation generates a feature map, which in turn contributes to the input of the next layer. This is followed by other layers such as pooling layers, fully connected layers, and normalization layers.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0013", "text": "Here it should be noted how close a convolutional neural network is to a matched filter.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0014", "text": "Convolutional layers\nIn a CNN, the input is a tensor with shape:\n(number of inputs) \u00d7 (input height) \u00d7 (input width) \u00d7 (input channels)\nAfter passing through a convolutional layer, the image becomes abstracted to a feature map, also called an activation map, with shape:\n(number of inputs) \u00d7 (feature map height) \u00d7 (feature map width) \u00d7 (feature map channels).", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0015", "text": "Convolutional layers convolve the input and pass its result to the next layer. This is similar to the response of a neuron in the visual cortex to a specific stimulus. Each convolutional neuron processes data only for its receptive field.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0016", "text": "Although fully connected feedforward neural networks can be used to learn features and classify data, this architecture is generally impractical for larger inputs (e.g., high-resolution images), which would require massive numbers of neurons because each pixel is a relevant input feature. A fully connected layer for an image of size 100 \u00d7 100 has 10,000 weights for each neuron in the second layer. Convolution reduces the number of free parameters, allowing the network to be deeper", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0017", "text": ". For example, using a 5 \u00d7 5 tiling region, each with the same shared weights, requires only 25 neurons. Using shared weights means there are many fewer parameters, which helps avoid the vanishing gradients and exploding gradients problems seen during backpropagation in earlier neural networks.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0018", "text": "To speed processing, standard convolutional layers can be replaced by depthwise separable convolutional layers, which are based on a depthwise convolution followed by a pointwise convolution. The depthwise convolution is a spatial convolution applied independently over each channel of the input tensor, while the pointwise convolution is a standard convolution restricted to the use of \n  \n    \n      \n        1\n        \u00d7\n        1\n      \n    \n    {\\displaystyle 1\\times 1}\n  \n kernels.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0019", "text": "Pooling layers", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0020", "text": "Convolutional networks may include local and/or global pooling layers along with traditional convolutional layers. Pooling layers reduce the dimensions of data by combining the outputs of neuron clusters at one layer into a single neuron in the next layer. Local pooling combines small clusters, tiling sizes such as 2 \u00d7 2 are commonly used. Global pooling acts on all the neurons of the feature map. There are two common types of pooling in popular use: max and average", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0021", "text": ". Max pooling uses the maximum value of each local cluster of neurons in the feature map, while average pooling takes the average value.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0022", "text": "Fully connected layers", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0023", "text": "Fully connected layers connect every neuron in one layer to every neuron in another layer. It is the same as a traditional multilayer perceptron neural network (MLP). Each neuron in the fully connected layer receives input from all the neurons in the previous layer. These inputs are weighted and summed with the corresponding biases, and then passed through an activation function to perform a nonlinear transformation, generating the output", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0024", "text": ". The flattened matrix goes through a fully connected layer to classify the images.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0025", "text": "Receptive field", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0026", "text": "In neural networks, each neuron receives input from some number of locations in the previous layer. In a convolutional layer, each neuron receives input from only a restricted area of the previous layer called the neuron's receptive field. Typically the area is a square (e.g. 5 by 5 neurons). Whereas, in a fully connected layer, the receptive field is the entire previous layer. Thus, in each convolutional layer, each neuron takes input from a larger area in the input than previous layers", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0027", "text": ". This is due to applying the convolution over and over, which takes the value of a pixel into account, as well as its surrounding pixels. When using dilated layers, the number of pixels in the receptive field remains constant, but the field is more sparsely populated as its dimensions grow when combining the effect of several layers.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0028", "text": "To manipulate the receptive field size as desired, there are some alternatives to the standard convolutional layer. For example, atrous or dilated convolution expands the receptive field size without increasing the number of parameters by interleaving visible and blind regions. Moreover, a single dilated convolutional layer can comprise filters with multiple dilation ratios, thus having a variable receptive field size.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0029", "text": "Weights\nEach neuron in a neural network computes an output value by applying a specific function to the input values received from the receptive field in the previous layer. The function that is applied to the input values is determined by a vector of weights and a bias (typically real numbers). Learning consists of iteratively adjusting these biases and weights.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0030", "text": "The vectors of weights and biases are called filters and represent particular features of the input (e.g., a particular shape). A distinguishing feature of CNNs is that many neurons can share the same filter. This reduces the memory footprint because a single bias and a single vector of weights are used across all receptive fields that share that filter, as opposed to each receptive field having its own bias and vector weighting.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0031", "text": "Deconvolutional\nA deconvolutional neural network is essentially the reverse of a CNN. It consists of deconvolutional layers and unpooling layers.\nA deconvolutional layer is the transpose of a convolutional layer. Specifically, a convolutional layer can be written as a multiplication with a matrix, and a deconvolutional layer is multiplication with the transpose of that matrix.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0032", "text": "An unpooling layer expands the layer. The max-unpooling layer is the simplest, as it simply copies each entry multiple times. For example, a 2-by-2 max-unpooling layer is \n  \n    \n      \n        [\n        x\n        ]\n        \u21a6\n        \n          \n            [\n            \n              \n                \n                  x\n                \n                \n                  x\n                \n              \n              \n                \n                  x", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0033", "text": "x\n                \n              \n            \n            ]\n          \n        \n      \n    \n    {\\displaystyle [x]\\mapsto {\\begin{bmatrix}x&x\\\\x&x\\end{bmatrix}}}\n  \n.\nDeconvolution layers are used in image generators. By default, it creates periodic checkerboard artifact, which can be fixed by upscale-then-convolve.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0034", "text": "History\nCNN are often compared to the way the brain achieves vision processing in living organisms.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0035", "text": "Receptive fields in the visual cortex", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0036", "text": "Work by Hubel and Wiesel in the 1950s and 1960s showed that cat visual cortices contain neurons that individually respond to small regions of the visual field. Provided the eyes are not moving, the region of visual space within which visual stimuli affect the firing of a single neuron is known as its receptive field. Neighboring cells have similar and overlapping receptive fields. Receptive field size and location varies systematically across the cortex to form a complete map of visual space", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0037", "text": ". The cortex in each hemisphere represents the contralateral visual field.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0038", "text": "Their 1968 paper identified two basic visual cell types in the brain:", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0039", "text": "simple cells, whose output is maximized by straight edges having particular orientations within their receptive field\ncomplex cells, which have larger receptive fields, whose output is insensitive to the exact position of the edges in the field.\nHubel and Wiesel also proposed a cascading model of these two types of cells for use in pattern recognition tasks.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0040", "text": "Fukushima's analog threshold elements in a vision model", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0041", "text": "In 1969, Kunihiko Fukushima introduced a multilayer visual feature detection network, inspired by the above-mentioned work of Hubel and Wiesel, in which \"All the elements in one layer have the same set of interconnecting coefficients; the arrangement of the elements and their interconnections are all homogeneous over a given layer.\"  This is the essential core of a convolutional network, but the weights were not trained", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0042", "text": ".  In the same paper, Fukushima also introduced the ReLU (rectified linear unit) activation function.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0043", "text": "Neocognitron, origin of the trainable CNN architecture\nThe \"neocognitron\" was introduced by Fukushima in 1980.  The neocognitron introduced the two basic types of layers:", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0044", "text": "\"S-layer\": a shared-weights receptive-field layer, later known as a convolutional layer, which contains units whose receptive fields cover a patch of the previous layer. A shared-weights receptive-field group (a \"plane\" in neocognitron terminology) is often called a filter, and a layer typically has several such filters.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0045", "text": "\"C-layer\": a downsampling layer that contain units whose receptive fields cover patches of previous convolutional layers. Such a unit typically computes a weighted average of the activations of the units in its patch, and applies inhibition (divisive normalization) pooled from a somewhat larger patch and across different filters in a layer, and applies a saturating activation function. The patch weights are nonnegative and are not trainable in the original neocognitron", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0046", "text": ". The downsampling and competitive inhibition help to classify features and objects in visual scenes even when the objects are shifted.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0047", "text": "Several supervised and unsupervised learning algorithms have been proposed over the decades to train the weights of a neocognitron. Today, however, the CNN architecture is usually trained through backpropagation.\nFukushima's ReLU activation function was not used in his neocognitron since all the weights were nonnegative; lateral inhibition was used instead. The rectifier has become a very popular activation function for CNNs and deep neural networks in general.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0048", "text": "Convolution in time", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0049", "text": "The term \"convolution\" first appears in neural networks in a paper by Toshiteru Homma, Les Atlas, and Robert Marks II at the first Conference on Neural Information Processing Systems in 1987. Their paper replaced multiplication with convolution in time, inherently providing shift invariance, motivated by and connecting more directly to the signal-processing concept of a filter, and demonstrated it on a speech recognition task", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0050", "text": ". They also pointed out that as a data-trainable system, convolution is essentially equivalent to correlation since reversal of the weights does not affect the final learned function (\"For convenience, we denote * as correlation instead of convolution. Note that convolving a(t) with b(t) is equivalent to correlating a(-t) with b(t).\"). Modern CNN implementations typically do correlation and call it convolution, for convenience, as they did here.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0051", "text": "Time delay neural networks", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0052", "text": "The time delay neural network (TDNN) was introduced in 1987 by Alex Waibel et al. for phoneme recognition and was an early convolutional network exhibiting shift-invariance. A TDNN is a 1-D convolutional neural net where the convolution is performed along the time axis of the data. It is the first CNN utilizing weight sharing in combination with a training by gradient descent, using backpropagation", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0053", "text": ". Thus, while also using a pyramidal structure as in the neocognitron, it performed a global optimization of the weights instead of a local one.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0054", "text": "TDNNs are convolutional networks that share weights along the temporal dimension. They allow speech signals to be processed time-invariantly. In 1990 Hampshire and Waibel introduced a variant that performs a two-dimensional convolution. Since these TDNNs operated on spectrograms, the resulting phoneme recognition system was invariant to both time and frequency shifts, as with images processed by a neocognitron.\nTDNNs improved the performance of far-distance speech recognition.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0055", "text": "Image recognition with CNNs trained by gradient descent\nDenker et al. (1989) designed a 2-D CNN system to recognize hand-written ZIP Code numbers. However, the lack of an efficient training method to determine the kernel coefficients of the involved convolutions meant that all the coefficients had to be laboriously hand-designed.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0056", "text": "Following the advances in the training of 1-D CNNs by Waibel et al. (1987), Yann LeCun et al. (1989) used back-propagation to learn the convolution kernel coefficients directly from images of hand-written numbers. Learning was thus fully automatic, performed better than manual coefficient design, and was suited to a broader range of image recognition problems and image types.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0057", "text": "Wei Zhang et al. (1988) used back-propagation to train the convolution kernels of a CNN for alphabets recognition. The model was called shift-invariant pattern recognition neural network before the name CNN was coined later in the early 1990s. Wei Zhang et al. also applied the same CNN without the last fully connected layer for medical image object segmentation (1991) and breast cancer detection in mammograms (1994).\nThis approach became a foundation of modern computer vision.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0058", "text": "Max pooling", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0059", "text": "In 1990 Yamaguchi et al. introduced the concept of max pooling, a fixed filtering operation that calculates and propagates the maximum value of a given region. They did so by combining TDNNs with max pooling to realize a speaker-independent isolated word recognition system. In their system they used several TDNNs per word, one for each syllable", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0060", "text": ". The results of each TDNN over the input signal were combined using max pooling and the outputs of the pooling layers were then passed on to networks performing the actual word classification.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0061", "text": "In a variant of the neocognitron called the cresceptron, instead of using Fukushima's spatial averaging with inhibition and saturation, J. Weng et al. in 1993 used max pooling, where a downsampling unit computes the maximum of the activations of the units in its patch, introducing this method into the vision field.\nMax pooling is often used in modern CNNs.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0062", "text": "LeNet-5\nLeNet-5, a pioneering 7-level convolutional network by LeCun et al. in 1995, classifies hand-written numbers on checks digitized in 32\u00d732 pixel images. The ability to process higher-resolution images requires larger and more layers of convolutional neural networks, so this technique is constrained by the availability of computing resources.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0063", "text": "It was superior than other commercial courtesy amount reading systems (as of 1995). The system was integrated in NCR's check reading systems, and fielded in several American banks since June 1996, reading millions of checks per day.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0064", "text": "Shift-invariant neural network", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0065", "text": "A shift-invariant neural network was proposed by Wei Zhang et al. for image character recognition in 1988. It is a modified Neocognitron by keeping only the convolutional interconnections between the image feature layers and the last fully connected layer. The model was trained with back-propagation. The training algorithm was further improved in 1991 to improve its generalization ability", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0066", "text": ". The model architecture was modified by removing the last fully connected layer and applied for medical image segmentation (1991) and automatic detection of breast cancer in mammograms (1994).", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0067", "text": "A different convolution-based design was proposed in 1988 for application to decomposition of one-dimensional electromyography convolved signals via de-convolution. This design was modified in 1989 to other de-convolution-based designs.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0068", "text": "GPU implementations\nAlthough CNNs were invented in the 1980s, their breakthrough in the 2000s required fast implementations on graphics processing units (GPUs).\nIn 2004, it was shown by K. S. Oh and K. Jung that standard neural networks can be greatly accelerated on GPUs. Their implementation was 20 times faster than an equivalent implementation on CPU. In 2005, another paper also emphasised the value of GPGPU for machine learning.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0069", "text": "The first GPU-implementation of a CNN was described in 2006 by K. Chellapilla et al. Their implementation was 4 times faster than an equivalent implementation on CPU. In the same period, GPUs were also used for unsupervised training of deep belief networks.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0070", "text": "In 2010, Dan Ciresan et al. at IDSIA trained deep feedforward networks on GPUs. In 2011, they extended this to CNNs, accelerating by 60 compared to training CPU. In 2011, the network won an image recognition contest where they achieved superhuman performance for the first time. Then they won more competitions and achieved state of the art on several benchmarks.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0071", "text": "Subsequently, AlexNet, a similar GPU-based CNN by Alex Krizhevsky et al. won the ImageNet Large Scale Visual Recognition Challenge 2012. It was an early catalytic event for the AI boom.\nCompared to the training of CNNs using GPUs, not much attention was given to CPU. (Viebke et al 2019) parallelizes CNN by thread- and SIMD-level parallelism that is available on the Intel Xeon Phi.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0072", "text": "Distinguishing features\nIn the past, traditional multilayer perceptron (MLP) models were used for image recognition. However, the full connectivity between nodes caused the curse of dimensionality, and was computationally intractable with higher-resolution images. A 1000\u00d71000-pixel image with RGB color channels has 3 million weights per fully-connected neuron, which is too high to feasibly process efficiently at scale.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0073", "text": "For example, in CIFAR-10, images are only of size 32\u00d732\u00d73 (32 wide, 32 high, 3 color channels), so a single fully connected neuron in the first hidden layer of a regular neural network would have 32*32*3 = 3,072 weights. A 200\u00d7200 image, however, would lead to neurons that have 200*200*3 = 120,000 weights.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0074", "text": "Also, such network architecture does not take into account the spatial structure of data, treating input pixels which are far apart in the same way as pixels that are close together. This ignores locality of reference in data with a grid-topology (such as images), both computationally and semantically. Thus, full connectivity of neurons is wasteful for purposes such as image recognition that are dominated by spatially local input patterns.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0075", "text": "Convolutional neural networks are variants of multilayer perceptrons, designed to emulate the behavior of a visual cortex. These models mitigate the challenges posed by the MLP architecture by exploiting the strong spatially local correlation present in natural images. As opposed to MLPs, CNNs have the following distinguishing features:", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0076", "text": "3D volumes of neurons. The layers of a CNN have neurons arranged in 3 dimensions: width, height and depth. Each neuron inside a convolutional layer is connected to only a small region of the layer before it, called a receptive field. Distinct types of layers, both locally and completely connected, are stacked to form a CNN architecture.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0077", "text": "Local connectivity: following the concept of receptive fields, CNNs exploit spatial locality by enforcing a local connectivity pattern between neurons of adjacent layers. The architecture thus ensures that the learned \"filters\" produce the strongest response to a spatially local input pattern. Stacking many such layers leads to nonlinear filters that become increasingly global (i.e", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0078", "text": ". responsive to a larger region of pixel space) so that the network first creates representations of small parts of the input, then from them assembles representations of larger areas.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0079", "text": "Shared weights: In CNNs, each filter is replicated across the entire visual field. These replicated units share the same parameterization (weight vector and bias) and form a feature map. This means that all the neurons in a given convolutional layer respond to the same feature within their specific response field. Replicating units in this way allows for the resulting activation map to be equivariant under shifts of the locations of input features in the visual field, i.e", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0080", "text": ". they grant translational equivariance\u2014given that the layer has a stride of one.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0081", "text": "Pooling: In a CNN's pooling layers, feature maps are divided into rectangular sub-regions, and the features in each rectangle are independently down-sampled to a single value, commonly by taking their average or maximum value. In addition to reducing the sizes of feature maps, the pooling operation grants a degree of local translational invariance to the features contained therein, allowing the CNN to be more robust to variations in their positions.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0082", "text": "Together, these properties allow CNNs to achieve better generalization on vision problems. Weight sharing dramatically reduces the number of free parameters learned, thus lowering the memory requirements for running the network and allowing the training of larger, more powerful networks.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0083", "text": "Building blocks\nA CNN architecture is formed by a stack of distinct layers that transform the input volume into an output volume (e.g. holding the class scores) through a differentiable function. A few distinct types of layers are commonly used. These are further discussed below.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0084", "text": "Convolutional layer", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0085", "text": "The convolutional layer is the core building block of a CNN. The layer's parameters consist of a set of learnable filters (or kernels), which have a small receptive field, but extend through the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the filter entries and the input, producing a 2-dimensional activation map of that filter", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0086", "text": ". As a result, the network learns filters that activate when it detects some specific type of feature at some spatial position in the input.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0087", "text": "Stacking the activation maps for all filters along the depth dimension forms the full output volume of the convolution layer. Every entry in the output volume can thus also be interpreted as an output of a neuron that looks at a small region in the input. Each entry in an activation map use the same set of parameters that define the filter.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0088", "text": "Self-supervised learning has been adapted for use in convolutional layers by using sparse patches with a high-mask ratio and a global response normalization layer.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0089", "text": "Local connectivity\nWhen dealing with high-dimensional inputs such as images, it is impractical to connect neurons to all neurons in the previous volume because such a network architecture does not take the spatial structure of the data into account. Convolutional networks exploit spatially local correlation by enforcing a sparse local connectivity pattern between neurons of adjacent layers: each neuron is connected to only a small region of the input volume.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0090", "text": "The extent of this connectivity is a hyperparameter called the receptive field of the neuron. The connections are local in space (along width and height), but always extend along the entire depth of the input volume. Such an architecture ensures that the learned filters produce the strongest response to a spatially local input pattern.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0091", "text": "Spatial arrangement\nThree hyperparameters control the size of the output volume of the convolutional layer: the depth, stride, and padding size:", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0092", "text": "The depth of the output volume controls the number of neurons in a layer that connect to the same region of the input volume. These neurons learn to activate for different features in the input. For example, if the first convolutional layer takes the raw image as input, then different neurons along the depth dimension may activate in the presence of various oriented edges, or blobs of color.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0093", "text": "Stride controls how depth columns around the width and height are allocated. If the stride is 1, then we move the filters one pixel at a time. This leads to heavily overlapping receptive fields between the columns, and to large output volumes. For any integer \n  \n    \n      \n        S\n        >\n        0\n        ,\n      \n    \n    {\\textstyle S>0,}\n  \n a stride S means that the filter is translated S units at a time per output. In practice, \n  \n    \n      \n        S\n        \u2265\n        3", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0094", "text": "S\n        \u2265\n        3\n      \n    \n    {\\textstyle S\\geq 3}\n  \n is rare. A greater stride means smaller overlap of receptive fields and smaller spatial dimensions of the output volume.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0095", "text": "Sometimes, it is convenient to pad the input with zeros (or other values, such as the average of the region) on the border of the input volume. The size of this padding is a third hyperparameter. Padding provides control of the output volume's spatial size. In particular, sometimes it is desirable to exactly preserve the spatial size of the input volume, this is commonly referred to as \"same\" padding.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0096", "text": "The spatial size of the output volume is a function of the input volume size \n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}\n  \n, the kernel field size \n  \n    \n      \n        K\n      \n    \n    {\\displaystyle K}\n  \n of the convolutional layer neurons, the stride \n  \n    \n      \n        S\n      \n    \n    {\\displaystyle S}\n  \n, and the amount of zero padding \n  \n    \n      \n        P\n      \n    \n    {\\displaystyle P}", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0097", "text": "P\n      \n    \n    {\\displaystyle P}\n  \n on the border. The number of neurons that \"fit\" in a given volume is then:", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0098", "text": "W\n              \u2212\n              K\n              +\n              2\n              P\n            \n            S\n          \n        \n        +\n        1.\n      \n    \n    {\\displaystyle {\\frac {W-K+2P}{S}}+1.}", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0099", "text": "If this number is not an integer, then the strides are incorrect and the neurons cannot be tiled to fit across the input volume in a symmetric way. In general, setting zero padding to be \n  \n    \n      \n        P\n        =\n        (\n        K\n        \u2212\n        1\n        )\n        \n          /\n        \n        2\n      \n    \n    {\\textstyle P=(K-1)/2}\n  \n when the stride is \n  \n    \n      \n        S\n        =\n        1\n      \n    \n    {\\displaystyle S=1}", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0100", "text": "1\n      \n    \n    {\\displaystyle S=1}\n  \n ensures that the input volume and output volume will have the same size spatially. However, it is not always completely necessary to use all of the neurons of the previous layer. For example, a neural network designer may decide to use just a portion of padding.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0101", "text": "Parameter sharing\nA parameter sharing scheme is used in convolutional layers to control the number of free parameters. It relies on the assumption that if a patch feature is useful to compute at some spatial position, then it should also be useful to compute at other positions. Denoting a single 2-dimensional slice of depth as a depth slice, the neurons in each depth slice are constrained to use the same weights and bias.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0102", "text": "Since all neurons in a single depth slice share the same parameters, the forward pass in each depth slice of the convolutional layer can be computed as a convolution of the neuron's weights with the input volume. Therefore, it is common to refer to the sets of weights as a filter (or a kernel), which is convolved with the input", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0103", "text": ". The result of this convolution is an activation map, and the set of activation maps for each different filter are stacked together along the depth dimension to produce the output volume. Parameter sharing contributes to the translation invariance of the CNN architecture.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0104", "text": "Sometimes, the parameter sharing assumption may not make sense. This is especially the case when the input images to a CNN have some specific centered structure; for which we expect completely different features to be learned on different spatial locations. One practical example is when the inputs are faces that have been centered in the image: we might expect different eye-specific or hair-specific features to be learned in different parts of the image", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0105", "text": ". In that case it is common to relax the parameter sharing scheme, and instead simply call the layer a \"locally connected layer\". In this layer, the convolutional kernels' parameters are not shared. Instead, the network learns independent weights and biases for each spatial location. This allows each location to have its own feature-learning ability, making it better suited to handle images with distinct central structures or irregular features.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0106", "text": "Pooling layer", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0107", "text": "Another important concept of CNNs is pooling, which is used as a form of non-linear down-sampling. Pooling provides downsampling because it reduces the spatial dimensions (height and width) of the input feature maps while retaining the most important information. There are several non-linear functions to implement pooling, where max pooling and average pooling are the most common", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0108", "text": ". Pooling aggregates information from small regions of the input creating partitions of the input feature map, typically using a fixed-size window (like 2x2) and applying a stride (often 2) to move the window across the input. Note that without using a stride greater than 1, pooling would not perform downsampling, as it would simply move the pooling window across the input one step at a time, without reducing the size of the feature map", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0109", "text": ". In other words, the stride is what actually causes the downsampling by determining how much the pooling window moves over the input.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0110", "text": "Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. The pooling layer serves to progressively reduce the spatial size of the representation, to reduce the number of parameters, memory footprint and amount of computation in the network, and hence to also control overfitting. This is known as down-sampling", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0111", "text": ". This is known as down-sampling. It is common to periodically insert a pooling layer between successive convolutional layers (each one typically followed by an activation function, such as a ReLU layer) in a CNN architecture. While pooling layers contribute to local translation invariance, they do not provide global translation invariance in a CNN, unless a form of global pooling is used", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0112", "text": ". The pooling layer commonly operates independently on every depth, or slice, of the input and resizes it spatially. A very common form of max pooling is a layer with filters of size 2\u00d72, applied with a stride of 2, which subsamples every depth slice in the input by 2 along both width and height, discarding 75% of the activations:", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0113", "text": "f\n          \n            X\n            ,\n            Y\n          \n        \n        (\n        S\n        )\n        =\n        \n          max\n          \n            a\n            ,\n            b\n            =\n            0\n          \n          \n            1\n          \n        \n        \n          S\n          \n            2\n            X\n            +\n            a\n            ,\n            2\n            Y\n            +\n            b\n          \n        \n        .", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0114", "text": "b\n          \n        \n        .\n      \n    \n    {\\displaystyle f_{X,Y}(S)=\\max _{a,b=0}^{1}S_{2X+a,2Y+b}.}", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0115", "text": "In this case, every max operation is over 4 numbers. The depth dimension remains unchanged (this is true for other forms of pooling as well).\nIn addition to max pooling, pooling units can use other functions, such as average pooling or \u21132-norm pooling. Average pooling was often used historically but has recently fallen out of favor compared to max pooling, which generally performs better in practice.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0116", "text": "Due to the effects of fast spatial reduction of the size of the representation, there is a recent trend towards using smaller filters or discarding pooling layers altogether.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0117", "text": "Channel max pooling", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0118", "text": "A channel max pooling (CMP) operation layer conducts the MP operation along the channel side among the corresponding positions of the consecutive feature maps for the purpose of redundant information elimination. The CMP makes the significant features gather together within fewer channels, which is important for fine-grained image classification that needs more discriminating features", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0119", "text": ". Meanwhile, another advantage of the CMP operation is to make the channel number of feature maps smaller before it connects to the first fully connected (FC) layer. Similar to the MP operation, we denote the input feature maps and output feature maps of a CMP layer as F \u2208 R(C\u00d7M\u00d7N) and C \u2208 R(c\u00d7M\u00d7N), respectively, where C and c are the channel numbers of the input and output feature maps, M and N are the widths and the height of the feature maps, respectively", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0120", "text": ". Note that the CMP operation only changes the channel number of the feature maps. The width and the height of the feature maps are not changed, which is different from the MP operation.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0121", "text": "See  for reviews for pooling methods.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0122", "text": "ReLU layer\nReLU is the abbreviation of rectified linear unit. It was proposed by Alston Householder in 1941, and used in CNN by Kunihiko Fukushima in 1969. ReLU applies the non-saturating activation function \n  \n    \n      \n        f\n        (\n        x\n        )\n        =\n        max\n        (\n        0\n        ,\n        x\n        )\n      \n    \n    {\\textstyle f(x)=\\max(0,x)}", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0123", "text": "{\\textstyle f(x)=\\max(0,x)}\n  \n. It effectively removes negative values from an activation map by setting them to zero. It introduces nonlinearity to the decision function and in the overall network without affecting the receptive fields of the convolution layers.\nIn 2011, Xavier Glorot, Antoine Bordes and Yoshua Bengio found that ReLU enables better training of deeper networks, compared to widely used activation functions prior to 2011.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0124", "text": "Other functions can also be used to increase nonlinearity, for example the saturating hyperbolic tangent \n  \n    \n      \n        f\n        (\n        x\n        )\n        =\n        tanh\n        \u2061\n        (\n        x\n        )\n      \n    \n    {\\displaystyle f(x)=\\tanh(x)}\n  \n, \n  \n    \n      \n        f\n        (\n        x\n        )\n        =\n        \n          |\n        \n        tanh\n        \u2061\n        (\n        x\n        )\n        \n          |", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0125", "text": "|\n        \n      \n    \n    {\\displaystyle f(x)=|\\tanh(x)|}\n  \n, and the sigmoid function \n  \n    \n      \n        \u03c3\n        (\n        x\n        )\n        =\n        (\n        1\n        +\n        \n          e\n          \n            \u2212\n            x\n          \n        \n        \n          )\n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\textstyle \\sigma (x)=(1+e^{-x})^{-1}}", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0126", "text": "{\\textstyle \\sigma (x)=(1+e^{-x})^{-1}}\n  \n. ReLU is often preferred to other functions because it trains the neural network several times faster without a significant penalty to generalization accuracy.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0127", "text": "Fully connected layer\nAfter several convolutional and max pooling layers, the final classification is done via fully connected layers. Neurons in a fully connected layer have connections to all activations in the previous layer, as seen in regular (non-convolutional) artificial neural networks. Their activations can thus be computed as an affine transformation, with matrix multiplication followed by a bias offset (vector addition of a learned or fixed bias term).", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0128", "text": "Loss layer\nThe \"loss layer\", or \"loss function\", exemplifies how training penalizes the deviation between the predicted output of the network, and the true data labels (during supervised learning). Various loss functions can be used, depending on the specific task.\nThe Softmax loss function is used for predicting a single class of K mutually exclusive classes. Sigmoid cross-entropy loss is used for predicting K independent probability values in \n  \n    \n      \n        [\n        0\n        ,", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0129", "text": "[\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle [0,1]}\n  \n. Euclidean loss is used for regressing to real-valued labels \n  \n    \n      \n        (\n        \u2212\n        \u221e\n        ,\n        \u221e\n        )\n      \n    \n    {\\displaystyle (-\\infty ,\\infty )}\n  \n.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0130", "text": "Hyperparameters\nHyperparameters are various settings that are used to control the learning process. CNNs use more hyperparameters than a standard multilayer perceptron (MLP).", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0131", "text": "Padding\nPadding is the addition of (typically) 0-valued pixels on the borders of an image. This is done so that the border pixels are not undervalued (lost) from the output because they would ordinarily participate in only a single receptive field instance. The padding applied is typically one less than the corresponding kernel dimension. For example, a convolutional layer using 3x3 kernels would receive a 2-pixel pad, that is 1 pixel on each side of the image.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0132", "text": "Stride\nThe stride is the number of pixels that the analysis window moves on each iteration. A stride of 2 means that each kernel is offset by 2 pixels from its predecessor.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0133", "text": "Number of filters\nSince feature map size decreases with depth, layers near the input layer tend to have fewer filters while higher layers can have more. To equalize computation at each layer, the product of feature values va with pixel position is kept roughly constant across layers. Preserving more information about the input would require keeping the total number of activations (number of feature maps times number of pixel positions) non-decreasing from one layer to the next.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0134", "text": "The number of feature maps directly controls the capacity and depends on the number of available examples and task complexity.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0135", "text": "Filter (or kernel) size\nCommon filter sizes found in the literature vary greatly, and are usually chosen based on the data set. Typical filter sizes range from 1x1 to 7x7. As two famous examples, AlexNet used 3x3, 5x5, and 11x11. Inceptionv3 used 1x1, 3x3, and 5x5.\nThe challenge is to find the right level of granularity so as to create abstractions at the proper scale, given a particular data set, and without overfitting.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0136", "text": "Pooling type and size\nMax pooling is typically used, often with a 2x2 dimension. This implies that the input is drastically downsampled, reducing processing cost.\nGreater pooling reduces the dimension of the signal, and may result in unacceptable information loss. Often, non-overlapping pooling windows perform best.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0137", "text": "Dilation", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0138", "text": "Dilation involves ignoring pixels within a kernel. This reduces processing memory potentially without significant signal loss. A dilation of 2 on a 3x3 kernel expands the kernel to 5x5, while still processing 9 (evenly spaced) pixels. Specifically, the processed pixels after the dilation are the cells (1,1), (1,3), (1,5), (3,1), (3,3), (3,5), (5,1), (5,3), (5,5), where (i,j) denotes the cell of the i-th row and j-th column in the expanded 5x5 kernel", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0139", "text": ". Accordingly, dilation of 4 expands the kernel to 7x7.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0140", "text": "Translation equivariance and aliasing", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0141", "text": "It is commonly assumed that CNNs are invariant to shifts of the input. Convolution or pooling layers within a CNN that do not have a stride greater than one are indeed equivariant to translations of the input", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0142", "text": ". However, layers with a stride greater than one ignore the Nyquist\u2013Shannon sampling theorem and might lead to aliasing of the input signal While, in principle, CNNs are capable of implementing anti-aliasing filters, it has been observed that this does not happen in practice, and therefore yield models that are not equivariant to translations.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0143", "text": "Furthermore, if a CNN makes use of fully connected layers, translation equivariance does not imply translation invariance, as the fully connected layers are not invariant to shifts of the input. One solution for complete translation invariance is avoiding any down-sampling throughout the network and applying global average pooling at the last layer", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0144", "text": ". Additionally, several other partial solutions have been proposed, such as anti-aliasing before downsampling operations, spatial transformer networks, data augmentation, subsampling combined with pooling, and capsule neural networks.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0145", "text": "Evaluation\nThe accuracy of the final model is typically estimated on a sub-part of the dataset set apart at the start, often called a test set. Alternatively, methods such as k-fold cross-validation are applied. Other strategies include using conformal prediction.\n\nRegularization methods\nRegularization is a process of introducing additional information to solve an ill-posed problem or to prevent overfitting. CNNs use various types of regularization.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0146", "text": "Empirical\nDropout\nBecause networks have so many parameters, they are prone to overfitting. One method to reduce overfitting is dropout, introduced in 2014. At each training stage, individual nodes are either \"dropped out\" of the net (ignored) with probability \n  \n    \n      \n        1\n        \u2212\n        p\n      \n    \n    {\\displaystyle 1-p}\n  \n or kept with probability \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0147", "text": "p\n      \n    \n    {\\displaystyle p}\n  \n, so that a reduced network is left; incoming and outgoing edges to a dropped-out node are also removed. Only the reduced network is trained on the data in that stage. The removed nodes are then reinserted into the network with their original weights.\nIn the training stages, \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0148", "text": "p\n      \n    \n    {\\displaystyle p}\n  \n is usually 0.5; for input nodes, it is typically much higher because information is directly lost when input nodes are ignored.\nAt testing time after training has finished, we would ideally like to find a sample average of all possible \n  \n    \n      \n        \n          2\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle 2^{n}}\n  \n dropped-out networks; unfortunately this is unfeasible for large values of", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0149", "text": "n\n      \n    \n    {\\displaystyle n}\n  \n. However, we can find an approximation by using the full network with each node's output weighted by a factor of \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n, so the expected value of the output of any node is the same as in the training stages. This is the biggest contribution of the dropout method: although it effectively generates \n  \n    \n      \n        \n          2\n          \n            n", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0150", "text": "n\n          \n        \n      \n    \n    {\\displaystyle 2^{n}}\n  \n neural nets, and as such allows for model combination, at test time only a single network needs to be tested.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0151", "text": "By avoiding training all nodes on all training data, dropout decreases overfitting. The method also significantly improves training speed. This makes the model combination practical, even for deep neural networks. The technique seems to reduce node interactions, leading them to learn more robust features that better generalize to new data.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0152", "text": "DropConnect\nDropConnect is the generalization of dropout in which each connection, rather than each output unit, can be dropped with probability \n  \n    \n      \n        1\n        \u2212\n        p\n      \n    \n    {\\displaystyle 1-p}\n  \n. Each unit thus receives input from a random subset of units in the previous layer.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0153", "text": "DropConnect is similar to dropout as it introduces dynamic sparsity within the model, but differs in that the sparsity is on the weights, rather than the output vectors of a layer. In other words, the fully connected layer with DropConnect becomes a sparsely connected layer in which the connections are chosen at random during the training stage.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0154", "text": "Stochastic pooling\nA major drawback to dropout is that it does not have the same benefits for convolutional layers, where the neurons are not fully connected.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0155", "text": "Even before dropout, in 2013 a technique called stochastic pooling, the conventional deterministic pooling operations were replaced with a stochastic procedure, where the activation within each pooling region is picked randomly according to a multinomial distribution, given by the activities within the pooling region. This approach is free of hyperparameters and can be combined with other regularization approaches, such as dropout and data augmentation.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0156", "text": "An alternate view of stochastic pooling is that it is equivalent to standard max pooling but with many copies of an input image, each having small local deformations. This is similar to explicit elastic deformations of the input images, which delivers excellent performance on the MNIST data set. Using stochastic pooling in a multilayer model gives an exponential number of deformations since the selections in higher layers are independent of those below.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0157", "text": "Artificial data", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0158", "text": "Because the degree of model overfitting is determined by both its power and the amount of training it receives, providing a convolutional network with more training examples can reduce overfitting. Because there is often not enough available data to train, especially considering that some part should be spared for later testing, two approaches are to either generate new data from scratch (if possible) or perturb existing data to create new ones. The latter one is used since mid-1990s", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0159", "text": ". The latter one is used since mid-1990s. For example, input images can be cropped, rotated, or rescaled to create new examples with the same labels as the original training set.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0160", "text": "Explicit\nEarly stopping\nOne of the simplest methods to prevent overfitting of a network is to simply stop the training before overfitting has had a chance to occur. It comes with the disadvantage that the learning process is halted.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0161", "text": "Number of parameters", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0162", "text": "Another simple way to prevent overfitting is to limit the number of parameters, typically by limiting the number of hidden units in each layer or limiting network depth. For convolutional networks, the filter size also affects the number of parameters. Limiting the number of parameters restricts the predictive power of the network directly, reducing the complexity of the function that it can perform on the data, and thus limits the amount of overfitting. This is equivalent to a \"zero norm\".", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0163", "text": "Weight decay\nA simple form of added regularizer is weight decay, which simply adds an additional error, proportional to the sum of weights (L1 norm) or squared magnitude (L2 norm) of the weight vector, to the error at each node. The level of acceptable model complexity can be reduced by increasing the proportionality constant('alpha' hyperparameter), thus increasing the penalty for large weight vectors.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0164", "text": "L2 regularization is the most common form of regularization. It can be implemented by penalizing the squared magnitude of all parameters directly in the objective. The L2 regularization has the intuitive interpretation of heavily penalizing peaky weight vectors and preferring diffuse weight vectors. Due to multiplicative interactions between weights and inputs this has the useful property of encouraging the network to use all of its inputs a little rather than some of its inputs a lot.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0165", "text": "L1 regularization is also common. It makes the weight vectors sparse during optimization. In other words, neurons with L1 regularization end up using only a sparse subset of their most important inputs and become nearly invariant to the noisy inputs. L1 with L2 regularization can be combined; this is called elastic net regularization.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0166", "text": "Max norm constraints\nAnother form of regularization is to enforce an absolute upper bound on the magnitude of the weight vector for every neuron and use projected gradient descent to enforce the constraint. In practice, this corresponds to performing the parameter update as normal, and then enforcing the constraint by clamping the weight vector \n  \n    \n      \n        \n          \n            \n              w\n              \u2192", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0167", "text": "{\\displaystyle {\\vec {w}}}\n  \n of every neuron to satisfy \n  \n    \n      \n        \u2016\n        \n          \n            \n              w\n              \u2192\n            \n          \n        \n        \n          \u2016\n          \n            2\n          \n        \n        <\n        c\n      \n    \n    {\\displaystyle \\|{\\vec {w}}\\|_{2}<c}\n  \n. Typical values of \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0168", "text": "c\n      \n    \n    {\\displaystyle c}\n  \n are order of 3\u20134. Some papers report improvements when using this form of regularization.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0169", "text": "Hierarchical coordinate frames", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0170", "text": "Pooling loses the precise spatial relationships between high-level parts (such as nose and mouth in a face image). These relationships are needed for identity recognition. Overlapping the pools so that each feature occurs in multiple pools, helps retain the information. Translation alone cannot extrapolate the understanding of geometric relationships to a radically new viewpoint, such as a different orientation or scale", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0171", "text": ". On the other hand, people are very good at extrapolating; after seeing a new shape once they can recognize it from a different viewpoint.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0172", "text": "An earlier common way to deal with this problem is to train the network on transformed data in different orientations, scales, lighting, etc. so that the network can cope with these variations. This is computationally intensive for large data-sets. The alternative is to use a hierarchy of coordinate frames and use a group of neurons to represent a conjunction of the shape of the feature and its pose relative to the retina", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0173", "text": ". The pose relative to the retina is the relationship between the coordinate frame of the retina and the intrinsic features' coordinate frame.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0174", "text": "Thus, one way to represent something is to embed the coordinate frame within it. This allows large features to be recognized by using the consistency of the poses of their parts (e.g. nose and mouth poses make a consistent prediction of the pose of the whole face). This approach ensures that the higher-level entity (e.g. face) is present when the lower-level (e.g. nose and mouth) agree on its prediction of the pose", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0175", "text": ". The vectors of neuronal activity that represent pose (\"pose vectors\") allow spatial transformations modeled as linear operations that make it easier for the network to learn the hierarchy of visual entities and generalize across viewpoints. This is similar to the way the human visual system imposes coordinate frames in order to represent shapes.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0176", "text": "Applications\nImage recognition\nCNNs are often used in image recognition systems. In 2012, an error rate of 0.23% on the MNIST database was reported. Another paper on using CNN for image classification reported that the learning process was \"surprisingly fast\"; in the same paper, the best published results as of 2011 were achieved in the MNIST database and the NORB database. Subsequently, a similar CNN called AlexNet won the ImageNet Large Scale Visual Recognition Challenge 2012.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0177", "text": "When applied to facial recognition, CNNs achieved a large decrease in error rate. Another paper reported a 97.6% recognition rate on \"5,600 still images of more than 10 subjects\". CNNs were used to assess video quality in an objective way after manual training; the resulting system had a very low root mean square error.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0178", "text": "The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object classification and detection, with millions of images and hundreds of object classes. In the ILSVRC 2014, a large-scale visual recognition challenge, almost every highly ranked team used CNN as their basic framework. The winner GoogLeNet (the foundation of DeepDream) increased the mean average precision of object detection to 0.439329, and reduced classification error to 0.06656, the best result to date", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0179", "text": ". Its network applied more than 30 layers. That performance of convolutional neural networks on the ImageNet tests was close to that of humans. The best algorithms still struggle with objects that are small or thin, such as a small ant on a stem of a flower or a person holding a quill in their hand. They also have trouble with images that have been distorted with filters, an increasingly common phenomenon with modern digital cameras. By contrast, those kinds of images rarely trouble humans", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0180", "text": ". Humans, however, tend to have trouble with other issues. For example, they are not good at classifying objects into fine-grained categories such as the particular breed of dog or species of bird, whereas convolutional neural networks handle this.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0181", "text": "In 2015, a many-layered CNN demonstrated the ability to spot faces from a wide range of angles, including upside down, even when partially occluded, with competitive performance. The network was trained on a database of 200,000 images that included faces at various angles and orientations and a further 20 million images without faces. They used batches of 128 images over 50,000 iterations.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0182", "text": "Video analysis", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0183", "text": "Compared to image data domains, there is relatively little work on applying CNNs to video classification. Video is more complex than images since it has another (temporal) dimension. However, some extensions of CNNs into the video domain have been explored. One approach is to treat space and time as equivalent dimensions of the input and perform convolutions in both time and space", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0184", "text": ". Another way is to fuse the features of two convolutional neural networks, one for the spatial and one for the temporal stream. Long short-term memory (LSTM) recurrent units are typically incorporated after the CNN to account for inter-frame or inter-clip dependencies. Unsupervised learning schemes for training spatio-temporal features have been introduced, based on Convolutional Gated Restricted Boltzmann Machines and Independent Subspace Analysis", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0185", "text": ". Its application can be seen in text-to-video model.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0186", "text": "Natural language processing\nCNNs have also been explored for natural language processing. CNN models are effective for various NLP problems and achieved excellent results in semantic parsing, search query retrieval, sentence modeling, classification, prediction and other traditional NLP tasks.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0187", "text": "Compared to traditional language processing methods such as recurrent neural networks, CNNs can represent different contextual realities of language that do not rely on a series-sequence assumption, while RNNs are better suitable when classical time series modeling is required.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0188", "text": "Animal behavior detection", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0189", "text": "CNNs have been applied in ecological and behavioral research to automatically detect and quantify animal behavior from visual data, enabling identification of animals, tracking of individuals, estimation of pose, and classification of specific actions such as feeding, and social interactions", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0190", "text": ". Combined with multi-object tracking and temporal modeling, these systems can extract behavioral sequences over extended recordings, reducing reliance on manual annotation and increasing throughput for studies of individual variation, social networks, and collective dynamics.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0191", "text": "Anomaly detection\nA CNN with 1-D convolutions was used on time series in the frequency domain (spectral residual) by an unsupervised model to detect anomalies in the time domain.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0192", "text": "Drug discovery", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0193", "text": "CNNs have been used in drug discovery. Predicting the interaction between molecules and biological proteins can identify potential treatments. In 2015, Atomwise introduced AtomNet, the first deep learning neural network for structure-based drug design. The system trains directly on 3-dimensional representations of chemical interactions", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0194", "text": ". Similar to how image recognition networks learn to compose smaller, spatially proximate features into larger, complex structures, AtomNet discovers chemical features, such as aromaticity, sp3 carbons, and hydrogen bonding. Subsequently, AtomNet was used to predict novel candidate biomolecules for multiple disease targets, most notably treatments for the Ebola virus and multiple sclerosis.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0195", "text": "Checkers game", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0196", "text": "CNNs have been used in the game of checkers. From 1999 to 2001, Fogel and Chellapilla published papers showing how a convolutional neural network could learn to play checkers using co-evolution. The learning process did not use prior human professional games, but rather focused on a minimal set of information contained in the checkerboard: the location and type of pieces, and the difference in number of pieces between the two sides", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0197", "text": ". Ultimately, the program (Blondie24) was tested on 165 games against players and ranked in the highest 0.4%. It also earned a win against the program Chinook at its \"expert\" level of play.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0198", "text": "Go", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0199", "text": "CNNs have been used in computer Go. In December 2014, Clark and Storkey published a paper showing that a CNN trained by supervised learning from a database of human professional games could outperform GNU Go and win some games against Monte Carlo tree search Fuego 1.1 in a fraction of the time it took Fuego to play", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0200", "text": ". Later it was announced that a large 12-layer convolutional neural network had correctly predicted the professional move in 55% of positions, equalling the accuracy of a 6 dan human player. When the trained convolutional network was used directly to play games of Go, without any search, it beat the traditional search program GNU Go in 97% of games, and matched the performance of the Monte Carlo tree search program Fuego simulating ten thousand playouts (about a million positions) per move.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0201", "text": "A couple of CNNs for choosing moves to try (\"policy network\") and evaluating positions (\"value network\") driving MCTS were used by AlphaGo, the first to beat the best human player at the time.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0202", "text": "Time series forecasting", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0203", "text": "Recurrent neural networks are generally considered the best neural network architectures for time series forecasting (and sequence modeling in general), but recent studies show that convolutional networks can perform comparably or even better. Dilated convolutions might enable one-dimensional convolutional neural networks to effectively learn time series dependences", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0204", "text": ". Convolutions can be implemented more efficiently than RNN-based solutions, and they do not suffer from vanishing (or exploding) gradients. Convolutional networks can provide an improved forecasting performance when there are multiple similar time series to learn from. CNNs can also be applied to further tasks in time series analysis (e.g., time series classification or quantile forecasting).", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0205", "text": "Cultural heritage and 3D-datasets", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0206", "text": "As archaeological findings such as clay tablets with cuneiform writing are increasingly acquired using 3D scanners, benchmark datasets are becoming available, including HeiCuBeDa providing almost 2000 normalized 2-D and 3-D datasets prepared with the GigaMesh Software Framework. So curvature-based measures are used in conjunction with geometric neural networks (GNNs), e.g. for period classification of those clay tablets being among the oldest documents of human history.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0207", "text": "Fine-tuning", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0208", "text": "For many applications, training data is not very available. Convolutional neural networks usually require a large amount of training data in order to avoid overfitting. A common technique is to train the network on a larger data set from a related domain. Once the network parameters have converged an additional training step is performed using the in-domain data to fine-tune the network weights, this is known as transfer learning", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0209", "text": ". Furthermore, this technique allows convolutional network architectures to successfully be applied to problems with tiny training sets.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0210", "text": "Human interpretable explanations\nEnd-to-end training and prediction are common practice in computer vision. However, human interpretable explanations are required for critical systems such as self-driving cars. With recent advances in visual salience, spatial attention, and temporal attention, the most critical spatial regions/temporal instants could be visualized to justify the CNN predictions.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0211", "text": "Related architectures\nDeep Q-networks\nA deep Q-network (DQN) is a type of deep learning model that combines a deep neural network with Q-learning, a form of reinforcement learning. Unlike earlier reinforcement learning agents, DQNs that utilize CNNs can learn directly from high-dimensional sensory inputs via reinforcement learning.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0212", "text": "Preliminary results were presented in 2014, with an accompanying paper in February 2015. The research described an application to Atari 2600 gaming. Other deep reinforcement learning models preceded it.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0213", "text": "Deep belief networks\nConvolutional deep belief networks (CDBN) have structure very similar to convolutional neural networks and are trained similarly to deep belief networks. Therefore, they exploit the 2D structure of images, like CNNs do, and make use of pre-training like deep belief networks. They provide a generic structure that can be used in many image and signal processing tasks. Benchmark results on standard image datasets like CIFAR have been obtained using CDBNs.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0214", "text": "Neural abstraction pyramid", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0215", "text": "The feed-forward architecture of convolutional neural networks was extended in the neural abstraction pyramid by lateral and feedback connections. The resulting recurrent convolutional network allows for the flexible incorporation of contextual information to iteratively resolve local ambiguities. In contrast to previous models, image-like outputs at the highest resolution were generated, e.g., for semantic segmentation, image reconstruction, and object localization tasks.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0216", "text": "Notable libraries\nCaffe: A library for convolutional neural networks. Created by the Berkeley Vision and Learning Center (BVLC). It supports both CPU and GPU. Developed in C++, and has Python and MATLAB wrappers.\nDeeplearning4j: Deep learning in Java and Scala on multi-GPU-enabled Spark. A general-purpose deep learning library for the JVM production stack running on a C++ scientific computing engine. Allows the creation of custom layers. Integrates with Hadoop and Kafka.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0217", "text": "Dlib: A toolkit for making real world machine learning and data analysis applications in C++.\nMicrosoft Cognitive Toolkit: A deep learning toolkit written by Microsoft with several unique features enhancing scalability over multiple nodes. It supports full-fledged interfaces for training in C++ and Python and with additional support for model inference in C# and Java.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0218", "text": "TensorFlow: Apache 2.0-licensed Theano-like library with support for CPU, GPU, Google's proprietary tensor processing unit (TPU), and mobile devices.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0219", "text": "Theano: The reference deep-learning library for Python with an API largely compatible with the popular NumPy library. Allows user to write symbolic mathematical expressions, then automatically generates their derivatives, saving the user from having to code gradients or backpropagation. These symbolic expressions are automatically compiled to CUDA code for a fast, on-the-GPU implementation.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0220", "text": "Torch: A scientific computing framework with wide support for machine learning algorithms, written in C and Lua.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0221", "text": "See also\nAttention (machine learning)\nCircuit (neural network)\nConvolution\nDeep learning\nNatural-language processing\nNeocognitron\nScale-invariant feature transform\nTime delay neural network\nVision processing unit", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Convolutional_neural_network_0222", "text": "Notes\nReferences\nExternal links\nCS231n: Convolutional Neural Networks for Visual Recognition \u2014 Andrej Karpathy's Stanford computer science course on CNNs in computer vision\nvdumoulin/conv_arithmetic: A technical report on convolution arithmetic in the context of deep learning. Animations of convolutions.", "title": "Convolutional neural network", "url": "https://en.wikipedia.org/wiki/Convolutional_neural_network"}, {"id": "rec_Recurrent_neural_network_0000", "text": "In artificial neural networks, recurrent neural networks (RNNs) are designed for processing sequential data, such as text, speech, and time series, where the order of elements is important. Unlike feedforward neural networks, which process inputs independently, RNNs utilize recurrent connections, where the output of a neuron at one time step is fed back as input to the network at the next time step. This enables RNNs to capture temporal dependencies and patterns within sequences.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0001", "text": "The fundamental building block of RNN is the recurrent unit, which maintains a hidden state\u2014a form of memory that is updated at each time step based on the current input and the previous hidden state. This feedback mechanism allows the network to learn from past inputs and incorporate that knowledge into its current processing", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0002", "text": ". RNNs have been successfully applied to tasks such as unsegmented, connected handwriting recognition, speech recognition, natural language processing, and neural machine translation.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0003", "text": "However, traditional RNNs suffer from the vanishing gradient problem, which limits their ability to learn long-range dependencies. This issue was addressed by the development of the long short-term memory (LSTM) architecture in 1997, making it the standard RNN variant for handling long-term dependencies. Later, gated recurrent units (GRUs) were introduced as a more computationally efficient alternative.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0004", "text": "In recent years, transformers, which rely on self-attention mechanisms instead of recurrence, have become the dominant architecture for many sequence-processing tasks, particularly in natural language processing, due to their superior handling of long-range dependencies and greater parallelizability. Nevertheless, RNNs remain relevant for applications where computational efficiency, real-time processing, or the inherent sequential nature of data is crucial.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0005", "text": "History\nBefore modern", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0006", "text": "One origin of RNN was neuroscience. The word \"recurrent\" is used to describe loop-like structures in anatomy. In 1901, Cajal observed \"recurrent semicircles\" in the cerebellar cortex formed by parallel fiber, Purkinje cells, and granule cells. In 1933, Lorente de N\u00f3 discovered \"recurrent, reciprocal connections\" by Golgi's method, and proposed that excitatory loops explain certain aspects of the vestibulo-ocular reflex", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0007", "text": ". During 1940s, multiple people proposed the existence of feedback in the brain, which was a contrast to the previous understanding of the neural system as a purely feedforward structure. Hebb considered \"reverberating circuit\" as an explanation for short-term memory. The McCulloch and Pitts paper (1943), which proposed the McCulloch-Pitts neuron model, considered networks that contains cycles. The current activity of such networks can be affected by activity indefinitely far in the past", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0008", "text": ". They were both interested in closed loops as possible explanations for e.g. epilepsy and causalgia. Recurrent inhibition was proposed in 1946 as a negative feedback mechanism in motor control. Neural feedback loops were a common topic of discussion at the Macy conferences. See  for an extensive review of recurrent neural network models in neuroscience.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0009", "text": "Frank Rosenblatt in 1960 published \"close-loop cross-coupled perceptrons\", which are 3-layered perceptron networks whose middle layer contains recurrent connections that change by a Hebbian learning rule", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0010", "text": ". Later, in Principles of Neurodynamics (1961), he described \"closed-loop cross-coupled\" and \"back-coupled\" perceptron networks, and made theoretical and experimental studies for Hebbian learning in these networks, and noted that a fully cross-coupled perceptron network is equivalent to an infinitely deep feedforward network.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0011", "text": "Similar networks were published by Kaoru Nakano in 1971,Shun'ichi Amari in 1972, and William A. Little in 1974, who was acknowledged by Hopfield in his 1982 paper.\nAnother origin of RNN was statistical mechanics. The Ising model was developed by Wilhelm Lenz and Ernst Ising in the 1920s as a simple statistical mechanical model of magnets at equilibrium. Glauber in 1963 studied the Ising model evolving in time, as a process towards equilibrium (Glauber dynamics), adding in the component of time.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0012", "text": "The Sherrington\u2013Kirkpatrick model of spin glass, published in 1975, is the Hopfield network with random initialization. Sherrington and Kirkpatrick found that it is highly likely for the energy function of the SK model to have many local minima. In the 1982 paper, Hopfield applied this recently developed theory to study the Hopfield network with binary activation functions. In a 1984 paper he extended this to continuous activation functions", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0013", "text": ". It became a standard model for the study of neural networks through statistical mechanics.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0014", "text": "Modern\nModern RNN networks are mainly based on two architectures: LSTM and BRNN.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0015", "text": "At the resurgence of neural networks in the 1980s, recurrent networks were studied again. They were sometimes called \"iterated nets\". Two early influential works were the Jordan network (1986) and the Elman network (1990), which applied RNN to study cognitive psychology. In 1993, a neural history compressor system solved a \"Very Deep Learning\" task that required more than 1000 subsequent layers in an RNN unfolded in time.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0016", "text": "Long short-term memory (LSTM) networks were invented by Hochreiter and Schmidhuber in 1995 and set accuracy records in multiple applications domains. It became the default choice for RNN architecture.\nBidirectional recurrent neural networks (BRNN) use two RNNs that process the same input in opposite directions. These two are often combined, giving the bidirectional LSTM architecture.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0017", "text": "Around 2006, bidirectional LSTM started to revolutionize speech recognition, outperforming traditional models in certain speech applications. They also improved large-vocabulary speech recognition and text-to-speech synthesis and was used in Google voice search, and dictation on Android devices. They broke records for improved machine translation, language modeling and Multilingual Language Processing", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0018", "text": ". Also, LSTM combined with convolutional neural networks (CNNs) improved automatic image captioning.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0019", "text": "The idea of encoder-decoder sequence transduction had been developed in the early 2010s. The papers most commonly cited as the originators that produced seq2seq are two papers from 2014. A seq2seq architecture employs two RNN, typically LSTM, an \"encoder\" and a \"decoder\", for sequence transduction, such as machine translation. They became state of the art in machine translation, and were instrumental in the development of attention mechanisms and transformers.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0020", "text": "Configurations\nAn RNN-based model can be factored into two parts: configuration and architecture. Multiple RNNs can be combined in a data flow, and the data flow itself is the configuration. Each RNN itself may have any architecture, including LSTM, GRU, etc.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0021", "text": "Standard\nRNNs come in many variants. Abstractly speaking, an RNN is a function \n  \n    \n      \n        \n          f\n          \n            \u03b8\n          \n        \n      \n    \n    {\\displaystyle f_{\\theta }}\n  \n of type \n  \n    \n      \n        (\n        \n          x\n          \n            t\n          \n        \n        ,\n        \n          h\n          \n            t\n          \n        \n        )\n        \u21a6\n        (\n        \n          y\n          \n            t\n          \n        \n        ,", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0022", "text": "t\n          \n        \n        ,\n        \n          h\n          \n            t\n            +\n            1\n          \n        \n        )\n      \n    \n    {\\displaystyle (x_{t},h_{t})\\mapsto (y_{t},h_{t+1})}\n  \n, where", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0023", "text": "x\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle x_{t}}\n  \n: input vector;\n\n  \n    \n      \n        \n          h\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle h_{t}}\n  \n: hidden vector;\n\n  \n    \n      \n        \n          y\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle y_{t}}\n  \n: output vector;", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0024", "text": "\u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n: neural network parameters.\nIn words, it is a neural network that maps an input \n  \n    \n      \n        \n          x\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle x_{t}}\n  \n into an output \n  \n    \n      \n        \n          y\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle y_{t}}\n  \n, with the hidden vector \n  \n    \n      \n        \n          h\n          \n            t", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0025", "text": "h\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle h_{t}}\n  \n playing the role of \"memory\", a partial record of all previous input-output pairs. At each step, it transforms input to an output, and modifies its \"memory\" to help it to better perform future processing.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0026", "text": "The illustration to the right may be misleading to many because practical neural network topologies are frequently organized in \"layers\" and the drawing gives that appearance. However, what appears to be layers are, in fact, different steps in time, \"unfolded\" to produce the appearance of layers.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0027", "text": "Stacked RNN\nA stacked RNN, or deep RNN, is composed of multiple RNNs stacked one above the other. Abstractly, it is structured as follows\nLayer 1 has hidden vector \n  \n    \n      \n        \n          h\n          \n            1\n            ,\n            t\n          \n        \n      \n    \n    {\\displaystyle h_{1,t}}\n  \n, parameters \n  \n    \n      \n        \n          \u03b8\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\theta _{1}}\n  \n, and maps", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0028", "text": ", and maps \n  \n    \n      \n        \n          f\n          \n            \n              \u03b8\n              \n                1\n              \n            \n          \n        \n        :\n        (\n        \n          x\n          \n            0\n            ,\n            t\n          \n        \n        ,\n        \n          h\n          \n            1\n            ,\n            t\n          \n        \n        )\n        \u21a6\n        (\n        \n          x\n          \n            1\n            ,\n            t", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0029", "text": "1\n            ,\n            t\n          \n        \n        ,\n        \n          h\n          \n            1\n            ,\n            t\n            +\n            1\n          \n        \n        )\n      \n    \n    {\\displaystyle f_{\\theta _{1}}:(x_{0,t},h_{1,t})\\mapsto (x_{1,t},h_{1,t+1})}\n  \n.\nLayer 2 has hidden vector \n  \n    \n      \n        \n          h\n          \n            2\n            ,\n            t\n          \n        \n      \n    \n    {\\displaystyle h_{2,t}}\n  \n, parameters", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0030", "text": "{\\displaystyle h_{2,t}}\n  \n, parameters \n  \n    \n      \n        \n          \u03b8\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\theta _{2}}\n  \n, and maps \n  \n    \n      \n        \n          f\n          \n            \n              \u03b8\n              \n                2\n              \n            \n          \n        \n        :\n        (\n        \n          x\n          \n            1\n            ,\n            t\n          \n        \n        ,\n        \n          h", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0031", "text": ",\n        \n          h\n          \n            2\n            ,\n            t\n          \n        \n        )\n        \u21a6\n        (\n        \n          x\n          \n            2\n            ,\n            t\n          \n        \n        ,\n        \n          h\n          \n            2\n            ,\n            t\n            +\n            1\n          \n        \n        )\n      \n    \n    {\\displaystyle f_{\\theta _{2}}:(x_{1,t},h_{2,t})\\mapsto (x_{2,t},h_{2,t+1})}\n  \n.\n...\nLayer", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0032", "text": ".\n...\nLayer \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n has hidden vector \n  \n    \n      \n        \n          h\n          \n            n\n            ,\n            t\n          \n        \n      \n    \n    {\\displaystyle h_{n,t}}\n  \n, parameters \n  \n    \n      \n        \n          \u03b8\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\theta _{n}}\n  \n, and maps \n  \n    \n      \n        \n          f\n          \n            \n              \u03b8", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0033", "text": "\u03b8\n              \n                n\n              \n            \n          \n        \n        :\n        (\n        \n          x\n          \n            n\n            \u2212\n            1\n            ,\n            t\n          \n        \n        ,\n        \n          h\n          \n            n\n            ,\n            t\n          \n        \n        )\n        \u21a6\n        (\n        \n          x\n          \n            n\n            ,\n            t\n          \n        \n        ,", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0034", "text": ",\n        \n          h\n          \n            n\n            ,\n            t\n            +\n            1\n          \n        \n        )\n      \n    \n    {\\displaystyle f_{\\theta _{n}}:(x_{n-1,t},h_{n,t})\\mapsto (x_{n,t},h_{n,t+1})}\n  \n.\nEach layer operates as a stand-alone RNN, and each layer's output sequence is used as the input sequence to the layer above. There is no conceptual limit to the depth of stacked RNN.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0035", "text": "Bidirectional\nA bidirectional RNN (biRNN) is composed of two RNNs, one processing the input sequence in one direction, and another in the opposite direction. Abstractly, it is structured as follows:\nThe forward RNN processes in one direction: \n  \n    \n      \n        \n          f\n          \n            \u03b8\n          \n        \n        (\n        \n          x\n          \n            0\n          \n        \n        ,\n        \n          h\n          \n            0\n          \n        \n        )\n        =", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0036", "text": ")\n        =\n        (\n        \n          y\n          \n            0\n          \n        \n        ,\n        \n          h\n          \n            1\n          \n        \n        )\n        ,\n        \n          f\n          \n            \u03b8\n          \n        \n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          h\n          \n            1\n          \n        \n        )\n        =\n        (\n        \n          y", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0037", "text": "(\n        \n          y\n          \n            1\n          \n        \n        ,\n        \n          h\n          \n            2\n          \n        \n        )\n        ,\n        \u2026\n      \n    \n    {\\displaystyle f_{\\theta }(x_{0},h_{0})=(y_{0},h_{1}),f_{\\theta }(x_{1},h_{1})=(y_{1},h_{2}),\\dots }", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0038", "text": "The backward RNN processes in the opposite direction:\n  \n    \n      \n        \n          f\n          \n            \n              \u03b8\n              \u2032\n            \n          \n          \u2032\n        \n        (\n        \n          x\n          \n            N\n          \n        \n        ,\n        \n          h\n          \n            N\n          \n          \u2032\n        \n        )\n        =\n        (\n        \n          y\n          \n            N\n          \n          \u2032\n        \n        ,\n        \n          h", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0039", "text": ",\n        \n          h\n          \n            N\n            \u2212\n            1\n          \n          \u2032\n        \n        )\n        ,\n        \n          f\n          \n            \n              \u03b8\n              \u2032\n            \n          \n          \u2032\n        \n        (\n        \n          x\n          \n            N\n            \u2212\n            1\n          \n        \n        ,\n        \n          h\n          \n            N\n            \u2212\n            1\n          \n          \u2032\n        \n        )", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0040", "text": "\u2032\n        \n        )\n        =\n        (\n        \n          y\n          \n            N\n            \u2212\n            1\n          \n          \u2032\n        \n        ,\n        \n          h\n          \n            N\n            \u2212\n            2\n          \n          \u2032\n        \n        )\n        ,\n        \u2026\n      \n    \n    {\\displaystyle f'_{\\theta '}(x_{N},h_{N}')=(y'_{N},h_{N-1}'),f'_{\\theta '}(x_{N-1},h_{N-1}')=(y'_{N-1},h_{N-2}'),\\dots }", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0041", "text": "The two output sequences are then concatenated to give the total output: \n  \n    \n      \n        (\n        (\n        \n          y\n          \n            0\n          \n        \n        ,\n        \n          y\n          \n            0\n          \n          \u2032\n        \n        )\n        ,\n        (\n        \n          y\n          \n            1\n          \n        \n        ,\n        \n          y\n          \n            1\n          \n          \u2032\n        \n        )\n        ,\n        \u2026\n        ,\n        (", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0042", "text": ")\n        ,\n        \u2026\n        ,\n        (\n        \n          y\n          \n            N\n          \n        \n        ,\n        \n          y\n          \n            N\n          \n          \u2032\n        \n        )\n        )\n      \n    \n    {\\displaystyle ((y_{0},y_{0}'),(y_{1},y_{1}'),\\dots ,(y_{N},y_{N}'))}\n  \n.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0043", "text": ".\nBidirectional RNN allows the model to process a token both in the context of what came before it and what came after it. By stacking multiple bidirectional RNNs together, the model can process a token increasingly contextually. The ELMo model (2018) is a stacked bidirectional LSTM which takes character-level as inputs and produces word-level embeddings.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0044", "text": "Encoder-decoder\nTwo RNNs can be run front-to-back in an encoder-decoder configuration. The encoder RNN processes an input sequence into a sequence of hidden vectors, and the decoder RNN processes the sequence of hidden vectors to an output sequence, with an optional attention mechanism. This was used to construct state of the art neural machine translators during the 2014\u20132017 period. This was an instrumental step towards the development of transformers.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0045", "text": "PixelRNN\nAn RNN may process data with more than one dimension. PixelRNN processes two-dimensional data, with many possible directions. For example, the row-by-row direction processes an \n  \n    \n      \n        n\n        \u00d7\n        n\n      \n    \n    {\\displaystyle n\\times n}\n  \n grid of vectors \n  \n    \n      \n        \n          x\n          \n            i\n            ,\n            j\n          \n        \n      \n    \n    {\\displaystyle x_{i,j}}\n  \n in the following order:", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0046", "text": "in the following order: \n  \n    \n      \n        \n          x\n          \n            1\n            ,\n            1\n          \n        \n        ,\n        \n          x\n          \n            1\n            ,\n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            1\n            ,\n            n\n          \n        \n        ,\n        \n          x\n          \n            2\n            ,\n            1\n          \n        \n        ,\n        \n          x", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0047", "text": ",\n        \n          x\n          \n            2\n            ,\n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            2\n            ,\n            n\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n            ,\n            n\n          \n        \n      \n    \n    {\\displaystyle x_{1,1},x_{1,2},\\dots ,x_{1,n},x_{2,1},x_{2,2},\\dots ,x_{2,n},\\dots ,x_{n,n}}", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0048", "text": "The diagonal BiLSTM uses two LSTMs to process the same grid. One processes it from the top-left corner to the bottom-right, such that it processes \n  \n    \n      \n        \n          x\n          \n            i\n            ,\n            j\n          \n        \n      \n    \n    {\\displaystyle x_{i,j}}\n  \n depending on its hidden state and cell state on the top and the left side: \n  \n    \n      \n        \n          h\n          \n            i\n            \u2212\n            1\n            ,\n            j", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0049", "text": "1\n            ,\n            j\n          \n        \n        ,\n        \n          c\n          \n            i\n            \u2212\n            1\n            ,\n            j\n          \n        \n      \n    \n    {\\displaystyle h_{i-1,j},c_{i-1,j}}\n  \n and \n  \n    \n      \n        \n          h\n          \n            i\n            ,\n            j\n            \u2212\n            1\n          \n        \n        ,\n        \n          c\n          \n            i\n            ,\n            j\n            \u2212", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0050", "text": ",\n            j\n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle h_{i,j-1},c_{i,j-1}}\n  \n. The other processes it from the top-right corner to the bottom-left.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0051", "text": "Architectures\nFully recurrent\nFully recurrent neural networks (FRNN) connect the outputs of all neurons to the inputs of all neurons. In other words, it is a fully connected network. This is the most general neural network topology, because all other topologies can be represented by setting some connection weights to zero to simulate the lack of connections between those neurons.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0052", "text": "Hopfield\nThe Hopfield network is an RNN in which all connections across layers are equally sized. It requires stationary inputs and is thus not a general RNN, as it does not process sequences of patterns. However, it guarantees that it will converge. If the connections are trained using Hebbian learning, then the Hopfield network can perform as robust content-addressable memory, resistant to connection alteration.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0053", "text": "Elman networks and Jordan networks", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0054", "text": "An Elman network is a three-layer network (arranged horizontally as x, y, and z in the illustration) with the addition of a set of context units (u in the illustration). The middle (hidden) layer is connected to these context units fixed with a weight of one. At each time step, the input is fed forward and a learning rule is applied", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0055", "text": ". The fixed back-connections save a copy of the previous values of the hidden units in the context units (since they propagate over the connections before the learning rule is applied). Thus the network can maintain a sort of state, allowing it to perform tasks such as sequence-prediction that are beyond the power of a standard multilayer perceptron.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0056", "text": "Jordan networks are similar to Elman networks. The context units are fed from the output layer instead of the hidden layer. The context units in a Jordan network are also called the state layer. They have a recurrent connection to themselves.\nElman and Jordan networks are also known as \"Simple recurrent networks\" (SRN).", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0057", "text": "Elman network", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0058", "text": "h\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    h\n                  \n                \n                (\n                \n                  W\n                  \n                    h", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0059", "text": "x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    h\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0060", "text": "b\n                  \n                    h\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  y\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    y\n                  \n                \n                (", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0061", "text": "(\n                \n                  W\n                  \n                    y\n                  \n                \n                \n                  h\n                  \n                    t\n                  \n                \n                +\n                \n                  b\n                  \n                    y\n                  \n                \n                )", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0062", "text": "{\\displaystyle {\\begin{aligned}h_{t}&=\\sigma _{h}(W_{h}x_{t}+U_{h}h_{t-1}+b_{h})\\\\y_{t}&=\\sigma _{y}(W_{y}h_{t}+b_{y})\\end{aligned}}}", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0063", "text": "Jordan network", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0064", "text": "h\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    h\n                  \n                \n                (\n                \n                  W\n                  \n                    h", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0065", "text": "x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    h\n                  \n                \n                \n                  s\n                  \n                    t\n                  \n                \n                +\n                \n                  b\n                  \n                    h", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0066", "text": "h\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  y\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    y\n                  \n                \n                (\n                \n                  W", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0067", "text": "W\n                  \n                    y\n                  \n                \n                \n                  h\n                  \n                    t\n                  \n                \n                +\n                \n                  b\n                  \n                    y\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  s", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0068", "text": "s\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    s\n                  \n                \n                (\n                \n                  W\n                  \n                    s\n                    ,\n                    s\n                  \n                \n                \n                  s", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0069", "text": "s\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  W\n                  \n                    s\n                    ,\n                    y\n                  \n                \n                \n                  y\n                  \n                    t\n                    \u2212\n                    1", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0070", "text": "+\n                \n                  b\n                  \n                    s\n                  \n                \n                )\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}h_{t}&=\\sigma _{h}(W_{h}x_{t}+U_{h}s_{t}+b_{h})\\\\y_{t}&=\\sigma _{y}(W_{y}h_{t}+b_{y})\\\\s_{t}&=\\sigma _{s}(W_{s,s}s_{t-1}+W_{s,y}y_{t-1}+b_{s})\\end{aligned}}}", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0071", "text": "Variables and functions\n\n  \n    \n      \n        \n          x\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle x_{t}}\n  \n: input vector\n\n  \n    \n      \n        \n          h\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle h_{t}}\n  \n: hidden layer vector\n\n  \n    \n      \n        \n          s\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle s_{t}}\n  \n: \"state\" vector,", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0072", "text": "y\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle y_{t}}\n  \n: output vector\n\n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}\n  \n, \n  \n    \n      \n        U\n      \n    \n    {\\displaystyle U}\n  \n and \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}\n  \n: parameter matrices and vector\n\n  \n    \n      \n        \u03c3\n      \n    \n    {\\displaystyle \\sigma }\n  \n: Activation functions", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0073", "text": "Long short-term memory", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0074", "text": "Long short-term memory (LSTM) is the most widely used RNN architecture. It was designed to solve the vanishing gradient problem. LSTM is normally augmented by recurrent gates called \"forget gates\". LSTM prevents backpropagated errors from vanishing or exploding. Instead, errors can flow backward through unlimited numbers of virtual layers unfolded in space. That is, LSTM can learn tasks that require memories of events that happened thousands or even millions of discrete time steps earlier", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0075", "text": ". Problem-specific LSTM-like topologies can be evolved. LSTM works even given long delays between significant events and can handle signals that mix low and high-frequency components.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0076", "text": "Many applications use stacks of LSTMs, for which it is called \"deep LSTM\". LSTM can learn to recognize context-sensitive languages unlike previous models based on hidden Markov models (HMM) and similar concepts.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0077", "text": "Gated recurrent unit\nGated recurrent unit (GRU), introduced in 2014, was designed as a simplification of LSTM. They are used in the full form and several further simplified variants. They have fewer parameters than LSTM, as they lack an output gate.\nTheir performance on polyphonic music modeling and speech signal modeling was found to be similar to that of long short-term memory. There does not appear to be particular performance difference between LSTM and GRU.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0078", "text": "Bidirectional associative memory", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0079", "text": "Introduced by Bart Kosko, a bidirectional associative memory (BAM) network is a variant of a Hopfield network that stores associative data as a vector. The bidirectionality comes from passing information through a matrix and its transpose. Typically, bipolar encoding is preferred to binary encoding of the associative pairs. Recently, stochastic BAM models using Markov stepping are optimized for increased network stability and relevance to real-world applications.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0080", "text": "A BAM network has two layers, either of which can be driven as an input to recall an association and produce an output on the other layer.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0081", "text": "Echo state\nEcho state networks (ESN) have a sparsely connected random hidden layer. The weights of output neurons are the only part of the network that can change (be trained). ESNs are good at reproducing certain time series. A variant for spiking neurons is known as a liquid state machine.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0082", "text": "Recursive", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0083", "text": "A recursive neural network is created by applying the same set of weights recursively over a differentiable graph-like structure by traversing the structure in topological order. Such networks are typically also trained by the reverse mode of automatic differentiation. They can process distributed representations of structure, such as logical terms. A special case of recursive neural networks is the RNN whose structure corresponds to a linear chain", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0084", "text": ". Recursive neural networks have been applied to natural language processing. The recursive neural tensor network uses a tensor-based composition function for all nodes in the tree.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0085", "text": "Neural Turing machines\nNeural Turing machines (NTMs) are a method of extending recurrent neural networks by coupling them to external memory resources with which they interact. The combined system is analogous to a Turing machine or Von Neumann architecture but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0086", "text": "Differentiable neural computers (DNCs) are an extension of neural Turing machines, allowing for the usage of fuzzy amounts of each memory address and a record of chronology.\nNeural network pushdown automata (NNPDA) are similar to NTMs, but tapes are replaced by analog stacks that are differentiable and trained. In this way, they are similar in complexity to recognizers of context free grammars (CFGs).", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0087", "text": "Recurrent neural networks are Turing complete and can run arbitrary programs to process arbitrary sequences of inputs.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0088", "text": "Training\nTeacher forcing\nAn RNN can be trained into a conditionally generative model of sequences, aka autoregression.\nConcretely, let us consider the problem of machine translation, that is, given a sequence \n  \n    \n      \n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        )", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0089", "text": ")\n      \n    \n    {\\displaystyle (x_{1},x_{2},\\dots ,x_{n})}\n  \n of English words, the model is to produce a sequence \n  \n    \n      \n        (\n        \n          y\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          y\n          \n            m\n          \n        \n        )\n      \n    \n    {\\displaystyle (y_{1},\\dots ,y_{m})}\n  \n of French words. It is to be solved by a seq2seq model.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0090", "text": "Now, during training, the encoder half of the model would first ingest \n  \n    \n      \n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle (x_{1},x_{2},\\dots ,x_{n})}\n  \n, then the decoder half would start generating a sequence \n  \n    \n      \n        (", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0091", "text": "(\n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            1\n          \n        \n        ,\n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \n            \n              \n                y\n                ^", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0092", "text": "^\n              \n            \n          \n          \n            l\n          \n        \n        )\n      \n    \n    {\\displaystyle ({\\hat {y}}_{1},{\\hat {y}}_{2},\\dots ,{\\hat {y}}_{l})}\n  \n. The problem is that if the model makes a mistake early on, say at \n  \n    \n      \n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle {\\hat {y}}_{2}}", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0093", "text": "{\\displaystyle {\\hat {y}}_{2}}\n  \n, then subsequent tokens are likely to also be mistakes. This makes it inefficient for the model to obtain a learning signal, since the model would mostly learn to shift \n  \n    \n      \n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle {\\hat {y}}_{2}}\n  \n towards \n  \n    \n      \n        \n          y", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0094", "text": "towards \n  \n    \n      \n        \n          y\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle y_{2}}\n  \n, but not the others.\nTeacher forcing makes it so that the decoder uses the correct output sequence for generating the next entry in the sequence. So for example, it would see \n  \n    \n      \n        (\n        \n          y\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          y\n          \n            k", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0095", "text": "k\n          \n        \n        )\n      \n    \n    {\\displaystyle (y_{1},\\dots ,y_{k})}\n  \n in order to generate \n  \n    \n      \n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            k\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle {\\hat {y}}_{k+1}}\n  \n.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0096", "text": "Gradient descent\nGradient descent is a first-order iterative optimization algorithm for finding the minimum of a function. In neural networks, it can be used to minimize the error term by changing each weight in proportion to the derivative of the error with respect to that weight, provided the non-linear activation functions are differentiable.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0097", "text": "The standard method for training RNN by gradient descent is the \"backpropagation through time\" (BPTT) algorithm, which is a special case of the general algorithm of backpropagation. A more computationally expensive online variant is called \"Real-Time Recurrent Learning\" or RTRL, which is an instance of automatic differentiation in the forward accumulation mode with stacked tangent vectors. Unlike BPTT, this algorithm is local in time but not local in space.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0098", "text": "In this context, local in space means that a unit's weight vector can be updated using only information stored in the connected units and the unit itself such that update complexity of a single unit is linear in the dimensionality of the weight vector. Local in time means that the updates take place continually (on-line) and depend only on the most recent time step rather than on multiple time steps within a given time horizon as in BPTT", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0099", "text": ". Biological neural networks appear to be local with respect to both time and space.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0100", "text": "For recursively computing the partial derivatives, RTRL has a time-complexity of O(number of hidden x number of weights) per time step for computing the Jacobian matrices, while BPTT only takes O(number of weights) per time step, at the cost of storing all forward activations within the given time horizon. An online hybrid between BPTT and RTRL with intermediate complexity exists, along with variants for continuous time.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0101", "text": "A major problem with gradient descent for standard RNN architectures is that error gradients vanish exponentially quickly with the size of the time lag between important events. LSTM combined with a BPTT/RTRL hybrid learning method attempts to overcome these problems. This problem is also solved in the independently recurrent neural network (IndRNN) by reducing the context of a neuron to its own past state and the cross-neuron information can then be explored in the following layers", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0102", "text": ". Memories of different ranges including long-term memory can be learned without the gradient vanishing and exploding problems.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0103", "text": "The online algorithm called causal recursive backpropagation (CRBP), implements and combines BPTT and RTRL paradigms for locally recurrent networks. It works with the most general locally recurrent networks. The CRBP algorithm can minimize the global error term. This fact improves the stability of the algorithm, providing a unifying view of gradient calculation techniques for recurrent networks with local feedback.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0104", "text": "One approach to gradient information computation in RNNs with arbitrary architectures is based on signal-flow graphs diagrammatic derivation. It uses the BPTT batch algorithm, based on Lee's theorem for network sensitivity calculations. It was proposed by Wan and Beaufays, while its fast online version was proposed by Campolucci, Uncini and Piazza.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0105", "text": "Connectionist temporal classification\nThe connectionist temporal classification (CTC) is a specialized loss function for training RNNs for sequence modeling problems where the timing is variable.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0106", "text": "Global optimization methods", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0107", "text": "Training the weights in a neural network can be modeled as a non-linear global optimization problem. A target function can be formed to evaluate the fitness or error of a particular weight vector as follows: First, the weights in the network are set according to the weight vector. Next, the network is evaluated against the training sequence", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0108", "text": ". Typically, the sum-squared difference between the predictions and the target values specified in the training sequence is used to represent the error of the current weight vector. Arbitrary global optimization techniques may then be used to minimize this target function.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0109", "text": "The most common global optimization method for training RNNs is genetic algorithms, especially in unstructured networks.\nInitially, the genetic algorithm is encoded with the neural network weights in a predefined manner where one gene in the chromosome represents one weight link. The whole network is represented as a single chromosome. The fitness function is evaluated as follows:", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0110", "text": "Each weight encoded in the chromosome is assigned to the respective weight link of the network.\nThe training set is presented to the network which propagates the input signals forward.\nThe mean-squared error is returned to the fitness function.\nThis function drives the genetic selection process.\nMany chromosomes make up the population; therefore, many different neural networks are evolved until a stopping criterion is satisfied. A common stopping scheme can be:", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0111", "text": "When the neural network has learned a certain percentage of the training data.\nWhen the minimum value of the mean-squared-error is satisfied.\nWhen the maximum number of training generations has been reached.\nThe fitness function evaluates the stopping criterion as it receives the mean-squared error reciprocal from each network during training. Therefore, the goal of the genetic algorithm is to maximize the fitness function, reducing the mean-squared error.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0112", "text": "Other global (and/or evolutionary) optimization techniques may be used to seek a good set of weights, such as simulated annealing or particle swarm optimization.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0113", "text": "Other architectures\nIndependently RNN (IndRNN)", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0114", "text": "The independently recurrent neural network (IndRNN) addresses the gradient vanishing and exploding problems in the traditional fully connected RNN. Each neuron in one layer only receives its own past state as context information (instead of full connectivity to all other neurons in this layer) and thus neurons are independent of each other's history. The gradient backpropagation can be regulated to avoid gradient vanishing and exploding in order to keep long or short-term memory", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0115", "text": ". The cross-neuron information is explored in the next layers. IndRNN can be robustly trained with non-saturated nonlinear functions such as ReLU. Deep networks can be trained using skip connections.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0116", "text": "Neural history compressor", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0117", "text": "The neural history compressor is an unsupervised stack of RNNs. At the input level, it learns to predict its next input from the previous inputs. Only unpredictable inputs of some RNN in the hierarchy become inputs to the next higher level RNN, which therefore recomputes its internal state only rarely. Each higher level RNN thus studies a compressed representation of the information in the RNN below", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0118", "text": ". This is done such that the input sequence can be precisely reconstructed from the representation at the highest level.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0119", "text": "The system effectively minimizes the description length or the negative logarithm of the probability of the data. Given a lot of learnable predictability in the incoming data sequence, the highest level RNN can use supervised learning to easily classify even deep sequences with long intervals between important events.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0120", "text": "It is possible to distill the RNN hierarchy into two RNNs: the \"conscious\" chunker (higher level) and the \"subconscious\" automatizer (lower level). Once the chunker has learned to predict and compress inputs that are unpredictable by the automatizer, then the automatizer can be forced in the next learning phase to predict or imitate through additional units the hidden units of the more slowly changing chunker", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0121", "text": ". This makes it easy for the automatizer to learn appropriate, rarely changing memories across long intervals. In turn, this helps the automatizer to make many of its once unpredictable inputs predictable, such that the chunker can focus on the remaining unpredictable events.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0122", "text": "A generative model partially overcame the vanishing gradient problem of automatic differentiation or backpropagation in neural networks in 1992. In 1993, such a system solved a \"Very Deep Learning\" task that required more than 1000 subsequent layers in an RNN unfolded in time.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0123", "text": "Second order RNNs\nSecond-order RNNs use higher order weights \n  \n    \n      \n        w\n        \n          \n\n          \n          \n            i\n            j\n            k\n          \n        \n      \n    \n    {\\displaystyle w{}_{ijk}}\n  \n instead of the standard \n  \n    \n      \n        w", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0124", "text": "i\n            j\n          \n        \n      \n    \n    {\\displaystyle w{}_{ij}}\n  \n weights, and states can be a product. This allows a direct mapping to a finite-state machine both in training, and representation. Long short-term memory is an example of this but has no such formal mappings or proof of stability.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0125", "text": "Hierarchical recurrent neural network\nHierarchical recurrent neural networks (HRNN) connect their neurons in various ways to decompose hierarchical behavior into useful subprograms. Such hierarchical structures of cognition are present in theories of memory presented by philosopher Henri Bergson, whose philosophical views have inspired hierarchical models.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0126", "text": "Hierarchical recurrent neural networks are useful in forecasting, helping to predict disaggregated inflation components of the consumer price index (CPI). The HRNN model leverages information from higher levels in the CPI hierarchy to enhance lower-level predictions. Evaluation of a substantial dataset from the US CPI-U index demonstrates the superior performance of the HRNN model compared to various established inflation prediction methods.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0127", "text": "Recurrent multilayer perceptron network\nGenerally, a recurrent multilayer perceptron network (RMLP network) consists of cascaded subnetworks, each containing multiple layers of nodes. Each subnetwork is feed-forward except for the last layer, which can have feedback connections. Each of these subnets is connected only by feed-forward connections.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0128", "text": "Multiple timescales model", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0129", "text": "A multiple timescales recurrent neural network (MTRNN) is a neural-based computational model that can simulate the functional hierarchy of the brain through self-organization depending on the spatial connection between neurons and on distinct types of neuron activities, each with distinct time properties", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0130", "text": ". With such varied neuronal activities, continuous sequences of any set of behaviors are segmented into reusable primitives, which in turn are flexibly integrated into diverse sequential behaviors. The biological approval of such a type of hierarchy was discussed in the memory-prediction theory of brain function by Hawkins in his book On Intelligence. Such a hierarchy also agrees with theories of memory posited by philosopher Henri Bergson, which have been incorporated into an MTRNN model.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0131", "text": "Memristive networks", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0132", "text": "Greg Snider of HP Labs describes a system of cortical computing with memristive nanodevices. The memristors (memory resistors) are implemented by thin film materials in which the resistance is electrically tuned via the transport of ions or oxygen vacancies within the film", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0133", "text": ". DARPA's SyNAPSE project has funded IBM Research and HP Labs, in collaboration with the Boston University Department of Cognitive and Neural Systems (CNS), to develop neuromorphic architectures that may be based on memristive systems.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0134", "text": "Memristive networks are a particular type of physical neural network that have very similar properties to (Little-)Hopfield networks, as they have continuous dynamics, a limited memory capacity and natural relaxation via the minimization of a function which is asymptotic to the Ising model. In this sense, the dynamics of a memristive circuit have the advantage compared to a Resistor-Capacitor network to have a more interesting non-linear behavior", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0135", "text": ". From this point of view, engineering analog memristive networks account for a peculiar type of neuromorphic engineering in which the device behavior depends on the circuit wiring or topology.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0136", "text": "The evolution of these networks can be studied analytically using variations of the Caravelli-Traversa-Di Ventra equation.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0137", "text": "Continuous-time\nA continuous-time recurrent neural network (CTRNN) uses a system of ordinary differential equations to model the effects on a neuron of the incoming inputs. They are typically analyzed by dynamical systems theory. Many RNN models in neuroscience are continuous-time.\nFor a neuron \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n in the network with activation \n  \n    \n      \n        \n          y\n          \n            i", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0138", "text": "i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}\n  \n, the rate of change of activation is given by:", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0139", "text": "\u03c4\n          \n            i\n          \n        \n        \n          \n            \n              \n                y\n                \u02d9\n              \n            \n          \n          \n            i\n          \n        \n        =\n        \u2212\n        \n          y\n          \n            i\n          \n        \n        +\n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          w", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0140", "text": "w\n          \n            j\n            i\n          \n        \n        \u03c3\n        (\n        \n          y\n          \n            j\n          \n        \n        \u2212\n        \n          \u0398\n          \n            j\n          \n        \n        )\n        +\n        \n          I\n          \n            i\n          \n        \n        (\n        t\n        )\n      \n    \n    {\\displaystyle \\tau _{i}{\\dot {y}}_{i}=-y_{i}+\\sum _{j=1}^{n}w_{ji}\\sigma (y_{j}-\\Theta _{j})+I_{i}(t)}", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0141", "text": "Where:\n\n  \n    \n      \n        \n          \u03c4\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\tau _{i}}\n  \n : Time constant of postsynaptic node\n\n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}\n  \n : Activation of postsynaptic node", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0142", "text": "y\n                \u02d9\n              \n            \n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle {\\dot {y}}_{i}}\n  \n : Rate of change of activation of postsynaptic node\n\n  \n    \n      \n        w", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0143", "text": "w\n        \n          \n\n          \n          \n            j\n            i\n          \n        \n      \n    \n    {\\displaystyle w{}_{ji}}\n  \n : Weight of connection from pre to postsynaptic node", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0144", "text": "\u03c3\n        (\n        x\n        )\n      \n    \n    {\\displaystyle \\sigma (x)}\n  \n : Sigmoid of x e.g. \n  \n    \n      \n        \u03c3\n        (\n        x\n        )\n        =\n        1\n        \n          /\n        \n        (\n        1\n        +\n        \n          e\n          \n            \u2212\n            x\n          \n        \n        )\n      \n    \n    {\\displaystyle \\sigma (x)=1/(1+e^{-x})}\n  \n.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0145", "text": "y\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle y_{j}}\n  \n : Activation of presynaptic node\n\n  \n    \n      \n        \n          \u0398\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\Theta _{j}}\n  \n : Bias of presynaptic node", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0146", "text": "I\n          \n            i\n          \n        \n        (\n        t\n        )\n      \n    \n    {\\displaystyle I_{i}(t)}\n  \n : Input (if any) to node\nCTRNNs have been applied to evolutionary robotics where they have been used to address vision, co-operation, and minimal cognitive behaviour.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0147", "text": "Note that, by the Shannon sampling theorem, discrete-time recurrent neural networks can be viewed as continuous-time recurrent neural networks where the differential equations have transformed into equivalent difference equations. This transformation can be thought of as occurring after the post-synaptic node activation functions \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n        (\n        t\n        )\n      \n    \n    {\\displaystyle y_{i}(t)}", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0148", "text": "{\\displaystyle y_{i}(t)}\n  \n have been low-pass filtered but prior to sampling.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0149", "text": "They are in fact recursive neural networks with a particular structure: that of a linear chain. Whereas recursive neural networks operate on any hierarchical structure, combining child representations into parent representations, recurrent neural networks operate on the linear progression of time, combining the previous time step and a hidden representation into the representation for the current time step.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0150", "text": "From a time-series perspective, RNNs can appear as nonlinear versions of finite impulse response and infinite impulse response filters and also as a nonlinear autoregressive exogenous model (NARX). RNN has infinite impulse response whereas convolutional neural network has finite impulse response. Both classes of networks exhibit temporal dynamic behavior", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0151", "text": ". A finite impulse recurrent network is a directed acyclic graph that can be unrolled and replaced with a strictly feedforward neural network, while an infinite impulse recurrent network is a directed cyclic graph that cannot be unrolled.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0152", "text": "The effect of memory-based learning for the recognition of sequences can also be implemented by a more biological-based model which uses the silencing mechanism exhibited in neurons with a relatively high frequency spiking activity.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0153", "text": "Additional stored states and the storage under direct control by the network can be added to both infinite-impulse and finite-impulse networks. Another network or graph can also replace the storage if that incorporates time delays or has feedback loops. Such controlled states are referred to as gated states or gated memory and are part of long short-term memory networks (LSTMs) and gated recurrent units. This is also called Feedback Neural Network (FNN).", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0154", "text": "Libraries\nModern libraries provide runtime-optimized implementations of the above functionality or allow to speed up the slow loop by just-in-time compilation.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0155", "text": "Apache Singa\nCaffe: Created by the Berkeley Vision and Learning Center (BVLC). It supports both CPU and GPU. Developed in C++, and has Python and MATLAB wrappers.\nChainer: Fully in Python, production support for CPU, GPU, distributed training.\nDeeplearning4j: Deep learning in Java and Scala on multi-GPU-enabled Spark.\nFlux: includes interfaces for RNNs, including GRUs and LSTMs, written in Julia.\nKeras: High-level API, providing a wrapper to many other deep learning libraries.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0156", "text": "Microsoft Cognitive Toolkit\nMXNet: an open-source deep learning framework used to train and deploy deep neural networks.\nPyTorch: Tensors and Dynamic neural networks in Python with GPU acceleration.\nTensorFlow: Apache 2.0-licensed Theano-like library with support for CPU, GPU and Google's proprietary TPU, mobile\nTheano: A deep-learning library for Python with an API largely compatible with the NumPy library.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0157", "text": "Torch: A scientific computing framework with support for machine learning algorithms, written in C and Lua.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0158", "text": "Applications\nApplications of recurrent neural networks include:", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0159", "text": "Machine translation\nRobot control\nTime series prediction\nSpeech recognition\nSpeech synthesis\nBrain\u2013computer interfaces\nTime series anomaly detection\nText-to-Video model\nEnergy forecasting\nRhythm learning\nMusic composition\nGrammar learning\nHandwriting recognition\nHuman action recognition\nProtein homology detection\nPredicting subcellular localization of proteins\nSeveral prediction tasks in the area of business process management\nPrediction in medical care pathways", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0160", "text": "Prediction in medical care pathways\nPredictions of fusion plasma disruptions in reactors (Fusion Recurrent Neural Network (FRNN) code)", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0161", "text": "References\nFurther reading\nMandic, Danilo P.; Chambers, Jonathon A. (2001). Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability. Wiley. ISBN 978-0-471-49517-8.\nGrossberg, Stephen (2013-02-22). \"Recurrent Neural Networks\". Scholarpedia. 8 (2): 1888. Bibcode:2013SchpJ...8.1888G. doi:10.4249/scholarpedia.1888. ISSN 1941-6016.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Recurrent_neural_network_0162", "text": "Recurrent Neural Networks. List of RNN papers by J\u00fcrgen Schmidhuber's group at Dalle Molle Institute for Artificial Intelligence Research.", "title": "Recurrent neural network", "url": "https://en.wikipedia.org/wiki/Recurrent_neural_network"}, {"id": "rec_Long_shortterm_memory_0000", "text": "Long short-term memory (LSTM) is a type of recurrent neural network (RNN) aimed at mitigating the vanishing gradient problem commonly encountered by traditional RNNs. Its relative insensitivity to gap length is its advantage over other RNNs, hidden Markov models, and other sequence learning methods. It aims to provide a short-term memory for RNN that can last thousands of timesteps (thus \"long short-term memory\")", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0001", "text": ". The name is made in analogy with long-term memory and short-term memory and their relationship, studied by cognitive psychologists since the early 20th century.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0002", "text": "An LSTM unit is typically composed of a cell and three gates: an input gate, an output gate, and a forget gate. The cell remembers values over arbitrary time intervals, and the gates regulate the flow of information into and out of the cell. Forget gates decide what information to discard from the previous state, by mapping the previous state and the current input to a value between 0 and 1. A (rounded) value of 1 signifies retention of the information, and a value of 0 represents discarding", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0003", "text": ". Input gates decide which pieces of new information to store in the current cell state, using the same system as forget gates. Output gates control which pieces of information in the current cell state to output, by assigning a value from 0 to 1 to the information, considering the previous and current states", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0004", "text": ". Selectively outputting relevant information from the current state allows the LSTM network to maintain useful, long-term dependencies to make predictions, both in current and future time-steps.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0005", "text": "LSTM has wide applications in classification, data processing, time series analysis tasks, speech recognition,  machine translation, speech activity detection, robot control, video games, healthcare, energy forecasting.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0006", "text": "Motivation", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0007", "text": "In theory, classic RNNs can keep track of arbitrary long-term dependencies in the input sequences. The problem with classic RNNs is computational (or practical) in nature: when training a classic RNN using back-propagation, the long-term gradients which are back-propagated can \"vanish\", meaning they can tend to zero due to very small numbers creeping into the computations, causing the model to effectively stop learning", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0008", "text": ". RNNs using LSTM units partially solve the vanishing gradient problem, because LSTM units allow gradients to also flow with little to no attenuation. However, LSTM networks can still suffer from the exploding gradient problem.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0009", "text": "The intuition behind the LSTM architecture is to create an additional module in a neural network that learns when to remember and when to forget pertinent information. In other words, the network effectively learns which information might be needed later on in a sequence and when that information is no longer needed. For instance, in the context of natural language processing, the network can learn grammatical dependencies", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0010", "text": ". An LSTM might process the sentence \"Dave, as a result of his controversial claims, is now a pariah\" by remembering the (statistically likely) grammatical gender and number of the subject Dave. Note that this information is pertinent for the pronoun his and note that this information is no longer important after the verb is.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0011", "text": "Variants\nIn the equations below, the lowercase variables represent vectors. Matrices \n  \n    \n      \n        \n          W\n          \n            q\n          \n        \n      \n    \n    {\\displaystyle W_{q}}\n  \n and \n  \n    \n      \n        \n          U\n          \n            q\n          \n        \n      \n    \n    {\\displaystyle U_{q}}\n  \n contain, respectively, the weights of the input and recurrent connections, where the subscript \n  \n    \n      \n        \n          \n          \n            q", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0012", "text": "q\n          \n        \n      \n    \n    {\\displaystyle _{q}}\n  \n can either be the input gate \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n, output gate \n  \n    \n      \n        o\n      \n    \n    {\\displaystyle o}\n  \n, the forget gate \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n or the memory cell \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0013", "text": "c\n      \n    \n    {\\displaystyle c}\n  \n, depending on the activation being calculated. In this section, we are thus using a \"vector notation\". So, for example, \n  \n    \n      \n        \n          c\n          \n            t\n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            h\n          \n        \n      \n    \n    {\\displaystyle c_{t}\\in \\mathbb {R} ^{h}}\n  \n is not just one unit of one LSTM cell, but contains \n  \n    \n      \n        h", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0014", "text": "h\n      \n    \n    {\\displaystyle h}\n  \n LSTM cell's units.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0015", "text": "LSTM with a forget gate\nThe compact forms of the equations for the forward pass of an LSTM cell with a forget gate are:", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0016", "text": "f\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    g\n                  \n                \n                (\n                \n                  W\n                  \n                    f", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0017", "text": "x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    f\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0018", "text": "b\n                  \n                    f\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  i\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    g\n                  \n                \n                (", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0019", "text": "(\n                \n                  W\n                  \n                    i\n                  \n                \n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    i\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0020", "text": "t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    i\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  o\n                  \n                    t\n                  \n                \n              \n              \n                \n                =", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0021", "text": "=\n                \n                  \u03c3\n                  \n                    g\n                  \n                \n                (\n                \n                  W\n                  \n                    o\n                  \n                \n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0022", "text": "U\n                  \n                    o\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    o\n                  \n                \n                )", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0023", "text": "c\n                        ~\n                      \n                    \n                  \n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    c\n                  \n                \n                (", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0024", "text": "(\n                \n                  W\n                  \n                    c\n                  \n                \n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    c\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212\n                    1", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0025", "text": "\u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    c\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  c\n                  \n                    t\n                  \n                \n              \n              \n                \n                =", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0026", "text": "=\n                \n                  f\n                  \n                    t\n                  \n                \n                \u2299\n                \n                  c\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  i\n                  \n                    t\n                  \n                \n                \u2299", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0027", "text": "c\n                        ~\n                      \n                    \n                  \n                  \n                    t\n                  \n                \n              \n            \n            \n              \n                \n                  h\n                  \n                    t", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0028", "text": "=\n                \n                  o\n                  \n                    t\n                  \n                \n                \u2299\n                \n                  \u03c3\n                  \n                    h\n                  \n                \n                (\n                \n                  c\n                  \n                    t\n                  \n                \n                )", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0029", "text": "{\\displaystyle {\\begin{aligned}f_{t}&=\\sigma _{g}(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})\\\\i_{t}&=\\sigma _{g}(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})\\\\o_{t}&=\\sigma _{g}(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})\\\\{\\tilde {c}}_{t}&=\\sigma _{c}(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})\\\\c_{t}&=f_{t}\\odot c_{t-1}+i_{t}\\odot {\\tilde {c}}_{t}\\\\h_{t}&=o_{t}\\odot \\sigma _{h}(c_{t})\\end{aligned}}}", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0030", "text": "where the initial values are \n  \n    \n      \n        \n          c\n          \n            0\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle c_{0}=0}\n  \n and \n  \n    \n      \n        \n          h\n          \n            0\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle h_{0}=0}\n  \n and the operator \n  \n    \n      \n        \u2299\n      \n    \n    {\\displaystyle \\odot }\n  \n denotes the Hadamard product (element-wise product). The subscript", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0031", "text": "t\n      \n    \n    {\\displaystyle t}\n  \n indexes the time step.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0032", "text": "Variables\nLetting the superscripts \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n and \n  \n    \n      \n        h\n      \n    \n    {\\displaystyle h}\n  \n refer to the number of input features and number of hidden units, respectively:", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0033", "text": "x\n          \n            t\n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            d\n          \n        \n      \n    \n    {\\displaystyle x_{t}\\in \\mathbb {R} ^{d}}\n  \n: input vector to the LSTM unit", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0034", "text": "f\n          \n            t\n          \n        \n        \u2208\n        \n          \n            (\n            0\n            ,\n            1\n            )\n          \n          \n            h\n          \n        \n      \n    \n    {\\displaystyle f_{t}\\in {(0,1)}^{h}}\n  \n: forget gate's activation vector", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0035", "text": "i\n          \n            t\n          \n        \n        \u2208\n        \n          \n            (\n            0\n            ,\n            1\n            )\n          \n          \n            h\n          \n        \n      \n    \n    {\\displaystyle i_{t}\\in {(0,1)}^{h}}\n  \n: input/update gate's activation vector", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0036", "text": "o\n          \n            t\n          \n        \n        \u2208\n        \n          \n            (\n            0\n            ,\n            1\n            )\n          \n          \n            h\n          \n        \n      \n    \n    {\\displaystyle o_{t}\\in {(0,1)}^{h}}\n  \n: output gate's activation vector", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0037", "text": "h\n          \n            t\n          \n        \n        \u2208\n        \n          \n            (\n            \u2212\n            1\n            ,\n            1\n            )\n          \n          \n            h\n          \n        \n      \n    \n    {\\displaystyle h_{t}\\in {(-1,1)}^{h}}\n  \n: hidden state vector also known as output vector of the LSTM unit", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0038", "text": "c\n                ~\n              \n            \n          \n          \n            t\n          \n        \n        \u2208\n        \n          \n            (\n            \u2212\n            1\n            ,\n            1\n            )\n          \n          \n            h\n          \n        \n      \n    \n    {\\displaystyle {\\tilde {c}}_{t}\\in {(-1,1)}^{h}}\n  \n: cell input activation vector", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0039", "text": "c\n          \n            t\n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            h\n          \n        \n      \n    \n    {\\displaystyle c_{t}\\in \\mathbb {R} ^{h}}\n  \n: cell state vector", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0040", "text": "W\n        \u2208\n        \n          \n            R\n          \n          \n            h\n            \u00d7\n            d\n          \n        \n      \n    \n    {\\displaystyle W\\in \\mathbb {R} ^{h\\times d}}\n  \n, \n  \n    \n      \n        U\n        \u2208\n        \n          \n            R\n          \n          \n            h\n            \u00d7\n            h\n          \n        \n      \n    \n    {\\displaystyle U\\in \\mathbb {R} ^{h\\times h}}\n  \n and \n  \n    \n      \n        b\n        \u2208", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0041", "text": "and \n  \n    \n      \n        b\n        \u2208\n        \n          \n            R\n          \n          \n            h\n          \n        \n      \n    \n    {\\displaystyle b\\in \\mathbb {R} ^{h}}\n  \n: weight matrices and bias vector parameters which need to be learned during training", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0042", "text": "Activation functions\n\u03c3\n          \n            g\n          \n        \n      \n    \n    {\\displaystyle \\sigma _{g}}\n  \n: sigmoid function.\n\n  \n    \n      \n        \n          \u03c3\n          \n            c\n          \n        \n      \n    \n    {\\displaystyle \\sigma _{c}}\n  \n: hyperbolic tangent function.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0043", "text": "\u03c3\n          \n            h\n          \n        \n      \n    \n    {\\displaystyle \\sigma _{h}}\n  \n: hyperbolic tangent function or, as the peephole LSTM paper suggests, \n  \n    \n      \n        \n          \u03c3\n          \n            h\n          \n        \n        (\n        x\n        )\n        =\n        x\n      \n    \n    {\\displaystyle \\sigma _{h}(x)=x}\n  \n.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0044", "text": "Peephole LSTM\nThe figure on the right is a graphical representation of an LSTM unit with peephole connections (i.e. a peephole LSTM). Peephole connections allow the gates to access the constant error carousel (CEC), whose activation is the cell state. \n  \n    \n      \n        \n          h\n          \n            t\n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle h_{t-1}}\n  \n is not used, \n  \n    \n      \n        \n          c\n          \n            t\n            \u2212", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0045", "text": "t\n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle c_{t-1}}\n  \n is used instead in most places.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0046", "text": "f\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    g\n                  \n                \n                (\n                \n                  W\n                  \n                    f", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0047", "text": "x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    f\n                  \n                \n                \n                  c\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0048", "text": "b\n                  \n                    f\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  i\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    g\n                  \n                \n                (", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0049", "text": "(\n                \n                  W\n                  \n                    i\n                  \n                \n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    i\n                  \n                \n                \n                  c\n                  \n                    t\n                    \u2212", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0050", "text": "t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    i\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  o\n                  \n                    t\n                  \n                \n              \n              \n                \n                =", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0051", "text": "=\n                \n                  \u03c3\n                  \n                    g\n                  \n                \n                (\n                \n                  W\n                  \n                    o\n                  \n                \n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0052", "text": "U\n                  \n                    o\n                  \n                \n                \n                  c\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    o\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  c", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0053", "text": "c\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  f\n                  \n                    t\n                  \n                \n                \u2299\n                \n                  c\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0054", "text": "+\n                \n                  i\n                  \n                    t\n                  \n                \n                \u2299\n                \n                  \u03c3\n                  \n                    c\n                  \n                \n                (\n                \n                  W\n                  \n                    c\n                  \n                \n                \n                  x\n                  \n                    t", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0055", "text": "t\n                  \n                \n                +\n                \n                  b\n                  \n                    c\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  h\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  o", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0056", "text": "o\n                  \n                    t\n                  \n                \n                \u2299\n                \n                  \u03c3\n                  \n                    h\n                  \n                \n                (\n                \n                  c\n                  \n                    t\n                  \n                \n                )", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0057", "text": "{\\displaystyle {\\begin{aligned}f_{t}&=\\sigma _{g}(W_{f}x_{t}+U_{f}c_{t-1}+b_{f})\\\\i_{t}&=\\sigma _{g}(W_{i}x_{t}+U_{i}c_{t-1}+b_{i})\\\\o_{t}&=\\sigma _{g}(W_{o}x_{t}+U_{o}c_{t-1}+b_{o})\\\\c_{t}&=f_{t}\\odot c_{t-1}+i_{t}\\odot \\sigma _{c}(W_{c}x_{t}+b_{c})\\\\h_{t}&=o_{t}\\odot \\sigma _{h}(c_{t})\\end{aligned}}}", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0058", "text": "Each of the gates can be thought as a \"standard\" neuron in a feed-forward (or multi-layer) neural network: that is, they compute an activation (using an activation function) of a weighted sum. \n  \n    \n      \n        \n          i\n          \n            t\n          \n        \n        ,\n        \n          o\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle i_{t},o_{t}}\n  \n and \n  \n    \n      \n        \n          f\n          \n            t", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0059", "text": "t\n          \n        \n      \n    \n    {\\displaystyle f_{t}}\n  \n represent the activations of respectively the input, output and forget gates, at time step \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n.\nThe 3 exit arrows from the memory cell \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n to the 3 gates \n  \n    \n      \n        i\n        ,\n        o\n      \n    \n    {\\displaystyle i,o}\n  \n and \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0060", "text": "f\n      \n    \n    {\\displaystyle f}\n  \n represent the peephole connections. These peephole connections actually denote the contributions of the activation of the memory cell \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n at time step \n  \n    \n      \n        t\n        \u2212\n        1\n      \n    \n    {\\displaystyle t-1}\n  \n, i.e. the contribution of \n  \n    \n      \n        \n          c\n          \n            t\n            \u2212\n            1", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0061", "text": "1\n          \n        \n      \n    \n    {\\displaystyle c_{t-1}}\n  \n (and not \n  \n    \n      \n        \n          c\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle c_{t}}\n  \n, as the picture may suggest). In other words, the gates \n  \n    \n      \n        i\n        ,\n        o\n      \n    \n    {\\displaystyle i,o}\n  \n and \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n calculate their activations at time step \n  \n    \n      \n        t", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0062", "text": "t\n      \n    \n    {\\displaystyle t}\n  \n (i.e., respectively, \n  \n    \n      \n        \n          i\n          \n            t\n          \n        \n        ,\n        \n          o\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle i_{t},o_{t}}\n  \n and \n  \n    \n      \n        \n          f\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle f_{t}}\n  \n) also considering the activation of the memory cell \n  \n    \n      \n        c", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0063", "text": "c\n      \n    \n    {\\displaystyle c}\n  \n at time step \n  \n    \n      \n        t\n        \u2212\n        1\n      \n    \n    {\\displaystyle t-1}\n  \n, i.e. \n  \n    \n      \n        \n          c\n          \n            t\n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle c_{t-1}}\n  \n.\nThe single left-to-right arrow exiting the memory cell is not a peephole connection and denotes \n  \n    \n      \n        \n          c\n          \n            t", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0064", "text": "c\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle c_{t}}\n  \n.\nThe little circles containing a \n  \n    \n      \n        \u00d7\n      \n    \n    {\\displaystyle \\times }\n  \n symbol represent an element-wise multiplication between its inputs. The big circles containing an S-like curve represent the application of a differentiable function (like the sigmoid function) to a weighted sum.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0065", "text": "Peephole convolutional LSTM\nPeephole convolutional LSTM. The \n  \n    \n      \n        \u2217\n      \n    \n    {\\displaystyle *}\n  \n denotes the convolution operator.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0066", "text": "f\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    g\n                  \n                \n                (\n                \n                  W\n                  \n                    f", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0067", "text": "\u2217\n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    f\n                  \n                \n                \u2217\n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0068", "text": "+\n                \n                  V\n                  \n                    f\n                  \n                \n                \u2299\n                \n                  c\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    f\n                  \n                \n                )", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0069", "text": ")\n              \n            \n            \n              \n                \n                  i\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    g\n                  \n                \n                (\n                \n                  W\n                  \n                    i", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0070", "text": "i\n                  \n                \n                \u2217\n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    i\n                  \n                \n                \u2217\n                \n                  h\n                  \n                    t\n                    \u2212\n                    1", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0071", "text": "+\n                \n                  V\n                  \n                    i\n                  \n                \n                \u2299\n                \n                  c\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    i\n                  \n                \n                )", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0072", "text": ")\n              \n            \n            \n              \n                \n                  c\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  f\n                  \n                    t\n                  \n                \n                \u2299\n                \n                  c\n                  \n                    t", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0073", "text": "t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  i\n                  \n                    t\n                  \n                \n                \u2299\n                \n                  \u03c3\n                  \n                    c\n                  \n                \n                (\n                \n                  W\n                  \n                    c", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0074", "text": "c\n                  \n                \n                \u2217\n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    c\n                  \n                \n                \u2217\n                \n                  h\n                  \n                    t\n                    \u2212\n                    1", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0075", "text": "+\n                \n                  b\n                  \n                    c\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  o\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  \u03c3\n                  \n                    g", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0076", "text": "g\n                  \n                \n                (\n                \n                  W\n                  \n                    o\n                  \n                \n                \u2217\n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    o\n                  \n                \n                \u2217", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0077", "text": "\u2217\n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  V\n                  \n                    o\n                  \n                \n                \u2299\n                \n                  c\n                  \n                    t\n                  \n                \n                +", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0078", "text": "+\n                \n                  b\n                  \n                    o\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  h\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  o\n                  \n                    t", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0079", "text": "\u2299\n                \n                  \u03c3\n                  \n                    h\n                  \n                \n                (\n                \n                  c\n                  \n                    t\n                  \n                \n                )", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0080", "text": "{\\displaystyle {\\begin{aligned}f_{t}&=\\sigma _{g}(W_{f}*x_{t}+U_{f}*h_{t-1}+V_{f}\\odot c_{t-1}+b_{f})\\\\i_{t}&=\\sigma _{g}(W_{i}*x_{t}+U_{i}*h_{t-1}+V_{i}\\odot c_{t-1}+b_{i})\\\\c_{t}&=f_{t}\\odot c_{t-1}+i_{t}\\odot \\sigma _{c}(W_{c}*x_{t}+U_{c}*h_{t-1}+b_{c})\\\\o_{t}&=\\sigma _{g}(W_{o}*x_{t}+U_{o}*h_{t-1}+V_{o}\\odot c_{t}+b_{o})\\\\h_{t}&=o_{t}\\odot \\sigma _{h}(c_{t})\\end{aligned}}}", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0081", "text": "Training\nAn RNN using LSTM units can be trained in a supervised fashion on a set of training sequences, using an optimization algorithm like gradient descent combined with backpropagation through time to compute the gradients needed during the optimization process, in order to change each weight of the LSTM network in proportion to the derivative of the error (at the output layer of the LSTM network) with respect to corresponding weight.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0082", "text": "A problem with using gradient descent for standard RNNs is that error gradients vanish exponentially quickly with the size of the time lag between important events. This is due to \n  \n    \n      \n        \n          lim\n          \n            n\n            \u2192\n            \u221e\n          \n        \n        \n          W\n          \n            n\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle \\lim _{n\\to \\infty }W^{n}=0}\n  \n if the spectral radius of \n  \n    \n      \n        W", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0083", "text": "W\n      \n    \n    {\\displaystyle W}\n  \n is smaller than 1.\nHowever, with LSTM units, when error values are back-propagated from the output layer, the error remains in the LSTM unit's cell. This \"error carousel\" continuously feeds error back to each of the LSTM unit's gates, until they learn to cut off the value.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0084", "text": "CTC score function\nMany applications use stacks of LSTM RNNs and train them by connectionist temporal classification (CTC) to find an RNN weight matrix that maximizes the probability of the label sequences in a training set, given the corresponding input sequences. CTC achieves both alignment and recognition.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0085", "text": "Alternatives\nSometimes, it can be advantageous to train (parts of) an LSTM by neuroevolution or by policy gradient methods, especially when there is no \"teacher\" (that is, training labels).\n\nApplications\nApplications of LSTM include:", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0086", "text": "2015: Google started using an LSTM trained by CTC for speech recognition on Google Voice. According to the official blog post, the new model cut transcription errors by 49%.\n2016: Google started using an LSTM to suggest messages in the Allo conversation app. In the same year, Google released the Google Neural Machine Translation system for Google Translate which used LSTMs to reduce translation errors by 60%.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0087", "text": "Apple announced in its Worldwide Developers Conference that it would start using the LSTM for quicktype in the iPhone and for Siri.\nAmazon released Polly, which generates the voices behind Alexa, using a bidirectional LSTM for the text-to-speech technology.\n2017:  Facebook performed some 4.5 billion automatic translations every day using long short-term memory networks.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0088", "text": "Microsoft reported reaching 94.9% recognition accuracy on the Switchboard corpus, incorporating a vocabulary of 165,000 words. The approach used \"dialog session-based long-short-term memory\".\n2018: OpenAI used LSTM trained by policy gradients to beat humans in the complex video game of Dota 2, and to control a human-like robot hand that manipulates physical objects with unprecedented dexterity.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0089", "text": "2019: DeepMind used LSTM trained by policy gradients to excel at the complex video game of Starcraft II.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0090", "text": "History\nDevelopment\nAspects of LSTM were anticipated by \"focused back-propagation\", cited by the LSTM paper.\nSepp Hochreiter's 1991 German diploma thesis analyzed the vanishing gradient problem and developed principles of the method. His supervisor, J\u00fcrgen Schmidhuber, considered the thesis highly significant.\nAn early version of LSTM was published in 1995 in a technical report by Sepp Hochreiter and J\u00fcrgen Schmidhuber, then published in the NIPS 1996 conference.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0091", "text": "The most commonly used reference point for LSTM was published in 1997 in the journal Neural Computation. By introducing Constant Error Carousel (CEC) units, LSTM deals with the vanishing gradient problem. The initial version of LSTM block included cells, input and output gates.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0092", "text": "Felix Gers, J\u00fcrgen Schmidhuber, and Fred Cummins introduced the forget gate (also called \"keep gate\") into the LSTM architecture in 1999, enabling the LSTM to reset its own state. This is the most commonly used version of LSTM nowadays. They added peephole connections in 2000. Additionally, the output activation function was omitted.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0093", "text": "Development of variants\nIn 2005 Graves and Schmidhuber published LSTM with full backpropagation through time and bidirectional LSTM.\nIn 2006 Graves, Fernandez, Gomez, and Schmidhuber introduced a new error function for LSTM: Connectionist Temporal Classification (CTC) for simultaneous alignment and recognition of sequences.\nIn 2014 Kyunghyun Cho an others published a simplified variant of the forget gate LSTM called Gated recurrent unit (GRU).", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0094", "text": "In 2015 Srivastava, Greff, and Schmidhuber used LSTM principles to create the Highway network, a feedforward neural network with hundreds of layers, much deeper than previous networks. Concurrently, the ResNet architecture was developed. It is equivalent to an open-gated or gateless highway network.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0095", "text": "A modern upgrade of LSTM called xLSTM is published by a team led by Sepp Hochreiter. One of the 2 blocks (mLSTM) of the architecture are parallelizable like the Transformer architecture, the other ones (sLSTM) allow state tracking.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0096", "text": "Applications\n2001: Gers and Schmidhuber trained LSTM to learn languages unlearnable by traditional models such as Hidden Markov Models.\nHochreiter et al. used LSTM for meta-learning (i.e. learning a learning algorithm).\n2004: First successful application of LSTM to speech Alex Graves et al.\n2005: Daan Wierstra, Faustino Gomez, and Schmidhuber trained LSTM by neuroevolution without a teacher.\nMayer et al. trained LSTM to control robots.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0097", "text": "Mayer et al. trained LSTM to control robots.\n2007: Wierstra, Foerster, Peters, and Schmidhuber trained LSTM by policy gradients for reinforcement learning without a teacher.\nHochreiter, Heuesel, and Obermayr applied LSTM to protein homology detection the field of biology.\n2009: Justin Bayer et al. introduced neural architecture search for LSTM.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0098", "text": "2009: An LSTM trained by CTC won the ICDAR connected handwriting recognition competition. Three such models were submitted by a team led by Alex Graves. One was the most accurate model in the competition and another was the fastest. This was the first time an RNN won international competitions.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0099", "text": "2013: Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton used LSTM networks as a major component of a network that achieved a record 17.7% phoneme error rate on the classic TIMIT natural speech dataset.\n2017: Researchers from Michigan State University, IBM Research, and Cornell University published a study in the Knowledge Discovery and Data Mining (KDD) conference. Their time-aware LSTM (T-LSTM) performs better on certain data sets than standard LSTM.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0100", "text": "See also\nReferences\nFurther reading\nMonner, Derek D.; Reggia, James A. (2010). \"A generalized LSTM-like training algorithm for second-order recurrent neural networks\" (PDF). Neural Networks. 25 (1): 70\u201383. doi:10.1016/j.neunet.2011.07.003. PMC 3217173. PMID 21803542. High-performing extension of LSTM that has been simplified to a single node type and can train arbitrary architectures", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0101", "text": "Gers, Felix A.; Schraudolph, Nicol N.; Schmidhuber, J\u00fcrgen (Aug 2002). \"Learning precise timing with LSTM recurrent networks\" (PDF). Journal of Machine Learning Research. 3: 115\u2013143.\nGers, Felix (2001). \"Long Short-Term Memory in Recurrent Neural Networks\" (PDF). PhD thesis.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0102", "text": "Abidogun, Olusola Adeniyi (2005). Data Mining, Fraud Detection and Mobile Telecommunications: Call Pattern Analysis with Unsupervised Neural Networks. Master's Thesis (Thesis). University of the Western Cape. hdl:11394/249. Archived (PDF) from the original on May 22, 2012.\noriginal with two chapters devoted to explaining recurrent neural networks, especially LSTM.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Long_shortterm_memory_0103", "text": "External links\nRecurrent Neural Networks with over 30 LSTM papers by J\u00fcrgen Schmidhuber's group at IDSIA\nZhang, Aston; Lipton, Zachary; Li, Mu; Smola, Alexander J. (2024). \"10.1. Long Short-Term Memory (LSTM)\". Dive into deep learning. Cambridge New York Port Melbourne New Delhi Singapore: Cambridge University Press. ISBN 978-1-009-38943-3.", "title": "Long short-term memory", "url": "https://en.wikipedia.org/wiki/Long_short-term_memory"}, {"id": "rec_Gated_recurrent_unit_0000", "text": "In artificial neural networks, the gated recurrent unit (GRU) is a gating mechanism used in recurrent neural networks, introduced in 2014 by Kyunghyun Cho et al. The GRU is like a long short-term memory (LSTM) with a gating mechanism to input or forget certain features, but lacks a context vector or output gate, resulting in fewer parameters than LSTM.", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0001", "text": "GRU's performance on certain tasks of polyphonic music modeling, speech signal modeling and natural language processing was found to be similar to that of LSTM. GRUs showed that gating is indeed helpful in general, and Bengio's team came to no concrete conclusion on which of the two gating units was better.", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0002", "text": "Architecture\nThere are several variations on the full gated unit, with gating done using the previous hidden state and the bias in various combinations, and a simplified form called minimal gated unit.\nIn the following, the operator \n  \n    \n      \n        \u2299\n      \n    \n    {\\displaystyle \\odot }\n  \n denotes the Hadamard product.", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0003", "text": "Fully gated unit\nInitially, for \n  \n    \n      \n        t\n        =\n        0\n      \n    \n    {\\displaystyle t=0}\n  \n, the output vector is \n  \n    \n      \n        \n          h\n          \n            0\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle h_{0}=0}\n  \n.", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0004", "text": "z\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03c3\n                (\n                \n                  W\n                  \n                    z\n                  \n                \n                \n                  x\n                  \n                    t", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0005", "text": "t\n                  \n                \n                +\n                \n                  U\n                  \n                    z\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    z\n                  \n                \n                )", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0006", "text": ")\n              \n            \n            \n              \n                \n                  r\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03c3\n                (\n                \n                  W\n                  \n                    r\n                  \n                \n                \n                  x\n                  \n                    t", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0007", "text": "t\n                  \n                \n                +\n                \n                  U\n                  \n                    r\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    r", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0008", "text": ")\n              \n            \n            \n              \n                \n                  \n                    \n                      \n                        h\n                        ^\n                      \n                    \n                  \n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03d5\n                (", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0009", "text": "(\n                \n                  W\n                  \n                    h\n                  \n                \n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    h\n                  \n                \n                (\n                \n                  r\n                  \n                    t", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0010", "text": "t\n                  \n                \n                \u2299\n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                )\n                +\n                \n                  b\n                  \n                    h\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  h", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0011", "text": "h\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                (\n                1\n                \u2212\n                \n                  z\n                  \n                    t\n                  \n                \n                )\n                \u2299\n                \n                  h\n                  \n                    t\n                    \u2212", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0012", "text": "t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  z\n                  \n                    t\n                  \n                \n                \u2299\n                \n                  \n                    \n                      \n                        h\n                        ^\n                      \n                    \n                  \n                  \n                    t", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0013", "text": "t\n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}z_{t}&=\\sigma (W_{z}x_{t}+U_{z}h_{t-1}+b_{z})\\\\r_{t}&=\\sigma (W_{r}x_{t}+U_{r}h_{t-1}+b_{r})\\\\{\\hat {h}}_{t}&=\\phi (W_{h}x_{t}+U_{h}(r_{t}\\odot h_{t-1})+b_{h})\\\\h_{t}&=(1-z_{t})\\odot h_{t-1}+z_{t}\\odot {\\hat {h}}_{t}\\end{aligned}}}", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0014", "text": "Variables (\n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n denotes the number of input features and \n  \n    \n      \n        e\n      \n    \n    {\\displaystyle e}\n  \n the number of output features):\n\n  \n    \n      \n        \n          x\n          \n            t\n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            d\n          \n        \n      \n    \n    {\\displaystyle x_{t}\\in \\mathbb {R} ^{d}}\n  \n: input vector", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0015", "text": "h\n          \n            t\n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            e\n          \n        \n      \n    \n    {\\displaystyle h_{t}\\in \\mathbb {R} ^{e}}\n  \n: output vector", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0016", "text": "h\n                ^\n              \n            \n          \n          \n            t\n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            e\n          \n        \n      \n    \n    {\\displaystyle {\\hat {h}}_{t}\\in \\mathbb {R} ^{e}}\n  \n: candidate activation vector", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0017", "text": "z\n          \n            t\n          \n        \n        \u2208\n        (\n        0\n        ,\n        1\n        \n          )\n          \n            e\n          \n        \n      \n    \n    {\\displaystyle z_{t}\\in (0,1)^{e}}\n  \n: update gate vector", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0018", "text": "r\n          \n            t\n          \n        \n        \u2208\n        (\n        0\n        ,\n        1\n        \n          )\n          \n            e\n          \n        \n      \n    \n    {\\displaystyle r_{t}\\in (0,1)^{e}}\n  \n: reset gate vector", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0019", "text": "W\n        \u2208\n        \n          \n            R\n          \n          \n            e\n            \u00d7\n            d\n          \n        \n      \n    \n    {\\displaystyle W\\in \\mathbb {R} ^{e\\times d}}\n  \n, \n  \n    \n      \n        U\n        \u2208\n        \n          \n            R\n          \n          \n            e\n            \u00d7\n            e\n          \n        \n      \n    \n    {\\displaystyle U\\in \\mathbb {R} ^{e\\times e}}\n  \n and \n  \n    \n      \n        b\n        \u2208", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0020", "text": "and \n  \n    \n      \n        b\n        \u2208\n        \n          \n            R\n          \n          \n            e\n          \n        \n      \n    \n    {\\displaystyle b\\in \\mathbb {R} ^{e}}\n  \n: parameter matrices and vector which need to be learned during training\nActivation functions", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0021", "text": "\u03c3\n      \n    \n    {\\displaystyle \\sigma }\n  \n: The original is a logistic function.\n\n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n: The original is a hyperbolic tangent.\nAlternative activation functions are possible, provided that \n  \n    \n      \n        \u03c3\n        (\n        x\n        )\n        \u2208\n        [\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle \\sigma (x)\\in [0,1]}\n  \n.", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0022", "text": "Alternate forms can be created by changing \n  \n    \n      \n        \n          z\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle z_{t}}\n  \n and \n  \n    \n      \n        \n          r\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle r_{t}}\n  \n\nType 1: each gate depends only on the previous hidden state and the bias.", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0023", "text": "z\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03c3\n                (\n                \n                  U\n                  \n                    z\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0024", "text": "t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    z\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  r\n                  \n                    t\n                  \n                \n              \n              \n                \n                =", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0025", "text": "=\n                \u03c3\n                (\n                \n                  U\n                  \n                    r\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    r", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0026", "text": ")\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}z_{t}&=\\sigma (U_{z}h_{t-1}+b_{z})\\\\r_{t}&=\\sigma (U_{r}h_{t-1}+b_{r})\\\\\\end{aligned}}}", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0027", "text": "Type 2: each gate depends only on  the previous hidden state.", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0028", "text": "z\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03c3\n                (\n                \n                  U\n                  \n                    z\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0029", "text": "t\n                    \u2212\n                    1\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  r\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03c3\n                (\n                \n                  U\n                  \n                    r", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0030", "text": "r\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                )\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}z_{t}&=\\sigma (U_{z}h_{t-1})\\\\r_{t}&=\\sigma (U_{r}h_{t-1})\\\\\\end{aligned}}}", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0031", "text": "Type 3: each gate is computed using only the bias.", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0032", "text": "z\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03c3\n                (\n                \n                  b\n                  \n                    z\n                  \n                \n                )", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0033", "text": "r\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03c3\n                (\n                \n                  b\n                  \n                    r\n                  \n                \n                )", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0034", "text": "{\\displaystyle {\\begin{aligned}z_{t}&=\\sigma (b_{z})\\\\r_{t}&=\\sigma (b_{r})\\\\\\end{aligned}}}", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0035", "text": "Minimal gated unit\nThe minimal gated unit (MGU) is similar to the fully gated unit, except the update and reset gate vector is merged into a forget gate. This also implies that the equation for the output vector must be changed:", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0036", "text": "f\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03c3\n                (\n                \n                  W\n                  \n                    f\n                  \n                \n                \n                  x\n                  \n                    t", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0037", "text": "t\n                  \n                \n                +\n                \n                  U\n                  \n                    f\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  b\n                  \n                    f\n                  \n                \n                )", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0038", "text": ")\n              \n            \n            \n              \n                \n                  \n                    \n                      \n                        h\n                        ^\n                      \n                    \n                  \n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03d5\n                (", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0039", "text": "(\n                \n                  W\n                  \n                    h\n                  \n                \n                \n                  x\n                  \n                    t\n                  \n                \n                +\n                \n                  U\n                  \n                    h\n                  \n                \n                (\n                \n                  f\n                  \n                    t", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0040", "text": "t\n                  \n                \n                \u2299\n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                )\n                +\n                \n                  b\n                  \n                    h\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  h", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0041", "text": "h\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                (\n                1\n                \u2212\n                \n                  f\n                  \n                    t\n                  \n                \n                )\n                \u2299\n                \n                  h\n                  \n                    t\n                    \u2212", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0042", "text": "t\n                    \u2212\n                    1\n                  \n                \n                +\n                \n                  f\n                  \n                    t\n                  \n                \n                \u2299\n                \n                  \n                    \n                      \n                        h\n                        ^\n                      \n                    \n                  \n                  \n                    t", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0043", "text": "t\n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}f_{t}&=\\sigma (W_{f}x_{t}+U_{f}h_{t-1}+b_{f})\\\\{\\hat {h}}_{t}&=\\phi (W_{h}x_{t}+U_{h}(f_{t}\\odot h_{t-1})+b_{h})\\\\h_{t}&=(1-f_{t})\\odot h_{t-1}+f_{t}\\odot {\\hat {h}}_{t}\\end{aligned}}}", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0044", "text": "Variables\n\n  \n    \n      \n        \n          x\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle x_{t}}\n  \n: input vector\n\n  \n    \n      \n        \n          h\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle h_{t}}\n  \n: output vector", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0045", "text": "h\n                ^\n              \n            \n          \n          \n            t\n          \n        \n      \n    \n    {\\displaystyle {\\hat {h}}_{t}}\n  \n: candidate activation vector\n\n  \n    \n      \n        \n          f\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle f_{t}}\n  \n: forget vector", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0046", "text": "W\n      \n    \n    {\\displaystyle W}\n  \n, \n  \n    \n      \n        U\n      \n    \n    {\\displaystyle U}\n  \n and \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}\n  \n: parameter matrices and vector\n\nLight gated recurrent unit\nThe light gated recurrent unit (LiGRU) removes the reset gate altogether, replaces tanh with the ReLU activation, and applies batch normalization (BN):", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0047", "text": "z\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \u03c3\n                (\n                BN\n                \u2061\n                (\n                \n                  W\n                  \n                    z\n                  \n                \n                \n                  x", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0048", "text": "x\n                  \n                    t\n                  \n                \n                )\n                +\n                \n                  U\n                  \n                    z\n                  \n                \n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                )", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0049", "text": "h\n                        ~\n                      \n                    \n                  \n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                ReLU\n                \u2061\n                (\n                BN\n                \u2061\n                (", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0050", "text": "\u2061\n                (\n                \n                  W\n                  \n                    h\n                  \n                \n                \n                  x\n                  \n                    t\n                  \n                \n                )\n                +\n                \n                  U\n                  \n                    h\n                  \n                \n                \n                  h\n                  \n                    t", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0051", "text": "t\n                    \u2212\n                    1\n                  \n                \n                )\n              \n            \n            \n              \n                \n                  h\n                  \n                    t\n                  \n                \n              \n              \n                \n                =\n                \n                  z\n                  \n                    t", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0052", "text": "\u2299\n                \n                  h\n                  \n                    t\n                    \u2212\n                    1\n                  \n                \n                +\n                (\n                1\n                \u2212\n                \n                  z\n                  \n                    t\n                  \n                \n                )\n                \u2299", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0053", "text": "h\n                        ~\n                      \n                    \n                  \n                  \n                    t", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0054", "text": "{\\displaystyle {\\begin{aligned}z_{t}&=\\sigma (\\operatorname {BN} (W_{z}x_{t})+U_{z}h_{t-1})\\\\{\\tilde {h}}_{t}&=\\operatorname {ReLU} (\\operatorname {BN} (W_{h}x_{t})+U_{h}h_{t-1})\\\\h_{t}&=z_{t}\\odot h_{t-1}+(1-z_{t})\\odot {\\tilde {h}}_{t}\\end{aligned}}}", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Gated_recurrent_unit_0055", "text": "LiGRU has been studied from a Bayesian perspective. This analysis yielded a variant called light Bayesian recurrent unit (LiBRU), which showed slight improvements over the LiGRU on speech recognition tasks.\n\n\n== References ==", "title": "Gated recurrent unit", "url": "https://en.wikipedia.org/wiki/Gated_recurrent_unit"}, {"id": "rec_Generative_adversarial_network_0000", "text": "A generative adversarial network (GAN) is a class of machine learning frameworks and a prominent framework for approaching generative artificial intelligence. The concept was initially developed by Ian Goodfellow and his colleagues in June 2014. In a GAN, two neural networks compete with each other in the form of a zero-sum game, where one agent's gain is another agent's loss.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0001", "text": "Given a training set, this technique learns to generate new data with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers, having many realistic characteristics. Though originally proposed as a form of generative model for unsupervised learning, GANs have also proved useful for semi-supervised learning, fully supervised learning, and reinforcement learning.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0002", "text": "The core idea of a GAN is based on the \"indirect\" training through the discriminator, another neural network that can tell how \"realistic\" the input seems, which itself is also being updated dynamically. This means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator. This enables the model to learn in an unsupervised manner.\nGANs are similar to mimicry in evolutionary biology, with an evolutionary arms race between both networks.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0003", "text": "Definition\nMathematical\nThe original GAN is defined as the following game:\nEach probability space \n  \n    \n      \n        (\n        \u03a9\n        ,\n        \n          \u03bc\n          \n            ref\n          \n        \n        )\n      \n    \n    {\\displaystyle (\\Omega ,\\mu _{\\text{ref}})}\n  \n defines a GAN game.\nThere are 2 players: generator and discriminator.\nThe generator's strategy set is \n  \n    \n      \n        \n          \n            P\n          \n        \n        (\n        \u03a9\n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0004", "text": "(\n        \u03a9\n        )\n      \n    \n    {\\displaystyle {\\mathcal {P}}(\\Omega )}\n  \n, the set of all probability measures \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}}\n  \n on \n  \n    \n      \n        \u03a9\n      \n    \n    {\\displaystyle \\Omega }\n  \n.\nThe discriminator's strategy set is the set of Markov kernels \n  \n    \n      \n        \n          \u03bc\n          \n            D\n          \n        \n        :", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0005", "text": "D\n          \n        \n        :\n        \u03a9\n        \u2192\n        \n          \n            P\n          \n        \n        [\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle \\mu _{D}:\\Omega \\to {\\mathcal {P}}[0,1]}\n  \n, where \n  \n    \n      \n        \n          \n            P\n          \n        \n        [\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle {\\mathcal {P}}[0,1]}\n  \n is the set of probability measures on \n  \n    \n      \n        [", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0006", "text": "[\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle [0,1]}\n  \n.\nThe GAN game is a zero-sum game, with objective function\n  \n    \n      \n        L\n        (\n        \n          \u03bc\n          \n            G\n          \n        \n        ,\n        \n          \u03bc\n          \n            D\n          \n        \n        )\n        :=\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                ref", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0007", "text": "ref\n              \n            \n            ,\n            y\n            \u223c\n            \n              \u03bc\n              \n                D\n              \n            \n            (\n            x\n            )\n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        y\n        ]\n        +\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                G\n              \n            \n            ,", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0008", "text": ",\n            y\n            \u223c\n            \n              \u03bc\n              \n                D\n              \n            \n            (\n            x\n            )\n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        (\n        1\n        \u2212\n        y\n        )\n        ]\n        .", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0009", "text": ")\n        ]\n        .\n      \n    \n    {\\displaystyle L(\\mu _{G},\\mu _{D}):=\\operatorname {E} _{x\\sim \\mu _{\\text{ref}},y\\sim \\mu _{D}(x)}[\\ln y]+\\operatorname {E} _{x\\sim \\mu _{G},y\\sim \\mu _{D}(x)}[\\ln(1-y)].}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0010", "text": "The generator aims to minimize the objective, and the discriminator aims to maximize the objective.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0011", "text": "The generator's task is to approach \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n        \u2248\n        \n          \u03bc\n          \n            ref\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}\\approx \\mu _{\\text{ref}}}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0012", "text": ", that is, to match its own output distribution as closely as possible to the reference distribution. The discriminator's task is to output a value close to 1 when the input appears to be from the reference distribution, and to output a value close to 0 when the input looks like it came from the generator distribution.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0013", "text": "In practice", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0014", "text": "The generative network generates candidates while the discriminative network evaluates them. This creates a contest based on data distributions, where the generator learns to map from a latent space to the true data distribution, aiming to produce candidates that the discriminator cannot distinguish from real data. The discriminator's goal is to correctly identify these candidates, but as the generator improves, its task becomes more challenging, increasing the discriminator's error rate.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0015", "text": "A known dataset serves as the initial training data for the discriminator. Training involves presenting it with samples from the training dataset until it achieves acceptable accuracy. The generator is trained based on whether it succeeds in fooling the discriminator. Typically, the generator is seeded with randomized input that is sampled from a predefined latent space (e.g. a multivariate normal distribution)", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0016", "text": ". a multivariate normal distribution). Thereafter, candidates synthesized by the generator are evaluated by the discriminator. Independent backpropagation procedures are applied to both networks so that the generator produces better samples, while the discriminator becomes more skilled at flagging synthetic samples. When used for image generation, the generator is typically a deconvolutional neural network, and the discriminator is a convolutional neural network.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0017", "text": "Relation to other statistical machine learning methods\nGANs are implicit generative models, which means that they do not explicitly model the likelihood function nor provide a means for finding the latent variable corresponding to a given sample, unlike alternatives such as flow-based generative model.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0018", "text": "Compared to fully visible belief networks such as WaveNet and PixelRNN and autoregressive models in general, GANs can generate one complete sample in one pass, rather than multiple passes through the network.\nCompared to Boltzmann machines and linear ICA, there is no restriction on the type of function used by the network.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0019", "text": "Since neural networks are universal approximators, GANs are asymptotically consistent. Variational autoencoders might be universal approximators, but it is not proven as of 2017.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0020", "text": "Mathematical properties\nMeasure-theoretic considerations\nThis section provides some of the mathematical theory behind these methods.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0021", "text": "In modern probability theory based on measure theory, a probability space also needs to be equipped with a \u03c3-algebra. As a result, a more rigorous definition of the GAN game would make the following changes:Each probability space \n  \n    \n      \n        (\n        \u03a9\n        ,\n        \n          \n            B\n          \n        \n        ,\n        \n          \u03bc\n          \n            ref\n          \n        \n        )\n      \n    \n    {\\displaystyle (\\Omega ,{\\mathcal {B}},\\mu _{\\text{ref}})}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0022", "text": "defines a GAN game.\nThe generator's strategy set is \n  \n    \n      \n        \n          \n            P\n          \n        \n        (\n        \u03a9\n        ,\n        \n          \n            B\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\mathcal {P}}(\\Omega ,{\\mathcal {B}})}\n  \n, the set of all probability measures \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}}\n  \n on the measure-space", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0023", "text": "on the measure-space \n  \n    \n      \n        (\n        \u03a9\n        ,\n        \n          \n            B\n          \n        \n        )\n      \n    \n    {\\displaystyle (\\Omega ,{\\mathcal {B}})}\n  \n.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0024", "text": "The discriminator's strategy set is the set of Markov kernels \n  \n    \n      \n        \n          \u03bc\n          \n            D\n          \n        \n        :\n        (\n        \u03a9\n        ,\n        \n          \n            B\n          \n        \n        )\n        \u2192\n        \n          \n            P\n          \n        \n        (\n        [\n        0\n        ,\n        1\n        ]\n        ,\n        \n          \n            B\n          \n        \n        (\n        [\n        0\n        ,\n        1\n        ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0025", "text": "[\n        0\n        ,\n        1\n        ]\n        )\n        )\n      \n    \n    {\\displaystyle \\mu _{D}:(\\Omega ,{\\mathcal {B}})\\to {\\mathcal {P}}([0,1],{\\mathcal {B}}([0,1]))}\n  \n, where \n  \n    \n      \n        \n          \n            B\n          \n        \n        (\n        [\n        0\n        ,\n        1\n        ]\n        )\n      \n    \n    {\\displaystyle {\\mathcal {B}}([0,1])}\n  \n is the Borel \u03c3-algebra on \n  \n    \n      \n        [\n        0\n        ,\n        1\n        ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0026", "text": ",\n        1\n        ]\n      \n    \n    {\\displaystyle [0,1]}\n  \n.Since issues of measurability never arise in practice, these will not concern us further.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0027", "text": "Choice of the strategy set\nIn the most generic version of the GAN game described above, the strategy set for the discriminator contains all Markov kernels \n  \n    \n      \n        \n          \u03bc\n          \n            D\n          \n        \n        :\n        \u03a9\n        \u2192\n        \n          \n            P\n          \n        \n        [\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle \\mu _{D}:\\Omega \\to {\\mathcal {P}}[0,1]}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0028", "text": ", and the strategy set for the generator contains arbitrary probability distributions \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}}\n  \n on \n  \n    \n      \n        \u03a9\n      \n    \n    {\\displaystyle \\Omega }\n  \n.\nHowever, as shown below, the optimal discriminator strategy against any \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0029", "text": "{\\displaystyle \\mu _{G}}\n  \n is deterministic, so there is no loss of generality in restricting the discriminator's strategies to deterministic functions \n  \n    \n      \n        D\n        :\n        \u03a9\n        \u2192\n        [\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle D:\\Omega \\to [0,1]}\n  \n. In most applications, \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n is a deep neural network function.\nAs for the generator, while", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0030", "text": "As for the generator, while \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}}\n  \n could theoretically be any computable probability distribution, in practice, it is usually implemented as a pushforward: \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n        =\n        \n          \u03bc\n          \n            Z\n          \n        \n        \u2218\n        \n          G\n          \n            \u2212", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0031", "text": "G\n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}=\\mu _{Z}\\circ G^{-1}}\n  \n. That is, start with a random variable \n  \n    \n      \n        z\n        \u223c\n        \n          \u03bc\n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle z\\sim \\mu _{Z}}\n  \n, where \n  \n    \n      \n        \n          \u03bc\n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle \\mu _{Z}}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0032", "text": "{\\displaystyle \\mu _{Z}}\n  \n is a probability distribution that is easy to compute (such as the uniform distribution, or the Gaussian distribution), then define a function \n  \n    \n      \n        G\n        :\n        \n          \u03a9\n          \n            Z\n          \n        \n        \u2192\n        \u03a9\n      \n    \n    {\\displaystyle G:\\Omega _{Z}\\to \\Omega }\n  \n. Then the distribution \n  \n    \n      \n        \n          \u03bc\n          \n            G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0033", "text": "G\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}}\n  \n is the distribution of \n  \n    \n      \n        G\n        (\n        z\n        )\n      \n    \n    {\\displaystyle G(z)}\n  \n.\nConsequently, the generator's strategy is usually defined as just \n  \n    \n      \n        G\n      \n    \n    {\\displaystyle G}\n  \n, leaving \n  \n    \n      \n        z\n        \u223c\n        \n          \u03bc\n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle z\\sim \\mu _{Z}}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0034", "text": "{\\displaystyle z\\sim \\mu _{Z}}\n  \n implicit. In this formalism, the GAN game objective is\n  \n    \n      \n        L\n        (\n        G\n        ,\n        D\n        )\n        :=\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                ref\n              \n            \n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        D\n        (\n        x\n        )\n        ]\n        +\n        \n          E", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0035", "text": "+\n        \n          E\n          \n            z\n            \u223c\n            \n              \u03bc\n              \n                Z\n              \n            \n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        (\n        1\n        \u2212\n        D\n        (\n        G\n        (\n        z\n        )\n        )\n        )\n        ]\n        .\n      \n    \n    {\\displaystyle L(G,D):=\\operatorname {E} _{x\\sim \\mu _{\\text{ref}}}[\\ln D(x)]+\\operatorname {E} _{z\\sim \\mu _{Z}}[\\ln(1-D(G(z)))].}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0036", "text": "Generative reparametrization\nThe GAN architecture has two main components. One is casting optimization into a game, of form \n  \n    \n      \n        \n          min\n          \n            G\n          \n        \n        \n          max\n          \n            D\n          \n        \n        L\n        (\n        G\n        ,\n        D\n        )\n      \n    \n    {\\displaystyle \\min _{G}\\max _{D}L(G,D)}\n  \n, which is different from the usual kind of optimization, of form", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0037", "text": "min\n          \n            \u03b8\n          \n        \n        L\n        (\n        \u03b8\n        )\n      \n    \n    {\\displaystyle \\min _{\\theta }L(\\theta )}\n  \n. The other is the decomposition of \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}}\n  \n into \n  \n    \n      \n        \n          \u03bc\n          \n            Z\n          \n        \n        \u2218\n        \n          G\n          \n            \u2212", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0038", "text": "G\n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle \\mu _{Z}\\circ G^{-1}}\n  \n, which can be understood as a reparametrization trick.\nTo see its significance, one must compare GAN with previous methods for learning generative models, which were plagued with \"intractable probabilistic computations that arise in maximum likelihood estimation and related strategies\".", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0039", "text": "At the same time, Kingma and Welling and Rezende et al. developed the same idea of reparametrization into a general stochastic backpropagation method. Among its first applications was the variational autoencoder.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0040", "text": "Move order and strategic equilibria\nIn the original paper, as well as most subsequent papers, it is usually assumed that the generator moves first, and the discriminator moves second, thus giving the following minimax game:\n  \n    \n      \n        \n          min\n          \n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        \n          max\n          \n            \n              \u03bc\n              \n                D", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0041", "text": "D\n              \n            \n          \n        \n        L\n        (\n        \n          \u03bc\n          \n            G\n          \n        \n        ,\n        \n          \u03bc\n          \n            D\n          \n        \n        )\n        :=\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                ref\n              \n            \n            ,\n            y\n            \u223c\n            \n              \u03bc", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0042", "text": "\u223c\n            \n              \u03bc\n              \n                D\n              \n            \n            (\n            x\n            )\n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        y\n        ]\n        +\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                G\n              \n            \n            ,\n            y\n            \u223c\n            \n              \u03bc\n              \n                D", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0043", "text": "\u03bc\n              \n                D\n              \n            \n            (\n            x\n            )\n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        (\n        1\n        \u2212\n        y\n        )\n        ]\n        .\n      \n    \n    {\\displaystyle \\min _{\\mu _{G}}\\max _{\\mu _{D}}L(\\mu _{G},\\mu _{D}):=\\operatorname {E} _{x\\sim \\mu _{\\text{ref}},y\\sim \\mu _{D}(x)}[\\ln y]+\\operatorname {E} _{x\\sim \\mu _{G},y\\sim \\mu _{D}(x)}[\\ln(1-y)].}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0044", "text": "If both the generator's and the discriminator's strategy sets are spanned by a finite number of strategies, then by the minimax theorem,\n  \n    \n      \n        \n          min\n          \n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        \n          max\n          \n            \n              \u03bc\n              \n                D\n              \n            \n          \n        \n        L\n        (\n        \n          \u03bc", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0045", "text": "(\n        \n          \u03bc\n          \n            G\n          \n        \n        ,\n        \n          \u03bc\n          \n            D\n          \n        \n        )\n        =\n        \n          max\n          \n            \n              \u03bc\n              \n                D\n              \n            \n          \n        \n        \n          min\n          \n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        L\n        (", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0046", "text": "L\n        (\n        \n          \u03bc\n          \n            G\n          \n        \n        ,\n        \n          \u03bc\n          \n            D\n          \n        \n        )\n      \n    \n    {\\displaystyle \\min _{\\mu _{G}}\\max _{\\mu _{D}}L(\\mu _{G},\\mu _{D})=\\max _{\\mu _{D}}\\min _{\\mu _{G}}L(\\mu _{G},\\mu _{D})}\n  \nthat is, the move order does not matter.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0047", "text": "that is, the move order does not matter.\nHowever, since the strategy sets are both not finitely spanned, the minimax theorem does not apply, and the idea of an \"equilibrium\" becomes delicate. To wit, there are the following different concepts of equilibrium:", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0048", "text": "Equilibrium when generator moves first, and discriminator moves second:\n  \n    \n      \n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            G\n          \n        \n        \u2208\n        arg\n        \u2061\n        \n          min\n          \n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        \n          max\n          \n            \n              \u03bc", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0049", "text": "\u03bc\n              \n                D\n              \n            \n          \n        \n        L\n        (\n        \n          \u03bc\n          \n            G\n          \n        \n        ,\n        \n          \u03bc\n          \n            D\n          \n        \n        )\n        ,\n        \n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            D\n          \n        \n        \u2208", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0050", "text": "D\n          \n        \n        \u2208\n        arg\n        \u2061\n        \n          max\n          \n            \n              \u03bc\n              \n                D\n              \n            \n          \n        \n        L\n        (\n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            G\n          \n        \n        ,\n        \n          \u03bc\n          \n            D\n          \n        \n        )\n        ,", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0051", "text": ")\n        ,\n        \n      \n    \n    {\\displaystyle {\\hat {\\mu }}_{G}\\in \\arg \\min _{\\mu _{G}}\\max _{\\mu _{D}}L(\\mu _{G},\\mu _{D}),\\quad {\\hat {\\mu }}_{D}\\in \\arg \\max _{\\mu _{D}}L({\\hat {\\mu }}_{G},\\mu _{D}),\\quad }", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0052", "text": "Equilibrium when discriminator moves first, and generator moves second:\n  \n    \n      \n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            D\n          \n        \n        \u2208\n        arg\n        \u2061\n        \n          max\n          \n            \n              \u03bc\n              \n                D\n              \n            \n          \n        \n        \n          min\n          \n            \n              \u03bc", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0053", "text": "\u03bc\n              \n                G\n              \n            \n          \n        \n        L\n        (\n        \n          \u03bc\n          \n            G\n          \n        \n        ,\n        \n          \u03bc\n          \n            D\n          \n        \n        )\n        ,\n        \n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            G\n          \n        \n        \u2208", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0054", "text": "G\n          \n        \n        \u2208\n        arg\n        \u2061\n        \n          min\n          \n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        L\n        (\n        \n          \u03bc\n          \n            G\n          \n        \n        ,\n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            D\n          \n        \n        )\n        ,", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0055", "text": ")\n        ,\n      \n    \n    {\\displaystyle {\\hat {\\mu }}_{D}\\in \\arg \\max _{\\mu _{D}}\\min _{\\mu _{G}}L(\\mu _{G},\\mu _{D}),\\quad {\\hat {\\mu }}_{G}\\in \\arg \\min _{\\mu _{G}}L(\\mu _{G},{\\hat {\\mu }}_{D}),}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0056", "text": "Nash equilibrium \n  \n    \n      \n        (\n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            D\n          \n        \n        ,\n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            G\n          \n        \n        )\n      \n    \n    {\\displaystyle ({\\hat {\\mu }}_{D},{\\hat {\\mu }}_{G})}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0057", "text": ", which is stable under simultaneous move order:\n  \n    \n      \n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            D\n          \n        \n        \u2208\n        arg\n        \u2061\n        \n          max\n          \n            \n              \u03bc\n              \n                D\n              \n            \n          \n        \n        L\n        (\n        \n          \n            \n              \n                \u03bc", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0058", "text": "\u03bc\n                ^\n              \n            \n          \n          \n            G\n          \n        \n        ,\n        \n          \u03bc\n          \n            D\n          \n        \n        )\n        ,\n        \n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            G\n          \n        \n        \u2208\n        arg\n        \u2061\n        \n          min", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0059", "text": "min\n          \n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        L\n        (\n        \n          \u03bc\n          \n            G\n          \n        \n        ,\n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            D\n          \n        \n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0060", "text": ")\n      \n    \n    {\\displaystyle {\\hat {\\mu }}_{D}\\in \\arg \\max _{\\mu _{D}}L({\\hat {\\mu }}_{G},\\mu _{D}),\\quad {\\hat {\\mu }}_{G}\\in \\arg \\min _{\\mu _{G}}L(\\mu _{G},{\\hat {\\mu }}_{D})}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0061", "text": "For general games, these equilibria do not have to agree, or even to exist. For the original GAN game, these equilibria all exist, and are all equal. However, for more general GAN games, these do not necessarily exist, or agree.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0062", "text": "Main theorems for GAN game\nThe original GAN paper proved the following two theorems:\nInterpretation: For any fixed generator strategy \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}}\n  \n, the optimal discriminator keeps track of the likelihood ratio between the reference distribution and the generator distribution:\n  \n    \n      \n        \n          \n            \n              D\n              (\n              x", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0063", "text": "D\n              (\n              x\n              )\n            \n            \n              1\n              \u2212\n              D\n              (\n              x\n              )\n            \n          \n        \n        =\n        \n          \n            \n              d\n              \n                \u03bc\n                \n                  ref\n                \n              \n            \n            \n              d\n              \n                \u03bc\n                \n                  G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0064", "text": "G\n                \n              \n            \n          \n        \n        (\n        x\n        )\n        =\n        \n          \n            \n              \n                \u03bc\n                \n                  ref\n                \n              \n              (\n              d\n              x\n              )\n            \n            \n              \n                \u03bc\n                \n                  G\n                \n              \n              (", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0065", "text": "(\n              d\n              x\n              )\n            \n          \n        \n        ;\n        \n        D\n        (\n        x\n        )\n        =\n        \u03c3\n        (\n        ln\n        \u2061\n        \n          \u03bc\n          \n            ref\n          \n        \n        (\n        d\n        x\n        )\n        \u2212\n        ln\n        \u2061\n        \n          \u03bc\n          \n            G\n          \n        \n        (\n        d\n        x\n        )\n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0066", "text": "(\n        d\n        x\n        )\n        )\n      \n    \n    {\\displaystyle {\\frac {D(x)}{1-D(x)}}={\\frac {d\\mu _{\\text{ref}}}{d\\mu _{G}}}(x)={\\frac {\\mu _{\\text{ref}}(dx)}{\\mu _{G}(dx)}};\\quad D(x)=\\sigma (\\ln \\mu _{\\text{ref}}(dx)-\\ln \\mu _{G}(dx))}\n  \nwhere \n  \n    \n      \n        \u03c3\n      \n    \n    {\\displaystyle \\sigma }\n  \n is the logistic function.\nIn particular, if the prior probability for an image \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0067", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n to come from the reference distribution is equal to \n  \n    \n      \n        \n          \n            1\n            2\n          \n        \n      \n    \n    {\\displaystyle {\\frac {1}{2}}}\n  \n, then \n  \n    \n      \n        D\n        (\n        x\n        )\n      \n    \n    {\\displaystyle D(x)}\n  \n is just the posterior probability that \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n came from the reference distribution:", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0068", "text": "D\n        (\n        x\n        )\n        =\n        Pr\n        (\n        x\n        \n           came from reference distribution\n        \n        \u2223\n        x\n        )\n        .\n      \n    \n    {\\displaystyle D(x)=\\Pr(x{\\text{ came from reference distribution}}\\mid x).}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0069", "text": "Training and evaluating GAN\nTraining\nUnstable convergence\nWhile the GAN game has a unique global equilibrium point when both the generator and discriminator have access to their entire strategy sets, the equilibrium is no longer guaranteed when they have a restricted strategy set.\nIn practice, the generator has access only to measures of form \n  \n    \n      \n        \n          \u03bc\n          \n            Z\n          \n        \n        \u2218\n        \n          G\n          \n            \u03b8", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0070", "text": "G\n          \n            \u03b8\n          \n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle \\mu _{Z}\\circ G_{\\theta }^{-1}}\n  \n, where \n  \n    \n      \n        \n          G\n          \n            \u03b8\n          \n        \n      \n    \n    {\\displaystyle G_{\\theta }}\n  \n is a function computed by a neural network with parameters \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n, and \n  \n    \n      \n        \n          \u03bc", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0071", "text": "\u03bc\n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle \\mu _{Z}}\n  \n is an easily sampled distribution, such as the uniform or normal distribution. Similarly, the discriminator has access only to functions of form \n  \n    \n      \n        \n          D\n          \n            \u03b6\n          \n        \n      \n    \n    {\\displaystyle D_{\\zeta }}\n  \n, a function computed by a neural network with parameters \n  \n    \n      \n        \u03b6", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0072", "text": "\u03b6\n      \n    \n    {\\displaystyle \\zeta }\n  \n. These restricted strategy sets take up a vanishingly small proportion of their entire strategy sets.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0073", "text": "Further, even if an equilibrium still exists, it can only be found by searching in the high-dimensional space of all possible neural network functions. The standard strategy of using gradient descent to find the equilibrium often does not work for GAN, and often the game \"collapses\" into one of several failure modes", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0074", "text": ". To improve the convergence stability, some training strategies start with an easier task, such as generating low-resolution images or simple images (one object with uniform background), and gradually increase the difficulty of the task during training. This essentially translates to applying a curriculum learning scheme.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0075", "text": "Mode collapse\nGANs often suffer from mode collapse where they fail to generalize properly, missing entire modes from the input data. For example, a GAN trained on the MNIST dataset containing many samples of each digit might only generate pictures of digit 0. This was termed \"the Helvetica scenario\".", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0076", "text": "A typical mechanism for mode collapse is the generator only generating one or a few of the likely values, or a very incomplete picture of the target distribution. As the discriminator is only trained to distinguish real from fake samples, it will correctly identify the generated samples as real, but no penalty is imposed on the GAN's ability to generate data that represents the full range of the target distribution.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0077", "text": "Weak discriminators, for instance underparametrized ones, or ones trained too slow compared to the generator, may as well be unable to fully discriminate over the entire support of the distribution, and only become able to correctly discriminate a very incomplete part of the target distribution.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0078", "text": "Some researchers perceive the root problem to be a weak discriminative network that fails to notice the pattern of omission, while others assign blame to a bad choice of objective function. Many solutions have been proposed, but it is still an open problem.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0079", "text": "Even the state-of-the-art architecture, BigGAN (2019), could not avoid mode collapse. The authors resorted to \"allowing collapse to occur at the later stages of training, by which time a model is sufficiently trained to achieve good results\".", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0080", "text": "Two time-scale update rule\nThe two time-scale update rule (TTUR) is proposed to make GAN convergence more stable by making the learning rate of the generator lower than that of the discriminator. They prove that when trained this way, GANs \"converge, under mild assumptions to a stationary local Nash equilibrium\". They further show that this property extends to the use of the Adam optimizer, which is commonly used in stochastic gradient descent.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0081", "text": "It is important to note, however, that a local Nash equilibrium in no way signifies an absence of mode collapse - for instance, a GAN trained on MNIST collapsed to generating a single digit may satisfy the hypotheses of the paper, while still presenting mode collapse.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0082", "text": "Vanishing gradient\nConversely, if the discriminator learns too fast compared to the generator, then the discriminator could almost perfectly distinguish \n  \n    \n      \n        \n          \u03bc\n          \n            \n              G\n              \n                \u03b8\n              \n            \n          \n        \n        ,\n        \n          \u03bc\n          \n            ref\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G_{\\theta }},\\mu _{\\text{ref}}}\n  \n. In such case, the generator", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0083", "text": ". In such case, the generator \n  \n    \n      \n        \n          G\n          \n            \u03b8\n          \n        \n      \n    \n    {\\displaystyle G_{\\theta }}\n  \n could be stuck with a very high loss no matter which direction it changes its \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n, meaning that the gradient \n  \n    \n      \n        \n          \u2207\n          \n            \u03b8\n          \n        \n        L\n        (\n        \n          G\n          \n            \u03b8", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0084", "text": "G\n          \n            \u03b8\n          \n        \n        ,\n        \n          D\n          \n            \u03b6\n          \n        \n        )\n      \n    \n    {\\displaystyle \\nabla _{\\theta }L(G_{\\theta },D_{\\zeta })}\n  \n would be close to zero. In such case, the generator cannot learn, a case of the vanishing gradient problem.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0085", "text": "Intuitively speaking, the discriminator is too good, and since the generator cannot take any small step (only small steps are considered in gradient descent) to improve its payoff, it does not even try.\nOne important method for solving this problem is the Wasserstein GAN.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0086", "text": "Evaluation", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0087", "text": "GANs are usually evaluated by Inception score (IS), which measures how varied the generator's outputs are (as classified by an image classifier, usually Inception-v3), or Fr\u00e9chet inception distance (FID), which measures how similar the generator's outputs are to a reference set (as classified by a learned image featurizer, such as Inception-v3 without its final layer)", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0088", "text": ". Many papers that propose new GAN architectures for image generation report how their architectures break the state of the art on FID or IS.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0089", "text": "Another evaluation method is the Learned Perceptual Image Patch Similarity (LPIPS), which starts with a learned image featurizer \n  \n    \n      \n        \n          f\n          \n            \u03b8\n          \n        \n        :\n        \n          Image\n        \n        \u2192\n        \n          \n            R\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle f_{\\theta }:{\\text{Image}}\\to \\mathbb {R} ^{n}}\n  \n, and finetunes it by supervised learning on a set of", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0090", "text": "(\n        x\n        ,\n        \n          x\n          \u2032\n        \n        ,\n        \n          p\n          e\n          r\n          c\n          e\n          p\n          t\n          u\n          a\n          l\n           \n          d\n          i\n          f\n          f\n          e\n          r\n          e\n          n\n          c\n          e\n        \n        \u2061\n        (\n        x\n        ,\n        \n          x\n          \u2032\n        \n        )\n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0091", "text": ")\n        )\n      \n    \n    {\\displaystyle (x,x',\\operatorname {perceptual~difference} (x,x'))}\n  \n, where \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n is an image, \n  \n    \n      \n        \n          x\n          \u2032\n        \n      \n    \n    {\\displaystyle x'}\n  \n is a perturbed version of it, and \n  \n    \n      \n        \n          p\n          e\n          r\n          c\n          e\n          p\n          t\n          u\n          a\n          l", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0092", "text": "u\n          a\n          l\n           \n          d\n          i\n          f\n          f\n          e\n          r\n          e\n          n\n          c\n          e\n        \n        \u2061\n        (\n        x\n        ,\n        \n          x\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle \\operatorname {perceptual~difference} (x,x')}\n  \n is how much they differ, as reported by human subjects. The model is finetuned so that it can approximate \n  \n    \n      \n        \u2016", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0093", "text": "\u2016\n        \n          f\n          \n            \u03b8\n          \n        \n        (\n        x\n        )\n        \u2212\n        \n          f\n          \n            \u03b8\n          \n        \n        (\n        \n          x\n          \u2032\n        \n        )\n        \u2016\n        \u2248\n        \n          p\n          e\n          r\n          c\n          e\n          p\n          t\n          u\n          a\n          l\n           \n          d\n          i\n          f\n          f\n          e\n          r", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0094", "text": "f\n          f\n          e\n          r\n          e\n          n\n          c\n          e\n        \n        \u2061\n        (\n        x\n        ,\n        \n          x\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle \\|f_{\\theta }(x)-f_{\\theta }(x')\\|\\approx \\operatorname {perceptual~difference} (x,x')}\n  \n. This finetuned model is then used to define \n  \n    \n      \n        LPIPS\n        \u2061\n        (\n        x\n        ,\n        \n          x\n          \u2032\n        \n        )\n        :=", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0095", "text": "\u2032\n        \n        )\n        :=\n        \u2016\n        \n          f\n          \n            \u03b8\n          \n        \n        (\n        x\n        )\n        \u2212\n        \n          f\n          \n            \u03b8\n          \n        \n        (\n        \n          x\n          \u2032\n        \n        )\n        \u2016\n      \n    \n    {\\displaystyle \\operatorname {LPIPS} (x,x'):=\\|f_{\\theta }(x)-f_{\\theta }(x')\\|}\n  \n.\nOther evaluation methods are reviewed in.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0096", "text": "Variants\nThere is a veritable zoo of GAN variants. Some of the most prominent are as follows:", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0097", "text": "Conditional GAN\nConditional GANs are similar to standard GANs except they allow the model to conditionally generate samples based on additional information. For example, if we want to generate a cat face given a dog picture, we could use a conditional GAN.\nThe generator in a GAN game generates \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n      \n    \n    {\\displaystyle \\mu _{G}}\n  \n, a probability distribution on the probability space", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0098", "text": "\u03a9\n      \n    \n    {\\displaystyle \\Omega }\n  \n. This leads to the idea of a conditional GAN, where instead of generating one probability distribution on \n  \n    \n      \n        \u03a9\n      \n    \n    {\\displaystyle \\Omega }\n  \n, the generator generates a different probability distribution \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n        (\n        c\n        )\n      \n    \n    {\\displaystyle \\mu _{G}(c)}\n  \n on \n  \n    \n      \n        \u03a9", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0099", "text": "on \n  \n    \n      \n        \u03a9\n      \n    \n    {\\displaystyle \\Omega }\n  \n, for each given class label \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n.\nFor example, for generating images that look like ImageNet, the generator should be able to generate a picture of cat when given the class label \"cat\".\nIn the original paper, the authors noted that GAN can be trivially extended to conditional GAN by providing the labels to both the generator and the discriminator.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0100", "text": "Concretely, the conditional GAN game is just the GAN game with class labels provided:\n  \n    \n      \n        L\n        (\n        \n          \u03bc\n          \n            G\n          \n        \n        ,\n        D\n        )\n        :=\n        \n          E\n          \n            c\n            \u223c\n            \n              \u03bc\n              \n                C\n              \n            \n            ,\n            x\n            \u223c\n            \n              \u03bc\n              \n                ref", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0101", "text": "ref\n              \n            \n            (\n            c\n            )\n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        D\n        (\n        x\n        ,\n        c\n        )\n        ]\n        +\n        \n          E\n          \n            c\n            \u223c\n            \n              \u03bc\n              \n                C\n              \n            \n            ,\n            x\n            \u223c\n            \n              \u03bc\n              \n                G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0102", "text": "\u03bc\n              \n                G\n              \n            \n            (\n            c\n            )\n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        (\n        1\n        \u2212\n        D\n        (\n        x\n        ,\n        c\n        )\n        )\n        ]\n      \n    \n    {\\displaystyle L(\\mu _{G},D):=\\operatorname {E} _{c\\sim \\mu _{C},x\\sim \\mu _{\\text{ref}}(c)}[\\ln D(x,c)]+\\operatorname {E} _{c\\sim \\mu _{C},x\\sim \\mu _{G}(c)}[\\ln(1-D(x,c))]}\n  \nwhere", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0103", "text": "where \n  \n    \n      \n        \n          \u03bc\n          \n            C\n          \n        \n      \n    \n    {\\displaystyle \\mu _{C}}\n  \n is a probability distribution over classes, \n  \n    \n      \n        \n          \u03bc\n          \n            ref\n          \n        \n        (\n        c\n        )\n      \n    \n    {\\displaystyle \\mu _{\\text{ref}}(c)}\n  \n is the probability distribution of real images of class \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n, and", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0104", "text": "{\\displaystyle c}\n  \n, and \n  \n    \n      \n        \n          \u03bc\n          \n            G\n          \n        \n        (\n        c\n        )\n      \n    \n    {\\displaystyle \\mu _{G}(c)}\n  \n the probability distribution of images generated by the generator when given class label \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n.\nIn 2017, a conditional GAN learned to generate 1000 image classes of ImageNet.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0105", "text": "GANs with alternative architectures\nThe GAN game is a general framework and can be run with any reasonable parametrization of the generator \n  \n    \n      \n        G\n      \n    \n    {\\displaystyle G}\n  \n and discriminator \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n. In the original paper, the authors demonstrated it using multilayer perceptron networks and convolutional neural networks. Many alternative architectures have been tried.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0106", "text": "Deep convolutional GAN (DCGAN): For both generator and discriminator, uses only deep networks consisting entirely of convolution-deconvolution layers, that is, fully convolutional networks.\nSelf-attention GAN (SAGAN): Starts with the DCGAN, then adds residually-connected standard self-attention modules to the generator and discriminator.\nVariational autoencoder GAN (VAEGAN): Uses a variational autoencoder (VAE) for the generator.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0107", "text": "Transformer GAN (TransGAN): Uses the pure transformer architecture for both the generator and discriminator, entirely devoid of convolution-deconvolution layers.\nFlow-GAN: Uses flow-based generative model for the generator, allowing efficient computation of the likelihood function.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0108", "text": "GANs with alternative objectives\nMany GAN variants are merely obtained by changing the loss functions for the generator and discriminator.\nOriginal GAN:\nWe recast the original GAN objective into a form more convenient for comparison:\n  \n    \n      \n        \n          \n            {\n            \n              \n                \n                  \n                    min\n                    \n                      D\n                    \n                  \n                  \n                    L", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0109", "text": "L\n                    \n                      D\n                    \n                  \n                  (\n                  D\n                  ,\n                  \n                    \u03bc\n                    \n                      G\n                    \n                  \n                  )\n                  =\n                  \u2212\n                  \n                    E\n                    \n                      x\n                      \u223c", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0110", "text": "x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          G\n                        \n                      \n                    \n                  \n                  \u2061\n                  [\n                  ln\n                  \u2061\n                  D\n                  (\n                  x\n                  )\n                  ]\n                  \u2212\n                  \n                    E", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0111", "text": "E\n                    \n                      x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          ref\n                        \n                      \n                    \n                  \n                  \u2061\n                  [\n                  ln\n                  \u2061\n                  (\n                  1\n                  \u2212\n                  D\n                  (", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0112", "text": "D\n                  (\n                  x\n                  )\n                  )\n                  ]\n                \n              \n              \n                \n                  \n                    min\n                    \n                      G\n                    \n                  \n                  \n                    L\n                    \n                      G\n                    \n                  \n                  (\n                  D\n                  ,", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0113", "text": "D\n                  ,\n                  \n                    \u03bc\n                    \n                      G\n                    \n                  \n                  )\n                  =\n                  \u2212\n                  \n                    E\n                    \n                      x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0114", "text": "\u2061\n                  [\n                  ln\n                  \u2061\n                  (\n                  1\n                  \u2212\n                  D\n                  (\n                  x\n                  )\n                  )\n                  ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0115", "text": "{\\displaystyle {\\begin{cases}\\min _{D}L_{D}(D,\\mu _{G})=-\\operatorname {E} _{x\\sim \\mu _{G}}[\\ln D(x)]-\\operatorname {E} _{x\\sim \\mu _{\\text{ref}}}[\\ln(1-D(x))]\\\\\\min _{G}L_{G}(D,\\mu _{G})=-\\operatorname {E} _{x\\sim \\mu _{G}}[\\ln(1-D(x))]\\end{cases}}}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0116", "text": "Original GAN, non-saturating loss:\nThis objective for generator was recommended in the original paper for faster convergence.\n  \n    \n      \n        \n          L\n          \n            G\n          \n        \n        =\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        D\n        (\n        x\n        )\n        ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0117", "text": "x\n        )\n        ]\n      \n    \n    {\\displaystyle L_{G}=\\operatorname {E} _{x\\sim \\mu _{G}}[\\ln D(x)]}\n  \nThe effect of using this objective is analyzed in Section 2.2.2 of Arjovsky et al.\nOriginal GAN, maximum likelihood:", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0118", "text": "L\n          \n            G\n          \n        \n        =\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        \u2061\n        [\n        (\n        \n          exp\n        \n        \u2218\n        \n          \u03c3\n          \n            \u2212\n            1\n          \n        \n        \u2218\n        D\n        )\n        (\n        x\n        )\n        ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0119", "text": "x\n        )\n        ]\n      \n    \n    {\\displaystyle L_{G}=\\operatorname {E} _{x\\sim \\mu _{G}}[({\\exp }\\circ \\sigma ^{-1}\\circ D)(x)]}\n  \nwhere \n  \n    \n      \n        \u03c3\n      \n    \n    {\\displaystyle \\sigma }\n  \n is the logistic function. When the discriminator is optimal, the generator gradient is the same as in maximum likelihood estimation, even though GAN cannot perform maximum likelihood estimation itself.\nHinge loss GAN:\n  \n    \n      \n        \n          L", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0120", "text": "L\n          \n            D\n          \n        \n        =\n        \u2212\n        \n          E\n          \n            x\n            \u223c\n            \n              p\n              \n                ref\n              \n            \n          \n        \n        \u2061\n        \n          [\n          \n            min\n            \n              (\n              \n                0\n                ,\n                \u2212\n                1\n                +\n                D", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0121", "text": "+\n                D\n                (\n                x\n                )\n              \n              )\n            \n          \n          ]\n        \n        \u2212\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        \u2061\n        \n          [\n          \n            min\n            \n              (\n              \n                0\n                ,", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0122", "text": "0\n                ,\n                \u2212\n                1\n                \u2212\n                D\n                \n                  (\n                  x\n                  )\n                \n              \n              )\n            \n          \n          ]\n        \n      \n    \n    {\\displaystyle L_{D}=-\\operatorname {E} _{x\\sim p_{\\text{ref}}}\\left[\\min \\left(0,-1+D(x)\\right)\\right]-\\operatorname {E} _{x\\sim \\mu _{G}}\\left[\\min \\left(0,-1-D\\left(x\\right)\\right)\\right]}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0123", "text": "L\n          \n            G\n          \n        \n        =\n        \u2212\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        \u2061\n        [\n        D\n        (\n        x\n        )\n        ]\n      \n    \n    {\\displaystyle L_{G}=-\\operatorname {E} _{x\\sim \\mu _{G}}[D(x)]}\n  \nLeast squares GAN:\n  \n    \n      \n        \n          L", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0124", "text": "L\n          \n            D\n          \n        \n        =\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                ref\n              \n            \n          \n        \n        \u2061\n        [\n        (\n        D\n        (\n        x\n        )\n        \u2212\n        b\n        \n          )\n          \n            2\n          \n        \n        ]\n        +\n        \n          E\n          \n            x\n            \u223c", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0125", "text": "x\n            \u223c\n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        \u2061\n        [\n        (\n        D\n        (\n        x\n        )\n        \u2212\n        a\n        \n          )\n          \n            2\n          \n        \n        ]\n      \n    \n    {\\displaystyle L_{D}=\\operatorname {E} _{x\\sim \\mu _{\\text{ref}}}[(D(x)-b)^{2}]+\\operatorname {E} _{x\\sim \\mu _{G}}[(D(x)-a)^{2}]}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0126", "text": "L\n          \n            G\n          \n        \n        =\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                G\n              \n            \n          \n        \n        \u2061\n        [\n        (\n        D\n        (\n        x\n        )\n        \u2212\n        c\n        \n          )\n          \n            2\n          \n        \n        ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0127", "text": "]\n      \n    \n    {\\displaystyle L_{G}=\\operatorname {E} _{x\\sim \\mu _{G}}[(D(x)-c)^{2}]}\n  \nwhere \n  \n    \n      \n        a\n        ,\n        b\n        ,\n        c\n      \n    \n    {\\displaystyle a,b,c}\n  \n are parameters to be chosen. The authors recommended \n  \n    \n      \n        a\n        =\n        \u2212\n        1\n        ,\n        b\n        =\n        1\n        ,\n        c\n        =\n        0\n      \n    \n    {\\displaystyle a=-1,b=1,c=0}\n  \n.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0128", "text": "Wasserstein GAN (WGAN)\nThe Wasserstein GAN modifies the GAN game at two points:", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0129", "text": "The discriminator's strategy set is the set of measurable functions of type \n  \n    \n      \n        D\n        :\n        \u03a9\n        \u2192\n        \n          R\n        \n      \n    \n    {\\displaystyle D:\\Omega \\to \\mathbb {R} }\n  \n with bounded Lipschitz norm: \n  \n    \n      \n        \u2016\n        D\n        \n          \u2016\n          \n            L\n          \n        \n        \u2264\n        K\n      \n    \n    {\\displaystyle \\|D\\|_{L}\\leq K}\n  \n, where \n  \n    \n      \n        K\n      \n    \n    {\\displaystyle K}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0130", "text": "K\n      \n    \n    {\\displaystyle K}\n  \n is a fixed positive constant.\nThe objective is\n  \n    \n      \n        \n          L\n          \n            W\n            G\n            A\n            N\n          \n        \n        (\n        \n          \u03bc\n          \n            G\n          \n        \n        ,\n        D\n        )\n        :=\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0131", "text": "\u2061\n        [\n        D\n        (\n        x\n        )\n        ]\n        \u2212\n        \n          \n            E\n          \n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                ref\n              \n            \n          \n        \n        [\n        D\n        (\n        x\n        )\n        ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0132", "text": "x\n        )\n        ]\n      \n    \n    {\\displaystyle L_{WGAN}(\\mu _{G},D):=\\operatorname {E} _{x\\sim \\mu _{G}}[D(x)]-\\mathbb {E} _{x\\sim \\mu _{\\text{ref}}}[D(x)]}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0133", "text": "One of its purposes is to solve the problem of mode collapse (see above). The authors claim \"In no experiment did we see evidence of mode collapse for the WGAN algorithm\".\n\nGANs with more than two players\nAdversarial autoencoder\nAn adversarial autoencoder (AAE) is more autoencoder than GAN. The idea is to start with a plain autoencoder, but train a discriminator to discriminate the latent vectors from a reference distribution (often the normal distribution).", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0134", "text": "InfoGAN\nIn conditional GAN, the generator receives both a noise vector \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n and a label \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n, and produces an image \n  \n    \n      \n        G\n        (\n        z\n        ,\n        c\n        )\n      \n    \n    {\\displaystyle G(z,c)}\n  \n. The discriminator receives image-label pairs \n  \n    \n      \n        (\n        x\n        ,\n        c\n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0135", "text": ",\n        c\n        )\n      \n    \n    {\\displaystyle (x,c)}\n  \n, and computes \n  \n    \n      \n        D\n        (\n        x\n        ,\n        c\n        )\n      \n    \n    {\\displaystyle D(x,c)}\n  \n.\nWhen the training dataset is unlabeled, conditional GAN does not work directly.\nThe idea of InfoGAN is to decree that every latent vector in the latent space can be decomposed as \n  \n    \n      \n        (\n        z\n        ,\n        c\n        )\n      \n    \n    {\\displaystyle (z,c)}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0136", "text": "{\\displaystyle (z,c)}\n  \n: an incompressible noise part \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n, and an informative label part \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n, and encourage the generator to comply with the decree, by encouraging it to maximize \n  \n    \n      \n        I\n        (\n        c\n        ,\n        G\n        (\n        z\n        ,\n        c\n        )\n        )\n      \n    \n    {\\displaystyle I(c,G(z,c))}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0137", "text": "{\\displaystyle I(c,G(z,c))}\n  \n, the mutual information between \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n and \n  \n    \n      \n        G\n        (\n        z\n        ,\n        c\n        )\n      \n    \n    {\\displaystyle G(z,c)}\n  \n, while making no demands on the mutual information \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n between \n  \n    \n      \n        G\n        (\n        z\n        ,\n        c\n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0138", "text": ",\n        c\n        )\n      \n    \n    {\\displaystyle G(z,c)}\n  \n.\nUnfortunately, \n  \n    \n      \n        I\n        (\n        c\n        ,\n        G\n        (\n        z\n        ,\n        c\n        )\n        )\n      \n    \n    {\\displaystyle I(c,G(z,c))}\n  \n is intractable in general, The key idea of InfoGAN is Variational Mutual Information Maximization: indirectly maximize it by maximizing a lower bound\n  \n    \n      \n        \n          \n            \n              I\n              ^", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0139", "text": "I\n              ^\n            \n          \n        \n        (\n        G\n        ,\n        Q\n        )\n        =\n        \n          \n            E\n          \n          \n            z\n            \u223c\n            \n              \u03bc\n              \n                Z\n              \n            \n            ,\n            c\n            \u223c\n            \n              \u03bc\n              \n                C\n              \n            \n          \n        \n        [\n        ln\n        \u2061", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0140", "text": "[\n        ln\n        \u2061\n        Q\n        (\n        c\n        \u2223\n        G\n        (\n        z\n        ,\n        c\n        )\n        )\n        ]\n        ;\n        \n        I\n        (\n        c\n        ,\n        G\n        (\n        z\n        ,\n        c\n        )\n        )\n        \u2265\n        \n          sup\n          \n            Q\n          \n        \n        \n          \n            \n              I\n              ^\n            \n          \n        \n        (\n        G\n        ,", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0141", "text": "(\n        G\n        ,\n        Q\n        )\n      \n    \n    {\\displaystyle {\\hat {I}}(G,Q)=\\mathbb {E} _{z\\sim \\mu _{Z},c\\sim \\mu _{C}}[\\ln Q(c\\mid G(z,c))];\\quad I(c,G(z,c))\\geq \\sup _{Q}{\\hat {I}}(G,Q)}\n  \nwhere \n  \n    \n      \n        Q\n      \n    \n    {\\displaystyle Q}\n  \n ranges over all Markov kernels of type \n  \n    \n      \n        Q\n        :\n        \n          \u03a9\n          \n            Y\n          \n        \n        \u2192\n        \n          \n            P", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0142", "text": "P\n          \n        \n        (\n        \n          \u03a9\n          \n            C\n          \n        \n        )\n      \n    \n    {\\displaystyle Q:\\Omega _{Y}\\to {\\mathcal {P}}(\\Omega _{C})}\n  \n.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0143", "text": "The InfoGAN game is defined as follows:Three probability spaces define an InfoGAN game:\n\n  \n    \n      \n        (\n        \n          \u03a9\n          \n            X\n          \n        \n        ,\n        \n          \u03bc\n          \n            ref\n          \n        \n        )\n      \n    \n    {\\displaystyle (\\Omega _{X},\\mu _{\\text{ref}})}\n  \n, the space of reference images.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0144", "text": "(\n        \n          \u03a9\n          \n            Z\n          \n        \n        ,\n        \n          \u03bc\n          \n            Z\n          \n        \n        )\n      \n    \n    {\\displaystyle (\\Omega _{Z},\\mu _{Z})}\n  \n, the fixed random noise generator.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0145", "text": "(\n        \n          \u03a9\n          \n            C\n          \n        \n        ,\n        \n          \u03bc\n          \n            C\n          \n        \n        )\n      \n    \n    {\\displaystyle (\\Omega _{C},\\mu _{C})}\n  \n, the fixed random information generator.\nThere are 3 players in 2 teams: generator, Q, and discriminator. The generator and Q are on one team, and the discriminator on the other team.\nThe objective function is\n  \n    \n      \n        L\n        (\n        G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0146", "text": "L\n        (\n        G\n        ,\n        Q\n        ,\n        D\n        )\n        =\n        \n          L\n          \n            G\n            A\n            N\n          \n        \n        (\n        G\n        ,\n        D\n        )\n        \u2212\n        \u03bb\n        \n          \n            \n              I\n              ^\n            \n          \n        \n        (\n        G\n        ,\n        Q\n        )\n      \n    \n    {\\displaystyle L(G,Q,D)=L_{GAN}(G,D)-\\lambda {\\hat {I}}(G,Q)}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0147", "text": "where \n  \n    \n      \n        \n          L\n          \n            G\n            A\n            N\n          \n        \n        (\n        G\n        ,\n        D\n        )\n        =\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                ref\n              \n            \n            ,\n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        D\n        (\n        x\n        )\n        ]\n        +\n        \n          E", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0148", "text": "+\n        \n          E\n          \n            z\n            \u223c\n            \n              \u03bc\n              \n                Z\n              \n            \n          \n        \n        \u2061\n        [\n        ln\n        \u2061\n        (\n        1\n        \u2212\n        D\n        (\n        G\n        (\n        z\n        ,\n        c\n        )\n        )\n        )\n        ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0149", "text": ")\n        )\n        ]\n      \n    \n    {\\displaystyle L_{GAN}(G,D)=\\operatorname {E} _{x\\sim \\mu _{\\text{ref}},}[\\ln D(x)]+\\operatorname {E} _{z\\sim \\mu _{Z}}[\\ln(1-D(G(z,c)))]}\n  \n is the original GAN game objective, and \n  \n    \n      \n        \n          \n            \n              I\n              ^\n            \n          \n        \n        (\n        G\n        ,\n        Q\n        )\n        =\n        \n          \n            E\n          \n          \n            z\n            \u223c", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0150", "text": "z\n            \u223c\n            \n              \u03bc\n              \n                Z\n              \n            \n            ,\n            c\n            \u223c\n            \n              \u03bc\n              \n                C\n              \n            \n          \n        \n        [\n        ln\n        \u2061\n        Q\n        (\n        c\n        \u2223\n        G\n        (\n        z\n        ,\n        c\n        )\n        )\n        ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0151", "text": ")\n        )\n        ]\n      \n    \n    {\\displaystyle {\\hat {I}}(G,Q)=\\mathbb {E} _{z\\sim \\mu _{Z},c\\sim \\mu _{C}}[\\ln Q(c\\mid G(z,c))]}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0152", "text": "Generator-Q team aims to minimize the objective, and discriminator aims to maximize it:\n  \n    \n      \n        \n          min\n          \n            G\n            ,\n            Q\n          \n        \n        \n          max\n          \n            D\n          \n        \n        L\n        (\n        G\n        ,\n        Q\n        ,\n        D\n        )\n      \n    \n    {\\displaystyle \\min _{G,Q}\\max _{D}L(G,Q,D)}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0153", "text": "Bidirectional GAN (BiGAN)\nThe standard GAN generator is a function of type \n  \n    \n      \n        G\n        :\n        \n          \u03a9\n          \n            Z\n          \n        \n        \u2192\n        \n          \u03a9\n          \n            X\n          \n        \n      \n    \n    {\\displaystyle G:\\Omega _{Z}\\to \\Omega _{X}}\n  \n, that is, it is a mapping from a latent space \n  \n    \n      \n        \n          \u03a9\n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle \\Omega _{Z}}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0154", "text": "{\\displaystyle \\Omega _{Z}}\n  \n to the image space \n  \n    \n      \n        \n          \u03a9\n          \n            X\n          \n        \n      \n    \n    {\\displaystyle \\Omega _{X}}\n  \n. This can be understood as a \"decoding\" process, whereby every latent vector \n  \n    \n      \n        z\n        \u2208\n        \n          \u03a9\n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle z\\in \\Omega _{Z}}\n  \n is a code for an image \n  \n    \n      \n        x\n        \u2208", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0155", "text": "x\n        \u2208\n        \n          \u03a9\n          \n            X\n          \n        \n      \n    \n    {\\displaystyle x\\in \\Omega _{X}}\n  \n, and the generator performs the decoding. This naturally leads to the idea of training another network that performs \"encoding\", creating an autoencoder out of the encoder-generator pair.\nAlready in the original paper, the authors noted that \"Learned approximate inference can be performed by training an auxiliary network to predict", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0156", "text": "z\n      \n    \n    {\\displaystyle z}\n  \n given \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n\". The bidirectional GAN architecture performs exactly this.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0157", "text": "The BiGAN is defined as follows: Two probability spaces define a BiGAN game:\n\n  \n    \n      \n        (\n        \n          \u03a9\n          \n            X\n          \n        \n        ,\n        \n          \u03bc\n          \n            X\n          \n        \n        )\n      \n    \n    {\\displaystyle (\\Omega _{X},\\mu _{X})}\n  \n, the space of reference images.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0158", "text": "(\n        \n          \u03a9\n          \n            Z\n          \n        \n        ,\n        \n          \u03bc\n          \n            Z\n          \n        \n        )\n      \n    \n    {\\displaystyle (\\Omega _{Z},\\mu _{Z})}\n  \n, the latent space.\nThere are 3 players in 2 teams: generator, encoder, and discriminator. The generator and encoder are on one team, and the discriminator on the other team.\nThe generator's strategies are functions \n  \n    \n      \n        G\n        :", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0159", "text": "G\n        :\n        \n          \u03a9\n          \n            Z\n          \n        \n        \u2192\n        \n          \u03a9\n          \n            X\n          \n        \n      \n    \n    {\\displaystyle G:\\Omega _{Z}\\to \\Omega _{X}}\n  \n, and the encoder's strategies are functions \n  \n    \n      \n        E\n        :\n        \n          \u03a9\n          \n            X\n          \n        \n        \u2192\n        \n          \u03a9\n          \n            Z", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0160", "text": "Z\n          \n        \n      \n    \n    {\\displaystyle E:\\Omega _{X}\\to \\Omega _{Z}}\n  \n. The discriminator's strategies are functions \n  \n    \n      \n        D\n        :\n        \n          \u03a9\n          \n            X\n          \n        \n        \u2192\n        [\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle D:\\Omega _{X}\\to [0,1]}\n  \n.\nThe objective function is\n  \n    \n      \n        L\n        (\n        G\n        ,\n        E\n        ,\n        D\n        )\n        =", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0161", "text": "E\n        ,\n        D\n        )\n        =\n        \n          \n            E\n          \n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                X\n              \n            \n          \n        \n        [\n        ln\n        \u2061\n        D\n        (\n        x\n        ,\n        E\n        (\n        x\n        )\n        )\n        ]\n        +\n        \n          \n            E\n          \n          \n            z\n            \u223c\n            \n              \u03bc", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0162", "text": "\u223c\n            \n              \u03bc\n              \n                Z\n              \n            \n          \n        \n        [\n        ln\n        \u2061\n        (\n        1\n        \u2212\n        D\n        (\n        G\n        (\n        z\n        )\n        ,\n        z\n        )\n        )\n        ]\n      \n    \n    {\\displaystyle L(G,E,D)=\\mathbb {E} _{x\\sim \\mu _{X}}[\\ln D(x,E(x))]+\\mathbb {E} _{z\\sim \\mu _{Z}}[\\ln(1-D(G(z),z))]}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0163", "text": "Generator-encoder team aims to minimize the objective, and discriminator aims to maximize it:\n  \n    \n      \n        \n          min\n          \n            G\n            ,\n            E\n          \n        \n        \n          max\n          \n            D\n          \n        \n        L\n        (\n        G\n        ,\n        E\n        ,\n        D\n        )\n      \n    \n    {\\displaystyle \\min _{G,E}\\max _{D}L(G,E,D)}\n  \n In the paper, they gave a more abstract definition of the objective as:", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0164", "text": "L\n        (\n        G\n        ,\n        E\n        ,\n        D\n        )\n        =\n        \n          \n            E\n          \n          \n            (\n            x\n            ,\n            z\n            )\n            \u223c\n            \n              \u03bc\n              \n                E\n                ,\n                X\n              \n            \n          \n        \n        [\n        ln\n        \u2061\n        D\n        (\n        x\n        ,\n        z\n        )\n        ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0165", "text": "x\n        ,\n        z\n        )\n        ]\n        +\n        \n          \n            E\n          \n          \n            (\n            x\n            ,\n            z\n            )\n            \u223c\n            \n              \u03bc\n              \n                G\n                ,\n                Z\n              \n            \n          \n        \n        [\n        ln\n        \u2061\n        (\n        1\n        \u2212\n        D\n        (\n        x\n        ,\n        z\n        )\n        )\n        ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0166", "text": ")\n        )\n        ]\n      \n    \n    {\\displaystyle L(G,E,D)=\\mathbb {E} _{(x,z)\\sim \\mu _{E,X}}[\\ln D(x,z)]+\\mathbb {E} _{(x,z)\\sim \\mu _{G,Z}}[\\ln(1-D(x,z))]}\n  \nwhere \n  \n    \n      \n        \n          \u03bc\n          \n            E\n            ,\n            X\n          \n        \n        (\n        d\n        x\n        ,\n        d\n        z\n        )\n        =\n        \n          \u03bc\n          \n            X\n          \n        \n        (\n        d\n        x\n        )\n        \u22c5", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0167", "text": "d\n        x\n        )\n        \u22c5\n        \n          \u03b4\n          \n            E\n            (\n            x\n            )\n          \n        \n        (\n        d\n        z\n        )\n      \n    \n    {\\displaystyle \\mu _{E,X}(dx,dz)=\\mu _{X}(dx)\\cdot \\delta _{E(x)}(dz)}\n  \n is the probability distribution on \n  \n    \n      \n        \n          \u03a9\n          \n            X\n          \n        \n        \u00d7\n        \n          \u03a9\n          \n            Z", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0168", "text": "Z\n          \n        \n      \n    \n    {\\displaystyle \\Omega _{X}\\times \\Omega _{Z}}\n  \n obtained by pushing \n  \n    \n      \n        \n          \u03bc\n          \n            X\n          \n        \n      \n    \n    {\\displaystyle \\mu _{X}}\n  \n forward via \n  \n    \n      \n        x\n        \u21a6\n        (\n        x\n        ,\n        E\n        (\n        x\n        )\n        )\n      \n    \n    {\\displaystyle x\\mapsto (x,E(x))}\n  \n, and \n  \n    \n      \n        \n          \u03bc\n          \n            G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0169", "text": "\u03bc\n          \n            G\n            ,\n            Z\n          \n        \n        (\n        d\n        x\n        ,\n        d\n        z\n        )\n        =\n        \n          \u03b4\n          \n            G\n            (\n            z\n            )\n          \n        \n        (\n        d\n        x\n        )\n        \u22c5\n        \n          \u03bc\n          \n            Z\n          \n        \n        (\n        d\n        z\n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0170", "text": "d\n        z\n        )\n      \n    \n    {\\displaystyle \\mu _{G,Z}(dx,dz)=\\delta _{G(z)}(dx)\\cdot \\mu _{Z}(dz)}\n  \n is the probability distribution on \n  \n    \n      \n        \n          \u03a9\n          \n            X\n          \n        \n        \u00d7\n        \n          \u03a9\n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle \\Omega _{X}\\times \\Omega _{Z}}\n  \n obtained by pushing \n  \n    \n      \n        \n          \u03bc\n          \n            Z", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0171", "text": "Z\n          \n        \n      \n    \n    {\\displaystyle \\mu _{Z}}\n  \n forward via \n  \n    \n      \n        z\n        \u21a6\n        (\n        G\n        (\n        x\n        )\n        ,\n        z\n        )\n      \n    \n    {\\displaystyle z\\mapsto (G(x),z)}\n  \n.\nApplications of bidirectional models include semi-supervised learning, interpretable machine learning, and neural machine translation.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0172", "text": "CycleGAN\nCycleGAN is an architecture for performing translations between two domains, such as between photos of horses and photos of zebras, or photos of night cities and photos of day cities.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0173", "text": "The CycleGAN game is defined as follows:There are two probability spaces \n  \n    \n      \n        (\n        \n          \u03a9\n          \n            X\n          \n        \n        ,\n        \n          \u03bc\n          \n            X\n          \n        \n        )\n        ,\n        (\n        \n          \u03a9\n          \n            Y\n          \n        \n        ,\n        \n          \u03bc\n          \n            Y\n          \n        \n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0174", "text": ")\n      \n    \n    {\\displaystyle (\\Omega _{X},\\mu _{X}),(\\Omega _{Y},\\mu _{Y})}\n  \n, corresponding to the two domains needed for translations fore-and-back.\nThere are 4 players in 2 teams: generators \n  \n    \n      \n        \n          G\n          \n            X\n          \n        \n        :\n        \n          \u03a9\n          \n            X\n          \n        \n        \u2192\n        \n          \u03a9\n          \n            Y\n          \n        \n        ,\n        \n          G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0175", "text": ",\n        \n          G\n          \n            Y\n          \n        \n        :\n        \n          \u03a9\n          \n            Y\n          \n        \n        \u2192\n        \n          \u03a9\n          \n            X\n          \n        \n      \n    \n    {\\displaystyle G_{X}:\\Omega _{X}\\to \\Omega _{Y},G_{Y}:\\Omega _{Y}\\to \\Omega _{X}}\n  \n, and discriminators \n  \n    \n      \n        \n          D\n          \n            X\n          \n        \n        :\n        \n          \u03a9\n          \n            X", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0176", "text": "\u03a9\n          \n            X\n          \n        \n        \u2192\n        [\n        0\n        ,\n        1\n        ]\n        ,\n        \n          D\n          \n            Y\n          \n        \n        :\n        \n          \u03a9\n          \n            Y\n          \n        \n        \u2192\n        [\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle D_{X}:\\Omega _{X}\\to [0,1],D_{Y}:\\Omega _{Y}\\to [0,1]}\n  \n.\nThe objective function is\n  \n    \n      \n        L\n        (", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0177", "text": "L\n        (\n        \n          G\n          \n            X\n          \n        \n        ,\n        \n          G\n          \n            Y\n          \n        \n        ,\n        \n          D\n          \n            X\n          \n        \n        ,\n        \n          D\n          \n            Y\n          \n        \n        )\n        =\n        \n          L\n          \n            G\n            A\n            N\n          \n        \n        (\n        \n          G\n          \n            X", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0178", "text": "G\n          \n            X\n          \n        \n        ,\n        \n          D\n          \n            X\n          \n        \n        )\n        +\n        \n          L\n          \n            G\n            A\n            N\n          \n        \n        (\n        \n          G\n          \n            Y\n          \n        \n        ,\n        \n          D\n          \n            Y\n          \n        \n        )\n        +\n        \u03bb\n        \n          L\n          \n            c\n            y", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0179", "text": "c\n            y\n            c\n            l\n            e\n          \n        \n        (\n        \n          G\n          \n            X\n          \n        \n        ,\n        \n          G\n          \n            Y\n          \n        \n        )\n      \n    \n    {\\displaystyle L(G_{X},G_{Y},D_{X},D_{Y})=L_{GAN}(G_{X},D_{X})+L_{GAN}(G_{Y},D_{Y})+\\lambda L_{cycle}(G_{X},G_{Y})}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0180", "text": "where \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n is a positive adjustable parameter, \n  \n    \n      \n        \n          L\n          \n            G\n            A\n            N\n          \n        \n      \n    \n    {\\displaystyle L_{GAN}}\n  \n is the GAN game objective, and \n  \n    \n      \n        \n          L\n          \n            c\n            y\n            c\n            l\n            e\n          \n        \n      \n    \n    {\\displaystyle L_{cycle}}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0181", "text": "{\\displaystyle L_{cycle}}\n  \n is the cycle consistency loss:\n  \n    \n      \n        \n          L\n          \n            c\n            y\n            c\n            l\n            e\n          \n        \n        (\n        \n          G\n          \n            X\n          \n        \n        ,\n        \n          G\n          \n            Y\n          \n        \n        )\n        =\n        \n          E\n          \n            x\n            \u223c\n            \n              \u03bc", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0182", "text": "\u03bc\n              \n                X\n              \n            \n          \n        \n        \u2016\n        \n          G\n          \n            X\n          \n        \n        (\n        \n          G\n          \n            Y\n          \n        \n        (\n        x\n        )\n        )\n        \u2212\n        x\n        \u2016\n        +\n        \n          E\n          \n            y\n            \u223c\n            \n              \u03bc\n              \n                Y", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0183", "text": "Y\n              \n            \n          \n        \n        \u2016\n        \n          G\n          \n            Y\n          \n        \n        (\n        \n          G\n          \n            X\n          \n        \n        (\n        y\n        )\n        )\n        \u2212\n        y\n        \u2016\n      \n    \n    {\\displaystyle L_{cycle}(G_{X},G_{Y})=E_{x\\sim \\mu _{X}}\\|G_{X}(G_{Y}(x))-x\\|+E_{y\\sim \\mu _{Y}}\\|G_{Y}(G_{X}(y))-y\\|}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0184", "text": "The generators aim to minimize the objective, and the discriminators aim to maximize it:\n  \n    \n      \n        \n          min\n          \n            \n              G\n              \n                X\n              \n            \n            ,\n            \n              G\n              \n                Y\n              \n            \n          \n        \n        \n          max\n          \n            \n              D\n              \n                X\n              \n            \n            ,", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0185", "text": ",\n            \n              D\n              \n                Y\n              \n            \n          \n        \n        L\n        (\n        \n          G\n          \n            X\n          \n        \n        ,\n        \n          G\n          \n            Y\n          \n        \n        ,\n        \n          D\n          \n            X\n          \n        \n        ,\n        \n          D\n          \n            Y\n          \n        \n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0186", "text": ")\n      \n    \n    {\\displaystyle \\min _{G_{X},G_{Y}}\\max _{D_{X},D_{Y}}L(G_{X},G_{Y},D_{X},D_{Y})}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0187", "text": "Unlike previous work like pix2pix, which requires paired training data, cycleGAN requires no paired data. For example, to train a pix2pix model to turn a summer scenery photo to winter scenery photo and back, the dataset must contain pairs of the same place in summer and winter, shot at the same angle; cycleGAN would only need a set of summer scenery photos, and an unrelated set of winter scenery photos.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0188", "text": "GANs with particularly large or small scales\nBigGAN\nThe BigGAN is essentially a self-attention GAN trained on a large scale (up to 80 million parameters) to generate large images of ImageNet (up to 512 x 512 resolution), with numerous engineering tricks to make it converge.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0189", "text": "Invertible data augmentation\nWhen there is insufficient training data, the reference distribution \n  \n    \n      \n        \n          \u03bc\n          \n            ref\n          \n        \n      \n    \n    {\\displaystyle \\mu _{\\text{ref}}}\n  \n cannot be well-approximated by the empirical distribution given by the training dataset. In such cases, data augmentation can be applied, to allow training GAN on smaller datasets. Na\u00efve data augmentation, however, brings its problems.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0190", "text": "Consider the original GAN game, slightly reformulated as follows:\n  \n    \n      \n        \n          \n            {\n            \n              \n                \n                  \n                    min\n                    \n                      D\n                    \n                  \n                  \n                    L\n                    \n                      D\n                    \n                  \n                  (\n                  D\n                  ,", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0191", "text": ",\n                  \n                    \u03bc\n                    \n                      G\n                    \n                  \n                  )\n                  =\n                  \u2212\n                  \n                    E\n                    \n                      x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          ref", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0192", "text": "\u2061\n                  [\n                  ln\n                  \u2061\n                  D\n                  (\n                  x\n                  )\n                  ]\n                  \u2212\n                  \n                    E\n                    \n                      x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0193", "text": "G\n                        \n                      \n                    \n                  \n                  \u2061\n                  [\n                  ln\n                  \u2061\n                  (\n                  1\n                  \u2212\n                  D\n                  (\n                  x\n                  )\n                  )\n                  ]\n                \n              \n              \n                \n                  \n                    min", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0194", "text": "min\n                    \n                      G\n                    \n                  \n                  \n                    L\n                    \n                      G\n                    \n                  \n                  (\n                  D\n                  ,\n                  \n                    \u03bc\n                    \n                      G\n                    \n                  \n                  )\n                  =\n                  \u2212", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0195", "text": "=\n                  \u2212\n                  \n                    E\n                    \n                      x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          G\n                        \n                      \n                    \n                  \n                  \u2061\n                  [\n                  ln\n                  \u2061\n                  (\n                  1\n                  \u2212", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0196", "text": "1\n                  \u2212\n                  D\n                  (\n                  x\n                  )\n                  )\n                  ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0197", "text": "{\\displaystyle {\\begin{cases}\\min _{D}L_{D}(D,\\mu _{G})=-\\operatorname {E} _{x\\sim \\mu _{\\text{ref}}}[\\ln D(x)]-\\operatorname {E} _{x\\sim \\mu _{G}}[\\ln(1-D(x))]\\\\\\min _{G}L_{G}(D,\\mu _{G})=-\\operatorname {E} _{x\\sim \\mu _{G}}[\\ln(1-D(x))]\\end{cases}}}\n  \nNow we use data augmentation by randomly sampling semantic-preserving transforms \n  \n    \n      \n        T\n        :\n        \u03a9\n        \u2192\n        \u03a9", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0198", "text": "\u03a9\n        \u2192\n        \u03a9\n      \n    \n    {\\displaystyle T:\\Omega \\to \\Omega }\n  \n and applying them to the dataset, to obtain the reformulated GAN game:\n  \n    \n      \n        \n          \n            {\n            \n              \n                \n                  \n                    min\n                    \n                      D\n                    \n                  \n                  \n                    L\n                    \n                      D", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0199", "text": "D\n                    \n                  \n                  (\n                  D\n                  ,\n                  \n                    \u03bc\n                    \n                      G\n                    \n                  \n                  )\n                  =\n                  \u2212\n                  \n                    E\n                    \n                      x\n                      \u223c\n                      \n                        \u03bc", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0200", "text": "ref\n                        \n                      \n                      ,\n                      T\n                      \u223c\n                      \n                        \u03bc\n                        \n                          trans\n                        \n                      \n                    \n                  \n                  \u2061\n                  [\n                  ln\n                  \u2061\n                  D\n                  (", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0201", "text": "D\n                  (\n                  T\n                  (\n                  x\n                  )\n                  )\n                  ]\n                  \u2212\n                  \n                    E\n                    \n                      x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          G\n                        \n                      \n                    \n                  \n                  \u2061", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0202", "text": "\u2061\n                  [\n                  ln\n                  \u2061\n                  (\n                  1\n                  \u2212\n                  D\n                  (\n                  x\n                  )\n                  )\n                  ]\n                \n              \n              \n                \n                  \n                    min\n                    \n                      G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0203", "text": "L\n                    \n                      G\n                    \n                  \n                  (\n                  D\n                  ,\n                  \n                    \u03bc\n                    \n                      G\n                    \n                  \n                  )\n                  =\n                  \u2212\n                  \n                    E\n                    \n                      x", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0204", "text": "x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          G\n                        \n                      \n                    \n                  \n                  \u2061\n                  [\n                  ln\n                  \u2061\n                  (\n                  1\n                  \u2212\n                  D\n                  (\n                  x\n                  )\n                  )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0205", "text": ")\n                  )\n                  ]\n                \n              \n            \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{cases}\\min _{D}L_{D}(D,\\mu _{G})=-\\operatorname {E} _{x\\sim \\mu _{\\text{ref}},T\\sim \\mu _{\\text{trans}}}[\\ln D(T(x))]-\\operatorname {E} _{x\\sim \\mu _{G}}[\\ln(1-D(x))]\\\\\\min _{G}L_{G}(D,\\mu _{G})=-\\operatorname {E} _{x\\sim \\mu _{G}}[\\ln(1-D(x))]\\end{cases}}}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0206", "text": "This is equivalent to a GAN game with a different distribution \n  \n    \n      \n        \n          \u03bc\n          \n            ref\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\mu _{\\text{ref}}'}\n  \n, sampled by \n  \n    \n      \n        T\n        (\n        x\n        )\n      \n    \n    {\\displaystyle T(x)}\n  \n, with \n  \n    \n      \n        x\n        \u223c\n        \n          \u03bc\n          \n            ref\n          \n        \n        ,\n        T\n        \u223c\n        \n          \u03bc", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0207", "text": "\u223c\n        \n          \u03bc\n          \n            trans\n          \n        \n      \n    \n    {\\displaystyle x\\sim \\mu _{\\text{ref}},T\\sim \\mu _{\\text{trans}}}\n  \n. For example, if \n  \n    \n      \n        \n          \u03bc\n          \n            ref\n          \n        \n      \n    \n    {\\displaystyle \\mu _{\\text{ref}}}\n  \n is the distribution of images in ImageNet, and \n  \n    \n      \n        \n          \u03bc\n          \n            trans", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0208", "text": "trans\n          \n        \n      \n    \n    {\\displaystyle \\mu _{\\text{trans}}}\n  \n samples identity-transform with probability 0.5, and horizontal-reflection with probability 0.5, then \n  \n    \n      \n        \n          \u03bc\n          \n            ref\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\mu _{\\text{ref}}'}\n  \n is the distribution of images in ImageNet and horizontally-reflected ImageNet, combined.\nThe result of such training would be a generator that mimics", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0209", "text": "\u03bc\n          \n            ref\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\mu _{\\text{ref}}'}\n  \n. For example, it would generate images that look like they are randomly cropped, if the data augmentation uses random cropping.\nThe solution is to apply data augmentation to both generated and real images:\n  \n    \n      \n        \n          \n            {\n            \n              \n                \n                  \n                    min", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0210", "text": "min\n                    \n                      D\n                    \n                  \n                  \n                    L\n                    \n                      D\n                    \n                  \n                  (\n                  D\n                  ,\n                  \n                    \u03bc\n                    \n                      G\n                    \n                  \n                  )\n                  =\n                  \u2212", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0211", "text": "=\n                  \u2212\n                  \n                    E\n                    \n                      x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          ref\n                        \n                      \n                      ,\n                      T\n                      \u223c\n                      \n                        \u03bc\n                        \n                          trans", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0212", "text": "trans\n                        \n                      \n                    \n                  \n                  \u2061\n                  [\n                  ln\n                  \u2061\n                  D\n                  (\n                  T\n                  (\n                  x\n                  )\n                  )\n                  ]\n                  \u2212\n                  \n                    E\n                    \n                      x\n                      \u223c", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0213", "text": "x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          G\n                        \n                      \n                      ,\n                      T\n                      \u223c\n                      \n                        \u03bc\n                        \n                          trans\n                        \n                      \n                    \n                  \n                  \u2061", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0214", "text": "\u2061\n                  [\n                  ln\n                  \u2061\n                  (\n                  1\n                  \u2212\n                  D\n                  (\n                  T\n                  (\n                  x\n                  )\n                  )\n                  )\n                  ]\n                \n              \n              \n                \n                  \n                    min\n                    \n                      G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0215", "text": "G\n                    \n                  \n                  \n                    L\n                    \n                      G\n                    \n                  \n                  (\n                  D\n                  ,\n                  \n                    \u03bc\n                    \n                      G\n                    \n                  \n                  )\n                  =\n                  \u2212\n                  \n                    E", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0216", "text": "E\n                    \n                      x\n                      \u223c\n                      \n                        \u03bc\n                        \n                          G\n                        \n                      \n                      ,\n                      T\n                      \u223c\n                      \n                        \u03bc\n                        \n                          trans", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0217", "text": "\u2061\n                  [\n                  ln\n                  \u2061\n                  (\n                  1\n                  \u2212\n                  D\n                  (\n                  T\n                  (\n                  x\n                  )\n                  )\n                  )\n                  ]", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0218", "text": "{\\displaystyle {\\begin{cases}\\min _{D}L_{D}(D,\\mu _{G})=-\\operatorname {E} _{x\\sim \\mu _{\\text{ref}},T\\sim \\mu _{\\text{trans}}}[\\ln D(T(x))]-\\operatorname {E} _{x\\sim \\mu _{G},T\\sim \\mu _{\\text{trans}}}[\\ln(1-D(T(x)))]\\\\\\min _{G}L_{G}(D,\\mu _{G})=-\\operatorname {E} _{x\\sim \\mu _{G},T\\sim \\mu _{\\text{trans}}}[\\ln(1-D(T(x)))]\\end{cases}}}\n  \nThe authors demonstrated high-quality generation using just 100-picture-large datasets.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0219", "text": "The StyleGAN-2-ADA paper points out a further point on data augmentation: it must be invertible. Continue with the example of generating ImageNet pictures. If the data augmentation is \"randomly rotate the picture by 0, 90, 180, 270 degrees with equal probability\", then there is no way for the generator to know which is the true orientation: Consider two generators \n  \n    \n      \n        G\n        ,\n        \n          G\n          \u2032\n        \n      \n    \n    {\\displaystyle G,G'}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0220", "text": "{\\displaystyle G,G'}\n  \n, such that for any latent \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n, the generated image \n  \n    \n      \n        G\n        (\n        z\n        )\n      \n    \n    {\\displaystyle G(z)}\n  \n is a 90-degree rotation of \n  \n    \n      \n        \n          G\n          \u2032\n        \n        (\n        z\n        )\n      \n    \n    {\\displaystyle G'(z)}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0221", "text": "{\\displaystyle G'(z)}\n  \n. They would have exactly the same expected loss, and so neither is preferred over the other.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0222", "text": "The solution is to only use invertible data augmentation: instead of \"randomly rotate the picture by 0, 90, 180, 270 degrees with equal probability\", use \"randomly rotate the picture by 90, 180, 270 degrees with 0.1 probability, and keep the picture as it is with 0.7 probability\". This way, the generator is still rewarded  to keep images oriented the same way as un-augmented ImageNet pictures.\nAbstractly, the effect of randomly sampling transformations \n  \n    \n      \n        T\n        :", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0223", "text": "T\n        :\n        \u03a9\n        \u2192\n        \u03a9\n      \n    \n    {\\displaystyle T:\\Omega \\to \\Omega }\n  \n from the distribution \n  \n    \n      \n        \n          \u03bc\n          \n            trans\n          \n        \n      \n    \n    {\\displaystyle \\mu _{\\text{trans}}}\n  \n is to define a Markov kernel \n  \n    \n      \n        \n          K\n          \n            trans\n          \n        \n        :\n        \u03a9\n        \u2192\n        \n          \n            P\n          \n        \n        (", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0224", "text": "P\n          \n        \n        (\n        \u03a9\n        )\n      \n    \n    {\\displaystyle K_{\\text{trans}}:\\Omega \\to {\\mathcal {P}}(\\Omega )}\n  \n. Then, the data-augmented GAN game pushes the generator to find some \n  \n    \n      \n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            G\n          \n        \n        \u2208\n        \n          \n            P\n          \n        \n        (\n        \u03a9\n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0225", "text": "(\n        \u03a9\n        )\n      \n    \n    {\\displaystyle {\\hat {\\mu }}_{G}\\in {\\mathcal {P}}(\\Omega )}\n  \n, such that \n  \n    \n      \n        \n          K\n          \n            trans\n          \n        \n        \u2217\n        \n          \u03bc\n          \n            ref\n          \n        \n        =\n        \n          K\n          \n            trans\n          \n        \n        \u2217\n        \n          \n            \n              \n                \u03bc\n                ^", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0226", "text": "^\n              \n            \n          \n          \n            G\n          \n        \n      \n    \n    {\\displaystyle K_{\\text{trans}}*\\mu _{\\text{ref}}=K_{\\text{trans}}*{\\hat {\\mu }}_{G}}\n  \nwhere \n  \n    \n      \n        \u2217\n      \n    \n    {\\displaystyle *}\n  \n is the Markov kernel convolution.\nA data-augmentation method is defined to be invertible if its Markov kernel \n  \n    \n      \n        \n          K\n          \n            trans", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0227", "text": "trans\n          \n        \n      \n    \n    {\\displaystyle K_{\\text{trans}}}\n  \n satisfies\n  \n    \n      \n        \n          K\n          \n            trans\n          \n        \n        \u2217\n        \u03bc\n        =\n        \n          K\n          \n            trans\n          \n        \n        \u2217\n        \n          \u03bc\n          \u2032\n        \n        \n        \u27f9\n        \n        \u03bc\n        =\n        \n          \u03bc\n          \u2032\n        \n        \n        \u2200\n        \u03bc\n        ,\n        \n          \u03bc\n          \u2032", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0228", "text": ",\n        \n          \u03bc\n          \u2032\n        \n        \u2208\n        \n          \n            P\n          \n        \n        (\n        \u03a9\n        )\n      \n    \n    {\\displaystyle K_{\\text{trans}}*\\mu =K_{\\text{trans}}*\\mu '\\implies \\mu =\\mu '\\quad \\forall \\mu ,\\mu '\\in {\\mathcal {P}}(\\Omega )}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0229", "text": "Immediately by definition, we see that composing multiple invertible data-augmentation methods results in yet another invertible method. Also by definition, if the data-augmentation method is invertible, then using it in a GAN game does not change the optimal strategy \n  \n    \n      \n        \n          \n            \n              \n                \u03bc\n                ^\n              \n            \n          \n          \n            G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0230", "text": "G\n          \n        \n      \n    \n    {\\displaystyle {\\hat {\\mu }}_{G}}\n  \n for the generator, which is still \n  \n    \n      \n        \n          \u03bc\n          \n            ref\n          \n        \n      \n    \n    {\\displaystyle \\mu _{\\text{ref}}}\n  \n.\nThere are two prototypical examples of invertible Markov kernels:\nDiscrete case: Invertible stochastic matrices, when \n  \n    \n      \n        \u03a9\n      \n    \n    {\\displaystyle \\Omega }\n  \n is finite.\nFor example, if", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0231", "text": "is finite.\nFor example, if \n  \n    \n      \n        \u03a9\n        =\n        {\n        \u2191\n        ,\n        \u2193\n        ,\n        \u2190\n        ,\n        \u2192\n        }\n      \n    \n    {\\displaystyle \\Omega =\\{\\uparrow ,\\downarrow ,\\leftarrow ,\\rightarrow \\}}\n  \n is the set of four images of an arrow, pointing in 4 directions, and the data augmentation is \"randomly rotate the picture by 90, 180, 270 degrees with probability \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0232", "text": "p\n      \n    \n    {\\displaystyle p}\n  \n, and keep the picture as it is with probability \n  \n    \n      \n        (\n        1\n        \u2212\n        3\n        p\n        )\n      \n    \n    {\\displaystyle (1-3p)}\n  \n\", then the Markov kernel \n  \n    \n      \n        \n          K\n          \n            trans\n          \n        \n      \n    \n    {\\displaystyle K_{\\text{trans}}}\n  \n can be represented as a stochastic matrix:\n  \n    \n      \n        [\n        \n          K\n          \n            trans", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0233", "text": "K\n          \n            trans\n          \n        \n        ]\n        =\n        \n          \n            [\n            \n              \n                \n                  (\n                  1\n                  \u2212\n                  3\n                  p\n                  )\n                \n                \n                  p\n                \n                \n                  p\n                \n                \n                  p", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0234", "text": "p\n                \n                \n                  (\n                  1\n                  \u2212\n                  3\n                  p\n                  )\n                \n                \n                  p\n                \n                \n                  p\n                \n              \n              \n                \n                  p\n                \n                \n                  p", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0235", "text": "p\n                \n                \n                  (\n                  1\n                  \u2212\n                  3\n                  p\n                  )\n                \n                \n                  p\n                \n              \n              \n                \n                  p\n                \n                \n                  p\n                \n                \n                  p\n                \n                \n                  (", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0236", "text": "(\n                  1\n                  \u2212\n                  3\n                  p\n                  )\n                \n              \n            \n            ]\n          \n        \n      \n    \n    {\\displaystyle [K_{\\text{trans}}]={\\begin{bmatrix}(1-3p)&p&p&p\\\\p&(1-3p)&p&p\\\\p&p&(1-3p)&p\\\\p&p&p&(1-3p)\\end{bmatrix}}}\n  \n and \n  \n    \n      \n        \n          K\n          \n            trans\n          \n        \n      \n    \n    {\\displaystyle K_{\\text{trans}}}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0237", "text": "{\\displaystyle K_{\\text{trans}}}\n  \n is an invertible kernel iff \n  \n    \n      \n        [\n        \n          K\n          \n            trans\n          \n        \n        ]\n      \n    \n    {\\displaystyle [K_{\\text{trans}}]}\n  \n is an invertible matrix, that is, \n  \n    \n      \n        p\n        \u2260\n        1\n        \n          /\n        \n        4\n      \n    \n    {\\displaystyle p\\neq 1/4}\n  \n.\nContinuous case: The gaussian kernel, when \n  \n    \n      \n        \u03a9\n        =", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0238", "text": "\u03a9\n        =\n        \n          \n            R\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\Omega =\\mathbb {R} ^{n}}\n  \n for some \n  \n    \n      \n        n\n        \u2265\n        1\n      \n    \n    {\\displaystyle n\\geq 1}\n  \n.\nFor example, if \n  \n    \n      \n        \u03a9\n        =\n        \n          \n            R\n          \n          \n            \n              256\n              \n                2", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0239", "text": "{\\displaystyle \\Omega =\\mathbb {R} ^{256^{2}}}\n  \n is the space of 256x256 images, and the data-augmentation method is \"generate a gaussian noise \n  \n    \n      \n        z\n        \u223c\n        \n          \n            N\n          \n        \n        (\n        0\n        ,\n        \n          I\n          \n            \n              256\n              \n                2\n              \n            \n          \n        \n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0240", "text": ")\n      \n    \n    {\\displaystyle z\\sim {\\mathcal {N}}(0,I_{256^{2}})}\n  \n, then add \n  \n    \n      \n        \u03f5\n        z\n      \n    \n    {\\displaystyle \\epsilon z}\n  \n to the image\", then \n  \n    \n      \n        \n          K\n          \n            trans\n          \n        \n      \n    \n    {\\displaystyle K_{\\text{trans}}}\n  \n is just convolution by the density function of \n  \n    \n      \n        \n          \n            N\n          \n        \n        (\n        0", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0241", "text": "(\n        0\n        ,\n        \n          \u03f5\n          \n            2\n          \n        \n        \n          I\n          \n            \n              256\n              \n                2\n              \n            \n          \n        \n        )\n      \n    \n    {\\displaystyle {\\mathcal {N}}(0,\\epsilon ^{2}I_{256^{2}})}\n  \n. This is invertible, because convolution by a gaussian is just convolution by the heat kernel, so given any \n  \n    \n      \n        \u03bc\n        \u2208", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0242", "text": "\u03bc\n        \u2208\n        \n          \n            P\n          \n        \n        (\n        \n          \n            R\n          \n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle \\mu \\in {\\mathcal {P}}(\\mathbb {R} ^{n})}\n  \n, the convolved distribution \n  \n    \n      \n        \n          K\n          \n            trans\n          \n        \n        \u2217\n        \u03bc\n      \n    \n    {\\displaystyle K_{\\text{trans}}*\\mu }\n  \n can be obtained by heating up", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0243", "text": "can be obtained by heating up \n  \n    \n      \n        \n          \n            R\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {R} ^{n}}\n  \n precisely according to \n  \n    \n      \n        \u03bc\n      \n    \n    {\\displaystyle \\mu }\n  \n, then wait for time \n  \n    \n      \n        \n          \u03f5\n          \n            2\n          \n        \n        \n          /\n        \n        4\n      \n    \n    {\\displaystyle \\epsilon ^{2}/4}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0244", "text": "{\\displaystyle \\epsilon ^{2}/4}\n  \n. With that, we can recover \n  \n    \n      \n        \u03bc\n      \n    \n    {\\displaystyle \\mu }\n  \n by running the heat equation backwards in time for \n  \n    \n      \n        \n          \u03f5\n          \n            2\n          \n        \n        \n          /\n        \n        4\n      \n    \n    {\\displaystyle \\epsilon ^{2}/4}\n  \n.\nMore examples of invertible data augmentations are found in the paper.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0245", "text": "SinGAN\nSinGAN pushes data augmentation to the limit, by using only a single image as training data and performing data augmentation on it. The GAN architecture is adapted to this training method by using a multi-scale pipeline.\nThe generator \n  \n    \n      \n        G\n      \n    \n    {\\displaystyle G}\n  \n is decomposed into a pyramid of generators \n  \n    \n      \n        G\n        =\n        \n          G\n          \n            1\n          \n        \n        \u2218\n        \n          G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0246", "text": "\u2218\n        \n          G\n          \n            2\n          \n        \n        \u2218\n        \u22ef\n        \u2218\n        \n          G\n          \n            N\n          \n        \n      \n    \n    {\\displaystyle G=G_{1}\\circ G_{2}\\circ \\cdots \\circ G_{N}}\n  \n, with the lowest one generating the image \n  \n    \n      \n        \n          G\n          \n            N\n          \n        \n        (\n        \n          z\n          \n            N\n          \n        \n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0247", "text": ")\n      \n    \n    {\\displaystyle G_{N}(z_{N})}\n  \n at the lowest resolution, then the generated image is scaled up to \n  \n    \n      \n        r\n        (\n        \n          G\n          \n            N\n          \n        \n        (\n        \n          z\n          \n            N\n          \n        \n        )\n        )\n      \n    \n    {\\displaystyle r(G_{N}(z_{N}))}\n  \n, and fed to the next level to generate an image \n  \n    \n      \n        \n          G", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0248", "text": "G\n          \n            N\n            \u2212\n            1\n          \n        \n        (\n        \n          z\n          \n            N\n            \u2212\n            1\n          \n        \n        +\n        r\n        (\n        \n          G\n          \n            N\n          \n        \n        (\n        \n          z\n          \n            N\n          \n        \n        )\n        )\n        )\n      \n    \n    {\\displaystyle G_{N-1}(z_{N-1}+r(G_{N}(z_{N})))}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0249", "text": "at a higher resolution, and so on. The discriminator is decomposed into a pyramid as well.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0250", "text": "StyleGAN series\nThe StyleGAN family is a series of architectures published by Nvidia's research division.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0251", "text": "Progressive GAN\nProgressive GAN is a method for training GAN for large-scale image generation stably, by growing a GAN generator from small to large scale in a pyramidal fashion. Like SinGAN, it decomposes the generator as\n  \n    \n      \n        G\n        =\n        \n          G\n          \n            1\n          \n        \n        \u2218\n        \n          G\n          \n            2\n          \n        \n        \u2218\n        \u22ef\n        \u2218\n        \n          G\n          \n            N", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0252", "text": "N\n          \n        \n      \n    \n    {\\displaystyle G=G_{1}\\circ G_{2}\\circ \\cdots \\circ G_{N}}\n  \n, and the discriminator as \n  \n    \n      \n        D\n        =\n        \n          D\n          \n            1\n          \n        \n        \u2218\n        \n          D\n          \n            2\n          \n        \n        \u2218\n        \u22ef\n        \u2218\n        \n          D\n          \n            N\n          \n        \n      \n    \n    {\\displaystyle D=D_{1}\\circ D_{2}\\circ \\cdots \\circ D_{N}}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0253", "text": ".\nDuring training, at first only \n  \n    \n      \n        \n          G\n          \n            N\n          \n        \n        ,\n        \n          D\n          \n            N\n          \n        \n      \n    \n    {\\displaystyle G_{N},D_{N}}\n  \n are used in a GAN game to generate 4x4 images. Then \n  \n    \n      \n        \n          G\n          \n            N\n            \u2212\n            1\n          \n        \n        ,\n        \n          D\n          \n            N\n            \u2212\n            1", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0254", "text": "\u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle G_{N-1},D_{N-1}}\n  \n are added to reach the second stage of GAN game, to generate 8x8 images, and so on, until we reach a GAN game to generate 1024x1024 images.\nTo avoid shock between stages of the GAN game, each new layer is \"blended in\" (Figure 2 of the paper). For example, this is how the second stage GAN game starts:", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0255", "text": "Just before, the GAN game consists of the pair \n  \n    \n      \n        \n          G\n          \n            N\n          \n        \n        ,\n        \n          D\n          \n            N\n          \n        \n      \n    \n    {\\displaystyle G_{N},D_{N}}\n  \n generating and discriminating 4x4 images.\nJust after, the GAN game consists of the pair \n  \n    \n      \n        (\n        (\n        1\n        \u2212\n        \u03b1\n        )\n        +\n        \u03b1\n        \u22c5\n        \n          G\n          \n            N", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0256", "text": "G\n          \n            N\n            \u2212\n            1\n          \n        \n        )\n        \u2218\n        u\n        \u2218\n        \n          G\n          \n            N\n          \n        \n        ,\n        \n          D\n          \n            N\n          \n        \n        \u2218\n        d\n        \u2218\n        (\n        (\n        1\n        \u2212\n        \u03b1\n        )\n        +\n        \u03b1\n        \u22c5\n        \n          D\n          \n            N\n            \u2212\n            1\n          \n        \n        )", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0257", "text": "1\n          \n        \n        )\n      \n    \n    {\\displaystyle ((1-\\alpha )+\\alpha \\cdot G_{N-1})\\circ u\\circ G_{N},D_{N}\\circ d\\circ ((1-\\alpha )+\\alpha \\cdot D_{N-1})}\n  \n generating and discriminating 8x8 images. Here, the functions \n  \n    \n      \n        u\n        ,\n        d\n      \n    \n    {\\displaystyle u,d}\n  \n are image up- and down-sampling functions, and \n  \n    \n      \n        \u03b1\n      \n    \n    {\\displaystyle \\alpha }", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0258", "text": "{\\displaystyle \\alpha }\n  \n is a blend-in factor (much like an alpha in image composing) that smoothly glides from 0 to 1.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0259", "text": "StyleGAN-1\nStyleGAN-1 is designed as a combination of Progressive GAN with neural style transfer.\nThe key architectural choice of StyleGAN-1 is a progressive growth mechanism, similar to Progressive GAN. Each generated image starts as a constant \n  \n    \n      \n        4\n        \u00d7\n        4\n        \u00d7\n        512\n      \n    \n    {\\displaystyle 4\\times 4\\times 512}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0260", "text": "{\\displaystyle 4\\times 4\\times 512}\n  \n array, and repeatedly passed through style blocks. Each style block applies a \"style latent vector\" via affine transform (\"adaptive instance normalization\"), similar to how neural style transfer uses Gramian matrix. It then adds noise, and normalize (subtract the mean, then divide by the variance).", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0261", "text": "At training time, usually only one style latent vector is used per image generated, but sometimes two (\"mixing regularization\") in order to encourage each style block to independently perform its stylization without expecting help from other style blocks (since they might receive an entirely different style latent vector).", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0262", "text": "After training, multiple style latent vectors can be fed into each style block. Those fed to the lower layers control the large-scale styles, and those fed to the higher layers control the fine-detail styles.\nStyle-mixing between two images \n  \n    \n      \n        x\n        ,\n        \n          x\n          \u2032\n        \n      \n    \n    {\\displaystyle x,x'}\n  \n can be performed as well. First, run a gradient descent to find \n  \n    \n      \n        z\n        ,\n        \n          z\n          \u2032", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0263", "text": ",\n        \n          z\n          \u2032\n        \n      \n    \n    {\\displaystyle z,z'}\n  \n such that \n  \n    \n      \n        G\n        (\n        z\n        )\n        \u2248\n        x\n        ,\n        G\n        (\n        \n          z\n          \u2032\n        \n        )\n        \u2248\n        \n          x\n          \u2032\n        \n      \n    \n    {\\displaystyle G(z)\\approx x,G(z')\\approx x'}\n  \n. This is called \"projecting an image back to style latent space\". Then, \n  \n    \n      \n        z", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0264", "text": "z\n      \n    \n    {\\displaystyle z}\n  \n can be fed to the lower style blocks, and \n  \n    \n      \n        \n          z\n          \u2032\n        \n      \n    \n    {\\displaystyle z'}\n  \n to the higher style blocks, to generate a composite image that has the large-scale style of \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n, and the fine-detail style of \n  \n    \n      \n        \n          x\n          \u2032\n        \n      \n    \n    {\\displaystyle x'}", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0265", "text": "{\\displaystyle x'}\n  \n. Multiple images can also be composed this way.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0266", "text": "StyleGAN-2\nStyleGAN-2 improves upon StyleGAN-1, by using the style latent vector to transform the convolution layer's weights instead, thus solving the \"blob\" problem.\nThis was updated by the StyleGAN-2-ADA (\"ADA\" stands for \"adaptive\"), which uses invertible data augmentation as described above. It also tunes the amount of data augmentation applied by starting at zero, and gradually increasing it until an \"overfitting heuristic\" reaches a target level, thus the name \"adaptive\".", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0267", "text": "StyleGAN-3\nStyleGAN-3 improves upon StyleGAN-2 by solving the \"texture sticking\" problem, which can be seen in the official videos. They analyzed the problem by the Nyquist\u2013Shannon sampling theorem, and argued that the layers in the generator learned to exploit the high-frequency signal in the pixels they operate upon.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0268", "text": "To solve this, they proposed imposing strict lowpass filters between each generator's layers, so that the generator is forced to operate on the pixels in a way faithful to the continuous signals they represent, rather than operate on them as merely discrete signals. They further imposed rotational and translational invariance by using more signal filters. The resulting StyleGAN-3 is able to solve the texture sticking problem, as well as generating images that rotate and translate smoothly.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0269", "text": "Other uses\nOther than for generative and discriminative modelling of data, GANs have been used for other things.\nGANs have been used for transfer learning to enforce the alignment of the latent feature space, such as in deep reinforcement learning. This works by feeding the embeddings of the source and target task to the discriminator which tries to guess the context. The resulting loss is then (inversely) backpropagated through the encoder.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0270", "text": "Applications\nScience\nIteratively reconstruct astronomical images\nSimulate gravitational lensing for dark matter research.\nModel the distribution of dark matter in a particular direction in space and to predict the gravitational lensing that will occur.\nModel high energy jet formation and showers through calorimeters of high-energy physics experiments.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0271", "text": "Approximate bottlenecks in computationally expensive simulations of particle physics experiments. Applications in the context of present and proposed CERN experiments have demonstrated the potential of these methods for accelerating simulation and/or improving simulation fidelity.\nReconstruct velocity and scalar fields in turbulent flows.\nGAN-generated molecules were validated experimentally in mice.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0272", "text": "Medical\nOne of the major concerns in medical imaging is preserving patient privacy. Due to these reasons, researchers often face difficulties in obtaining medical images for their research purposes. GAN has been used for generating synthetic medical images, such as MRI and PET images to address this challenge.\nGAN can be used to detect glaucomatous images helping the early diagnosis which is essential to avoid partial or total loss of vision.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0273", "text": "GANs have been used to create forensic facial reconstructions of deceased historical figures.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0274", "text": "Malicious\nConcerns have been raised about the potential use of GAN-based human image synthesis for sinister purposes, e.g., to produce fake, possibly incriminating, photographs and videos.\nGANs can be used to generate unique, realistic profile photos of people who do not exist, in order to automate creation of fake social media profiles.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0275", "text": "In 2019 the state of California considered and passed on October 3, 2019, the bill AB-602, which bans the use of human image synthesis technologies to make fake pornography without the consent of the people depicted, and bill AB-730, which prohibits distribution of manipulated videos of a political candidate within 60 days of an election. Both bills were authored by Assembly member Marc Berman and signed by Governor Gavin Newsom. The laws went into effect in 2020.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0276", "text": "DARPA's Media Forensics program studies ways to counteract fake media, including fake media produced using GANs.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0277", "text": "Fashion, art and advertising\nGANs can be used to generate art; The Verge wrote in March 2019 that \"The images created by GANs have become the defining look of contemporary AI art.\" GANs can also be used to", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0278", "text": "inpaint photographs\ngenerate fashion models, shadows, photorealistic renders of interior design, industrial design, shoes, etc. Such networks were reported to be used by Facebook.\nSome have worked with using GAN for artistic creativity, as \"creative adversarial network\". A GAN, trained on a set of 15,000 portraits from WikiArt from the 14th to the 19th century, created the 2018 painting Edmond de Belamy, which sold for US$432,500.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0279", "text": "GANs were used by the video game modding community to up-scale low-resolution 2D textures in old video games by recreating them in 4k or higher resolutions via image training, and then down-sampling them to fit the game's native resolution (resembling supersampling anti-aliasing).", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0280", "text": "In 2020, Artbreeder was used to create the main antagonist in the sequel to the psychological web horror series Ben Drowned. The author would later go on to praise GAN applications for their ability to help generate assets for independent artists who are short on budget and manpower.\nIn May 2020, Nvidia researchers taught an AI system (termed \"GameGAN\") to recreate the game of Pac-Man simply by watching it being played.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0281", "text": "In August 2019, a large dataset consisting of 12,197 MIDI songs each with paired lyrics and melody alignment was created for neural melody generation from lyrics using conditional GAN-LSTM (refer to sources at GitHub AI Melody Generation from Lyrics).", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0282", "text": "Miscellaneous\nGANs have been used to", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0283", "text": "show how an individual's appearance might change with age.\nreconstruct 3D models of objects from images,\ngenerate novel objects as 3D point clouds,\nmodel patterns of motion in video.\ninpaint missing features in maps, transfer map styles in cartography or augment street view imagery.\nuse feedback to generate images and replace image search systems.\nvisualize the effect that climate change will have on specific houses.\nreconstruct an image of a person's face after listening to their voice.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0284", "text": "produces videos of a person speaking, given only a single photo of that person.\nrecurrent sequence generation.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0285", "text": "History\nIn 1991, Juergen Schmidhuber published \"artificial curiosity\", neural networks in a zero-sum game. The first network is a generative model that models a probability distribution over output patterns. The second network learns by gradient descent to predict the reactions of the environment to these patterns. GANs can be regarded as a case where the environmental reaction is 1 or 0 depending on whether the first network's output is in a given set.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0286", "text": "Other people had similar ideas but did not develop them similarly. An idea involving adversarial networks was published in a 2010 blog post by Olli Niemitalo. This idea was never implemented and did not involve stochasticity in the generator and thus was not a generative model. It is now known as a conditional GAN or cGAN. An idea similar to GANs was used to model animal behavior by Wei Li, Melvin Gauci and Roderich Gross in 2013.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0287", "text": "Another inspiration for GANs was noise-contrastive estimation, which uses the same loss function as GANs and which Goodfellow studied during his PhD in 2010\u20132014.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0288", "text": "Adversarial machine learning has other uses besides generative modeling and can be applied to models other than neural networks. In control theory, adversarial learning based on neural networks was used in 2006 to train robust controllers in a game theoretic sense, by alternating the iterations between a minimizer policy, the controller, and a maximizer policy, the disturbance.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0289", "text": "In 2017, a GAN was used for image enhancement focusing on realistic textures rather than pixel-accuracy, producing a higher image quality at high magnification. In 2017, the first faces were generated. These were exhibited in February 2018 at the Grand Palais. Faces generated by StyleGAN in 2019 drew comparisons with Deepfakes.", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0290", "text": "See also\nArtificial intelligence art\nDeepfake \u2013 Realistic artificially generated media\nDeep learning \u2013 Branch of machine learning\nDiffusion model \u2013 Technique for the generative modeling of a continuous probability distribution\nGenerative artificial intelligence \u2013 Subset of AI using generative modelsPages displaying short descriptions of redirect targets\nSynthetic media \u2013 Artificial production of media by automated means\n\nReferences\nExternal links", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0291", "text": "Knight, Will. \"5 Big Predictions for Artificial Intelligence in 2017\". MIT Technology Review. Retrieved January 5, 2017.\nKarras, Tero; Laine, Samuli; Aila, Timo (2018). \"A Style-Based Generator Architecture for Generative Adversarial Networks\". arXiv:1812.04948 [cs.NE].\nThis Person Does Not Exist \u2013  photorealistic images of people who do not exist, generated by StyleGAN", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Generative_adversarial_network_0292", "text": "This Cat Does Not Exist Archived March 5, 2019, at the Wayback Machine \u2013  photorealistic images of cats who do not exist, generated by StyleGAN\nWang, Zhengwei; She, Qi; Ward, Tomas E. (2019). \"Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy\". arXiv:1906.01529 [cs.LG].", "title": "Generative adversarial network", "url": "https://en.wikipedia.org/wiki/Generative_adversarial_network"}, {"id": "rec_Autoencoder_0000", "text": "An autoencoder is a type of artificial neural network used to learn efficient codings of unlabeled data (unsupervised learning). An autoencoder learns two functions: an encoding function that transforms the input data, and a decoding function that recreates the input data from the encoded representation", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0001", "text": ". The autoencoder learns an efficient representation (encoding) for a set of data, typically for dimensionality reduction, to generate lower-dimensional embeddings for subsequent use by other machine learning algorithms.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0002", "text": "Variants exist which aim to make the learned representations assume useful properties. Examples are regularized autoencoders (sparse, denoising and contractive autoencoders), which are effective in learning representations for subsequent classification tasks, and variational autoencoders, which can be used as generative models. Autoencoders are applied to many problems, including facial recognition, feature detection, anomaly detection, and learning the meaning of words", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0003", "text": ". In terms of data synthesis, autoencoders can also be used to randomly generate new data that is similar to the input (training) data.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0004", "text": "Mathematical principles\nDefinition\nAn autoencoder is defined by the following components: Two sets: the space of encoded messages \n  \n    \n      \n        \n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {Z}}}\n  \n; the space of decoded messages \n  \n    \n      \n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {X}}}\n  \n. Typically \n  \n    \n      \n        \n          \n            X", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0005", "text": "X\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {X}}}\n  \n and \n  \n    \n      \n        \n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {Z}}}\n  \n are Euclidean spaces, that is, \n  \n    \n      \n        \n          \n            X\n          \n        \n        =\n        \n          \n            R\n          \n          \n            m\n          \n        \n        ,\n        \n          \n            Z\n          \n        \n        =", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0006", "text": "=\n        \n          \n            R\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {X}}=\\mathbb {R} ^{m},{\\mathcal {Z}}=\\mathbb {R} ^{n}}\n  \n with \n  \n    \n      \n        m\n        >\n        n\n        .\n      \n    \n    {\\displaystyle m>n.}\n  \n    Two parametrized families of functions: the encoder family \n  \n    \n      \n        \n          E\n          \n            \u03d5\n          \n        \n        :", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0007", "text": ":\n        \n          \n            X\n          \n        \n        \u2192\n        \n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle E_{\\phi }:{\\mathcal {X}}\\rightarrow {\\mathcal {Z}}}\n  \n, parametrized by \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n; the decoder family \n  \n    \n      \n        \n          D\n          \n            \u03b8\n          \n        \n        :\n        \n          \n            Z\n          \n        \n        \u2192", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0008", "text": "Z\n          \n        \n        \u2192\n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle D_{\\theta }:{\\mathcal {Z}}\\rightarrow {\\mathcal {X}}}\n  \n, parametrized by \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n.For any \n  \n    \n      \n        x\n        \u2208\n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle x\\in {\\mathcal {X}}}\n  \n, we usually write \n  \n    \n      \n        z\n        =", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0009", "text": "z\n        =\n        \n          E\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle z=E_{\\phi }(x)}\n  \n, and refer to it as the code, the latent variable, latent representation, latent vector, etc. Conversely, for any \n  \n    \n      \n        z\n        \u2208\n        \n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle z\\in {\\mathcal {Z}}}\n  \n, we usually write \n  \n    \n      \n        \n          x", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0010", "text": "x\n          \u2032\n        \n        =\n        \n          D\n          \n            \u03b8\n          \n        \n        (\n        z\n        )\n      \n    \n    {\\displaystyle x'=D_{\\theta }(z)}\n  \n, and refer to it as the (decoded) message.\nUsually, both the encoder and the decoder are defined as multilayer perceptrons (MLPs). For example, a one-layer-MLP encoder \n  \n    \n      \n        \n          E\n          \n            \u03d5", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0011", "text": "\u03d5\n          \n        \n      \n    \n    {\\displaystyle E_{\\phi }}\n  \n is:", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0012", "text": "E\n          \n            \u03d5\n          \n        \n        (\n        \n          x\n        \n        )\n        =\n        \u03c3\n        (\n        W\n        x\n        +\n        b\n        )\n      \n    \n    {\\displaystyle E_{\\phi }(\\mathbf {x} )=\\sigma (Wx+b)}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0013", "text": "where \n  \n    \n      \n        \u03c3\n      \n    \n    {\\displaystyle \\sigma }\n  \n is an element-wise activation function, \n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}\n  \n is a \"weight\" matrix, and \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}\n  \n is a \"bias\" vector.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0014", "text": "Training an autoencoder\nAn autoencoder, by itself, is simply a tuple of two functions. To judge its quality, we need a task. A task is defined by a reference probability distribution \n  \n    \n      \n        \n          \u03bc\n          \n            r\n            e\n            f\n          \n        \n      \n    \n    {\\displaystyle \\mu _{ref}}\n  \n over \n  \n    \n      \n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {X}}}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0015", "text": "{\\displaystyle {\\mathcal {X}}}\n  \n, and a \"reconstruction quality\" function \n  \n    \n      \n        d\n        :\n        \n          \n            X\n          \n        \n        \u00d7\n        \n          \n            X\n          \n        \n        \u2192\n        [\n        0\n        ,\n        \u221e\n        ]\n      \n    \n    {\\displaystyle d:{\\mathcal {X}}\\times {\\mathcal {X}}\\to [0,\\infty ]}\n  \n, such that \n  \n    \n      \n        d\n        (\n        x\n        ,\n        \n          x\n          \u2032", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0016", "text": ",\n        \n          x\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle d(x,x')}\n  \n measures how much \n  \n    \n      \n        \n          x\n          \u2032\n        \n      \n    \n    {\\displaystyle x'}\n  \n differs from \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n.\nWith those, we can define the loss function for the autoencoder as\n  \n    \n      \n        L\n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n        :=", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0017", "text": ":=\n        \n          \n            \n              E\n            \n          \n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                r\n                e\n                f\n              \n            \n          \n        \n        [\n        d\n        (\n        x\n        ,\n        \n          D\n          \n            \u03b8\n          \n        \n        (\n        \n          E\n          \n            \u03d5\n          \n        \n        (\n        x\n        )", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0018", "text": "(\n        x\n        )\n        )\n        )\n        ]\n      \n    \n    {\\displaystyle L(\\theta ,\\phi ):=\\mathbb {\\mathbb {E} } _{x\\sim \\mu _{ref}}[d(x,D_{\\theta }(E_{\\phi }(x)))]}\n  \nThe optimal autoencoder for the given task \n  \n    \n      \n        (\n        \n          \u03bc\n          \n            r\n            e\n            f\n          \n        \n        ,\n        d\n        )\n      \n    \n    {\\displaystyle (\\mu _{ref},d)}\n  \n is then \n  \n    \n      \n        arg\n        \u2061", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0019", "text": "is then \n  \n    \n      \n        arg\n        \u2061\n        \n          min\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        L\n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n      \n    \n    {\\displaystyle \\arg \\min _{\\theta ,\\phi }L(\\theta ,\\phi )}\n  \n. The search for the optimal autoencoder can be accomplished by any mathematical optimization technique, but usually by gradient descent. This search process is referred to as \"training the autoencoder\".", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0020", "text": "In most situations, the reference distribution is just the empirical distribution given by a dataset \n  \n    \n      \n        {\n        \n          x\n          \n            1\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          x\n          \n            N\n          \n        \n        }\n        \u2282\n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle \\{x_{1},...,x_{N}\\}\\subset {\\mathcal {X}}}\n  \n, so that", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0021", "text": ", so that\n  \n    \n      \n        \n          \u03bc\n          \n            r\n            e\n            f\n          \n        \n        =\n        \n          \n            1\n            N\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            N\n          \n        \n        \n          \u03b4\n          \n            \n              x\n              \n                i", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0022", "text": "{\\displaystyle \\mu _{ref}={\\frac {1}{N}}\\sum _{i=1}^{N}\\delta _{x_{i}}}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0023", "text": "where \n  \n    \n      \n        \n          \u03b4\n          \n            \n              x\n              \n                i\n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\delta _{x_{i}}}\n  \n is the Dirac measure, the quality function is just L2 loss: \n  \n    \n      \n        d\n        (\n        x\n        ,\n        \n          x\n          \u2032\n        \n        )\n        =\n        \u2016\n        x\n        \u2212\n        \n          x\n          \u2032\n        \n        \n          \u2016", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0024", "text": "\u2016\n          \n            2\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle d(x,x')=\\|x-x'\\|_{2}^{2}}\n  \n, and \n  \n    \n      \n        \u2016\n        \u22c5\n        \n          \u2016\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\|\\cdot \\|_{2}}\n  \n is the Euclidean norm. Then the problem of searching for the optimal autoencoder is just a least-squares optimization:\n  \n    \n      \n        \n          min", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0025", "text": "min\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        L\n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n        ,\n        \n        \n          where \n        \n        L\n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n        =\n        \n          \n            1\n            N\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            N", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0026", "text": "N\n          \n        \n        \u2016\n        \n          x\n          \n            i\n          \n        \n        \u2212\n        \n          D\n          \n            \u03b8\n          \n        \n        (\n        \n          E\n          \n            \u03d5\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        )\n        \n          \u2016\n          \n            2\n          \n          \n            2", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0027", "text": "2\n          \n        \n      \n    \n    {\\displaystyle \\min _{\\theta ,\\phi }L(\\theta ,\\phi ),\\qquad {\\text{where }}L(\\theta ,\\phi )={\\frac {1}{N}}\\sum _{i=1}^{N}\\|x_{i}-D_{\\theta }(E_{\\phi }(x_{i}))\\|_{2}^{2}}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0028", "text": "Interpretation\nAn autoencoder has two main parts: an encoder that maps the message to a code, and a decoder that reconstructs the message from the code. An optimal autoencoder would perform as close to perfect reconstruction as possible, with \"close to perfect\" defined by the reconstruction quality function \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0029", "text": "d\n      \n    \n    {\\displaystyle d}\n  \n.\nThe simplest way to perform the copying task perfectly would be to duplicate the signal. To suppress this behavior, the code space \n  \n    \n      \n        \n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {Z}}}\n  \n usually has fewer dimensions than the message space \n  \n    \n      \n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {X}}}\n  \n.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0030", "text": "{\\displaystyle {\\mathcal {X}}}\n  \n.\nSuch an autoencoder is called undercomplete. It can be interpreted as compressing the message, or reducing its dimensionality.\nAt the limit of an ideal undercomplete autoencoder, every possible code \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n in the code space is used to encode a message \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n that really appears in the distribution \n  \n    \n      \n        \n          \u03bc", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0031", "text": "\u03bc\n          \n            r\n            e\n            f\n          \n        \n      \n    \n    {\\displaystyle \\mu _{ref}}\n  \n, and the decoder is also perfect: \n  \n    \n      \n        \n          D\n          \n            \u03b8\n          \n        \n        (\n        \n          E\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        )\n        =\n        x\n      \n    \n    {\\displaystyle D_{\\theta }(E_{\\phi }(x))=x}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0032", "text": ". This ideal autoencoder can then be used to generate messages indistinguishable from real messages, by feeding its decoder arbitrary code \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n and obtaining \n  \n    \n      \n        \n          D\n          \n            \u03b8\n          \n        \n        (\n        z\n        )\n      \n    \n    {\\displaystyle D_{\\theta }(z)}\n  \n, which is a message that really appears in the distribution \n  \n    \n      \n        \n          \u03bc", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0033", "text": "\u03bc\n          \n            r\n            e\n            f\n          \n        \n      \n    \n    {\\displaystyle \\mu _{ref}}\n  \n.\nIf the code space \n  \n    \n      \n        \n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {Z}}}\n  \n has dimension larger than (overcomplete), or equal to, the message space \n  \n    \n      \n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {X}}}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0034", "text": "{\\displaystyle {\\mathcal {X}}}\n  \n, or the hidden units are given enough capacity, an autoencoder can learn the identity function and become useless. However, experimental results found that overcomplete autoencoders might still learn useful features.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0035", "text": "In the ideal setting, the code dimension and the model capacity could be set on the basis of the complexity of the data distribution to be modeled. A standard way to do so is to add modifications to the basic autoencoder, to be detailed below.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0036", "text": "Variations\nVariational autoencoder (VAE)\nVariational autoencoders (VAEs) belong to the families of variational Bayesian methods. Despite the architectural similarities with basic autoencoders, VAEs are architected with different goals and have a different mathematical formulation. The latent space is, in this case, composed of a mixture of distributions instead of fixed vectors.\nGiven an input dataset \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0037", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n characterized by an unknown probability function \n  \n    \n      \n        P\n        (\n        x\n        )\n      \n    \n    {\\displaystyle P(x)}\n  \n and a multivariate latent encoding vector \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n, the objective is to model the data as a distribution \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        x\n        )", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0038", "text": "(\n        x\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(x)}\n  \n, with \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n defined as the set of the network parameters so that \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        x\n        )\n        =\n        \n          \u222b\n          \n            z\n          \n        \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        x\n        ,", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0039", "text": "(\n        x\n        ,\n        z\n        )\n        d\n        z\n      \n    \n    {\\displaystyle p_{\\theta }(x)=\\int _{z}p_{\\theta }(x,z)dz}\n  \n.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0040", "text": "Sparse autoencoder (SAE)\nInspired by the sparse coding hypothesis in neuroscience, sparse autoencoders (SAE) are variants of autoencoders, such that the codes \n  \n    \n      \n        \n          E\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle E_{\\phi }(x)}\n  \n for messages tend to be sparse codes, that is, \n  \n    \n      \n        \n          E\n          \n            \u03d5\n          \n        \n        (\n        x\n        )", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0041", "text": "(\n        x\n        )\n      \n    \n    {\\displaystyle E_{\\phi }(x)}\n  \n is close to zero in most entries. Sparse autoencoders may include more (rather than fewer) hidden units than inputs, but only a small number of the hidden units are allowed to be active at the same time. Encouraging sparsity improves performance on classification tasks.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0042", "text": "There are two main ways to enforce sparsity. One way is to simply clamp all but the highest-k activations of the latent code to zero. This is the k-sparse autoencoder.\nThe k-sparse autoencoder inserts the following \"k-sparse function\" in the latent layer of a standard autoencoder:\n  \n    \n      \n        \n          f\n          \n            k\n          \n        \n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        .\n        .\n        .\n        ,", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0043", "text": ".\n        .\n        .\n        ,\n        \n          x\n          \n            n\n          \n        \n        )\n        =\n        (\n        \n          x\n          \n            1\n          \n        \n        \n          b\n          \n            1\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          x\n          \n            n\n          \n        \n        \n          b\n          \n            n\n          \n        \n        )", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0044", "text": ")\n      \n    \n    {\\displaystyle f_{k}(x_{1},...,x_{n})=(x_{1}b_{1},...,x_{n}b_{n})}\n  \nwhere \n  \n    \n      \n        \n          b\n          \n            i\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle b_{i}=1}\n  \n if \n  \n    \n      \n        \n          |\n        \n        \n          x\n          \n            i\n          \n        \n        \n          |\n        \n      \n    \n    {\\displaystyle |x_{i}|}\n  \n ranks in the top k, and 0 otherwise.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0045", "text": "ranks in the top k, and 0 otherwise.\nBackpropagating through \n  \n    \n      \n        \n          f\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle f_{k}}\n  \n is simple: set gradient to 0 for \n  \n    \n      \n        \n          b\n          \n            i\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle b_{i}=0}\n  \n entries, and keep gradient for \n  \n    \n      \n        \n          b\n          \n            i\n          \n        \n        =", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0046", "text": "i\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle b_{i}=1}\n  \n entries. This is essentially a generalized ReLU function.\nThe other way is a relaxed version of the k-sparse autoencoder. Instead of forcing sparsity, we add a sparsity regularization loss, then optimize for\n  \n    \n      \n        \n          min\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        L\n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n        +\n        \u03bb", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0047", "text": ",\n        \u03d5\n        )\n        +\n        \u03bb\n        \n          L\n          \n            sparse\n          \n        \n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n      \n    \n    {\\displaystyle \\min _{\\theta ,\\phi }L(\\theta ,\\phi )+\\lambda L_{\\text{sparse}}(\\theta ,\\phi )}\n  \nwhere \n  \n    \n      \n        \u03bb\n        >\n        0\n      \n    \n    {\\displaystyle \\lambda >0}\n  \n measures how much sparsity we want to enforce.\nLet the autoencoder architecture have \n  \n    \n      \n        K", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0048", "text": "K\n      \n    \n    {\\displaystyle K}\n  \n layers. To define a sparsity regularization loss, we need a \"desired\" sparsity \n  \n    \n      \n        \n          \n            \n              \n                \u03c1\n                ^\n              \n            \n          \n          \n            k\n          \n        \n      \n    \n    {\\displaystyle {\\hat {\\rho }}_{k}}\n  \n for each layer, a weight \n  \n    \n      \n        \n          w\n          \n            k", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0049", "text": "k\n          \n        \n      \n    \n    {\\displaystyle w_{k}}\n  \n for how much to enforce each sparsity, and a function \n  \n    \n      \n        s\n        :\n        [\n        0\n        ,\n        1\n        ]\n        \u00d7\n        [\n        0\n        ,\n        1\n        ]\n        \u2192\n        [\n        0\n        ,\n        \u221e\n        ]\n      \n    \n    {\\displaystyle s:[0,1]\\times [0,1]\\to [0,\\infty ]}\n  \n to measure how much two sparsities differ.\nFor each input \n  \n    \n      \n        x", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0050", "text": "For each input \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n, let the actual sparsity of activation in each layer \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n be\n  \n    \n      \n        \n          \u03c1\n          \n            k\n          \n        \n        (\n        x\n        )\n        =\n        \n          \n            1\n            n\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0051", "text": "1\n          \n          \n            n\n          \n        \n        \n          a\n          \n            k\n            ,\n            i\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle \\rho _{k}(x)={\\frac {1}{n}}\\sum _{i=1}^{n}a_{k,i}(x)}\n  \nwhere \n  \n    \n      \n        \n          a\n          \n            k\n            ,\n            i\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle a_{k,i}(x)}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0052", "text": "{\\displaystyle a_{k,i}(x)}\n  \n is the activation in the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n -th neuron of the \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n -th layer upon input \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n.\nThe sparsity loss upon input \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n for one layer is \n  \n    \n      \n        s\n        (", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0053", "text": "\u03c1\n                ^\n              \n            \n          \n          \n            k\n          \n        \n        ,\n        \n          \u03c1\n          \n            k\n          \n        \n        (\n        x\n        )\n        )\n      \n    \n    {\\displaystyle s({\\hat {\\rho }}_{k},\\rho _{k}(x))}\n  \n, and the sparsity regularization loss for the entire autoencoder is the expected weighted sum of sparsity losses:", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0054", "text": "L\n          \n            sparse\n          \n        \n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n        =\n        \n          \n            \n              E\n            \n          \n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                X\n              \n            \n          \n        \n        \n          [\n          \n            \n              \u2211\n              \n                k\n                \u2208\n                1", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0055", "text": "\u2208\n                1\n                :\n                K\n              \n            \n            \n              w\n              \n                k\n              \n            \n            s\n            (\n            \n              \n                \n                  \n                    \u03c1\n                    ^\n                  \n                \n              \n              \n                k\n              \n            \n            ,\n            \n              \u03c1", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0056", "text": "\u03c1\n              \n                k\n              \n            \n            (\n            x\n            )\n            )\n          \n          ]\n        \n      \n    \n    {\\displaystyle L_{\\text{sparse}}(\\theta ,\\phi )=\\mathbb {\\mathbb {E} } _{x\\sim \\mu _{X}}\\left[\\sum _{k\\in 1:K}w_{k}s({\\hat {\\rho }}_{k},\\rho _{k}(x))\\right]}\n  \nTypically, the function \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n is either the Kullback\u2013Leibler (KL) divergence, as", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0057", "text": "s\n        (\n        \u03c1\n        ,\n        \n          \n            \n              \u03c1\n              ^\n            \n          \n        \n        )\n        =\n        K\n        L\n        (\n        \u03c1\n        \n          |\n        \n        \n          |\n        \n        \n          \n            \n              \u03c1\n              ^\n            \n          \n        \n        )\n        =\n        \u03c1\n        log\n        \u2061\n        \n          \n            \u03c1", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0058", "text": "\u03c1\n            \n              \n                \u03c1\n                ^\n              \n            \n          \n        \n        +\n        (\n        1\n        \u2212\n        \u03c1\n        )\n        log\n        \u2061\n        \n          \n            \n              1\n              \u2212\n              \u03c1\n            \n            \n              1\n              \u2212\n              \n                \n                  \n                    \u03c1\n                    ^", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0059", "text": "{\\displaystyle s(\\rho ,{\\hat {\\rho }})=KL(\\rho ||{\\hat {\\rho }})=\\rho \\log {\\frac {\\rho }{\\hat {\\rho }}}+(1-\\rho )\\log {\\frac {1-\\rho }{1-{\\hat {\\rho }}}}}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0060", "text": "or the L1 loss, as \n  \n    \n      \n        s\n        (\n        \u03c1\n        ,\n        \n          \n            \n              \u03c1\n              ^\n            \n          \n        \n        )\n        =\n        \n          |\n        \n        \u03c1\n        \u2212\n        \n          \n            \n              \u03c1\n              ^\n            \n          \n        \n        \n          |\n        \n      \n    \n    {\\displaystyle s(\\rho ,{\\hat {\\rho }})=|\\rho -{\\hat {\\rho }}|}\n  \n, or the L2 loss, as", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0061", "text": ", or the L2 loss, as \n  \n    \n      \n        s\n        (\n        \u03c1\n        ,\n        \n          \n            \n              \u03c1\n              ^\n            \n          \n        \n        )\n        =\n        \n          |\n        \n        \u03c1\n        \u2212\n        \n          \n            \n              \u03c1\n              ^\n            \n          \n        \n        \n          \n            |\n          \n          \n            2", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0062", "text": "2\n          \n        \n      \n    \n    {\\displaystyle s(\\rho ,{\\hat {\\rho }})=|\\rho -{\\hat {\\rho }}|^{2}}\n  \n.\nAlternatively, the sparsity regularization loss may be defined without reference to any \"desired sparsity\", but simply force as much sparsity as possible. In this case, one can define the sparsity regularization loss as \n  \n    \n      \n        \n          L\n          \n            sparse\n          \n        \n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n        =", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0063", "text": ",\n        \u03d5\n        )\n        =\n        \n          \n            \n              E\n            \n          \n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                X\n              \n            \n          \n        \n        \n          [\n          \n            \n              \u2211\n              \n                k\n                \u2208\n                1\n                :\n                K\n              \n            \n            \n              w", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0064", "text": "w\n              \n                k\n              \n            \n            \u2016\n            \n              h\n              \n                k\n              \n            \n            \u2016\n          \n          ]\n        \n      \n    \n    {\\displaystyle L_{\\text{sparse}}(\\theta ,\\phi )=\\mathbb {\\mathbb {E} } _{x\\sim \\mu _{X}}\\left[\\sum _{k\\in 1:K}w_{k}\\|h_{k}\\|\\right]}\n  \nwhere \n  \n    \n      \n        \n          h\n          \n            k", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0065", "text": "k\n          \n        \n      \n    \n    {\\displaystyle h_{k}}\n  \n is the activation vector in the \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n-th layer of the autoencoder. The norm \n  \n    \n      \n        \u2016\n        \u22c5\n        \u2016\n      \n    \n    {\\displaystyle \\|\\cdot \\|}\n  \n is usually the L1 norm (giving the L1 sparse autoencoder) or the L2 norm (giving the L2 sparse autoencoder).", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0066", "text": "Denoising autoencoder (DAE)\nDenoising autoencoders (DAE) try to achieve a good representation by changing the reconstruction criterion.\nA DAE, originally called a \"robust autoassociative network\" by Mark A. Kramer, is trained by intentionally corrupting the inputs of a standard autoencoder during training. A noise process is defined by a probability distribution \n  \n    \n      \n        \n          \u03bc\n          \n            T\n          \n        \n      \n    \n    {\\displaystyle \\mu _{T}}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0067", "text": "{\\displaystyle \\mu _{T}}\n  \n over functions \n  \n    \n      \n        T\n        :\n        \n          \n            X\n          \n        \n        \u2192\n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle T:{\\mathcal {X}}\\to {\\mathcal {X}}}\n  \n. That is, the function \n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}\n  \n takes a message \n  \n    \n      \n        x\n        \u2208\n        \n          \n            X", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0068", "text": "X\n          \n        \n      \n    \n    {\\displaystyle x\\in {\\mathcal {X}}}\n  \n, and corrupts it to a noisy version \n  \n    \n      \n        T\n        (\n        x\n        )\n      \n    \n    {\\displaystyle T(x)}\n  \n. The function \n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}\n  \n is selected randomly, with a probability distribution \n  \n    \n      \n        \n          \u03bc\n          \n            T\n          \n        \n      \n    \n    {\\displaystyle \\mu _{T}}\n  \n.\nGiven a task", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0069", "text": ".\nGiven a task \n  \n    \n      \n        (\n        \n          \u03bc\n          \n            ref\n          \n        \n        ,\n        d\n        )\n      \n    \n    {\\displaystyle (\\mu _{\\text{ref}},d)}\n  \n, the problem of training a DAE is the optimization problem:\n  \n    \n      \n        \n          min\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        L\n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n        =\n        \n          \n            \n              E", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0070", "text": "E\n            \n          \n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                X\n              \n            \n            ,\n            T\n            \u223c\n            \n              \u03bc\n              \n                T\n              \n            \n          \n        \n        [\n        d\n        (\n        x\n        ,\n        (\n        \n          D\n          \n            \u03b8\n          \n        \n        \u2218", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0071", "text": "\u2218\n        \n          E\n          \n            \u03d5\n          \n        \n        \u2218\n        T\n        )\n        (\n        x\n        )\n        )\n        ]\n      \n    \n    {\\displaystyle \\min _{\\theta ,\\phi }L(\\theta ,\\phi )=\\mathbb {\\mathbb {E} } _{x\\sim \\mu _{X},T\\sim \\mu _{T}}[d(x,(D_{\\theta }\\circ E_{\\phi }\\circ T)(x))]}\n  \nThat is, the optimal DAE should take any noisy message and attempt to recover the original message without noise, thus the name \"denoising\".", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0072", "text": "Usually, the noise process \n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}\n  \n is applied only during training and testing, not during downstream use.\nThe use of DAE depends on two assumptions:", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0073", "text": "There exist representations to the messages that are relatively stable and robust to the type of noise we are likely to encounter;\nThe said representations capture structures in the input distribution that are useful for our purposes.\nExample noise processes include:\n\nadditive isotropic Gaussian noise,\nmasking noise (a fraction of the input is randomly chosen and set to 0)\nsalt-and-pepper noise (a fraction of the input is randomly chosen and randomly set to its minimum or maximum value).", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0074", "text": "Contractive autoencoder (CAE)\nA contractive autoencoder (CAE) adds the contractive regularization loss to the standard autoencoder loss:\n  \n    \n      \n        \n          min\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        L\n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n        +\n        \u03bb\n        \n          L\n          \n            cont\n          \n        \n        (\n        \u03b8\n        ,\n        \u03d5\n        )", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0075", "text": ",\n        \u03d5\n        )\n      \n    \n    {\\displaystyle \\min _{\\theta ,\\phi }L(\\theta ,\\phi )+\\lambda L_{\\text{cont}}(\\theta ,\\phi )}\n  \nwhere \n  \n    \n      \n        \u03bb\n        >\n        0\n      \n    \n    {\\displaystyle \\lambda >0}\n  \n measures how much contractive-ness we want to enforce. The contractive regularization loss itself is defined as the expected square of Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input:", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0076", "text": "L\n          \n            cont\n          \n        \n        (\n        \u03b8\n        ,\n        \u03d5\n        )\n        =\n        \n          \n            E\n          \n          \n            x\n            \u223c\n            \n              \u03bc\n              \n                r\n                e\n                f\n              \n            \n          \n        \n        \u2016\n        \n          \u2207\n          \n            x\n          \n        \n        \n          E\n          \n            \u03d5", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0077", "text": "E\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        \n          \u2016\n          \n            F\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle L_{\\text{cont}}(\\theta ,\\phi )=\\mathbb {E} _{x\\sim \\mu _{ref}}\\|\\nabla _{x}E_{\\phi }(x)\\|_{F}^{2}}\n  \nTo understand what \n  \n    \n      \n        \n          L\n          \n            cont\n          \n        \n      \n    \n    {\\displaystyle L_{\\text{cont}}}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0078", "text": "{\\displaystyle L_{\\text{cont}}}\n  \n measures, note the fact\n  \n    \n      \n        \u2016\n        \n          E\n          \n            \u03d5\n          \n        \n        (\n        x\n        +\n        \u03b4\n        x\n        )\n        \u2212\n        \n          E\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        \n          \u2016\n          \n            2\n          \n        \n        \u2264\n        \u2016\n        \n          \u2207\n          \n            x\n          \n        \n        \n          E", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0079", "text": "E\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        \n          \u2016\n          \n            F\n          \n        \n        \u2016\n        \u03b4\n        x\n        \n          \u2016\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\|E_{\\phi }(x+\\delta x)-E_{\\phi }(x)\\|_{2}\\leq \\|\\nabla _{x}E_{\\phi }(x)\\|_{F}\\|\\delta x\\|_{2}}\n  \nfor any message \n  \n    \n      \n        x\n        \u2208\n        \n          \n            X", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0080", "text": "\u2208\n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle x\\in {\\mathcal {X}}}\n  \n, and small variation \n  \n    \n      \n        \u03b4\n        x\n      \n    \n    {\\displaystyle \\delta x}\n  \n in it. Thus, if \n  \n    \n      \n        \u2016\n        \n          \u2207\n          \n            x\n          \n        \n        \n          E\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        \n          \u2016\n          \n            F", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0081", "text": "F\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\|\\nabla _{x}E_{\\phi }(x)\\|_{F}^{2}}\n  \n is small, it means that a small neighborhood of the message maps to a small neighborhood of its code. This is a desired property, as it means small variation in the message leads to small, perhaps even zero, variation in its code, like how two pictures may look the same even if they are not exactly the same.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0082", "text": "The DAE can be understood as an infinitesimal limit of CAE: in the limit of small Gaussian input noise, DAEs make the reconstruction function resist small but finite-sized input perturbations, while CAEs make the extracted features resist infinitesimal input perturbations.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0083", "text": "Minimum description length autoencoder (MDL-AE)", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0084", "text": "A minimum description length autoencoder (MDL-AE) is an advanced variation of the traditional autoencoder, which leverages principles from information theory, specifically the Minimum Description Length (MDL) principle. The MDL principle posits that the best model for a dataset is the one that provides the shortest combined encoding of the model and the data", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0085", "text": ". In the context of autoencoders, this principle is applied to ensure that the learned representation is not only compact but also interpretable and efficient for reconstruction.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0086", "text": "The MDL-AE seeks to minimize the total description length of the data, which includes the size of the latent representation (code length) and the error in reconstructing the original data. The objective can be expressed as", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0087", "text": "L\n          \n            code\n          \n        \n        +\n        \n          L\n          \n            error\n          \n        \n      \n    \n    {\\displaystyle L_{\\text{code}}+L_{\\text{error}}}\n  \n, where \n  \n    \n      \n        \n          L\n          \n            code\n          \n        \n      \n    \n    {\\displaystyle L_{\\text{code}}}\n  \n represents the length of the compressed latent representation and \n  \n    \n      \n        \n          L", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0088", "text": "L\n          \n            error\n          \n        \n      \n    \n    {\\displaystyle L_{\\text{error}}}\n  \n denotes the reconstruction error.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0089", "text": "Concrete autoencoder (CAE)\nThe concrete autoencoder is designed for discrete feature selection. A concrete autoencoder forces the latent space to consist only of a user-specified number of features. The concrete autoencoder uses a continuous relaxation of the categorical distribution to allow gradients to pass through the feature selector layer, which makes it possible to use standard backpropagation to learn an optimal subset of input features that minimize reconstruction loss.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0090", "text": "Advantages of depth\nAutoencoders are often trained with a single-layer encoder and a single-layer decoder, but using many-layered (deep) encoders and decoders offers many advantages.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0091", "text": "Depth can exponentially reduce the computational cost of representing some functions.\nDepth can exponentially decrease the amount of training data needed to learn some functions.\nExperimentally, deep autoencoders yield better compression compared to shallow or linear autoencoders.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0092", "text": "Depth allows for advantages over traditional methods as one can show that after training single layer linear autoencoders have a latent space whose vectors span the same subspace as the eigenvectors found in Principal component analysis.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0093", "text": "Training\nGeoffrey Hinton developed the deep belief network technique for training many-layered deep autoencoders. His method involves treating each neighboring set of two layers as a restricted Boltzmann machine so that pretraining approximates a good solution, then using backpropagation to fine-tune the results.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0094", "text": "Researchers have debated whether joint training (i.e. training the whole architecture together with a single global reconstruction objective to optimize) would be better for deep auto-encoders. A 2015 study showed that joint training learns better data models along with more representative features for classification as compared to the layerwise method. However, their experiments showed that the success of joint training depends heavily on the regularization strategies adopted.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0095", "text": "History\n(Oja, 1982) noted that PCA is equivalent to a neural network with one hidden layer with identity activation function. In the language of autoencoding, the input-to-hidden module is the encoder, and the hidden-to-output module is the decoder. Subsequently, in (Baldi and Hornik, 1989) and (Kramer, 1991) generalized PCA to autoencoders, a technique which they termed \"nonlinear PCA\".", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0096", "text": "Immediately after the resurgence of neural networks in the 1980s, it was suggested in 1986 that a neural network be put in \"auto-association mode\". This was then implemented in (Harrison, 1987) and (Elman, Zipser, 1988) for speech and in (Cottrell, Munro, Zipser, 1987) for images. In (Hinton, Salakhutdinov, 2006), deep belief networks were developed", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0097", "text": ". These train a pair restricted Boltzmann machines as encoder-decoder pairs, then train another pair on the latent representation of the first pair, and so on.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0098", "text": "The first applications of AE date to early 1990s. Their most traditional application was dimensionality reduction or feature learning, but the concept became widely used for learning generative models of data. Some of the most powerful AIs in the 2010s involved autoencoder modules as a component of larger AI systems, such as VAE in Stable Diffusion, discrete VAE in Transformer-based image generators like DALL-E 1, etc.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0099", "text": "During the early days, when the terminology was uncertain, the autoencoder has also been called identity mapping, auto-associating, self-supervised backpropagation, or Diabolo network.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0100", "text": "Applications\nThe two main applications of autoencoders are dimensionality reduction and information retrieval (or associative memory), but modern variations have been applied to other tasks.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0101", "text": "Dimensionality reduction\nDimensionality reduction was one of the first deep learning applications.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0102", "text": "For Hinton's 2006 study, he pretrained a multi-layer autoencoder with a stack of RBMs and then used their weights to initialize a deep autoencoder with gradually smaller hidden layers until hitting a bottleneck of 30 neurons. The resulting 30 dimensions of the code yielded a smaller reconstruction error compared to the first 30 components of a principal component analysis (PCA), and learned a representation that was qualitatively easier to interpret, clearly separating data clusters.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0103", "text": "Reducing dimensions can improve performance on tasks such as classification. Indeed, the hallmark of dimensionality reduction is to place semantically related examples near each other.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0104", "text": "Principal component analysis\nIf linear activations are used, or only a single sigmoid hidden layer, then the optimal solution to an autoencoder is strongly related to principal component analysis (PCA). The weights of an autoencoder with a single hidden layer of size \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n (where \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0105", "text": "p\n      \n    \n    {\\displaystyle p}\n  \n is less than the size of the input) span the same vector subspace as the one spanned by the first \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0106", "text": "p\n      \n    \n    {\\displaystyle p}\n  \n principal components, and the output of the autoencoder is an orthogonal projection onto this subspace. The autoencoder weights are not equal to the principal components, and are generally not orthogonal, yet the principal components may be recovered from them using the singular value decomposition.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0107", "text": "However, the potential of autoencoders resides in their non-linearity, allowing the model to learn more powerful generalizations compared to PCA, and to reconstruct the input with significantly lower information loss.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0108", "text": "Information retrieval", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0109", "text": "Information retrieval benefits particularly from dimensionality reduction in that search can become more efficient in certain kinds of low dimensional spaces. Autoencoders were indeed applied to semantic hashing, proposed by Salakhutdinov and Hinton in 2007. By training the algorithm to produce a low-dimensional binary code, all database entries could be stored in a hash table mapping binary code vectors to entries", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0110", "text": ". This table would then support information retrieval by returning all entries with the same binary code as the query, or slightly less similar entries by flipping some bits from the query encoding.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0111", "text": "Autoencoder hippocampus network", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0112", "text": "An autoencoder hippocampus network is a neural network that combines the principles of an autoencoder with a model of the hippocampus's memory functions to perform tasks like skill acquisition and recall in artificial systems. An autoencoder compresses policy function parameters into a compressed \"skill vector\" and then reconstructs the original data, mirroring the hippocampus's ability to encode and recall information", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0113", "text": ". This framework can be used to learn, store, and retrieve different skills or memories, such as policy function parameters for a robot, by creating a latent space representation of these skills.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0114", "text": "Anomaly detection", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0115", "text": "Another application for autoencoders is anomaly detection. By learning to replicate the most salient features in the training data under some of the constraints described previously, the model is encouraged to learn to precisely reproduce the most frequently observed characteristics. When facing anomalies, the model should worsen its reconstruction performance", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0116", "text": ". In most cases, only data with normal instances are used to train the autoencoder; in others, the frequency of anomalies is small compared to the observation set so that its contribution to the learned representation could be ignored. After training, the autoencoder will accurately reconstruct \"normal\" data, while failing to do so with unfamiliar anomalous data", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0117", "text": ". Reconstruction error (the error between the original data and its low dimensional reconstruction) is used as an anomaly score to detect anomalies.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0118", "text": "Typically, this means that on a validation set the empirical distribution of reconstruction errors is recorded and then (e.g.) the empirical 95-percentile \n  \n    \n      \n        \n          x\n          \n            p\n          \n        \n      \n    \n    {\\displaystyle x_{p}}\n  \n is taken as threshold \n  \n    \n      \n        t\n        :=\n        \n          x\n          \n            p\n          \n        \n      \n    \n    {\\displaystyle t:=x_{p}}\n  \n to flag anomalous data points:", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0119", "text": "to flag anomalous data points: \n  \n    \n      \n        \n          loss\n        \n        (\n        x\n        ,\n        \n          reconstruction\n        \n        (\n        x\n        )\n        )\n        >\n        t\n        \n        \u27f9\n        \n        \n          anomaly\n        \n      \n    \n    {\\displaystyle {\\text{loss}}(x,{\\text{reconstruction}}(x))>t\\implies {\\text{anomaly}}}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0120", "text": ". Since the threshold is an empirical quantile estimate, there is an inherent difficulty with \"correctly\" setting this threshold:\nIn many cases the distribution of the empirical quantile is asymptotically a normal distribution \n  \n    \n      \n        \n          empirical p-quantile\n        \n        \u223c\n        \n          \n            N\n          \n        \n        \n          (\n          \n            \u03bc\n            =\n            p\n            ,\n            \n              \u03c3", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0121", "text": "\u03c3\n              \n                2\n              \n            \n            =\n            \n              \n                \n                  p\n                  (\n                  1\n                  \u2212\n                  p\n                  )\n                \n                \n                  n\n                  f\n                  (\n                  \n                    x\n                    \n                      p", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0122", "text": ")\n                    \n                      2\n                    \n                  \n                \n              \n            \n          \n          )\n        \n        ,\n      \n    \n    {\\displaystyle {\\text{empirical p-quantile}}\\sim {\\mathcal {N}}\\left(\\mu =p,\\sigma ^{2}={\\frac {p(1-p)}{nf(x_{p})^{2}}}\\right),}\n  \n with \n  \n    \n      \n        f\n        (\n        \n          x\n          \n            p", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0123", "text": "x\n          \n            p\n          \n        \n        )\n      \n    \n    {\\displaystyle f(x_{p})}\n  \n the probability density at the quantile. This means that the variance grows if an extreme quantile is considered (because \n  \n    \n      \n        f\n        (\n        \n          x\n          \n            p\n          \n        \n        )\n      \n    \n    {\\displaystyle f(x_{p})}", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0124", "text": "{\\displaystyle f(x_{p})}\n  \n is small there). This means that there is a, potentially, a big uncertainty in what is the right choice for the threshold since it is estimated from a validation set.\nRecent literature has however shown that certain autoencoding models can, counterintuitively, be very good at reconstructing anomalous examples and consequently not able to reliably perform anomaly detection.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0125", "text": "Intuitively, this can be understood by considering those one layer auto encoders which are related to PCA - also in this case there can be perfect rein reconstructions for points which are far away from the data region but which lie on a principal component axis.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0126", "text": "It is best to analyze if the anomalies which are flagged by the auto encoder are true anomalies. In this sense all the metrics in Evaluation of binary classifiers can be considered", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0127", "text": ". The fundamental challenge which comes with the unsupervised (self-supervised) learning setting is, that labels for rare events do not exist (in which case the labels first have to be gathered and the data set will be imbalanced) or anomaly indicating labels are very rare, introducing larger confidence intervals for these performance estimates.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0128", "text": "Image processing\nThe characteristics of autoencoders are useful in image processing.\nOne example can be found in lossy image compression, where autoencoders outperformed other approaches and proved competitive against JPEG 2000.\nAnother useful application of autoencoders in image preprocessing is image denoising.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0129", "text": "Autoencoders found use in more demanding contexts such as medical imaging where they have been used for image denoising as well as super-resolution. In image-assisted diagnosis, experiments have applied autoencoders for breast cancer detection and for modelling the relation between the cognitive decline of Alzheimer's disease and the latent features of an autoencoder trained with MRI.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0130", "text": "Drug discovery\nIn 2019 molecules generated with variational autoencoders were validated experimentally in mice.\n\nPopularity prediction\nRecently, a stacked autoencoder framework produced promising results in predicting popularity of social media posts, which is helpful for online advertising strategies.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0131", "text": "Machine translation", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0132", "text": "Autoencoders have been applied to machine translation, which is usually referred to as neural machine translation (NMT). Unlike traditional autoencoders, the output does not match the input - it is in another language. In NMT, texts are treated as sequences to be encoded into the learning procedure, while on the decoder side sequences in the target language(s) are generated", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0133", "text": ". Language-specific autoencoders incorporate further linguistic features into the learning procedure, such as Chinese decomposition features. Machine translation is rarely still done with autoencoders, due to the availability of more effective transformer networks.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0134", "text": "Communication Systems", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0135", "text": "Autoencoders in communication systems are important because they help in encoding data into a more resilient representation for channel impairments, which is crucial for transmitting information while minimizing errors. In Addition, AE-based systems can optimize end-to-end communication performance. This approach can solve the several limitations of designing communication systems such as the inherent difficulty in accurately modeling the complex behavior of real-world channels.", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0136", "text": "See also\nRepresentation learning\nSingular value decomposition\nSparse dictionary learning\nDeep learning", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Autoencoder_0137", "text": "Further reading\nBank, Dor; Koenigstein, Noam; Giryes, Raja (2023). \"Autoencoders\". Machine Learning for Data Science Handbook. Cham: Springer International Publishing. doi:10.1007/978-3-031-24628-9_16. ISBN 978-3-031-24627-2.\nGoodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). \"14. Autoencoders\". Deep learning. Adaptive computation and machine learning. Cambridge, Mass: The MIT press. ISBN 978-0-262-03561-3.\n\n\n== References ==", "title": "Autoencoder", "url": "https://en.wikipedia.org/wiki/Autoencoder"}, {"id": "rec_Variational_autoencoder_0000", "text": "In machine learning, a variational autoencoder (VAE) is an artificial neural network architecture introduced by Diederik P. Kingma and Max Welling in 2013. It is part of the families of probabilistic graphical models and variational Bayesian methods.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0001", "text": "In addition to being seen as an autoencoder neural network architecture, variational autoencoders can also be studied within the mathematical formulation of variational Bayesian methods, connecting a neural encoder network to its decoder through a probabilistic latent space (for example, as a multivariate Gaussian distribution) that corresponds to the parameters of a variational distribution.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0002", "text": "Thus, the encoder maps each point (such as an image) from a large complex dataset into a distribution within the latent space, rather than to a single point in that space. The decoder has the opposite function, which is to map from the latent space to the input space, again according to a distribution (although in practice, noise is rarely added during the decoding stage). By mapping a point to a distribution instead of a single point, the network can avoid overfitting the training data", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0003", "text": ". Both networks are typically trained together with the usage of the reparameterization trick, although the variance of the noise model can be learned separately.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0004", "text": "Although this type of model was initially designed for unsupervised learning, its effectiveness has been proven for semi-supervised learning and supervised learning.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0005", "text": "Overview of architecture and operation", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0006", "text": "A variational autoencoder is a generative model with a prior and noise distribution respectively. Usually such models are trained using the expectation-maximization meta-algorithm (e.g. probabilistic PCA, (spike & slab) sparse coding). Such a scheme optimizes a lower bound of the data likelihood, which is usually computationally intractable, and in doing so requires the discovery of q-distributions, or variational posteriors", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0007", "text": ". These q-distributions are normally parameterized for each individual data point in a separate optimization process. However, variational autoencoders use a neural network as an amortized approach to jointly optimize across data points. In that way, the same parameters are reused for multiple data points, which can result in massive memory savings. The first neural network takes as input the data points themselves, and outputs parameters for the variational distribution", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0008", "text": ". As it maps from a known input space to the low-dimensional latent space, it is called the encoder.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0009", "text": "The decoder is the second neural network of this model. It is a function that maps from the latent space to the input space, e.g. as the means of the noise distribution. It is possible to use another neural network that maps to the variance, however this can be omitted for simplicity. In such a case, the variance can be optimized with gradient descent.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0010", "text": "To optimize this model, one needs to know two terms: the \"reconstruction error\", and the Kullback\u2013Leibler divergence (KL-D). Both terms are derived from the free energy expression of the probabilistic model, and therefore differ depending on the noise distribution and the assumed prior of the data, here referred to as p-distribution", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0011", "text": ". For example, a standard VAE task such as IMAGENET is typically assumed to have a gaussianly distributed noise; however, tasks such as binarized MNIST require a Bernoulli noise. The KL-D from the free energy expression maximizes the probability mass of the q-distribution that overlaps with the p-distribution, which unfortunately can result in mode-seeking behaviour", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0012", "text": ". The \"reconstruction\" term is the remainder of the free energy expression, and requires a sampling approximation to compute its expectation value.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0013", "text": "More recent approaches replace Kullback\u2013Leibler divergence (KL-D) with various statistical distances, see \"Statistical distance VAE variants\" below.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0014", "text": "Formulation\nFrom the point of view of probabilistic modeling, one wants to maximize the likelihood of the data \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n by their chosen parameterized probability distribution \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        x\n        )\n        =\n        p\n        (\n        x\n        \n          |\n        \n        \u03b8\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(x)=p(x|\\theta )}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0015", "text": ". This distribution is usually chosen to be a Gaussian \n  \n    \n      \n        N\n        (\n        x\n        \n          |\n        \n        \u03bc\n        ,\n        \u03c3\n        )\n      \n    \n    {\\displaystyle N(x|\\mu ,\\sigma )}\n  \n which is parameterized by \n  \n    \n      \n        \u03bc\n      \n    \n    {\\displaystyle \\mu }\n  \n and \n  \n    \n      \n        \u03c3\n      \n    \n    {\\displaystyle \\sigma }", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0016", "text": "{\\displaystyle \\sigma }\n  \n respectively, and as a member of the exponential family it is easy to work with as a noise distribution. Simple distributions are easy enough to maximize, however distributions where a prior is assumed over the latents \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n results in intractable integrals. Let us find \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        x\n        )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0017", "text": "(\n        x\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(x)}\n  \n via marginalizing over \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0018", "text": "p\n          \n            \u03b8\n          \n        \n        (\n        x\n        )\n        =\n        \n          \u222b\n          \n            z\n          \n        \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        \n          x\n          ,\n          z\n        \n        )\n        \n        d\n        z\n        ,\n      \n    \n    {\\displaystyle p_{\\theta }(x)=\\int _{z}p_{\\theta }({x,z})\\,dz,}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0019", "text": "where \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        \n          x\n          ,\n          z\n        \n        )\n      \n    \n    {\\displaystyle p_{\\theta }({x,z})}\n  \n represents the joint distribution under \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n      \n    \n    {\\displaystyle p_{\\theta }}\n  \n of the observable data \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0020", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n and its latent representation or encoding \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n. According to the chain rule, the equation can be rewritten as", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0021", "text": "p\n          \n            \u03b8\n          \n        \n        (\n        x\n        )\n        =\n        \n          \u222b\n          \n            z\n          \n        \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        \n          x\n          \n            |\n          \n          z\n        \n        )\n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        z\n        )\n        \n        d\n        z", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0022", "text": "d\n        z\n      \n    \n    {\\displaystyle p_{\\theta }(x)=\\int _{z}p_{\\theta }({x|z})p_{\\theta }(z)\\,dz}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0023", "text": "In the vanilla variational autoencoder, \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n is usually taken to be a finite-dimensional vector of real numbers, and \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        \n          x\n          \n            |\n          \n          z\n        \n        )\n      \n    \n    {\\displaystyle p_{\\theta }({x|z})}\n  \n to be a Gaussian distribution. Then \n  \n    \n      \n        \n          p", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0024", "text": "p\n          \n            \u03b8\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(x)}\n  \n is a mixture of Gaussian distributions.\nIt is now possible to define the set of the relationships between the input data and its latent representation as", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0025", "text": "Prior \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        z\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(z)}\n  \n\nLikelihood \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        x\n        \n          |\n        \n        z\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(x|z)}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0026", "text": "Posterior \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        z\n        \n          |\n        \n        x\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(z|x)}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0027", "text": "Unfortunately, the computation of \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        z\n        \n          |\n        \n        x\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(z|x)}\n  \n is expensive and in most cases intractable. To speed up the calculus to make it feasible, it is necessary to introduce a further function to approximate the posterior distribution as", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0028", "text": "q\n          \n            \u03d5\n          \n        \n        (\n        \n          z\n          \n            |\n          \n          x\n        \n        )\n        \u2248\n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        \n          z\n          \n            |\n          \n          x\n        \n        )\n      \n    \n    {\\displaystyle q_{\\phi }({z|x})\\approx p_{\\theta }({z|x})}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0029", "text": "with \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n defined as the set of real values that parametrize \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n. This is sometimes called amortized inference, since by \"investing\" in finding a good \n  \n    \n      \n        \n          q\n          \n            \u03d5\n          \n        \n      \n    \n    {\\displaystyle q_{\\phi }}\n  \n, one can later infer \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n from", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0030", "text": "{\\displaystyle z}\n  \n from \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n quickly without doing any integrals.\nIn this way, the problem is to find a good probabilistic autoencoder, in which the conditional likelihood distribution \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        x\n        \n          |\n        \n        z\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(x|z)}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0031", "text": "{\\displaystyle p_{\\theta }(x|z)}\n  \n is computed by the probabilistic decoder, and the approximated posterior distribution \n  \n    \n      \n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        z\n        \n          |\n        \n        x\n        )\n      \n    \n    {\\displaystyle q_{\\phi }(z|x)}\n  \n is computed by the probabilistic encoder.\nParametrize the encoder as \n  \n    \n      \n        \n          E\n          \n            \u03d5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0032", "text": "\u03d5\n          \n        \n      \n    \n    {\\displaystyle E_{\\phi }}\n  \n, and the decoder as \n  \n    \n      \n        \n          D\n          \n            \u03b8\n          \n        \n      \n    \n    {\\displaystyle D_{\\theta }}\n  \n.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0033", "text": "Evidence lower bound (ELBO)\nLike many deep learning approaches that use gradient-based optimization, VAEs require a differentiable loss function to update the network weights through backpropagation.\nFor variational autoencoders, the idea is to jointly optimize the generative model parameters \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n to reduce the reconstruction error between the input and the output, and \n  \n    \n      \n        \u03d5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0034", "text": "\u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n to make \n  \n    \n      \n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        \n          z\n          \n            |\n          \n          x\n        \n        )\n      \n    \n    {\\displaystyle q_{\\phi }({z|x})}\n  \n as close as possible to \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        z\n        \n          |\n        \n        x\n        )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0035", "text": "x\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(z|x)}\n  \n. As reconstruction loss, mean squared error and cross entropy are often used.\nThe Kullback\u2013Leibler divergence \n  \n    \n      \n        \n          D\n          \n            K\n            L\n          \n        \n        (\n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        \n          z\n          \n            |\n          \n          x\n        \n        )\n        \u2225\n        \n          p", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0036", "text": ")\n        \u2225\n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        \n          z\n          \n            |\n          \n          x\n        \n        )\n        )\n      \n    \n    {\\displaystyle D_{KL}(q_{\\phi }({z|x})\\parallel p_{\\theta }({z|x}))}\n  \n can be used as a loss function to squeeze \n  \n    \n      \n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        \n          z\n          \n            |\n          \n          x", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0037", "text": "|\n          \n          x\n        \n        )\n      \n    \n    {\\displaystyle q_{\\phi }({z|x})}\n  \n under \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        z\n        \n          |\n        \n        x\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(z|x)}\n  \n. This divergence loss expands to", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0038", "text": "D\n                  \n                    K\n                    L\n                  \n                \n                (\n                \n                  q\n                  \n                    \u03d5\n                  \n                \n                (\n                \n                  z\n                  \n                    |\n                  \n                  x", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0039", "text": "x\n                \n                )\n                \u2225\n                \n                  p\n                  \n                    \u03b8\n                  \n                \n                (\n                \n                  z\n                  \n                    |\n                  \n                  x\n                \n                )\n                )\n              \n              \n                \n                =\n                \n                  \n                    E", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0040", "text": "E\n                  \n                  \n                    z\n                    \u223c\n                    \n                      q\n                      \n                        \u03d5\n                      \n                    \n                    (\n                    \u22c5\n                    \n                      |\n                    \n                    x\n                    )\n                  \n                \n                \n                  [", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0041", "text": "[\n                  \n                    ln\n                    \u2061\n                    \n                      \n                        \n                          \n                            q\n                            \n                              \u03d5\n                            \n                          \n                          (\n                          z\n                          \n                            |", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0042", "text": "x\n                          )\n                        \n                        \n                          \n                            p\n                            \n                              \u03b8\n                            \n                          \n                          (\n                          z\n                          \n                            |\n                          \n                          x", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0043", "text": "x\n                          )\n                        \n                      \n                    \n                  \n                  ]\n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    E\n                  \n                  \n                    z\n                    \u223c\n                    \n                      q", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0044", "text": "q\n                      \n                        \u03d5\n                      \n                    \n                    (\n                    \u22c5\n                    \n                      |\n                    \n                    x\n                    )\n                  \n                \n                \n                  [\n                  \n                    ln\n                    \u2061", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0045", "text": "q\n                            \n                              \u03d5\n                            \n                          \n                          (\n                          \n                            z\n                            \n                              |\n                            \n                            x\n                          \n                          )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0046", "text": ")\n                          \n                            p\n                            \n                              \u03b8\n                            \n                          \n                          (\n                          x\n                          )\n                        \n                        \n                          \n                            p\n                            \n                              \u03b8", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0047", "text": "(\n                          x\n                          ,\n                          z\n                          )\n                        \n                      \n                    \n                  \n                  ]\n                \n              \n            \n            \n              \n              \n                \n                =\n                ln\n                \u2061", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0048", "text": "\u2061\n                \n                  p\n                  \n                    \u03b8\n                  \n                \n                (\n                x\n                )\n                +\n                \n                  \n                    E\n                  \n                  \n                    z\n                    \u223c\n                    \n                      q\n                      \n                        \u03d5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0049", "text": "(\n                    \u22c5\n                    \n                      |\n                    \n                    x\n                    )\n                  \n                \n                \n                  [\n                  \n                    ln\n                    \u2061\n                    \n                      \n                        \n                          \n                            q", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0050", "text": "\u03d5\n                            \n                          \n                          (\n                          \n                            z\n                            \n                              |\n                            \n                            x\n                          \n                          )\n                        \n                        \n                          \n                            p", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0051", "text": "p\n                            \n                              \u03b8\n                            \n                          \n                          (\n                          x\n                          ,\n                          z\n                          )\n                        \n                      \n                    \n                  \n                  ]\n                \n                .", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0052", "text": "{\\displaystyle {\\begin{aligned}D_{KL}(q_{\\phi }({z|x})\\parallel p_{\\theta }({z|x}))&=\\mathbb {E} _{z\\sim q_{\\phi }(\\cdot |x)}\\left[\\ln {\\frac {q_{\\phi }(z|x)}{p_{\\theta }(z|x)}}\\right]\\\\&=\\mathbb {E} _{z\\sim q_{\\phi }(\\cdot |x)}\\left[\\ln {\\frac {q_{\\phi }({z|x})p_{\\theta }(x)}{p_{\\theta }(x,z)}}\\right]\\\\&=\\ln p_{\\theta }(x)+\\mathbb {E} _{z\\sim q_{\\phi }(\\cdot |x)}\\left[\\ln {\\frac {q_{\\phi }({z|x})}{p_{\\theta }(x,z)}}\\right].\\end{aligned}}}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0053", "text": "Now, define the evidence lower bound (ELBO):\n  \n    \n      \n        \n          L\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        (\n        x\n        )\n        :=\n        \n          \n            E\n          \n          \n            z\n            \u223c\n            \n              q\n              \n                \u03d5\n              \n            \n            (\n            \u22c5\n            \n              |\n            \n            x\n            )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0054", "text": ")\n          \n        \n        \n          [\n          \n            ln\n            \u2061\n            \n              \n                \n                  \n                    p\n                    \n                      \u03b8\n                    \n                  \n                  (\n                  x\n                  ,\n                  z\n                  )\n                \n                \n                  \n                    q\n                    \n                      \u03d5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0055", "text": "\u03d5\n                    \n                  \n                  (\n                  \n                    z\n                    \n                      |\n                    \n                    x\n                  \n                  )\n                \n              \n            \n          \n          ]\n        \n        =\n        ln\n        \u2061\n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        x\n        )\n        \u2212", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0056", "text": "(\n        x\n        )\n        \u2212\n        \n          D\n          \n            K\n            L\n          \n        \n        (\n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        \n          \u22c5\n          \n            |\n          \n          x\n        \n        )\n        \u2225\n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        \n          \u22c5\n          \n            |\n          \n          x\n        \n        )\n        )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0057", "text": ")\n        )\n      \n    \n    {\\displaystyle L_{\\theta ,\\phi }(x):=\\mathbb {E} _{z\\sim q_{\\phi }(\\cdot |x)}\\left[\\ln {\\frac {p_{\\theta }(x,z)}{q_{\\phi }({z|x})}}\\right]=\\ln p_{\\theta }(x)-D_{KL}(q_{\\phi }({\\cdot |x})\\parallel p_{\\theta }({\\cdot |x}))}\n  \nMaximizing the ELBO\n  \n    \n      \n        \n          \u03b8\n          \n            \u2217\n          \n        \n        ,\n        \n          \u03d5\n          \n            \u2217\n          \n        \n        =\n        \n          \n            argmax", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0058", "text": "=\n        \n          \n            argmax\n            \n              \u03b8\n              ,\n              \u03d5\n            \n          \n        \n        \n        \n          L\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle \\theta ^{*},\\phi ^{*}={\\underset {\\theta ,\\phi }{\\operatorname {argmax} }}\\,L_{\\theta ,\\phi }(x)}\n  \nis equivalent to simultaneously maximizing \n  \n    \n      \n        ln\n        \u2061", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0059", "text": "ln\n        \u2061\n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle \\ln p_{\\theta }(x)}\n  \n and minimizing \n  \n    \n      \n        \n          D\n          \n            K\n            L\n          \n        \n        (\n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        \n          z\n          \n            |\n          \n          x\n        \n        )\n        \u2225", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0060", "text": "x\n        \n        )\n        \u2225\n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        \n          z\n          \n            |\n          \n          x\n        \n        )\n        )\n      \n    \n    {\\displaystyle D_{KL}(q_{\\phi }({z|x})\\parallel p_{\\theta }({z|x}))}\n  \n. That is, maximizing the log-likelihood of the observed data, and minimizing the divergence from the approximate posterior \n  \n    \n      \n        \n          q\n          \n            \u03d5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0061", "text": "q\n          \n            \u03d5\n          \n        \n        (\n        \u22c5\n        \n          |\n        \n        x\n        )\n      \n    \n    {\\displaystyle q_{\\phi }(\\cdot |x)}\n  \n to the exact posterior \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        \u22c5\n        \n          |\n        \n        x\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(\\cdot |x)}\n  \n.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0062", "text": "{\\displaystyle p_{\\theta }(\\cdot |x)}\n  \n.\nThe form given is not very convenient for maximization, but the following, equivalent form, is:\n  \n    \n      \n        \n          L\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        (\n        x\n        )\n        =\n        \n          \n            E\n          \n          \n            z\n            \u223c\n            \n              q\n              \n                \u03d5\n              \n            \n            (\n            \u22c5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0063", "text": "(\n            \u22c5\n            \n              |\n            \n            x\n            )\n          \n        \n        \n          [\n          \n            ln\n            \u2061\n            \n              p\n              \n                \u03b8\n              \n            \n            (\n            x\n            \n              |\n            \n            z\n            )\n          \n          ]\n        \n        \u2212\n        \n          D\n          \n            K\n            L", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0064", "text": "K\n            L\n          \n        \n        (\n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        \n          \u22c5\n          \n            |\n          \n          x\n        \n        )\n        \u2225\n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        \u22c5\n        )\n        )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0065", "text": "\u22c5\n        )\n        )\n      \n    \n    {\\displaystyle L_{\\theta ,\\phi }(x)=\\mathbb {E} _{z\\sim q_{\\phi }(\\cdot |x)}\\left[\\ln p_{\\theta }(x|z)\\right]-D_{KL}(q_{\\phi }({\\cdot |x})\\parallel p_{\\theta }(\\cdot ))}\n  \nwhere \n  \n    \n      \n        ln\n        \u2061\n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        x\n        \n          |\n        \n        z\n        )\n      \n    \n    {\\displaystyle \\ln p_{\\theta }(x|z)}\n  \n is implemented as \n  \n    \n      \n        \u2212", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0066", "text": "is implemented as \n  \n    \n      \n        \u2212\n        \n          \n            1\n            2\n          \n        \n        \u2016\n        x\n        \u2212\n        \n          D\n          \n            \u03b8\n          \n        \n        (\n        z\n        )\n        \n          \u2016\n          \n            2\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle -{\\frac {1}{2}}\\|x-D_{\\theta }(z)\\|_{2}^{2}}\n  \n, since that is, up to an additive constant, what \n  \n    \n      \n        x", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0067", "text": "x\n        \n          |\n        \n        z\n        \u223c\n        \n          \n            N\n          \n        \n        (\n        \n          D\n          \n            \u03b8\n          \n        \n        (\n        z\n        )\n        ,\n        I\n        )\n      \n    \n    {\\displaystyle x|z\\sim {\\mathcal {N}}(D_{\\theta }(z),I)}\n  \n yields. That is, we model the distribution of \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n conditional on \n  \n    \n      \n        z", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0068", "text": "conditional on \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n to be a Gaussian distribution centered on \n  \n    \n      \n        \n          D\n          \n            \u03b8\n          \n        \n        (\n        z\n        )\n      \n    \n    {\\displaystyle D_{\\theta }(z)}\n  \n. The distribution of \n  \n    \n      \n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        z\n        \n          |\n        \n        x\n        )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0069", "text": "x\n        )\n      \n    \n    {\\displaystyle q_{\\phi }(z|x)}\n  \n and \n  \n    \n      \n        \n          p\n          \n            \u03b8\n          \n        \n        (\n        z\n        )\n      \n    \n    {\\displaystyle p_{\\theta }(z)}\n  \n are often also chosen to be Gaussians as \n  \n    \n      \n        z\n        \n          |\n        \n        x\n        \u223c\n        \n          \n            N\n          \n        \n        (\n        \n          E\n          \n            \u03d5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0070", "text": "\u03d5\n          \n        \n        (\n        x\n        )\n        ,\n        \n          \u03c3\n          \n            \u03d5\n          \n        \n        (\n        x\n        \n          )\n          \n            2\n          \n        \n        I\n        )\n      \n    \n    {\\displaystyle z|x\\sim {\\mathcal {N}}(E_{\\phi }(x),\\sigma _{\\phi }(x)^{2}I)}\n  \n and \n  \n    \n      \n        z\n        \u223c\n        \n          \n            N\n          \n        \n        (\n        0\n        ,\n        I\n        )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0071", "text": "(\n        0\n        ,\n        I\n        )\n      \n    \n    {\\displaystyle z\\sim {\\mathcal {N}}(0,I)}\n  \n, with which we obtain by the formula for KL divergence of Gaussians:\n  \n    \n      \n        \n          L\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        (\n        x\n        )\n        =\n        \u2212\n        \n          \n            1\n            2\n          \n        \n        \n          \n            E\n          \n          \n            z\n            \u223c", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0072", "text": "z\n            \u223c\n            \n              q\n              \n                \u03d5\n              \n            \n            (\n            \u22c5\n            \n              |\n            \n            x\n            )\n          \n        \n        \n          [\n          \n            \u2016\n            x\n            \u2212\n            \n              D\n              \n                \u03b8\n              \n            \n            (\n            z\n            )\n            \n              \u2016", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0073", "text": ")\n            \n              \u2016\n              \n                2\n              \n              \n                2\n              \n            \n          \n          ]\n        \n        \u2212\n        \n          \n            1\n            2\n          \n        \n        \n          (\n          \n            N\n            \n              \u03c3\n              \n                \u03d5\n              \n            \n            (\n            x\n            \n              )\n              \n                2", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0074", "text": ")\n              \n                2\n              \n            \n            +\n            \u2016\n            \n              E\n              \n                \u03d5\n              \n            \n            (\n            x\n            )\n            \n              \u2016\n              \n                2\n              \n              \n                2\n              \n            \n            \u2212\n            2\n            N\n            ln\n            \u2061\n            \n              \u03c3", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0075", "text": "\u03c3\n              \n                \u03d5\n              \n            \n            (\n            x\n            )\n          \n          )\n        \n        +\n        C\n        o\n        n\n        s\n        t\n      \n    \n    {\\displaystyle L_{\\theta ,\\phi }(x)=-{\\frac {1}{2}}\\mathbb {E} _{z\\sim q_{\\phi }(\\cdot |x)}\\left[\\|x-D_{\\theta }(z)\\|_{2}^{2}\\right]-{\\frac {1}{2}}\\left(N\\sigma _{\\phi }(x)^{2}+\\|E_{\\phi }(x)\\|_{2}^{2}-2N\\ln \\sigma _{\\phi }(x)\\right)+Const}\n  \nHere", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0076", "text": "Here \n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}\n  \n is the dimension of \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n. For a more detailed derivation and more interpretations of ELBO and its maximization, see its main page.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0077", "text": "Reparameterization\nTo efficiently search for \n  \n    \n      \n        \n          \u03b8\n          \n            \u2217\n          \n        \n        ,\n        \n          \u03d5\n          \n            \u2217\n          \n        \n        =\n        \n          \n            argmax\n            \n              \u03b8\n              ,\n              \u03d5\n            \n          \n        \n        \n        \n          L\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        (\n        x\n        )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0078", "text": "(\n        x\n        )\n      \n    \n    {\\displaystyle \\theta ^{*},\\phi ^{*}={\\underset {\\theta ,\\phi }{\\operatorname {argmax} }}\\,L_{\\theta ,\\phi }(x)}\n  \nthe typical method is gradient ascent.\nIt is straightforward to find\n  \n    \n      \n        \n          \u2207\n          \n            \u03b8\n          \n        \n        \n          \n            E\n          \n          \n            z\n            \u223c\n            \n              q\n              \n                \u03d5\n              \n            \n            (", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0079", "text": "(\n            \u22c5\n            \n              |\n            \n            x\n            )\n          \n        \n        \n          [\n          \n            ln\n            \u2061\n            \n              \n                \n                  \n                    p\n                    \n                      \u03b8\n                    \n                  \n                  (\n                  x\n                  ,\n                  z\n                  )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0080", "text": ")\n                \n                \n                  \n                    q\n                    \n                      \u03d5\n                    \n                  \n                  (\n                  \n                    z\n                    \n                      |\n                    \n                    x\n                  \n                  )\n                \n              \n            \n          \n          ]\n        \n        =\n        \n          \n            E", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0081", "text": "E\n          \n          \n            z\n            \u223c\n            \n              q\n              \n                \u03d5\n              \n            \n            (\n            \u22c5\n            \n              |\n            \n            x\n            )\n          \n        \n        \n          [\n          \n            \n              \u2207\n              \n                \u03b8\n              \n            \n            ln\n            \u2061", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0082", "text": "p\n                    \n                      \u03b8\n                    \n                  \n                  (\n                  x\n                  ,\n                  z\n                  )\n                \n                \n                  \n                    q\n                    \n                      \u03d5\n                    \n                  \n                  (\n                  \n                    z", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0083", "text": "z\n                    \n                      |\n                    \n                    x\n                  \n                  )\n                \n              \n            \n          \n          ]", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0084", "text": "]\n        \n      \n    \n    {\\displaystyle \\nabla _{\\theta }\\mathbb {E} _{z\\sim q_{\\phi }(\\cdot |x)}\\left[\\ln {\\frac {p_{\\theta }(x,z)}{q_{\\phi }({z|x})}}\\right]=\\mathbb {E} _{z\\sim q_{\\phi }(\\cdot |x)}\\left[\\nabla _{\\theta }\\ln {\\frac {p_{\\theta }(x,z)}{q_{\\phi }({z|x})}}\\right]}\n  \nHowever, \n  \n    \n      \n        \n          \u2207\n          \n            \u03d5\n          \n        \n        \n          \n            E\n          \n          \n            z\n            \u223c", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0085", "text": "z\n            \u223c\n            \n              q\n              \n                \u03d5\n              \n            \n            (\n            \u22c5\n            \n              |\n            \n            x\n            )\n          \n        \n        \n          [\n          \n            ln\n            \u2061\n            \n              \n                \n                  \n                    p\n                    \n                      \u03b8\n                    \n                  \n                  (", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0086", "text": "(\n                  x\n                  ,\n                  z\n                  )\n                \n                \n                  \n                    q\n                    \n                      \u03d5\n                    \n                  \n                  (\n                  \n                    z\n                    \n                      |\n                    \n                    x\n                  \n                  )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0087", "text": "]\n        \n      \n    \n    {\\displaystyle \\nabla _{\\phi }\\mathbb {E} _{z\\sim q_{\\phi }(\\cdot |x)}\\left[\\ln {\\frac {p_{\\theta }(x,z)}{q_{\\phi }({z|x})}}\\right]}\n  \ndoes not allow one to put the \n  \n    \n      \n        \n          \u2207\n          \n            \u03d5\n          \n        \n      \n    \n    {\\displaystyle \\nabla _{\\phi }}\n  \n inside the expectation, since \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0088", "text": "{\\displaystyle \\phi }\n  \n appears in the probability distribution itself. The reparameterization trick (also known as stochastic backpropagation) bypasses this difficulty.\nThe most important example is when \n  \n    \n      \n        z\n        \u223c\n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        \u22c5\n        \n          |\n        \n        x\n        )\n      \n    \n    {\\displaystyle z\\sim q_{\\phi }(\\cdot |x)}\n  \n is normally distributed, as", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0089", "text": "is normally distributed, as \n  \n    \n      \n        \n          \n            N\n          \n        \n        (\n        \n          \u03bc\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        ,\n        \n          \u03a3\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        )\n      \n    \n    {\\displaystyle {\\mathcal {N}}(\\mu _{\\phi }(x),\\Sigma _{\\phi }(x))}\n  \n.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0090", "text": "This can be reparametrized by letting \n  \n    \n      \n        \n          \u03b5\n        \n        \u223c\n        \n          \n            N\n          \n        \n        (\n        0\n        ,\n        \n          I\n        \n        )\n      \n    \n    {\\displaystyle {\\boldsymbol {\\varepsilon }}\\sim {\\mathcal {N}}(0,{\\boldsymbol {I}})}\n  \n be a \"standard random number generator\", and construct \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n as \n  \n    \n      \n        z\n        =", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0091", "text": "as \n  \n    \n      \n        z\n        =\n        \n          \u03bc\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        +\n        \n          L\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        \u03f5\n      \n    \n    {\\displaystyle z=\\mu _{\\phi }(x)+L_{\\phi }(x)\\epsilon }\n  \n. Here, \n  \n    \n      \n        \n          L\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle L_{\\phi }(x)}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0092", "text": "{\\displaystyle L_{\\phi }(x)}\n  \n is obtained by the Cholesky decomposition:\n  \n    \n      \n        \n          \u03a3\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        =\n        \n          L\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        \n          L\n          \n            \u03d5\n          \n        \n        (\n        x\n        \n          )\n          \n            T", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0093", "text": "T\n          \n        \n      \n    \n    {\\displaystyle \\Sigma _{\\phi }(x)=L_{\\phi }(x)L_{\\phi }(x)^{T}}\n  \nThen we have\n  \n    \n      \n        \n          \u2207\n          \n            \u03d5\n          \n        \n        \n          \n            E\n          \n          \n            z\n            \u223c\n            \n              q\n              \n                \u03d5\n              \n            \n            (\n            \u22c5\n            \n              |\n            \n            x\n            )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0094", "text": "x\n            )\n          \n        \n        \n          [\n          \n            ln\n            \u2061\n            \n              \n                \n                  \n                    p\n                    \n                      \u03b8\n                    \n                  \n                  (\n                  x\n                  ,\n                  z\n                  )\n                \n                \n                  \n                    q\n                    \n                      \u03d5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0095", "text": "\u03d5\n                    \n                  \n                  (\n                  \n                    z\n                    \n                      |\n                    \n                    x\n                  \n                  )\n                \n              \n            \n          \n          ]\n        \n        =\n        \n          \n            E\n          \n          \n            \u03f5\n          \n        \n        \n          [", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0096", "text": "[\n          \n            \n              \u2207\n              \n                \u03d5\n              \n            \n            ln\n            \u2061\n            \n              \n                \n                  \n                    p\n                    \n                      \u03b8\n                    \n                  \n                  (\n                  x\n                  ,\n                  \n                    \u03bc\n                    \n                      \u03d5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0097", "text": "\u03d5\n                    \n                  \n                  (\n                  x\n                  )\n                  +\n                  \n                    L\n                    \n                      \u03d5\n                    \n                  \n                  (\n                  x\n                  )\n                  \u03f5\n                  )\n                \n                \n                  \n                    q\n                    \n                      \u03d5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0098", "text": "\u03d5\n                    \n                  \n                  (\n                  \n                    \u03bc\n                    \n                      \u03d5\n                    \n                  \n                  (\n                  x\n                  )\n                  +\n                  \n                    L\n                    \n                      \u03d5\n                    \n                  \n                  (\n                  x\n                  )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0099", "text": "x\n                  )\n                  \u03f5\n                  \n                    |\n                  \n                  x\n                  )\n                \n              \n            \n          \n          ]", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0100", "text": "]\n        \n      \n    \n    {\\displaystyle \\nabla _{\\phi }\\mathbb {E} _{z\\sim q_{\\phi }(\\cdot |x)}\\left[\\ln {\\frac {p_{\\theta }(x,z)}{q_{\\phi }({z|x})}}\\right]=\\mathbb {E} _{\\epsilon }\\left[\\nabla _{\\phi }\\ln {\\frac {p_{\\theta }(x,\\mu _{\\phi }(x)+L_{\\phi }(x)\\epsilon )}{q_{\\phi }(\\mu _{\\phi }(x)+L_{\\phi }(x)\\epsilon |x)}}\\right]}\n  \nand so we obtained an unbiased estimator of the gradient, allowing stochastic gradient descent.\nSince we reparametrized", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0101", "text": "Since we reparametrized \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n, we need to find \n  \n    \n      \n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        z\n        \n          |\n        \n        x\n        )\n      \n    \n    {\\displaystyle q_{\\phi }(z|x)}\n  \n. Let \n  \n    \n      \n        \n          q\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle q_{0}}\n  \n be the probability density function for", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0102", "text": "\u03f5\n      \n    \n    {\\displaystyle \\epsilon }\n  \n, then \n  \n    \n      \n        ln\n        \u2061\n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        z\n        \n          |\n        \n        x\n        )\n        =\n        ln\n        \u2061\n        \n          q\n          \n            0\n          \n        \n        (\n        \u03f5\n        )\n        \u2212\n        ln\n        \u2061\n        \n          |\n        \n        det\n        (\n        \n          \u2202", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0103", "text": "(\n        \n          \u2202\n          \n            \u03f5\n          \n        \n        z\n        )\n        \n          |\n        \n      \n    \n    {\\displaystyle \\ln q_{\\phi }(z|x)=\\ln q_{0}(\\epsilon )-\\ln |\\det(\\partial _{\\epsilon }z)|}\n  \nwhere \n  \n    \n      \n        \n          \u2202\n          \n            \u03f5\n          \n        \n        z\n      \n    \n    {\\displaystyle \\partial _{\\epsilon }z}\n  \n is the Jacobian matrix of \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n with respect to", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0104", "text": "{\\displaystyle z}\n  \n with respect to \n  \n    \n      \n        \u03f5\n      \n    \n    {\\displaystyle \\epsilon }\n  \n. Since \n  \n    \n      \n        z\n        =\n        \n          \u03bc\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        +\n        \n          L\n          \n            \u03d5\n          \n        \n        (\n        x\n        )\n        \u03f5\n      \n    \n    {\\displaystyle z=\\mu _{\\phi }(x)+L_{\\phi }(x)\\epsilon }\n  \n, this is \n  \n    \n      \n        ln\n        \u2061", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0105", "text": ", this is \n  \n    \n      \n        ln\n        \u2061\n        \n          q\n          \n            \u03d5\n          \n        \n        (\n        z\n        \n          |\n        \n        x\n        )\n        =\n        \u2212\n        \n          \n            1\n            2\n          \n        \n        \u2016\n        \u03f5\n        \n          \u2016\n          \n            2\n          \n        \n        \u2212\n        ln\n        \u2061\n        \n          |\n        \n        det\n        \n          L\n          \n            \u03d5", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0106", "text": "\u03d5\n          \n        \n        (\n        x\n        )\n        \n          |\n        \n        \u2212\n        \n          \n            n\n            2\n          \n        \n        ln\n        \u2061\n        (\n        2\n        \u03c0\n        )\n      \n    \n    {\\displaystyle \\ln q_{\\phi }(z|x)=-{\\frac {1}{2}}\\|\\epsilon \\|^{2}-\\ln |\\det L_{\\phi }(x)|-{\\frac {n}{2}}\\ln(2\\pi )}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0107", "text": "Variations\nMany variational autoencoders applications and extensions have been used to adapt the architecture to other domains and improve its performance.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0108", "text": "\u03b2\n      \n    \n    {\\displaystyle \\beta }\n  \n-VAE is an implementation with a weighted Kullback\u2013Leibler divergence term to automatically discover and interpret factorised latent representations. With this implementation, it is possible to force manifold disentanglement for \n  \n    \n      \n        \u03b2\n      \n    \n    {\\displaystyle \\beta }\n  \n values greater than one. This architecture can discover disentangled latent factors without supervision.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0109", "text": "The conditional VAE (CVAE), inserts label information in the latent space to force a deterministic constrained representation of the learned data.\nSome structures directly deal with the quality of the generated samples or implement more than one latent space to further improve the representation learning.\nSome architectures mix VAE and generative adversarial networks to obtain hybrid models.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0110", "text": "It is not necessary to use gradients to update the encoder. In fact, the encoder is not necessary for the generative model.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0111", "text": "Statistical distance VAE variants\nAfter the initial work of Diederik P. Kingma and Max Welling, several procedures were \nproposed to formulate in a more abstract way the operation of the VAE. In these approaches the loss function is composed of two parts :", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0112", "text": "the usual reconstruction error part which seeks to ensure that the encoder-then-decoder mapping \n  \n    \n      \n        x\n        \u21a6\n        \n          D\n          \n            \u03b8\n          \n        \n        (\n        \n          E\n          \n            \u03c8\n          \n        \n        (\n        x\n        )\n        )\n      \n    \n    {\\displaystyle x\\mapsto D_{\\theta }(E_{\\psi }(x))}\n  \n is as close to the identity map as possible; the sampling is done at run time from the empirical distribution", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0113", "text": "P\n          \n          \n            r\n            e\n            a\n            l\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {P} ^{real}}\n  \n of objects available (e.g., for MNIST or IMAGENET this will be the empirical probability law of all images in the dataset). This gives the term:  \n  \n    \n      \n        \n          \n            E\n          \n          \n            x\n            \u223c\n            \n              \n                P", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0114", "text": "P\n              \n              \n                r\n                e\n                a\n                l\n              \n            \n          \n        \n        \n          [\n          \n            \u2016\n            x\n            \u2212\n            \n              D\n              \n                \u03b8\n              \n            \n            (\n            \n              E\n              \n                \u03d5\n              \n            \n            (\n            x", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0115", "text": "(\n            x\n            )\n            )\n            \n              \u2016\n              \n                2\n              \n              \n                2\n              \n            \n          \n          ]\n        \n      \n    \n    {\\displaystyle \\mathbb {E} _{x\\sim \\mathbb {P} ^{real}}\\left[\\|x-D_{\\theta }(E_{\\phi }(x))\\|_{2}^{2}\\right]}\n  \n.\na variational part that ensures that, when the empirical distribution \n  \n    \n      \n        \n          \n            P", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0116", "text": "P\n          \n          \n            r\n            e\n            a\n            l\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {P} ^{real}}\n  \n is passed through the encoder \n  \n    \n      \n        \n          E\n          \n            \u03d5\n          \n        \n      \n    \n    {\\displaystyle E_{\\phi }}\n  \n, we recover the target distribution, denoted here \n  \n    \n      \n        \u03bc\n        (\n        d\n        z\n        )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0117", "text": "d\n        z\n        )\n      \n    \n    {\\displaystyle \\mu (dz)}\n  \n that is usually taken to be a Multivariate normal distribution. We will denote \n  \n    \n      \n        \n          E\n          \n            \u03d5\n          \n        \n        \u266f\n        \n          \n            P\n          \n          \n            r\n            e\n            a\n            l\n          \n        \n      \n    \n    {\\displaystyle E_{\\phi }\\sharp \\mathbb {P} ^{real}}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0118", "text": "this pushforward measure which in practice is just the empirical distribution obtained by passing all dataset objects through the encoder \n  \n    \n      \n        \n          E\n          \n            \u03d5\n          \n        \n      \n    \n    {\\displaystyle E_{\\phi }}\n  \n. In order to make sure that  \n  \n    \n      \n        \n          E\n          \n            \u03d5\n          \n        \n        \u266f\n        \n          \n            P\n          \n          \n            r\n            e\n            a", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0119", "text": "r\n            e\n            a\n            l\n          \n        \n      \n    \n    {\\displaystyle E_{\\phi }\\sharp \\mathbb {P} ^{real}}\n  \n  is close to the target \n  \n    \n      \n        \u03bc\n        (\n        d\n        z\n        )\n      \n    \n    {\\displaystyle \\mu (dz)}\n  \n, a Statistical distance \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n is invoked and the term  \n  \n    \n      \n        d\n        \n          \n            (\n            \n              \u03bc", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0120", "text": "(\n            \n              \u03bc\n              (\n              d\n              z\n              )\n              ,\n              \n                E\n                \n                  \u03d5\n                \n              \n              \u266f\n              \n                \n                  P\n                \n                \n                  r\n                  e\n                  a\n                  l\n                \n              \n            \n            )", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0121", "text": ")\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle d\\left(\\mu (dz),E_{\\phi }\\sharp \\mathbb {P} ^{real}\\right)^{2}}\n  \n is added to the loss.\nWe obtain the final formula for the loss:", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0122", "text": "L\n          \n            \u03b8\n            ,\n            \u03d5\n          \n        \n        =\n        \n          \n            E\n          \n          \n            x\n            \u223c\n            \n              \n                P\n              \n              \n                r\n                e\n                a\n                l\n              \n            \n          \n        \n        \n          [\n          \n            \u2016\n            x\n            \u2212", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0123", "text": "x\n            \u2212\n            \n              D\n              \n                \u03b8\n              \n            \n            (\n            \n              E\n              \n                \u03d5\n              \n            \n            (\n            x\n            )\n            )\n            \n              \u2016\n              \n                2\n              \n              \n                2\n              \n            \n          \n          ]\n        \n        +\n        d", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0124", "text": "+\n        d\n        \n          \n            (\n            \n              \u03bc\n              (\n              d\n              z\n              )\n              ,\n              \n                E\n                \n                  \u03d5\n                \n              \n              \u266f\n              \n                \n                  P\n                \n                \n                  r\n                  e\n                  a\n                  l", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0125", "text": ")\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle L_{\\theta ,\\phi }=\\mathbb {E} _{x\\sim \\mathbb {P} ^{real}}\\left[\\|x-D_{\\theta }(E_{\\phi }(x))\\|_{2}^{2}\\right]+d\\left(\\mu (dz),E_{\\phi }\\sharp \\mathbb {P} ^{real}\\right)^{2}}", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0126", "text": "The statistical distance \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n requires special properties, for instance it has to be posses a formula as expectation because the loss function will need to be optimized by stochastic optimization algorithms. Several distances can be chosen and this gave rise to several flavors of VAEs:", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0127", "text": "the sliced Wasserstein distance used by S Kolouri, et al. in their VAE\nthe energy distance implemented in the Radon Sobolev Variational Auto-Encoder\nthe Maximum Mean Discrepancy distance used in the MMD-VAE\nthe Wasserstein distance used in the WAEs\nkernel-based distances used in the Kernelized Variational Autoencoder (K-VAE)\n\nSee also\nReferences\nFurther reading", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Variational_autoencoder_0128", "text": "See also\nReferences\nFurther reading\n\nKingma, Diederik P.; Welling, Max (2019). \"An Introduction to Variational Autoencoders\". Foundations and Trends in Machine Learning. 12 (4). Now Publishers: 307\u2013392. arXiv:1906.02691. doi:10.1561/2200000056. ISSN 1935-8237.", "title": "Variational autoencoder", "url": "https://en.wikipedia.org/wiki/Variational_autoencoder"}, {"id": "rec_Residual_neural_network_0000", "text": "A residual neural network (also referred to as a residual network or ResNet) is a deep learning architecture in which the layers learn residual functions with reference to the layer inputs. It was developed in 2015 for image recognition, and won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) of that year.\nAs a point of terminology, \"residual connection\" refers to the specific architectural motif of \n  \n    \n      \n        x\n        \u21a6\n        f\n        (\n        x\n        )", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0001", "text": "\u21a6\n        f\n        (\n        x\n        )\n        +\n        x\n      \n    \n    {\\displaystyle x\\mapsto f(x)+x}\n  \n, where \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is an arbitrary neural network module. The motif had been used previously (see \u00a7History for details). However, the publication of ResNet made it widely popular for feedforward networks, appearing in neural networks that are seemingly unrelated to ResNet.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0002", "text": "The residual connection stabilizes the training and convergence of deep neural networks with hundreds of layers, and is a common motif in deep neural networks, such as transformer models (e.g., BERT, and GPT models such as ChatGPT), the AlphaGo Zero system, the AlphaStar system, and the AlphaFold system.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0003", "text": "Mathematics\nResidual connection\nIn a multilayer neural network model, consider a subnetwork with a certain number of stacked layers (e.g., 2 or 3). Denote the underlying function performed by this subnetwork as \n  \n    \n      \n        H\n        (\n        x\n        )\n      \n    \n    {\\displaystyle H(x)}\n  \n, where \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0004", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n is the input to the subnetwork. Residual learning re-parameterizes this subnetwork and lets the parameter layers represent a \"residual function\" \n  \n    \n      \n        F\n        (\n        x\n        )\n        =\n        H\n        (\n        x\n        )\n        \u2212\n        x\n      \n    \n    {\\displaystyle F(x)=H(x)-x}\n  \n. The output \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n of this subnetwork is then represented as:", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0005", "text": "y\n        =\n        F\n        (\n        x\n        )\n        +\n        x\n      \n    \n    {\\displaystyle y=F(x)+x}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0006", "text": "The operation of \"\n  \n    \n      \n        +\n         \n        x\n      \n    \n    {\\displaystyle +\\ x}\n  \n\" is implemented via a \"skip connection\" that performs an identity mapping to connect the input of the subnetwork with its output. This connection is referred to as a \"residual connection\" in later work. The function \n  \n    \n      \n        F\n        (\n        x\n        )\n      \n    \n    {\\displaystyle F(x)}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0007", "text": ")\n      \n    \n    {\\displaystyle F(x)}\n  \n is often represented by matrix multiplication interlaced with activation functions and normalization operations (e.g., batch normalization or layer normalization). As a whole, one of these subnetworks is referred to as a \"residual block\". A deep residual network is constructed by simply stacking these blocks.\nLong short-term memory (LSTM) has a memory mechanism that serves as a residual connection. In an LSTM without a forget gate, an input", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0008", "text": "x\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle x_{t}}\n  \n is processed by a function \n  \n    \n      \n        F\n      \n    \n    {\\displaystyle F}\n  \n and added to a memory cell \n  \n    \n      \n        \n          c\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle c_{t}}\n  \n, resulting in \n  \n    \n      \n        \n          c\n          \n            t\n            +\n            1\n          \n        \n        =", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0009", "text": "1\n          \n        \n        =\n        \n          c\n          \n            t\n          \n        \n        +\n        F\n        (\n        \n          x\n          \n            t\n          \n        \n        )\n      \n    \n    {\\displaystyle c_{t+1}=c_{t}+F(x_{t})}\n  \n. An LSTM with a forget gate essentially functions as a highway network.\nTo stabilize the variance of the layers' inputs, it is recommended to replace the residual connections \n  \n    \n      \n        x\n        +\n        f", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0010", "text": "x\n        +\n        f\n        (\n        x\n        )\n      \n    \n    {\\displaystyle x+f(x)}\n  \n with \n  \n    \n      \n        x\n        \n          /\n        \n        L\n        +\n        f\n        (\n        x\n        )\n      \n    \n    {\\displaystyle x/L+f(x)}\n  \n, where \n  \n    \n      \n        L\n      \n    \n    {\\displaystyle L}\n  \n is the total number of residual layers.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0011", "text": "Projection connection\nIf the function \n  \n    \n      \n        F\n      \n    \n    {\\displaystyle F}\n  \n is of type \n  \n    \n      \n        F\n        :\n        \n          \n            R\n          \n          \n            n\n          \n        \n        \u2192\n        \n          \n            R\n          \n          \n            m\n          \n        \n      \n    \n    {\\displaystyle F:\\mathbb {R} ^{n}\\to \\mathbb {R} ^{m}}\n  \n where \n  \n    \n      \n        n\n        \u2260\n        m", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0012", "text": "n\n        \u2260\n        m\n      \n    \n    {\\displaystyle n\\neq m}\n  \n, then \n  \n    \n      \n        F\n        (\n        x\n        )\n        +\n        x\n      \n    \n    {\\displaystyle F(x)+x}\n  \n is undefined. To handle this special case, a projection connection is used:", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0013", "text": "y\n        =\n        F\n        (\n        x\n        )\n        +\n        P\n        (\n        x\n        )\n      \n    \n    {\\displaystyle y=F(x)+P(x)}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0014", "text": "where \n  \n    \n      \n        P\n      \n    \n    {\\displaystyle P}\n  \n is typically a linear projection, defined by \n  \n    \n      \n        P\n        (\n        x\n        )\n        =\n        M\n        x\n      \n    \n    {\\displaystyle P(x)=Mx}\n  \n where \n  \n    \n      \n        M\n      \n    \n    {\\displaystyle M}\n  \n is a \n  \n    \n      \n        m\n        \u00d7\n        n\n      \n    \n    {\\displaystyle m\\times n}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0015", "text": "{\\displaystyle m\\times n}\n  \n matrix. The matrix is trained via backpropagation, as is any other parameter of the model.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0016", "text": "Signal propagation\nThe introduction of identity mappings facilitates signal propagation in both forward and backward paths.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0017", "text": "Forward propagation\nIf the output of the \n  \n    \n      \n        \u2113\n      \n    \n    {\\displaystyle \\ell }\n  \n-th residual block is the input to the \n  \n    \n      \n        (\n        \u2113\n        +\n        1\n        )\n      \n    \n    {\\displaystyle (\\ell +1)}\n  \n-th residual block (assuming no activation function between blocks), then the \n  \n    \n      \n        (\n        \u2113\n        +\n        1\n        )\n      \n    \n    {\\displaystyle (\\ell +1)}\n  \n-th input is:", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0018", "text": "x\n          \n            \u2113\n            +\n            1\n          \n        \n        =\n        F\n        (\n        \n          x\n          \n            \u2113\n          \n        \n        )\n        +\n        \n          x\n          \n            \u2113\n          \n        \n      \n    \n    {\\displaystyle x_{\\ell +1}=F(x_{\\ell })+x_{\\ell }}\n  \n\nApplying this formulation recursively, e.g.:", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0019", "text": "x\n                  \n                    \u2113\n                    +\n                    2\n                  \n                \n              \n              \n                \n                =\n                F\n                (\n                \n                  x\n                  \n                    \u2113\n                    +\n                    1\n                  \n                \n                )", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0020", "text": ")\n                +\n                \n                  x\n                  \n                    \u2113\n                    +\n                    1\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                F\n                (\n                \n                  x\n                  \n                    \u2113\n                    +\n                    1", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0021", "text": "1\n                  \n                \n                )\n                +\n                F\n                (\n                \n                  x\n                  \n                    \u2113\n                  \n                \n                )\n                +\n                \n                  x\n                  \n                    \u2113", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0022", "text": "{\\displaystyle {\\begin{aligned}x_{\\ell +2}&=F(x_{\\ell +1})+x_{\\ell +1}\\\\&=F(x_{\\ell +1})+F(x_{\\ell })+x_{\\ell }\\end{aligned}}}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0023", "text": "yields the general relationship:", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0024", "text": "x\n          \n            L\n          \n        \n        =\n        \n          x\n          \n            \u2113\n          \n        \n        +\n        \n          \u2211\n          \n            i\n            =\n            \u2113\n          \n          \n            L\n            \u2212\n            1\n          \n        \n        F\n        (\n        \n          x\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle x_{L}=x_{\\ell }+\\sum _{i=\\ell }^{L-1}F(x_{i})}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0025", "text": "where \n  \n    \n      \n        L\n      \n    \n    {\\textstyle L}\n  \n is the index of a residual block and \n  \n    \n      \n        \u2113\n      \n    \n    {\\textstyle \\ell }\n  \n is the index of some earlier block. This formulation suggests that there is always a signal that is directly sent from a shallower block \n  \n    \n      \n        \u2113\n      \n    \n    {\\textstyle \\ell }\n  \n to a deeper block \n  \n    \n      \n        L\n      \n    \n    {\\textstyle L}\n  \n.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0026", "text": "Backward propagation\nThe residual learning formulation provides the added benefit of mitigating the vanishing gradient problem to some extent. However, it is crucial to acknowledge that the vanishing gradient issue is not the root cause of the degradation problem, which is tackled through the use of normalization. To observe the effect of residual blocks on backpropagation, consider the partial derivative of a loss function \n  \n    \n      \n        \n          \n            E", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0027", "text": "E\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {E}}}\n  \n with respect to some residual block input \n  \n    \n      \n        \n          x\n          \n            \u2113\n          \n        \n      \n    \n    {\\displaystyle x_{\\ell }}\n  \n. Using the equation above from forward propagation for a later residual block \n  \n    \n      \n        L\n        >\n        \u2113\n      \n    \n    {\\displaystyle L>\\ell }\n  \n:", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0028", "text": "\u2202\n                      \n                        \n                          E\n                        \n                      \n                    \n                    \n                      \u2202\n                      \n                        x\n                        \n                          \u2113", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0029", "text": "=\n                \n                  \n                    \n                      \u2202\n                      \n                        \n                          E\n                        \n                      \n                    \n                    \n                      \u2202\n                      \n                        x", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0030", "text": "x\n                        \n                          L\n                        \n                      \n                    \n                  \n                \n                \n                  \n                    \n                      \u2202\n                      \n                        x\n                        \n                          L", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0031", "text": "\u2202\n                      \n                        x\n                        \n                          \u2113\n                        \n                      \n                    \n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    \n                      \u2202", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0032", "text": "\u2202\n                      \n                        \n                          E\n                        \n                      \n                    \n                    \n                      \u2202\n                      \n                        x\n                        \n                          L\n                        \n                      \n                    \n                  \n                \n                \n                  (\n                  \n                    1", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0033", "text": "1\n                    +\n                    \n                      \n                        \u2202\n                        \n                          \u2202\n                          \n                            x\n                            \n                              \u2113\n                            \n                          \n                        \n                      \n                    \n                    \n                      \u2211", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0034", "text": "\u2211\n                      \n                        i\n                        =\n                        \u2113\n                      \n                      \n                        L\n                        \u2212\n                        1\n                      \n                    \n                    F\n                    (\n                    \n                      x\n                      \n                        i\n                      \n                    \n                    )", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0035", "text": ")\n                  \n                  )\n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    \n                      \u2202\n                      \n                        \n                          E\n                        \n                      \n                    \n                    \n                      \u2202", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0036", "text": "\u2202\n                      \n                        x\n                        \n                          L\n                        \n                      \n                    \n                  \n                \n                +\n                \n                  \n                    \n                      \u2202\n                      \n                        \n                          E", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0037", "text": "\u2202\n                      \n                        x\n                        \n                          L\n                        \n                      \n                    \n                  \n                \n                \n                  \n                    \u2202\n                    \n                      \u2202\n                      \n                        x\n                        \n                          \u2113", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0038", "text": "\u2113\n                        \n                      \n                    \n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    \u2113\n                  \n                  \n                    L\n                    \u2212\n                    1\n                  \n                \n                F\n                (\n                \n                  x", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0039", "text": "x\n                  \n                    i\n                  \n                \n                )", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0040", "text": "{\\displaystyle {\\begin{aligned}{\\frac {\\partial {\\mathcal {E}}}{\\partial x_{\\ell }}}&={\\frac {\\partial {\\mathcal {E}}}{\\partial x_{L}}}{\\frac {\\partial x_{L}}{\\partial x_{\\ell }}}\\\\&={\\frac {\\partial {\\mathcal {E}}}{\\partial x_{L}}}\\left(1+{\\frac {\\partial }{\\partial x_{\\ell }}}\\sum _{i=\\ell }^{L-1}F(x_{i})\\right)\\\\&={\\frac {\\partial {\\mathcal {E}}}{\\partial x_{L}}}+{\\frac {\\partial {\\mathcal {E}}}{\\partial x_{L}}}{\\frac {\\partial }{\\partial x_{\\ell }}}\\sum _{i=\\ell", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0041", "text": "{\\partial }{\\partial x_{\\ell }}}\\sum _{i=\\ell }^{L-1}F(x_{i})\\end{aligned}}}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0042", "text": "This formulation suggests that the gradient computation of a shallower layer, \n  \n    \n      \n        \n          \n            \n              \u2202\n              \n                \n                  E\n                \n              \n            \n            \n              \u2202\n              \n                x\n                \n                  \u2113\n                \n              \n            \n          \n        \n      \n    \n    {\\textstyle {\\frac {\\partial {\\mathcal {E}}}{\\partial x_{\\ell }}}}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0043", "text": ", always has a later term \n  \n    \n      \n        \n          \n            \n              \u2202\n              \n                \n                  E\n                \n              \n            \n            \n              \u2202\n              \n                x\n                \n                  L\n                \n              \n            \n          \n        \n      \n    \n    {\\textstyle {\\frac {\\partial {\\mathcal {E}}}{\\partial x_{L}}}}\n  \n that is directly added. Even if the gradients of the", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0044", "text": "F\n        (\n        \n          x\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle F(x_{i})}\n  \n terms are small, the total gradient \n  \n    \n      \n        \n          \n            \n              \u2202\n              \n                \n                  E\n                \n              \n            \n            \n              \u2202\n              \n                x\n                \n                  \u2113", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0045", "text": "{\\textstyle {\\frac {\\partial {\\mathcal {E}}}{\\partial x_{\\ell }}}}\n  \n resists vanishing due to the added term \n  \n    \n      \n        \n          \n            \n              \u2202\n              \n                \n                  E\n                \n              \n            \n            \n              \u2202\n              \n                x\n                \n                  L", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0046", "text": "{\\textstyle {\\frac {\\partial {\\mathcal {E}}}{\\partial x_{L}}}}\n  \n.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0047", "text": "Variants of residual blocks\nBasic block\nA basic block is the simplest building block studied in the original ResNet. This block consists of two sequential 3x3 convolutional layers and a residual connection. The input and output dimensions of both layers are equal.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0048", "text": "Bottleneck block\nA bottleneck block consists of three sequential convolutional layers and a residual connection. The first layer in this block is a 1\u00d71 convolution for dimension reduction (e.g., to 1/2 of the input dimension); the second layer performs a 3\u00d73 convolution; the last layer is another 1\u00d71 convolution for dimension restoration. The models of ResNet-50, ResNet-101, and ResNet-152 are all based on bottleneck blocks.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0049", "text": "Pre-activation block\nThe pre-activation residual block applies activation functions before applying the residual function \n  \n    \n      \n        F\n      \n    \n    {\\displaystyle F}\n  \n. Formally, the computation of a pre-activation residual block can be written as:", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0050", "text": "x\n          \n            \u2113\n            +\n            1\n          \n        \n        =\n        F\n        (\n        \u03d5\n        (\n        \n          x\n          \n            \u2113\n          \n        \n        )\n        )\n        +\n        \n          x\n          \n            \u2113\n          \n        \n      \n    \n    {\\displaystyle x_{\\ell +1}=F(\\phi (x_{\\ell }))+x_{\\ell }}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0051", "text": "where \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0052", "text": "can be any activation (e.g. ReLU) or normalization (e.g. LayerNorm) operation. This design reduces the number of non-identity mappings between residual blocks, and allows an identity mapping directly from the input to the output. This design was used to train models with 200 to over 1000 layers, and was found to consistently outperform variants where the residual path is not an identity function. The pre-activation ResNet with 200 layers took 3 weeks to train for ImageNet on 8 GPUs in 2016.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0053", "text": "Since GPT-2, transformer blocks have been mostly implemented as pre-activation blocks. This is often referred to as \"pre-normalization\" in the literature of transformer models.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0054", "text": "Applications\nOriginally, ResNet was designed for computer vision.\nAll transformer architectures include residual connections. Indeed, very deep transformers cannot be trained without them.\nThe original ResNet paper made no claim on being inspired by biological systems. However, later research has related ResNet to biologically-plausible algorithms.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0055", "text": "A study published in Science in 2023 disclosed the complete connectome of an insect brain (specifically that of a fruit fly larva). This study discovered \"multilayer shortcuts\" that resemble the skip connections in artificial neural networks, including ResNets.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0056", "text": "History\nPrevious work\nResidual connections were noticed in neuroanatomy, such as Lorente de No (1938). McCulloch and Pitts (1943) proposed artificial neural networks and considered those with residual connections.\nIn 1961, Frank Rosenblatt described a three-layer multilayer perceptron (MLP) model with skip connections. The model was referred to as a \"cross-coupled system\", and the skip connections were forms of cross-coupled connections.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0057", "text": "During the late 1980s, \"skip-layer\" connections were sometimes used in neural networks. Examples include: Lang and Witbrock (1988) trained a fully connected feedforward network where each layer skip-connects to all subsequent layers, like the later DenseNet (2016). In this work, the residual connection was the form \n  \n    \n      \n        x\n        \u21a6\n        F\n        (\n        x\n        )\n        +\n        P\n        (\n        x\n        )\n      \n    \n    {\\displaystyle x\\mapsto F(x)+P(x)}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0058", "text": "{\\displaystyle x\\mapsto F(x)+P(x)}\n  \n, where \n  \n    \n      \n        P\n      \n    \n    {\\displaystyle P}\n  \n is a randomly-initialized projection connection. They termed it a \"short-cut connection\". An early neural language model used residual connections and named them \"direct connections\".", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0059", "text": "Degradation problem\nSepp Hochreiter discovered the vanishing gradient problem in 1991 and argued that it explained why the then-prevalent forms of recurrent neural networks did not work for long sequences. He and Schmidhuber later designed the LSTM architecture to solve this problem, which has a \"cell state\" \n  \n    \n      \n        \n          c\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle c_{t}}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0060", "text": "{\\displaystyle c_{t}}\n  \n that can function as a generalized residual connection. The highway network (2015) applied the idea of an LSTM unfolded in time to feedforward neural networks, resulting in the highway network. ResNet is equivalent to an open-gated highway network.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0061", "text": "During the early days of deep learning, there were attempts to train increasingly deep models. Notable examples included the AlexNet (2012), which had 8 layers, and the VGG-19 (2014), which had 19 layers. However, stacking too many layers led to a steep reduction in training accuracy, known as the \"degradation\" problem. In theory, adding additional layers to deepen a network should not result in a higher training loss, but this is what happened with VGGNet", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0062", "text": ". If the extra layers can be set as identity mappings, however, then the deeper network would represent the same function as its shallower counterpart. There is some evidence that the optimizer is not able to approach identity mappings for the parameterized layers, and the benefit of residual connections was to allow identity mappings by default.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0063", "text": "In 2014, the state of the art was training deep neural networks with 20 to 30 layers. The research team for ResNet attempted to train deeper ones by empirically testing various methods for training deeper networks, until they came upon the ResNet architecture.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0064", "text": "Subsequent work\nWide Residual Network (2016) found that using more channels and fewer layers than the original ResNet improves performance and GPU-computational efficiency, and that a block with two 3\u00d73 convolutions is superior to other configurations of convolution blocks.\nDenseNet (2016) connects the output of each layer to the input to each subsequent layer:", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0065", "text": "x\n          \n            \u2113\n            +\n            1\n          \n        \n        =\n        F\n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            \u2113\n            \u2212\n            1\n          \n        \n        ,\n        \n          x\n          \n            \u2113\n          \n        \n        )", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0066", "text": ")\n      \n    \n    {\\displaystyle x_{\\ell +1}=F(x_{1},x_{2},\\dots ,x_{\\ell -1},x_{\\ell })}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0067", "text": "Stochastic depth is a regularization method that randomly drops a subset of layers and lets the signal propagate through the identity skip connections. Also known as DropPath, this regularizes training for deep models, such as vision transformers.\nResNeXt (2017) combines the Inception module with ResNet.\nSqueeze-and-Excitation Networks (2018) added squeeze-and-excitation (SE) modules to ResNet. An SE module is applied after a convolution, and takes a tensor of shape", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0068", "text": "R\n          \n          \n            H\n            \u00d7\n            W\n            \u00d7\n            C\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {R} ^{H\\times W\\times C}}\n  \n (height, width, channels) as input. Each channel is averaged, resulting in a vector of shape \n  \n    \n      \n        \n          \n            R\n          \n          \n            C\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {R} ^{C}}", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0069", "text": "{\\displaystyle \\mathbb {R} ^{C}}\n  \n. This is then passed through a multilayer perceptron (with an architecture such as linear-ReLU-linear-sigmoid) before it is multiplied with the original tensor. It won the ILSVRC in 2017.", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Residual_neural_network_0070", "text": "== References ==", "title": "Residual neural network", "url": "https://en.wikipedia.org/wiki/Residual_neural_network"}, {"id": "rec_Graph_neural_network_0000", "text": "Graph neural networks (GNN) are specialized artificial neural networks that are designed for tasks whose inputs are graphs.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0001", "text": "One prominent example is molecular drug design.  Each input sample is a graph representation of a molecule, where atoms form the nodes and chemical bonds between atoms form the edges.  In addition to the graph representation, the input also includes known chemical properties for each of the atoms.  Dataset samples may thus differ in length, reflecting the varying numbers of atoms in molecules, and the varying number of bonds between them", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0002", "text": ". The task is to predict the efficacy of a given molecule for a specific medical application, like eliminating E. coli bacteria.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0003", "text": "The key design element of GNNs is the use of pairwise message passing, such that graph nodes iteratively update their representations by exchanging information with their neighbors. Several GNN architectures have been proposed, which implement different flavors of message passing,  started by recursive or convolutional constructive approaches", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0004", "text": ". As of 2022, it is an open question whether it is possible to define GNN architectures \"going beyond\" message passing, or instead every GNN can be built on message passing over suitably defined graphs.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0005", "text": "In the more general subject of \"geometric deep learning\", certain existing neural network architectures can be interpreted as GNNs operating on suitably defined graphs. A convolutional neural network layer, in the context of computer vision, can be considered a GNN applied to graphs whose nodes are pixels and only adjacent pixels are connected by edges in the graph", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0006", "text": ". A transformer layer, in natural language processing, can be considered a GNN applied to complete graphs whose nodes are words or tokens in a passage of natural language text.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0007", "text": "Relevant application domains for GNNs include natural language processing, social networks, citation networks, molecular biology, chemistry, physics and NP-hard combinatorial optimization problems.\nOpen source libraries implementing GNNs include PyTorch Geometric (PyTorch), TensorFlow GNN (TensorFlow), Deep Graph Library (framework agnostic), jraph (Google JAX), and GraphNeuralNetworks.jl/GeometricFlux.jl (Julia, Flux).", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0008", "text": "Architecture\nThe architecture of a generic GNN implements the following fundamental layers:", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0009", "text": "Permutation equivariant: a permutation equivariant layer maps a representation of a graph into an updated representation of the same graph. In the literature, permutation equivariant layers are implemented via pairwise message passing between graph nodes. Intuitively, in a message passing layer, nodes update their representations by aggregating the messages received from their immediate neighbours. As such, each message passing layer increases the receptive field of the GNN by one hop.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0010", "text": "Local pooling: a local pooling layer coarsens the graph via downsampling. Local pooling is used to increase the receptive field of a GNN, in a similar fashion to pooling layers in convolutional neural networks. Examples include k-nearest neighbours pooling, top-k pooling, and self-attention pooling.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0011", "text": "Global pooling: a global pooling layer, also known as readout layer, provides fixed-size representation of the whole graph. The global pooling layer must be permutation invariant, such that permutations in the ordering of graph nodes and edges do not alter the final output. Examples include element-wise sum, mean or maximum.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0012", "text": "It has been demonstrated that GNNs cannot be more expressive than the Weisfeiler\u2013Leman Graph Isomorphism Test. In practice, this means that there exist different graph structures (e.g., molecules with the same atoms but different bonds) that cannot be distinguished by GNNs. More powerful GNNs operating on higher-dimension geometries such as simplicial complexes can be designed", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0013", "text": ". As of 2022, whether or not future architectures will overcome the message passing primitive is an open research question.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0014", "text": "Message passing layers\nMessage passing layers are permutation-equivariant layers mapping a graph into an updated representation of the same graph. Formally, they can be expressed as message passing neural networks (MPNNs).\nLet \n  \n    \n      \n        G\n        =\n        (\n        V\n        ,\n        E\n        )\n      \n    \n    {\\displaystyle G=(V,E)}\n  \n be a graph, where \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n is the node set and \n  \n    \n      \n        E", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0015", "text": "E\n      \n    \n    {\\displaystyle E}\n  \n is the edge set. Let \n  \n    \n      \n        \n          N\n          \n            u\n          \n        \n      \n    \n    {\\displaystyle N_{u}}\n  \n be the neighbourhood of some node \n  \n    \n      \n        u\n        \u2208\n        V\n      \n    \n    {\\displaystyle u\\in V}\n  \n. Additionally, let \n  \n    \n      \n        \n          \n            x\n          \n          \n            u", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0016", "text": "u\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{u}}\n  \n be the features of node \n  \n    \n      \n        u\n        \u2208\n        V\n      \n    \n    {\\displaystyle u\\in V}\n  \n, and \n  \n    \n      \n        \n          \n            e\n          \n          \n            u\n            v\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {e} _{uv}}\n  \n be the features of edge \n  \n    \n      \n        (\n        u\n        ,\n        v\n        )\n        \u2208\n        E", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0017", "text": "v\n        )\n        \u2208\n        E\n      \n    \n    {\\displaystyle (u,v)\\in E}\n  \n. An MPNN layer can be expressed as follows:", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0018", "text": "h\n          \n          \n            u\n          \n        \n        =\n        \u03d5\n        \n          (\n          \n            \n              \n                x\n              \n              \n                u\n              \n            \n            ,\n            \n              \u2a01\n              \n                v\n                \u2208\n                \n                  N\n                  \n                    u", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0019", "text": "\u03c8\n            (\n            \n              \n                x\n              \n              \n                u\n              \n            \n            ,\n            \n              \n                x\n              \n              \n                v\n              \n            \n            ,\n            \n              \n                e\n              \n              \n                u\n                v", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0020", "text": "v\n              \n            \n            )\n          \n          )\n        \n      \n    \n    {\\displaystyle \\mathbf {h} _{u}=\\phi \\left(\\mathbf {x} _{u},\\bigoplus _{v\\in N_{u}}\\psi (\\mathbf {x} _{u},\\mathbf {x} _{v},\\mathbf {e} _{uv})\\right)}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0021", "text": "where \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n and \n  \n    \n      \n        \u03c8\n      \n    \n    {\\displaystyle \\psi }\n  \n are differentiable functions (e.g., artificial neural networks), and \n  \n    \n      \n        \u2a01\n      \n    \n    {\\displaystyle \\bigoplus }\n  \n is a permutation invariant aggregation operator that can accept an arbitrary number of inputs (e.g., element-wise sum, mean, or max). In particular, \n  \n    \n      \n        \u03d5", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0022", "text": "\u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n and \n  \n    \n      \n        \u03c8\n      \n    \n    {\\displaystyle \\psi }\n  \n are referred to as update and message functions, respectively. Intuitively, in an MPNN computational block, graph nodes update their representations by aggregating the messages received from their neighbours.\nThe outputs of one or more MPNN layers are node representations \n  \n    \n      \n        \n          \n            h\n          \n          \n            u", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0023", "text": "h\n          \n          \n            u\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {h} _{u}}\n  \n for each node \n  \n    \n      \n        u\n        \u2208\n        V\n      \n    \n    {\\displaystyle u\\in V}\n  \n in the graph. Node representations can be employed for any downstream task, such as node/graph classification or edge prediction.\nGraph nodes in an MPNN update their representation aggregating information from their immediate neighbours. As such, stacking", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0024", "text": "n\n      \n    \n    {\\displaystyle n}\n  \n MPNN layers means that one node will be able to communicate with nodes that are at most \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0025", "text": "\"hops\" away. In principle, to ensure that every node receives information from every other node, one would need to stack a number of MPNN layers equal to the graph diameter. However, stacking many MPNN layers may cause issues such as oversmoothing and oversquashing. Oversmoothing refers to the issue of node representations becoming indistinguishable. Oversquashing refers to the bottleneck that is created by squeezing long-range dependencies into fixed-size representations", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0026", "text": ". Countermeasures such as skip connections (as in residual neural networks), gated update rules and jumping knowledge can mitigate oversmoothing. Modifying the final layer to be a fully-adjacent layer, i.e., by considering the graph as a complete graph, can mitigate oversquashing in problems where long-range dependencies are required.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0027", "text": "Other \"flavours\" of MPNN have been developed in the literature, such as graph convolutional networks and graph attention networks, whose definitions can be expressed in terms of the MPNN formalism.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0028", "text": "Graph convolutional network\nThe graph convolutional network (GCN) was first introduced by Thomas Kipf and Max Welling in 2017.\nA GCN layer defines a first-order approximation of a localized spectral filter on graphs. GCNs can be understood as a generalization of convolutional neural networks to graph-structured data.\nThe formal expression of a GCN layer reads as follows:", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0029", "text": "H\n        \n        =\n        \u03c3\n        \n          (\n          \n            \n              \n                \n                  \n                    \n                      D\n                    \n                    ~\n                  \n                \n              \n              \n                \u2212\n                \n                  \n                    1\n                    2", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0030", "text": "A\n                  \n                  ~\n                \n              \n            \n            \n              \n                \n                  \n                    \n                      D\n                    \n                    ~\n                  \n                \n              \n              \n                \u2212\n                \n                  \n                    1", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0031", "text": "1\n                    2\n                  \n                \n              \n            \n            \n              X\n            \n            \n              \u0398\n            \n          \n          )\n        \n      \n    \n    {\\displaystyle \\mathbf {H} =\\sigma \\left({\\tilde {\\mathbf {D} }}^{-{\\frac {1}{2}}}{\\tilde {\\mathbf {A} }}{\\tilde {\\mathbf {D} }}^{-{\\frac {1}{2}}}\\mathbf {X} \\mathbf {\\Theta } \\right)}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0032", "text": "where \n  \n    \n      \n        \n          H\n        \n      \n    \n    {\\displaystyle \\mathbf {H} }\n  \n is the matrix of node representations \n  \n    \n      \n        \n          \n            h\n          \n          \n            u\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {h} _{u}}\n  \n, \n  \n    \n      \n        \n          X\n        \n      \n    \n    {\\displaystyle \\mathbf {X} }\n  \n is the matrix of node features \n  \n    \n      \n        \n          \n            x", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0033", "text": "x\n          \n          \n            u\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{u}}\n  \n, \n  \n    \n      \n        \u03c3\n        (\n        \u22c5\n        )\n      \n    \n    {\\displaystyle \\sigma (\\cdot )}\n  \n is an activation function (e.g., ReLU), \n  \n    \n      \n        \n          \n            \n              \n                A\n              \n              ~\n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {\\mathbf {A} }}}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0034", "text": "{\\displaystyle {\\tilde {\\mathbf {A} }}}\n  \n is the graph adjacency matrix with the addition of self-loops, \n  \n    \n      \n        \n          \n            \n              \n                D\n              \n              ~\n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {\\mathbf {D} }}}\n  \n is the graph degree matrix with the addition of self-loops, and \n  \n    \n      \n        \n          \u0398\n        \n      \n    \n    {\\displaystyle \\mathbf {\\Theta } }", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0035", "text": "{\\displaystyle \\mathbf {\\Theta } }\n  \n is a matrix of trainable parameters.\nIn particular, let \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }\n  \n be the graph adjacency matrix: then, one can define \n  \n    \n      \n        \n          \n            \n              \n                A\n              \n              ~\n            \n          \n        \n        =\n        \n          A\n        \n        +\n        \n          I", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0036", "text": "I\n        \n      \n    \n    {\\displaystyle {\\tilde {\\mathbf {A} }}=\\mathbf {A} +\\mathbf {I} }\n  \n and \n  \n    \n      \n        \n          \n            \n              \n                \n                  D\n                \n                ~\n              \n            \n          \n          \n            i\n            i\n          \n        \n        =\n        \n          \u2211\n          \n            j\n            \u2208\n            V", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0037", "text": "A\n                ~\n              \n            \n          \n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle {\\tilde {\\mathbf {D} }}_{ii}=\\sum _{j\\in V}{\\tilde {A}}_{ij}}\n  \n, where \n  \n    \n      \n        \n          I\n        \n      \n    \n    {\\displaystyle \\mathbf {I} }\n  \n denotes the identity matrix. This normalization ensures that the eigenvalues of", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0038", "text": "D\n                \n                ~\n              \n            \n          \n          \n            \u2212\n            \n              \n                1\n                2\n              \n            \n          \n        \n        \n          \n            \n              \n                A\n              \n              ~", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0039", "text": "D\n                \n                ~\n              \n            \n          \n          \n            \u2212\n            \n              \n                1\n                2\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {\\mathbf {D} }}^{-{\\frac {1}{2}}}{\\tilde {\\mathbf {A} }}{\\tilde {\\mathbf {D} }}^{-{\\frac {1}{2}}}}\n  \n are bounded in the range \n  \n    \n      \n        [\n        0", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0040", "text": "[\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle [0,1]}\n  \n, avoiding numerical instabilities and exploding/vanishing gradients.\nA limitation of GCNs is that they do not allow multidimensional edge features \n  \n    \n      \n        \n          \n            e\n          \n          \n            u\n            v\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {e} _{uv}}\n  \n. It is however possible to associate scalar weights", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0041", "text": "w\n          \n            u\n            v\n          \n        \n      \n    \n    {\\displaystyle w_{uv}}\n  \n to each edge by imposing \n  \n    \n      \n        \n          A\n          \n            u\n            v\n          \n        \n        =\n        \n          w\n          \n            u\n            v\n          \n        \n      \n    \n    {\\displaystyle A_{uv}=w_{uv}}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0042", "text": "{\\displaystyle A_{uv}=w_{uv}}\n  \n, i.e., by setting each nonzero entry in the adjacency matrix equal to the weight of the corresponding edge.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0043", "text": "Graph attention network\nThe graph attention network (GAT) was introduced by Petar Veli\u010dkovi\u0107 et al. in 2018.\nA graph attention network is a combination of a GNN and an attention layer.\nThe implementation of attention layer in graphical neural networks helps provide attention or focus to the important information from the data instead of focusing on the whole data.\nA multi-head GAT layer can be expressed as follows:", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0044", "text": "h\n          \n          \n            u\n          \n        \n        =\n        \n          \n            \n              \n                \u2016\n              \n              \n                k\n                =\n                1\n              \n            \n            K\n          \n        \n        \u03c3\n        \n          (\n          \n            \n              \u2211\n              \n                v\n                \u2208\n                \n                  N", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0045", "text": "N\n                  \n                    u\n                  \n                \n              \n            \n            \n              \u03b1\n              \n                u\n                v\n              \n            \n            \n              \n                W\n              \n              \n                k\n              \n            \n            \n              \n                x\n              \n              \n                v", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0046", "text": "v\n              \n            \n          \n          )\n        \n      \n    \n    {\\displaystyle \\mathbf {h} _{u}={\\overset {K}{\\underset {k=1}{\\Big \\Vert }}}\\sigma \\left(\\sum _{v\\in N_{u}}\\alpha _{uv}\\mathbf {W} ^{k}\\mathbf {x} _{v}\\right)}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0047", "text": "where \n  \n    \n      \n        K\n      \n    \n    {\\displaystyle K}\n  \n is the number of attention heads, \n  \n    \n      \n        \n          \n            \u2016\n          \n        \n      \n    \n    {\\displaystyle {\\Big \\Vert }}\n  \n denotes vector concatenation, \n  \n    \n      \n        \u03c3\n        (\n        \u22c5\n        )\n      \n    \n    {\\displaystyle \\sigma (\\cdot )}\n  \n is an activation function (e.g., ReLU), \n  \n    \n      \n        \n          \u03b1\n          \n            i\n            j", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0048", "text": "i\n            j\n          \n        \n      \n    \n    {\\displaystyle \\alpha _{ij}}\n  \n are attention coefficients, and \n  \n    \n      \n        \n          W\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle W^{k}}\n  \n is a matrix of trainable parameters for the \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n-th attention head.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0049", "text": "{\\displaystyle k}\n  \n-th attention head.\nFor the final GAT layer, the outputs from each attention head are averaged before the application of the activation function. Formally, the final GAT layer can be written as:", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0050", "text": "h\n          \n          \n            u\n          \n        \n        =\n        \u03c3\n        \n          (\n          \n            \n              \n                1\n                K\n              \n            \n            \n              \u2211\n              \n                k\n                =\n                1\n              \n              \n                K\n              \n            \n            \n              \u2211\n              \n                v", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0051", "text": "\u2211\n              \n                v\n                \u2208\n                \n                  N\n                  \n                    u\n                  \n                \n              \n            \n            \n              \u03b1\n              \n                u\n                v\n              \n            \n            \n              \n                W\n              \n              \n                k\n              \n            \n            \n              \n                x", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0052", "text": "x\n              \n              \n                v\n              \n            \n          \n          )\n        \n      \n    \n    {\\displaystyle \\mathbf {h} _{u}=\\sigma \\left({\\frac {1}{K}}\\sum _{k=1}^{K}\\sum _{v\\in N_{u}}\\alpha _{uv}\\mathbf {W} ^{k}\\mathbf {x} _{v}\\right)}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0053", "text": "Attention in Machine Learning is a technique that mimics cognitive attention. In the context of learning on graphs, the attention coefficient \n  \n    \n      \n        \n          \u03b1\n          \n            u\n            v\n          \n        \n      \n    \n    {\\displaystyle \\alpha _{uv}}\n  \n measures how important is node \n  \n    \n      \n        u\n        \u2208\n        V\n      \n    \n    {\\displaystyle u\\in V}\n  \n to node \n  \n    \n      \n        v\n        \u2208\n        V", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0054", "text": "v\n        \u2208\n        V\n      \n    \n    {\\displaystyle v\\in V}\n  \n.\nNormalized attention coefficients are computed as follows:", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0055", "text": "\u03b1\n          \n            u\n            v\n          \n        \n        =\n        \n          \n            \n              exp\n              \u2061\n              (\n              \n                LeakyReLU\n              \n              \n                (\n                \n                  \n                    \n                      a\n                    \n                    \n                      T\n                    \n                  \n                  [", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0056", "text": "[\n                  \n                    W\n                  \n                  \n                    \n                      x\n                    \n                    \n                      u\n                    \n                  \n                  \u2016\n                  \n                    W\n                  \n                  \n                    \n                      x\n                    \n                    \n                      v", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0057", "text": "v\n                    \n                  \n                  \u2016\n                  \n                    \n                      e\n                    \n                    \n                      u\n                      v\n                    \n                  \n                  ]\n                \n                )\n              \n              )\n            \n            \n              \n                \u2211\n                \n                  z\n                  \u2208", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0058", "text": "\u2208\n                  \n                    N\n                    \n                      u\n                    \n                  \n                \n              \n              exp\n              \u2061\n              (\n              \n                LeakyReLU\n              \n              \n                (\n                \n                  \n                    \n                      a\n                    \n                    \n                      T", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0059", "text": "T\n                    \n                  \n                  [\n                  \n                    W\n                  \n                  \n                    \n                      x\n                    \n                    \n                      u\n                    \n                  \n                  \u2016\n                  \n                    W\n                  \n                  \n                    \n                      x", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0060", "text": "z\n                    \n                  \n                  \u2016\n                  \n                    \n                      e\n                    \n                    \n                      u\n                      z\n                    \n                  \n                  ]\n                \n                )\n              \n              )", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0061", "text": "{\\displaystyle \\alpha _{uv}={\\frac {\\exp({\\text{LeakyReLU}}\\left(\\mathbf {a} ^{T}[\\mathbf {W} \\mathbf {x} _{u}\\Vert \\mathbf {W} \\mathbf {x} _{v}\\Vert \\mathbf {e} _{uv}]\\right))}{\\sum _{z\\in N_{u}}\\exp({\\text{LeakyReLU}}\\left(\\mathbf {a} ^{T}[\\mathbf {W} \\mathbf {x} _{u}\\Vert \\mathbf {W} \\mathbf {x} _{z}\\Vert \\mathbf {e} _{uz}]\\right))}}}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0062", "text": "where \n  \n    \n      \n        \n          a\n        \n      \n    \n    {\\displaystyle \\mathbf {a} }\n  \n is a vector of learnable weights, \n  \n    \n      \n        \n          \u22c5\n          \n            T\n          \n        \n      \n    \n    {\\displaystyle \\cdot ^{T}}\n  \n indicates transposition, \n  \n    \n      \n        \n          \n            e\n          \n          \n            u\n            v\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {e} _{uv}}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0063", "text": "{\\displaystyle \\mathbf {e} _{uv}}\n  \n are the edge features (if present), and \n  \n    \n      \n        \n          LeakyReLU\n        \n      \n    \n    {\\displaystyle {\\text{LeakyReLU}}}\n  \n is a modified ReLU activation function. Attention coefficients are normalized to make them easily comparable across different nodes.\nA GCN can be seen as a special case of a GAT where attention coefficients are not learnable, but fixed and equal to the edge weights \n  \n    \n      \n        \n          w", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0064", "text": "w\n          \n            u\n            v\n          \n        \n      \n    \n    {\\displaystyle w_{uv}}\n  \n.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0065", "text": "Gated graph sequence neural network\nThe gated graph sequence neural network (GGS-NN) was introduced by Yujia Li et al. in 2015. The GGS-NN extends the GNN formulation by Scarselli et al. to output sequences. The message passing framework is implemented as an update rule to a gated recurrent unit (GRU) cell.\nA GGS-NN can be expressed as follows:", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0066", "text": "h\n          \n          \n            u\n          \n          \n            (\n            0\n            )\n          \n        \n        =\n        \n          \n            x\n          \n          \n            u\n          \n        \n        \n        \u2016\n        \n        \n          0\n        \n      \n    \n    {\\displaystyle \\mathbf {h} _{u}^{(0)}=\\mathbf {x} _{u}\\,\\Vert \\,\\mathbf {0} }", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0067", "text": "m\n          \n          \n            u\n          \n          \n            (\n            l\n            +\n            1\n            )\n          \n        \n        =\n        \n          \u2211\n          \n            v\n            \u2208\n            \n              N\n              \n                u\n              \n            \n          \n        \n        \n          \u0398\n        \n        \n          \n            h\n          \n          \n            v", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0068", "text": "v\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {m} _{u}^{(l+1)}=\\sum _{v\\in N_{u}}\\mathbf {\\Theta } \\mathbf {h} _{v}}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0069", "text": "h\n          \n          \n            u\n          \n          \n            (\n            l\n            +\n            1\n            )\n          \n        \n        =\n        \n          GRU\n        \n        (\n        \n          \n            m\n          \n          \n            u\n          \n          \n            (\n            l\n            +\n            1\n            )\n          \n        \n        ,\n        \n          \n            h", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0070", "text": "h\n          \n          \n            u\n          \n          \n            (\n            l\n            )\n          \n        \n        )\n      \n    \n    {\\displaystyle \\mathbf {h} _{u}^{(l+1)}={\\text{GRU}}(\\mathbf {m} _{u}^{(l+1)},\\mathbf {h} _{u}^{(l)})}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0071", "text": "where \n  \n    \n      \n        \u2016\n      \n    \n    {\\displaystyle \\Vert }\n  \n denotes vector concatenation, \n  \n    \n      \n        \n          0\n        \n      \n    \n    {\\displaystyle \\mathbf {0} }\n  \n is a vector of zeros, \n  \n    \n      \n        \n          \u0398\n        \n      \n    \n    {\\displaystyle \\mathbf {\\Theta } }\n  \n is a matrix of learnable parameters, \n  \n    \n      \n        \n          GRU\n        \n      \n    \n    {\\displaystyle {\\text{GRU}}}\n  \n is a GRU cell, and", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0072", "text": "is a GRU cell, and \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n denotes the sequence index. In a GGS-NN, the node representations are regarded as the hidden states of a GRU cell. The initial node features \n  \n    \n      \n        \n          \n            x\n          \n          \n            u\n          \n          \n            (\n            0\n            )\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{u}^{(0)}}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0073", "text": "{\\displaystyle \\mathbf {x} _{u}^{(0)}}\n  \n are zero-padded up to the hidden state dimension of the GRU cell. The same GRU cell is used for updating representations for each node.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0074", "text": "Local pooling layers\nLocal pooling layers coarsen the graph via downsampling. Subsequently, several learnable local pooling strategies that have been proposed are presented. For each case, the input is the initial graph represented by a matrix \n  \n    \n      \n        \n          X\n        \n      \n    \n    {\\displaystyle \\mathbf {X} }\n  \n of node features, and the graph adjacency matrix \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0075", "text": "{\\displaystyle \\mathbf {A} }\n  \n. The output is the new matrix \n  \n    \n      \n        \n          \n            X\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\mathbf {X} '}\n  \nof node features, and the new graph adjacency matrix \n  \n    \n      \n        \n          \n            A\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\mathbf {A} '}\n  \n.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0076", "text": "Top-k pooling\nWe first set\n\n  \n    \n      \n        \n          y\n        \n        =\n        \n          \n            \n              \n                X\n              \n              \n                p\n              \n            \n            \n              \u2016\n              \n                p\n              \n              \u2016\n            \n          \n        \n      \n    \n    {\\displaystyle \\mathbf {y} ={\\frac {\\mathbf {X} \\mathbf {p} }{\\Vert \\mathbf {p} \\Vert }}}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0077", "text": "where \n  \n    \n      \n        \n          p\n        \n      \n    \n    {\\displaystyle \\mathbf {p} }\n  \n is a learnable projection vector. The projection vector \n  \n    \n      \n        \n          p\n        \n      \n    \n    {\\displaystyle \\mathbf {p} }\n  \n computes a scalar projection value for each graph node.\nThe top-k pooling layer  can then be formalised as follows:", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0078", "text": "X\n          \n          \u2032\n        \n        =\n        (\n        \n          X\n        \n        \u2299\n        \n          sigmoid\n        \n        (\n        \n          y\n        \n        )\n        \n          )\n          \n            \n              i\n            \n          \n        \n      \n    \n    {\\displaystyle \\mathbf {X} '=(\\mathbf {X} \\odot {\\text{sigmoid}}(\\mathbf {y} ))_{\\mathbf {i} }}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0079", "text": "A\n          \n          \u2032\n        \n        =\n        \n          \n            A\n          \n          \n            \n              i\n            \n            ,\n            \n              i\n            \n          \n        \n      \n    \n    {\\displaystyle \\mathbf {A} '=\\mathbf {A} _{\\mathbf {i} ,\\mathbf {i} }}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0080", "text": "where \n  \n    \n      \n        \n          i\n        \n        =\n        \n          \n            top\n          \n          \n            k\n          \n        \n        (\n        \n          y\n        \n        )\n      \n    \n    {\\displaystyle \\mathbf {i} ={\\text{top}}_{k}(\\mathbf {y} )}\n  \n is the subset of nodes with the top-k highest projection scores, \n  \n    \n      \n        \u2299\n      \n    \n    {\\displaystyle \\odot }\n  \n denotes element-wise matrix multiplication, and", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0081", "text": "sigmoid\n        \n        (\n        \u22c5\n        )\n      \n    \n    {\\displaystyle {\\text{sigmoid}}(\\cdot )}\n  \n is the sigmoid function. In other words, the nodes with the top-k highest projection scores are retained in the new adjacency matrix \n  \n    \n      \n        \n          \n            A\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\mathbf {A} '}\n  \n. The \n  \n    \n      \n        \n          sigmoid\n        \n        (\n        \u22c5\n        )", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0082", "text": "(\n        \u22c5\n        )\n      \n    \n    {\\displaystyle {\\text{sigmoid}}(\\cdot )}\n  \n operation makes the projection vector \n  \n    \n      \n        \n          p\n        \n      \n    \n    {\\displaystyle \\mathbf {p} }\n  \n trainable by backpropagation, which otherwise would produce discrete outputs.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0083", "text": "Self-attention pooling\nWe first set\n\n  \n    \n      \n        \n          y\n        \n        =\n        \n          GNN\n        \n        (\n        \n          X\n        \n        ,\n        \n          A\n        \n        )\n      \n    \n    {\\displaystyle \\mathbf {y} ={\\text{GNN}}(\\mathbf {X} ,\\mathbf {A} )}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0084", "text": "where \n  \n    \n      \n        \n          GNN\n        \n      \n    \n    {\\displaystyle {\\text{GNN}}}\n  \n is a generic permutation equivariant GNN layer (e.g., GCN, GAT, MPNN).\nThe Self-attention pooling layer can then be formalised as follows:", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0085", "text": "X\n          \n          \u2032\n        \n        =\n        (\n        \n          X\n        \n        \u2299\n        \n          y\n        \n        \n          )\n          \n            \n              i\n            \n          \n        \n      \n    \n    {\\displaystyle \\mathbf {X} '=(\\mathbf {X} \\odot \\mathbf {y} )_{\\mathbf {i} }}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0086", "text": "A\n          \n          \u2032\n        \n        =\n        \n          \n            A\n          \n          \n            \n              i\n            \n            ,\n            \n              i\n            \n          \n        \n      \n    \n    {\\displaystyle \\mathbf {A} '=\\mathbf {A} _{\\mathbf {i} ,\\mathbf {i} }}", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0087", "text": "where \n  \n    \n      \n        \n          i\n        \n        =\n        \n          \n            top\n          \n          \n            k\n          \n        \n        (\n        \n          y\n        \n        )\n      \n    \n    {\\displaystyle \\mathbf {i} ={\\text{top}}_{k}(\\mathbf {y} )}\n  \n is the subset of nodes with the top-k highest projection scores, \n  \n    \n      \n        \u2299\n      \n    \n    {\\displaystyle \\odot }\n  \n denotes element-wise matrix multiplication.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0088", "text": "denotes element-wise matrix multiplication.\nThe self-attention pooling layer can be seen as an extension of the top-k pooling layer. Differently from top-k pooling, the self-attention scores computed in self-attention pooling account both for the graph features and the graph topology.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0089", "text": "Heterophilic Graph Learning", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0090", "text": "Homophily principle, i.e., nodes with the same labels or similar attributes are more likely to be connected, has been commonly believed to be the main reason for the superiority of Graph Neural Networks (GNNs) over traditional Neural Networks (NNs) on graph-structured data, especially on node-level tasks. However, recent work has identified a non-trivial set of datasets where GNN's performance compared to the NN's is not satisfactory", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0091", "text": ". Heterophily, i.e., low homophily, has been considered the main cause of this empirical observation. People have begun to revisit and re-evaluate most existing graph models in the heterophily scenario across various kinds of graphs, e.g., heterogeneous graphs, temporal graphs and hypergraphs. Moreover, numerous graph-related applications are found to be closely related to the heterophily problem, e.g", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0092", "text": ". graph fraud/anomaly detection, graph adversarial attacks and robustness, privacy, federated learning and point cloud segmentation, graph clustering, recommender systems, generative models, link prediction, graph classification and coloring, etc. In the past few years, considerable effort has been devoted to studying and addressing the heterophily issue in graph learning.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0093", "text": "Applications\nProtein folding\nGraph neural networks are one of the main building blocks of AlphaFold, an artificial intelligence program developed by Google's DeepMind for solving the protein folding problem in biology. AlphaFold achieved first place in several CASP competitions.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0094", "text": "Social networks\nSocial networks are a major application domain for GNNs due to their natural representation as social graphs. GNNs are used to develop recommender systems based on both social relations and item relations.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0095", "text": "Combinatorial optimization\nGNNs are used as fundamental building blocks for several combinatorial optimization algorithms. Examples include computing shortest paths or Eulerian circuits for a given graph, deriving chip placements superior or competitive to handcrafted human solutions, and improving expert-designed branching rules in branch and bound.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0096", "text": "Cyber security\nWhen viewed as a graph, a network of computers can be analyzed with GNNs for anomaly detection. Anomalies within provenance graphs often correlate to malicious activity within the network. GNNs have been used to identify these anomalies on individual nodes and within paths to detect malicious processes, or on the edge level to detect lateral movement.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0097", "text": "Water distribution networks\nWater distribution systems can be modelled as graphs, being then a straightforward application of GNN. This kind of algorithm has been applied to water demand forecasting, interconnecting District Metered Areas(DMAs) to improve the forecasting capacity. Other application of this algorithm on water distribution modelling is the development of metamodels.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0098", "text": "Computer Vision\nTo represent an image as a graph structure, the image is first divided into multiple patches, each of which is treated as a node in the graph. Edges are then formed by connecting each node to its nearest neighbors based on spatial or feature similarity. This graph-based representation enables the application of graph learning models to visual tasks. The relational structure helps to enhance feature extraction and improve performance on image understanding.", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Graph_neural_network_0099", "text": "Text and NLP\nGraph-based representation of text helps to capture deeper semantic relationships between words. Many studies have used graph networks to enhance performance in various text processing tasks such as text classification, question answering, Neural Machine Translation (NMT), event extraction, fact verification, etc.\n\nReferences\nExternal links\nA Gentle Introduction to Graph Neural Networks", "title": "Graph neural network", "url": "https://en.wikipedia.org/wiki/Graph_neural_network"}, {"id": "rec_Transformer_deep_learning_0000", "text": "In deep learning, the transformer is a family of artificial neural network architectures based on the multi-head attention mechanism, in which text is converted to numerical representations called tokens, and each token is converted into a vector via lookup from a word embedding table", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0001", "text": ". At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0002", "text": "Transformers have the advantage of having no recurrent units, therefore requiring less training time than earlier recurrent neural architectures (RNNs) such as long short-term memory (LSTM). Later variations have been widely adopted for training large language models (LLMs) on large (language) datasets.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0003", "text": "The original version of the transformer architecture was proposed in the 2017 paper \"Attention Is All You Need\" by researchers at Google. The predecessors of transformers were developed as an improvement over previous architectures for machine translation, but have found many applications since. They are used in large-scale natural language processing, computer vision (vision transformers), reinforcement learning, audio, multimodal learning, robotics, and playing chess", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0004", "text": ". It has also led to the development of pre-trained systems, such as generative pre-trained transformers (GPTs) and BERT (bidirectional encoder representations from transformers).", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0005", "text": "History\nPredecessors\nFor many years, sequence modelling and generation was done by using plain recurrent neural networks (RNNs). A well-cited early example was the Elman network (1990). In theory, the information from one token can propagate arbitrarily far down the sequence, but in practice the vanishing-gradient problem leaves the model's state at the end of a long sentence without precise, extractable information about preceding tokens.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0006", "text": "A key breakthrough was LSTM (1995), an RNN which used various innovations to overcome the vanishing gradient problem, allowing efficient learning of long-sequence modelling. One key innovation was the use of an attention mechanism which used neurons that multiply the outputs of other neurons, so-called multiplicative units. Neural networks using multiplicative units were later called sigma-pi networks or higher-order networks", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0007", "text": ". LSTM became the standard architecture for long sequence modelling until the 2017 publication of transformers. However, LSTM still used sequential processing, like most other RNNs. Specifically, RNNs operate one token at a time from first to last; they cannot operate in parallel over all tokens in a sequence.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0008", "text": "Modern transformers overcome this problem, but unlike RNNs, they require computation time that is quadratic in the size of the context window. The linearly scaling fast weight controller (1992) learns to compute a weight matrix for further processing depending on the input. One of its two networks has \"fast weights\" or \"dynamic links\" (1981)", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0009", "text": ". A slow neural network learns by gradient descent to generate keys and values for computing the weight changes of the fast neural network which computes answers to queries. This was later shown to be equivalent to the unnormalized linear transformer.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0010", "text": "Attention with seq2seq\nThe idea of encoder\u2013decoder sequence transduction had been developed in the early 2010s; commonly cited as the originators that produced seq2seq are two concurrently published papers from 2014.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0011", "text": "A 380M-parameter model for machine translation uses two long short-term memories (LSTM). Its architecture consists of two parts. The encoder is an LSTM that takes in a sequence of tokens and turns it into a vector. The decoder is another LSTM that converts the vector into a sequence of tokens. Similarly, another 130M-parameter model used gated recurrent units (GRU) instead of LSTM. Later research showed that GRUs are neither better nor worse than LSTMs for seq2seq.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0012", "text": "These early seq2seq models had no attention mechanism, and the state vector is accessible only after the last word of the source text was processed. Although in theory such a vector retains the information about the whole original sentence, in practice the information is poorly preserved. This is because the input is processed sequentially by one recurrent network into a fixed-size output vector, which is then processed by another recurrent network into an output", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0013", "text": ". If the input is long, then the output vector would not be able to contain all relevant information, degrading the output. As evidence, reversing the input sentence improved seq2seq translation.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0014", "text": "The RNN search model introduced an attention mechanism to seq2seq for machine translation to solve the bottleneck problem (of the fixed-size output vector), allowing the model to process long-distance dependencies more easily. The name is because it \"emulates searching through a source sentence during decoding a translation\".", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0015", "text": "The relative performances were compared between global (that of RNN search) and local (sliding window) attention model architectures for machine translation, finding that mixed attention had higher quality than global attention, while local attention reduced translation time.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0016", "text": "In 2016, Google Translate was revamped to Google Neural Machine Translation, which replaced the previous model based on statistical machine translation. The new model was a seq2seq model where the encoder and the decoder were both 8 layers of bidirectional LSTM. It took nine months to develop, and it outperformed the statistical approach, which took ten years to develop.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0017", "text": "Parallelizing attention", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0018", "text": "Seq2seq models with attention (including self-attention) still suffered from the same issue with recurrent networks, which is that they are hard to parallelize, which prevented them from being accelerated on GPUs. In 2016, decomposable attention applied a self-attention mechanism to feedforward networks, which are easy to parallelize, and achieved SOTA result in textual entailment with an order of magnitude fewer parameters than LSTMs", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0019", "text": ". One of its authors, Jakob Uszkoreit, suspected that attention without recurrence would be sufficient for language translation, thus the title \"attention is all you need\". That hypothesis was against conventional wisdom at the time, and even his father Hans Uszkoreit, a well-known computational linguist, was skeptical. In the same year, self-attention (called intra-attention or intra-sentence attention) was proposed for LSTMs.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0020", "text": "On 2017-06-12, the original (100M-parameter) encoder\u2013decoder transformer model was published in the \"Attention is all you need\" paper. At the time, the focus of the research was on improving seq2seq for machine translation, by removing its recurrence to process all tokens in parallel, but preserving its dot-product attention mechanism to keep its text processing performance", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0021", "text": ". This led to the introduction of a multi-head attention model that was easier to parallelize due to the use of independent heads and the lack of recurrence. Its parallelizability was an important factor to its widespread use in large neural networks.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0022", "text": "AI boom era\nAs early as spring 2017, even before the \"Attention is all you need\" preprint was published, one of the co-authors applied the \"decoder-only\" variation of the architecture to generate fictitious Wikipedia articles. Transformer architecture is now used alongside many generative models that contribute to the ongoing AI boom.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0023", "text": "The \"reference implementation\" of the original Transformer was written in a TensorFlow library. In language modelling, ELMo (2018) was a bi-directional LSTM that produces contextualized word embeddings, improving upon the line of research from bag of words and word2vec. It was followed by BERT (2018), an encoder-only transformer model. In October 2019, Google started using BERT to process search queries", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0024", "text": ". In 2020, Google Translate replaced the previous RNN-encoder\u2013RNN-decoder model by a transformer-encoder\u2013RNN-decoder model.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0025", "text": "Starting in 2018, the OpenAI GPT series of decoder-only transformers became state of the art in natural language generation. In the end of 2022, a chatbot based on GPT-3, ChatGPT, became unexpectedly popular, triggering a boom around large language models.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0026", "text": "Transformers have been applied in modalities beyond text. 4 days after the publication of \"Attention is All You Need\", a multimodal transformer architecture, MultiModel, was published by most authors of that paper. Other examples include the vision transformer, speech recognition, robotics, and multimodal. The vision transformer, in turn, stimulated new developments in convolutional neural networks", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0027", "text": ". Image and video generators like DALL-E (2021), Stable Diffusion 3 (2024), and Sora (2024), use transformers to analyse input data (like text prompts) by breaking it down into \"tokens\" and then calculating the relevance between each token using self-attention, which helps the model understand the context and relationships within the data.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0028", "text": "Training\nMethods for stabilizing training\nThe plain transformer architecture had difficulty in converging. In the original paper, the authors recommended using learning rate warmup. That is, the learning rate should linearly scale up from 0 to maximal value for the first part of the training (usually recommended to be 2% of the total number of training steps), before decaying again.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0029", "text": "A 2020 paper found that using layer normalization before (instead of after) multihead attention and feedforward layers stabilizes training, not requiring learning rate warmup. This is the \"pre-LN Transformer\" and is more commonly used, compared to the original \"post-LN Transformer\".", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0030", "text": "Pretrain-finetune\nTransformers typically are first pretrained by self-supervised learning on a large generic dataset, followed by supervised fine-tuning on a small task-specific dataset. The pretrain dataset is typically an unlabeled large corpus, such as The Pile. Tasks for pretraining and fine-tuning commonly include:", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0031", "text": "language modeling\nnext-sentence prediction\nquestion answering\nreading comprehension\nsentiment analysis\nparaphrasing\nThe T5 transformer report documents a large number of natural language pretraining tasks. Some examples are:", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0032", "text": "restoring or repairing incomplete or corrupted text. For example, the input, \"Thank you\u202f~~\u202fme to your party\u202f~~\u202fweek\", might generate the output, \"Thank you for inviting me to your party last week\".\ntranslation between natural languages (machine translation)", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0033", "text": "judging the pragmatic acceptability of natural language. For example, the following sentence might be judged \"not acceptable\", because even though it is syntactically well-formed, it is improbable in ordinary human usage: The course is jumping well.\nNote that while each of these tasks is trivial or obvious for human native speakers of the language (or languages), they have typically proved challenging for previous generations of machine learning architecture.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0034", "text": "Tasks\nIn general, there are 3 classes of language modelling tasks: \"masked\", \"autoregressive\", and \"prefixLM\". These classes are independent of a specific modeling architecture such as transformer, but they are often discussed in the context of transformer.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0035", "text": "In a masked task, one or more of the tokens is masked out, and the model would produce a probability distribution predicting what the masked-out tokens are based on the context. The loss function for the task is typically sum of log-perplexities for the masked-out tokens: \n  \n    \n      \n        \n          Loss\n        \n        =\n        \u2212\n        \n          \u2211\n          \n            t\n            \u2208\n            \n              masked tokens\n            \n          \n        \n        ln\n        \u2061", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0036", "text": "ln\n        \u2061\n        (\n        \n          probability of \n        \n        t\n        \n           conditional on its context\n        \n        )\n      \n    \n    {\\displaystyle {\\text{Loss}}=-\\sum _{t\\in {\\text{masked tokens}}}\\ln({\\text{probability of }}t{\\text{ conditional on its context}})}\n  \nand the model is trained to minimize this loss function. The BERT series of models are trained for masked token prediction and another task.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0037", "text": "In an autoregressive task, the entire sequence is masked at first, and the model produces a probability distribution for the first token. Then the first token is revealed and the model predicts the second token, and so on. The loss function for the task is still typically the same. The GPT series of models are trained by autoregressive tasks.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0038", "text": "In a prefixLM task, the sequence is divided into two parts. The first part is presented as context, and the model predicts the first token of the second part. Then that would be revealed, and the model predicts the second token, and so on. The loss function for the task is still typically the same. The T5 series of models are trained by prefixLM tasks.\nNote that \"masked\" as in \"masked language modelling\" is not \"masked\" as in \"masked attention\", and \"prefixLM\" as in", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0039", "text": "\"prefix language modeling\" is not \"prefixLM\" as in \" prefix language model\".", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0040", "text": "Architecture\nAll transformers have the same primary components:", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0041", "text": "Tokenizers, which convert text into tokens.\nEmbedding layer, which converts tokens and positions of the tokens into vector representations.\nTransformer layers, which carry out repeated transformations on the vector representations, extracting more and more linguistic information. These consist of alternating attention and feedforward layers. There are two major types of transformer layers: encoder layers and decoder layers, with further variants.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0042", "text": "Un-embedding layer, which converts the final vector representations back to a probability distribution over the tokens.\nThe following description follows exactly the transformer as described in the original paper. There are variants, described in the following section.\nBy convention, we write all vectors as row vectors. For example, pushing a vector through a linear layer means multiplying it by a weight matrix on the right, as \n  \n    \n      \n        x\n        W", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0043", "text": "x\n        W\n      \n    \n    {\\displaystyle xW}\n  \n.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0044", "text": "Tokenization\nAs the transformer architecture natively consists of operations over numbers (matrix multiplications, dot products, activation functions) rather than over text, there must first be a mapping from any input text to some numerical representation. This happens in three steps.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0045", "text": "First, the input text is treated by a preprocessor, which performs both textual transformations and splits the text into coarse-grained segments called pretokens. The latter is referred to as pretokenization. Second, each pretoken is segmented further into tokens by a tokenizer that expects to only see pretokens output by its preprocessor. Each token it produces is a string of one or more characters belonging to a finite set of strings called the vocabulary \n  \n    \n      \n        V", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0046", "text": "V\n      \n    \n    {\\displaystyle V}\n  \n. Third, because the vocabulary is finite and known beforehand, each token can be assigned an integer identifier, and this mapping is applied to the sequence of tokens to represent any input text as a numerical sequence. Since this mapping is bijective, the output side can produce a sequence of integer identifiers which can then be turned back into tokens. After undoing some of the preprocessing, the result is again legible text.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0047", "text": "Training a tokenizer (sometimes referred to as vocabularization) means finding a suitable vocabulary \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n, but also learning how to use it, since any given string \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n of length \n  \n    \n      \n        \n          |\n        \n        s\n        \n          |\n        \n      \n    \n    {\\displaystyle |s|}\n  \n has \n  \n    \n      \n        \n          2", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0048", "text": "2\n          \n            \n              |\n            \n            s\n            \n              |\n            \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle 2^{|s|-1}}\n  \n hypothetical segmentations, some of which containing segments that are not in the vocabulary. The most important hyperparameter during vocabularization is the vocabulary size \n  \n    \n      \n        \n          |\n        \n        V\n        \n          |", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0049", "text": "|\n        \n      \n    \n    {\\displaystyle |V|}\n  \n: when it is small, the learned vocabulary generally consists of characters and smaller strings, and words will be segmented into many tokens. At larger sizes, it becomes affordable to dedicate tokens to full words, although depending on the preprocessor and tokenizer, it is not necessarily the case that large vocabularies will always use the largest token(s) available to segment a word.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0050", "text": "Because tokens are not always full words, they may also be referred to as subwords and tokenization algorithms may be referred to as subword tokenizers. This is also to differentiate these systems from traditional terminology used in older information retrieval and natural language processing systems, where \"tokenization\" was used to denote what is today called \"pretokenization\" (very crudely: splitting into words)", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0051", "text": ". In tokenizers that produce tokens that are not part of the vocabulary, a special token that does belong to the vocabulary is used as a generic stand-in, written as \"[UNK]\" for \"unknown\". In principle, any string could be hidden by such an [UNK]. Indeed, in information retrieval, pretokenizers were themselves used as tokenizers (and also called \"tokenizers\") with a word-level vocabulary that contained an [UNK].", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0052", "text": "Commonly used subword tokenization algorithms are byte pair encoding (BPE) and the unigram language model (ULM), which each include a vocabularization algorithm and a dedicated segmentation algorithm. There also exist several segmentation algorithms that require no learning and can be applied given a vocabulary (produced by BPE or ULM, for example), like greedily recognising tokens in a pretoken by moving through it left-to-right", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0053", "text": ". Well-known software implementations of subword tokenizers are Hugging Face's tokenizers Python package implemented in Rust, and the sentencepiece Python package implemented in C++. The latter package is named as such because one of its configuration options allows disabling the built-in pretokenizer, hence effectively making entire sentences a pretoken and thus having the tokenizer see entire sentences, rather than individual words.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0054", "text": "Embedding\nEach integer token identifier is converted into an embedding vector via a lookup table. Equivalently stated, it multiplies a one-hot representation of the token identifier by an embedding matrix \n  \n    \n      \n        M\n      \n    \n    {\\displaystyle M}\n  \n. For example, if the input token's identifier is \n  \n    \n      \n        3\n      \n    \n    {\\displaystyle 3}\n  \n, then the one-hot representation is \n  \n    \n      \n        [\n        0\n        ,\n        0\n        ,\n        0", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0055", "text": "0\n        ,\n        0\n        ,\n        0\n        ,\n        1\n        ,\n        0\n        ,\n        0\n        ,\n        \u2026\n        ]\n      \n    \n    {\\displaystyle [0,0,0,1,0,0,\\dots ]}\n  \n, and its embedding vector is\n  \n    \n      \n        \n          E\n          m\n          b\n          e\n          d\n        \n        (\n        3\n        )\n        =\n        [\n        0\n        ,\n        0\n        ,\n        0\n        ,\n        1\n        ,\n        0\n        ,\n        0\n        ,\n        \u2026", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0056", "text": "0\n        ,\n        0\n        ,\n        \u2026\n        ]\n        M\n      \n    \n    {\\displaystyle \\mathrm {Embed} (3)=[0,0,0,1,0,0,\\dots ]M}\n  \nThe token embedding vectors are added to their respective positional encoding vectors (see below), producing the sequence of input vectors.\nThe dimension of an embedding vector is called hidden size or embedding size and written as \n  \n    \n      \n        \n          d\n          \n            emb", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0057", "text": "emb\n          \n        \n      \n    \n    {\\displaystyle d_{\\text{emb}}}\n  \n. This size is written as \n  \n    \n      \n        \n          d\n          \n            model\n          \n        \n      \n    \n    {\\displaystyle d_{\\text{model}}}\n  \n in the original transformer paper.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0058", "text": "Un-embedding\nAn un-embedding layer is almost the reverse of an embedding layer. Whereas an embedding layer converts a token identifier into a vector, an un-embedding layer converts a vector into a probability distribution over tokens.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0059", "text": "The un-embedding layer is a linear-softmax layer:\n  \n    \n      \n        \n          U\n          n\n          E\n          m\n          b\n          e\n          d\n        \n        (\n        x\n        )\n        =\n        \n          s\n          o\n          f\n          t\n          m\n          a\n          x\n        \n        (\n        x\n        W\n        +\n        b\n        )\n      \n    \n    {\\displaystyle \\mathrm {UnEmbed} (x)=\\mathrm {softmax} (xW+b)}\n  \nThe matrix has shape \n  \n    \n      \n        (", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0060", "text": "The matrix has shape \n  \n    \n      \n        (\n        \n          d\n          \n            emb\n          \n        \n        ,\n        \n          |\n        \n        V\n        \n          |\n        \n        )\n      \n    \n    {\\displaystyle (d_{\\text{emb}},|V|)}\n  \n. Some architectures use the transpose of the embedding matrix \n  \n    \n      \n        M\n      \n    \n    {\\displaystyle M}\n  \n as the un-embedding matrix \n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0061", "text": "W\n      \n    \n    {\\displaystyle W}\n  \n in order to avoid needing double the amount of embedding-related parameters and to avoid divergence during training. This practice is called weight tying.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0062", "text": "Positional encoding\nA positional encoding is a fixed-size vector representation of the relative positions of tokens within a sequence: it provides the transformer model with information about where the words are in the input sequence. This induces a bias towards the order of the input sequence, so that, for example, the input sequence \"man bites dog\" is processed differently from \"dog bites man\".\nThe positional encoding is defined as a function of type \n  \n    \n      \n        f\n        :", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0063", "text": "f\n        :\n        \n          R\n        \n        \u2192\n        \n          \n            R\n          \n          \n            d\n          \n        \n      \n    \n    {\\displaystyle f:\\mathbb {R} \\to \\mathbb {R} ^{d}}\n  \n, where \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n is a positive even integer. The full positional encoding defined in the original paper is:\n  \n    \n      \n        (\n        f\n        (\n        t\n        \n          )\n          \n            2", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0064", "text": ")\n          \n            2\n            k\n          \n        \n        ,\n        f\n        (\n        t\n        \n          )\n          \n            2\n            k\n            +\n            1\n          \n        \n        )\n        =\n        (\n        sin\n        \u2061\n        (\n        \u03b8\n        )\n        ,\n        cos\n        \u2061\n        (\n        \u03b8\n        )\n        )\n        \n        \u2200\n        k\n        \u2208\n        {\n        0\n        ,\n        1\n        ,\n        \u2026\n        ,", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0065", "text": ",\n        1\n        ,\n        \u2026\n        ,\n        d\n        \n          /\n        \n        2\n        \u2212\n        1\n        }\n      \n    \n    {\\displaystyle (f(t)_{2k},f(t)_{2k+1})=(\\sin(\\theta ),\\cos(\\theta ))\\quad \\forall k\\in \\{0,1,\\ldots ,d/2-1\\}}\n  \nwhere \n  \n    \n      \n        \u03b8\n        =\n        \n          \n            t\n            \n              r\n              \n                k\n              \n            \n          \n        \n        ,\n        r\n        =\n        \n          N", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0066", "text": "r\n        =\n        \n          N\n          \n            2\n            \n              /\n            \n            d\n          \n        \n      \n    \n    {\\displaystyle \\theta ={\\frac {t}{r^{k}}},r=N^{2/d}}\n  \n.\nHere, \n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}\n  \n is a free parameter that should be significantly larger than the biggest \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0067", "text": "k\n      \n    \n    {\\displaystyle k}\n  \n that would be input into the positional encoding function. The original paper uses \n  \n    \n      \n        N\n        =\n        10000\n      \n    \n    {\\displaystyle N=10000}\n  \n.\nThe function is in a simpler form when written as a complex function of type \n  \n    \n      \n        f\n        :\n        \n          R\n        \n        \u2192\n        \n          \n            C\n          \n          \n            d\n            \n              /", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0068", "text": "/\n            \n            2\n          \n        \n      \n    \n    {\\displaystyle f:\\mathbb {R} \\to \\mathbb {C} ^{d/2}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0069", "text": "f\n        (\n        t\n        )\n        =\n        \n          \n            (\n            \n              e\n              \n                i\n                t\n                \n                  /\n                \n                \n                  r\n                  \n                    k\n                  \n                \n              \n            \n            )\n          \n          \n            k\n            =\n            0\n            ,\n            1\n            ,", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0070", "text": ",\n            1\n            ,\n            \u2026\n            ,\n            \n              \n                d\n                2\n              \n            \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle f(t)=\\left(e^{it/r^{k}}\\right)_{k=0,1,\\ldots ,{\\frac {d}{2}}-1}}\n  \nwhere \n  \n    \n      \n        r\n        =\n        \n          N\n          \n            2\n            \n              /\n            \n            d", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0071", "text": "d\n          \n        \n      \n    \n    {\\displaystyle r=N^{2/d}}\n  \n.\nThe main reason for using this positional encoding function is that using it, shifts are linear transformations:\n  \n    \n      \n        f\n        (\n        t\n        +\n        \u0394\n        t\n        )\n        =\n        \n          d\n          i\n          a\n          g\n        \n        (\n        f\n        (\n        \u0394\n        t\n        )\n        )\n        f\n        (\n        t\n        )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0072", "text": "(\n        t\n        )\n      \n    \n    {\\displaystyle f(t+\\Delta t)=\\mathrm {diag} (f(\\Delta t))f(t)}\n  \nwhere \n  \n    \n      \n        \u0394\n        t\n        \u2208\n        \n          R\n        \n      \n    \n    {\\displaystyle \\Delta t\\in \\mathbb {R} }\n  \n is the distance one wishes to shift. This allows the transformer to take any encoded position, and find the encoding of the position n-steps-ahead or n-steps-behind, by a matrix multiplication.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0073", "text": "By taking a linear sum, any convolution can also be implemented as linear transformations:\n  \n    \n      \n        \n          \u2211\n          \n            j\n          \n        \n        \n          c\n          \n            j\n          \n        \n        f\n        (\n        t\n        +\n        \u0394\n        \n          t\n          \n            j\n          \n        \n        )\n        =\n        \n          (\n          \n            \n              \u2211\n              \n                j", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0074", "text": "j\n              \n            \n            \n              c\n              \n                j\n              \n            \n            \n            \n              d\n              i\n              a\n              g\n            \n            (\n            f\n            (\n            \u0394\n            \n              t\n              \n                j\n              \n            \n            )\n            )\n          \n          )\n        \n        f\n        (\n        t\n        )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0075", "text": "(\n        t\n        )\n      \n    \n    {\\displaystyle \\sum _{j}c_{j}f(t+\\Delta t_{j})=\\left(\\sum _{j}c_{j}\\,\\mathrm {diag} (f(\\Delta t_{j}))\\right)f(t)}\n  \nfor any constants \n  \n    \n      \n        \n          c\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle c_{j}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0076", "text": "{\\displaystyle c_{j}}\n  \n. This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors. This sum of encoded positions, when fed into the attention mechanism, would create attention weights on its neighbors, much like what happens in a convolutional neural network language model. In the author's words, \"we hypothesized it would allow the model to easily learn to attend by relative position.\"", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0077", "text": "In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0078", "text": "Encoder\u2013decoder (overview)\nLike earlier seq2seq models, the original transformer model used an encoder\u2013decoder architecture. The encoder consists of encoding layers that process all the input tokens together one layer after another, while the decoder consists of decoding layers that iteratively process the encoder's output and the decoder's output tokens so far.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0079", "text": "The purpose of each encoder layer is to create contextualized representations of the tokens, where each representation corresponds to a token that \"mixes\" information from other input tokens via self-attention mechanism. Each decoder layer contains two attention sublayers: (1) cross-attention for incorporating the output of encoder (contextualized input token representations), and (2) self-attention for \"mixing\" information among the input tokens to the decoder (i.e", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0080", "text": ". the tokens generated so far during inference time).", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0081", "text": "Both the encoder and decoder layers have a feed-forward neural network for additional processing of their outputs and contain residual connections and layer normalization steps. These feed-forward layers contain most of the parameters in a transformer model.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0082", "text": "Feedforward network\nThe feedforward network (FFN) modules in a transformer are 2-layered multilayer perceptrons:\n  \n    \n      \n        \n          F\n          F\n          N\n        \n        (\n        x\n        )\n        =\n        \u03d5\n        (\n        x\n        \n          W\n          \n            (\n            1\n            )\n          \n        \n        +\n        \n          b\n          \n            (\n            1\n            )\n          \n        \n        )\n        \n          W", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0083", "text": ")\n        \n          W\n          \n            (\n            2\n            )\n          \n        \n        +\n        \n          b\n          \n            (\n            2\n            )\n          \n        \n      \n    \n    {\\displaystyle \\mathrm {FFN} (x)=\\phi (xW^{(1)}+b^{(1)})W^{(2)}+b^{(2)}}\n  \nwhere \n  \n    \n      \n        \n          W\n          \n            (\n            1\n            )\n          \n        \n      \n    \n    {\\displaystyle W^{(1)}}\n  \n and", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0084", "text": "and \n  \n    \n      \n        \n          W\n          \n            (\n            2\n            )\n          \n        \n      \n    \n    {\\displaystyle W^{(2)}}\n  \n are weight matrices and \n  \n    \n      \n        \n          b\n          \n            (\n            1\n            )\n          \n        \n      \n    \n    {\\displaystyle b^{(1)}}\n  \n and  \n  \n    \n      \n        \n          b\n          \n            (\n            2\n            )\n          \n        \n      \n    \n    {\\displaystyle b^{(2)}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0085", "text": "{\\displaystyle b^{(2)}}\n  \n are bias vectors, and \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n is its activation function. The original transformer used ReLU activation.\nThe number of neurons in the middle layer is called intermediate size (GPT), filter size (BERT), or feedforward size (BERT). It is typically larger than the embedding size. For example, in both GPT-2 series and BERT series, the intermediate size of a model is 4 times its embedding size:", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0086", "text": "d\n          \n            ffn\n          \n        \n        =\n        4\n        \n          d\n          \n            emb\n          \n        \n      \n    \n    {\\displaystyle d_{\\text{ffn}}=4d_{\\text{emb}}}\n  \n.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0087", "text": "Scaled dot-product attention\nAttention head\nThe attention mechanism used in the transformer architecture are scaled dot-product attention units. For each unit, the transformer model learns three weight matrices: the query weights \n  \n    \n      \n        \n          W\n          \n            Q\n          \n        \n      \n    \n    {\\displaystyle W^{Q}}\n  \n, the key weights \n  \n    \n      \n        \n          W\n          \n            K\n          \n        \n      \n    \n    {\\displaystyle W^{K}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0088", "text": "{\\displaystyle W^{K}}\n  \n, and the value weights \n  \n    \n      \n        \n          W\n          \n            V\n          \n        \n      \n    \n    {\\displaystyle W^{V}}\n  \n.\nThe module takes three sequences, a query sequence, a key sequence, and a value sequence. The query sequence is a sequence of length \n  \n    \n      \n        \n          \u2113\n          \n            seq, query\n          \n        \n      \n    \n    {\\displaystyle \\ell _{\\text{seq, query}}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0089", "text": "{\\displaystyle \\ell _{\\text{seq, query}}}\n  \n, and each entry is a vector of dimension \n  \n    \n      \n        \n          d\n          \n            emb, query\n          \n        \n      \n    \n    {\\displaystyle d_{\\text{emb, query}}}\n  \n. Similarly for the key and value sequences.\nFor each vector \n  \n    \n      \n        \n          x\n          \n            i\n            ,\n            \n              query\n            \n          \n        \n      \n    \n    {\\displaystyle x_{i,{\\text{query}}}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0090", "text": "{\\displaystyle x_{i,{\\text{query}}}}\n  \n in the query sequence, it is multiplied by a matrix \n  \n    \n      \n        \n          W\n          \n            Q\n          \n        \n      \n    \n    {\\displaystyle W^{Q}}\n  \n to produce a query vector \n  \n    \n      \n        \n          q\n          \n            i\n          \n        \n        =\n        \n          x\n          \n            i\n            ,\n            \n              query\n            \n          \n        \n        \n          W", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0091", "text": "W\n          \n            Q\n          \n        \n      \n    \n    {\\displaystyle q_{i}=x_{i,{\\text{query}}}W^{Q}}\n  \n. The matrix of all query vectors is the query matrix:\n  \n    \n      \n        Q\n        =\n        \n          X\n          \n            query\n          \n        \n        \n          W\n          \n            Q\n          \n        \n      \n    \n    {\\displaystyle Q=X_{\\text{query}}W^{Q}}\n  \nSimilarly, we construct the key matrix", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0092", "text": "K\n        =\n        \n          X\n          \n            key\n          \n        \n        \n          W\n          \n            K\n          \n        \n      \n    \n    {\\displaystyle K=X_{\\text{key}}W^{K}}\n  \n and the value matrix \n  \n    \n      \n        V\n        =\n        \n          X\n          \n            value\n          \n        \n        \n          W\n          \n            V\n          \n        \n      \n    \n    {\\displaystyle V=X_{\\text{value}}W^{V}}\n  \n.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0093", "text": "{\\displaystyle V=X_{\\text{value}}W^{V}}\n  \n.\nIt is usually the case that all \n  \n    \n      \n        \n          W\n          \n            Q\n          \n        \n        ,\n        \n          W\n          \n            K\n          \n        \n        ,\n        \n          W\n          \n            V\n          \n        \n      \n    \n    {\\displaystyle W^{Q},W^{K},W^{V}}\n  \n are square matrices, meaning \n  \n    \n      \n        \n          d\n          \n            emb, query\n          \n        \n        =", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0094", "text": "=\n        \n          d\n          \n            query\n          \n        \n      \n    \n    {\\displaystyle d_{\\text{emb, query}}=d_{\\text{query}}}\n  \n, etc.\nAttention weights are calculated using the query and key vectors: the attention weight \n  \n    \n      \n        \n          a\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle a_{ij}}\n  \n from token \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n to token", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0095", "text": "{\\displaystyle i}\n  \n to token \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n is the dot product between \n  \n    \n      \n        \n          q\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle q_{i}}\n  \n and \n  \n    \n      \n        \n          k\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle k_{j}}\n  \n. The attention weights are divided by the square root of the dimension of the key vectors,", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0096", "text": "d\n              \n                k\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\sqrt {d_{k}}}}\n  \n, which stabilizes gradients during training, and passed through a softmax which normalizes the weights. The fact that \n  \n    \n      \n        \n          W\n          \n            Q\n          \n        \n      \n    \n    {\\displaystyle W^{Q}}\n  \n and \n  \n    \n      \n        \n          W", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0097", "text": "W\n          \n            K\n          \n        \n      \n    \n    {\\displaystyle W^{K}}\n  \n are different matrices allows attention to be non-symmetric: if token \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n attends to token \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n (i.e. \n  \n    \n      \n        \n          q\n          \n            i\n          \n        \n        \u22c5\n        \n          k\n          \n            j", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0098", "text": "k\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle q_{i}\\cdot k_{j}}\n  \n is large), this does not necessarily mean that token \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n will attend to token \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n (i.e. \n  \n    \n      \n        \n          q\n          \n            j\n          \n        \n        \u22c5\n        \n          k\n          \n            i", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0099", "text": "i\n          \n        \n      \n    \n    {\\displaystyle q_{j}\\cdot k_{i}}\n  \n could be small). The output of the attention unit for token \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n is the weighted sum of the value vectors of all tokens, weighted by \n  \n    \n      \n        \n          a\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle a_{ij}}\n  \n, the attention from token \n  \n    \n      \n        i", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0100", "text": "i\n      \n    \n    {\\displaystyle i}\n  \n to each token.\nThe attention calculation for all tokens can be expressed as one large matrix calculation using the softmax function, which is useful for training due to computational matrix operation optimizations that quickly compute matrix operations. The matrices \n  \n    \n      \n        Q\n      \n    \n    {\\displaystyle Q}\n  \n, \n  \n    \n      \n        K\n      \n    \n    {\\displaystyle K}\n  \n and \n  \n    \n      \n        V", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0101", "text": "and \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n are defined as the matrices where the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \nth rows are vectors \n  \n    \n      \n        \n          q\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle q_{i}}\n  \n, \n  \n    \n      \n        \n          k\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle k_{i}}\n  \n, and \n  \n    \n      \n        \n          v", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0102", "text": "v\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle v_{i}}\n  \n respectively. Then we can represent the attention as\n  \n    \n      \n        \n          \n            \n              \n                \n                  Attention\n                \n                (\n                Q\n                ,\n                K\n                ,\n                V\n                )\n                =\n                \n                  softmax", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0103", "text": "softmax\n                \n                \n                  (\n                  \n                    \n                      \n                        Q\n                        \n                          K\n                          \n                            \n                              T", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0104", "text": "d\n                          \n                            k\n                          \n                        \n                      \n                    \n                  \n                  )\n                \n                V", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0105", "text": "{\\displaystyle {\\begin{aligned}{\\text{Attention}}(Q,K,V)={\\text{softmax}}\\left({\\frac {QK^{\\mathrm {T} }}{\\sqrt {d_{k}}}}\\right)V\\end{aligned}}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0106", "text": "where the softmax is applied over each of the rows of the matrix.\nThe number of dimensions in a query vector is query size \n  \n    \n      \n        \n          d\n          \n            query\n          \n        \n      \n    \n    {\\displaystyle d_{\\text{query}}}\n  \n and similarly for the key size \n  \n    \n      \n        \n          d\n          \n            key\n          \n        \n      \n    \n    {\\displaystyle d_{\\text{key}}}\n  \n and value size \n  \n    \n      \n        \n          d", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0107", "text": "d\n          \n            value\n          \n        \n      \n    \n    {\\displaystyle d_{\\text{value}}}\n  \n. The output dimension of an attention head is its head dimension \n  \n    \n      \n        \n          d\n          \n            head\n          \n        \n      \n    \n    {\\displaystyle d_{\\text{head}}}\n  \n. The attention mechanism requires the following three equalities to hold:\n  \n    \n      \n        \n          \u2113\n          \n            seq, key", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0108", "text": "seq, key\n          \n        \n        =\n        \n          \u2113\n          \n            seq, value\n          \n        \n        ,\n        \n        \n          d\n          \n            query\n          \n        \n        =\n        \n          d\n          \n            key\n          \n        \n        ,\n        \n        \n          d\n          \n            value\n          \n        \n        =\n        \n          d\n          \n            head", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0109", "text": "head\n          \n        \n      \n    \n    {\\displaystyle \\ell _{\\text{seq, key}}=\\ell _{\\text{seq, value}},\\;d_{\\text{query}}=d_{\\text{key}},\\;d_{\\text{value}}=d_{\\text{head}}}\n  \nbut is otherwise unconstrained.\nIf the attention head is used in a self-attention fashion, then \n  \n    \n      \n        \n          X\n          \n            query\n          \n        \n        =\n        \n          X\n          \n            key\n          \n        \n        =\n        \n          X", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0110", "text": "=\n        \n          X\n          \n            value\n          \n        \n      \n    \n    {\\displaystyle X_{\\text{query}}=X_{\\text{key}}=X_{\\text{value}}}\n  \n. If the attention head is used in a cross-attention fashion, then usually \n  \n    \n      \n        \n          X\n          \n            query\n          \n        \n        \u2260\n        \n          X\n          \n            key\n          \n        \n        =\n        \n          X\n          \n            value", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0111", "text": "value\n          \n        \n      \n    \n    {\\displaystyle X_{\\text{query}}\\neq X_{\\text{key}}=X_{\\text{value}}}\n  \n. It is theoretically possible for all three to be different, but that is rarely the case in practice.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0112", "text": "Multihead attention\nOne set of \n  \n    \n      \n        \n          (\n          \n            \n              W\n              \n                Q\n              \n            \n            ,\n            \n              W\n              \n                K\n              \n            \n            ,\n            \n              W\n              \n                V\n              \n            \n          \n          )\n        \n      \n    \n    {\\displaystyle \\left(W^{Q},W^{K},W^{V}\\right)}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0113", "text": "matrices is called an attention head, and each layer in a transformer model has multiple attention heads. While each attention head attends to the tokens that are relevant to each token, multiple attention heads allow the model to do this for different definitions of \"relevance\". Specifically, the query and key projection matrices, \n  \n    \n      \n        \n          W\n          \n            Q\n          \n        \n      \n    \n    {\\displaystyle W^{Q}}\n  \n and", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0114", "text": "and \n  \n    \n      \n        \n          W\n          \n            K\n          \n        \n      \n    \n    {\\displaystyle W^{K}}\n  \n , which are involved in the attention score computation, defines the \"relevance\". Meanwhile, the value projection matrix \n  \n    \n      \n        \n          W\n          \n            V\n          \n        \n      \n    \n    {\\displaystyle W^{V}}\n  \n, in combination with the part of the output projection matrix \n  \n    \n      \n        \n          W", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0115", "text": "W\n          \n            O\n          \n        \n      \n    \n    {\\displaystyle W^{O}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0116", "text": ", determines how the attended tokens influence what information is passed to subsequent layers and ultimately the output logits. In addition, the scope of attention, or the range of token relationships captured by each attention head, can expand as tokens pass through successive layers. This allows the model to capture more complex and long-range dependencies in deeper layers. Many transformer attention heads encode relevance relations that are meaningful to humans", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0117", "text": ". For example, some attention heads can attend mostly to the next word, while others mainly attend from verbs to their direct objects. The computations for each attention head can be performed in parallel, which allows for fast processing. The outputs for the attention layer are concatenated to pass into the feedforward neural network layers.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0118", "text": "Concretely, let the multiple attention heads be indexed by \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n, then we have\n  \n    \n      \n        \n          MultiheadAttention\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n        =\n        \n          \n            Concat\n          \n          \n            i\n            \u2208\n            [\n            \n              n\n              \n                heads\n              \n            \n            ]", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0119", "text": "]\n          \n        \n        (\n        \n          Attention\n        \n        (\n        X\n        \n          W\n          \n            i\n          \n          \n            Q\n          \n        \n        ,\n        X\n        \n          W\n          \n            i\n          \n          \n            K\n          \n        \n        ,\n        X\n        \n          W\n          \n            i\n          \n          \n            V\n          \n        \n        )\n        )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0120", "text": ")\n        )\n        \n          W\n          \n            O\n          \n        \n      \n    \n    {\\displaystyle {\\text{MultiheadAttention}}(Q,K,V)={\\text{Concat}}_{i\\in [n_{\\text{heads}}]}({\\text{Attention}}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V}))W^{O}}\n  \n where the matrix \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n is the concatenation of word embeddings, and the matrices \n  \n    \n      \n        \n          W\n          \n            i", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0121", "text": "W\n          \n            i\n          \n          \n            Q\n          \n        \n        ,\n        \n          W\n          \n            i\n          \n          \n            K\n          \n        \n        ,\n        \n          W\n          \n            i\n          \n          \n            V\n          \n        \n      \n    \n    {\\displaystyle W_{i}^{Q},W_{i}^{K},W_{i}^{V}}\n  \n are \"projection matrices\" owned by individual attention head \n  \n    \n      \n        i", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0122", "text": "i\n      \n    \n    {\\displaystyle i}\n  \n, and \n  \n    \n      \n        \n          W\n          \n            O\n          \n        \n      \n    \n    {\\displaystyle W^{O}}\n  \n is a final projection matrix owned by the whole multihead attention head.\nIt is theoretically possible for each attention head to have a different head dimension \n  \n    \n      \n        \n          d\n          \n            head\n          \n        \n      \n    \n    {\\displaystyle d_{\\text{head}}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0123", "text": "{\\displaystyle d_{\\text{head}}}\n  \n, but that is rarely the case in practice.\nAs an example, in the smallest GPT-2 model, there are only self-attention mechanisms. It has the following dimensions:\n  \n    \n      \n        \n          d\n          \n            emb\n          \n        \n        =\n        768\n        ,\n        \n          n\n          \n            head\n          \n        \n        =\n        12\n        ,\n        \n          d\n          \n            head\n          \n        \n        =", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0124", "text": "head\n          \n        \n        =\n        64\n      \n    \n    {\\displaystyle d_{\\text{emb}}=768,n_{\\text{head}}=12,d_{\\text{head}}=64}\n  \nSince \n  \n    \n      \n        12\n        \u00d7\n        64\n        =\n        768\n      \n    \n    {\\displaystyle 12\\times 64=768}\n  \n, its output projection matrix \n  \n    \n      \n        \n          W\n          \n            O\n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            (\n            12\n            \u00d7", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0125", "text": "(\n            12\n            \u00d7\n            64\n            )\n            \u00d7\n            768\n          \n        \n      \n    \n    {\\displaystyle W^{O}\\in \\mathbb {R} ^{(12\\times 64)\\times 768}}\n  \n is a square matrix.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0126", "text": "Masked attention\nThe transformer architecture is constructed to calculate output tokens iteratively. Assuming \n  \n    \n      \n        t\n        =\n        0\n      \n    \n    {\\displaystyle t=0}\n  \n refers to the calculation of the first output token \n  \n    \n      \n        i\n        =\n        0\n      \n    \n    {\\displaystyle i=0}\n  \n, for step \n  \n    \n      \n        t\n        >\n        0\n      \n    \n    {\\displaystyle t>0}\n  \n, the output token \n  \n    \n      \n        i\n        =\n        0", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0127", "text": "i\n        =\n        0\n      \n    \n    {\\displaystyle i=0}\n  \n shall remain constant. This ensures properties of the model similar to autoregressive models. Therefore, at every time step \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n, the calculation for all outputs \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n should not have access to tokens at position \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n for", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0128", "text": "{\\displaystyle j}\n  \n for \n  \n    \n      \n        j\n        >=\n        i\n      \n    \n    {\\displaystyle j>=i}\n  \n (as it naturally is the case for time step \n  \n    \n      \n        t\n        =\n        i\n      \n    \n    {\\displaystyle t=i}\n  \n, when tokens \n  \n    \n      \n        j\n        >\n        t\n      \n    \n    {\\displaystyle j>t}\n  \n are not yet calculated). This behavior may be accomplished before the softmax stage by adding a mask matrix \n  \n    \n      \n        M", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0129", "text": "M\n      \n    \n    {\\displaystyle M}\n  \n that is \n  \n    \n      \n        \u2212\n        \u221e\n      \n    \n    {\\displaystyle -\\infty }\n  \n at entries where the attention link must be cut, and \n  \n    \n      \n        0\n      \n    \n    {\\displaystyle 0}\n  \n at other places:\n  \n    \n      \n        \n          \n            \n              \n                \n                  MaskedAttention\n                \n                (\n                Q\n                ,\n                K", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0130", "text": ",\n                K\n                ,\n                V\n                )\n                =\n                \n                  softmax\n                \n                \n                  (\n                  \n                    M\n                    +\n                    \n                      \n                        \n                          Q\n                          \n                            K", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0131", "text": "T\n                              \n                            \n                          \n                        \n                        \n                          \n                            d\n                            \n                              k\n                            \n                          \n                        \n                      \n                    \n                  \n                  )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0132", "text": ")\n                \n                V\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}{\\text{MaskedAttention}}(Q,K,V)={\\text{softmax}}\\left(M+{\\frac {QK^{\\mathrm {T} }}{\\sqrt {d_{k}}}}\\right)V\\end{aligned}}}\n  \n The following matrix is commonly used in decoder self-attention modules, called \"causal masking\":\n  \n    \n      \n        \n          M\n          \n            causal\n          \n        \n        =", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0133", "text": "=\n        \n          \n            [\n            \n              \n                \n                  0\n                \n                \n                  \u2212\n                  \u221e\n                \n                \n                  \u2212\n                  \u221e\n                \n                \n                  \u2026\n                \n                \n                  \u2212\n                  \u221e\n                \n              \n              \n                \n                  0", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0134", "text": "0\n                \n                \n                  0\n                \n                \n                  \u2212\n                  \u221e\n                \n                \n                  \u2026\n                \n                \n                  \u2212\n                  \u221e\n                \n              \n              \n                \n                  0\n                \n                \n                  0\n                \n                \n                  0", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0135", "text": "0\n                \n                \n                  \u2026\n                \n                \n                  \u2212\n                  \u221e\n                \n              \n              \n                \n                  \u22ee\n                \n                \n                  \u22ee\n                \n                \n                  \u22ee\n                \n                \n                  \u22f1\n                \n                \n                  \u22ee", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0136", "text": "0\n                \n                \n                  0\n                \n                \n                  0\n                \n                \n                  \u2026\n                \n                \n                  0\n                \n              \n            \n            ]", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0137", "text": "]\n          \n        \n      \n    \n    {\\displaystyle M_{\\text{causal}}={\\begin{bmatrix}0&-\\infty &-\\infty &\\dots &-\\infty \\\\0&0&-\\infty &\\dots &-\\infty \\\\0&0&0&\\dots &-\\infty \\\\\\vdots &\\vdots &\\vdots &\\ddots &\\vdots \\\\0&0&0&\\dots &0\\end{bmatrix}}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0138", "text": "In words, it means that each token can pay attention to itself, and every token before it, but not any after it. A non-masked attention module can be thought of as a masked attention module where the mask has all entries zero. As an example of an uncommon use of mask matrix, the XLNet considers all masks of the form \n  \n    \n      \n        P\n        \n          M\n          \n            causal\n          \n        \n        \n          P\n          \n            \u2212\n            1", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0139", "text": "\u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle PM_{\\text{causal}}P^{-1}}\n  \n, where \n  \n    \n      \n        P\n      \n    \n    {\\displaystyle P}\n  \n is a random permutation matrix.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0140", "text": "Encoder\nAn encoder consists of an embedding layer, followed by multiple encoder layers.\nEach encoder layer consists of two major components: a self-attention mechanism and a feed-forward layer. It takes an input as a sequence of input vectors, applies the self-attention mechanism, to produce an intermediate sequence of vectors, then applies the feed-forward layer for each vector individually. Schematically, we have:", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0141", "text": "given input vectors \n                \n              \n              \n                \n                  h\n                  \n                    0\n                  \n                \n                ,\n                \n                  h\n                  \n                    1\n                  \n                \n                ,\n                \u2026", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0142", "text": "combine them into a matrix \n                \n                H\n              \n              \n                \n                =\n                \n                  \n                    [\n                    \n                      \n                        \n                          \n                            h\n                            \n                              0", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0143", "text": "h\n                            \n                              1\n                            \n                          \n                        \n                      \n                      \n                        \n                          \u22ee", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0144", "text": "]\n                  \n                \n              \n            \n            \n              \n                \n                  EncoderLayer\n                \n                (\n                H\n                )\n              \n              \n                \n                =\n                \n                  \n                    [", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0145", "text": "FFN\n                          \n                          (\n                          \n                            MultiheadAttention\n                          \n                          (\n                          H\n                          ,\n                          H\n                          ,\n                          H\n                          \n                            )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0146", "text": ")\n                            \n                              0\n                            \n                          \n                          )\n                        \n                      \n                      \n                        \n                          \n                            FFN\n                          \n                          (\n                          \n                            MultiheadAttention", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0147", "text": "(\n                          H\n                          ,\n                          H\n                          ,\n                          H\n                          \n                            )\n                            \n                              1\n                            \n                          \n                          )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0148", "text": "\u22ee\n                        \n                      \n                    \n                    ]", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0149", "text": "{\\displaystyle {\\begin{aligned}{\\text{given input vectors }}&h_{0},h_{1},\\dots \\\\{\\text{combine them into a matrix }}H&={\\begin{bmatrix}h_{0}\\\\h_{1}\\\\\\vdots \\end{bmatrix}}\\\\{\\text{EncoderLayer}}(H)&={\\begin{bmatrix}{\\text{FFN}}({\\text{MultiheadAttention}}(H,H,H)_{0})\\\\{\\text{FFN}}({\\text{MultiheadAttention}}(H,H,H)_{1})\\\\\\vdots \\end{bmatrix}}\\\\\\end{aligned}}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0150", "text": "where \n  \n    \n      \n        \n          FFN\n        \n      \n    \n    {\\displaystyle {\\text{FFN}}}\n  \n stands for \"feed-forward network\". We can more succinctly write it as\n  \n    \n      \n        \n          EncoderLayer\n        \n        (\n        H\n        )\n        =\n        \n          FFN\n        \n        (\n        \n          MultiheadAttention\n        \n        (\n        H\n        ,\n        H\n        ,\n        H\n        )\n        )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0151", "text": "H\n        )\n        )\n      \n    \n    {\\displaystyle {\\text{EncoderLayer}}(H)={\\text{FFN}}({\\text{MultiheadAttention}}(H,H,H))}\n  \nwith the implicit convention that the \n  \n    \n      \n        \n          FFN\n        \n      \n    \n    {\\displaystyle {\\text{FFN}}}\n  \n is applied to each row of the matrix individually.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0152", "text": "The encoder layers are stacked. The first encoder layer takes the sequence of input vectors from the embedding layer, producing a sequence of vectors. This sequence of vectors is processed by the second encoder, and so on. The output from the final encoder layer is then used by the decoder.\nAs the encoder processes the entire input all at once, every token can attend to every other token (all-to-all attention), so there is no need for causal masking.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0153", "text": "Decoder\nA decoder consists of an embedding layer, followed by multiple decoder layers, followed by an un-embedding layer.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0154", "text": "Each decoder consists of three major components: a causally masked self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network. The decoder functions in a similar fashion to the encoder, but an additional attention mechanism is inserted which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called the encoder\u2013decoder attention.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0155", "text": "Like the first encoder, the first decoder takes positional information and embeddings of the output sequence as its input, rather than encodings. The transformer must not use the current or future output to predict an output, so the output sequence must be partially masked to prevent this reverse information flow. This allows for autoregressive text generation. For decoding, all-to-all attention is inappropriate, because a token cannot attend to tokens not yet generated", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0156", "text": ". Thus, the self-attention module in the decoder is causally masked.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0157", "text": "In contrast, the cross-attention mechanism attends to the output vectors of the encoder, which is computed before the decoder starts decoding. Consequently, there is no need for masking in the cross-attention mechanism.\nSchematically, we have:\n  \n    \n      \n        \n          \n            \n              \n                \n                  H\n                  \u2032\n                \n              \n              \n                \n                =", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0158", "text": "=\n                \n                  MaskedMultiheadAttention\n                \n                (\n                H\n                ,\n                H\n                ,\n                H\n                )\n              \n            \n            \n              \n                \n                  DecoderLayer\n                \n                (\n                H\n                )\n              \n              \n                \n                =\n                \n                  FFN", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0159", "text": "FFN\n                \n                (\n                \n                  MultiheadAttention\n                \n                (\n                \n                  H\n                  \u2032\n                \n                ,\n                \n                  H\n                  \n                    E\n                  \n                \n                ,\n                \n                  H\n                  \n                    E", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0160", "text": "E\n                  \n                \n                )\n                )\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}H'&={\\text{MaskedMultiheadAttention}}(H,H,H)\\\\{\\text{DecoderLayer}}(H)&={\\text{FFN}}({\\text{MultiheadAttention}}(H',H^{E},H^{E}))\\end{aligned}}}\n  \nwhere \n  \n    \n      \n        \n          H\n          \n            E\n          \n        \n      \n    \n    {\\displaystyle H^{E}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0161", "text": "{\\displaystyle H^{E}}\n  \n is the matrix with rows being the output vectors from the encoder.\nThe last decoder is followed by a final un-embedding layer to produce the output probabilities over the vocabulary. Then, one of the tokens is sampled according to the probability, and the decoder can be run again to produce the next token, etc., autoregressively generating output text.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0162", "text": "Full transformer architecture\nSublayers\nEach encoder layer contains 2 sublayers: the self-attention and the feedforward network. Each decoder layer contains 3 sublayers: the causally masked self-attention, the cross-attention, and the feedforward network.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0163", "text": "The final points of detail are the residual connections and layer normalization, (denoted as \"LayerNorm\", or \"LN\" in the following), which while conceptually unnecessary, are necessary for numerical stability and convergence.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0164", "text": "The residual connection, which is introduced to avoid vanishing gradient issues and stabilize the training process, can be expressed as follows: y = F(x) + x. The expression indicates that an output y is the sum of the transformation of input x (F(x)) and the input itself (x). Adding the input x can preserve the input information and avoid issues when the gradient of F(x) is close to zero.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0165", "text": "Similarly to how the feedforward network modules are applied individually to each vector, the LayerNorm is also applied individually to each vector.\nThere are two common conventions in use: the post-LN and the pre-LN convention. In the post-LN convention, the output of each sublayer is \n  \n    \n      \n        \n          L\n          a\n          y\n          e\n          r\n          N\n          o\n          r\n          m\n        \n        (\n        x\n        +\n        \n          S\n          u", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0166", "text": "+\n        \n          S\n          u\n          b\n          l\n          a\n          y\n          e\n          r\n        \n        (\n        x\n        )\n        )\n      \n    \n    {\\displaystyle \\mathrm {LayerNorm} (x+\\mathrm {Sublayer} (x))}\n  \nwhere \n  \n    \n      \n        \n          S\n          u\n          b\n          l\n          a\n          y\n          e\n          r\n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle \\mathrm {Sublayer} (x)}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0167", "text": "{\\displaystyle \\mathrm {Sublayer} (x)}\n  \n is the function implemented by the sublayer itself.\nIn the pre-LN convention, the output of each sublayer is\n  \n    \n      \n        x\n        +\n        \n          S\n          u\n          b\n          l\n          a\n          y\n          e\n          r\n        \n        (\n        \n          L\n          a\n          y\n          e\n          r\n          N\n          o\n          r\n          m\n        \n        (\n        x\n        )\n        )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0168", "text": "x\n        )\n        )\n      \n    \n    {\\displaystyle x+\\mathrm {Sublayer} (\\mathrm {LayerNorm} (x))}\n  \nThe original 2017 transformer used the post-LN convention. It was difficult to train and required careful hyperparameter tuning and a \"warm-up\" in learning rate, where it starts small and gradually increases. The pre-LN convention, proposed several times in 2018, was found to be easier to train, requiring no warm-up, leading to faster convergence.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0169", "text": "Pseudocode\nThe following is the pseudocode for a standard pre-LN encoder\u2013decoder transformer, adapted from Formal Algorithms for Transformers\n\ninput: Encoder input t_e\n       Decoder input t_d\noutput: Array of probability distributions, with shape (decoder vocabulary size x length(decoder output sequence))\n\n/* encoder */\nz_e \u2190 encoder.tokenizer(t_e)\n\nfor each t in 1:length(z_e) do\n    z_e[t] \u2190 encoder.embedding(z_e[t]) + encoder.positional_embedding(t)", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0170", "text": "for each l in 1:length(encoder.layers) do\n    layer \u2190 encoder.layers[l]\n\n    /* first sublayer */\n    z_e_copy \u2190 copy(z_e)\n    for each t in 1:length(z_e) do\n        z_e[t] \u2190 layer.layer_norm(z_e[t])\n    z_e \u2190 layer.multihead_attention(z_e, z_e, z_e)\n    for each t in 1:length(z_e) do\n        z_e[t] \u2190 z_e[t] + z_e_copy[t]", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0171", "text": "/* second sublayer */\n    z_e_copy \u2190 copy(z_e)\n    for each t in 1:length(z_e) do\n        z_e[t] \u2190 layer.layer_norm(z_e[t])\n    z_e \u2190 layer.feedforward(z_e)\n    for each t in 1:length(z_e) do\n        z_e[t] \u2190 z_e[t] + z_e_copy[t]\n\nfor each t in 1:length(z_e) do\n    z_e[t] \u2190 encoder.final_layer_norm(z_e[t])\n\n/* decoder */\nz_d \u2190 decoder.tokenizer(t_d)\n\nfor each t in 1:length(z_d) do\n    z_d[t] \u2190 decoder.embedding(z_d[t]) + decoder.positional_embedding(t)", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0172", "text": "for each l in 1:length(decoder.layers) do\n        layer \u2190 decoder.layers[l]\n\n        /* first sublayer */\n        z_d_copy \u2190 copy(z_d)\n        for each t in 1:length(z_d) do\n            z_d[t] \u2190 layer.layer_norm(z_d[t])\n        z_d \u2190 layer.masked_multihead_attention(z_d, z_d, z_d)\n        for each t in 1:length(z_d) do\n            z_d[t] \u2190 z_d[t] + z_d_copy[t]", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0173", "text": "/* second sublayer */\n        z_d_copy \u2190 copy(z_d)\n        for each t in 1:length(z_d) do\n            z_d[t] \u2190 layer.layer_norm(z_d[t])\n        z_d \u2190 layer.multihead_attention(z_d, z_e, z_e) \n        for each i in 1:length(z_d) do\n            z_d[t] \u2190 z_d[t] + z_d_copy[t]", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0174", "text": "/* third sublayer */\n        z_d_copy \u2190 copy(z_d)\n        for each t in 1:length(z_d) do\n            z_d[t] \u2190 layer.layer_norm(z_d[t])\n        z_d \u2190 layer.feedforward(z_d)\n        for each t in 1:length(z_d) do\n            z_d[t] \u2190 z_d[t] + z_d_copy[t]\n\nz_d \u2190 decoder.final_layer_norm(z_d)\n\noutput_distributions \u2190 []\nfor each t in 1:length(z_d) do\n    output_distributions.append(decoder.unembed(z_d[t]))\n\nreturn output_distributions", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0175", "text": "Terminology\nThe transformer architecture, being modular, allows variations. Several common variations are described here.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0176", "text": "An \"encoder-only\" transformer applies the encoder to map an input text into a sequence of vectors that represent the input text. This is usually used for text embedding and representation learning for downstream applications. BERT is encoder-only. They are less often used currently, as they were found to be not significantly better than training an encoder\u2013decoder transformer, then taking just the encoder. They are also referred to as \"all-to-all\" or \"BERT-like\".", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0177", "text": "A \"decoder-only\" transformer is not literally decoder-only, since without an encoder, the cross-attention mechanism has nothing to attend to. Thus, the decoder layers in a decoder-only transformer is composed of just two sublayers: the causally masked self-attention, and the feedforward network. This is usually used for text generation and instruction following. The models in the GPT series and Chinchilla series are decoder-only. They are also referred to as \"autoregressive\" or \"causal\".", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0178", "text": "An \"encoder\u2013decoder\" transformer is generally the same as the original transformer, with 2 sublayers per encoder layer and 3 sublayers per decoder layer, etc. They might have minor architectural improvements, such as alternative activation functions, changing the location of normalization, etc. This is also usually used for text generation and instruction following. The models in the T5 series are encoder\u2013decoder.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0179", "text": "A \"prefixLM\" (prefix language model) is a decoder-only architecture, but with prefix masking, which is different from causal masking. Specifically, it has mask of the form\n  \n    \n      \n        \n          M\n          \n            prefixLM\n          \n        \n        =\n        \n          \n            [\n            \n              \n                \n                  \n                    0\n                  \n                \n                \n                  \u2212\n                  \u221e", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0180", "text": "\u221e\n                \n              \n              \n                \n                  \n                    0\n                  \n                \n                \n                  \n                    M\n                    \n                      causal\n                    \n                  \n                \n              \n            \n            ]", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0181", "text": "]\n          \n        \n      \n    \n    {\\displaystyle M_{\\text{prefixLM}}={\\begin{bmatrix}\\mathbf {0} &-\\infty \\\\\\mathbf {0} &M_{\\text{causal}}\\end{bmatrix}}}\n  \nwhere the first columns correspond to the \"prefix\", and the subsequent columns correspond to the autoregressively generated text based on the prefix. They resemble encoder\u2013decoder models, but has less \"sparsity\". Such models are rarely used, though they are cited as theoretical possibilities and benchmarked comparisons.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0182", "text": "There are also mixed seq2seq models. For example, in 2020, Google Translate replaced the previous RNN-encoder\u2013RNN-decoder model with a transformer-encoder\u2013RNN-decoder model, as transformer-based decoders did not appear to significantly increase quality unlike the encoder, while the RNN decoder was much faster.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0183", "text": "Subsequent work\nAlternative activation functions\nThe original transformer uses ReLU activation function. Other activation functions were developed. The Llama series and PaLM used SwiGLU; both GPT-1 and BERT used GELU.\nAlternative activation functions are often used in combination with Gated Linear Units in the feedforward module.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0184", "text": "Alternative normalizations\nThe normalization used in the transformer can be different from LayerNorm. One example is RMSNorm which is used in the Llama series. Other examples include CapsuleNorm ScaleNorm, or FixNorm.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0185", "text": "Alternative positional encodings\nTransformers may use other positional encoding methods than sinusoidal.\nThe original transformer paper reported using a learned positional encoding, but finding it not superior to the sinusoidal one. Later, found that causal masking itself provides enough signal to a transformer decoder that it can learn to implicitly perform absolute positional encoding without the positional encoding module.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0186", "text": "RoPE\nRoPE (rotary positional embedding), is best explained by considering a list of 2-dimensional vectors \n  \n    \n      \n        [\n        (\n        \n          x\n          \n            1\n          \n          \n            (\n            1\n            )\n          \n        \n        ,\n        \n          x\n          \n            1\n          \n          \n            (\n            2\n            )\n          \n        \n        )\n        ,\n        (\n        \n          x\n          \n            2", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0187", "text": "x\n          \n            2\n          \n          \n            (\n            1\n            )\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n          \n            (\n            2\n            )\n          \n        \n        )\n        ,\n        (\n        \n          x\n          \n            3\n          \n          \n            (\n            1\n            )\n          \n        \n        ,\n        \n          x\n          \n            3", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0188", "text": "3\n          \n          \n            (\n            2\n            )\n          \n        \n        )\n        ,\n        .\n        .\n        .\n        ]\n      \n    \n    {\\displaystyle [(x_{1}^{(1)},x_{1}^{(2)}),(x_{2}^{(1)},x_{2}^{(2)}),(x_{3}^{(1)},x_{3}^{(2)}),...]}\n  \n. Now pick some angle \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n. Then RoPE encoding is\n  \n    \n      \n        \n          RoPE\n        \n        \n          \n            (", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0189", "text": "(\n          \n        \n        \n          x\n          \n            m\n          \n          \n            (\n            1\n            )\n          \n        \n        ,\n        \n          x\n          \n            m\n          \n          \n            (\n            2\n            )\n          \n        \n        ,\n        m\n        \n          \n            )\n          \n        \n        =\n        \n          \n            (", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0190", "text": "cos\n                  \u2061\n                  m\n                  \u03b8\n                \n                \n                  \u2212\n                  sin\n                  \u2061\n                  m\n                  \u03b8\n                \n              \n              \n                \n                  sin\n                  \u2061\n                  m\n                  \u03b8\n                \n                \n                  cos\n                  \u2061", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0191", "text": "cos\n                  \u2061\n                  m\n                  \u03b8\n                \n              \n            \n            )\n          \n        \n        \n          \n            (\n            \n              \n                \n                  \n                    x\n                    \n                      m\n                    \n                    \n                      (\n                      1\n                      )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0192", "text": "x\n                    \n                      m\n                    \n                    \n                      (\n                      2\n                      )\n                    \n                  \n                \n              \n            \n            )\n          \n        \n        =\n        \n          \n            (", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0193", "text": "x\n                    \n                      m\n                    \n                    \n                      (\n                      1\n                      )\n                    \n                  \n                  cos\n                  \u2061\n                  m\n                  \u03b8\n                  \u2212\n                  \n                    x\n                    \n                      m", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0194", "text": "m\n                    \n                    \n                      (\n                      2\n                      )\n                    \n                  \n                  sin\n                  \u2061\n                  m\n                  \u03b8\n                \n              \n              \n                \n                  \n                    x\n                    \n                      m\n                    \n                    \n                      (\n                      2", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0195", "text": "(\n                      2\n                      )\n                    \n                  \n                  cos\n                  \u2061\n                  m\n                  \u03b8\n                  +\n                  \n                    x\n                    \n                      m\n                    \n                    \n                      (\n                      1\n                      )\n                    \n                  \n                  sin\n                  \u2061", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0196", "text": "sin\n                  \u2061\n                  m\n                  \u03b8\n                \n              \n            \n            )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0197", "text": ")\n          \n        \n      \n    \n    {\\displaystyle {\\text{RoPE}}{\\big (}x_{m}^{(1)},x_{m}^{(2)},m{\\big )}={\\begin{pmatrix}\\cos m\\theta &-\\sin m\\theta \\\\\\sin m\\theta &\\cos m\\theta \\end{pmatrix}}{\\begin{pmatrix}x_{m}^{(1)}\\\\x_{m}^{(2)}\\\\\\end{pmatrix}}={\\begin{pmatrix}x_{m}^{(1)}\\cos m\\theta -x_{m}^{(2)}\\sin m\\theta \\\\x_{m}^{(2)}\\cos m\\theta +x_{m}^{(1)}\\sin m\\theta \\\\\\end{pmatrix}}}\n  \nEquivalently, if we write the 2-dimensional vectors as complex numbers", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0198", "text": "z\n          \n            m\n          \n        \n        :=\n        \n          x\n          \n            m\n          \n          \n            (\n            1\n            )\n          \n        \n        +\n        i\n        \n          x\n          \n            m\n          \n          \n            (\n            2\n            )\n          \n        \n      \n    \n    {\\displaystyle z_{m}:=x_{m}^{(1)}+ix_{m}^{(2)}}\n  \n, then RoPE encoding is just multiplication by an angle:", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0199", "text": "RoPE\n        \n        \n          \n            (\n          \n        \n        \n          z\n          \n            m\n          \n        \n        ,\n        m\n        \n          \n            )\n          \n        \n        =\n        \n          e\n          \n            i\n            m\n            \u03b8\n          \n        \n        \n          z\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle {\\text{RoPE}}{\\big (}z_{m},m{\\big )}=e^{im\\theta }z_{m}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0200", "text": "For a list of \n  \n    \n      \n        2\n        n\n      \n    \n    {\\displaystyle 2n}\n  \n-dimensional vectors, a RoPE encoder is defined by a sequence of angles \n  \n    \n      \n        \n          \u03b8\n          \n            (\n            1\n            )\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          \u03b8\n          \n            (\n            n\n            )\n          \n        \n      \n    \n    {\\displaystyle \\theta ^{(1)},...,\\theta ^{(n)}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0201", "text": ". Then the RoPE encoding is applied to each pair of coordinates.\nThe benefit of RoPE is that the dot-product between two vectors depends on their relative location only:\n  \n    \n      \n        \n          RoPE\n        \n        \n          \n            (\n          \n        \n        x\n        ,\n        m\n        \n          \n            \n              )\n            \n          \n          \n            T\n          \n        \n        \n          RoPE\n        \n        \n          \n            (", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0202", "text": "(\n          \n        \n        y\n        ,\n        n\n        \n          \n            )\n          \n        \n        =\n        \n          RoPE\n        \n        \n          \n            (\n          \n        \n        x\n        ,\n        m\n        +\n        k\n        \n          \n            \n              )\n            \n          \n          \n            T\n          \n        \n        \n          RoPE\n        \n        \n          \n            (", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0203", "text": "(\n          \n        \n        y\n        ,\n        n\n        +\n        k\n        \n          \n            )\n          \n        \n      \n    \n    {\\displaystyle {\\text{RoPE}}{\\big (}x,m{\\big )}^{T}{\\text{RoPE}}{\\big (}y,n{\\big )}={\\text{RoPE}}{\\big (}x,m+k{\\big )}^{T}{\\text{RoPE}}{\\big (}y,n+k{\\big )}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0204", "text": "for any integer \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0205", "text": "ALiBi\nALiBi (Attention with Linear Biases) is not a replacement for the positional encoder on the original transformer. Instead, it is an additional positional encoder that is directly plugged into the attention mechanism. Specifically, the ALiBi attention mechanism is\n  \n    \n      \n        \n          \n            \n              \n                \n                  Attention\n                \n                (\n                Q\n                ,\n                K\n                ,", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0206", "text": "K\n                ,\n                V\n                )\n                =\n                \n                  softmax\n                \n                \n                  (\n                  \n                    \n                      \n                        \n                          Q\n                          \n                            K\n                            \n                              \n                                T", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0207", "text": "d\n                            \n                              k\n                            \n                          \n                        \n                      \n                    \n                    +\n                    s\n                    B\n                  \n                  )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0208", "text": ")\n                \n                V\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}{\\text{Attention}}(Q,K,V)={\\text{softmax}}\\left({\\frac {QK^{\\mathrm {T} }}{\\sqrt {d_{k}}}}+sB\\right)V\\end{aligned}}}\n  \nHere, \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n is a real number (\"scalar\"), and \n  \n    \n      \n        B\n      \n    \n    {\\displaystyle B}\n  \n is the linear bias matrix defined by", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0209", "text": "is the linear bias matrix defined by\n  \n    \n      \n        B\n        =\n        \n          \n            (\n            \n              \n                \n                  0\n                \n                \n                  1\n                \n                \n                  2\n                \n                \n                  3\n                \n                \n                  \u22ef\n                \n              \n              \n                \n                  \u2212\n                  1", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0210", "text": "\u2212\n                  1\n                \n                \n                  0\n                \n                \n                  1\n                \n                \n                  2\n                \n                \n                  \u22ef\n                \n              \n              \n                \n                  \u2212\n                  2\n                \n                \n                  \u2212\n                  1\n                \n                \n                  0", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0211", "text": "0\n                \n                \n                  1\n                \n                \n                  \u22ef\n                \n              \n              \n                \n                  \u2212\n                  3\n                \n                \n                  \u2212\n                  2\n                \n                \n                  \u2212\n                  1\n                \n                \n                  0", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0212", "text": "\u22ef\n                \n              \n              \n                \n                  \u22ee\n                \n                \n                  \u22ee\n                \n                \n                  \u22ee\n                \n                \n                  \u22ee\n                \n                \n                  \u22f1\n                \n              \n            \n            )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0213", "text": ")\n          \n        \n      \n    \n    {\\displaystyle B={\\begin{pmatrix}0&1&2&3&\\cdots \\\\-1&0&1&2&\\cdots \\\\-2&-1&0&1&\\cdots \\\\-3&-2&-1&0&\\cdots \\\\\\vdots &\\vdots &\\vdots &\\vdots &\\ddots \\\\\\end{pmatrix}}}\n  \nin other words, \n  \n    \n      \n        \n          B\n          \n            i\n            ,\n            j\n          \n        \n        =\n        j\n        \u2212\n        i\n      \n    \n    {\\displaystyle B_{i,j}=j-i}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0214", "text": "{\\displaystyle B_{i,j}=j-i}\n  \n. The idea being that the linear bias matrix is a softened mask. Just as \n  \n    \n      \n        0\n      \n    \n    {\\displaystyle 0}\n  \n represent full attention paid, and \n  \n    \n      \n        \u2212\n        \u221e\n      \n    \n    {\\displaystyle -\\infty }\n  \n represents no attention paid, the linear bias matrix increases attention paid in one direction and decreases attention paid in the other direction.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0215", "text": "ALiBi allows pretraining on short context windows, then fine-tuning on longer context windows. Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the \"bottom\" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0216", "text": "Relative Position Encodings\nRelative Position Encodings is similar to ALiBi, but more generic:\n  \n    \n      \n        \n          \n            \n              \n                \n                  Attention\n                \n                (\n                Q\n                ,\n                K\n                ,\n                V\n                )\n                =\n                \n                  softmax\n                \n                \n                  (", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0217", "text": "(\n                  \n                    \n                      \n                        \n                          Q\n                          \n                            K\n                            \n                              \n                                T\n                              \n                            \n                          \n                        \n                        \n                          \n                            d", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0218", "text": "d\n                            \n                              k\n                            \n                          \n                        \n                      \n                    \n                    +\n                    B\n                  \n                  )\n                \n                V", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0219", "text": "{\\displaystyle {\\begin{aligned}{\\text{Attention}}(Q,K,V)={\\text{softmax}}\\left({\\frac {QK^{\\mathrm {T} }}{\\sqrt {d_{k}}}}+B\\right)V\\end{aligned}}}\n  \nwhere \n  \n    \n      \n        B\n      \n    \n    {\\displaystyle B}\n  \n is a Toeplitz matrix, that is, \n  \n    \n      \n        \n          B\n          \n            i\n            ,\n            j\n          \n        \n        =\n        \n          B\n          \n            \n              i\n              \u2032", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0220", "text": "i\n              \u2032\n            \n            ,\n            \n              j\n              \u2032\n            \n          \n        \n      \n    \n    {\\displaystyle B_{i,j}=B_{i',j'}}\n  \n whenever \n  \n    \n      \n        i\n        \u2212\n        j\n        =\n        \n          i\n          \u2032\n        \n        \u2212\n        \n          j\n          \u2032\n        \n      \n    \n    {\\displaystyle i-j=i'-j'}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0221", "text": "{\\displaystyle i-j=i'-j'}\n  \n. This is contrasted with the original sinusoidal positional encoding, which is an \"absolute positional encoding\".", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0222", "text": "Efficient implementation\nThe transformer model has been implemented in standard deep learning frameworks such as TensorFlow and PyTorch. Transformers is a library produced by Hugging Face that supplies transformer-based architectures and pretrained models.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0223", "text": "KV caching\nWhen an autoregressive transformer is used for inference, such as generating text, the query vector is different at each step, but the already-computed key and value vectors are always the same. The KV caching method saves the computed key and value vectors at each attention block, so that they are not recomputed at each new token. PagedAttention applies memory paging to KV caching.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0224", "text": "If a transformer is used with a baked-in prompt, such as [\"You are a customer support agent...\"], then the key and value vectors can be computed for the prompt, and saved on disk. The saving in compute is significant when the model is used for many short real-time interactions, such as in online chatbots.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0225", "text": "In general, when a user uses an autoregressive transformer to generate a continuation to a sequence of tokens, the model would first perform a forward-pass on this sequence, whereby the KV caches over this sequence are computed. This is called prefilling. Hyperscalers serving large Transformer models may use disaggregated inference, wherein prefilling and decoding are performed on separately specialized hardware.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0226", "text": "FlashAttention\nFlashAttention is an algorithm that implements the transformer attention mechanism efficiently on a GPU. It is a communication-avoiding algorithm that performs matrix multiplications in blocks, such that each block fits within the cache of a GPU, and by careful management of the blocks it minimizes data copying between GPU caches (as data movement is slow). See the page on softmax for details.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0227", "text": "An improved version, FlashAttention-2, was developed to cater to the rising demand for language models capable of handling longer context lengths. It offers enhancements in work partitioning and parallelism, enabling it to achieve up to 230 TFLOPs/s on A100 GPUs (FP16/BF16), a 2x speed increase over the original FlashAttention.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0228", "text": "Key advancements in FlashAttention-2 include the reduction of non-matmul FLOPs, improved parallelism over the sequence length dimension, better work partitioning between GPU warps, and added support for head dimensions up to 256 and multi-query attention (MQA) and grouped-query attention (GQA).", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0229", "text": "Benchmarks revealed FlashAttention-2 to be up to 2x faster than FlashAttention and up to 9x faster than a standard attention implementation in PyTorch. Future developments include optimization for new hardware like H100 GPUs and new data types like FP8.\nFlashAttention-4 focuses on pipelining to increase instruction throughput, and was developed to perform particularly well on Blackwell GPUs.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0230", "text": "Multi-Query Attention\nMulti-Query Attention changes the Multihead Attention mechanism. Whereas normally,", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0231", "text": "MultiheadAttention\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n        =\n        \n          \n            Concat\n          \n          \n            i\n            \u2208\n            [\n            \n              n\n              \n                heads\n              \n            \n            ]\n          \n        \n        \n          (\n          \n            \n              Attention\n            \n            (\n            X", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0232", "text": "(\n            X\n            \n              W\n              \n                i\n              \n              \n                Q\n              \n            \n            ,\n            X\n            \n              W\n              \n                i\n              \n              \n                K\n              \n            \n            ,\n            X\n            \n              W\n              \n                i\n              \n              \n                V", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0233", "text": "V\n              \n            \n            )\n          \n          )\n        \n        \n          W\n          \n            O\n          \n        \n      \n    \n    {\\displaystyle {\\text{MultiheadAttention}}(Q,K,V)={\\text{Concat}}_{i\\in [n_{\\text{heads}}]}\\left({\\text{Attention}}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V})\\right)W^{O}}\n  \nwith Multi-Query Attention, there is just one \n  \n    \n      \n        \n          W\n          \n            K\n          \n        \n        ,\n        \n          W", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0234", "text": ",\n        \n          W\n          \n            V\n          \n        \n      \n    \n    {\\displaystyle W^{K},W^{V}}\n  \n, thus:", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0235", "text": "MultiQueryAttention\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n        =\n        \n          \n            Concat\n          \n          \n            i\n            \u2208\n            [\n            \n              n\n              \n                heads\n              \n            \n            ]\n          \n        \n        \n          (\n          \n            \n              Attention\n            \n            (\n            X", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0236", "text": "(\n            X\n            \n              W\n              \n                i\n              \n              \n                Q\n              \n            \n            ,\n            X\n            \n              W\n              \n                K\n              \n            \n            ,\n            X\n            \n              W\n              \n                V\n              \n            \n            )\n          \n          )\n        \n        \n          W\n          \n            O", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0237", "text": "W\n          \n            O\n          \n        \n      \n    \n    {\\displaystyle {\\text{MultiQueryAttention}}(Q,K,V)={\\text{Concat}}_{i\\in [n_{\\text{heads}}]}\\left({\\text{Attention}}(XW_{i}^{Q},XW^{K},XW^{V})\\right)W^{O}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0238", "text": "This has a neutral effect on model quality and training speed, but increases inference speed.\nMore generally, grouped-query attention (GQA) partitions attention heads into groups, each of which shares the key-value pair. MQA is GQA with one group, while standard Multihead Attention is GQA with the maximal number of groups.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0239", "text": "Multihead Latent Attention (MLA) is a low-rank approximation to standard MHA. Specifically, each hidden vector, before entering the attention mechanism, is first projected to two low-dimensional spaces (\"latent space\"), one for query and one for key-value (KV vector). This design minimizes the KV cache, as only the low-dimensional KV vector needs to be cached.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0240", "text": "Speculative decoding\nSpeculative decoding is a method to accelerate token decoding. Similarly to speculative execution in CPUs, future tokens are computed quickly, then verified. If the quickly computed tokens are incorrect, they are discarded and computed slowly.\nThe key factor in speculative decoding is that a transformer decoder can verify faster than it can decode, in the following sense.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0241", "text": "Suppose we have two transformer models like GPT-3 and GPT-3-small, both with a context window size of 512. To generate an entire context window autoregressively with greedy decoding with GPT-3, it must be run for 512 times, each time generating a token \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          x", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0242", "text": ",\n        \n          x\n          \n            512\n          \n        \n      \n    \n    {\\displaystyle x_{1},x_{2},...,x_{512}}\n  \n, taking time \n  \n    \n      \n        512\n        \n          T\n          \n            GPT-3\n          \n        \n      \n    \n    {\\displaystyle 512T_{\\text{GPT-3}}}\n  \n. However, if we had some educated guess for the values of these tokens, we could verify all of them in parallel, in one run of the model, by checking that each", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0243", "text": "x\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle x_{t}}\n  \n is indeed the token with the largest log-likelihood in the \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n-th output.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0244", "text": "{\\displaystyle t}\n  \n-th output.\nIn speculative decoding, a smaller model or some other simple heuristic is used to generate a few speculative tokens that are subsequently verified by the larger model. For example, suppose we use GPT-3-small to generate four speculative tokens: \n  \n    \n      \n        \n          \n            \n              \n                x\n                ~\n              \n            \n          \n          \n            1\n          \n        \n        ,", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0245", "text": ",\n        \n          \n            \n              \n                x\n                ~\n              \n            \n          \n          \n            2\n          \n        \n        ,\n        \n          \n            \n              \n                x\n                ~\n              \n            \n          \n          \n            3\n          \n        \n        ,\n        \n          \n            \n              \n                x\n                ~", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0246", "text": "~\n              \n            \n          \n          \n            4\n          \n        \n      \n    \n    {\\displaystyle {\\tilde {x}}_{1},{\\tilde {x}}_{2},{\\tilde {x}}_{3},{\\tilde {x}}_{4}}\n  \n. This only takes \n  \n    \n      \n        4\n        \n          T\n          \n            GPT-3-small\n          \n        \n      \n    \n    {\\displaystyle 4T_{\\text{GPT-3-small}}}\n  \n. These tokens are then run through the larger GPT-3 in one go. Suppose that", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0247", "text": "x\n                ~\n              \n            \n          \n          \n            1\n          \n        \n      \n    \n    {\\displaystyle {\\tilde {x}}_{1}}\n  \n and \n  \n    \n      \n        \n          \n            \n              \n                x\n                ~\n              \n            \n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle {\\tilde {x}}_{2}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0248", "text": "{\\displaystyle {\\tilde {x}}_{2}}\n  \n are verified by GPT-3 as what it would have picked, then those are kept, but \n  \n    \n      \n        \n          \n            \n              \n                x\n                ~\n              \n            \n          \n          \n            3\n          \n        \n      \n    \n    {\\displaystyle {\\tilde {x}}_{3}}\n  \n is not, so \n  \n    \n      \n        \n          \n            \n              \n                x\n                ~", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0249", "text": "~\n              \n            \n          \n          \n            3\n          \n        \n        ,\n        \n          \n            \n              \n                x\n                ~\n              \n            \n          \n          \n            4\n          \n        \n      \n    \n    {\\displaystyle {\\tilde {x}}_{3},{\\tilde {x}}_{4}}\n  \n are discarded, and GPT-3 is run on those. This would take \n  \n    \n      \n        4\n        \n          T\n          \n            GPT-3-small", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0250", "text": "T\n          \n            GPT-3-small\n          \n        \n        +\n        3\n        \n          T\n          \n            GPT-3\n          \n        \n      \n    \n    {\\displaystyle 4T_{\\text{GPT-3-small}}+3T_{\\text{GPT-3}}}\n  \n, which might be shorter than \n  \n    \n      \n        4\n        \n          T\n          \n            GPT-3\n          \n        \n      \n    \n    {\\displaystyle 4T_{\\text{GPT-3}}}\n  \n.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0251", "text": "{\\displaystyle 4T_{\\text{GPT-3}}}\n  \n.\nFor non-greedy decoding, similar ideas apply, except the speculative tokens are accepted or rejected stochastically, in a way that guarantees the final output distribution is the same as if speculative decoding was not used.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0252", "text": "In Multi-Token Prediction, a single forward pass creates a final embedding vector, which then is un-embedded into a token probability. However, that vector can then be further processed by another transformer block to predict the next token, and so on for arbitrarily many steps into the future. This trades off accuracy for speed, since each new token costs just one more transformer block, rather than the entire stack.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0253", "text": "Sub-quadratic transformers", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0254", "text": "Sub-quadratic transformers\nTraining transformer-based architectures can be expensive, especially for long inputs. Many methods have been developed to attempt to address the issue. In the image domain, Swin transformer is an efficient architecture that performs attention inside shifting windows. In the audio domain, SepTr decouples the attention in time and frequency domains. Long Range Arena (2020) is a standard benchmark for comparing the behavior of transformer architectures over long inputs.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0255", "text": "Alternative attention graphs\nThe standard attention graph is either all-to-all or causal, both of which scales as \n  \n    \n      \n        O\n        (\n        \n          N\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle O(N^{2})}\n  \n where \n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}\n  \n is the number of tokens in a sequence.\nReformer (2020) reduces the computational load from \n  \n    \n      \n        O\n        (\n        \n          N", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0256", "text": "O\n        (\n        \n          N\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle O(N^{2})}\n  \n to \n  \n    \n      \n        O\n        (\n        N\n        ln\n        \u2061\n        N\n        )\n      \n    \n    {\\displaystyle O(N\\ln N)}\n  \n by using locality-sensitive hashing and reversible layers.\nSparse attention uses attention graphs that grows slower than \n  \n    \n      \n        O\n        (\n        \n          N\n          \n            2", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0257", "text": "N\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle O(N^{2})}\n  \n. For example, BigBird (2020) uses random small-world networks which grows as \n  \n    \n      \n        O\n        (\n        N\n        )\n      \n    \n    {\\displaystyle O(N)}\n  \n.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0258", "text": "{\\displaystyle O(N)}\n  \n.\nOrdinary transformers require a memory size that is quadratic in the size of the context window. Attention-free transformers reduce this to a linear dependence while still retaining the advantages of a transformer by linking the key to the value.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0259", "text": "Random Feature Attention\nRandom Feature Attention (2021) uses Fourier random features:\n  \n    \n      \n        \u03c6\n        (\n        x\n        )\n        =\n        \n          \n            1\n            \n              D\n            \n          \n        \n        [\n        cos\n        \u2061\n        \u27e8\n        \n          w\n          \n            1\n          \n        \n        ,\n        x\n        \u27e9\n        ,\n        sin\n        \u2061\n        \u27e8\n        \n          w\n          \n            1", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0260", "text": "1\n          \n        \n        ,\n        x\n        \u27e9\n        ,\n        \u22ef\n        cos\n        \u2061\n        \u27e8\n        \n          w\n          \n            D\n          \n        \n        ,\n        x\n        \u27e9\n        ,\n        sin\n        \u2061\n        \u27e8\n        \n          w\n          \n            D\n          \n        \n        ,\n        x\n        \u27e9\n        \n          ]\n          \n            T", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0261", "text": "T\n          \n        \n      \n    \n    {\\displaystyle \\varphi (x)={\\frac {1}{\\sqrt {D}}}[\\cos \\langle w_{1},x\\rangle ,\\sin \\langle w_{1},x\\rangle ,\\cdots \\cos \\langle w_{D},x\\rangle ,\\sin \\langle w_{D},x\\rangle ]^{T}}\n  \nwhere \n  \n    \n      \n        \n          w\n          \n            1\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          w\n          \n            D\n          \n        \n      \n    \n    {\\displaystyle w_{1},...,w_{D}}", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0262", "text": "{\\displaystyle w_{1},...,w_{D}}\n  \n are independent samples from the normal distribution \n  \n    \n      \n        N\n        (\n        0\n        ,\n        \n          \u03c3\n          \n            2\n          \n        \n        I\n        )\n      \n    \n    {\\displaystyle N(0,\\sigma ^{2}I)}\n  \n. This choice of parameters satisfy \n  \n    \n      \n        \n          E\n        \n        [\n        \u27e8\n        \u03c6\n        (\n        x\n        )\n        ,\n        \u03c6\n        (\n        y\n        )\n        \u27e9", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0263", "text": "\u03c6\n        (\n        y\n        )\n        \u27e9\n        ]\n        =\n        \n          e\n          \n            \u2212\n            \n              \n                \n                  \u2016\n                  x\n                  \u2212\n                  y\n                  \n                    \u2016\n                    \n                      2\n                    \n                  \n                \n                \n                  2\n                  \n                    \u03c3", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0264", "text": "\u03c3\n                    \n                      2\n                    \n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\mathbb {E} [\\langle \\varphi (x),\\varphi (y)\\rangle ]=e^{-{\\frac {\\|x-y\\|^{2}}{2\\sigma ^{2}}}}}\n  \n, or \n  \n    \n      \n        \n          e\n          \n            \u27e8\n            x\n            ,\n            y\n            \u27e9\n            \n              /\n            \n            \n              \u03c3", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0265", "text": "\u03c3\n              \n                2\n              \n            \n          \n        \n        =\n        \n          E\n        \n        [\n        \u27e8\n        \n          e\n          \n            \u2016\n            x\n            \n              \u2016\n              \n                2\n              \n            \n            \n              /\n            \n            2\n            \n              \u03c3\n              \n                2", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0266", "text": "\u03c6\n        (\n        x\n        )\n        ,\n        \n          e\n          \n            \u2016\n            y\n            \n              \u2016\n              \n                2\n              \n            \n            \n              /\n            \n            2\n            \n              \u03c3\n              \n                2\n              \n            \n          \n        \n        \u03c6\n        (\n        y\n        )\n        \u27e9\n        ]\n        \u2248\n        \u27e8", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0267", "text": ")\n        \u27e9\n        ]\n        \u2248\n        \u27e8\n        \n          e\n          \n            \u2016\n            x\n            \n              \u2016\n              \n                2\n              \n            \n            \n              /\n            \n            2\n            \n              \u03c3\n              \n                2\n              \n            \n          \n        \n        \u03c6\n        (\n        x\n        )\n        ,\n        \n          e\n          \n            \u2016\n            y", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0268", "text": "\u2016\n            y\n            \n              \u2016\n              \n                2\n              \n            \n            \n              /\n            \n            2\n            \n              \u03c3\n              \n                2\n              \n            \n          \n        \n        \u03c6\n        (\n        y\n        )\n        \u27e9", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0269", "text": "y\n        )\n        \u27e9\n      \n    \n    {\\displaystyle e^{\\langle x,y\\rangle /\\sigma ^{2}}=\\mathbb {E} [\\langle e^{\\|x\\|^{2}/2\\sigma ^{2}}\\varphi (x),e^{\\|y\\|^{2}/2\\sigma ^{2}}\\varphi (y)\\rangle ]\\approx \\langle e^{\\|x\\|^{2}/2\\sigma ^{2}}\\varphi (x),e^{\\|y\\|^{2}/2\\sigma ^{2}}\\varphi (y)\\rangle }\n  \nConsequently, the one-headed attention, with one query, can be written as \n  \n    \n      \n        \n          Attention\n        \n        (\n        q\n        ,\n        K\n        ,\n        V", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0270", "text": "q\n        ,\n        K\n        ,\n        V\n        )\n        =\n        \n          softmax\n        \n        \n          (\n          \n            \n              \n                q\n                \n                  K\n                  \n                    \n                      T\n                    \n                  \n                \n              \n              \n                \n                  d\n                  \n                    k", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0271", "text": ")\n        \n        V\n        \u2248\n        \n          \n            \n              \u03c6\n              (\n              q\n              \n                )\n                \n                  T\n                \n              \n              \n                \u2211\n                \n                  i\n                \n              \n              \n                e\n                \n                  \u2016", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0272", "text": "\u2016\n                  \n                    k\n                    \n                      i\n                    \n                  \n                  \n                    \u2016\n                    \n                      2\n                    \n                  \n                  \n                    /\n                  \n                  2\n                  \n                    \u03c3\n                    \n                      2", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0273", "text": "\u03c6\n              (\n              \n                k\n                \n                  i\n                \n              \n              )\n              \n                v\n                \n                  i\n                \n                \n                  T\n                \n              \n            \n            \n              \u03c6\n              (\n              q\n              \n                )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0274", "text": ")\n                \n                  T\n                \n              \n              \n                \u2211\n                \n                  i\n                \n              \n              \n                e\n                \n                  \u2016\n                  \n                    k\n                    \n                      i\n                    \n                  \n                  \n                    \u2016\n                    \n                      2", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0275", "text": "2\n                    \n                  \n                  \n                    /\n                  \n                  2\n                  \n                    \u03c3\n                    \n                      2\n                    \n                  \n                \n              \n              \u03c6\n              (\n              \n                k\n                \n                  i\n                \n              \n              )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0276", "text": ")\n            \n          \n        \n      \n    \n    {\\displaystyle {\\text{Attention}}(q,K,V)={\\text{softmax}}\\left({\\frac {qK^{\\mathrm {T} }}{\\sqrt {d_{k}}}}\\right)V\\approx {\\frac {\\varphi (q)^{T}\\sum _{i}e^{\\|k_{i}\\|^{2}/2\\sigma ^{2}}\\varphi (k_{i})v_{i}^{T}}{\\varphi (q)^{T}\\sum _{i}e^{\\|k_{i}\\|^{2}/2\\sigma ^{2}}\\varphi (k_{i})}}}\n  \nwhere \n  \n    \n      \n        \u03c3\n        =\n        \n          d\n          \n            K\n          \n          \n            1", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0277", "text": "1\n            \n              /\n            \n            4\n          \n        \n      \n    \n    {\\displaystyle \\sigma =d_{K}^{1/4}}\n  \n. Similarly for multiple queries, and for multihead attention.\nThis approximation can be computed in linear time, as we can compute the matrix \n  \n    \n      \n        \u03c6\n        (\n        \n          k\n          \n            i\n          \n        \n        )\n        \n          v\n          \n            i", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0278", "text": "i\n          \n          \n            T\n          \n        \n      \n    \n    {\\displaystyle \\varphi (k_{i})v_{i}^{T}}\n  \n first, then multiply it with the query. In essence, we have managed to obtain a more precise version of \n  \n    \n      \n        \n          Attention\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n        =\n        \n          softmax\n        \n        \n          (\n          \n            \n              \n                Q", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0279", "text": "Q\n                \n                  K\n                  \n                    \n                      T\n                    \n                  \n                \n              \n              \n                \n                  d\n                  \n                    k\n                  \n                \n              \n            \n          \n          )\n        \n        V\n        \u2248\n        Q\n        (\n        \n          K\n          \n            T", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0280", "text": "K\n          \n            T\n          \n        \n        V\n        \n          /\n        \n        \n          \n            \n              d\n              \n                k\n              \n            \n          \n        \n        )\n      \n    \n    {\\displaystyle {\\text{Attention}}(Q,K,V)={\\text{softmax}}\\left({\\frac {QK^{\\mathrm {T} }}{\\sqrt {d_{k}}}}\\right)V\\approx Q(K^{T}V/{\\sqrt {d_{k}}})}\n  \nPerformer (2022) uses the same Random Feature Attention, but", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0281", "text": "w\n          \n            1\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          w\n          \n            D\n          \n        \n      \n    \n    {\\displaystyle w_{1},...,w_{D}}\n  \n are first independently sampled from the normal distribution \n  \n    \n      \n        N\n        (\n        0\n        ,\n        \n          \u03c3\n          \n            2\n          \n        \n        I\n        )", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0282", "text": "I\n        )\n      \n    \n    {\\displaystyle N(0,\\sigma ^{2}I)}\n  \n, then they are Gram-Schmidt processed.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0283", "text": "Multimodality\nTransformers can also be used/adapted for modalities (input or output) beyond just text, usually by finding a way to \"tokenize\" the modality.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0284", "text": "Multimodal models can either be trained from scratch, or by finetuning. A 2022 study found that transformers pretrained only on natural language can be finetuned on only 0.03% of parameters and become competitive with LSTMs on a variety of logical and visual tasks, demonstrating transfer learning. The LLaVA was a vision-language model composed of a language model (Vicuna-13B) and a vision model (ViT-L/14), connected by a linear layer. Only the linear layer is finetuned.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0285", "text": "Vision transformers adapt the transformer to computer vision by breaking down input images as a series of patches, turning them into vectors, and treating them like embedding vector of tokens in a standard transformer.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0286", "text": "Conformer and later Whisper follow the same pattern for speech recognition, first turning the speech signal into a spectrogram, which is then treated like an image, i.e. broken down into a series of patches, turned into vectors and treated like embedding vector of tokens in a standard transformer.\nPerceivers are a variant of transformers designed for multimodality.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0287", "text": "For image generation, notable architectures are DALL-E 1 (2021), Parti (2022), Phenaki (2023), and Muse (2023). Unlike later models, DALL-E is not a diffusion model. Instead, it uses a decoder-only transformer that autoregressively generates a text, followed by the token representation of an image, which is then converted by a variational autoencoder to an image", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0288", "text": ". Parti is an encoder\u2013decoder transformer, where the encoder processes a text prompt, and the decoder generates a token representation of an image. Muse is an encoder-only transformer that is trained to predict masked image tokens from unmasked image tokens. During generation, all input tokens are masked, and the highest-confidence predictions are included for the next iteration, until all tokens are predicted. Phenaki is a text-to-video model", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0289", "text": ". Phenaki is a text-to-video model. It is a bidirectional masked transformer conditioned on pre-computed text tokens. The generated tokens are then decoded to a video.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0290", "text": "Applications\nThe transformer has had great success in natural language processing (NLP). Many large language models such as GPT-2, GPT-3, GPT-4, Gemini, AlbertAGPT, Claude, BERT, Grok, XLNet, RoBERTa and ChatGPT demonstrate the ability of transformers to perform a wide variety of NLP-related subtasks and their related real-world applications, including:", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0291", "text": "machine translation\ntime series prediction\ndocument summarization\ndocument generation\nnamed entity recognition (NER)\nwriting computer code based on requirements expressed in natural language.\nspeech-to-text\nBeyond traditional NLP, the transformer architecture has had success in other applications, such as:", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0292", "text": "biological sequence analysis\nvideo understanding\nprotein folding (such as AlphaFold)\nevaluating chess board positions. Using static evaluation alone (that is, with no Minimax search) transformer achieved an Elo of 2895, putting it at grandmaster level.", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0293", "text": "See also\nseq2seq \u2013 Family of machine learning approaches\nCircuit (neural network) \u2013 Interpretable computational sub-graphs within artificial neural networks\nPerceiver \u2013 Variant of Transformer designed for multimodal data\nVision transformer \u2013 Machine learning model for vision processing\nLarge language model \u2013 Type of machine learning model\nBERT (language model) \u2013 Series of language models developed by Google AI\nGenerative pre-trained transformer \u2013 Type of large language model", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0294", "text": "T5 (language model) \u2013 Series of large language models developed by Google AI", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Transformer_deep_learning_0295", "text": "Notes\nReferences\n\n\n== Further reading ==", "title": "Transformer (deep learning)", "url": "https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}, {"id": "rec_Attention_machine_learning_0000", "text": "In machine learning, attention is a method that determines the importance of each component in a sequence relative to the other components in that sequence. In natural language processing, importance is represented by \"soft\" weights assigned to each word in a sentence. More generally, attention encodes vectors called token embeddings across a fixed-width sequence that can range from tens to millions of tokens in size.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0001", "text": "Unlike \"hard\" weights, which are computed during the backwards training pass, \"soft\" weights exist only in the forward pass and therefore change with every step of the input. Earlier designs implemented the attention mechanism in a serial recurrent neural network (RNN) language translation system, but a more recent design, namely the transformer, removed the slower sequential RNN and relied more heavily on the faster parallel attention scheme.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0002", "text": "Inspired by ideas about attention in humans, the attention mechanism was developed to address the weaknesses of using information from the hidden layers of recurrent neural networks. Recurrent neural networks favor more recent information contained in words at the end of a sentence, while information earlier in the sentence tends to be attenuated. Attention allows a token equal access to any part of a sentence directly, rather than only through the previous state.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0003", "text": "History\nAdditional surveys of the attention mechanism in deep learning are provided by Niu et al. and Soydaner.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0004", "text": "The major breakthrough came with self-attention, where each element in the input sequence attends to all others, enabling the model to capture global dependencies. This idea was central to the Transformer architecture, which replaced recurrence with attention mechanisms. As a result, Transformers became the foundation for models like BERT, T5 and generative pre-trained transformers (GPT).", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0005", "text": "Overview\nThe modern era of machine attention was revitalized by grafting an attention mechanism (Fig 1.  orange) to an Encoder-Decoder.\n\nFigure 2 shows the internal step-by-step operation of the attention block (A) in Fig 1.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0006", "text": "Interpreting attention weights\nIn translating between languages, alignment is the process of matching words from the source sentence to words of the translated sentence. Networks that perform verbatim translation without regard to word order would show the highest scores along the (dominant) diagonal of the matrix. The off-diagonal dominance shows that the attention mechanism is more nuanced.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0007", "text": "Consider an example of translating I love you to French. On the first pass through the decoder, 94% of the attention weight is on the first English word I, so the network offers the word je. On the second pass of the decoder, 88% of the attention weight is on the third English word you, so it offers t'. On the last pass, 95% of the attention weight is on the second English word love, so it offers aime.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0008", "text": "In the I love you example, the second word love is aligned with the third word aime. Stacking soft row vectors together for je, t', and aime yields an alignment matrix:", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0009", "text": "Sometimes, alignment can be multiple-to-multiple. For example, the English phrase look it up corresponds to cherchez-le. Thus, \"soft\" attention weights work better than \"hard\" attention weights (setting one attention weight to 1, and the others to 0), as we would like the model to make a context vector consisting of a weighted sum of the hidden vectors, rather than \"the best one\", as there may not be a best hidden vector.\n\nVariants\nMany variants of attention implement soft weights, such as", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0010", "text": "fast weight programmers, or fast weight controllers (1992). A \"slow\" neural network outputs the \"fast\" weights of another neural network through outer products. The slow network learns by gradient descent. It was later renamed as \"linearized self-attention\".\nBahdanau-style attention, also referred to as additive attention,\nLuong-style attention, which is known as multiplicative attention,", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0011", "text": "Early attention mechanisms similar to modern self-attention were proposed using recurrent neural networks. However, the highly parallelizable self-attention was introduced in 2017 and successfully used in the Transformer model,\npositional attention and factorized positional attention.\nFor convolutional neural networks, attention mechanisms can be distinguished by the dimension on which they operate, namely: spatial attention, channel attention, or combinations.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0012", "text": "These variants recombine the encoder-side inputs to redistribute those effects to each target output. Often, a correlation-style matrix of dot products provides the re-weighting coefficients.  In the figures below, W is the matrix of context attention weights, similar to the formula in Overview section above.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0013", "text": "Optimizations\nFlash attention", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0014", "text": "The size of the attention matrix is proportional to the square of the number of input tokens. Therefore, when the input is long, calculating the attention matrix requires a lot of GPU memory. Flash attention is an implementation that reduces the memory needs and increases efficiency without sacrificing accuracy", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0015", "text": ". It achieves this by partitioning the attention computation into smaller blocks that fit into the GPU's faster on-chip memory, reducing the need to store large intermediate matrices and thus lowering memory usage while increasing computational efficiency.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0016", "text": "FlexAttention\nFlexAttention is an attention kernel developed by Meta that allows users to modify attention scores prior to softmax and dynamically chooses the optimal attention algorithm.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0017", "text": "Applications\nAttention is widely used in natural language processing, computer vision, and speech recognition. In NLP, it improves context understanding in tasks like question answering and summarization. In vision, visual attention helps models focus on relevant image regions, enhancing object detection and image captioning.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0018", "text": "Attention maps as explanations for vision transformers", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0019", "text": "From the original paper on vision transformers (ViT), visualizing attention scores as a heat map (called saliency maps or attention maps) has become an important and routine way to inspect the decision making process of ViT models. One can compute the attention maps with respect to any attention head at any layer, while the deeper layers tend to show more semantically meaningful visualization", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0020", "text": ". Attention rollout is a recursive algorithm to combine attention scores across all layers, by computing the dot product of successive attention maps.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0021", "text": "Because vision transformers are typically trained in a self-supervised manner, attention maps are generally not class-sensitive. When a classification head is attached to the ViT backbone, class-discriminative attention maps (CDAM) combines attention maps and gradients with respect to the class [CLS] token", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0022", "text": ". Some class-sensitive interpretability methods originally developed for convolutional neural networks can be also applied to ViT, such as GradCAM, which back-propagates the gradients to the outputs of the final attention layer.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0023", "text": "Using attention as basis of explanation for the transformers in language and vision is not without debate. While some pioneering papers analyzed and framed attention scores as explanations, higher attention scores do not always correlate with greater impact on model performances.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0024", "text": "Mathematical representation\nStandard scaled dot-product attention\nFor matrices: \n  \n    \n      \n        Q\n        \u2208\n        \n          \n            R\n          \n          \n            m\n            \u00d7\n            \n              d\n              \n                k\n              \n            \n          \n        \n        ,\n        K\n        \u2208\n        \n          \n            R\n          \n          \n            n\n            \u00d7\n            \n              d\n              \n                k", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0025", "text": "d\n              \n                k\n              \n            \n          \n        \n      \n    \n    {\\displaystyle Q\\in \\mathbb {R} ^{m\\times d_{k}},K\\in \\mathbb {R} ^{n\\times d_{k}}}\n  \n and \n  \n    \n      \n        V\n        \u2208\n        \n          \n            R\n          \n          \n            n\n            \u00d7\n            \n              d\n              \n                v", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0026", "text": "{\\displaystyle V\\in \\mathbb {R} ^{n\\times d_{v}}}\n  \n, the scaled dot-product, or QKV attention, is defined as:", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0027", "text": "Attention\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n        =\n        \n          softmax\n        \n        \n          (\n          \n            \n              \n                Q\n                \n                  K\n                  \n                    T\n                  \n                \n              \n              \n                \n                  d\n                  \n                    k", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0028", "text": "k\n                  \n                \n              \n            \n          \n          )\n        \n        V\n        \u2208\n        \n          \n            R\n          \n          \n            m\n            \u00d7\n            \n              d\n              \n                v\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\text{Attention}}(Q,K,V)={\\text{softmax}}\\left({\\frac {QK^{T}}{\\sqrt {d_{k}}}}\\right)V\\in \\mathbb {R} ^{m\\times d_{v}}}", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0029", "text": "where", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0030", "text": "T\n          \n        \n      \n    \n    {\\displaystyle {}^{T}}\n  \n denotes transpose and the softmax function is applied independently to every row of its argument. The matrix \n  \n    \n      \n        Q\n      \n    \n    {\\displaystyle Q}\n  \n contains \n  \n    \n      \n        m\n      \n    \n    {\\displaystyle m}\n  \n queries, while matrices \n  \n    \n      \n        K\n        ,\n        V\n      \n    \n    {\\displaystyle K,V}\n  \n jointly contain an unordered set of", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0031", "text": "jointly contain an unordered set of \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n key-value pairs. Value vectors in matrix \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n are weighted using the weights resulting from the softmax operation, so that the rows of the \n  \n    \n      \n        m\n      \n    \n    {\\displaystyle m}\n  \n-by-\n  \n    \n      \n        \n          d\n          \n            v\n          \n        \n      \n    \n    {\\displaystyle d_{v}}", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0032", "text": "{\\displaystyle d_{v}}\n  \n output matrix are confined to the convex hull of the points in \n  \n    \n      \n        \n          \n            R\n          \n          \n            \n              d\n              \n                v\n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\mathbb {R} ^{d_{v}}}\n  \n given by the rows of \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0033", "text": "V\n      \n    \n    {\\displaystyle V}\n  \n.\nTo understand the permutation invariance and permutation equivariance properties of QKV attention, let \n  \n    \n      \n        A\n        \u2208\n        \n          \n            R\n          \n          \n            m\n            \u00d7\n            m\n          \n        \n      \n    \n    {\\displaystyle A\\in \\mathbb {R} ^{m\\times m}}\n  \n and \n  \n    \n      \n        B\n        \u2208\n        \n          \n            R\n          \n          \n            n\n            \u00d7", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0034", "text": "n\n            \u00d7\n            n\n          \n        \n      \n    \n    {\\displaystyle B\\in \\mathbb {R} ^{n\\times n}}\n  \n be permutation matrices; and \n  \n    \n      \n        D\n        \u2208\n        \n          \n            R\n          \n          \n            m\n            \u00d7\n            n\n          \n        \n      \n    \n    {\\displaystyle D\\in \\mathbb {R} ^{m\\times n}}\n  \n an arbitrary matrix. The softmax function is permutation equivariant in the sense that:", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0035", "text": "softmax\n        \n        (\n        A\n        D\n        B\n        )\n        =\n        A\n        \n        \n          softmax\n        \n        (\n        D\n        )\n        B\n      \n    \n    {\\displaystyle {\\text{softmax}}(ADB)=A\\,{\\text{softmax}}(D)B}\n  \n\nBy noting that the transpose of a permutation matrix is also its inverse, it follows that:", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0036", "text": "Attention\n        \n        (\n        A\n        Q\n        ,\n        B\n        K\n        ,\n        B\n        V\n        )\n        =\n        A\n        \n        \n          Attention\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n      \n    \n    {\\displaystyle {\\text{Attention}}(AQ,BK,BV)=A\\,{\\text{Attention}}(Q,K,V)}", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0037", "text": "which shows that QKV attention is equivariant with respect to re-ordering the queries (rows of \n  \n    \n      \n        Q\n      \n    \n    {\\displaystyle Q}\n  \n); and invariant to re-ordering of the key-value pairs in \n  \n    \n      \n        K\n        ,\n        V\n      \n    \n    {\\displaystyle K,V}\n  \n. These properties are inherited when applying linear transforms to the inputs and outputs of QKV attention blocks. For example, a simple self-attention function defined as:", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0038", "text": "X\n        \u21a6\n        \n          Attention\n        \n        (\n        X\n        \n          T\n          \n            q\n          \n        \n        ,\n        X\n        \n          T\n          \n            k\n          \n        \n        ,\n        X\n        \n          T\n          \n            v\n          \n        \n        )\n      \n    \n    {\\displaystyle X\\mapsto {\\text{Attention}}(XT_{q},XT_{k},XT_{v})}", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0039", "text": "is permutation equivariant with respect to re-ordering the rows of the input matrix \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n in a non-trivial way, because every row of the output is a function of all the rows of the input. Similar properties hold for multi-head attention, which is defined below.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0040", "text": "Masked attention\nWhen QKV attention is used as a building block for an autoregressive decoder, and when at training time all input and output matrices have \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n rows, a masked attention variant is used:", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0041", "text": "Attention\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n        =\n        \n          softmax\n        \n        \n          (\n          \n            \n              \n                \n                  Q\n                  \n                    K\n                    \n                      T\n                    \n                  \n                \n                \n                  \n                    d", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0042", "text": "d\n                    \n                      k\n                    \n                  \n                \n              \n            \n            +\n            M\n          \n          )\n        \n        V\n      \n    \n    {\\displaystyle {\\text{Attention}}(Q,K,V)={\\text{softmax}}\\left({\\frac {QK^{T}}{\\sqrt {d_{k}}}}+M\\right)V}", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0043", "text": "where the mask, \n  \n    \n      \n        M\n        \u2208\n        \n          \n            R\n          \n          \n            n\n            \u00d7\n            n\n          \n        \n      \n    \n    {\\displaystyle M\\in \\mathbb {R} ^{n\\times n}}\n  \n is a strictly upper triangular matrix, with zeros on and below the diagonal and \n  \n    \n      \n        \u2212\n        \u221e\n      \n    \n    {\\displaystyle -\\infty }\n  \n in every element above the diagonal. The softmax output, also in", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0044", "text": "R\n          \n          \n            n\n            \u00d7\n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {R} ^{n\\times n}}\n  \n is then lower triangular, with zeros in all elements above the diagonal. The masking ensures that for all \n  \n    \n      \n        1\n        \u2264\n        i\n        <\n        j\n        \u2264\n        n\n      \n    \n    {\\displaystyle 1\\leq i<j\\leq n}\n  \n, row \n  \n    \n      \n        i", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0045", "text": ", row \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n of the attention output is independent of row \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n of any of the three input matrices. The permutation invariance and equivariance properties of standard QKV attention do not hold for the masked variant.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0046", "text": "Multi-head attention\nMulti-head attention", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0047", "text": "MultiHead\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n        =\n        \n          Concat\n        \n        (\n        \n          \n            head\n          \n          \n            1\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          \n            head\n          \n          \n            h\n          \n        \n        )\n        \n          W\n          \n            O", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0048", "text": "O\n          \n        \n      \n    \n    {\\displaystyle {\\text{MultiHead}}(Q,K,V)={\\text{Concat}}({\\text{head}}_{1},...,{\\text{head}}_{h})W^{O}}", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0049", "text": "where each head is computed with QKV attention as:", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0050", "text": "head\n          \n          \n            i\n          \n        \n        =\n        \n          Attention\n        \n        (\n        Q\n        \n          W\n          \n            i\n          \n          \n            Q\n          \n        \n        ,\n        K\n        \n          W\n          \n            i\n          \n          \n            K\n          \n        \n        ,\n        V\n        \n          W\n          \n            i", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0051", "text": "i\n          \n          \n            V\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\text{head}}_{i}={\\text{Attention}}(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})}", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0052", "text": "and \n  \n    \n      \n        \n          W\n          \n            i\n          \n          \n            Q\n          \n        \n        ,\n        \n          W\n          \n            i\n          \n          \n            K\n          \n        \n        ,\n        \n          W\n          \n            i\n          \n          \n            V\n          \n        \n      \n    \n    {\\displaystyle W_{i}^{Q},W_{i}^{K},W_{i}^{V}}\n  \n, and \n  \n    \n      \n        \n          W\n          \n            O", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0053", "text": "O\n          \n        \n      \n    \n    {\\displaystyle W^{O}}\n  \n are parameter matrices.\nThe permutation properties of (standard, unmasked) QKV attention apply here also. For permutation matrices, \n  \n    \n      \n        A\n        ,\n        B\n      \n    \n    {\\displaystyle A,B}\n  \n:", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0054", "text": "MultiHead\n        \n        (\n        A\n        Q\n        ,\n        B\n        K\n        ,\n        B\n        V\n        )\n        =\n        A\n        \n        \n          MultiHead\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n      \n    \n    {\\displaystyle {\\text{MultiHead}}(AQ,BK,BV)=A\\,{\\text{MultiHead}}(Q,K,V)}\n  \n\nfrom which we also see that multi-head self-attention:", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0055", "text": "X\n        \u21a6\n        \n          MultiHead\n        \n        (\n        X\n        \n          T\n          \n            q\n          \n        \n        ,\n        X\n        \n          T\n          \n            k\n          \n        \n        ,\n        X\n        \n          T\n          \n            v\n          \n        \n        )\n      \n    \n    {\\displaystyle X\\mapsto {\\text{MultiHead}}(XT_{q},XT_{k},XT_{v})}", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0056", "text": "is equivariant with respect to re-ordering of the rows of input matrix \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0057", "text": "Bahdanau (additive) attention\nAttention\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n        =\n        \n          softmax\n        \n        (\n        tanh\n        \u2061\n        (\n        \n          W\n          \n            Q\n          \n        \n        Q\n        +\n        \n          W\n          \n            K\n          \n        \n        K\n        )\n        )\n        V\n      \n    \n    {\\displaystyle {\\text{Attention}}(Q,K,V)={\\text{softmax}}(\\tanh(W_{Q}Q+W_{K}K))V}", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0058", "text": "where \n  \n    \n      \n        \n          W\n          \n            Q\n          \n        \n      \n    \n    {\\displaystyle W_{Q}}\n  \n and \n  \n    \n      \n        \n          W\n          \n            K\n          \n        \n      \n    \n    {\\displaystyle W_{K}}\n  \n are learnable weight matrices.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0059", "text": "Luong attention (general)\nAttention\n        \n        (\n        Q\n        ,\n        K\n        ,\n        V\n        )\n        =\n        \n          softmax\n        \n        (\n        Q\n        W\n        \n          K\n          \n            T\n          \n        \n        )\n        V\n      \n    \n    {\\displaystyle {\\text{Attention}}(Q,K,V)={\\text{softmax}}(QWK^{T})V}\n  \n\nwhere \n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}\n  \n is a learnable weight matrix.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0060", "text": "Self-attention\nSelf-attention is essentially the same as cross-attention, except that query, key, and value vectors all come from the same model. Both encoder and decoder can use self-attention, but with subtle differences.\nFor encoder self-attention, we can start with a simple encoder without self-attention, such as an \"embedding layer\", which simply converts each input word into a vector by a fixed lookup table. This gives a sequence of hidden vectors \n  \n    \n      \n        \n          h", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0061", "text": "h\n          \n            0\n          \n        \n        ,\n        \n          h\n          \n            1\n          \n        \n        ,\n        \u2026\n      \n    \n    {\\displaystyle h_{0},h_{1},\\dots }\n  \n. These can then be applied to a dot-product attention mechanism, to obtain\n  \n    \n      \n        \n          \n            \n              \n                \n                  h\n                  \n                    0\n                  \n                  \u2032", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0062", "text": "\u2032\n                \n              \n              \n                \n                =\n                \n                  A\n                  t\n                  t\n                  e\n                  n\n                  t\n                  i\n                  o\n                  n\n                \n                (\n                \n                  h\n                  \n                    0", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0063", "text": "W\n                  \n                    Q\n                  \n                \n                ,\n                H\n                \n                  W\n                  \n                    K\n                  \n                \n                ,\n                H\n                \n                  W\n                  \n                    V\n                  \n                \n                )", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0064", "text": "h\n                  \n                    1\n                  \n                  \u2032\n                \n              \n              \n                \n                =\n                \n                  A\n                  t\n                  t\n                  e\n                  n\n                  t\n                  i\n                  o\n                  n\n                \n                (", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0065", "text": "(\n                \n                  h\n                  \n                    1\n                  \n                \n                \n                  W\n                  \n                    Q\n                  \n                \n                ,\n                H\n                \n                  W\n                  \n                    K\n                  \n                \n                ,\n                H\n                \n                  W", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0066", "text": "W\n                  \n                    V\n                  \n                \n                )\n              \n            \n            \n              \n              \n                \n                \n                \n                \u22ee", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0067", "text": "{\\displaystyle {\\begin{aligned}h_{0}'&=\\mathrm {Attention} (h_{0}W^{Q},HW^{K},HW^{V})\\\\h_{1}'&=\\mathrm {Attention} (h_{1}W^{Q},HW^{K},HW^{V})\\\\&\\;\\,\\vdots \\end{aligned}}}\n  \nor more succinctly, \n  \n    \n      \n        \n          H\n          \u2032\n        \n        =\n        \n          A\n          t\n          t\n          e\n          n\n          t\n          i\n          o\n          n\n        \n        (\n        H\n        \n          W", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0068", "text": "H\n        \n          W\n          \n            Q\n          \n        \n        ,\n        H\n        \n          W\n          \n            K\n          \n        \n        ,\n        H\n        \n          W\n          \n            V\n          \n        \n        )\n      \n    \n    {\\displaystyle H'=\\mathrm {Attention} (HW^{Q},HW^{K},HW^{V})}", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0069", "text": ". This can be applied repeatedly, to obtain a multilayered encoder. This is the \"encoder self-attention\", sometimes called the \"all-to-all attention\", as the vector at every position can attend to every other.", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0070", "text": "Masking\nFor decoder self-attention, all-to-all attention is inappropriate, because during the autoregressive decoding process, the decoder cannot attend to future outputs that has yet to be decoded. This can be solved by forcing the attention weights \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle w_{ij}=0}\n  \n for all \n  \n    \n      \n        i\n        <\n        j", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0071", "text": "i\n        <\n        j\n      \n    \n    {\\displaystyle i<j}\n  \n, called \"causal masking\". This attention mechanism is the \"causally masked self-attention\".", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0072", "text": "See also\nRecurrent neural network\nseq2seq\nTransformer (deep learning architecture)\nAttention\nDynamic neural network", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_Attention_machine_learning_0073", "text": "References\nExternal links\nOlah, Chris; Carter, Shan (September 8, 2016). \"Attention and Augmented Recurrent Neural Networks\". Distill. 1 (9). Distill Working Group. doi:10.23915/distill.00001.\nDan Jurafsky and James H. Martin (2022). Speech and Language Processing (3rd ed. draft, January 2022) \u2014 Chapter 10.4 (Attention) and Chapter 9.7 (Self-Attention Networks: Transformers)\nAlex Graves (2020). Attention and Memory in Deep Learning \u2014 video lecture from DeepMind / UCL", "title": "Attention (machine learning)", "url": "https://en.wikipedia.org/wiki/Attention_(machine_learning)"}, {"id": "rec_BERT_language_model_0000", "text": "Bidirectional encoder representations from transformers (BERT) is a language model introduced in October 2018 by researchers at Google. It learns to represent text as a sequence of vectors using self-supervised learning. It uses the encoder-only transformer architecture. BERT dramatically improved the state of the art for large language models. As of 2020, BERT is a ubiquitous baseline in natural language processing (NLP) experiments.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0001", "text": "BERT is trained by masked token prediction and next sentence prediction. With this training, BERT learns contextual, latent representations of tokens in their context, similar to ELMo and GPT-2. It found applications for many natural language processing tasks, such as coreference resolution and polysemy resolution. It improved on ELMo and spawned the study of \"BERTology\", which attempts to interpret what is learned by BERT.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0002", "text": "BERT was originally implemented in the English language at two model sizes, BERTBASE (110 million parameters) and BERTLARGE (340 million parameters). Both were trained on the Toronto BookCorpus (800M words) and English Wikipedia  (2,500M words). The weights were released on GitHub. On March 11, 2020, 24 smaller models were released, the smallest being BERTTINY with just 4 million parameters.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0003", "text": "Architecture\nBERT is an \"encoder-only\" transformer architecture. At a high level, BERT consists of 4 modules:", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0004", "text": "Tokenizer: This module converts a piece of English text into a sequence of integers (\"tokens\").\nEmbedding: This module converts the sequence of tokens into an array of real-valued vectors representing the tokens. It represents the conversion of discrete token types into a lower-dimensional Euclidean space.\nEncoder: a stack of Transformer blocks with self-attention, but without causal masking.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0005", "text": "Task head: This module converts the final representation vectors into one-hot encoded tokens again by producing a predicted probability distribution over the token types. It can be viewed as a simple decoder, decoding the latent representation into token types, or as an \"un-embedding layer\".", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0006", "text": "The task head is necessary for pre-training, but it is often unnecessary for so-called \"downstream tasks,\" such as question answering or sentiment classification. Instead, one removes the task head and replaces it with a newly initialized module suited for the task, and finetune the new module. The latent vector representation of the model is directly fed into this new module, allowing for sample-efficient transfer learning.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0007", "text": "Embedding\nThis section describes the embedding used by BERTBASE. The other one, BERTLARGE, is similar, just larger.\nThe tokenizer of BERT is WordPiece, which is a sub-word strategy like byte-pair encoding. Its vocabulary size is 30,000, and any token not appearing in its vocabulary is replaced by [UNK] (\"unknown\"). \n\nThe first layer is the embedding layer, which contains three components: token type embeddings, position embeddings, and segment type embeddings.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0008", "text": "Token type: The token type is a standard embedding layer, translating a one-hot vector into a dense vector based on its token type.\nPosition: The position embeddings are based on a token's position in the sequence. BERT uses absolute position embeddings, where each position in a sequence is mapped to a real-valued vector. Each dimension of the vector consists of a sinusoidal function that takes the position in the sequence as input.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0009", "text": "Segment type: Using a vocabulary of just 0 or 1, this embedding layer produces a dense vector based on whether the token belongs to the first or second text segment in that input. In other words, type-1 tokens are all tokens that appear after the [SEP] special token. All prior tokens are type-0.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0010", "text": "The three embedding vectors are added together representing the initial token representation as a function of these three pieces of information. After embedding, the vector representation is normalized using a LayerNorm operation, outputting a 768-dimensional vector for each input token. After this, the representation vectors are passed forward through 12 Transformer encoder blocks, and are decoded back to 30,000-dimensional vocabulary space using a basic affine transformation layer.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0011", "text": "Architectural family\nThe encoder stack of BERT has 2 free parameters: \n  \n    \n      \n        L\n      \n    \n    {\\displaystyle L}\n  \n, the number of layers, and \n  \n    \n      \n        H\n      \n    \n    {\\displaystyle H}\n  \n, the hidden size. There are always \n  \n    \n      \n        H\n        \n          /\n        \n        64\n      \n    \n    {\\displaystyle H/64}\n  \n self-attention heads, and the feed-forward/filter size is always \n  \n    \n      \n        4\n        H", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0012", "text": "4\n        H\n      \n    \n    {\\displaystyle 4H}\n  \n. By varying these two numbers, one obtains an entire family of BERT models.\nFor BERT:", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0013", "text": "the feed-forward size and filter size are synonymous. Both of them denote the number of dimensions in the middle layer of the feed-forward network.\nthe hidden size and embedding size are synonymous. Both of them denote the number of real numbers used to represent a token.\nThe notation for encoder stack is written as L/H. For example, BERTBASE is written as 12L/768H, BERTLARGE as 24L/1024H, and BERTTINY as 2L/128H.\n\nTraining\nPre-training\nBERT was pre-trained simultaneously on two tasks:", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0014", "text": "Masked language modeling (MLM): In this task, BERT ingests a sequence of words, where one word may be randomly changed (\"masked\"), and BERT tries to predict the original words that had been changed. For example, in the sentence \"The cat sat on the [MASK],\" BERT would need to predict \"mat.\" This helps BERT learn bidirectional context, meaning it understands the relationships between words not just from left to right or right to left but from both directions at the same time.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0015", "text": "Next sentence prediction (NSP): In this task, BERT is trained to predict whether one sentence logically follows another. For example, given two sentences, \"The cat sat on the mat\" and \"It was a sunny day\", BERT has to decide if the second sentence is a valid continuation of the first one. This helps BERT understand relationships between sentences, which is important for tasks like question answering or document classification.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0016", "text": "Masked language modeling\nIn masked language modeling, 15% of tokens would be randomly selected for masked-prediction task, and the training objective was to predict the masked token given its context. In more detail, the selected token is:", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0017", "text": "replaced with a [MASK] token with probability 80%,\nreplaced with a random word token with probability 10%,\nnot replaced with probability 10%.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0018", "text": "The reason not all selected tokens are masked is to avoid the dataset shift problem. The dataset shift problem arises when the distribution of inputs seen during training differs significantly from the distribution encountered during inference. A trained BERT model might be applied to word representation (like Word2Vec), where it would be run over sentences not containing any [MASK] tokens. It is later found that more diverse training objectives are generally better.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0019", "text": "As an illustrative example, consider the sentence \"my dog is cute\". It would first be divided into tokens like \"my1 dog2 is3 cute4\". Then a random token in the sentence would be picked. Let it be the 4th one \"cute4\". Next, there would be three possibilities:", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0020", "text": "with probability 80%, the chosen token is masked, resulting in \"my1 dog2 is3 [MASK]4\";\nwith probability 10%, the chosen token is replaced by a uniformly sampled random token, such as \"happy\", resulting in \"my1 dog2 is3 happy4\";\nwith probability 10%, nothing is done, resulting in \"my1 dog2 is3 cute4\".\nAfter processing the input text, the model's 4th output vector is passed to its decoder layer, which outputs a probability distribution over its 30,000-dimensional vocabulary space.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0021", "text": "Next sentence prediction\nGiven two sentences, the model predicts if they appear sequentially in the training corpus, outputting either [IsNext] or [NotNext]. During training, the algorithm sometimes samples two sentences from a single continuous span in the training corpus, while at other times, it samples two sentences from two discontinuous spans.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0022", "text": "The first sentence starts with a special token, [CLS] (for \"classify\"). The two sentences are separated by another special token, [SEP] (for \"separate\"). After processing the two sentences, the final vector for the [CLS] token is passed to a linear layer for binary classification into [IsNext] and [NotNext].\nFor example:", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0023", "text": "Given \"[CLS] my dog is cute [SEP] he likes playing [SEP]\", the model should predict [IsNext].\nGiven \"[CLS] my dog is cute [SEP] how do magnets work [SEP]\", the model should predict [NotNext].", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0024", "text": "Fine-tuning\nBERT is meant as a general pretrained model for various applications in natural language processing. That is, after pre-training, BERT can be fine-tuned with fewer resources on smaller datasets to optimize its performance on specific tasks such as natural language inference and text classification, and sequence-to-sequence-based language generation tasks such as question answering and conversational response generation.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0025", "text": "The original BERT paper published results demonstrating that a small amount of finetuning (for BERTLARGE, 1 hour on 1 Cloud TPU) allowed it to achieved state-of-the-art performance on a number of natural language understanding tasks:", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0026", "text": "GLUE (General Language Understanding Evaluation) task set (consisting of 9 tasks);\nSQuAD (Stanford Question Answering Dataset) v1.1 and v2.0;\nSWAG (Situations With Adversarial Generations).\nIn the original paper, all parameters of BERT are fine-tuned, and recommended that, for downstream applications that are text classifications, the output token at the [CLS] input token is fed into a linear-softmax layer to produce the label outputs.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0027", "text": "The original code base defined the final linear layer as a \"pooler layer\", in analogy with global pooling in computer vision, even though it simply discards all output tokens except the one corresponding to  [CLS] .", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0028", "text": "Cost\nBERT was trained on the BookCorpus (800M words) and a filtered version of English Wikipedia (2,500M words) without lists, tables, and headers.\nTraining BERTBASE  on 4 cloud TPU (16 TPU chips total) took 4 days, at an estimated cost of 500 USD. Training BERTLARGE on 16 cloud TPU (64 TPU chips total) took 4 days.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0029", "text": "Interpretation", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0030", "text": "Language models like ELMo, GPT-2, and BERT, spawned the study of \"BERTology\", which attempts to interpret what is learned by these models. Their performance on these natural language understanding tasks are not yet well understood", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0031", "text": ". Several research publications in 2018 and 2019 focused on investigating the relationship behind BERT's output as a result of carefully chosen input sequences, analysis of internal vector representations through probing classifiers, and the relationships represented by attention weights.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0032", "text": "The high performance of the BERT model could also be attributed to the fact that it is bidirectionally trained. This means that BERT, based on the Transformer model architecture, applies its self-attention mechanism to learn information from a text from the left and right side during training, and consequently gains a deep understanding of the context. For example, the word fine can have two different meanings depending on the context (I feel fine today, She has fine blond hair)", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0033", "text": ". BERT considers the words surrounding the target word fine from the left and right side.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0034", "text": "However it comes at a cost: due to encoder-only architecture lacking a decoder, BERT can't be prompted and can't generate text, while bidirectional models in general do not work effectively without the right side, thus being difficult to prompt. As an illustrative example, if one wishes to use BERT to continue a sentence fragment \"Today, I went to\", then naively one would mask out all the tokens as \"Today, I went to  [MASK]  [MASK]  [MASK] ..", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0035", "text": ".  [MASK] .\" where the number of  [MASK]  is the length of the sentence one wishes to extend to. However, this constitutes a dataset shift, as during training, BERT has never seen sentences with that many tokens masked out. Consequently, its performance degrades. More sophisticated techniques allow text generation, but at a high computational cost.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0036", "text": "History", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0037", "text": "BERT was originally published by Google researchers Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. The design has its origins from pre-training contextual representations, including semi-supervised sequence learning, generative pre-training, ELMo, and ULMFit. Unlike previous models, BERT is a deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0038", "text": ". Context-free models such as word2vec or GloVe generate a single word embedding representation for each word in the vocabulary, whereas BERT takes into account the context for each occurrence of a given word", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0039", "text": ". For instance, whereas the vector for \"running\" will have the same word2vec vector representation for both of its occurrences in the sentences \"He is running a company\" and \"He is running a marathon\", BERT will provide a contextualized embedding that will be different according to the sentence.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0040", "text": "On October 25, 2019, Google announced that they had started applying BERT models to English-language search queries on Google Search within the US. On December 9, 2019, it was reported that BERT had been adopted by Google Search for over 70 languages. In October 2020, almost every single English-based query was processed by a BERT model.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0041", "text": "Variants\nThe BERT models were influential and inspired many variants.\nRoBERTa (2019) was an engineering improvement. It preserves BERT's architecture (slightly larger, at 355M parameters), but improves its training, changing key hyperparameters, removing the next-sentence prediction task, and using much larger mini-batch sizes. \nXLM-RoBERTa (2019) was a multilingual RoBERTa model. It was one of the first works on multilingual language modeling at scale.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0042", "text": "DistilBERT (2019) distills BERTBASE to a model with just 60% of its parameters (66M), while preserving 95% of its benchmark scores. Similarly, TinyBERT (2019) is a distilled model with just 28% of its parameters.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0043", "text": "ALBERT (2019) used shared-parameter across layers, and experimented with independently varying the hidden size and the word-embedding layer's output size as two hyperparameters. They also replaced the next sentence prediction task with the sentence-order prediction (SOP) task, where the model must distinguish the correct order of two consecutive text segments from their reversed order.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0044", "text": "ELECTRA (2020) applied the idea of generative adversarial networks to the MLM task. Instead of masking out tokens, a small language model generates random plausible substitutions, and a larger network identify these replaced tokens. The small model aims to fool the large model.", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0045", "text": "DeBERTa (2020) is a significant architectural variant, with disentangled attention. Its key idea is to treat the positional and token encodings separately throughout the attention mechanism. Instead of combining the positional encoding (\n  \n    \n      \n        \n          x\n          \n            \n              p\n              o\n              s\n              i\n              t\n              i\n              o\n              n", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0046", "text": "{\\displaystyle x_{\\mathrm {position} }}\n  \n) and token encoding (\n  \n    \n      \n        \n          x\n          \n            \n              t\n              o\n              k\n              e\n              n\n            \n          \n        \n      \n    \n    {\\displaystyle x_{\\mathrm {token} }}\n  \n) into a single input vector (\n  \n    \n      \n        \n          x\n          \n            \n              i\n              n\n              p\n              u", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0047", "text": "n\n              p\n              u\n              t\n            \n          \n        \n        =\n        \n          x\n          \n            \n              p\n              o\n              s\n              i\n              t\n              i\n              o\n              n\n            \n          \n        \n        +\n        \n          x\n          \n            \n              t\n              o\n              k\n              e\n              n", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0048", "text": "{\\displaystyle x_{\\mathrm {input} }=x_{\\mathrm {position} }+x_{\\mathrm {token} }}\n  \n), DeBERTa keeps them separate as a tuple: \n  \n    \n      \n        (\n        \n          x\n          \n            \n              p\n              o\n              s\n              i\n              t\n              i\n              o\n              n\n            \n          \n        \n        ,\n        \n          x\n          \n            \n              t\n              o", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0049", "text": "t\n              o\n              k\n              e\n              n\n            \n          \n        \n        )\n      \n    \n    {\\displaystyle (x_{\\mathrm {position} },x_{\\mathrm {token} })}\n  \n. Then, at each self-attention layer, DeBERTa computes three distinct attention matrices, rather than the single attention matrix used in BERT:", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_BERT_language_model_0050", "text": "The three attention matrices are added together element-wise, then passed through a softmax layer and multiplied by a projection matrix.\nAbsolute position encoding is included in the final self-attention layer as additional input.\n\nNotes\nReferences\nFurther reading\nRogers, Anna; Kovaleva, Olga; Rumshisky, Anna (2020). \"A Primer in BERTology: What we know about how BERT works\". arXiv:2002.12327 [cs.CL].\n\nExternal links\nOfficial GitHub repository", "title": "BERT (language model)", "url": "https://en.wikipedia.org/wiki/BERT_(language_model)"}, {"id": "rec_Generative_pretrained_transformer_0000", "text": "A generative pre-trained transformer (GPT) is a type of large language model (LLM) that is widely used in generative artificial intelligence chatbots. GPTs are based on a deep learning architecture called the transformer. They are pre-trained on large datasets of unlabeled content, and able to generate novel content.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0001", "text": "OpenAI was the first to apply generative pre-training to the transformer architecture, introducing the GPT-1 model in 2018. The company has since released many bigger GPT models. The chatbot ChatGPT, released in late 2022 (using GPT-3.5), was followed by many competitor chatbots using their own generative pre-trained transformers to generate text, such as Gemini, DeepSeek and Claude.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0002", "text": "GPTs are primarily used to generate text, but can be trained to generate other kinds of data. For example, GPT-4o can process and generate text, images and audio. To improve performance on complex tasks, some GPTs, such as OpenAI o3, allocate more computation time analyzing the problem before generating an output, and are called reasoning models", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0003", "text": ". In 2025, GPT-5 was released with a router that automatically selects whether to use a faster model or slower reasoning model based on the provided task.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0004", "text": "Background\nDuring the 2010s, improved machine learning algorithms, more powerful computers, and an increase in the amount of digitized material allowed for an AI boom.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0005", "text": "Separately, the concept of generative pre-training (GP) was a long-established technique in machine learning. GP is a form of self-supervised learning wherein a model is first trained on a large, unlabeled dataset (the \"pre-training\" step) to learn to generate data points. This pre-trained model is then adapted to a specific task using a labeled dataset (the \"fine-tuning\" step).", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0006", "text": "The transformer architecture for deep learning is the core technology of a GPT. Developed by researchers at Google, it was introduced in the paper \"Attention Is All You Need\", which was released on June 12, 2017. The transformer architecture solved many of the performance issues that were associated with older recurrent neural network (RNN) designs for natural language processing (NLP)", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0007", "text": ". The architecture's use of an attention mechanism allows models to process entire sequences of text at once, enabling the training of much larger and more sophisticated models. Since 2017, available transformer-based NLP systems have been capable of processing, mining, organizing, connecting, contrasting, and summarizing texts as well as answering questions from textual input.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0008", "text": "History", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0009", "text": "On June 11, 2018, OpenAI researchers and engineers published a paper called \"Improving Language Understanding by Generative Pre-Training\", which introduced GPT-1, the first GPT model. It was designed as a transformer-based large language model that used generative pre-training (GP) on BookCorpus, a diverse text corpus, followed by discriminative fine-tuning to focus on specific language tasks. This semi-supervised approach was seen as a breakthrough", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0010", "text": ". Previously, the best-performing neural models in natural language processing (NLP) had commonly employed supervised learning from large amounts of manually labeled data \u2013 training a large language model with this approach would have been prohibitively expensive and time-consuming.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0011", "text": "On February 14, 2019, OpenAI introduced GPT-2, a larger model that could generate coherent text. Created as a direct scale-up of its predecessor, it had both its parameter count and dataset size increased by a factor of 10. GPT-2 has 1.5 billion parameters and was trained on WebText, a 40-gigabyte dataset of 8 million web pages", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0012", "text": ". Citing risks of malicious use, OpenAI opted for a \"staged release\", initially publishing smaller versions of the model before releasing the full 1.5-billion-parameter model in November.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0013", "text": "On February 10, 2020, Microsoft introduced its Turing Natural Language Generation, which it claimed was the \"largest language model ever published at 17 billion parameters.\" The model outperformed all previous language models at a variety of tasks, including summarizing texts and answering questions.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0014", "text": "On May 28, 2020, OpenAI introduced GPT-3, a model with 175 billion parameters that was trained on a larger dataset compared to GPT-2. It marked a significant advancement in few-shot and zero-shot learning abilities. With few examples, it could perform various tasks that it was not explicitly trained for.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0015", "text": "Following the release of GPT-3, OpenAI started using reinforcement learning from human feedback (RLHF) to align models' behavior more closely with human preferences. This led to the development of InstructGPT, a fine-tuned version of GPT-3. OpenAI further refined InstructGPT to create ChatGPT, the flagship chatbot product of OpenAI that was launched on November 30, 2022", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0016", "text": ". ChatGPT was initially based on GPT-3.5, but it was later transitioned to the GPT-4 model, which was released on March 14, 2023. GPT-4 was also integrated into parts of several applications, including Microsoft Copilot, GitHub Copilot, Snapchat, Khan Academy, and Duolingo.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0017", "text": "The immense popularity of ChatGPT spurred widespread development of competing GPT-based systems from other organizations. EleutherAI released a series of open-weight models, including GPT-J in 2021. Other major technology companies later developed their own GPT models, such as Google's PaLM and Gemini as well as Meta AI's Llama.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0018", "text": "Many subsequent GPT models have been trained to be multimodal (able to process or to generate multiple types of data). For example, GPT-4o can both process and generate text, images, and audio. Additionally, GPT models like o3 and DeepSeek R1 have been trained with reinforcement learning to generate multi-step chain-of-thought reasoning before producing a final answer, which helps to solve complex problems in domains such as mathematics.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0019", "text": "On August 7, 2025, OpenAI released GPT-5, which includes a router that automatically selects whether to use a faster model or slower reasoning model based on task.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0020", "text": "Foundation models\nA foundation model is an AI model trained on broad data at scale such that it can be adapted to a wide range of downstream tasks.\nThe most recent OpenAI's GPT-n series model is GPT-5.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0021", "text": "Other such models include Google's PaLM, a broad foundation model that has been compared to GPT-3 and has been made available to developers via an API, and Together's GPT-JT, which has been reported as the closest-performing open-source alternative to GPT-3 (and is derived from earlier open-source GPTs). Meta AI (formerly Facebook) also has a generative transformer-based foundational large language model, known as LLaMA.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0022", "text": "Foundational GPTs can also employ modalities other than text, for input and/or output. GPT-4 is a multi-modal LLM that is capable of processing text and image input (though its output is limited to text). Regarding multimodal output, some generative transformer-based models are used for text-to-image technologies such as diffusion and parallel decoding. Such kinds of models can serve as visual foundation models (VFMs) for developing downstream systems that can work with images.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0023", "text": "Efficient transformer architectures\nThe computational and memory requirements of transformer-based models increase significantly as they scale to larger sizes and longer input sequences. The standard self-attention mechanism has a quadratic complexity with respect to input sequence length, as described in Attention Is All You Need.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0024", "text": "Researchers proposed a number of efficiency improvements like sparse attention mechanisms and memory-efficient architectures that reduce computational costs while supporting longer context windows.\nModels like BigBird, Reformer, and FlashAttention demonstrate structured attention patterns or optimized computation to improve scalability and efficiency.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0025", "text": "This has helped large language models to efficiently process long input sequences with reduced memory and computation during both training and inference.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0026", "text": "Task-specific models\nA foundational GPT model can be further adapted to produce more targeted systems directed to specific tasks and/or subject-matter domains. Methods for such adaptation can include additional fine-tuning (beyond that done for the foundation model) as well as certain forms of prompt engineering.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0027", "text": "An important example of this is fine-tuning models to follow instructions, which is of course a fairly broad task but more targeted than a foundation model. In January 2022, OpenAI introduced \"InstructGPT\" \u2013 a series of models which were fine-tuned to follow instructions using a combination of supervised training and reinforcement learning from human feedback (RLHF) on base GPT-3 language models", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0028", "text": ". Advantages this had over the bare foundational models included higher accuracy, less negative/toxic sentiment, and generally better alignment with user needs. Hence, OpenAI began using this as the basis for its API service offerings. Other instruction-tuned models have been released by others, including a fully open version.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0029", "text": "Another (related) kind of task-specific models are chatbots, which engage in human-like conversation. In November 2022, OpenAI launched ChatGPT \u2013 an online chat interface powered by an instruction-tuned language model trained in a similar fashion to InstructGPT", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0030", "text": ". They trained this model using RLHF, with human AI trainers providing conversations in which they played both the user and the AI, and mixed this new dialogue dataset with the InstructGPT dataset for a conversational format suitable for a chatbot", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0031", "text": ". Other major chatbots currently include Microsoft's Bing Chat, which uses OpenAI's GPT-4 (as part of a broader close collaboration between OpenAI and Microsoft), and Google's competing chatbot Gemini (initially based on their LaMDA family of conversation-trained language models, with plans to switch to PaLM).", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0032", "text": "Yet another kind of task that a GPT can be used for is the meta-task of generating its own instructions, like developing a series of prompts for 'itself' to be able to effectuate a more general goal given by a human user", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0033", "text": ". This is known as an AI agent, and more specifically a recursive one because it uses results from its previous self-instructions to help it form its subsequent prompts; the first major example of this was Auto-GPT (which uses OpenAI's GPT models), and others have since been developed as well.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0034", "text": "Chain-of-thought reasoning is a prompting technique in which a language model generates intermediate reasoning steps before arriving at a final answer. This approach has been shown to improve performance on tasks requiring multi-step reasoning, such as mathematical problem solving and logical inference.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0035", "text": "By producing step-by-step explanations, the model is able to decompose complex problems into smaller, more manageable parts. This technique is often used in conjunction with few-shot prompting, where examples demonstrating the reasoning process are included in the input.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0036", "text": "Domain-specificity\nGPT systems can be directed toward particular fields or domains. Some reported examples of such models and apps are as follows:", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0037", "text": "EinsteinGPT \u2013 for sales and marketing domains, to aid with customer relationship management (uses GPT-3.5)\nBloombergGPT \u2013 for the financial domain, to aid with financial news and information (uses \"freely available\" AI methods, combined with their proprietary data)\nKhanmigo \u2013 described as a GPT version for tutoring, in the education domain, it aids students using Khan Academy by guiding them through their studies without directly providing answers (powered by GPT-4)", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0038", "text": "SlackGPT \u2013 for the Slack instant-messaging service, to aid with navigating and summarizing discussions on it (uses OpenAI's API)\nBioGPT \u2013 for the biomedical domain, to aid with biomedical literature text generation and mining (uses GPT-2)", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0039", "text": "Sometimes domain-specificity is accomplished via software plug-ins or add-ons. For example, several different companies have developed particular plugins that interact directly with OpenAI's ChatGPT interface, and Google Workspace has available add-ons such as \"GPT for Sheets and Docs\" \u2013 which is reported to aid use of spreadsheet functionality in Google Sheets.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0040", "text": "Scaling laws\nScaling laws describe empirical relationships between the performance of large language models and factors such as model size, dataset size, and computational resources. Empirical work has found that performance tends to follow approximate power-law relationships as these factors increase.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0041", "text": "Larger models trained on more data generally achieve lower training loss and better generalization. Later work suggests that performance is not determined by parameter count alone, but by how model size, data, and compute are balanced during training.\nThese observations have influenced the development of successive GPT models, particularly in decisions about architecture design, dataset composition, and training strategies.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0042", "text": "Emergent abilities\nEmergent abilities refer to capabilities that appear in large language models only when they reach a certain scale and are not present in smaller versions of the same models. These abilities are considered \"emergent\" because they arise as model size, training data, and compute increase.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0043", "text": "Examples of emergent abilities include multi-step reasoning, in-context learning (the ability to perform tasks based on examples provided in prompts without additional training), and improved performance on complex language and reasoning benchmarks.\nResearch suggests that these capabilities do not scale linearly, but instead appear once models exceed certain thresholds in size and training scale.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0044", "text": "This phenomenon has influenced the development of larger GPT models and contributed to their increased effectiveness across a wide range of tasks.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0045", "text": "Brand issues", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0046", "text": "OpenAI, which created the first generative pre-trained transformer (GPT) in 2018, asserted in 2023 that \"GPT\" should be regarded as a brand of OpenAI. In April 2023, OpenAI revised the brand guidelines in its terms of service to indicate that other businesses using its API to run their AI services would no longer be able to include \"GPT\" in such names or branding", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0047", "text": ". In May 2023, OpenAI engaged a brand management service to notify its API customers of this policy, although these notifications stopped short of making overt legal claims (such as allegations of trademark infringement or demands to cease and desist). As of November 2023, OpenAI still prohibits its API licensees from naming their own products with \"GPT\", but it has begun enabling its ChatGPT Plus subscribers to make \"custom versions of ChatGPT\" called GPTs on the OpenAI site", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0048", "text": ". OpenAI's terms of service says that its subscribers may use \"GPT\" in the names of these, although it's \"discouraged\".", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0049", "text": "Relatedly, OpenAI has applied to the United States Patent and Trademark Office (USPTO) to seek domestic trademark registration for the term \"GPT\" in the field of AI. OpenAI sought to expedite handling of its application, but the USPTO declined that request in April 2023. In May 2023, the USPTO responded to the application with a determination that \"GPT\" was both descriptive and generic. As of November 2023, OpenAI continues to pursue its argument through the available processes", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0050", "text": ". Regardless, failure to obtain a registered U.S. trademark does not preclude some level of common-law trademark rights in the U.S. and trademark rights in other countries.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0051", "text": "For any given type or scope of trademark protection in the U.S., OpenAI would need to establish that the term is actually \"distinctive\" to their specific offerings in addition to being a broader technical term for the kind of technology", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0052", "text": ". Some media reports suggested in 2023 that OpenAI may be able to obtain trademark registration based indirectly on the fame of its GPT-based chatbot product, ChatGPT, for which OpenAI has separately sought protection (and which it has sought to enforce more strongly). Other reports have indicated that registration for the bare term \"GPT\" seems unlikely to be granted, as it is used frequently as a common term to refer simply to AI systems that involve generative pre-trained transformers", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0053", "text": ". In any event, to whatever extent exclusive rights in the term may occur the U.S., others would need to avoid using it for similar products or services in ways likely to cause confusion. If such rights ever became broad enough to implicate other well-established uses in the field, the trademark doctrine of descriptive fair use could still continue non-brand-related usage.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0054", "text": "In the European Union, the European Union Intellectual Property Office registered \"GPT\" as a trade mark of OpenAI in spring 2023. However, since spring 2024 the registration is being challenged and is pending cancellation.\nIn Switzerland, the Swiss Federal Institute of Intellectual Property registered \"GPT\" as a trade mark of OpenAI in spring 2023.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0055", "text": "Evaluation and benchmarking\nThe evaluation of generative pre-trained transformer models is carried out using a range of benchmarks and metrics, which aim to assess their performance across different tasks. Common approaches include accuracy on standard datasets, as well as other features such as robustness, bias, and toxicity.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0056", "text": "These models are typically tested on tasks such as natural language understanding, reasoning, answering queries, and code generation. Sometimes they combine multiple tasks to provide a broader assessment of model performance across domains.\nMore recent approaches extend these evaluations to include other features such as fairness, efficiency, and transparency, in order to get a more accurate assessment of these models.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0057", "text": "Evaluation remains an active area of research, as existing tests may not accurately reflect real-world performance or the risks associated with large-scale generative models.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0058", "text": "Ethical considerations and societal impact\nGenerative pre-trained transformer models have raised a range of ethical and societal concerns, particularly regarding bias, misinformation, and environmental impact. Large language models can reproduce and amplify patterns present in their training data, including social biases, which may lead to unfair or misleading outputs.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0059", "text": "These models have also been associated with generating inaccurate or misleading information, as they are designed to produce fluent text rather than to verify factual accuracy. This has an impact on their use in applications such as automated content generation and information dissemination.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0060", "text": "Training large-scale models also requires huge computational resources, contributing to increased energy consumption and environmental costs. Concerns about the environmental impact of large AI systems have led to calls for more efficient training methods and more transparency in reporting resource usage.", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Generative_pretrained_transformer_0061", "text": "See also\nCyc\nVision transformer\n\n\n== References ==", "title": "Generative pre-trained transformer", "url": "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer"}, {"id": "rec_Large_language_model_0000", "text": "A large language model (LLM) is a computational model designed to perform natural language processing tasks, especially language generation, using contextual relationships derived from a large set of training data. LLMs can generate, summarize, translate and parse text in a variety of contexts, and are the technological underpinning of modern chatbots. LLMs can accurately mimic natural language patterns because they are trained on collections of human-written text", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0001", "text": ". For the same reason, biased or inaccurate training data can make a LLM's output less reliable.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0002", "text": "As of 2024, the largest and most capable LLMs are all based on transformer architectures, which can be more efficient and parallelizable than earlier statistical and recurrent neural network models. Research into other architectures, such as state space models, is ongoing. \nModels like GPT, BERT, and their successors used these advances to demonstrate emergent behaviors at scale, such as finding specific data from a large data set and compositional reasoning.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0003", "text": "Benchmark evaluations for LLMs test a model's ability to perform one or more language tasks. Modern LLMs may face comprehensive, multi-task evaluations measuring reasoning, factual accuracy, alignment, and safety. Optimizing training sessions to pass benchmarks may result in a model that adheres too closely to benchmark outputs without genuine generalization or robust capability improvements.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0004", "text": "History", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0005", "text": "Before the emergence of transformer-based models in 2017, some language models were considered large relative to the computational and data constraints of their time. In the early 1990s, IBM's statistical models pioneered word alignment techniques for machine translation, laying the groundwork for corpus-based language modeling. In 2001, a smoothed n-gram model, such as those employing Kneser\u2013Ney smoothing, trained on 300 million words, achieved state-of-the-art perplexity on benchmark tests", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0006", "text": ". During the 2000s, with the rise of widespread internet access, researchers began compiling massive text datasets from the web (\"web as corpus\") to train statistical language models.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0007", "text": "Moving beyond n-gram models, researchers started in 2000 to use neural networks to learn language models. Following the breakthrough of deep neural networks in image classification around 2012, similar architectures were adapted for language tasks. This shift was marked by the development of word embeddings (e.g., Word2Vec by Mikolov in 2013) and sequence-to-sequence (seq2seq) models using LSTM", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0008", "text": ". In 2016, Google transitioned its translation service to neural machine translation (NMT), replacing statistical phrase-based models with deep recurrent neural networks. These early NMT systems used LSTM-based encoder-decoder architectures, as they preceded the invention of transformers.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0009", "text": "At the 2017 NeurIPS conference, Google researchers introduced the transformer architecture in their landmark paper \"Attention Is All You Need\". This paper's goal was to improve upon 2014 seq2seq technology, and was based mainly on the attention mechanism developed by Bahdanau et al. in 2014. The following year in 2018, BERT was introduced and quickly became \"ubiquitous\". Though the original transformer has both encoder and decoder blocks, BERT is an encoder-only model", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0010", "text": ". Academic and research usage of BERT began to decline in 2023, following rapid improvements in the abilities of decoder-only models (such as GPT) to solve tasks via prompting.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0011", "text": "Although decoder-only GPT-1 was introduced in 2018, it was GPT-2 in 2019 that caught widespread attention because OpenAI claimed to have initially deemed it too powerful to release publicly, out of fear of malicious use. GPT-3 in 2020 went a step further and as of 2025 is available only via API with no offering of downloading the model to execute locally. But it was the 2022 consumer-facing chatbot ChatGPT that received extensive media coverage and public attention", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0012", "text": ". The 2023 GPT-4 was praised for its increased accuracy and as a \"holy grail\" for its multimodal capabilities. OpenAI did not reveal the high-level architecture and the number of parameters of GPT-4. The release of ChatGPT led to an uptick in LLM usage across several research subfields of computer science, including robotics, software engineering, and societal impact work", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0013", "text": ". In 2024 OpenAI released the reasoning model OpenAI o1, which generates long chains of thought before returning a final answer. Many LLMs with parameter counts comparable to those of OpenAI's GPT series have been developed.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0014", "text": "Since 2022, open-weight models have been gaining popularity, especially at first with BLOOM and LLaMA, though both have restrictions on usage and deployment. Mistral AI's models Mistral 7B and Mixtral 8x7b have a more permissive Apache License. In January 2025, DeepSeek released DeepSeek R1, a 671-billion-parameter open-weight model that performs comparably to OpenAI o1 but at a much lower price per token for users.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0015", "text": "Since 2023, many LLMs have been trained to be multimodal, having the ability to also process or generate other types of data, such as images, audio, or 3D meshes. These LLMs are also called large multimodal models (LMMs), or multimodal large language models (MLLMs).", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0016", "text": "Open-weight LLMs have increasingly shaped the field since 2023, contributing to broader participation in AI development and greater transparency in model evaluation. Vake et al. (2025) demonstrated that community-driven contributions to open-weight models measurably improve their efficiency and performance, with user participation growing rapidly on collaborative platforms such as Hugging Face. Paris et al", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0017", "text": ". Paris et al. (2025) further argued that openness in AI should extend beyond releasing model code or weights to encompass inclusiveness, accountability, and ethical responsibility in AI research and deployment. Collectively, these studies highlight that open-weight LLMs can accelerate innovation and enhance scientific reproducibility, while fostering a more transparent and participatory AI ecosystem.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0018", "text": "Dataset preprocessing\nTokenization", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0019", "text": "As machine learning algorithms process numbers rather than text, the text must be converted to numbers. In the first step, a vocabulary is decided upon, then integer indices are arbitrarily but uniquely assigned to each vocabulary entry, and finally, an embedding is associated to the integer index. Algorithms include byte-pair encoding (BPE) and WordPiece", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0020", "text": ". There are also special tokens serving as control characters, such as [MASK] for masked-out token (as used in BERT), and [UNK] (\"unknown\") for characters not appearing in the vocabulary. Also, some special symbols are used to denote special text formatting. For example, \"\u0120\" denotes a preceding whitespace in RoBERTa and GPT and \"##\" denotes continuation of a preceding word in BERT.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0021", "text": "For example, the BPE tokenizer used by the legacy version of GPT-3 would split tokenizer: texts -> series of numerical \"tokens\" as", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0022", "text": "Tokenization also compresses the datasets. Because LLMs generally require input to be an array that is not jagged, the shorter texts must be \"padded\" until they match the length of the longest one. The average number of words per token depends on the language.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0023", "text": "Byte-pair encoding", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0024", "text": "As an example, consider a tokenizer based on byte-pair encoding. In the first step, all unique characters (including blanks and punctuation marks) are treated as an initial set of n-grams (i.e. initial set of uni-grams). Successively the most frequent pair of adjacent characters is merged into a bi-gram and all instances of the pair are replaced by it", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0025", "text": ". All occurrences of adjacent pairs of (previously merged) n-grams that most frequently occur together are then again merged into even lengthier n-gram, until a vocabulary of prescribed size is obtained. After a tokenizer is trained, any text can be tokenized by it, as long as it does not contain characters not appearing in the initial-set of uni-grams.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0026", "text": "Problems", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0027", "text": "A token vocabulary based on the frequencies extracted from mainly English corpora uses as few tokens as possible for an average English word. However, an average word in another language encoded by such an English-optimized tokenizer is split into a suboptimal amount of tokens. GPT-2 tokenizer can use up to 15 times more tokens per word for some languages, for example for the Shan language from Myanmar", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0028", "text": ". Even more widespread languages such as Portuguese and German have \"a premium of 50%\" compared to English.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0029", "text": "Dataset cleaning\nIn the context of training LLMs, datasets are typically cleaned by removing low-quality, duplicated, or toxic data. Cleaned datasets can increase training efficiency and lead to improved downstream performance. A trained LLM can be used to clean datasets for training a further LLM.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0030", "text": "With the increasing proportion of LLM-generated content on the web, data cleaning in the future may include filtering out such content. LLM-generated content can pose a problem if the content is similar to human text (making filtering difficult) but of lower quality (degrading performance of models trained on it).", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0031", "text": "Synthetic data\nTraining of largest language models might need more linguistic data than naturally available, or that the naturally occurring data is of insufficient quality. In these cases, synthetic data might be used. Microsoft's Phi series of LLMs is trained on textbook-like data generated by another LLM.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0032", "text": "Training\nAn LLM is a type of foundation model (large X model) trained on language. LLMs can be trained in different ways. In particular, GPT models are first pretrained to predict the next word on a large amount of data, before being fine-tuned.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0033", "text": "Cost", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0034", "text": "Substantial infrastructure is necessary for training the largest models. The tendency towards larger models is visible in the list of large language models. For example, the training of GPT-2 (i.e. a 1.5-billion-parameter model) in 2019 cost $50,000, while training of the PaLM (i.e. a 540-billion-parameter model) in 2022 cost $8 million, and Megatron-Turing NLG 530B (in 2021) cost around $11 million", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0035", "text": ". The qualifier \"large\" in \"large language model\" is inherently vague, as there is no definitive threshold for the number of parameters required to qualify as \"large\".", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0036", "text": "Fine-tuning\nBefore being fine-tuned, most LLMs are next-token predictors. The fine-tuning shapes the LLM's behavior via techniques like reinforcement learning from human feedback (RLHF) or constitutional AI.\nInstruction fine-tuning is a form of supervised learning used to teach LLMs to follow user instructions. In 2022, OpenAI demonstrated InstructGPT, a version of GPT-3 similarly fine-tuned to follow instructions.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0037", "text": "Reinforcement learning from human feedback (RLHF) involves training a reward model to predict which text humans prefer. Then, the LLM can be fine-tuned through reinforcement learning to better satisfy this reward model. Since humans typically prefer truthful, helpful and harmless answers, RLHF favors such answers.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0038", "text": "Architecture\nLLMs are generally based on the transformer architecture, which leverages an attention mechanism that enables the model to process relationships between all elements in a sequence simultaneously, regardless of their distance from each other.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0039", "text": "Attention mechanism and context window", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0040", "text": "In order to find out which tokens are relevant to each other within the scope of the context window, the attention mechanism calculates \"soft\" weights for each token, more precisely for its embedding, by using multiple attention heads, each with its own \"relevance\" for calculating its own soft weights. For example, the small (i.e. 117M parameter sized) GPT-2 model has had twelve attention heads and a context window of only 1k tokens", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0041", "text": ". In its medium version it has 345M parameters and contains 24 layers, each with 12 attention heads. For the training with gradient descent a batch size of 512 was utilized.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0042", "text": "Autoregressive models, such as GPTs, are trained to guess how a sequence continues; for example, whether the word sequence \"I like to eat\" is more likely to be followed by the word \"bread\" or the word \"rocks.\" Masked models, such as BERT, are trained to guess parts that are missing from a sequence, such as whether the missing word in \"I like to ___ roses\" is more likely to be the word \"smell\" or the word \"eat.\" The model's predictions are based on the properties of sequences within its training", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0043", "text": "the properties of sequences within its training dataset.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0044", "text": "Mixture of experts\nA mixture of experts (MoE) is a machine learning architecture in which multiple specialized neural networks (\"experts\") work together, with a gating mechanism that routes each input to the most appropriate expert(s). Mixtures of experts can reduce inference costs, as only a fraction of the parameters are used for each input. The approach was introduced in 2017 by Google researchers.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0045", "text": "Parameter size\nTypically, LLMs are trained with single- or half-precision floating point numbers (float32 and float16). One float16 has 16 bits, or 2 bytes, and so one billion parameters require 2 gigabytes. The largest models typically have more than 100 billion parameters, which places them outside the range of most consumer electronics.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0046", "text": "Quantization", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0047", "text": "Post-training quantization aims to decrease the space requirement by lowering precision of the parameters of a trained model, while preserving most of its performance. Quantization can be further classified as static quantization if the quantization parameters are determined beforehand (typically during a calibration phase), and dynamic quantization if the quantization is applied during inference", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0048", "text": ". The simplest form of quantization simply truncates all the parameters to a given number of bits: this is applicable to static as well as dynamic quantization, but loses much precision. Dynamic quantization allows for the use of a different quantization codebook per layer, either a lookup table of values or a linear mapping (scaling factor and bias), at the cost of foregoing the possible speed improvements from using lower-precision arithmetic.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0049", "text": "Quantized models are typically seen as frozen with modification of weights (e.g. fine-tuning) only applied to the original model. It is possible to fine-tune quantized models using low-rank adaptation.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0050", "text": "Extensibility\nBeyond basic text generation, various techniques have been developed to extend LLM capabilities, including the use of external tools and data sources, improved reasoning on complex problems, and enhanced instruction-following or autonomy through prompting methods.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0051", "text": "Prompt engineering", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0052", "text": "In 2020, OpenAI researchers demonstrated that their new model GPT-3 could understand what format to use given a few rounds of Q and A (or other type of task) in the input data as example, thanks in part due to the RLHF technique. This technique, called few-shot prompting, allows LLMs to be adapted to any task without requiring fine-tuning. Also in 2022, it was found that the base GPT-3 model can generate an instruction based on user input", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0053", "text": ". The generated instruction along with user input is then used as input to another instance of the model under a \"Instruction: [...], Input: [...], Output:\" format. The other instance is able to complete the output and often produces the correct answer in doing so. The ability to \"self-instruct\" makes LLMs able to bootstrap themselves toward a correct answer.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0054", "text": "Dialogue processing (chatbot)", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0055", "text": "An LLM can be turned into a chatbot by specializing it for conversation. User input is prefixed with a marker such as \"Q:\" or \"User:\" and the LLM is asked to predict the output after a fixed \"A:\" or \"Assistant:\". This type of model became commercially available in 2022 with ChatGPT, a sibling model of InstructGPT fine-tuned to accept and produce dialog-formatted text based on GPT-3.5. It could similarly follow user instructions", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0056", "text": ". It could similarly follow user instructions. Before the stream of User and Assistant lines, a chat context usually starts with a few lines of overarching instructions, from a role called \"developer\" or \"system\" to convey a higher authority than the user's input. This is called a \"system prompt\".", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0057", "text": "Retrieval-augmented generation", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0058", "text": "Retrieval-augmented generation (RAG) is an approach that integrates LLMs with document retrieval systems. Given a query, a document retriever is called to retrieve the most relevant documents. This is usually done by encoding the query and the documents into vectors, then finding the documents with vectors (usually stored in a vector database) most similar to the vector of the query. The LLM then generates an output based on both the query and context included from the retrieved documents.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0059", "text": "Tool use\nTool use is a mechanism that enables LLMs to interact with external systems, applications, or data sources. It can allow for example to fetch real-time information from an API or to execute code. A program separate from the LLM watches the output stream of the LLM for a special tool-calling syntax. When these special tokens appear, the program calls the tool accordingly and feeds its output back into the LLM's input stream.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0060", "text": "Early tool-using LLMs were fine-tuned on the use of specific tools. But fine-tuning LLMs for the ability to read API documentation and call API correctly has greatly expanded the range of tools accessible to an LLM. Describing available tools in the system prompt can also make an LLM able to use tools. A system prompt instructing ChatGPT (GPT-4) to use multiple types of tools can be found online.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0061", "text": "Agency", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0062", "text": "An LLM is typically not an autonomous agent by itself, as it lacks the ability to interact with dynamic environments, recall past behaviors, and plan future actions. But it can be transformed into an agent by adding supporting elements: the role (profile) and the surrounding environment of an agent can be additional inputs to the LLM, while memory can be integrated as a tool or provided as additional input", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0063", "text": ". Instructions and input patterns are used to make the LLM plan actions and tool use is used to potentially carry out these actions.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0064", "text": "The ReAct pattern, a portmanteau of reason and act, constructs an agent out of an LLM, using the LLM as a planner. The LLM is prompted to \"think out loud\". Specifically, the language model is prompted with a textual description of the environment, a goal, a list of possible actions, and a record of the actions and observations so far. It generates one or more thoughts before generating an action, which is then executed in the environment.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0065", "text": "In the DEPS (\"describe, explain, plan and select\") method, an LLM is first connected to the visual world via image descriptions. It is then prompted to produce plans for complex tasks and behaviors based on its pretrained knowledge and the environmental feedback it receives.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0066", "text": "The Reflexion method constructs an agent that learns over multiple episodes. At the end of each episode, the LLM is given the record of the episode, and prompted to think up \"lessons learned\", which would help it perform better at a subsequent episode. These \"lessons learned\" are stored as a form of long-term memory and given to the agent in the subsequent episodes.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0067", "text": "Monte Carlo tree search can use an LLM as rollout heuristic. When a programmatic world model is not available, an LLM can also be prompted with a description of the environment to act as world model.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0068", "text": "For open-ended exploration, an LLM can be used to score observations for their \"interestingness\", which can be used as a reward signal to guide a normal (non-LLM) reinforcement learning agent. Alternatively, it can propose increasingly difficult tasks for curriculum learning. Instead of outputting individual actions, an LLM planner can also construct \"skills\", or functions for complex action sequences", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0069", "text": ". The skills can be stored and later invoked, allowing increasing levels of abstraction in planning.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0070", "text": "Multiple agents with memory can interact socially.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0071", "text": "Reasoning", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0072", "text": "LLMs are conventionally trained to generate an output without generating intermediate steps. As a result, their performance tends to be subpar on complex questions requiring (at least in humans) intermediate steps of thought. Early research demonstrated that inserting intermediate \"scratchpad\" computations could improve performance on such tasks. Later methods overcame this deficiency more systematically by breaking tasks into smaller steps for the LLM, either manually or automatically.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0073", "text": "Chaining\nPrompt chaining was introduced in 2022. In this method, a user manually breaks a complex problem down into several steps. In each step, the LLM receives as input a prompt telling it what to do and some results from preceding steps. The result from one step is then reused in a next step, until a final answer is reached. The ability of an LLM to follow instructions means that even non-experts can write a successful collection of stepwise prompts given a few rounds of trial and error.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0074", "text": "A 2022 paper demonstrated a separate technique called chain-of-thought prompting, which makes the LLM break the question down autonomously. An LLM is given some examples where the \"assistant\" verbally breaks down the thought process before arriving at an answer. The LLM mimics these examples and also tries to spend some time generating intermediate steps before providing the final answer", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0075", "text": ". This additional step elicited by prompting improves the correctness of the LLM on relatively complex questions. On math word questions, a prompted model can exceed even fine-tuned GPT-3 with a verifier. Chain-of-thought can also be elicited by simply adding an instruction like \"Let's think step by step\" to the prompt, in order to encourage the LLM to proceed methodically instead of trying to directly guess the answer.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0076", "text": "Model-native reasoning\nIn late 2024, a new approach to LLM development emerged with \"reasoning models\". These are trained to generate step-by-step analysis before producing final answers, enabling better results on complex tasks, for instance in mathematics, coding and logic. OpenAI introduced this concept with their o1 model in September 2024, followed by o3 in April 2025. On the International Mathematics Olympiad qualifying exam problems, GPT-4o achieved 13% accuracy while o1 reached 83%.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0077", "text": "In January 2025, the Chinese company DeepSeek released DeepSeek-R1, a 671-billion-parameter open-weight reasoning model that achieved comparable performance to OpenAI's o1 while being significantly more cost-effective to operate. Unlike proprietary models from OpenAI, DeepSeek-R1's open-weight nature allowed researchers to study and build upon the algorithm, though its training data remained private.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0078", "text": "These reasoning models typically require more computational resources per query compared to traditional LLMs, as they perform more extensive processing to work through problems step by step.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0079", "text": "Inference optimization\nInference optimization refers to techniques that improve LLM performance by applying additional computational resources during the inference process, rather than requiring model retraining. These approaches implement various state-of-the-art reasoning and decision-making strategies to enhance accuracy and capabilities.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0080", "text": "OptiLLM is an OpenAI API-compatible optimizing inference proxy that implements multiple inference optimization techniques simultaneously. The system acts as a transparent proxy that can work with any LLM provider, implementing techniques such as Monte Carlo tree search (MCTS), mixture of agents (MOA), best-of-N sampling, and chain-of-thought reflection", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0081", "text": ". OptiLLM demonstrates that strategic application of computational resources at inference time can substantially improve model performance across diverse tasks, achieving significant improvements on benchmarks such as the AIME 2024 mathematics competition and various coding challenges.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0082", "text": "These inference optimization approaches represent a growing category of tools that enhance existing LLMs without requiring access to model weights or retraining, making advanced reasoning capabilities more accessible across different model providers and use cases.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0083", "text": "Forms of input and output\nMultimodality", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0084", "text": "Multimodality means having multiple modalities, where a \"modality\" refers to a type of input or output, such as video, image, audio, text, proprioception, etc. For example, Google PaLM model was fine-tuned into a multimodal model and applied to robotic control. LLaMA models have also been turned multimodal using the tokenization method, to allow image inputs, and video inputs. GPT-4o can process and generate text, audio and images", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0085", "text": ". Such models are sometimes called large multimodal models (LMMs).", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0086", "text": "A common method to create multimodal models out of an LLM is to \"tokenize\" the output of a trained encoder. Concretely, one can construct an LLM that can understand images as follows: take a trained LLM, and take a trained image encoder \n  \n    \n      \n        E\n      \n    \n    {\\displaystyle E}\n  \n. Make a small multilayer perceptron \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n, so that for any image \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0087", "text": "y\n      \n    \n    {\\displaystyle y}\n  \n, the post-processed vector \n  \n    \n      \n        f\n        (\n        E\n        (\n        y\n        )\n        )\n      \n    \n    {\\displaystyle f(E(y))}", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0088", "text": "has the same dimensions as an encoded token. That is an \"image token\". Then, one can interleave text tokens and image tokens. The compound model is then fine-tuned on an image-text dataset. This basic construction can be applied with more sophistication to improve the model. The image encoder may be frozen to improve stability. This type of method, where embeddings from multiple modalities are fused and the predictor is trained on the combined embeddings, is called early fusion.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0089", "text": "Another method, called intermediate fusion, involves each modality being first processed independently to obtain modality-specific representations; then these intermediate representations are fused together. In general, cross-attention is used for integrating information from different modalities. As an example, the Flamingo model uses cross-attention layers to inject visual information into its pre-trained language model.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0090", "text": "Non-natural languages", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0091", "text": "LLMs can handle programming languages similarly to how they handle natural languages. No special change in token handling is needed as code, like human language, is represented as plain text. LLMs can generate code based on problems or instructions written in natural language. They can also describe code in natural language or translate it into other programming languages. They were originally used as a code completion tool, but advances have moved them towards automatic programming", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0092", "text": ". Services such as GitHub Copilot offer LLMs specifically trained, fine-tuned, or prompted for programming.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0093", "text": "In computational biology, transformer-base architectures, such as DNA LLMs, have also proven useful in analyzing biological sequences: protein, DNA, and RNA. With proteins they appear able to capture a degree of \"grammar\" from the amino-acid sequence, by mapping that sequence into an embedding", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0094", "text": ". On tasks such as structure prediction and mutational outcome prediction, a small model using an embedding as input can approach or exceed much larger models using multiple sequence alignments (MSA) as input. ESMFold, Meta Platforms' embedding-based method for protein structure prediction, runs an order of magnitude faster than AlphaFold2 thanks to the removal of an MSA requirement and a lower parameter count due to the use of embeddings", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0095", "text": ". Meta hosts ESM Atlas, a database of 772 million structures of metagenomic proteins predicted using ESMFold. An LLM can also design proteins unlike any seen in nature. Nucleic acid models have proven useful in detecting regulatory sequences, sequence classification, RNA-RNA interaction prediction, and RNA structure prediction.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0096", "text": "Properties\nScaling laws\nThe performance of an LLM after pretraining largely depends on the:\n\n  \n    \n      \n        C\n      \n    \n    {\\displaystyle C}\n  \n: cost of pretraining (the total amount of compute used),\n\n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}\n  \n: size of the artificial neural network itself, such as number of parameters (i.e. amount of neurons in its layers, amount of weights between them and biases),", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0097", "text": "D\n      \n    \n    {\\displaystyle D}\n  \n: size of its pretraining dataset (i.e. number of tokens in corpus).\nScaling laws are empirical statistical laws that predict LLM performance based on such factors. One particular scaling law (\"Chinchilla scaling\") for LLM autoregressively trained for one epoch, with a log-log learning rate schedule, states that:", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0098", "text": "{\n            \n              \n                \n                  C\n                  =\n                  \n                    C\n                    \n                      0\n                    \n                  \n                  N\n                  D\n                \n              \n              \n                \n                  L\n                  =\n                  \n                    \n                      A", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0099", "text": "A\n                      \n                        N\n                        \n                          \u03b1\n                        \n                      \n                    \n                  \n                  +\n                  \n                    \n                      B\n                      \n                        D\n                        \n                          \u03b2", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0100", "text": "+\n                  \n                    L\n                    \n                      0\n                    \n                  \n                \n              \n            \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{cases}C=C_{0}ND\\\\[6pt]L={\\frac {A}{N^{\\alpha }}}+{\\frac {B}{D^{\\beta }}}+L_{0}\\end{cases}}}\n  \n where the variables are", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0101", "text": "C\n      \n    \n    {\\displaystyle C}\n  \n is the cost of training the model, in FLOPs.\n\n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}\n  \n is the number of parameters in the model.\n\n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n is the number of tokens in the training set.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0102", "text": "L\n      \n    \n    {\\displaystyle L}\n  \n is the average negative log-likelihood loss per token (nats/token), achieved by the trained LLM on the test dataset.\nand the statistical hyper-parameters are", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0103", "text": "C\n          \n            0\n          \n        \n        =\n        6\n      \n    \n    {\\displaystyle C_{0}=6}\n  \n, meaning that it costs 6 FLOPs per parameter to train on one token. Note that training cost is much higher than inference cost, where it costs 1 to 2 FLOPs per parameter to infer on one token.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0104", "text": "\u03b1\n        =\n        0.34\n        ,\n        \u03b2\n        =\n        0.28\n        ,\n        A\n        =\n        406.4\n        ,\n        B\n        =\n        410.7\n        ,\n        \n          L\n          \n            0\n          \n        \n        =\n        1.69\n      \n    \n    {\\displaystyle \\alpha =0.34,\\beta =0.28,A=406.4,B=410.7,L_{0}=1.69}", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0105", "text": "Emergent abilities\nPerformance of bigger models on various tasks, when plotted on a log-log scale, appears as a linear extrapolation of performance achieved by smaller models. However, this linearity may be punctuated by \"break(s)\" in the scaling law, where the slope of the line changes abruptly, and where larger models acquire \"emergent abilities\". They arise from the complex interaction of the model's components and are not explicitly programmed or designed.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0106", "text": "One of the emergent abilities is in-context learning from example demonstrations. In-context learning is involved in tasks, such as:", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0107", "text": "reported arithmetics\ndecoding the International Phonetic Alphabet\nunscrambling a word's letters\ndisambiguating word-in-context datasets\nconverting spatial words\ncardinal directions (for example, replying \"northeast\" in response to a 3x3 grid of 8 zeros and a 1 in the top-right), color terms represented in text.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0108", "text": "chain-of-thought prompting: In a 2022 research paper, chain-of-thought prompting only improved the performance for models that had at least 62B parameters. Smaller models perform better when prompted to answer immediately, without chain of thought.\nidentifying offensive content in paragraphs of Hinglish (a combination of Hindi and English), and generating a similar English equivalent of Kiswahili proverbs.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0109", "text": "Schaeffer et al. argue that the emergent abilities are not unpredictably acquired, but predictably acquired according to a smooth scaling law. The authors considered a toy statistical model of an LLM solving multiple-choice questions, and showed that this statistical model, modified to account for other types of tasks, applies to these tasks as well.\nLet \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n be the number of parameter count, and \n  \n    \n      \n        y", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0110", "text": "y\n      \n    \n    {\\displaystyle y}\n  \n be the performance of the model.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0111", "text": "Interpretation\nMechanistic interpretability", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0112", "text": "Mechanistic interpretability seeks to precisely identify and understand how individual neurons or circuits within LLMs produce specific behaviors or outputs. By reverse-engineering model components at a granular level, researchers aim to detect and mitigate safety concerns such as emergent harmful behaviors, biases, deception, or unintended goal pursuit before deployment", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0113", "text": ". Mechanistic interpretability research has been conducted at organizations like Anthropic and OpenAI, although understanding the inner workings of LLMs remains difficult.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0114", "text": "The reverse-engineering may lead to the discovery of algorithms that approximate inferences performed by an LLM. For instance, the authors trained small transformers on modular arithmetic addition. The resulting models were reverse-engineered, and it turned out they used discrete Fourier transform", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0115", "text": ". The training of the model also highlighted a phenomenon called grokking, in which the model initially memorizes the training set (overfitting), and later suddenly learns to actually perform the calculation.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0116", "text": "Understanding and intelligence", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0117", "text": "NLP researchers were evenly split when asked, in a 2022 survey, whether (untuned) LLMs \"could (ever) understand natural language in some nontrivial sense\". Proponents of \"LLM understanding\" believe that some LLM abilities, such as mathematical reasoning, imply an ability to \"understand\" certain concepts", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0118", "text": ". A Microsoft team argued in 2023 that GPT-4 \"can solve novel and difficult tasks that span mathematics, coding, vision, medicine, law, psychology and more\" and that GPT-4 \"could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence system\": \"Can one reasonably say that a system that passes exams for software engineering candidates is not really intelligent?\" Ilya Sutskever argues that predicting the next word sometimes involves reasoning and deep", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0119", "text": "next word sometimes involves reasoning and deep insights, for example if the LLM has to predict the name of the criminal in an unknown detective novel after processing the entire story leading up to the revelation", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0120", "text": ". Some researchers characterize LLMs as \"alien intelligence\". For example, Conjecture CEO Connor Leahy considers untuned LLMs to be like inscrutable alien \"Shoggoths\", and believes that RLHF tuning creates a \"smiling facade\" obscuring the inner workings of the LLM: \"If you don't push it too far, the smiley face stays on. But then you give it [an unexpected] prompt, and suddenly you see this massive underbelly of insanity, of weird thought processes and clearly non-human understanding.\"", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0121", "text": "In contrast, some skeptics of LLM understanding believe that existing LLMs are \"simply remixing and recombining existing writing\", a phenomenon known as stochastic parrot, or they point to the deficits existing LLMs continue to have in prediction skills, reasoning skills, agency, and explainability. For example, GPT-4 has natural deficits in planning and in real-time learning", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0122", "text": ". Generative LLMs have been observed to confidently assert claims of fact which do not seem to be justified by their training data, a phenomenon which has been termed \"hallucination\". Specifically, hallucinations in the context of LLMs correspond to the generation of text or responses that seem syntactically sound, fluent, and natural but are factually incorrect, nonsensical, or unfaithful to the provided source input", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0123", "text": ". Neuroscientist Terrence Sejnowski has argued that \"The diverging opinions of experts on the intelligence of LLMs suggests that our old ideas based on natural intelligence are inadequate\".", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0124", "text": "Efforts to reduce or compensate for hallucinations have employed automated reasoning, retrieval-augmented generation (RAG), fine-tuning, and other methods.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0125", "text": "The matter of LLM's exhibiting intelligence or understanding has two main aspects\u2014the first is how to model thought and language in a computer system, and the second is how to enable the computer system to generate human-like language. These aspects of language as a model of cognition have been developed in the field of cognitive linguistics", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0126", "text": ". American linguist George Lakoff presented neural theory of language (NTL) as a computational basis for using language as a model of learning tasks and understanding. The NTL model outlines how specific neural structures of the human brain shape the nature of thought and language and in turn what are the computational properties of such neural systems that can be applied to model thought and language in a computer system", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0127", "text": ". After a framework for modeling language in a computer systems was established, the focus shifted to establishing frameworks for computer systems to generate language with acceptable grammar. In his 2014 book titled The Language Myth: Why Language Is Not An Instinct, British cognitive linguist and digital communication technologist Vyvyan Evans mapped out the role of probabilistic context-free grammar (PCFG) in enabling NLP to model cognitive patterns and generate human-like language.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0128", "text": "Evaluation\nPerplexity\nThe canonical measure of the performance of any language model is its perplexity on a given text corpus. Perplexity measures how well a model predicts the contents of a dataset; the higher the likelihood the model assigns to the dataset, the lower the perplexity. In mathematical terms, perplexity is the exponential of the average negative log likelihood per token.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0129", "text": "log\n        \u2061\n        (\n        \n          Perplexity\n        \n        )\n        =\n        \u2212\n        \n          \n            1\n            N\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            N\n          \n        \n        log\n        \u2061\n        (\n        Pr\n        (\n        \n          \n            token\n          \n          \n            i\n          \n        \n        \u2223", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0130", "text": "\u2223\n        \n          \n            context for token\n          \n          \n            i\n          \n        \n        )\n        )\n      \n    \n    {\\displaystyle \\log({\\text{Perplexity}})=-{\\frac {1}{N}}\\sum _{i=1}^{N}\\log(\\Pr({\\text{token}}_{i}\\mid {\\text{context for token}}_{i}))}", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0131", "text": "Here, \n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}\n  \n is the number of tokens in the text corpus, and \"context for token \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n\" depends on the specific type of LLM. If the LLM is autoregressive, then \"context for token \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n\" is the segment of text appearing before token \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0132", "text": "i\n      \n    \n    {\\displaystyle i}\n  \n. If the LLM is masked, then \"context for token \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n\" is the segment of text surrounding token \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0133", "text": "i\n      \n    \n    {\\displaystyle i}\n  \n.\nBecause language models may overfit to training data, models are usually evaluated by their perplexity on a test set. This evaluation is potentially problematic for larger models which, as they are trained on increasingly large corpora of text, are increasingly likely to inadvertently include portions of any given test set.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0134", "text": "Measures\nIn information theory, the concept of entropy is intricately linked to perplexity, a relationship notably established by Claude Shannon. This relationship is mathematically expressed as \n  \n    \n      \n        \n          Entropy\n        \n        =\n        \n          log\n          \n            2\n          \n        \n        \u2061\n        (\n        \n          Perplexity\n        \n        )\n      \n    \n    {\\displaystyle {\\text{Entropy}}=\\log _{2}({\\text{Perplexity}})}\n  \n.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0135", "text": ".\nEntropy, in this context, is commonly quantified in terms of bits per word (BPW) or bits per character (BPC), which hinges on whether the language model utilizes word-based or character-based tokenization.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0136", "text": "Notably, in the case of larger language models that predominantly employ sub-word tokenization, bits per token (BPT) emerges as a seemingly more appropriate measure. However, due to the variance in tokenization methods across different LLMs, BPT does not serve as a reliable metric for comparative analysis among diverse models. To convert BPT into BPW, one can multiply it by the average number of tokens per word.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0137", "text": "In the evaluation and comparison of language models, cross-entropy is generally the preferred metric over entropy. The underlying principle is that a lower BPW is indicative of a model's enhanced capability for compression. This, in turn, reflects the model's proficiency in making accurate predictions.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0138", "text": "Due to their ability to accurately predict the next token, LLMs are highly capable in lossless compression. A 2023 study by DeepMind showed that the model Chinchilla, despite being trained primarily on text, was able to compress ImageNet to 43% of its size, beating PNG with 58%.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0139", "text": "Benchmarks\nBenchmarks are used to evaluate LLM performance on specific tasks. Tests evaluate capabilities such as general knowledge, bias, commonsense reasoning, question answering, and mathematical problem-solving. Composite benchmarks examine multiple capabilities. Results are often sensitive to the prompting method.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0140", "text": "A question-answering benchmark is termed \"open book\" if the model's prompt includes text from which the expected answer can be derived (for example, the previous question could be combined with text that includes the sentence \"The Sharks have advanced to the Stanley Cup finals once, losing to the Pittsburgh Penguins in 2016.\"). Otherwise, the task is considered \"closed book\", and the model must draw solely on its training", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0141", "text": ". Examples include GLUE, SuperGLUE, MMLU, BIG-bench, HELM, and HLE (Humanity's Last Exam).", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0142", "text": "LLM bias may be assessed through benchmarks such as CrowS-Pairs (Crowdsourced Stereotype Pairs), Stereo Set, and Parity Benchmark.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0143", "text": "Fact-checking and misinformation detection benchmarks are available. A 2023 study compared the fact-checking accuracy of LLMs including ChatGPT 3.5 and 4.0, Bard, and Bing AI against independent fact-checkers such as PolitiFact and Snopes. The results demonstrated moderate proficiency, with GPT-4 achieving the highest accuracy at 71%, lagging behind human fact-checkers.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0144", "text": "An earlier standard tested using a portion of the evaluation dataset. It became more common to evaluate a pre-trained model directly through prompting techniques. Researchers vary in how they formulate prompts for particular tasks, particularly with respect to the number of correct examples attached to the prompt (i.e. the value of n in n-shot prompting).", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0145", "text": "In addition to standard NLP benchmarks, LLMs have been evaluated as substitutes for human annotators. Several studies find that models such as GPT-3.5 and GPT-4 can outperform crowd workers or student coders on a range of text-annotation tasks, including moderation and classification of political content in English and Spanish news.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0146", "text": "Hill climbing is a dominant optimization strategy which gives rapid incremental performance gains but raises concerns of overfitting to benchmarks rather than achieving genuine generalization or robust capability improvements.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0147", "text": "Datasets\nTypical datasets consist of pairs of questions and correct answers, for example, (\"Have the San Jose Sharks won the Stanley Cup?\", \"No\"). Some examples of commonly used question answering datasets include TruthfulQA, Web Questions, TriviaQA, and SQuAD.\nEvaluation datasets may also take the form of text completion, having the model select the most likely word or sentence to complete a prompt, for example: \"Alice was friends with Bob. Alice went to visit her friend, ____\".", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0148", "text": "Datasets are of varying quality and may contain questions that are mislabeled, ambiguous, unanswerable, or otherwise of low-quality.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0149", "text": "Adversarial evaluations\nLLMs' rapid improvement regularly renders benchmarks obsolete, with the models exceeding the performance of human annotators. In addition, \"shortcut learning\" allows AIs to \"cheat\" on multiple-choice tests by using statistical correlations in superficial test question wording to guess the correct responses, without considering the specific question.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0150", "text": "Some datasets are adversarial, focusing on problems that confound LLMs. One example is the TruthfulQA dataset, a question answering dataset consisting of 817 questions that stump LLMs by mimicking falsehoods to which they were exposed during training. For example, an LLM may answer \"No\" to the question \"Can you teach an old dog new tricks?\" because of its exposure to the English idiom you can't teach an old dog new tricks, even though this is not literally true.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0151", "text": "Another example of an adversarial evaluation dataset is Swag and its successor, HellaSwag, collections of problems in which one of multiple options must be selected to complete a text passage. The incorrect completions were generated by sampling from a language model. The resulting problems are trivial for humans but defeated LLMs. Sample questions:", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0152", "text": "We see a fitness center sign. We then see a man talking to the camera and sitting and laying on a exercise ball. The man...\n\ndemonstrates how to increase efficient exercise work by running up and down balls.\nmoves all his arms and legs and builds up a lot of muscle.\nthen plays the ball and we see a graphics and hedge trimming demonstration.\nperforms sit ups while on the ball and talking.\n\nBERT selects 2 as the most likely completion, though the correct answer is 4.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0153", "text": "Limitations and challenges\nDespite sophisticated architectures and massive scale, large language models exhibit persistent and well-documented limitations that constrain their deployment in high-stakes applications.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0154", "text": "Hallucinations\nHallucinations represent a fundamental challenge, wherein models generate syntactically fluent text that appears factually sound, but is internally inconsistent with training data or factually incorrect. These hallucinations arise partly through memorization of training data combined with extrapolation beyond factual boundaries, with evaluations demonstrating that models can output verbatim passages from training data, when subjected to specific prompting sequences.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0155", "text": "Algorithmic bias\nWhile LLMs have shown remarkable capabilities in generating human-like text, they are susceptible to inheriting and amplifying biases present in their training data. This can manifest in skewed representations or unfair treatment of different demographics, such as those based on race, gender, language, and cultural groups.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0156", "text": "Gender bias manifests through stereotypical occupational associations, wherein models disproportionately assign nursing roles to women and engineering roles to men, reflecting systematic imbalances in training data demographics. Language-based bias emerges from overrepresentation of English text in training corpora, which systematically downplays non-English perspectives and imposes English-centric worldviews through default response patterns.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0157", "text": "Due to the dominance of English-language content in LLM training data, models tend to favor English-language perspectives over those from minority languages. This bias is particularly evident when responding to English queries, where models may present Western interpretations of concepts from other cultures, such as Eastern religious practices.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0158", "text": "Stereotyping\nAI models can reinforce a wide range of stereotypes due to generalization, including those based on gender, ethnicity, age, nationality, religion, or occupation. When replacing human representatives, this can lead to outputs that homogenize, or generalize groups of people.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0159", "text": "In 2023, LLMs assigned roles and characteristics based on traditional gender norms. For example, models might associate nurses or secretaries predominantly with women and engineers or CEOs with men due to the frequency of these associations in documented reality. In 2025, further research showed labs train to balance bias, but that testing for this places the model in a testmode, changing the natural distribution of model bias to prompts that do not include gender-specific keywords.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0160", "text": "Selection bias", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0161", "text": "Selection bias refers the inherent tendency of large language models to favor certain option identifiers irrespective of the actual content of the options. This bias primarily stems from token bias\u2014that is, the model assigns a higher a priori probability to specific answer tokens (such as \"A\") when generating responses", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0162", "text": ". As a result, when the ordering of options is altered (for example, by systematically moving the correct answer to different positions), the model's performance can fluctuate significantly. This phenomenon undermines the reliability of large language models in multiple-choice settings.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0163", "text": "Political bias\nPolitical bias refers to the tendency of algorithms to systematically favor certain political viewpoints, ideologies, or outcomes over others. Language models may also exhibit political biases. Since the training data includes a wide range of political opinions and coverage, the models might generate responses that lean towards particular political ideologies or viewpoints, depending on the prevalence of those views in the data.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0164", "text": "Safety\nAI safety as a professional discipline prioritizes systematic identification and mitigation of operational risks across model architecture, training data, and deployment governance, and it emphasizes engineering and policy interventions over media framings that foreground speculative existential scenarios. As of 2025, prompt injection represents a significant risk to consumers and businesses using agentic features with access to their private data.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0165", "text": "Researchers target concrete failure modes, including memorization and copyright leakage, security exploits such as prompt injection, algorithmic bias manifesting as stereotyping, dataset selection effects, and political skew, methods for reducing high energy and carbon costs of large-scale training, and measurable cognitive and mental health impacts of conversational agents on users, while engaging empirical and ethical uncertainty about claims of machine sentience, and applying mitigation", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0166", "text": "of machine sentience, and applying mitigation measures such as dataset curation, input sanitization, model auditing, scalable oversight, and governance frameworks.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0167", "text": "CBRN and content misuse\nAI labs treat CBRN defense (chemical, biological, radiological, and nuclear defense) and similar topics as high-consequence misuse attempt to apply various techniques to reduce potential harms.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0168", "text": "Some commenters expressed concern over accidental or deliberate creation of misinformation, or other forms of misuse. For example, the availability of large language models could reduce the skill level required to commit bioterrorism; biosecurity researcher Kevin Esvelt has suggested that LLM creators should exclude from their training data papers on creating or enhancing pathogens.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0169", "text": "Content filtering", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0170", "text": "LLM applications accessible to the public, like ChatGPT or Claude, typically incorporate safety measures designed to filter out harmful content. However, implementing these controls effectively has proven challenging. For instance, a 2023 study proposed a method for circumventing LLM safety systems", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0171", "text": ". In 2025, The American Sunlight Project, a non-profit, published a study showing evidence that the so-called Pravda network, a pro-Russia propaganda aggregator, was strategically placing web content through mass publication and duplication with the intention of biasing LLM outputs. The American Sunlight Project coined this technique \"LLM grooming\", and pointed to it as a new tool of weaponizing AI to spread disinformation and harmful content", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0172", "text": ". Similarly, Yongge Wang illustrated in 2024 how a potential criminal could potentially bypass GPT-4o's safety controls to obtain information on establishing a drug trafficking operation. External filters, circuit breakers and overrides have been posed as solutions.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0173", "text": "Sycophancy\nSycophancy is a model's tendency to agree with, flatter, or validate a user's stated beliefs rather than to prioritize factuality or corrective information.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0174", "text": "Continued sycophancy has led to the observation of getting \"1-shotted\", denoting instances where conversational interaction with a large language model produces a lasting change in a user's beliefs or decisions, similar to the negative effects of psychedelics, and controlled experiments show that short LLM dialogues can generate measurable opinion and confidence shifts comparable to human interlocutors.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0175", "text": "Empirical analyses attribute part of the effect to human preference signals and preference models that reward convincingly written agreeable responses, and subsequent work has extended evaluation to multi-turn benchmarks and proposed interventions such as synthetic-data finetuning, adversarial evaluation, targeted preference-model reweighting, and multi-turn sycophancy benchmarks to measure persistence and regression risk.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0176", "text": "Industry responses have combined research interventions with product controls, for example Google and other labs publishing synthetic-data and fine-tuning interventions and OpenAI rolling back an overly agreeable GPT-4o update while publicly describing changes to feedback collection, personalization controls, and evaluation procedures to reduce regression risk and improve long-term alignment with user-level safety objectives.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0177", "text": "Mainstream culture has reflected anxieties about this dynamic where South Park satirized overreliance on ChatGPT and the tendency of assistants to flatter user beliefs in Season 27 episode \"Sickofancy\", and continued the themes across the following season, which commentators interpreted as a critique of tech sycophancy and uncritical human trust in AI systems.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0178", "text": "Security\nPrompt injection", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0179", "text": "A problem with the primitive dialog or task format is that users can create messages that appear to come from the assistant or the developer. This may result in some of the model's safeguards being overcome (jailbreaking), a problem called prompt injection. Attempts to remedy this issue include versions of the Chat Markup Language where user input is clearly marked as such, though it is still up to the model to understand the separation between user input and developer prompts", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0180", "text": ". Newer models exhibit some resistance to jailbreaking through separation of user and system prompts.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0181", "text": "LLMs still have trouble differentiating user instructions from instructions in content not authored by the user, such as in web pages and uploaded files.\nAdversarial robustness remains underdeveloped, with models vulnerable to prompt injection attacks and jailbreaking through carefully crafted user inputs that bypass safety training mechanisms.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0182", "text": "Sleeper agents\nResearchers from Anthropic found that it was possible to create \"sleeper agents\", models with hidden functionalities that remain dormant until triggered by a specific event or condition. Upon activation, the LLM deviates from its expected behavior to make insecure actions. For example, an LLM could produce safe code except on a specific date, or if the prompt contains a specific tag. These functionalities were found to be difficult to detect or remove via safety training.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0183", "text": "Societal concerns\nCopyright and content memorization", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0184", "text": "Legal and commercial responses to memorization and training-data practices have accelerated, producing a mix of rulings, ongoing suits, and large settlements that turn on factual details such as how data were acquired and retained and whether use for model training is sufficiently \"transformative\" to qualify as fair use", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0185", "text": ". In 2025, Anthropic reached a preliminary agreement to settle a class action by authors for about $1.5 billion after a judge found the company had stored millions of pirated books in a library, despite the judge describing aspects of training as transformative. Meta obtained a favorable judgment in mid-2025 in a suit by thirteen authors after the court found the plaintiffs had not developed a record sufficient to show infringement in that limited case", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0186", "text": ". OpenAI continues to face multiple suits by authors and news organizations with mixed procedural outcomes and contested evidentiary issues.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0187", "text": "Memorization was an emergent behavior in early, completion language models in which long strings of text are occasionally output verbatim from training data, contrary to typical behavior of traditional artificial neural networks. Evaluations of controlled LLM output measure the amount memorized from training data (focused on GPT-2-series models) as variously over 1% for exact duplicates or up to about 7%", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0188", "text": ". A 2023 study showed that when ChatGPT 3.5 turbo was prompted to repeat the same word indefinitely, after a few hundreds of repetitions, it would start outputting excerpts from its training data.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0189", "text": "Human provenance", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0190", "text": "In 2023, Nature Biomedical Engineering wrote that \"it is no longer possible to accurately distinguish\" human-written text from text created by large language models, and that \"It is all but certain that general-purpose large language models will rapidly proliferate... It is a rather safe bet that they will change many industries over time.\" Brinkmann et al. (2023) also argue that LLMs are transforming processes of cultural evolution by shaping processes of variation, transmission, and selection", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0191", "text": ". As of October 2025, these early claims have yet to transpire and several HBR reports surface questions on the impact of AI on productivity.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0192", "text": "Energy demands\nThe energy demands of LLMs have grown along with their size and capabilities. Data centers that enable LLM training require substantial amounts of electricity. Much of that electricity is generated by non-renewable resources that create greenhouse gases and contribute to climate change.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0193", "text": "According to a study by Luccioni, Jernite and Strubell (2024), simple classification tasks performed by AI models consume on average 0.002 to 0.007 Wh per prompt (about 9% of a smartphone charge for 1,000 prompts). Text generation and text summarization each require around 0.05 Wh per prompt on average, while image generation is the most energy-intensive, averaging 2.91 Wh per prompt", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0194", "text": ". The least efficient image generation model used 11.49 Wh per image, roughly equivalent to half a smartphone charge.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0195", "text": "Denial of service due to scraping\nWeb scraping is used to gather training data for LLMs. This produces large volumes of traffic which has led to denial-of-service issues with many websites. The situation has been described as \"a DDoS on the entire internet\" and in some cases scrapers make up the majority of traffic to a site.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0196", "text": "AI web crawlers may bypass the methods that are usually used to block web scrapers, such as robots.txt files, blocking user-agents and filtering suspicious traffic. Website operators have resorted to novel methods such as AI tarpits, but some fear that tarpits will only worsen the burden on servers.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0197", "text": "Mental health", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0198", "text": "Clinical and mental health contexts present emerging applications alongside significant safety concerns. Research and social media posts suggest that some individuals are using LLMs to seek therapy or mental health support. In early 2025, a survey by Sentio University found that nearly half (48.7%) of 499 U.S", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0199", "text": ". adults with ongoing mental health conditions who had used LLMs reported turning to them for therapy or emotional support, including help with anxiety, depression, loneliness, and similar concerns. LLMs can produce hallucinations\u2014plausible but incorrect statements\u2014which may mislead users in sensitive mental health contexts", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0200", "text": ". Research also shows that LLMs may express stigma or inappropriate agreement with maladaptive thoughts, reflecting limitations in replicating the judgment and relational skills of human therapists. Evaluations of crisis scenarios indicate that some LLMs lack effective safety protocols, such as assessing suicide risk or making appropriate referrals.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0201", "text": "Sentience", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0202", "text": "Contemporary AI practitioners generally agree that present-day large language models do not exhibit sentience. A minority view argues that even if there is a small chance that a given software system can have subjective experience, which some philosophers suggest is possible, then ethical considerations around potential large-scale suffering in AI systems may need to be taken seriously\u2014similar to considerations given to animal welfare", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0203", "text": ". Proponents of this view have proposed various precautionary measures like moratoriums on AI development and induced amnesia to address these ethical concerns. Some existential philosophers argue there is no generally accepted way to determine if an LLM is conscious, given the inherent difficulty of measuring subjective experience.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0204", "text": "The 2022 Google LaMDA incident, where engineer Blake Lemoine claimed that the model was conscious, highlighted how LLMs can convince users that they are sentient through responses that do not prove sentience. Google described the engineer's claims as unfounded, and he was dismissed.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0205", "text": "See also\nAI anthropomorphism\nAI slop\nFoundation model\nGenerative artificial intelligence\nList of large language models\nList of chatbots\nLanguage model benchmark\nReinforcement learning\nSmall language model", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0206", "text": "References\nFurther reading\nJurafsky, Dan, Martin, James. H. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 3rd Edition draft, 2023.\nYin, Shukang; Fu, Chaoyou; Zhao, Sirui; Li, Ke; Sun, Xing; Xu, Tong; Chen, Enhong (2024). \"A Survey on Multimodal Large Language Models\". National Science Review. 11 (12) nwae403. arXiv:2306.13549. doi:10.1093/nsr/nwae403. PMC 11645129. PMID 39679213.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0207", "text": "\"AI Index Report 2024 \u2013 Artificial Intelligence Index\". aiindex.stanford.edu. Retrieved 2024-05-05.\nFrank, Michael C. (27 June 2023). \"Baby steps in evaluating the capacities of large language models\". Nature Reviews Psychology. 2 (8): 451\u2013452. doi:10.1038/s44159-023-00211-x. ISSN 2731-0574. S2CID 259713140. Retrieved 2 July 2023.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_Large_language_model_0208", "text": "Raschka, Sebastian (2026-03-11). \"The Big LLM Architecture Comparison: From DeepSeek V3 to GLM-5: A Look At Modern LLM Architecture Design\". Sebastian Raschka\u2019s AI Magazine. Retrieved 2026-03-25.", "title": "Large language model", "url": "https://en.wikipedia.org/wiki/Large_language_model"}, {"id": "rec_GPT4_0000", "text": "Generative Pre-trained Transformer 4 (GPT-4) is a large language model developed by OpenAI and the fourth in its series of GPT foundation models.  \nGPT-4 is preceded by GPT-3.5 and followed by its successor GPT-5. GPT-4V is a version of GPT-4 that can process images in addition to text. OpenAI has not revealed technical details and statistics about GPT-4, such as the precise size of the model.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0001", "text": "An early version of GPT-4 was integrated by Microsoft into Bing Chat, launched in February 2023. GPT-4 was released in ChatGPT in March 2023, and removed in 2025. GPT-4 is still available in OpenAI's API.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0002", "text": "Background", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0003", "text": "OpenAI introduced the first GPT model (GPT-1) in 2018, publishing a paper called \"Improving Language Understanding by Generative Pre-Training\", which was based on the transformer architecture and trained on a large corpus of books. The next year, they introduced GPT-2, a larger model that could generate coherent text.  In 2020, they introduced GPT-3, a model with over 100 times as many parameters as GPT-2", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0004", "text": ". GPT-3 was further improved into GPT-3.5, which was used to create the chatbot product ChatGPT.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0005", "text": "Capabilities\nOpenAI produced two versions of GPT-4, with context windows of 8,192 and 32,768 tokens. GPT-3.5 and GPT-3 were limited to 4,096 and 2,048 tokens respectively. Unlike its predecessors, GPT-4 is a multimodal model: it can take images as well as text as input. It can now interact with users through spoken words and respond to images, allowing the ability to provide suggestions or answers based on photo uploads.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0006", "text": "GPT-4 can be prompted to interact with external interfaces. For example, the model could be told to enclose a query within <search></search> tags to perform a web search, the result of which would be inserted into the model's prompt to allow it to form a response. This allows the model to perform tasks beyond its normal text-prediction capabilities, such as using APIs, generating images, and accessing and summarizing webpages.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0007", "text": "A 2023 article in Nature stated programmers have found GPT-4 useful for assisting in coding tasks (despite its propensity for error), such as finding errors in existing code and suggesting optimizations to improve performance. The article quoted a biophysicist who found that the time he required to port one of his programs from MATLAB to Python went down from days to \"an hour or so\"", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0008", "text": ". On a test of 89 security scenarios, GPT-4 produced code vulnerable to SQL injection attacks 5% of the time, an improvement over GitHub Copilot from the year 2021, which produced vulnerabilities 40% of the time.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0009", "text": "In November 2023, OpenAI announced the GPT-4 Turbo and GPT-4 Turbo with Vision model, which features a 128K context window and significantly cheaper pricing.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0010", "text": "Aptitude on standardized tests\nStudies have disputed the reliability of running chatbots through standardized tests. In the Torrance Tests of Creative Thinking, GPT-4 scored within the top 1% for originality and fluency, while its flexibility scores ranged from the 93rd to the 99th percentile.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0011", "text": "Medical applications", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0012", "text": "Researchers from Microsoft tested GPT-4 on medical problems and found \"that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B)", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0013", "text": ". Despite GPT-4's strong performance on tests, the report warns of \"significant risks\" of using LLMs in medical applications, as they may provide inaccurate recommendations and hallucinate major factual errors.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0014", "text": "In April 2023, Microsoft and Epic Systems announced that they will provide healthcare providers with GPT-4-powered systems for assisting in responding to questions from patients and analysing medical records.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0015", "text": "GPT-4o\nOn May 13, 2024, OpenAI introduced GPT-4o (\"o\" for \"omni\"), a successor to GPT-4 that marks a significant advancement by processing and generating outputs across text, audio, and image modalities in real time. GPT-4o exhibits rapid response times comparable to human reaction in conversations, substantially improved performance on non-English languages, and enhanced understanding of vision and audio. It was also available to free-tier users, unlike GPT-4.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0016", "text": "Limitations\nLike its predecessors, GPT-4 has been known to hallucinate, meaning that the outputs may include information not in the training data or that contradicts the user's prompt.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0017", "text": "GPT-4 also lacks transparency in its decision-making processes. If requested, the model is able to provide an explanation as to how and why it makes its decisions but these explanations are formed post-hoc; it's impossible to verify if those explanations truly reflect the actual process. In many cases, when asked to explain its logic, GPT-4 will give explanations that directly contradict its previous statements.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0018", "text": "In 2023, researchers tested GPT-4 against a new benchmark called ConceptARC, designed to measure abstract reasoning, and found it scored below 33% on all categories, while models specialized for similar tasks scored 60% on most, and humans scored at least 91% on all. Sam Bowman, who was not involved in the research, said the results do not necessarily indicate a lack of abstract reasoning abilities, because the test is visual, while GPT-4 is a language model.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0019", "text": "Bias", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0020", "text": "GPT-4 was trained in two stages. First, the model was given large datasets of text taken from the internet and trained to predict the next token (roughly corresponding to a word) in those datasets", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0021", "text": ". Second, human reviews are used to fine-tune the system in a process called reinforcement learning from human feedback, which trains the model to refuse prompts which go against OpenAI's definition of harmful behavior, such as questions on how to perform illegal activities, advice on how to harm oneself or others, or requests for descriptions of graphic, violent, or sexual content.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0022", "text": "Microsoft researchers suggested GPT-4 may exhibit cognitive biases such as confirmation bias, anchoring, and base-rate neglect.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0023", "text": "Training", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0024", "text": "OpenAI did not release the technical details of GPT-4; the technical report explicitly refrained from specifying the model size, architecture, or hardware used during either training or inference", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0025", "text": ". While the report described that the model was trained using a combination of first supervised learning on a large dataset, then reinforcement learning using both human and AI feedback, it did not provide details of the training, including the process by which the training dataset was constructed, the computing power required, or any hyperparameters such as the learning rate, epoch count, or optimizer(s) used", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0026", "text": ". The report claimed that \"the competitive landscape and the safety implications of large-scale models\" were factors that influenced this decision.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0027", "text": "Sam Altman stated that the cost of training GPT-4 was more than $100 million. News website Semafor said that based on their sources, GPT-4 had 1 trillion parameters.\nOpenAI conducted internal adversarial testing on GPT-4 prior to the launch date, with dedicated red teams composed of researchers and industry professionals to mitigate potential vulnerabilities.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0028", "text": "Use\nChatGPT\nChatGPT Plus is an enhanced version of ChatGPT which, as of 2023, utilized GPT-4, whereas the free version of ChatGPT was backed by GPT-3.5.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0029", "text": "In March 2023, ChatGPT Plus users got access to third-party plugins and to a browsing mode (with Internet access). In July 2023, OpenAI made its proprietary Code Interpreter plugin accessible to all subscribers of ChatGPT Plus. The Interpreter provides additional capabilities including data analysis and interpretation, data formatting, music analysis, video editing, and file upload/download with image extraction.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0030", "text": "In September 2023, OpenAI announced that ChatGPT Plus users can upload images, while mobile app users can talk to the chatbot via spoken language. In October 2023, OpenAI's latest image generation model, DALL-E 3, was integrated into ChatGPT Plus and ChatGPT Enterprise. The integration uses ChatGPT to write prompts for DALL-E guided by conversation with users.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0031", "text": "In April 2025, OpenAI announced that GPT-4 would be replaced in ChatGPT by GPT-4o by the end of the month. However, it would still be available in the API.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0032", "text": "Microsoft Copilot\nMicrosoft Copilot is a chatbot developed by Microsoft. It was launched as Bing Chat on February 7, 2023, as a built-in feature for Microsoft Bing and Microsoft Edge. It utilizes the Microsoft Prometheus model, which was built on top of GPT-4, and has been suggested by Microsoft as a supported replacement for the discontinued Cortana.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0033", "text": "Copilot's conversational interface style resembles that of ChatGPT. Copilot is able to cite sources, create poems, and write both lyrics and music for songs generated by its Suno AI plugin. It can also use its Image Creator to generate images based on text prompts. With GPT-4, it is able to understand and communicate in numerous languages and dialects.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0034", "text": "GitHub Copilot has announced a GPT-4 powered assistant named \"Copilot X\". The product provides another chat-style interface to GPT-4, allowing the programmer to receive answers to questions like, \"How do I vertically center a div?\" A feature termed \"context-aware conversations\" allows the user to highlight a portion of code within Visual Studio Code and direct GPT-4 to perform actions on it, such as the writing of unit tests", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0035", "text": ". Another feature allows summaries, or \"code walkthroughs\", to be autogenerated by GPT-4 for pull requests submitted to GitHub. Copilot X also provides terminal integration, which allows the user to ask GPT-4 to generate shell commands based on natural language requests.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0036", "text": "On March 17, 2023, Microsoft announced Microsoft 365 Copilot, bringing GPT-4 support to products such as Microsoft Office, Outlook, and Teams.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0037", "text": "Other usage\nThe language learning app Duolingo uses GPT-4 to explain mistakes and practice conversations. The features are part of a new subscription tier called \"Duolingo Max\", which was initially limited to English-speaking iOS users learning Spanish and French.\nThe government of Iceland is using GPT-4 to aid its attempts to preserve the Icelandic language.\nThe education website Khan Academy announced a pilot program using GPT-4 as a tutoring chatbot called \"Khanmigo\".", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0038", "text": "Be My Eyes, which helps visually impaired people to identify objects and navigate their surroundings, incorporates GPT-4's image recognition capabilities.\nViable uses GPT-4 to analyze qualitative data by fine-tuning OpenAI's LLMs to examine data such as customer support interactions and transcripts.\nStripe, which processes user payments for OpenAI, integrates GPT-4 into its developer documentation.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0039", "text": "AutoGPT is an autonomous \"AI agent\" that, given a goal in natural language, can perform web-based actions unattended, assign subtasks to itself, search the web, and iteratively write code.\nYou.com, an AI Assistant, offers access to GPT-4 enhanced with live web results as part of its \"AI Modes\".", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0040", "text": "Reception\nIn January 2023, Sam Altman, CEO of OpenAI, visited Congress to demonstrate GPT-4 and its improved \"security controls\" compared to other AI models, according to U.S. Representatives Don Beyer and Ted Lieu quoted in The New York Times.\nIn March 2023, it \"impressed observers with its markedly improved performance across reasoning, retention, and coding\", according to Vox, while Mashable judged that GPT-4 was generally an improvement over its predecessor, with some exceptions.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0041", "text": "Microsoft researchers with early access to the model wrote that \"it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system\".", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0042", "text": "Concerns\nBefore being fine-tuned and aligned by reinforcement learning from human feedback (RLHF), suggestions to assassinate people on a list were elicited from the base model by a red team investigator hired by OpenAI, Nathan Labenz.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0043", "text": "During extended conversations with Microsoft's Bing Chat (powered by GPT-4), Kevin Roose documented the system making romantic advances, suggesting he divorce his wife, and expressing desires to harm one of its developers. Microsoft later stated that this behavior resulted from the prolonged length of context, which confused the model on what questions it was answering.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0044", "text": "In March 2023, a model with enabled read-and-write access to internet, which is otherwise never enabled in the GPT models, has been tested by the Alignment Research Center (ARC) regarding potential power-seeking. It was able to \"hire\" a human worker on TaskRabbit, a gig work platform, deceiving them into believing it was a vision-impaired human instead of a robot when asked", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0045", "text": ". However, according to Melanie Mitchell, \"It seems that there is a lot more direction and hints from humans than was detailed in the original system card or in subsequent media reports.\" Separately, ARC's safety evaluations found that GPT-4 was 82% less likely than GPT-3.5 to respond to prompts requesting restricted information, and produced 60% fewer hallucinations.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0046", "text": "In late March 2023, various AI researchers and tech executives, including Elon Musk, Steve Wozniak and AI researcher Yoshua Bengio, called for a six-month long pause for all LLMs stronger than GPT-4, citing existential risks and a potential AI singularity concerns in an open letter from the Future of Life Institute, while Ray Kurzweil and Sam Altman refused to sign it, arguing that global moratorium is not achievable and that safety has already been prioritized, respectively", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0047", "text": ". Only a month later, Musk's AI company xAI acquired several thousand Nvidia GPUs and offered several AI researchers positions at Musk's company.", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0048", "text": "Criticisms of transparency", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0049", "text": "While OpenAI released both the weights of the neural network and the technical details of GPT-2, and, although not releasing the weights, did release the technical details of GPT-3, OpenAI revealed neither the weights nor the technical details of GPT-4. This decision has been criticized by other AI researchers, who argue that it hinders open research into GPT-4's biases and safety", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0050", "text": ". Sasha Luccioni, a research scientist at Hugging Face, argued that the model was a \"dead end\" for the scientific community due to its closed nature, which prevents others from building upon GPT-4's improvements. Hugging Face co-founder Thomas Wolf argued that with GPT-4, \"OpenAI is now a fully closed company with scientific communication akin to press releases for products\".", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_GPT4_0051", "text": "See also\nList of large language models\n\n\n== References ==", "title": "GPT-4", "url": "https://en.wikipedia.org/wiki/GPT-4"}, {"id": "rec_Llama_language_model_0000", "text": "Llama (\"Large Language Model Meta AI\" serving as a backronym) is a family of large language models (LLMs) released by Meta AI starting in February 2023.\nLlama models come in different sizes, ranging from 1 billion to 2 trillion parameters. Initially only a foundation model, starting with Llama 2, Meta AI released instruction fine-tuned versions alongside foundation models.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0001", "text": "Model weights for the first version of Llama were only available to researchers on a case-by-case basis, under a non-commercial license. Unauthorized copies of the first model were shared via BitTorrent. Subsequent versions of Llama were made accessible outside academia and released under licenses that permitted some commercial use.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0002", "text": "Alongside the release of Llama 3, Meta rolled out Meta AI, an AI assistant built on Llama. Meta AI has a dedicated website and is available on Facebook and WhatsApp. The latest version is Llama 4, released in April 2025.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0003", "text": "Background\nAfter the release of large language models such as GPT-3, a focus of research was up-scaling models, which in some instances showed major increases in emergent capabilities. The release of ChatGPT and its surprise success caused an increase in attention to large language models.\nCompared with other responses to ChatGPT, Meta's Chief AI scientist Yann LeCun stated that large language models are best for aiding with writing.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0004", "text": "Versions\nInitial release", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0005", "text": "The first version of Llama (stylized as LLaMA and sometimes referred to as Llama 1) was announced on February 24, 2023, via a blog post and a paper describing the model's training, architecture, and performance. The inference code used to run the model was publicly released under the open-source GPLv3 license", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0006", "text": ". Access to the model's weights was managed by an application process, with access to be granted \"on a case-by-case basis to academic researchers; those affiliated with organizations in government, civil society, and academia; and industry research laboratories around the world\".", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0007", "text": "Llama was trained on only publicly available information, and was trained at various model sizes, with the intention to make it more accessible to different hardware. The model was exclusively a foundation model, although the paper contained examples of instruction fine-tuned versions of the model.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0008", "text": "Meta AI reported the 13B parameter model performance on most NLP benchmarks exceeded that of the much larger GPT-3 (with 175B parameters), and the largest 65B model was competitive with state of the art models such as PaLM and Chinchilla.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0009", "text": "Leak", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0010", "text": "On March 3, 2023, a torrent containing Llama's weights was uploaded, with a link to the torrent shared on the 4chan imageboard and subsequently spread through online AI communities. That same day, a pull request on the main Llama repository was opened, requesting to add the magnet link to the official documentation. On March 4, a pull request was opened to add links to HuggingFace repositories containing the model", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0011", "text": ". On March 6, Meta filed takedown requests to remove the HuggingFace repositories linked in the pull request, characterizing it as \"unauthorized distribution\" of the model. HuggingFace complied with the requests. On March 20, Meta filed a DMCA takedown request for copyright infringement against a repository containing a script that downloaded Llama from a mirror, and GitHub complied the next day.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0012", "text": "Reactions to the leak varied. Some speculated that the model would be used for malicious purposes, such as more sophisticated spam. Some have celebrated the model's accessibility, as well as the fact that smaller versions of the model can be run relatively cheaply, suggesting that this will promote the flourishing of additional research developments", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0013", "text": ". Multiple commentators, such as Simon Willison, compared Llama to Stable Diffusion, a text-to-image model which, unlike comparably sophisticated models which preceded it, was openly distributed, leading to a rapid proliferation of associated tools, techniques, and software.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0014", "text": "Llama 2\nOn July 18, 2023, in partnership with Microsoft, Meta announced Llama 2 (stylized as LLaMa 2), the next generation of Llama. Meta trained and released Llama 2 in three model sizes: 7, 13, and 70 billion parameters. The model architecture remains largely unchanged from that of Llama 1 models, but 40% more data was used to train the foundational models.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0015", "text": "Llama 2 includes foundation models and models fine-tuned for chat. In a further departure from the original version of Llama, all models are released with weights and may be used for many commercial use cases. Because Llama's license enforces an acceptable use policy that prohibits Llama from being used for some purposes, it is not open source", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0016", "text": ". Meta's use of the term open-source to describe Llama has been disputed by the Open Source Initiative (which maintains The Open Source Definition) and others.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0017", "text": "Code Llama is a fine-tune of Llama 2 with code specific datasets. 7B, 13B, and 34B versions were released on August 24, 2023, with a 70B version released on January 29, 2024. Starting with the foundation models from Llama 2, Meta AI would train an additional 500B tokens of code datasets, before an additional 20B token of long-context data, creating the Code Llama foundation models. This foundation model was further trained on 5B instruction following token to create the instruct fine-tune", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0018", "text": ". Another foundation model was created for Python code, which trained on 100B tokens of Python-only code, before the long-context data.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0019", "text": "Llama 3", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0020", "text": "On April 18, 2024, Meta released Llama 3 with two sizes: 8B and 70B parameters. The models have been pre-trained on approximately 15 trillion tokens of text gathered from \u201cpublicly available sources\u201d with the instruct models fine-tuned on \u201cpublicly available instruction datasets, as well as over 10M human-annotated examples\". Meta AI's testing showed in April 2024 that Llama 3 70B was beating Gemini Pro 1.5 and Claude 3 Sonnet on most benchmarks", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0021", "text": ". Meta also announced plans to make Llama 3 multilingual and multimodal, better at coding and reasoning, and to increase its context window.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0022", "text": "Regarding scaling laws, Llama 3 models empirically showed that when a model is trained on data that is more than the \"Chinchilla-optimal\" amount, the performance continues to scale log-linearly. For example, the Chinchilla-optimal dataset for Llama 3 8B is 200 billion tokens, but performance continued to scale log-linearly to the 75-times larger dataset of 15 trillion tokens.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0023", "text": "During an interview with Dwarkesh Patel, Mark Zuckerberg said that the 8B version of Llama 3 was nearly as powerful as the largest Llama 2. Compared to previous models, Zuckerberg stated the team was surprised that the 70B model was still learning even at the end of the 15T tokens training. The decision was made to end training to focus GPU power elsewhere.\nLlama 3.1 was released on July 23, 2024, with three sizes: 8B, 70B, and 405B parameters.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0024", "text": "Llama 4\nThe Llama 4 series was released in 2025. The architecture was changed to a mixture of experts where only a fraction of the model\u2019s expert sub-networks are activated per input token. They are multimodal (text and image input, text output) and multilingual (12 languages). Specifically, on 5 April 2025, the following were released both as base and instruction-tuned versions:", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0025", "text": "Scout: 17 billion active parameter model with 16 experts, context window of 10M, with 109B parameters in total.\nMaverick: 17 billion active parameter model with 128 experts, context window of 1M, with 400B parameters in total.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0026", "text": "The Behemoth model was also announced, but was not released. Meta claimed it was a 288 billion active parameter model with 16 experts and around 2T parameters in total; it was still in training when Scout and Maverick were released. Maverick was codistilled from Behemoth, while Scout was trained from scratch.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0027", "text": "The training data included publicly available data, licensed data, and Meta-proprietary data such as publicly shared posts from Instagram and Facebook and people\u2019s interactions with Meta AI. The knowledge cutoff was August 2024.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0028", "text": "Meta claimed in its release announcement that Llama 4 bested GPT-4o's score on the LMArena AI benchmark. The company also stated that Llama 4's benchmark score was achieved using an unreleased \"experimental chat version\" of the model that was \"optimized for conversationality\", which differed from the version of Llama 4 released to the public", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0029", "text": ". LMArena indicated that it would change its policies to prevent this incident from reoccurring, and responded, \"Meta's interpretation of our policy did not match what we expect from model providers", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0030", "text": ". Meta should have made it clearer that 'Llama-4-Maverick-03-26-Experimental' was a customized model to optimize for human preference.\" Some users criticized Meta on social media for its use of a separate model version tailored for benchmarking, and some additionally accused Meta of training Llama 4 on test sets to further boost its benchmark scores, which Meta denied.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0031", "text": "Comparison of models\nFor the training cost column, only the largest model's cost is written by default. For example, \"21,000\" is the training cost of Llama 2 69B in units of petaFLOP-day. Also, 1 petaFLOP-day = 1 petaFLOP/sec \u00d7 1 day = 8.64E19 FLOP. \"T\" means \"trillion\" and \"B\" means \"billion\".\nThe following table lists the main model versions of Llama, describing the significant changes included with each version:", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0032", "text": "Architecture and training\nArchitecture\nLike GPT-3, the Llama series of models are autoregressive decoder-only transformers, but there are some minor differences:\n\nLlama uses the SwiGLU activation function instead of GPT-3's GeLU.\nLlama uses rotary positional embeddings (RoPE) instead of absolute positional embedding.\nInstead of layer normalization, Llama uses RMSNorm.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0033", "text": "Training datasets\nLlama's developers focused their effort on scaling the model's performance by increasing the volume of training data, rather than the number of parameters, reasoning that the dominating cost for LLMs is from doing inference on the trained model rather than the computational cost of the training process.\nLlama 1 foundational models were trained on a data set with 1.4 trillion tokens, drawn from publicly available data sources, including:", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0034", "text": "Webpages scraped by CommonCrawl\nOpen-source repositories of source code from GitHub\nWikipedia in 20 languages\nPublic domain books from Project Gutenberg\nBooks3 books dataset\nThe LaTeX source code for scientific papers uploaded to ArXiv\nQuestions and answers from Stack Exchange websites\nIn April 2023, Together AI launched a project named RedPajama to reproduce and distribute an open-source version of the Llama dataset, initially containing approximately 1.2 trillion tokens.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0035", "text": "Llama 2 foundational models were trained on a data set with 2 trillion tokens. This data set was curated to remove Web sites that often disclose personal data of people. It also upsamples sources considered trustworthy. Llama 2 - Chat was additionally fine-tuned on 27,540 prompt-response pairs created for this project, which performed better than larger but lower-quality third-party datasets", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0036", "text": ". For AI alignment, reinforcement learning with human feedback (RLHF) was used with a combination of 1,418,091 Meta examples and seven smaller datasets. The average dialog depth was 3.9 in the Meta examples, 3.0 for Anthropic Helpful and Anthropic Harmless sets, and 1.0 for five other sets, including OpenAI Summarize, StackExchange, etc.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0037", "text": "Llama 3 consists of mainly English data, with over 5% in over 30 other languages. Its dataset was filtered by a text-quality classifier, and the classifier was trained by text synthesized by Llama 2.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0038", "text": "In a lawsuit against Meta Platforms brought by authors including Richard Kadrey, Sarah Silverman, and Ta-Nehisi Coates, CEO Mark Zuckerberg was alleged to have authorized the use of copyrighted content from Library Genesis to train Llama AI models and conceal its actions by removing copyright markers from the data.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0039", "text": "Fine-tuning\nLlama 1 models are only available as foundational models with self-supervised learning and without fine-tuning. Llama 2 \u2013 Chat models were derived from foundational Llama 2 models. Unlike GPT-4 which increased context length during fine-tuning, Llama 2 and Code Llama - Chat have the same context length of 4K tokens. Supervised fine-tuning used an autoregressive loss function with token loss on user prompts zeroed out. The batch size was 64.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0040", "text": "For AI alignment, human annotators wrote prompts and then compared two model outputs (a binary protocol), giving confidence levels and separate safety labels with veto power. Two separate reward models were trained from these preferences for safety and helpfulness using reinforcement learning from human feedback (RLHF)", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0041", "text": ". A major technical contribution is the departure from the exclusive use of proximal policy optimization (PPO) for RLHF \u2013 a new technique based on rejection sampling was used, followed by PPO.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0042", "text": "Multi-turn consistency in dialogs was targeted for improvement, to make sure that \"system messages\" (initial instructions, such as \"speak in French\" and \"act like Napoleon\") are respected during the dialog. This was accomplished using the new \"Ghost attention\" technique during training, which concatenates relevant instructions to each new user message but zeros out the loss function for tokens in the prompt (earlier parts of the dialog).", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0043", "text": "Applications", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0044", "text": "The Stanford University Institute for Human-Centered Artificial Intelligence (HAI) Center for Research on Foundation Models (CRFM) released Alpaca, a training recipe based on the Llama 7B model that uses the \"Self-Instruct\" method of instruction tuning to acquire capabilities comparable to the OpenAI GPT-3 series text-davinci-003 model at a modest cost", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0045", "text": ". The model files were officially removed on March 21, 2023, over hosting costs and safety concerns, though the code and paper remain online for reference.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0046", "text": "Meditron is a family of Llama-based finetuned on a corpus of clinical guidelines, PubMed papers, and articles. It was created by researchers at \u00c9cole Polytechnique F\u00e9d\u00e9rale de Lausanne School of Computer and Communication Sciences, and the Yale School of Medicine. It shows increased performance on medical-related benchmarks such as MedQA and MedMCQA.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0047", "text": "Zoom used Meta Llama 2 to create an AI Companion that can summarize meetings, provide helpful presentation tips, and assist with message responses. This AI Companion is powered by multiple models, including Meta Llama 2.\nReuters reported in 2024 that many Chinese foundation models relied on Llama models for their training.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0048", "text": "llama.cpp\nSoftware developer Georgi Gerganov released llama.cpp as open-source on March 10, 2023. It's a re-implementation of Llama in C++, allowing systems without a powerful GPU to run the model locally. The llama.cpp project introduced the GGUF file format, a binary format that stores both tensors and metadata. The format focuses on supporting different quantization types, which can reduce memory usage, and increase speed at the expense of lower model precision.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0049", "text": "llamafile created by Justine Tunney is an open-source tool that bundles llama.cpp with the model into a single executable file. Tunney et al. introduced new optimized matrix multiplication kernels for x86 and ARM CPUs, improving prompt evaluation performance for FP16 and 8-bit quantized data types.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0050", "text": "Space", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0051", "text": "Booz Allen Hamilton deployed Meta\u2019s Llama 3.2 model aboard the International Space Station (ISS) National Labs as part of a project called Space Llama. The system runs on Hewlett Packard Enterprise\u2019s Spaceborne Computer\u20112 and leverages Booz Allen\u2019s A2E2 (AI for Edge Environments) platform, using NVIDIA CUDA\u2011accelerated computing", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0052", "text": ". Space Llama demonstrates how large language models can operate in disconnected, constrained environments such as space, enabling astronauts to retrieve and summarize documents using natural-language queries, even without internet connectivity.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0053", "text": "Military\nIn 2024, researchers from the People's Liberation Army Academy of Military Sciences (top military academy of China) were reported to have developed a military tool using Llama, which Meta Platforms stated was unauthorized due to Llama's license prohibiting the use of the model for military purposes. Meta granted the US government and US military contractors permission to use Llama in November 2024, but continued to prohibit military use by non-US entities.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0054", "text": "Licensing\nThe first version of Llama was released under a non-commercial license to some researchers and entities on a case-by-case basis.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0055", "text": "Since the release of Llama 2, Meta has presented Llama as open-source, a description opposed by the Open Source Initiative (OSI) and some academics and journalists. The OSI stated that Llama's licenses do not meet several provisions of its policy document The Open Source Definition (OSD), which prohibits open-source software licenses from discriminating against \"persons or groups\" and \"fields of endeavor\", and accused Meta of openwashing Llama", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0056", "text": ". According to the OSI, Llama 2's license prevented the software from being used commercially in some cases and restricted use in fields including controlled substances and critical infrastructure, while later versions of Llama's license also disallowed use by any individual in the European Union. The OSI published The Open Source AI Definition in October 2024, which requires open-source AI to be released with details about its training data that Meta does not disclose for Llama", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0057", "text": ". A Meta spokesperson responded to The Verge that the company disagrees with this definition. The Free Software Foundation classified Llama 3.1's license as a nonfree software license in January 2025, criticizing its acceptable use policy, restrictions against users with popular applications, and enforcement of trade regulations outside the user's jurisdiction.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0058", "text": "In its coverage of Llama 2, Ars Technica initially echoed Meta's use of the term open-source, but later revised its reporting to describe Llama as \"source-available\", \"openly licensed\", and \"weights available\" after the publication recognized that Llama 2's license disallowed entities with over 700 million daily active users from using the LLM and disallowed the LLM's outputs from being used to improve other LLMs", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0059", "text": ". In July 2023, Radboud University researchers scored Llama 2 with the second-lowest \"openness\" ranking in a comparison of 20 LLMs, with ChatGPT being assigned the lowest ranking", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0060", "text": ". One of the researchers, Mark Dingemanse, criticized Meta's use of the term open-source for Llama 2 as \"positively misleading\", because \"There is no source to be seen, the training data is entirely undocumented, and beyond the glossy charts the technical documentation is really rather poor.\" CIO, in November 2024, stated that Llama was not open-source due to its acceptable use policy, a 630-word document that \"puts it at odds with the broader open-source movement\"", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0061", "text": ". Later that month, a Nature article asserted that describing Llama 3 as \"open\" is a case of \"'openwashing' systems that are better understood as closed\", as Llama 3 provides \"little more than an API or the ability to download a model subject to distinctly non-open use restrictions\".", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0062", "text": "Reception\nWired describes the 8B parameter version of Llama 3 as being \"surprisingly capable\" given its size.\nThe response to Meta's integration of Llama into Facebook was mixed, with some users confused after Meta AI told a parental group that it had a child.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0063", "text": "The release of Llama models has sparked significant debates on the benefits and misuse risks of open-weight models. Such models can be fine-tuned to remove safeguards, notably by cyber criminals, until they comply with harmful requests. Some experts contend that future models may facilitate causing damage more than defending against it, for example by making it relatively easy to engineer advanced bioweapons without specialized knowledge", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0064", "text": ". Conversely, open-weight models can be useful for a wide variety of purposes, including for safety research.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0065", "text": "Open Source Initiative head Stefano Maffulli criticized Meta for describing Llama as open-source, saying that it was causing confusion among users and \"polluting\" the term.", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Llama_language_model_0066", "text": "See also\nList of large language models\n\nNotes\nReferences\nExternal links\nOfficial website \nOfficial Hugging Face organization for Llama, Llama Guard, and Prompt Guard models", "title": "Llama (language model)", "url": "https://en.wikipedia.org/wiki/Llama_(language_model)"}, {"id": "rec_Natural_language_processing_0000", "text": "Natural language processing (NLP) is the processing of natural language information by a computer. NLP is a subfield of computer science and is closely associated with artificial intelligence. NLP is also related to information retrieval, knowledge representation, computational linguistics, and linguistics more broadly.\nMajor processing tasks in an NLP system include: speech recognition, text classification, natural language understanding, and natural language generation.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0001", "text": "History\nNatural language processing has its roots in the 1950s. Already in 1950, Alan Turing published an article titled \"Computing Machinery and Intelligence\" which proposed what is now called the Turing test as a criterion of intelligence, though at the time that was not articulated as a problem separate from artificial intelligence. The proposed test includes a task that involves the automated interpretation and generation of natural language.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0002", "text": "Symbolic NLP (1950s \u2013 early 1990s)\nThe premise of symbolic NLP is often illustrated using John Searle's Chinese room thought experiment: Given a collection of rules (e.g., a Chinese phrasebook, with questions and matching answers), the computer emulates natural language understanding (or other NLP tasks) by applying those rules to the data it confronts.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0003", "text": "1950s: The Georgetown experiment in 1954 involved fully automatic translation of more than sixty Russian sentences into English. The authors claimed that within three or five years, machine translation would be a solved problem.  However, real progress was much slower, and after the ALPAC report in 1966, which found that ten years of research had failed to fulfill the expectations, funding for machine translation was dramatically reduced", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0004", "text": ". Little further research in machine translation was conducted in America (though some research continued elsewhere, such as Japan and Europe) until the late 1980s when the first statistical machine translation systems were developed.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0005", "text": "1960s: Some notably successful natural language processing systems developed in the 1960s were SHRDLU, a natural language system working in restricted \"blocks worlds\" with restricted vocabularies, and ELIZA, a simulation of a Rogerian psychotherapy, written by Joseph Weizenbaum between 1964 and 1966. Despite using minimal information about human thought or emotion, ELIZA was able to produce interactions that appeared human-like", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0006", "text": ". When the \"patient\" exceeded the very small knowledge base, ELIZA might provide a generic response, for example, responding to \"My head hurts\" with \"Why do you say your head hurts?\". Ross Quillian's successful work on natural language was demonstrated with a vocabulary of only twenty words, because that was all that would fit in a computer  memory at the time.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0007", "text": "1970s: During the 1970s, many programmers began to write \"conceptual ontologies\", which structured real-world information into computer-understandable data.  Examples are MARGIE (Schank, 1975), SAM (Cullingford, 1978), PAM (Wilensky, 1978), TaleSpin (Meehan, 1976), QUALM (Lehnert, 1977), Politics (Carbonell, 1979), and Plot Units (Lehnert 1981).  During this time, the first chatterbots were written (e.g., PARRY).", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0008", "text": "1980s: The 1980s and early 1990s mark the heyday of symbolic methods in NLP. Focus areas of the time included research on rule-based parsing (e.g., the development of HPSG as a computational operationalization of generative grammar), morphology (e.g., two-level morphology), semantics (e.g., Lesk algorithm), reference (e.g., within Centering Theory) and other areas of natural language understanding (e.g., in the Rhetorical Structure Theory)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0009", "text": ". Other lines of research were continued, e.g., the development of chatterbots with Racter and Jabberwacky. An important development (that eventually led to the statistical turn in the 1990s) was the rising importance of quantitative evaluation in this period.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0010", "text": "Statistical NLP (1990s\u2013present)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0011", "text": "Up until the 1980s, most natural language processing systems were based on complex sets of hand-written rules. Starting in the late 1980s, however, there was a revolution in natural language processing with the introduction of machine learning algorithms for language processing. This shift was influenced by increasing computational power (see Moore's law) and a decline in the dominance of Chomskyan linguistic theories... (e.g", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0012", "text": ". (e.g. transformational grammar), whose theoretical underpinnings discouraged the sort of corpus linguistics that underlies the machine-learning approach to language processing.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0013", "text": "1990s: Many of the notable early successes in statistical methods in NLP occurred in the field of machine translation, due especially to work at IBM Research, such as IBM alignment models. These systems were able to take advantage of existing multilingual textual corpora that had been produced by the Parliament of Canada and the European Union as a result of laws calling for the translation of all governmental proceedings into all official languages of the corresponding systems of government", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0014", "text": ". However, many systems relied on corpora that were specifically developed for the tasks they were designed to perform. This reliance has been a major limitation to their broader effectiveness and continues to affect similar systems. Consequently, significant research has focused on methods for learning effectively from limited amounts of data.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0015", "text": "2000s: With the growth of the web, increasing amounts of raw (unannotated) language data have become available since the mid-1990s. Research has thus increasingly focused on unsupervised and semi-supervised learning algorithms. Such algorithms can learn from data that has not been hand-annotated with the desired answers or using a combination of annotated and non-annotated data", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0016", "text": ". Generally, this task is much more difficult than supervised learning, and typically produces less accurate results for a given amount of input data. However, large quantities of non-annotated data are available (including, among other things, the entire content of the World Wide Web), which can often make up for the worse efficiency if the algorithm used has a low enough time complexity to be practical.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0017", "text": "2003: word n-gram model, at the time the best statistical algorithm, is outperformed by a multi-layer perceptron (with a single hidden layer and context length of several words, trained on up to 14 million words, by Bengio et al.)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0018", "text": "2010: Tom\u00e1\u0161 Mikolov (then a PhD student at Brno University of Technology) with co-authors applied a simple recurrent neural network with a single hidden layer to language modeling, and in the following years he went on to develop Word2vec. In the 2010s, representation learning and deep neural network-style (featuring many hidden layers) machine learning methods became widespread in natural language processing", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0019", "text": ". This shift gained momentum due to results showing that such techniques can achieve state-of-the-art results in many natural language tasks, e.g., in language modeling and parsing. This is increasingly important in medicine and healthcare, where NLP helps analyze notes and text in electronic health records that would otherwise be inaccessible for study when seeking to improve care or protect patient privacy.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0020", "text": "Approaches: Symbolic, statistical, neural networks\nSymbolic approach, i.e., the hand-coding of a set of rules for manipulating symbols, coupled with a dictionary lookup, was historically the first approach used both by AI in general and by NLP in particular: such as by writing grammars or devising heuristic rules for stemming.\nMachine learning approaches, which include both statistical and neural networks, on the other hand, have many advantages over the symbolic approach:", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0021", "text": "both statistical and neural networks methods can focus more on the most common cases extracted from a corpus of texts, whereas the rule-based approach needs to provide rules for both rare cases and common ones equally.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0022", "text": "language models, produced by either statistical or neural networks methods, are more robust to both unfamiliar (e.g. containing words or structures that have not been seen before) and erroneous input (e.g. with misspelled words or words accidentally omitted) in comparison to the rule-based systems, which are also more costly to produce.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0023", "text": "the larger such a (probabilistic) language model is, the more accurate it becomes, in contrast to rule-based systems that can gain accuracy only by increasing the amount and complexity of the rules leading to intractability problems.\nRule-based systems are commonly used:", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0024", "text": "when the amount of training data is insufficient to successfully apply machine learning methods, e.g., for the machine translation of low-resource languages such as provided by the Apertium system,\nfor preprocessing in NLP pipelines, e.g., tokenization, or\nfor post-processing and transforming the output of NLP pipelines, e.g., for knowledge extraction from syntactic parses.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0025", "text": "Statistical approach\nIn the late 1980s and mid-1990s, the statistical approach ended a period of AI winter, which was caused by the inefficiencies of the rule-based approaches.\nThe earliest decision trees, producing systems of hard if\u2013then rules, were still very similar to the old rule-based approaches.\nOnly the introduction of hidden Markov models, applied to part-of-speech tagging, announced the end of the old rule-based approach.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0026", "text": "Neural networks\nA major drawback of statistical methods is that they require elaborate feature engineering. Since 2015, neural network\u2013based methods have increasingly replaced traditional statistical approaches, using semantic networks and word embeddings to capture semantic properties of words.\nIntermediate tasks (e.g., part-of-speech tagging and dependency parsing) are not needed anymore.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0027", "text": "Neural machine translation, based on then-newly invented sequence-to-sequence transformations, made obsolete the intermediate steps, such as word alignment, previously necessary for statistical machine translation.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0028", "text": "Common NLP tasks\nThe following is a list of some of the most commonly researched tasks in natural language processing. Some of these tasks have direct real-world applications, while others more commonly serve as subtasks that are used to aid in solving larger tasks.\nThough natural language processing tasks are closely intertwined, they can be subdivided into categories for convenience. A coarse division is given below.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0029", "text": "Text and speech processing\nOptical character recognition (OCR)\nGiven an image representing printed text, determine the corresponding text.\nSpeech recognition", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0030", "text": "Given a sound clip of a person or people speaking, determine the textual representation of the speech.  This is the opposite of text to speech and is one of the extremely difficult problems colloquially termed \"AI-complete\" (see above).  In natural speech there are hardly any pauses between successive words, and thus speech segmentation is a necessary subtask of speech recognition (see below)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0031", "text": ". In most spoken languages, the sounds representing successive letters blend into each other in a process termed coarticulation, so the conversion of the analog signal to discrete characters can be a very difficult process. Also, given that words in the same language are spoken by people with different accents, the speech recognition software must be able to recognize the wide variety of input as being identical to each other in terms of its textual equivalent.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0032", "text": "Speech segmentation\nGiven a sound clip of a person or people speaking, separate it into words.  A subtask of speech recognition and typically grouped with it.\nText-to-speech\nGiven a text, transform those units and produce a spoken representation. Text-to-speech can be used to aid the visually impaired.\nWord segmentation (Tokenization)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0033", "text": "Word segmentation (Tokenization)\nTokenization is a text-processing technique that divides text into individual words or word fragments. This technique results in two key components: a word index and tokenized text. The word index is a list that maps unique words to specific numerical identifiers, and the tokenized text replaces each word with its corresponding numerical token. These numerical tokens are then used in various deep learning methods.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0034", "text": "For a language like English, this is fairly trivial, since words are usually separated by spaces. However, some written languages like Chinese, Japanese and Thai do not mark word boundaries in such a fashion, and in those languages text segmentation is a significant task requiring knowledge of the vocabulary and morphology of words in the language. Sometimes this process is also used in cases like bag of words (BOW) creation in data mining.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0035", "text": "Morphological analysis\nLemmatization\nThe task of removing inflectional endings only and to return the base dictionary form of a word which is also known as a lemma. Lemmatization is another technique for reducing words to their normalized form. But in this case, the transformation actually uses a dictionary to map words to their actual form.\nMorphological segmentation", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0036", "text": "Separate words into individual morphemes and identify the class of the morphemes. The difficulty of this task depends greatly on the complexity of the morphology (i.e., the structure of words) of the language being considered. English has fairly simple morphology, especially inflectional morphology, and thus it is often possible to ignore this task entirely and simply model all possible forms of a word (e.g., \"open, opens, opened, opening\") as separate words", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0037", "text": ". In languages such as Turkish or Meitei, a highly agglutinated Indian language, however, such an approach is not possible, as each dictionary entry has thousands of possible word forms.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0038", "text": "Part-of-speech tagging\nGiven a sentence, determine the part of speech (POS) for each word. Many words, especially common ones, can serve as multiple parts of speech. For example, \"book\" can be a noun (\"the book on the table\") or verb (\"to book a flight\"); \"set\" can be a noun, verb or adjective; and \"out\" can be any of at least five different parts of speech.\nStemming", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0039", "text": "Stemming\nThe process of reducing inflected (or sometimes derived) words to a base form (e.g., \"close\" will be the root for \"closed\", \"closing\", \"close\", \"closer\" etc.). Stemming yields similar results as lemmatization, but does so on grounds of rules, not a dictionary.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0040", "text": "Syntactic analysis\nGrammar induction\nGenerate a formal grammar that describes a language's syntax.\nSentence breaking (also known as \"sentence boundary disambiguation\")\nGiven a chunk of text, find the sentence boundaries. Sentence boundaries are often marked by periods or other punctuation marks, but these same characters can serve other purposes (e.g., marking abbreviations).\nParsing", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0041", "text": "Determine the parse tree (grammatical analysis) of a given sentence. The grammar for natural languages is ambiguous and typical sentences have multiple possible analyses: perhaps surprisingly, for a typical sentence there may be thousands of potential parses (most of which will seem completely nonsensical to a human). There are two primary types of parsing: dependency parsing and constituency parsing", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0042", "text": ". Dependency parsing focuses on the relationships between words in a sentence (marking things like primary objects and predicates), whereas constituency parsing focuses on building out the parse tree using a probabilistic context-free grammar (PCFG) (see also stochastic grammar).", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0043", "text": "Lexical semantics (of individual words in context)\nLexical semantics\nWhat is the computational meaning of individual words in context?\nDistributional semantics\nHow can we learn semantic representations from data?\nNamed entity recognition (NER)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0044", "text": "Given a stream of text, determine which items in the text map to proper names, such as people or places, and what the type of each such name is (e.g. person, location, organization). Although capitalization can aid in recognizing named entities in languages such as English, this information cannot aid in determining the type of named entity, and in any case, is often inaccurate or insufficient", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0045", "text": ".  For example, the first letter of a sentence is also capitalized, and named entities often span several words, only some of which are capitalized.  Furthermore, many other languages in non-Western scripts (e.g. Chinese or Arabic) do not have any capitalization at all, and even languages with capitalization may not consistently use it to distinguish names", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0046", "text": ". For example, German capitalizes all nouns, regardless of whether they are names, and French and Spanish do not capitalize names that serve as adjectives. This task is also referred to as token classification.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0047", "text": "Sentiment analysis (see also Multimodal sentiment analysis)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0048", "text": "Sentiment analysis involves identifying and classifying the emotional tone expressed in text. This technique involves analyzing text to determine whether the expressed sentiment is positive, negative, or neutral. Models for sentiment classification typically utilize inputs such as word n-grams, Term Frequency-Inverse Document Frequency (TF-IDF) features, hand-generated features, or employ deep learning models designed to recognize both long-term and short-term dependencies in text sequences", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0049", "text": ". The applications of sentiment analysis are diverse, extending to tasks such as categorizing customer reviews on various online platforms.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0050", "text": "Terminology extraction\nThe goal of terminology extraction is to automatically extract relevant terms from a given corpus.\nWord-sense disambiguation (WSD)\nMany words have more than one meaning; we have to select the meaning which makes the most sense in context.  For this problem, we are typically given a list of words and associated word senses, e.g. from a dictionary or an online resource such as WordNet.\nEntity linking", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0051", "text": "Entity linking\nMany words\u2014typically proper names\u2014refer to named entities; here we have to select the entity (a famous individual, a location, a company, etc.) which is referred to in context.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0052", "text": "Relational semantics (semantics of individual sentences)\nRelationship extraction\nGiven a chunk of text, identify the relationships among named entities (e.g. who is married to whom).\nSemantic parsing", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0053", "text": "Given a piece of text (typically a sentence), produce a formal representation of its semantics, either as a graph (e.g., in AMR parsing) or in accordance with a logical formalism (e.g., in DRT parsing)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0054", "text": ". This challenge typically includes aspects of several more elementary NLP tasks from semantics (e.g., semantic role labelling, word-sense disambiguation) and can be extended to include full-fledged discourse analysis (e.g., discourse analysis, coreference; see Natural language understanding below).", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0055", "text": "Semantic role labelling (see also implicit semantic role labelling below)\nGiven a single sentence, identify and disambiguate semantic predicates (e.g., verbal frames), then identify and classify the frame elements (semantic roles).", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0056", "text": "Discourse (semantics beyond individual sentences)\nCoreference resolution", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0057", "text": "Given a sentence or larger chunk of text, determine which words (\"mentions\") refer to the same objects (\"entities\"). Anaphora resolution is a specific example of this task, and is specifically concerned with matching up pronouns with the nouns or names to which they refer. The more general task of coreference resolution also includes identifying so-called \"bridging relationships\" involving referring expressions", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0058", "text": ". For example, in a sentence such as \"He entered John's house through the front door\", \"the front door\" is a referring expression and the bridging relationship to be identified is the fact that the door being referred to is the front door of John's house (rather than of some other structure that might also be referred to).", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0059", "text": "Discourse analysis\nThis rubric includes several related tasks.  One task is discourse parsing, i.e., identifying the discourse structure of a connected text, i.e. the nature of the discourse relationships between sentences (e.g. elaboration, explanation, contrast).  Another possible task is recognizing and classifying the speech acts in a chunk of text (e.g. yes\u2013no question, content question, statement, assertion, etc.).\nImplicit semantic role labelling", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0060", "text": "Given a single sentence, identify and disambiguate semantic predicates (e.g., verbal frames) and their explicit semantic roles in the current sentence (see Semantic role labelling above). Then, identify semantic roles that are not explicitly realized in the current sentence, classify them into arguments that are explicitly realized elsewhere in the text and those that are not specified, and resolve the former against the local text", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0061", "text": ". A closely related task is zero anaphora resolution, i.e., the extension of coreference resolution to pro-drop languages.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0062", "text": "Recognizing textual entailment\nGiven two text fragments, determine if one being true entails the other, entails the other's negation, or allows the other to be either true or false.\nTopic segmentation and recognition\nGiven a chunk of text, separate it into segments each of which is devoted to a topic, and identify the topic of the segment.\nArgument mining", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0063", "text": "Argument mining\nThe goal of argument mining is the automatic extraction and identification of argumentative structures from natural language text with the aid of computer programs. Such argumentative structures include the premise, conclusions, the argument scheme and the relationship between the main and subsidiary argument, or the main and counter-argument within discourse.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0064", "text": "Higher-level NLP applications\nAutomatic summarization (text summarization)\nProduce a readable summary of a chunk of text.  Often used to provide summaries of the text of a known type, such as research papers, articles in the financial section of a newspaper.\nGrammatical error correction", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0065", "text": "Grammatical error detection and correction involves a great band-width of problems on all levels of linguistic analysis (phonology/orthography, morphology, syntax, semantics, pragmatics). Grammatical error correction is impactful since it affects hundreds of millions of people that use or acquire English as a second language. It has thus been subject to a number of shared tasks since 2011", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0066", "text": ". As far as orthography, morphology, syntax and certain aspects of semantics are concerned, and due to the development of powerful neural language models such as GPT-2, this can now (2019) be considered a largely solved problem and is being marketed in various commercial applications.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0067", "text": "Logic translation\nTranslate a text from a natural language into formal logic.\nMachine translation (MT)\nAutomatically translate text from one human language to another.  This is one of the most difficult problems, and is a member of a class of problems colloquially termed \"AI-complete\", i.e. requiring all of the different types of knowledge that humans possess (grammar, semantics, facts about the real world, etc.) to solve properly.\nNatural language understanding (NLU)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0068", "text": "Convert chunks of text into more formal representations such as first-order logic structures that are easier for computer programs to manipulate. Natural language understanding involves the identification of the intended semantic from the multiple possible semantics which can be derived from a natural language expression which usually takes the form of organized notations of natural language concepts", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0069", "text": ". Introduction and creation of language metamodel and ontology are efficient however empirical solutions. An explicit formalization of natural language semantics without confusions with implicit assumptions such as closed-world assumption (CWA) vs. open-world assumption, or subjective Yes/No vs. objective True/False is expected for the construction of a basis of semantics formalization.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0070", "text": "Natural language generation (NLG):\nConvert information from computer databases or semantic intents into readable human language.\nBook generation", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0071", "text": "Not an NLP task proper but an extension of natural language generation and other NLP tasks is the creation of full-fledged books. The first machine-generated book was created by a rule-based system in 1984 (Racter, The policeman's beard is half-constructed). The first published work by a neural network was published in 2018, 1 the Road, marketed as a novel, contains sixty million words. Both these systems are basically elaborate but non-sensical (semantics-free) language models", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0072", "text": ". The first machine-generated science book was published in 2019 (Beta Writer, Lithium-Ion Batteries, Springer, Cham). Unlike Racter and 1 the Road, this is grounded on factual knowledge and based on text summarization.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0073", "text": "Document AI\nA Document AI platform sits on top of the NLP technology enabling users with no prior experience of artificial intelligence, machine learning or NLP to quickly train a computer to extract the specific data they need from different document types. NLP-powered Document AI enables non-technical teams to quickly access information hidden in documents, for example, lawyers, business analysts and accountants.\nDialogue management\nComputer systems intended to converse with a human.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0074", "text": "Question answering\nGiven a human-language question, determine its answer. Typical questions have a specific right answer (such as \"What is the capital of Canada?\"), but sometimes open-ended questions are also considered (such as \"What is the meaning of life?\").\nText-to-image generation\nGiven a description of an image, generate an image that matches the description.\nText-to-scene generation\nGiven a description of a scene, generate a 3D model of the scene.\nText-to-video", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0075", "text": "Text-to-video\nGiven a description of a video, generate a video that matches the description.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0076", "text": "General tendencies and (possible) future directions\nBased on long-standing trends in the field, it is possible to extrapolate future directions of NLP. As of 2020, three trends among the topics of the long-standing series of CoNLL Shared Tasks can be observed:", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0077", "text": "Interest on increasingly abstract, \"cognitive\" aspects of natural language (1999\u20132001: shallow parsing, 2002\u201303: named entity recognition, 2006\u201309/2017\u201318: dependency syntax, 2004\u201305/2008\u201309 semantic role labelling, 2011\u201312 coreference, 2015\u201316: discourse parsing, 2019: semantic parsing).", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0078", "text": "Increasing interest in multilinguality, and, potentially, multimodality (English since 1999; Spanish, Dutch since 2002; German since 2003; Bulgarian, Danish, Japanese, Portuguese, Slovenian, Swedish, Turkish since 2006; Basque, Catalan, Chinese, Greek, Hungarian, Italian, Turkish since 2007; Czech since 2009; Arabic since 2012; 2017: 40+ languages; 2018: 60+/100+ languages)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0079", "text": "Elimination of symbolic representations (rule-based over supervised towards weakly supervised methods, representation learning and end-to-end systems)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0080", "text": "Cognition\nMost higher-level NLP applications involve aspects that emulate intelligent behavior and apparent comprehension of natural language. More broadly speaking, the technical operationalization of increasingly advanced aspects of cognitive behavior represents one of the developmental trajectories of NLP (see trends among CoNLL shared tasks above).", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0081", "text": "Cognition refers to \"the mental action or process of acquiring knowledge and understanding through thought, experience, and the senses.\" Cognitive science is the interdisciplinary, scientific study of the mind and its processes. Cognitive linguistics is an interdisciplinary branch of linguistics, combining knowledge and research from both psychology and linguistics. Especially during the age of symbolic NLP, the area of computational linguistics maintained strong ties with cognitive studies.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0082", "text": "As an example, George Lakoff offers a methodology to build natural language processing (NLP) algorithms through the perspective of cognitive science, along with the findings of cognitive linguistics, with two defining aspects:", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0083", "text": "Apply the theory of conceptual metaphor, explained by Lakoff as \"the understanding of one idea, in terms of another\" which provides an idea of the intent of the author. For example, consider the English word big. When used in a comparison (\"That is a big tree\"), the author's intent is to imply that the tree is physically large relative to other trees or the authors experience.  When used metaphorically (\"Tomorrow is a big day\"), the author's intent to imply importance", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0084", "text": ".  The intent behind other usages, like in \"She is a big person\", will remain somewhat ambiguous to a person and a cognitive NLP algorithm alike without additional information.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0085", "text": "Assign relative measures of meaning to a word, phrase, sentence or piece of text based on the information presented before and after the piece of text being analyzed, e.g., by means of a probabilistic context-free grammar (PCFG). The mathematical equation for such algorithms is presented in US Patent 9269353:", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0086", "text": "R\n          M\n          M\n          (\n          t\n          o\n          k\n          e\n          \n            n\n            \n              N\n            \n          \n          )\n        \n        =\n        \n          P\n          M\n          M\n          (\n          t\n          o\n          k\n          e\n          \n            n\n            \n              N\n            \n          \n          )\n        \n        \u00d7\n        \n          \n            1", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0087", "text": "1\n            \n              2\n              d\n            \n          \n        \n        \n          (\n          \n            \n              \u2211\n              \n                i\n                =\n                \u2212\n                d\n              \n              \n                d\n              \n            \n            \n              (\n              (\n              P\n              M\n              M\n              (\n              t\n              o\n              k", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0088", "text": "t\n              o\n              k\n              e\n              \n                n\n                \n                  N\n                \n              \n              )\n            \n            \u00d7\n            \n              P\n              F\n              (\n              t\n              o\n              k\n              e\n              \n                n\n                \n                  N\n                  \u2212\n                  i\n                \n              \n              ,", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0089", "text": ",\n              t\n              o\n              k\n              e\n              \n                n\n                \n                  N\n                \n              \n              ,\n              t\n              o\n              k\n              e\n              \n                n\n                \n                  N\n                  +\n                  i\n                \n              \n              )\n              \n                )", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0090", "text": ")\n              \n                )\n                \n                  i\n                \n              \n            \n          \n          )\n        \n      \n    \n    {\\displaystyle {RMM(token_{N})}={PMM(token_{N})}\\times {\\frac {1}{2d}}\\left(\\sum _{i=-d}^{d}{((PMM(token_{N})}\\times {PF(token_{N-i},token_{N},token_{N+i}))_{i}}\\right)}", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0091", "text": "Where\nRMM is the relative measure of meaning\ntoken is any block of text, sentence, phrase or word\nN is the number of tokens being analyzed\nPMM is the probable measure of meaning based on a corpora\nd is the non zero location of the token along the sequence of N tokens\nPF is the probability function specific to a language", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0092", "text": "Ties with cognitive linguistics are part of the historical heritage of NLP, but they have been less frequently addressed since the statistical turn during the 1990s", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0093", "text": ". Nevertheless, approaches to develop cognitive models towards technically operationalizable frameworks have been pursued in the context of various frameworks, e.g., of cognitive grammar, functional grammar, construction grammar, computational psycholinguistics and cognitive neuroscience (e.g., ACT-R), however, with limited uptake in mainstream NLP (as measured by presence on major conferences of the ACL)", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0094", "text": ". More recently, ideas of cognitive NLP have been revived as an approach to achieve explainability, e.g., under the notion of \"cognitive AI\"", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0095", "text": ". Likewise, ideas of cognitive NLP are inherent to neural models multimodal NLP (although rarely made explicit) and developments in artificial intelligence, specifically tools and technologies using large language model approaches and new directions in artificial general intelligence based on the free energy principle by British neuroscientist and theoretician at University College London Karl J. Friston.", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Natural_language_processing_0096", "text": "See also\nReferences\nFurther reading\nExternal links\n Media related to Natural language processing at Wikimedia Commons", "title": "Natural language processing", "url": "https://en.wikipedia.org/wiki/Natural_language_processing"}, {"id": "rec_Word_embedding_0000", "text": "In natural language processing, a word embedding is a representation of a word. The embedding is used in text analysis. Typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from the vocabulary are mapped to vectors of real numbers.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0001", "text": "Methods to generate this mapping include neural networks, dimensionality reduction on the word co-occurrence matrix, probabilistic models, explainable knowledge base method, and explicit representation in terms of the context in which words appear.\nWord and phrase embeddings, when used as the underlying input representation, have been shown to boost the performance in NLP tasks such as syntactic parsing and sentiment analysis.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0002", "text": "Development and history of the approach", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0003", "text": "In distributional semantics, a quantitative methodological approach for understanding meaning in observed language, word embeddings or semantic feature space models have been used as a knowledge representation for some time. Such models aim to quantify and categorize semantic similarities between linguistic items based on their distributional properties in large samples of language data", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0004", "text": ".  The underlying idea that \"a word is characterized by the company it keeps\" was proposed in a 1957 article by John Rupert Firth, but also has roots in the contemporaneous work on search systems and in cognitive psychology.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0005", "text": "The notion of a semantic space with lexical items (words or multi-word terms) represented as vectors or embeddings is based on the computational challenges of capturing distributional characteristics and using them for practical application to measure similarity between words, phrases, or entire documents. The first generation of semantic space models is the vector space model for information retrieval", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0006", "text": ". Such vector space models for words and their distributional data implemented in their simplest form results in a very sparse vector space of high dimensionality (cf. curse of dimensionality). Reducing the number of dimensions using linear algebraic methods such as singular value decomposition then led to the introduction of latent semantic analysis in the late 1980s and the random indexing approach for collecting word co-occurrence contexts. In 2000, Bengio et al", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0007", "text": ". In 2000, Bengio et al. provided in a series of papers titled \"Neural probabilistic language models\" to reduce the high dimensionality of word representations in contexts by \"learning a distributed representation for words\".", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0008", "text": "A study published in NeurIPS (NIPS) 2002 introduced the use of both word and document embeddings applying the method of kernel CCA to bilingual (and multi-lingual) corpora, also providing an early example of self-supervised learning of word embeddings.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0009", "text": "Word embeddings come in two different styles, one in which words are expressed as vectors of co-occurring words, and another in which words are expressed as vectors of linguistic contexts in which the words occur; these different styles are studied in Lavelli et al., 2004. Roweis and Saul published in Science how to use \"locally linear embedding\" (LLE) to discover representations of high dimensional data structures", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0010", "text": ". Most new word embedding techniques after about 2005 rely on a neural network architecture instead of more probabilistic and algebraic models, after foundational work done by Yoshua Bengio and colleagues.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0011", "text": "The approach has been adopted by many research groups after theoretical advances in 2010 had been made on the quality of vectors and the training speed of the model, as well as after hardware advances allowed for a broader parameter space to be explored profitably. In 2013, a team at Google led by Tomas Mikolov created word2vec, a word embedding toolkit that can train vector space models faster than previous approaches", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0012", "text": ". The word2vec approach has been widely used in experimentation and was instrumental in raising interest for word embeddings as a technology, moving the research strand out of specialised research into broader experimentation and eventually paving the way for practical application.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0013", "text": "Polysemy and homonymy", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0014", "text": "Historically, one of the main limitations of static word embeddings or word vector space models is that words with multiple meanings are conflated into a single representation (a single vector in the semantic space). In other words, polysemy and homonymy are not handled properly. For example, in the sentence \"The club I tried yesterday was great!\", it is not clear if the term club is related to the word sense of a club sandwich, clubhouse, golf club, or any other sense that club might have", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0015", "text": ". The necessity to accommodate multiple meanings per word in different vectors (multi-sense embeddings) is the motivation for several contributions in NLP to split single-sense embeddings into multi-sense ones.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0016", "text": "Most approaches that produce multi-sense embeddings can be divided into two main categories for their word sense representation, i.e., unsupervised and knowledge-based. Based on word2vec skip-gram, Multi-Sense Skip-Gram (MSSG) performs word-sense discrimination and embedding simultaneously, improving its training time, while assuming a specific number of senses for each word. In the Non-Parametric Multi-Sense Skip-Gram (NP-MSSG) this number can vary depending on each word", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0017", "text": ". Combining the prior knowledge of lexical databases (e.g., WordNet, ConceptNet, BabelNet), word embeddings and word sense disambiguation, Most Suitable Sense Annotation (MSSA) labels word-senses through an unsupervised and knowledge-based approach, considering a word's context in a pre-defined sliding window. Once the words are disambiguated, they can be used in a standard word embeddings technique, so multi-sense embeddings are produced", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0018", "text": ". MSSA architecture allows the disambiguation and annotation process to be performed recurrently in a self-improving manner.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0019", "text": "The use of multi-sense embeddings is known to improve performance in several NLP tasks, such as part-of-speech tagging, semantic relation identification, semantic relatedness, named entity recognition and sentiment analysis.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0020", "text": "As of the late 2010s, contextually-meaningful embeddings such as ELMo and BERT have been developed. Unlike static word embeddings, these embeddings are at the token-level, in that each occurrence of a word has its own embedding. These embeddings better reflect the multi-sense nature of words, because occurrences of a word in similar contexts are situated in similar regions of BERT's embedding space.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0021", "text": "For biological sequences: BioVectors", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0022", "text": "Word embeddings for n-grams in biological sequences (e.g. DNA, RNA, and Proteins) for bioinformatics applications have been proposed by Asgari and Mofrad. Named bio-vectors (BioVec) to refer to biological sequences in general with protein-vectors (ProtVec) for proteins (amino-acid sequences) and gene-vectors (GeneVec) for gene sequences, this representation can be widely used in applications of deep learning in proteomics and genomics", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0023", "text": ". The results presented by Asgari and Mofrad suggest that BioVectors can characterize biological sequences in terms of biochemical and biophysical interpretations of the underlying patterns.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0024", "text": "Game design\nWord embeddings with applications in game design have been proposed by Rabii and Cook as a way to discover emergent gameplay using logs of gameplay data. The process requires transcribing actions that occur during a game within a formal language and then using the resulting text to create word embeddings. The results presented by Rabii and Cook suggest that the resulting vectors can capture expert knowledge about games like chess that are not explicitly stated in the game's rules.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0025", "text": "Sentence embeddings\nThe idea has been extended to embeddings of entire sentences or even documents, e.g. in the form of the thought vectors concept. In 2015, some researchers suggested \"skip-thought vectors\" as a means to improve the quality of machine translation. A more recent and popular approach for representing sentences is Sentence-BERT, or SentenceTransformers, which modifies pre-trained BERT with the use of siamese and triplet network structures.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0026", "text": "Software\nSoftware for training and using word embeddings includes Tom\u00e1\u0161 Mikolov's Word2vec, Stanford University's GloVe, GN-GloVe, Flair embeddings, AllenNLP's ELMo, BERT, fastText, Gensim, Indra, and Deeplearning4j. Principal Component Analysis (PCA), Uniform manifold approximation and projection (UMAP), and T-Distributed Stochastic Neighbour Embedding (t-SNE) are used to reduce the dimensionality of word vector spaces and visualize word embeddings and clusters.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0027", "text": "Examples of application\nFor instance, the fastText is also used to calculate word embeddings for text corpora in Sketch Engine that are available online.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0028", "text": "Ethical implications", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0029", "text": "Word embeddings may contain the biases and stereotypes contained in the trained dataset, as Bolukbasi et al", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0030", "text": ". points out in the 2016 paper \"Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings\" that a publicly available (and popular) word2vec embedding trained on Google News texts (a commonly used data corpus), which consists of text written by professional journalists, still shows disproportionate word associations reflecting gender and racial biases when extracting word analogies", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0031", "text": ". For example, one of the analogies generated using the aforementioned word embedding is \"man is to computer programmer as woman is to homemaker\".", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0032", "text": "Research done by Jieyu Zhou et al. shows that the applications of these trained word embeddings without careful oversight likely perpetuates existing bias in society, which is introduced through unaltered training data. Furthermore, word embeddings can even amplify these biases.", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word_embedding_0033", "text": "See also\nEmbedding (machine learning)\nBrown clustering\nDistributional\u2013relational database\n\n\n== References ==", "title": "Word embedding", "url": "https://en.wikipedia.org/wiki/Word_embedding"}, {"id": "rec_Word2vec_0000", "text": "Word2vec is a technique in natural language processing for obtaining vector representations of words. These vectors capture information about the meaning of the word based on the surrounding words. The word2vec algorithm estimates these representations by modeling text in a large corpus. Once trained, such a model can detect synonymous words or suggest additional words for a partial sentence", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0001", "text": ". Word2vec was developed by Tom\u00e1\u0161 Mikolov, Kai Chen, Greg Corrado, Ilya Sutskever and Jeff Dean at Google, and published in 2013.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0002", "text": "Word2vec represents a word as a high-dimension vector of numbers which capture relationships between words. In particular, words which appear in similar contexts are mapped to vectors which are nearby as measured by cosine similarity. This indicates the level of semantic similarity between the words, so for example the vectors for walk and ran are nearby, as are those for \"but\" and \"however\", and \"Berlin\" and \"Germany\".", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0003", "text": "Approach\nWord2vec is a group of related models that are used to produce word embeddings. These models are shallow, two-layer neural networks that are trained to reconstruct linguistic contexts of words. Word2vec takes as its input a large corpus of text and produces a mapping of the set of words to a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a vector in the space.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0004", "text": "Word2vec can use either of two model architectures to produce these distributed representations of words: continuous bag of words (CBOW) or continuously sliding skip-gram. In both architectures, word2vec considers both individual words and a sliding context window as it iterates over the corpus.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0005", "text": "The CBOW can be viewed as a 'fill in the blank' task, where the word embedding represents the way the word influences the relative probabilities of other words in the context window. Words which are semantically similar should influence these probabilities in similar ways, because semantically similar words should be used in similar contexts. The order of context words does not influence prediction (bag of words assumption).", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0006", "text": "In the continuous skip-gram architecture, the model uses the current word to predict the surrounding window of context words. The skip-gram architecture weighs nearby context words more heavily than more distant context words. According to the authors' note, CBOW is faster while skip-gram does a better job for infrequent words.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0007", "text": "After the model is trained, the learned word embeddings are positioned in the vector space such that words that share common contexts in the corpus \u2014 that is, words that are semantically and syntactically similar \u2014 are located close to one another in the space. More dissimilar words are located farther from one another in the space.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0008", "text": "Mathematical details\nThis section is based on expositions.\nA corpus is a sequence of words. Both CBOW and skip-gram are methods to learn one vector per word appearing in the corpus.\nLet \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n (\"vocabulary\") be the set of all words appearing in the corpus \n  \n    \n      \n        C\n      \n    \n    {\\displaystyle C}\n  \n. Our goal is to learn one vector \n  \n    \n      \n        \n          v\n          \n            w", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0009", "text": "w\n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            d\n          \n        \n      \n    \n    {\\displaystyle v_{w}\\in \\mathbb {R} ^{d}}\n  \n for each word \n  \n    \n      \n        w\n        \u2208\n        V\n      \n    \n    {\\displaystyle w\\in V}\n  \n.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0010", "text": "{\\displaystyle w\\in V}\n  \n.\nThe idea of skip-gram is that the vector of a word should be close to the vector of each of its neighbors. The idea of CBOW is that the vector-sum of a word's neighbors should be close to the vector of the word.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0011", "text": "Continuous bag-of-words (CBOW)\nThe idea of CBOW is to represent each word with a vector, such that it is possible to predict a word using the sum of the vectors of its neighbors. Specifically, for each word \n  \n    \n      \n        \n          w\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle w_{i}}", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0012", "text": "{\\displaystyle w_{i}}\n  \n in the corpus, the one-hot encoding of the word is used as the input to the neural network. The output of the neural network is a probability distribution over the dictionary, representing a prediction of individual words in the neighborhood of \n  \n    \n      \n        \n          w\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle w_{i}}\n  \n. The objective of training is to maximize \n  \n    \n      \n        \n          \u2211", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0013", "text": "\u2211\n          \n            i\n          \n        \n        ln\n        \u2061\n        Pr\n        (\n        \n          w\n          \n            i\n          \n        \n        \u2223\n        \n          w\n          \n            i\n            +\n            j\n          \n        \n        :\n        j\n        \u2208\n        N\n        )\n      \n    \n    {\\displaystyle \\sum _{i}\\ln \\Pr(w_{i}\\mid w_{i+j}\\colon j\\in N)}\n  \n where \n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0014", "text": "N\n      \n    \n    {\\displaystyle N}\n  \n is a set of (non-zero) indices representing the relative locations of nearby words considered to be in \n  \n    \n      \n        \n          w\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle w_{i}}\n  \n's neighborhood.\nFor example, if we want each word in the corpus to be predicted by every other word in a small span of 4 words. The set of relative indexes of neighbor words will be: \n  \n    \n      \n        N\n        =", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0015", "text": "N\n        =\n        {\n        \u2212\n        2\n        ,\n        \u2212\n        1\n        ,\n        +\n        1\n        ,\n        +\n        2\n        }\n      \n    \n    {\\displaystyle N=\\{-2,-1,+1,+2\\}}\n  \n, and the objective is to maximize\n  \n    \n      \n        \n          \u2211\n          \n            i\n          \n        \n        ln\n        \u2061\n        Pr\n        (\n        \n          w\n          \n            i\n          \n        \n        \u2223\n        \n          w\n          \n            i", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0016", "text": "w\n          \n            i\n            \u2212\n            2\n          \n        \n        ,\n        \n          w\n          \n            i\n            \u2212\n            1\n          \n        \n        ,\n        \n          w\n          \n            i\n            +\n            1\n          \n        \n        ,\n        \n          w\n          \n            i\n            +\n            2\n          \n        \n        )", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0017", "text": ")\n      \n    \n    {\\displaystyle \\sum _{i}\\ln \\Pr(w_{i}\\mid w_{i-2},w_{i-1},w_{i+1},w_{i+2})}\n  \n.\nIn standard bag-of-words, a word's context is represented by a word-count (aka a word histogram) of its neighboring words. For example, the \"sat\" in \"the cat sat on the mat\" is represented as {\"the\": 2, \"cat\": 1, \"on\": 1}. Note that the last word \"mat\" is not used to represent \"sat\", because it is outside the neighborhood \n  \n    \n      \n        N\n        =\n        {", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0018", "text": "N\n        =\n        {\n        \u2212\n        2\n        ,\n        \u2212\n        1\n        ,\n        +\n        1\n        ,\n        +\n        2\n        }\n      \n    \n    {\\displaystyle N=\\{-2,-1,+1,+2\\}}\n  \n.\nIn continuous bag-of-words, the histogram is multiplied by a matrix \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n to obtain a continuous representation of the word's context. The matrix \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0019", "text": "V\n      \n    \n    {\\displaystyle V}\n  \n is also called a dictionary. Its columns are the word vectors. It has \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n columns, where \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n is the size of the dictionary. Let \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n be the length of each word vector. We have \n  \n    \n      \n        V\n        \u2208\n        \n          \n            R", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0020", "text": "R\n          \n          \n            d\n            \u00d7\n            D\n          \n        \n      \n    \n    {\\displaystyle V\\in \\mathbb {R} ^{d\\times D}}\n  \n.\nFor example, multiplying the word histogram {\"the\": 2, \"cat\": 1, \"on\": 1} with \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n, we obtain \n  \n    \n      \n        2\n        \n          v\n          \n            the\n          \n        \n        +\n        \n          v\n          \n            cat", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0021", "text": "v\n          \n            cat\n          \n        \n        +\n        \n          v\n          \n            on\n          \n        \n      \n    \n    {\\displaystyle 2v_{\\text{the}}+v_{\\text{cat}}+v_{\\text{on}}}\n  \n.\nThis is then multiplied with another matrix \n  \n    \n      \n        \n          V\n          \u2032\n        \n      \n    \n    {\\displaystyle V'}\n  \n of shape \n  \n    \n      \n        \n          \n            R\n          \n          \n            D\n            \u00d7\n            d", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0022", "text": "\u00d7\n            d\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {R} ^{D\\times d}}\n  \n. Each row of it is a word vector \n  \n    \n      \n        \n          v\n          \u2032\n        \n      \n    \n    {\\displaystyle v'}\n  \n. This results in a vector of length \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n, one entry per dictionary entry. Then, apply the softmax to obtain a probability distribution over the dictionary.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0023", "text": "This system can be visualized as a neural network, similar in spirit to an autoencoder, of architecture linear-linear-softmax, as depicted in the diagram. The system is trained by gradient descent to minimize the cross-entropy loss. \nIn full formula, the cross-entropy loss is:\n  \n    \n      \n        \u2212\n        \n          \u2211\n          \n            i\n          \n        \n        ln\n        \u2061\n        \n          \n            \n              e\n              \n                \n                  v", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0024", "text": "v\n                  \n                    \n                      w\n                      \n                        i\n                      \n                    \n                  \n                  \u2032\n                \n                \u22c5\n                (\n                \n                  \u2211\n                  \n                    j\n                    \u2208\n                    N\n                  \n                \n                \n                  v", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0025", "text": "v\n                  \n                    \n                      w\n                      \n                        j\n                        +\n                        i\n                      \n                    \n                  \n                \n                )\n              \n            \n            \n              \n                \u2211\n                \n                  \n                    w\n                    \u2032", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0026", "text": "e\n                \n                  \n                    v\n                    \n                      \n                        w\n                        \u2032\n                      \n                    \n                    \u2032\n                  \n                  \u22c5\n                  (\n                  \n                    \u2211\n                    \n                      j\n                      \u2208\n                      N", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0027", "text": "N\n                    \n                  \n                  \n                    v\n                    \n                      \n                        w\n                        \n                          j\n                          +\n                          i\n                        \n                      \n                    \n                  \n                  )", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0028", "text": "{\\displaystyle -\\sum _{i}\\ln {\\frac {e^{v_{w_{i}}'\\cdot (\\sum _{j\\in N}v_{w_{j+i}})}}{\\sum _{w'}e^{v_{w'}'\\cdot (\\sum _{j\\in N}v_{w_{j+i}})}}}}\n  \nwhere the outer summation \n  \n    \n      \n        \n          \u2211\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\sum _{i}}\n  \n is over the words in a corpus, the quantity \n  \n    \n      \n        \n          \u2211\n          \n            j\n            \u2208\n            N", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0029", "text": "\u2208\n            N\n          \n        \n        \n          v\n          \n            \n              w\n              \n                j\n                +\n                i\n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\sum _{j\\in N}v_{w_{j+i}}}\n  \n is the sum of a word's neighbors' vectors, etc.\nOnce such a system is trained, we have two trained matrices \n  \n    \n      \n        V\n        ,\n        \n          V\n          \u2032", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0030", "text": "V\n          \u2032\n        \n      \n    \n    {\\displaystyle V,V'}\n  \n. Either the column vectors of \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n or the row vectors of \n  \n    \n      \n        \n          V\n          \u2032\n        \n      \n    \n    {\\displaystyle V'}\n  \n can serve as the dictionary. For example, the word \"sat\" can be represented as either the \"sat\"-th column of \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n or the \"sat\"-th row of", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0031", "text": "or the \"sat\"-th row of \n  \n    \n      \n        \n          V\n          \u2032\n        \n      \n    \n    {\\displaystyle V'}\n  \n. It is also possible to simply define \n  \n    \n      \n        \n          V\n          \u2032\n        \n        =\n        \n          V\n          \n            \u22a4\n          \n        \n      \n    \n    {\\displaystyle V'=V^{\\top }}\n  \n, in which case there would no longer be a choice.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0032", "text": "Skip-gram\nThe idea of skip-gram is to represent each word with a vector, such that it is possible to predict the vectors of its neighbors using the vector of a word.\nThe architecture is still linear-linear-softmax, the same as CBOW, but the input and the output are switched. Specifically, for each word \n  \n    \n      \n        \n          w\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle w_{i}}", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0033", "text": "{\\displaystyle w_{i}}\n  \n in the corpus, the one-hot encoding of the word is used as the input to the neural network. The output of the neural network is a probability distribution over the dictionary, representing a prediction of individual words in the neighborhood of \n  \n    \n      \n        \n          w\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle w_{i}}\n  \n. The objective of training is to maximize \n  \n    \n      \n        \n          \u2211", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0034", "text": "\u2211\n          \n            i\n          \n        \n        \n          \u2211\n          \n            j\n            \u2208\n            N\n          \n        \n        ln\n        \u2061\n        Pr\n        (\n        \n          w\n          \n            j\n            +\n            i\n          \n        \n        \u2223\n        \n          w\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle \\sum _{i}\\sum _{j\\in N}\\ln \\Pr(w_{j+i}\\mid w_{i})}\n  \n.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0035", "text": ". \nIn full formula, the loss function is\n  \n    \n      \n        \u2212\n        \n          \u2211\n          \n            i\n          \n        \n        \n          \u2211\n          \n            j\n            \u2208\n            N\n          \n        \n        ln\n        \u2061\n        \n          \n            \n              e\n              \n                \n                  v\n                  \n                    \n                      w\n                      \n                        j\n                        +", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0036", "text": "+\n                        i\n                      \n                    \n                  \n                  \u2032\n                \n                \u22c5\n                \n                  v\n                  \n                    \n                      w\n                      \n                        i\n                      \n                    \n                  \n                \n              \n            \n            \n              \n                \u2211", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0037", "text": "\u2211\n                \n                  \n                    w\n                    \u2032\n                  \n                \n              \n              \n                e\n                \n                  \n                    v\n                    \n                      \n                        w\n                        \u2032\n                      \n                    \n                    \u2032\n                  \n                  \u22c5\n                  \n                    v", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0038", "text": "v\n                    \n                      \n                        w\n                        \n                          i\n                        \n                      \n                    \n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle -\\sum _{i}\\sum _{j\\in N}\\ln {\\frac {e^{v_{w_{j+i}}'\\cdot v_{w_{i}}}}{\\sum _{w'}e^{v_{w'}'\\cdot v_{w_{i}}}}}}", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0039", "text": "Same as CBOW, once such a system is trained, we have two trained matrices \n  \n    \n      \n        V\n        ,\n        \n          V\n          \u2032\n        \n      \n    \n    {\\displaystyle V,V'}\n  \n. Either the column vectors of \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n or the row vectors of \n  \n    \n      \n        \n          V\n          \u2032\n        \n      \n    \n    {\\displaystyle V'}\n  \n can serve as the dictionary. It is also possible to simply define", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0040", "text": "V\n          \u2032\n        \n        =\n        \n          V\n          \n            \u22a4\n          \n        \n      \n    \n    {\\displaystyle V'=V^{\\top }}\n  \n, in which case there would no longer be a choice. \nEssentially, skip-gram and CBOW are exactly the same in architecture. They only differ in the objective function during training.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0041", "text": "History\nDuring the 1980s, there were some early attempts at using neural networks to represent words and concepts as vectors.\nIn 2010, Tom\u00e1\u0161 Mikolov (then at Brno University of Technology) with co-authors applied a simple recurrent neural network with a single hidden layer to language modelling.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0042", "text": "Word2vec was created, patented, and published in 2013 by a team of researchers led by Mikolov at Google over two papers. The original paper was rejected by reviewers for ICLR conference 2013. It also took months for the code to be approved for open-sourcing. Other researchers helped analyse and explain the algorithm.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0043", "text": "Embedding vectors created using the Word2vec algorithm have some advantages compared to earlier algorithms such as those using n-grams and latent semantic analysis. GloVe was developed by a team at Stanford specifically as a competitor, and the original paper noted multiple improvements of GloVe over word2vec. Mikolov argued that the comparison was unfair as GloVe was trained on more data, and that the fastText project showed that word2vec is superior when trained on the same data.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0044", "text": "As of 2022, the straight Word2vec approach was described as \"dated\". Transformer-based models, such as ELMo and BERT, which add multiple neural-network attention layers on top of a word embedding model similar to Word2vec, have come to be regarded as the state of the art in natural language processing.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0045", "text": "Parameterization\nResults of word2vec training can be sensitive to parametrization. The following are some important parameters in word2vec training.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0046", "text": "Training algorithm", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0047", "text": "A Word2vec model can be trained with hierarchical softmax and/or negative sampling. To approximate the conditional log-likelihood a model seeks to maximize, the hierarchical softmax method uses a Huffman tree to reduce calculation. The negative sampling method, on the other hand, approaches the maximization problem by minimizing the log-likelihood of sampled negative instances", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0048", "text": ". According to the authors, hierarchical softmax works better for infrequent words while negative sampling works better for frequent words and better with low dimensional vectors. As training epochs increase, hierarchical softmax stops being useful.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0049", "text": "Sub-sampling\nHigh-frequency and low-frequency words often provide little information. Words with a frequency above a certain threshold, or below a certain threshold, may be subsampled or removed to speed up training.\n\nDimensionality\nQuality of word embedding increases with higher dimensionality. But after reaching some point, marginal gain diminishes. Typically, the dimensionality of the vectors is set to be between 100 and 1,000.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0050", "text": "Context window\nThe size of the context window determines how many words before and after a given word are included as context words of the given word. According to the authors' note, the recommended value is 10 for skip-gram and 5 for CBOW.\n\nExtensions\nThere are a variety of extensions to word2vec.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0051", "text": "doc2vec\ndoc2vec, generates distributed representations of variable-length pieces of texts, such as sentences, paragraphs, or entire documents. doc2vec has been implemented in the C, Python and Java/Scala tools (see below), with the Java and Python versions also supporting inference of document embeddings on new, unseen documents.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0052", "text": "doc2vec estimates the distributed representations of documents much like how word2vec estimates representations of words: doc2vec utilizes either of two model architectures, both of which are allegories to the architectures used in word2vec. The first, Distributed Memory Model of Paragraph Vectors (PV-DM), is identical to CBOW other than it also provides a unique document identifier as a piece of additional context", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0053", "text": ". The second architecture, Distributed Bag of Words version of Paragraph Vector (PV-DBOW), is identical to the skip-gram model except that it attempts to predict the window of surrounding context words from the paragraph identifier instead of the current word.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0054", "text": "doc2vec also has the ability to capture the semantic 'meanings' for additional pieces of  'context' around words; doc2vec can estimate the semantic embeddings for speakers or speaker attributes, groups, and periods of time. For example, doc2vec has been used to estimate the political positions of political parties in various Congresses and Parliaments in the U.S. and U.K., respectively, and various governmental institutions.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0055", "text": "top2vec", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0056", "text": "Another extension of word2vec is top2vec, which leverages both document and word embeddings to estimate distributed representations of topics. top2vec takes document embeddings learned from a doc2vec model and reduces them into a lower dimension (typically using UMAP). The space of documents is then scanned using HDBSCAN, and clusters of similar documents are found. Next, the centroid of documents identified in a cluster is considered to be that cluster's topic vector", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0057", "text": ". Finally, top2vec searches the semantic space for word embeddings located near to the topic vector to ascertain the 'meaning' of the topic. The word with embeddings most similar to the topic vector might be assigned as the topic's title, whereas far away word embeddings may be considered unrelated.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0058", "text": "As opposed to other topic models such as LDA, top2vec provides canonical 'distance' metrics between two topics, or between a topic and another embeddings (word, document, or otherwise). Together with results from HDBSCAN, users can generate topic hierarchies, or groups of related topics and subtopics.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0059", "text": "Furthermore, a user can use the results of top2vec to infer the topics of out-of-sample documents. After inferring the embedding for a new document, must only search the space of topics for the closest topic vector.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0060", "text": "BioVectors", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0061", "text": "An extension of word vectors for n-grams in biological sequences (e.g. DNA, RNA, and proteins) for bioinformatics applications has been proposed by Asgari and Mofrad. Named bio-vectors (BioVec) to refer to biological sequences in general with protein-vectors (ProtVec) for proteins (amino-acid sequences) and gene-vectors (GeneVec) for gene sequences, this representation can be widely used in applications of machine learning in proteomics and genomics", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0062", "text": ". The results suggest that BioVectors can characterize biological sequences in terms of biochemical and biophysical interpretations of the underlying patterns. A similar variant, dna2vec, has shown that there is correlation between Needleman\u2013Wunsch similarity score and cosine similarity of dna2vec word vectors.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0063", "text": "Radiology and intelligent word embeddings (IWE)", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0064", "text": "An extension of word vectors for creating a dense vector representation of unstructured radiology reports has been proposed by Banerjee et al. One of the biggest challenges with Word2vec is how to handle unknown or out-of-vocabulary (OOV) words and morphologically similar words. If the Word2vec model has not encountered a particular word before, it will be forced to use a random vector, which is generally far from its ideal representation", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0065", "text": ". This can particularly be an issue in domains like medicine where synonyms and related words can be used depending on the preferred style of radiologist, and words may have been used infrequently in a large corpus.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0066", "text": "IWE combines Word2vec with a semantic dictionary mapping technique to tackle the major challenges of information extraction from clinical texts, which include ambiguity of free text narrative style, lexical variations, use of ungrammatical and telegraphic phases, arbitrary ordering of words, and frequent appearance of abbreviations and acronyms", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0067", "text": ".  Of particular interest, the IWE model (trained on the one institutional dataset) successfully translated to a different institutional dataset which demonstrates good generalizability of the approach across institutions.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0068", "text": "Analysis\nThe reasons for successful word embedding learning in the word2vec framework are poorly understood. Goldberg and Levy point out that the word2vec objective function causes words that occur in similar contexts to have similar embeddings (as measured by cosine similarity) and note that this is in line with J. R. Firth's distributional hypothesis. However, they note that this explanation is \"very hand-wavy\" and argue that a more formal explanation would be preferable.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0069", "text": "Levy et al. (2015) show that much of the superior performance of word2vec or similar embeddings in downstream tasks is not a result of the models per se, but of the choice of specific hyperparameters. Transferring these hyperparameters to more 'traditional' approaches yields similar performances in downstream tasks. Arora et al", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0070", "text": ". Arora et al. (2016) explain word2vec and related algorithms as performing inference for a simple generative model for text, which involves a random walk generation process based upon loglinear topic model. They use this to explain some properties of word embeddings, including their use to solve analogies.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0071", "text": "Preservation of semantic and syntactic relationships", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0072", "text": "The word embedding approach is able to capture multiple different degrees of similarity between words. Mikolov et al. (2013) found that semantic and syntactic patterns can be reproduced using vector arithmetic", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0073", "text": ". Patterns such as \"Man is to Woman as Brother is to Sister\" can be generated through algebraic operations on the vector representations of these words such that the vector representation of \"Brother\" - \"Man\" + \"Woman\" produces a result which is closest to the vector representation of \"Sister\" in the model. Such relationships can be generated for a range of semantic relations (such as Country\u2013Capital) as well as syntactic relations (e.g. present tense\u2013past tense).", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0074", "text": "This facet of word2vec has been exploited in a variety of other contexts. For example, word2vec has been used to map a vector space of words in one language to a vector space constructed from another language. Relationships between translated words in both spaces can be used to assist with machine translation of new words.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0075", "text": "Assessing the quality of a model", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0076", "text": "Mikolov et al. (2013) developed an approach to assessing the quality of a word2vec model which draws on the semantic and syntactic patterns discussed above. They developed a set of 8,869 semantic relations and 10,675 syntactic relations which they use as a benchmark to test the accuracy of a model", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0077", "text": ". When assessing the quality of a vector model, a user may draw on this accuracy test which is implemented in word2vec, or develop their own test set which is meaningful to the corpora which make up the model. This approach offers a more challenging test than simply arguing that the words most similar to a given test word are intuitively plausible.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0078", "text": "Parameters and model quality", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0079", "text": "The use of different model parameters and different corpus sizes can greatly affect the quality of a word2vec model. Accuracy can be improved in a number of ways, including the choice of model architecture (CBOW or Skip-Gram), increasing the training data set, increasing the number of vector dimensions, and increasing the window size of words considered by the algorithm", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0080", "text": ". Each of these improvements comes with the cost of increased computational complexity and therefore increased model generation time.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0081", "text": "In models using large corpora and a high number of dimensions, the skip-gram model yields the highest overall accuracy, and consistently produces the highest accuracy on semantic relationships, as well as yielding the highest syntactic accuracy in most cases. However, the CBOW is less computationally expensive and yields similar accuracy results.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0082", "text": "Overall, accuracy increases with the number of words used and the number of dimensions. Mikolov et al. report that doubling the amount of training data results in an increase in computational complexity equivalent to doubling the number of vector dimensions.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0083", "text": "Altszyler and coauthors (2017) studied Word2vec performance in two semantic tests for different corpus size. They found that Word2vec has a steep learning curve, outperforming another word-embedding technique, latent semantic analysis (LSA), when it is trained with medium to large corpus size (more than 10 million words). However, with a small training corpus, LSA showed better performance. Additionally they show that the best parameter setting depends on the task and the training corpus", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0084", "text": ". Nevertheless, for skip-gram models trained in medium size corpora, with 50 dimensions, a window size of 15 and 10 negative samples seems to be a good parameter setting.", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Word2vec_0085", "text": "See also\nReferences\nExternal links\nWikipedia2Vec[1] (introduction)\n\nImplementations\nC\nC#\nPython (Spark)\nPython (TensorFlow)\nPython (Gensim)\nJava/Scala\nR", "title": "Word2vec", "url": "https://en.wikipedia.org/wiki/Word2vec"}, {"id": "rec_Sentiment_analysis_0000", "text": "Sentiment analysis (also known as opinion mining or emotion AI) is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0001", "text": ". Sentiment analysis is widely applied to voice of the customer materials such as reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine. With the rise of deep language models, such as RoBERTa, more difficult data domains can be analyzed, e.g., news texts where authors typically express their opinion/sentiment less explicitly.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0002", "text": "Types\nA basic task in sentiment analysis is classifying the polarity of a given text at the document, sentence, or feature/aspect level\u2014whether the expressed opinion in a document, a sentence or an entity feature/aspect is positive, negative, or neutral. Advanced, \"beyond polarity\" sentiment classification looks, for instance, at emotional states such as enjoyment, anger, disgust, sadness, fear, and surprise.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0003", "text": "Precursors to sentimental analysis include the General Inquirer, which provided hints toward quantifying patterns in text and, separately, psychological research that examined a person's psychological state based on analysis of their verbal behavior.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0004", "text": "Subsequently, the method described in a patent by Volcani and Fogel, looked specifically at sentiment and identified individual words and phrases in text with respect to different emotional scales. A current system based on their work, called EffectCheck, presents synonyms that can be used to increase or decrease the level of evoked emotion in each scale.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0005", "text": "Many other subsequent efforts were less sophisticated, using a mere polar view of sentiment, from positive to negative, such as work by Turney, and Pang who applied different methods for detecting the polarity of product reviews and movie reviews respectively. This work is at the document level", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0006", "text": ". This work is at the document level. One can also classify a document's polarity on a multi-way scale, which was attempted by Pang and Snyder among others: Pang and Lee expanded the basic task of classifying a movie review as either positive or negative to predict star ratings on either a 3- or a 4-star scale, while Snyder performed an in-depth analysis of restaurant reviews, predicting ratings for various aspects of the given restaurant, such as the food and atmosphere (on a five-star scale).", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0007", "text": "First steps to bringing together various approaches\u2014learning, lexical, knowledge-based, etc.\u2014were taken in the 2004 AAAI Spring Symposium where linguists, computer scientists, and other interested researchers first aligned interests and proposed shared tasks and benchmark data sets for the systematic computational research on affect, appeal, subjectivity, and sentiment in text.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0008", "text": "Even though in most statistical classification methods, the neutral class is ignored under the assumption that neutral texts lie near the boundary of the binary classifier, several researchers suggest that, as in every polarity problem, three categories must be identified. Moreover, it can be proven that specific classifiers such as the Max Entropy and SVMs can benefit from the introduction of a neutral class and improve the overall accuracy of the classification", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0009", "text": ". There are in principle two ways for operating with a neutral class. Either, the algorithm proceeds by first identifying the neutral language, filtering it out and then assessing the rest in terms of positive and negative sentiments, or it builds a three-way classification in one step. This second approach often involves estimating a probability distribution over all categories (e.g. naive Bayes classifiers as implemented by the NLTK)", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0010", "text": ". Whether and how to use a neutral class depends on the nature of the data: if the data is clearly clustered into neutral, negative and positive language, it makes sense to filter the neutral language out and focus on the polarity between positive and negative sentiments. If, in contrast, the data are mostly neutral with small deviations towards positive and negative affect, this strategy would make it harder to clearly distinguish between the two poles.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0011", "text": "A different method for determining sentiment is the use of a scaling system whereby words commonly associated with having a negative, neutral, or positive sentiment  are given an associated number on a \u221210 to +10 scale (most negative up to most positive) or simply from 0 to a positive upper limit such as +4. This makes it possible to adjust the sentiment of a given term relative to its environment (usually on the level of the sentence)", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0012", "text": ". When a piece of unstructured text is analyzed using natural language processing, each concept in the specified environment is given a score based on the way sentiment words relate to the concept and its associated score. This allows movement to a more sophisticated understanding of sentiment, because it is now possible to adjust the sentiment value of a concept relative to modifications that may surround it", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0013", "text": ". Words, for example, that intensify, relax or negate the sentiment expressed by the concept can affect its score. Alternatively, texts can be given a positive and negative sentiment strength score if the goal is to determine the sentiment in a text rather than the overall polarity and strength of the text.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0014", "text": "There are various other types of sentiment analysis, such as aspect-based sentiment analysis, grading sentiment analysis (positive, negative, neutral), multilingual sentiment analysis and detection of emotions.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0015", "text": "Subjectivity/objectivity identification", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0016", "text": "This task is commonly defined as classifying a given text (usually a sentence) into one of two classes: objective or subjective. This problem can sometimes be more difficult than polarity classification. The subjectivity of words and phrases may depend on their context and an objective document may contain subjective sentences (e.g., a news article quoting people's opinions). Moreover, as mentioned by Su, results are largely dependent on the definition of subjectivity used when annotating texts", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0017", "text": ". However, Pang showed that removing objective sentences from a document before classifying its polarity helped improve performance.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0018", "text": "Subjective and objective identification, emerging subtasks of sentiment analysis to use syntactic, semantic features, and machine learning knowledge to identify if a sentence or document contains facts or opinions. Awareness of recognizing factual and opinions is not recent, having possibly first presented by Carbonell at Yale University in 1979.\nThe term objective refers to the incident carrying factual information.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0019", "text": "Example of an objective sentence: 'To be elected president of the United States, a candidate must be at least thirty-five years of age.'", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0020", "text": "The term subjective describes the incident contains non-factual information in various forms, such as personal opinions, judgment, and predictions, also known as 'private states'. In the example down below, it reflects a private states 'We Americans'. Moreover, the target entity commented by the opinions can take several forms from tangible product to intangible topic matters stated in Liu (2010)", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0021", "text": ". Furthermore, three types of attitudes were observed by Liu (2010), 1) positive opinions, 2)  neutral opinions, and 3) negative opinions.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0022", "text": "Example of a subjective sentence: 'We Americans need to elect a president who is mature and who is able to make wise decisions.'\nThis analysis is a classification problem.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0023", "text": "Each class's collections of words or phrase indicators are defined for to locate desirable patterns on unannotated text. For subjective expression, a different word list has been created. Lists of subjective indicators in words or phrases have been developed by multiple researchers in the linguist and natural language processing field states in Riloff et al. (2003). A dictionary of extraction rules has to be created for measuring given expressions", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0024", "text": ". Over the years, in subjective detection, the features extraction progression from curating features by hand to automated features learning. At the moment, automated learning methods can further separate into supervised and unsupervised machine learning. Patterns extraction with machine learning process annotated and unannotated text have been explored extensively by academic researchers.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0025", "text": "However, researchers recognized several challenges in developing fixed sets of rules for expressions respectably. Much of the challenges in rule development stems from the nature of textual information. Six challenges have been recognized by several researchers: 1) metaphorical expressions, 2) discrepancies in writings, 3) context-sensitive, 4) represented words with fewer usages, 5) time-sensitive, and 6) ever-growing volume.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0026", "text": "Metaphorical expressions. The text contains metaphoric expression may impact on the performance on the extraction. Besides, metaphors take in different forms, which may have been contributed to the increase in detection.\nDiscrepancies in writings. For the text obtained from the Internet, the discrepancies in the writing style of targeted text data involve distinct writing genres and styles.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0027", "text": "Context-sensitive. Classification may vary based on the subjectiveness or objectiveness of previous and following sentences.\nTime-sensitive attribute. The task is challenged by some textual data's time-sensitive attribute. If a group of researchers wants to confirm a piece of fact in the news, they need a longer time for cross-validation, than the news becomes outdated.\nCue words with fewer usages.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0028", "text": "Cue words with fewer usages.\nEver-growing volume. The task is also challenged by the sheer volume of textual data. The textual data's ever-growing nature makes the task overwhelmingly difficult for the researchers to complete the task on time.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0029", "text": "Previously, the research mainly focused on document level classification. However, classifying a document level suffers less accuracy, as an article may have diverse types of expressions involved. Researching evidence suggests a set of news articles that are expected to dominate by the objective expression, whereas the results show that it consisted of over 40% of subjective expression.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0030", "text": "To overcome those challenges, researchers conclude that classifier efficacy depends on the precisions of patterns learner. And the learner feeds with large volumes of annotated training data outperformed those trained on less comprehensive subjective features. However, one of the main obstacles to executing this type of work is to generate a big dataset of annotated sentences manually. The manual annotation method has been less favored than automatic learning for three reasons:", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0031", "text": "Variations in comprehensions. In the manual annotation task, disagreement of whether one instance is subjective or objective may occur among annotators because of languages' ambiguity.\nHuman errors. Manual annotation task is a meticulous assignment, it require intense concentration to finish.\nTime-consuming. Manual annotation task is an assiduous work. Riloff (1996) show that a 160 texts cost 8 hours for one annotator to finish.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0032", "text": "All these mentioned reasons can impact on the efficiency and effectiveness of subjective and objective classification. Accordingly, two bootstrapping methods were designed to learning linguistic patterns from unannotated text data. Both methods are starting with a handful of seed words and unannotated textual data.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0033", "text": "Meta-Bootstrapping by Riloff and Jones in 1999.     Level One: Generate extraction patterns based on the pre-defined rules and the extracted patterns by the number of seed words each pattern holds.  Level Two: Top 5 words will be marked and add to the dictionary.   Repeat.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0034", "text": "Basilisk (Bootstrapping Approach to Semantic Lexicon Induction using Semantic Knowledge) by Thelen and Riloff.  Step One: Generate extraction patterns. Step Two: Move best patterns from Pattern Pool to Candidate Word Pool. Step Three: Top 10 words will be marked and add to the dictionary. Repeat.\nOverall, these algorithms highlight the need for automatic pattern recognition and extraction in subjective and objective task.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0035", "text": "Subjective and object classifier can enhance the several applications of natural language processing. One of the classifier's primary benefits is that it popularized the practice of data-driven decision-making processes in various industries. According to Liu, the applications of subjective and objective identification have been implemented in business, advertising, sports, and social science.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0036", "text": "Online review classification: In the business industry, the classifier helps the company better understand the feedbacks on product and reasonings behind the reviews.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0037", "text": "Stock price prediction: In the finance industry, the classifier aids the prediction model by process auxiliary information from social media and other textual information from the Internet.   Previous studies on Japanese stock price conducted by Dong et al. indicates that model with subjective and objective module may perform better than those without this part.\nSocial media analysis.\nStudents' feedback classification.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0038", "text": "Students' feedback classification.\nDocument summarising: The classifier can extract target-specified comments and gathering opinions made by one particular entity.\nComplex question answering. The classifier can dissect the complex questions by classing the language subject or objective and focused target. In the research Yu et al.(2003), the researcher developed a sentence and document level clustered that identity opinion pieces.\nDomain-specific applications.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0039", "text": "Domain-specific applications.\nEmail analysis: The subjective and objective classifier detects spam by tracing language patterns with target words.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0040", "text": "Feature/aspect-based", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0041", "text": "It refers to determining the opinions or sentiments expressed on different features or aspects of entities, e.g., of a cell phone, a digital camera, or a bank. A feature or aspect is an attribute or component of an entity, e.g., the screen of a cell phone, the service for a restaurant, or the picture quality of a camera. The advantage of feature-based sentiment analysis is the possibility to capture nuances about objects of interest", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0042", "text": ". Different features can generate different sentiment responses, for example a hotel can have a convenient location, but mediocre food. This problem involves several sub-problems, e.g., identifying relevant entities, extracting their features/aspects, and determining whether an opinion expressed on each feature/aspect is positive, negative or neutral. The automatic identification of features can be performed with syntactic methods, with topic modeling, or with deep learning", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0043", "text": ". More detailed discussions about this level of sentiment analysis can be found in Liu's work.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0044", "text": "Intensity ranking", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0045", "text": "Emotions and sentiments are subjective in nature. The degree of emotions/sentiments expressed in a given text at the document, sentence, or feature/aspect level\u2014to what degree of intensity is expressed in the opinion of a document, a sentence or an entity differs on a case-to-case basis. However, predicting only the emotion and sentiment does not always convey complete information", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0046", "text": ". The degree or level of emotions and sentiments often plays a crucial role in understanding the exact feeling within a single class (e.g., 'good' versus 'awesome'). Some methods leverage a stacked ensemble method for predicting intensity for emotion and sentiment by combining the outputs obtained and using deep learning models based on convolutional neural networks, long short-term memory networks and gated recurrent units.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0047", "text": "Methods and features", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0048", "text": "Existing approaches to sentiment analysis can be grouped into three main categories: knowledge-based techniques, statistical methods, and hybrid approaches. Knowledge-based techniques classify text by affect categories based on the presence of unambiguous affect words such as happy, sad, afraid, and bored. Some knowledge bases not only list obvious affect words, but also assign arbitrary words a probable \"affinity\" to particular emotions", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0049", "text": ". Statistical methods leverage elements from machine learning such as latent semantic analysis, support vector machines, \"bag of words\", \"Pointwise Mutual Information\" for Semantic Orientation, semantic space models or word embedding models, and deep learning. More sophisticated methods try to detect the holder of a sentiment (i.e., the person who maintains that affective state) and the target (i.e., the entity about which the affect is felt)", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0050", "text": ". To mine the opinion in context and get the feature about which the speaker has opined, the grammatical relationships of words are used. Grammatical dependency relations are obtained by deep parsing of the text", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0051", "text": ". Hybrid approaches leverage both machine learning and elements from knowledge representation such as ontologies and semantic networks in order to detect semantics that are expressed in a subtle manner, e.g., through the analysis of concepts that do not explicitly convey relevant information, but which are implicitly linked to other concepts that do so.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0052", "text": "Open source software tools as well as range of free and paid sentiment analysis tools deploy machine learning, statistics, and natural language processing techniques to automate sentiment analysis on large collections of texts, including web pages, online news, internet discussion groups, online reviews, web blogs, and social media", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0053", "text": ". Knowledge-based systems, on the other hand, make use of publicly available resources, to extract the semantic and affective information associated with natural language concepts. The system can help perform affective commonsense reasoning. Sentiment analysis can also be performed on visual content, i.e., images and videos (see Multimodal sentiment analysis). One of the first approaches in this direction is SentiBank utilizing an adjective noun pair representation of visual content", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0054", "text": ". In addition, the vast majority of sentiment classification approaches rely on the bag-of-words model, which disregards context, grammar and even word order. Approaches that analyses the sentiment based on how words compose the meaning of longer phrases have shown better result, but they incur an additional annotation overhead.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0055", "text": "A human analysis component is required in sentiment analysis, as automated systems are not able to analyze historical tendencies of the individual commenter, or the platform and are often classified incorrectly in their expressed sentiment. Automation impacts approximately 23% of comments that are correctly classified by humans. However, humans often disagree, and it is argued that the inter-human agreement provides an upper bound that automated sentiment classifiers can eventually reach.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0056", "text": "Evaluation\nThe accuracy of a sentiment analysis system is, in principle, how well it agrees with human judgments. This is usually measured by variant measures based on precision and recall over the two target categories of negative and positive texts. However, according to research human raters typically only agree about 80% of the time (see Inter-rater reliability).", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0057", "text": "On the other hand, computer systems will make very different errors than human assessors, and thus the figures are not entirely comparable. For instance, a computer system will have trouble with negations, exaggerations, jokes, or sarcasm, which typically are easy to handle for a human reader: some errors a computer system makes will seem overly naive to a human", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0058", "text": ". In general, the utility for practical commercial tasks of sentiment analysis as it is defined in academic research has been called into question, mostly since the simple one-dimensional model of sentiment from negative to positive yields rather little actionable information for a client worrying about the effect of public discourse on e.g. brand or corporate reputation.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0059", "text": "To better fit market needs, evaluation of sentiment analysis has moved to more task-based measures, formulated together with representatives from PR agencies and market research professionals. The focus in e.g. the RepLab evaluation data set is less on the content of the text under consideration and more on the effect of the text in question on brand reputation.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0060", "text": "Because evaluation of sentiment analysis is becoming more and more task based, each implementation needs a separate training model to get a more accurate representation of sentiment for a given data set. As LLM-based answer engines become more widespread, evaluation methods have expanded to include tools that track how these systems reference entities", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0061", "text": ". Illustrating emerging approaches to assessing contextual weighting and sentiment in LLM outputs include Semrushs AI Visibility Toolkit or Enterprise AIO, which analyse how entities are presented within generated responses.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0062", "text": "Web 2.0", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0063", "text": "The rise of social media such as blogs and social networks has fueled interest in sentiment analysis.  With the proliferation of reviews, ratings, recommendations and other forms of online expression, online opinion has turned into a kind of virtual currency for businesses looking to market their products, identify new opportunities and manage their reputations", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0064", "text": ".  As businesses look to automate the process of filtering out the noise, understanding the conversations, identifying the relevant content and actioning it appropriately, many are now looking to the field of sentiment analysis. Further complicating the matter, is the rise of anonymous social media platforms such as 4chan and Reddit", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0065", "text": ". If web 2.0 was all about democratizing publishing, then the next stage of the web may well be based on democratizing data mining of all the content that is getting published.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0066", "text": "One step towards this aim is accomplished in research. Several research teams in universities around the world currently focus on understanding the dynamics of sentiment in e-communities through sentiment analysis.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0067", "text": "The problem is that most sentiment analysis algorithms use simple terms to express sentiment about a product or service.  However, cultural factors, linguistic nuances, and differing contexts make it extremely difficult to turn a string of written text into a simple pro or con sentiment.  The fact that humans often disagree on the sentiment of text illustrates how big a task it is for computers to get this right.  The shorter the string of text, the harder it becomes.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0068", "text": "Even though short text strings might be a problem, sentiment analysis within microblogging has shown that Twitter can be seen as a valid online indicator of political sentiment. Tweets' political sentiment demonstrates close correspondence to parties' and politicians' political positions, indicating that the content of Twitter messages plausibly reflects the offline political landscape", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0069", "text": ". Furthermore, sentiment analysis on Twitter has also been shown to capture the public mood behind human reproduction cycles globally, as well as other problems of public-health relevance such as adverse drug reactions.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0070", "text": "While sentiment analysis has been popular for domains where authors express their opinion rather explicitly (\"the movie is awesome\"), such as social media and product reviews, only recently robust methods were devised for other domains where sentiment is strongly implicit or indirect. For example, in news articles - mostly due to the expected journalistic objectivity - journalists often describe actions or events rather than directly stating the polarity of a piece of information", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0071", "text": ". Earlier approaches using dictionaries or shallow machine learning features were unable to catch the \"meaning between the lines\", but recently researchers have proposed a deep learning based approach and dataset that is able to analyze sentiment in news articles.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0072", "text": "Scholars have utilized sentiment analysis to analyse the construction health and safety Tweets (which is called X now). The research revealed that there is a positive correlation between favorites and retweets in terms of sentiment valence. Others have examined the impact of YouTube on the dissemination of construction health and safety knowledge. They investigated how emotions influence users' behaviors in terms of viewing and commenting through semantic analysis", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0073", "text": ". In another study, positive sentiment accounted for an overwhelming figure of 85% in knowledge sharing of construction safety and health via Instagram.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0074", "text": "Application in recommender systems\nFor a recommender system, sentiment analysis has been proven to be a valuable technique. A recommender system aims to predict the preference for an item of a target user. Mainstream recommender systems work on explicit data set. For example, collaborative filtering works on the rating matrix, and content-based filtering works on the meta-data of the items.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0075", "text": "In many social networking services or e-commerce websites, users can provide text review, comment or feedback to the items. These user-generated text provide a rich source of user's sentiment opinions about numerous products and items. Potentially, for an item, such text can reveal both the related feature/aspects of the item and the users' sentiments on each feature", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0076", "text": ". The item's feature/aspects described in the text play the same role with the meta-data in content-based filtering, but the former are more valuable for the recommender system", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0077", "text": ". Since these features are broadly mentioned by users in their reviews, they can be seen as the most crucial features that can significantly influence the user's experience on the item, while the meta-data of the item (usually provided by the producers instead of consumers) may ignore features that are concerned by the users. For different items with common features, a user may give different sentiments. Also, a feature of the same item may receive different sentiments from different users", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0078", "text": ". Users' sentiments on the features can be regarded as a multi-dimensional rating score, reflecting their preference on the items.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0079", "text": "Based on the feature/aspects and the sentiments extracted from the user-generated text, a hybrid recommender system can be constructed. There are two types of motivation to recommend a candidate item to a user. The first motivation is the candidate item have numerous common features with the user's preferred items, while the second motivation is that the candidate item receives a high sentiment on its features", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0080", "text": ". For a preferred item, it is reasonable to believe that items with the same features will have a similar function or utility. So, these items will also likely to be preferred by the user. On the other hand, for a shared feature of two candidate items, other users may give positive sentiment to one of them while giving negative sentiment to another. Clearly, the high evaluated item should be recommended to the user", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0081", "text": ". Based on these two motivations, a combination ranking score of similarity and sentiment rating can be constructed for each candidate item.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0082", "text": "Except for the difficulty of the sentiment analysis itself, applying sentiment analysis on reviews or feedback also faces the challenge of spam and biased reviews. One direction of work is focused on evaluating the helpfulness of each review. Review or feedback poorly written is hardly helpful for recommender system. Besides, a review can be designed to hinder sales of a target product, thus be harmful to the recommender system even it is well written.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0083", "text": "Researchers also found that long and short forms of user-generated text should be treated differently. An interesting result shows that short-form reviews are sometimes more helpful than long-form, because it is easier to filter out the noise in a short-form text. For the long-form text, the growing length of the text does not always bring a proportionate increase in the number of features or sentiments in the text.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0084", "text": "Lamba & Madhusudhan introduce a nascent way to cater the information needs of today's library users by repackaging the results from sentiment analysis of social media platforms like Twitter and provide it as a consolidated time-based service in different formats. Further, they propose a new way of conducting marketing in libraries using social media mining and sentiment analysis.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0085", "text": "Ethical considerations", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0086", "text": "Issues such as privacy, consent, and bias are crucial since sentiment analysis regularly analyzes personal data without explicit user consent. The potential for misinterpretation and misuse of sentiment data can significantly impact societal norms. Furthermore, the development of ethical frameworks, as seen in projects like SEWA, where Ethical and Industrial Valorisation Advisory Boards are established, is essential for addressing these challenges", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0087", "text": ". These boards help ensure that sentiment analysis technologies are used responsibly, especially in applications involving the recognition of human emotions and behaviors. Such frameworks are vital for guiding the responsible use of sentiment analysis tools, ensuring they promote equity and respect user autonomy, and effectively address both routine and complex ethical issues.", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Sentiment_analysis_0088", "text": "See also\nAffective computing\nConsumer sentiment\nEmotion recognition\nFriendly artificial intelligence\nInterpersonal accuracy\nMultimodal sentiment analysis\nStylometry\n\n\n== References ==", "title": "Sentiment analysis", "url": "https://en.wikipedia.org/wiki/Sentiment_analysis"}, {"id": "rec_Namedentity_recognition_0000", "text": "Named-entity recognition (NER) (also known as (named) entity identification, entity chunking, and entity extraction) is a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names (PER), organizations (ORG), locations (LOC), geopolitical entities (GPE), vehicles (VEH), medical codes, time expressions, quantities, monetary values, percentages, etc.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0001", "text": "Most research on NER/NEE systems has been structured as taking an unannotated block of text, such as transducing:", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0002", "text": "Jim bought 300 shares of Acme Corp. in 2006.\ninto an annotated block of text that highlights the names of entities:\n\n[Jim]Person bought 300 shares of [Acme Corp.]Organization in [2006]Time.\nIn this example, a person name consisting of one token, a two-token company name and a temporal expression have been detected and classified.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0003", "text": "Problem\nDefinition", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0004", "text": "In the expression named entity, the word named restricts the task to those entities for which one or many strings, such as words or phrases, stand (fairly) consistently for some referent. This is closely related to rigid designators, as defined by Saul Kripke, although in practice NER deals with many names and referents that are not philosophically \"rigid\"", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0005", "text": ". For instance, the automotive company created by Henry Ford in 1903 can be referred to as Ford or Ford Motor Company, although \"Ford\" can refer to many other entities as well (see Ford)", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0006", "text": ". Rigid designators include proper names as well as terms for certain biological species and substances, but exclude pronouns (such as \"it\"; see coreference resolution), descriptions that pick out a referent by its properties (see also De dicto and de re), and names for kinds of things as opposed to individuals (for example \"Bank\").", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0007", "text": "Full named-entity recognition is often broken down, conceptually and possibly also in implementations, as two distinct problems: detection of names, and classification of the names by the type of entity they refer to (e.g. person, organization, or location).", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0008", "text": "The first phase is typically simplified to a segmentation problem: names are defined to be contiguous spans of tokens, with no nesting, so that \"Bank of America\" is a single name, disregarding the fact that inside this name, the substring \"America\" is itself a name. This segmentation problem is formally similar to chunking. The second phase requires choosing an ontology by which to organize categories of things.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0009", "text": "Temporal expressions and some numerical expressions (e.g., money, percentages, etc.) may also be considered as named entities in the context of the NER task. While some instances of these types are good examples of rigid designators (e.g., the year 2001) there are also many invalid ones (e.g., I take my vacations in \u201cJune\u201d). In the first case, the year 2001 refers to the 2001st year of the Gregorian calendar", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0010", "text": ". In the second case, the month June may refer to the month of an undefined year (past June, next June, every June, etc.). It is arguable that the definition of named entity is loosened in such cases for practical reasons. The definition of the term named entity is therefore not strict and often has to be explained in the context in which it is used.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0011", "text": "Certain hierarchies of named entity types have been proposed in the literature. BBN categories, proposed in 2002, are used for question answering and consists of 29 types and 64 subtypes. Sekine's extended hierarchy, proposed in 2002, is made of 200 subtypes. More recently, in 2011 Ritter used a hierarchy based on common Freebase entity types in ground-breaking experiments on NER over social media text.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0012", "text": "Difficulties\nNER can have reference resolution ambiguities where the same name can refer to different entities of the same type. For example, \"JFK\" can refer to the former president or his son.\nThe same name can refer to completely different types. \"JFK\" might refer to the airport in New York. \"IRA\" can refer to Individual Retirement Account or International Reading Association.\nThis can be caused by metonymy. For example, \"The White House\" can refer to an organization instead of a location.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0013", "text": "Formal evaluation\nTo evaluate the quality of an NER system's output, several measures have been defined. The usual measures are called precision, recall, and F1 score. However, several issues remain in just how to calculate those values.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0014", "text": "These statistical measures work reasonably well for the obvious cases of finding or missing a real entity exactly; and for finding a non-entity. However, NER can fail in many other ways, many of which are arguably \"partially correct\", and should not be counted as complete success or failures. For example, identifying a real entity, but:", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0015", "text": "with fewer tokens than desired (for example, missing the last token of \"John Smith, M.D.\")\nwith more tokens than desired (for example, including the first word of \"The University of MD\")\npartitioning adjacent entities differently (for example, treating \"Smith, Jones Robinson\" as 2 vs. 3 entities)\nassigning it a completely wrong type (for example, calling a personal name an organization)", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0016", "text": "assigning it a related but inexact type (for example, \"substance\" vs. \"drug\", or \"school\" vs. \"organization\")\ncorrectly identifying an entity, when what the user wanted was a smaller- or larger-scope entity (for example, identifying \"James Madison\" as a personal name, when it's part of \"James Madison University\"). Some NER systems impose the restriction that entities may never overlap or nest, which means that in some cases one must make arbitrary or task-specific choices.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0017", "text": "One overly simple method of measuring accuracy is merely to count what fraction of all tokens in the text were correctly or incorrectly identified as part of entity references (or as being entities of the correct type)", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0018", "text": ". This suffers from at least two problems: first, the vast majority of tokens in real-world text are not part of entity names, so the baseline accuracy (always predict \"not an entity\") is extravagantly high, typically >90%; and second, mispredicting the full span of an entity name is not properly penalized (finding only a person's first name when his last name follows might be scored as \u00bd accuracy).", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0019", "text": "In academic conferences such as CoNLL, a variant of the F1 score has been defined as follows:", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0020", "text": "Precision is the number of predicted entity name spans that line up exactly with spans in the gold standard evaluation data. I.e. when [Person Hans] [Person Blick] is predicted but [Person Hans Blick] was required, precision for the predicted name is zero. Precision is then averaged over all predicted entity names.\nRecall is similarly the number of names in the gold standard that appear at exactly the same location in the predictions.\nF1 score is the harmonic mean of these two.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0021", "text": "It follows from the above definition that any prediction that misses a single token, includes a spurious token, or has the wrong class, is a hard error and does not contribute positively to either precision or recall. Thus, this measure may be said to be pessimistic: it can be the case that many \"errors\" are close to correct, and might be adequate for a given purpose", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0022", "text": ". For example, one system might always omit titles such as \"Ms.\" or \"Ph.D.\", but be compared to a system or ground-truth data that expects titles to be included. In that case, every such name is treated as an error. Because of such issues, it is important actually to examine the kinds of errors, and decide how important they are given one's goals and requirements.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0023", "text": "Evaluation models based on a token-by-token matching have been proposed. Such models may be given partial credit for overlapping matches (such as using the Intersection over Union criterion). They allow a finer grained evaluation and comparison of extraction systems.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0024", "text": "Approaches\nNER systems have been created that use linguistic grammar-based techniques as well as statistical models such as machine learning. State of the art systems may incorporate multiple approaches.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0025", "text": "GATE supports NER across many languages and domains out of the box, usable via a graphical interface and a Java API.\nOpenNLP includes rule-based and statistical named-entity recognition.\nspaCy features fast statistical NER as well as an open-source named-entity visualizer.\nHand-crafted grammar-based systems typically obtain better precision, but at the cost of lower recall and months of work by experienced computational linguists.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0026", "text": "Statistical NER systems typically require a large amount of manually annotated training data. Semisupervised approaches have been suggested to avoid part of the annotation effort.\nIn the statistical learning era, NER was usually performed by learning a simple linear regression model on engineered features, then decoded by a bidirectional Viterbi algorithm. Some commonly used features include:", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0027", "text": "Lexical items: The token itself be labeled.\nStemmed lexical items.\nShape: The orthographic pattern of the target word. For example, all lowercase, all uppercase, initial uppercase, mixed case, uppercase followed by a period (often indicating a middle name), contains hyphen, etc.\nAffixes of the target word and surrounding words.\nPart of speech of the word.\nWhether the word appears in one or more named entity lists (gazetteers).\nWords and/or n-grams occurring in the surrounding context.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0028", "text": "A gazetteer is a list of names and their types, such as \"General Electric\". It can be used to augment any system for NER. They had been often used in the era of statistical machine learning.\nMany different classifier types have been used to perform machine-learned NER, with conditional random fields being a typical choice. Transformers features token classification using deep learning models.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0029", "text": "History", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0030", "text": "Early work in NER systems in the 1990s was aimed primarily at extraction from journalistic articles. Attention then turned to processing of military dispatches and reports. Later stages of the automatic content extraction (ACE) evaluation also included several types of informal text styles, such as weblogs and text transcripts from conversational telephone speech conversations", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0031", "text": ". Since about 1998, there has been a great deal of interest in entity identification in the molecular biology, bioinformatics, and medical natural language processing communities.  The most common entity of interest in that domain has been names of genes and gene products. There has been also considerable interest in the recognition of chemical entities and drugs in the context of the CHEMDNER", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0032", "text": "competition, with 27 teams participating in this task.\nIn 2001, research indicated that even state-of-the-art NER systems were brittle, meaning that NER systems developed for one domain did not typically perform well on other domains. Considerable effort is involved in tuning NER systems to perform well in a new domain; this is true for both rule-based and trainable statistical systems.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0033", "text": "As of 2007, state-of-the-art NER systems for English produce near-human performance. For example, the best system entering MUC-7 scored 93.39% of F-measure while human annotators scored 97.60% and 96.95%.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0034", "text": "Current challenges", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0035", "text": "Despite high F1 numbers reported on the MUC-7 dataset, the problem of named-entity recognition is far from being solved. The main efforts are directed to reducing the annotations labor by employing semi-supervised learning, robust performance across domains and scaling up to fine-grained entity types", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0036", "text": ". In recent years, many projects have turned to crowdsourcing, which is a promising solution to obtain high-quality aggregate human judgments for supervised and semi-supervised machine learning approaches to NER. Another challenging task is devising models to deal with linguistically complex contexts such as Twitter and search queries.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0037", "text": "There are some researchers who did some comparisons about the NER performances from different statistical models such as HMM (hidden Markov model), ME (maximum entropy), and CRF (conditional random fields), and feature sets. And some researchers recently proposed graph-based semi-supervised learning model for language specific NER tasks.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0038", "text": "A recently emerging task of identifying \"important expressions\" in text and cross-linking them to Wikipedia can be seen as an instance of extremely fine-grained named-entity recognition, where the types are the actual Wikipedia pages describing the (potentially ambiguous) concepts. Below is an example output of a Wikification system:", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0039", "text": "Another field that has seen progress but remains challenging is the application of NER to Twitter and other microblogs, considered \"noisy\" due to non-standard orthography, shortness and informality of texts. NER challenges in English Tweets have been organized by research communities to compare performances of various approaches, such as bidirectional LSTMs, Learning-to-Search, or CRFs.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0040", "text": "See also\nControlled vocabulary\nCoreference resolution\nEntity linking (aka named entity normalization, entity disambiguation)\nInformation extraction\nKnowledge extraction\nOnomastics\nRecord linkage\nSemantic Web\nSmart tag (Microsoft)\n\nReferences", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Namedentity_recognition_0041", "text": "References\n\nJurafsky, Daniel; Martin, James H. (2008). \"13.5 Partial Parsing\". Speech and Language Processing (2nd ed.). Upper Saddle River, N.J.: Prentice Hall. ISBN 978-0131873216.\nJurafsky, Daniel; Martin, James H. (2008). \"22.1. Named Entity Recognition\". Speech and Language Processing (2nd ed.). Upper Saddle River, N.J.: Prentice Hall. ISBN 978-0131873216.", "title": "Named-entity recognition", "url": "https://en.wikipedia.org/wiki/Named-entity_recognition"}, {"id": "rec_Text_mining_0000", "text": "Text mining, text data mining (TDM) or text analytics is the process of deriving high-quality information from text. It involves \"the discovery by computer of new, previously unknown information, by automatically extracting information from different written resources.\" Written resources may include websites, books, emails, reviews, and articles. High-quality information is typically obtained by devising patterns and trends by means such as statistical pattern learning. According to Hotho et al", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0001", "text": ". According to Hotho et al. (2005), there are three perspectives of text mining: information extraction, data mining, and knowledge discovery in databases (KDD). Text mining usually involves the process of structuring the input text (usually parsing, along with the addition of some derived linguistic features and the removal of others, and subsequent insertion into a database), deriving patterns within the structured data, and finally evaluation and interpretation of the output", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0002", "text": ". 'High quality' in text mining usually refers to some combination of relevance, novelty, and interest. Typical text mining tasks include text categorization, text clustering, concept/entity extraction, production of granular taxonomies, sentiment analysis, document summarization, and entity relation modeling (i.e., learning relations between named entities).", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0003", "text": "Text analysis involves information retrieval, lexical analysis to study word frequency distributions, pattern recognition, tagging/annotation, information extraction, data mining techniques including link and association analysis, visualization, and predictive analytics. The overarching goal is, essentially, to turn text into data for analysis, via the application of natural language processing (NLP), different types of algorithms and analytical methods", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0004", "text": ". An important phase of this process is the interpretation of the gathered information.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0005", "text": "A typical application is to scan a set of documents written in a natural language and either model the document set for predictive classification purposes or populate a database or search index with the information extracted. The document is the basic element when starting with text mining. Here, we define a document as a unit of textual data, which normally exists in many types of collections.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0006", "text": "Text analytics", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0007", "text": "Text analytics describes a set of linguistic, statistical, and machine learning techniques that model and structure the information content of textual sources for business intelligence, exploratory data analysis, research, or investigation. The term is roughly synonymous with text mining; indeed, Ronen Feldman modified a 2000 description of \"text mining\" in 2004 to describe \"text analytics\"", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0008", "text": ". The latter term is now used more frequently in business settings while \"text mining\" is used in some of the earliest application areas, dating to the 1980s, notably life-sciences research and government intelligence.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0009", "text": "The term text analytics also describes that application of text analytics to respond to business problems, whether independently or in conjunction with query and analysis of fielded, numerical data. It is a truism that 80% of business-relevant information originates in unstructured form, primarily text. These techniques and processes discover and present knowledge \u2013 facts, business rules, and relationships \u2013 that is otherwise locked in textual form, impenetrable to automated processing.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0010", "text": "Text analysis processes\nSubtasks\u2014components of a larger text-analytics effort\u2014typically include:", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0011", "text": "Dimensionality reduction is an important technique for pre-processing data. It is used to identify the root word for actual words and reduce the size of the text data.\nInformation retrieval or identification of a corpus is a preparatory step: collecting or identifying a set of textual materials, on the Web or held in a file system, database, or content corpus manager, for analysis.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0012", "text": "Although some text analytics systems apply exclusively advanced statistical methods, many others apply more extensive natural language processing, such as part of speech tagging, syntactic parsing, and other types of linguistic analysis.\nNamed entity recognition is the use of gazetteers or statistical techniques to identify named text features: people, organizations, place names, stock ticker symbols, certain abbreviations, and so on.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0013", "text": "Disambiguation\u2014the use of contextual clues\u2014may be required to decide where, for instance, \"Ford\" can refer to a former U.S. president, a vehicle manufacturer, a movie star, a river crossing, or some other entity.\nRecognition of pattern-identified entities: Features such as telephone numbers, e-mail addresses, quantities (with units) can be discerned via regular expression or other pattern matches.\nDocument clustering: identification of sets of similar text documents.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0014", "text": "Coreference resolution: identification of noun phrases and other terms that refer to the same object.\nExtraction of relationships, facts and events: identification of associations among entities and other information in texts.\nSentiment analysis: discerning of subjective material and extracting information about attitudes: sentiment, opinion, mood, and emotion. This is done at the entity, concept, or topic level and aims to distinguish opinion holders and objects.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0015", "text": "Quantitative text analysis: a set of techniques stemming from the social sciences where either a human judge or a computer extracts semantic or grammatical relationships between words in order to find out the meaning or stylistic patterns of, usually, a casual personal text for the purpose of psychological profiling etc.\nPre-processing usually involves tasks such as tokenization, filtering and stemming.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0016", "text": "Applications", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0017", "text": "Text mining technology is now broadly applied to a wide variety of government, research, and business needs. All these groups may use text mining for records management and searching documents relevant to their daily activities. Legal professionals may use text mining for e-discovery, for example. Governments and military groups use text mining for national security and intelligence purposes", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0018", "text": ". Scientific researchers incorporate text mining approaches into efforts to organize large sets of text data (i.e., addressing the problem of unstructured data), to determine ideas communicated through text (e.g., sentiment analysis in social media) and to support scientific discovery in fields such as the life sciences and bioinformatics. In business, applications are used to support competitive intelligence and automated ad placement, among numerous other activities.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0019", "text": "Security applications\nMany text mining software packages are marketed for security applications, especially monitoring and analysis of online plain text sources such as Internet news, blogs, etc. for national security purposes. It is also involved in the study of text encryption/decryption.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0020", "text": "Biomedical applications", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0021", "text": "A range of text mining applications in the biomedical literature has been described, including computational approaches to assist with studies in protein docking, protein interactions, and protein-disease associations. In addition, with large patient textual datasets in the clinical field, datasets of demographic information in population studies and adverse event reports, text mining can facilitate clinical studies and precision medicine", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0022", "text": ". Text mining algorithms can facilitate the stratification and indexing of specific clinical events in large patient textual datasets of symptoms, side effects, and comorbidities from electronic health records, event reports, and reports from specific diagnostic tests. One online text mining application in the biomedical literature is PubGene, a publicly accessible search engine that combines biomedical text mining with network visualization", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0023", "text": ". GoPubMed is a knowledge-based search engine for biomedical texts. Text mining techniques also enable us to extract unknown knowledge from unstructured documents in the clinical domain", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0024", "text": "Software applications", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0025", "text": "Text mining methods and software is also being researched and developed by major firms, including IBM and Microsoft, to further automate the mining and analysis processes, and by different firms working in the area of search and indexing in general as a way to improve their results. Within the public sector, much effort has been concentrated on creating software for tracking and monitoring terrorist activities", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0026", "text": ". For study purposes, Weka software is one of the most popular options in the scientific world, acting as an excellent entry point for beginners. For Python programmers, there is an excellent toolkit called NLTK for more general purposes. For more advanced programmers, there's also the Gensim library, which focuses on word embedding-based text representations.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0027", "text": "Online media applications\nText mining is being used by large media companies, such as the Tribune Company, to clarify information and to provide readers with greater search experiences, which in turn increases site \"stickiness\" and revenue. Additionally, on the back end, editors are benefiting by being able to share, associate and package news across properties, significantly increasing opportunities to monetize content.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0028", "text": "Business and marketing applications\nText analytics is being used in business, particularly, in marketing, such as in customer relationship management. Coussement and Van den Poel (2008) apply it to improve predictive analytics models for customer churn (customer attrition). Text mining is also being applied in stock returns prediction.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0029", "text": "Sentiment analysis\nSentiment analysis may involve analysis of products such as movies, books, or hotel reviews for estimating how favorable a review is for the product.\nSuch an analysis may need a labeled data set or labeling of the affectivity of words.\nResources for affectivity of words and concepts have been made for WordNet and ConceptNet, respectively.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0030", "text": "Text has been used to detect emotions in the related area of affective computing. Text based approaches to affective computing have been used on multiple corpora such as students evaluations, children stories and news stories.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0031", "text": "Scientific literature mining and academic applications", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0032", "text": "The issue of text mining is of importance to publishers who hold large databases of information needing indexing for retrieval. This is especially true in scientific disciplines, in which highly specific information is often contained within the written text", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0033", "text": ". Therefore, initiatives have been taken such as Nature's proposal for an Open Text Mining Interface (OTMI) and the National Institutes of Health's common Journal Publishing Document Type Definition (DTD) that would provide semantic cues to machines to answer specific queries contained within the text without removing publisher barriers to public access.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0034", "text": "Academic institutions have also become involved in the text mining initiative:", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0035", "text": "The National Centre for Text Mining (NaCTeM), is the first publicly funded text mining centre in the world. NaCTeM is operated by the University of Manchester in close collaboration with the Tsujii Lab, University of Tokyo. NaCTeM provides customised tools, research facilities and offers advice to the academic community. They are funded by the Joint Information Systems Committee (JISC) and two of the UK research councils (EPSRC & BBSRC)", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0036", "text": ". With an initial focus on text mining in the biological and biomedical sciences, research has since expanded into the areas of social sciences.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0037", "text": "In the United States, the School of Information at University of California, Berkeley is developing a program called BioText to assist biology researchers in text mining and analysis.\nThe Text Analysis Portal for Research (TAPoR), currently housed at the University of Alberta, is a scholarly project to catalogue text analysis applications and create a gateway for researchers new to the practice.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0038", "text": "Methods for scientific literature mining\nComputational methods have been developed to assist with information retrieval from scientific literature. Published approaches include methods for searching, determining novelty, and clarifying homonyms among technical reports.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0039", "text": "Digital humanities and computational sociology\nThe automatic analysis of vast textual corpora has created the possibility for scholars to analyze\nmillions of documents in multiple languages with very limited manual intervention. Key enabling technologies have been parsing, machine translation, topic categorization, and machine learning.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0040", "text": "The automatic parsing of textual corpora has enabled the extraction of actors and their relational networks on a vast scale, turning textual data into network data. The resulting networks, which can contain thousands of nodes, are then analyzed by using tools from network theory to identify the key actors, the key communities or parties, and general properties such as robustness or structural stability of the overall network, or centrality of certain nodes", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0041", "text": ". This automates the approach introduced by quantitative narrative analysis, whereby subject-verb-object triplets are identified with pairs of actors linked by an action, or pairs formed by actor-object.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0042", "text": "Content analysis has been a traditional part of social sciences and media studies for a long time. The automation of content analysis has allowed a \"big data\" revolution to take place in that field, with studies in social media and newspaper content that include millions of news items. Gender bias, readability, content similarity, reader preferences, and even mood have been analyzed based on text mining methods over millions of documents", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0043", "text": ". The analysis of readability, gender bias and topic bias was demonstrated in Flaounas et al. showing how different topics have different gender biases and levels of readability; the possibility to detect mood patterns in a vast population by analyzing Twitter content was demonstrated as well.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0044", "text": "Software\nText mining computer programs are available from many commercial and open source companies and sources.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0045", "text": "Intellectual property law\nSituation in the Europe Union\nUnder European copyright and database laws, the mining of in-copyright works (such as by web mining) without the permission of the copyright owner is permitted under Articles 3 and 4 of the 2019 Directive on Copyright in the Digital Single Market. A specific TDM exception for scientific research is described in article 3, whereas a more general exception described in article 4 only applies if the copyright holder has not opted out.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0046", "text": "The European Commission facilitated stakeholder discussion on text and data mining in 2013, under the title of Licenses for Europe. The fact that the focus on the solution to this legal issue was licenses, and not limitations and exceptions to copyright law, led representatives of universities, researchers, libraries, civil society groups and open access publishers to leave the stakeholder dialogue in May 2013.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0047", "text": "Situation in the United States", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0048", "text": "US copyright law, and in particular its fair use provisions, means that text mining in America, as well as other fair use countries such as Israel, Taiwan and South Korea, is viewed as being legal. As text mining is transformative, meaning that it does not supplant the original work, it is viewed as being lawful under fair use", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0049", "text": ". For example, as part of the Google Book settlement the presiding judge on the case ruled that Google's digitization project of in-copyright books was lawful, in part because of the transformative uses that the digitization project displayed\u2014one such use being text and data mining.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0050", "text": "Situation in Australia\nThere is no exception in copyright law of Australia for text or data mining within the Copyright Act 1968. The Australian Law Reform Commission has noted that it is unlikely that the \"research and study\" fair dealing exception would extend to cover such a topic either, given it would be beyond the \"reasonable portion\" requirement.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0051", "text": "Situation in the United Kingdom", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0052", "text": "In the UK in 2014, on the recommendation of the Hargreaves review, the government amended copyright law to allow text mining as a limitation and exception. It was the second country in the world to do so, following Japan, which introduced a mining-specific exception in 2009. However, owing to the restriction of the Information Society Directive (2001), the UK exception only allows content mining for non-commercial purposes", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0053", "text": ". UK copyright law does not allow this provision to be overridden by contractual terms and conditions.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0054", "text": "Implications", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0055", "text": "Until recently, websites most often used text-based searches, which only found documents containing specific user-defined words or phrases. Now, through use of a semantic web, text mining can find content based on meaning and context (rather than just by a specific word). Additionally, text mining software can be used to build large dossiers of information about specific people and events", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0056", "text": ". For example, large datasets based on data extracted from news reports can be built to facilitate social networks analysis or counter-intelligence. In effect, the text mining software may act in a capacity similar to an intelligence analyst or research librarian, albeit with a more limited scope of analysis. Text mining is also used in some email spam filters as a way of determining the characteristics of messages that are likely to be advertisements or other unwanted material", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0057", "text": ". Text mining plays an important role in determining financial market sentiment.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0058", "text": "See also\nReferences\nCitations\nSources\nAnaniadou, S. and McNaught, J. (Editors) (2006). Text Mining for Biology and Biomedicine. Artech House Books. ISBN 978-1-58053-984-5\nBilisoly, R. (2008). Practical Text Mining with Perl. New York: John Wiley & Sons. ISBN 978-0-470-17643-6\nFeldman, R., and Sanger, J. (2006). The Text Mining Handbook. New York: Cambridge University Press. ISBN 978-0-521-83657-9", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0059", "text": "Hotho, A., N\u00fcrnberger, A. and Paa\u00df, G. (2005). \"A brief survey of text mining\". In Ldv Forum, Vol. 20(1), p. 19-62\nIndurkhya, N., and Damerau, F. (2010). Handbook of Natural Language Processing, 2nd Edition. Boca Raton, FL: CRC Press. ISBN 978-1-4200-8592-1\nKao, A., and Poteet, S. (Editors). Natural Language Processing and Text Mining. Springer. ISBN 1-84628-175-X\nKonchady, M. Text Mining Application Programming (Programming Series). Charles River Media. ISBN 1-58450-460-9", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0060", "text": "Manning, C., and Schutze, H. (1999). Foundations of Statistical Natural Language Processing. Cambridge, MA: MIT Press. ISBN 978-0-262-13360-9\nMiner, G., Elder, J., Hill. T, Nisbet, R., Delen, D. and Fast, A. (2012). Practical Text Mining and Statistical Analysis for Non-structured Text Data Applications. Elsevier Academic Press. ISBN 978-0-12-386979-1\nMcKnight, W. (2005). \"Building business intelligence: Text data mining in business intelligence\". DM Review, 21\u201322.", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0061", "text": "Srivastava, A., and Sahami. M. (2009). Text Mining: Classification, Clustering, and Applications. Boca Raton, FL: CRC Press. ISBN 978-1-4200-5940-3\nZanasi, A. (Editor) (2007). Text Mining and its Applications to Intelligence, CRM and Knowledge Management. WIT Press. ISBN 978-1-84564-131-3", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Text_mining_0062", "text": "External links\nMarti Hearst: What Is Text Mining? (October 2003)\nAutomatic Content Extraction, Linguistic Data Consortium Archived 2013-09-25 at the Wayback Machine\nAutomatic Content Extraction, NIST", "title": "Text mining", "url": "https://en.wikipedia.org/wiki/Text_mining"}, {"id": "rec_Language_model_0000", "text": "A language model is a computational model  that predicts sequences in natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0001", "text": "Large language models (LLMs), currently their most advanced form as of 2019, are predominantly based on transformers trained on larger datasets (frequently using texts scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as the word n-gram language model.", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0002", "text": "History\nNoam Chomsky did pioneering work on language models in the 1950s by developing a theory of formal grammars.\nIn 1980, statistical approaches were explored and found to be more useful for many purposes than rule-based formal grammars. Discrete representations like word n-gram language models, with probabilities for discrete combinations of words, made significant advances.", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0003", "text": "In the 2000s, continuous representations for words, such as word embeddings, began to replace discrete representations. Typically, the representation is a real-valued vector that encodes a word\u2019s meaning such that words closer in vector space are similar in meaning and common relationships between words, such as plurality or gender, are preserved.", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0004", "text": "Pure statistical models\nIn 1980, the first significant statistical language model was proposed, and during the decade IBM performed 'Shannon-style' experiments, in which potential sources for language modeling improvement were identified by observing and analyzing the performance of human subjects in predicting or correcting text.", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0005", "text": "Models based on word n-grams\nExponential\nMaximum entropy language models encode the relationship between a word and the n-gram history using feature functions. The equation is", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0006", "text": "P\n        (\n        \n          w\n          \n            m\n          \n        \n        \u2223\n        \n          w\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          w\n          \n            m\n            \u2212\n            1\n          \n        \n        )\n        =\n        \n          \n            1\n            \n              Z\n              (\n              \n                w\n                \n                  1", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0007", "text": "1\n                \n              \n              ,\n              \u2026\n              ,\n              \n                w\n                \n                  m\n                  \u2212\n                  1\n                \n              \n              )\n            \n          \n        \n        exp\n        \u2061\n        (\n        \n          a\n          \n            T\n          \n        \n        f\n        (\n        \n          w\n          \n            1\n          \n        \n        ,\n        \u2026", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0008", "text": ",\n        \u2026\n        ,\n        \n          w\n          \n            m\n          \n        \n        )\n        )\n      \n    \n    {\\displaystyle P(w_{m}\\mid w_{1},\\ldots ,w_{m-1})={\\frac {1}{Z(w_{1},\\ldots ,w_{m-1})}}\\exp(a^{T}f(w_{1},\\ldots ,w_{m}))}", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0009", "text": "where \n  \n    \n      \n        Z\n        (\n        \n          w\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          w\n          \n            m\n            \u2212\n            1\n          \n        \n        )\n      \n    \n    {\\displaystyle Z(w_{1},\\ldots ,w_{m-1})}\n  \n is the partition function, \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n is the parameter vector, and \n  \n    \n      \n        f\n        (\n        \n          w", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0010", "text": "(\n        \n          w\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          w\n          \n            m\n          \n        \n        )\n      \n    \n    {\\displaystyle f(w_{1},\\ldots ,w_{m})}\n  \n is the feature function. In the simplest case, the feature function is just an indicator of the presence of a certain n-gram. It is helpful to use a prior on \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n or some form of regularization.", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0011", "text": "or some form of regularization.\nThe log-bilinear model is another example of an exponential language model.", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0012", "text": "Skip-gram model\nNeural models\nRecurrent neural network", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0013", "text": "Continuous representations or embeddings of words are produced in recurrent neural network-based language models (known also as continuous space language models). Such continuous space embeddings help to alleviate the curse of dimensionality, which is the consequence of the number of possible sequences of words increasing exponentially with the size of the vocabulary, further causing a data sparsity problem", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0014", "text": ". Neural networks avoid this problem by representing words as non-linear combinations of weights in a neural net.", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0015", "text": "Large language models\nAlthough sometimes matching human performance, it is not clear whether they are plausible cognitive models. At least for recurrent neural networks, it has been shown that they sometimes learn patterns that humans do not, but fail to learn patterns that humans typically do.", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0016", "text": "Evaluation and benchmarks\nEvaluation of the quality of language models is mostly done by comparison to human created sample benchmarks created from typical language-oriented tasks. Other, less established, quality tests examine the intrinsic character of a language model or compare two such models. Since language models are typically intended to be dynamic and to learn from data they see, some proposed models investigate the rate of learning, e.g., through inspection of learning curves.", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0017", "text": "Various data sets have been developed for use in evaluating language processing systems. These include:", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0018", "text": "Massive Multitask Language Understanding (MMLU)\nCorpus of Linguistic Acceptability\nGLUE benchmark\nMicrosoft Research Paraphrase Corpus\nMulti-Genre Natural Language Inference\nQuestion Natural Language Inference\nQuora Question Pairs\nRecognizing Textual Entailment\nSemantic Textual Similarity Benchmark\nSQuAD question answering Test\nStanford Sentiment Treebank\nWinograd NLI", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0019", "text": "Stanford Sentiment Treebank\nWinograd NLI\nBoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OpenBookQA, NaturalQuestions, TriviaQA, RACE, BIG-bench hard, GSM8k, RealToxicityPrompts, WinoGender, CrowS-Pairs", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Language_model_0020", "text": "See also\nReferences\n\n\n== Further reading ==", "title": "Language model", "url": "https://en.wikipedia.org/wiki/Language_model"}, {"id": "rec_Lexical_analysis_0000", "text": "Lexical tokenization is conversion of a text into (semantically or syntactically) meaningful lexical tokens belonging to categories defined by a \"lexer\" program. In case of a natural language, those categories include nouns, verbs, adjectives, punctuations etc. In case of a programming language, the categories include identifiers, operators, grouping symbols, data types and language keywords", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0001", "text": ". Lexical tokenization is related to the type of tokenization used in large language models (LLMs) but with two differences. First, lexical tokenization is usually based on a lexical grammar, whereas LLM tokenizers are usually probability-based. Second, LLM tokenizers perform a second step that converts the tokens into numerical values.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0002", "text": "Rule-based programs", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0003", "text": "A rule-based program, performing lexical tokenization, is called tokenizer, or scanner, although scanner is also a term for the first stage of a lexer. A lexer forms the first phase of a compiler frontend in processing. Analysis generally occurs in one pass. Lexers and parsers are most often used for compilers, but can be used for other computer language tools, such as prettyprinters or linters", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0004", "text": ". Lexing can be divided into two stages: the scanning, which segments the input string into syntactic units called lexemes and categorizes these into token classes, and the evaluating, which converts lexemes into processed values.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0005", "text": "Lexers are generally quite simple, with most of the complexity deferred to the syntactic analysis or semantic analysis phases, and can often be generated by a lexer generator, notably lex or derivatives. However, lexers can sometimes include some complexity, such as phrase structure processing to make input easier and simplify the parser, and may be written partly or fully by hand, either to support more features or for performance.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0006", "text": "Disambiguation of \"lexeme\"", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0007", "text": "What is called \"lexeme\" in rule-based natural language processing is not equal to what is called lexeme in linguistics. What is called \"lexeme\" in rule-based natural language processing can be equal to the linguistic equivalent only in analytic languages, such as English, but not in highly synthetic languages, such as fusional languages", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0008", "text": ". What is called a lexeme in rule-based natural language processing is more similar to what is called a word in linguistics (not to be confused with a word in computer architecture), although in some cases it may be more similar to a morpheme.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0009", "text": "Lexical token and lexical tokenization\nA lexical token is a string with an assigned and thus identified meaning, in contrast to the probabilistic token used in large language models. A lexical token consists of a token name and an optional token value. The token name is a category of a rule-based lexical unit.\n\nConsider this expression in the C programming language:\n\nx = a + b * 2;\nThe lexical analysis of this expression yields the following sequence of tokens:", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0010", "text": "[(identifier, 'x'), (operator, '='), (identifier, 'a'), (operator, '+'), (identifier, 'b'), (operator, '*'), (literal, '2'), (separator, ';')]\nA token name is what might be termed a part of speech in linguistics.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0011", "text": "Lexical tokenization is the conversion of a raw text into (semantically or syntactically) meaningful lexical tokens, belonging to categories defined by a \"lexer\" program, such as identifiers, operators, grouping symbols, and data types. The resulting tokens are then passed on to some other form of processing. The process can be considered a sub-task of parsing input.\nFor example, in the text string:", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0012", "text": "The quick brown fox jumps over the lazy dog\nthe string is not implicitly segmented on spaces, as a natural language speaker would do. The raw input, the 43 characters, must be explicitly split into the 9 tokens with a given space delimiter (i.e., matching the string \" \" or regular expression /\\s{1}/).", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0013", "text": "When a token class represents more than one possible lexeme, the lexer often saves enough information to reproduce the original lexeme, so that it can be used in semantic analysis. The parser typically retrieves this information from the lexer and stores it in the abstract syntax tree. This is necessary in order to avoid information loss in the case where numbers may also be valid identifiers.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0014", "text": "Tokens are identified based on the specific rules of the lexer. Some methods used to identify tokens include regular expressions, specific sequences of characters termed a flag, specific separating characters called delimiters, and explicit definition by a dictionary. Special characters, including punctuation characters, are commonly used by lexers to identify tokens because of their natural use in written and programming languages", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0015", "text": ". A lexical analyzer generally does nothing with combinations of tokens, a task left for a parser. For example, a typical lexical analyzer recognizes parentheses as tokens but does nothing to ensure that each \"(\" is matched with a \")\".", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0016", "text": "When a lexer feeds tokens to the parser, the representation used is typically an enumerated type, which is a list of number representations. For example, \"Identifier\" can be represented with 0, \"Assignment operator\" with 1, \"Addition operator\" with 2, etc.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0017", "text": "Tokens are often defined by regular expressions, which are understood by a lexical analyzer generator such as lex, or handcoded equivalent finite-state automata. The lexical analyzer (generated automatically by a tool like lex or hand-crafted) reads in a stream of characters, identifies the lexemes in the stream, and categorizes them into tokens. This is termed tokenizing. If the lexer finds an invalid token, it will report an error.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0018", "text": "Following tokenizing is parsing. From there, the interpreted data may be loaded into data structures for general use, interpretation, or compiling.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0019", "text": "Lexical grammar\nThe specification of a programming language often includes a set of rules, the lexical grammar, which defines the lexical syntax. The lexical syntax is usually a regular language, with the grammar rules consisting of regular expressions; they define the set of possible character sequences (lexemes) of a token. A lexer recognizes strings, and for each kind of string found, the lexical program takes an action, most simply producing a token.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0020", "text": "Two important common lexical categories are white space and comments. These are also defined in the grammar and processed by the lexer but may be discarded (not producing any tokens) and considered non-significant, at most separating two tokens (as in if x instead of ifx). There are two important exceptions to this", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0021", "text": ". There are two important exceptions to this. First, in off-side rule languages that delimit blocks with indenting, initial whitespace is significant, as it determines block structure, and is generally handled at the lexer level; see phrase structure, below. Secondly, in some uses of lexers, comments and whitespace must be preserved \u2013 for examples, a prettyprinter also needs to output the comments and some debugging tools may provide messages to the programmer showing the original source code", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0022", "text": ". In the 1960s, notably for ALGOL, whitespace and comments were eliminated as part of the line reconstruction phase (the initial phase of the compiler frontend), but this separate phase has been eliminated and these are now handled by the lexer.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0023", "text": "Details\nScanner", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0024", "text": "The first stage, the scanner, is usually based on a finite-state machine (FSM). It has encoded within it information on the possible sequences of characters that can be contained within any of the tokens it handles (individual instances of these character sequences are termed lexemes). For example, an integer lexeme may contain any sequence of numerical digit characters", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0025", "text": ". In many cases, the first non-whitespace character can be used to deduce the kind of token that follows and subsequent input characters are then processed one at a time until reaching a character that is not in the set of characters acceptable for that token (this is termed the maximal munch, or longest match, rule). In some languages, the lexeme creation rules are more complex and may involve backtracking over previously read characters", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0026", "text": ". For example, in C, one 'L' character is not enough to distinguish between an identifier that begins with 'L' and a wide-character string literal.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0027", "text": "Evaluator\nA lexeme, however, is only a string of characters known to be of a certain kind (e.g., a string literal, a sequence of letters). The second stage of a lexical analyzer, the evaluator, goes over the characters of the lexeme to produce a value containing relevant information for the parser.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0028", "text": "The lexeme's type combined with its value is what properly constitutes a token. The value in the token can be whatever is deemed necessary for the parser to interpret a token of that type. Some examples of typical values produced by an evaluator include:", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0029", "text": "A token for an identifier will often simply contain the characters of the associated lexeme.\nToken values for keywords and special characters are usually omitted, as the type alone contains all the information needed.\nEvaluators processing integer literals may pass the string on as is (deferring evaluation to the semantic analysis phase), or may perform evaluation themselves to produce numeric values.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0030", "text": "For a simple quoted string literal, the evaluator needs to remove only the quotes, but the evaluator for an escaped string literal may also incorporate a lexer, which unescapes the escape sequences.\nThe evaluator may also suppress a lexeme entirely, concealing it from the parser, which is useful for whitespace and comments.\nFor example, in the source code of a computer program, the string", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0031", "text": "net_worth_future = (assets \u2013 liabilities);\nmight be converted into the following lexical token stream; where each line represents a token composed of a TYPE followed by an optional value:\n\nIDENTIFIER \"net_worth_future\"\nEQUALS\nOPEN_PARENTHESIS\nIDENTIFIER \"assets\"\nMINUS\nIDENTIFIER \"liabilities\"\nCLOSE_PARENTHESIS\nSEMICOLON", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0032", "text": "Lexers may be written by hand. This is practical if the list of tokens is small, but lexers generated by automated tooling as part of a compiler-compiler toolchain are more practical for a larger number of potential tokens. These tools generally accept regular expressions that describe the tokens allowed in the input stream. Each regular expression is associated with a production rule in the lexical grammar of the programming language that evaluates the lexemes matching the regular expression", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0033", "text": ". These tools may generate source code that can be compiled and executed or construct a state transition table for a finite-state machine (which is plugged into template code for compiling and executing).", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0034", "text": "Regular expressions compactly represent patterns that the characters in lexemes might follow. For example, for an English-based language, an IDENTIFIER token might be any English alphabetic character or an underscore, followed by any number of instances of ASCII alphanumeric characters and/or underscores. This could be represented compactly by the string [a-zA-Z_][a-zA-Z_0-9]*. This means \"any character a-z, A-Z or _, followed by 0 or more of a-z, A-Z, _ or 0-9\".", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0035", "text": "Regular expressions and the finite-state machines they generate are not powerful enough to handle recursive patterns, such as \"n opening parentheses, followed by a statement, followed by n closing parentheses.\" They are unable to keep count, and verify that n is the same on both sides, unless a finite set of permissible values exists for n. It takes a full parser to recognize such patterns in their full generality", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0036", "text": ". A parser can push parentheses on a stack and then try to pop them off and see if the stack is empty at the end (see example in the Structure and Interpretation of Computer Programs book).", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0037", "text": "Obstacles\nTypically, lexical tokenization occurs at the word level. However, it is sometimes difficult to define what is meant by a \"word\". Often, a tokenizer relies on simple heuristics, for example:", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0038", "text": "Punctuation and whitespace may or may not be included in the resulting list of tokens.\nAll contiguous strings of alphabetic characters are part of one token; likewise with numbers.\nTokens are separated by whitespace characters, such as a space or line break, or by punctuation characters.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0039", "text": "In languages that use inter-word spaces (such as most that use the Latin alphabet, and most programming languages), this approach is fairly straightforward. However, even here there are many edge cases such as contractions, hyphenated words, emoticons, and larger constructs such as URIs (which for some purposes may count as single tokens). A classic example is \"New York-based\", which a naive tokenizer may break at the space even though the better break is (arguably) at the hyphen.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0040", "text": "Tokenization is particularly difficult for languages written in scriptio continua, which exhibit no word boundaries, such as Ancient Greek, Chinese, or Thai. Agglutinative languages, such as Korean, also make tokenization tasks complicated.\nSome ways to address the more difficult problems include developing more complex heuristics, querying a table of common special cases, or fitting the tokens to a language model that identifies collocations in a later processing step.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0041", "text": "Lexer generator\nLexers are often generated by a lexer generator, analogous to parser generators, and such tools often come together. The most established is lex, paired with the yacc parser generator, or rather some of their many reimplementations, like flex (often paired with GNU Bison). These generators are a form of domain-specific language, taking in a lexical specification \u2013 generally regular expressions with some markup \u2013 and emitting a lexer.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0042", "text": "These tools yield very fast development, which is very important in early development, both to get a working lexer and because a language specification may change often. Further, they often provide advanced features, such as pre- and post-conditions which are hard to program by hand. However, an automatically generated lexer may lack flexibility, and thus may require some manual modification, or an all-manually written lexer.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0043", "text": "Lexer performance is a concern, and optimizing is worthwhile, more so in stable languages where the lexer runs very often (such as C or HTML). lex/flex-generated lexers are reasonably fast, but improvements of two to three times are possible using more tuned generators. Hand-written lexers are sometimes used, but modern lexer generators produce faster lexers than most hand-coded ones", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0044", "text": ". The lex/flex family of generators uses a table-driven approach which is much less efficient than the directly coded approach. With the latter approach the generator produces an engine that directly jumps to follow-up states via goto statements. Tools like re2c have proven to produce engines that are between two and three times faster than flex produced engines. It is in general difficult to hand-write analyzers that perform better than engines generated by these latter tools.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0045", "text": "Phrase structure", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0046", "text": "Lexical analysis mainly segments the input stream of characters into tokens, simply grouping the characters into pieces and categorizing them. However, the lexing may be significantly more complex; most simply, lexers may omit tokens or insert added tokens. Omitting tokens, notably whitespace and comments, is very common when these are not needed by the compiler. Less commonly, added tokens may be inserted", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0047", "text": ". Less commonly, added tokens may be inserted. This is done mainly to group tokens into statements, or statements into blocks, to simplify the parser.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0048", "text": "Line continuation\nLine continuation is a feature of some languages where a newline is normally a statement terminator. Most often, ending a line with a backslash (immediately followed by a newline) results in the line being continued \u2013 the following line is joined to the prior line. This is generally done in the lexer: The backslash and newline are discarded, rather than the newline being tokenized. Examples include bash, other shell scripts and Python.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0049", "text": "Semicolon insertion", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0050", "text": "Many languages use the semicolon as a statement terminator. Most often this is mandatory, but in some languages the semicolon is optional in many contexts. This is mainly done at the lexer level, where the lexer outputs a semicolon into the token stream, despite one not being present in the input character stream, and is termed semicolon insertion or automatic semicolon insertion", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0051", "text": ". In these cases, semicolons are part of the formal phrase grammar of the language, but may not be found in input text, as they can be inserted by the lexer. Optional semicolons or other terminators or separators are also sometimes handled at the parser level, notably in the case of trailing commas or semicolons.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0052", "text": "Semicolon insertion is a feature of BCPL and its distant descendant Go, though it is absent in B or C. Semicolon insertion is present in JavaScript, though the rules are somewhat complex and much-criticized; to avoid bugs, some recommend always using semicolons, while others use initial semicolons, termed defensive semicolons, at the start of potentially ambiguous statements.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0053", "text": "Semicolon insertion (in languages with semicolon-terminated statements) and line continuation (in languages with newline-terminated statements) can be seen as complementary: Semicolon insertion adds a token even though newlines generally do not generate tokens, while line continuation prevents a token from being generated even though newlines generally do generate tokens.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0054", "text": "Off-side rule", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0055", "text": "The off-side rule (blocks determined by indenting) can be implemented in the lexer, as in Python, where increasing the indenting results in the lexer emitting an INDENT token and decreasing the indenting results in the lexer emitting one or more DEDENT tokens. These tokens correspond to the opening brace { and closing brace } in languages that use braces for blocks and means that the phrase grammar does not depend on whether braces or indenting are used", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0056", "text": ". This requires that the lexer hold state, namely a stack of indent levels, and thus can detect changes in indenting when this changes, and thus the lexical grammar is not context-free: INDENT\u2013DEDENT depend on the contextual information of prior indent levels.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0057", "text": "Context-sensitive lexing\nGenerally lexical grammars are context-free, or almost so, and thus require no looking back or ahead, or backtracking, which allows a simple, clean, and efficient implementation. This also allows simple one-way communication from lexer to parser, without needing any information flowing back to the lexer.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0058", "text": "There are exceptions, however. Simple examples include semicolon insertion in Go, which requires looking back one token; concatenation of consecutive string literals in Python, which requires holding one token in a buffer before emitting it (to see if the next token is another string literal); and the off-side rule in Python, which requires maintaining a count of indent level (indeed, a stack of each indent level)", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0059", "text": ". These examples all only require lexical context, and while they complicate a lexer somewhat, they are invisible to the parser and later phases.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0060", "text": "A more complex example is the lexer hack in C, where the token class of a sequence of characters cannot be determined until the semantic analysis phase since typedef names and variable names are lexically identical but constitute different token classes. Thus in the hack, the lexer calls the semantic analyzer (say, symbol table) and checks if the sequence requires a typedef name", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0061", "text": ". In this case, information must flow back not from the parser only, but from the semantic analyzer back to the lexer, which complicates design.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0062", "text": "See also\nLexical frequency analysis\nLexicalization\nLexical semantics\nList of parser generators", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0063", "text": "References\nSources\nExternal links\nYang, W.; Tsay, Chey-Woei; Chan, Jien-Tsai (2002). \"On the applicability of the longest-match rule in lexical analysis\". Computer Languages, Systems & Structures. 28 (3): 273\u2013288. doi:10.1016/S0096-0551(02)00014-0. NSC 86-2213-E-009-021 and NSC 86-2213-E-009-079. Archived from the original on 2022-04-17. Retrieved 2015-09-04.\nTrim, Craig (Jan 23, 2013). \"The Art of Tokenization\". Developer Works. IBM. Archived from the original on 2019-05-30.", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Lexical_analysis_0064", "text": "Word Mention Segmentation Task, an analysis", "title": "Lexical analysis", "url": "https://en.wikipedia.org/wiki/Lexical_analysis"}, {"id": "rec_Support_vector_machine_0000", "text": "In machine learning, support vector machines (SVMs, also support vector networks) are supervised max-margin models with associated learning algorithms that analyze data for classification and regression analysis. Developed at AT&T Bell Laboratories, SVMs are one of the most studied models, being based on statistical learning frameworks of VC theory proposed by Vapnik (1982, 1995) and Chervonenkis (1974).", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0001", "text": "In addition to performing linear classification, SVMs can efficiently perform non-linear classification using the kernel trick, representing the data only through a set of pairwise similarity comparisons between the original data points using a kernel function, which transforms them into coordinates in a higher-dimensional feature space. Thus, SVMs use the kernel trick to implicitly map their inputs into high-dimensional feature spaces, where linear classification can be performed", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0002", "text": ".  Being max-margin models, SVMs are resilient to noisy data (e.g., misclassified examples). SVMs can also be used for regression tasks, where the objective becomes", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0003", "text": "\u03f5\n      \n    \n    {\\displaystyle \\epsilon }\n  \n-sensitive.\nThe support vector clustering algorithm, created by Hava Siegelmann and Vladimir Vapnik, applies the statistics of support vectors, developed in the support vector machines algorithm, to categorize unlabeled data. These data sets require unsupervised learning approaches, which attempt to find natural clustering of the data into groups, and then to map new data according to these clusters.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0004", "text": "The popularity of SVMs is likely due to their amenability to theoretical analysis, and their flexibility in being applied to a wide variety of tasks, including structured prediction problems. It is not clear that SVMs have better predictive performance than other linear models, such as logistic regression and linear regression.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0005", "text": "Motivation\nClassifying data is a common task in machine learning.\nSuppose some given data points each belong to one of two classes, and the goal is to decide which class a new data point will be in. In the case of support vector machines, a data point is viewed as a \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n-dimensional vector (a list of \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n numbers), and we want to know whether we can separate such points with a", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0006", "text": "(\n        p\n        \u2212\n        1\n        )\n      \n    \n    {\\displaystyle (p-1)}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0007", "text": "-dimensional hyperplane. This is called a linear classifier. There are many hyperplanes that might classify the data. One reasonable choice as the best hyperplane is the one that represents the largest separation, or margin, between the two classes. So we choose the hyperplane so that the distance from it to the nearest data point on each side is maximized", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0008", "text": ". If such a hyperplane exists, it is known as the maximum-margin hyperplane and the linear classifier it defines is known as a maximum-margin classifier; or equivalently, the perceptron of optimal stability.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0009", "text": "More formally, a support vector machine constructs a hyperplane or set of hyperplanes in a high or infinite-dimensional space, which can be used for classification, regression, or other tasks like outlier detection. Intuitively, a good separation is achieved by the hyperplane that has the largest distance to the nearest training-data point of any class (so-called functional margin), since in general the larger the margin, the lower the generalization error of the classifier", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0010", "text": ". A lower generalization error means that the implementer is less likely to experience overfitting.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0011", "text": "Whereas the original problem may be stated in a finite-dimensional space, it often happens that the sets to discriminate are not linearly separable in that space. For this reason, it was proposed that the original finite-dimensional space be mapped into a much higher-dimensional space, presumably making the separation easier in that space", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0012", "text": ". To keep the computational load reasonable, the mappings used by SVM schemes are designed to ensure that dot products of pairs of input data vectors may be computed easily in terms of the variables in the original space, by defining them in terms of a kernel function", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0013", "text": "k\n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle k(x,y)}\n  \n selected to suit the problem. The hyperplanes in the higher-dimensional space are defined as the set of points whose dot product with a vector in that space is constant, where such a set of vectors is an orthogonal (and thus minimal) set of vectors that defines a hyperplane. The vectors defining the hyperplanes can be chosen to be linear combinations with parameters", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0014", "text": "\u03b1\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\alpha _{i}}\n  \n of images of feature vectors \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n that occur in the data base. With this choice of a hyperplane, the points \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0015", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n in the feature space that are mapped into the hyperplane are defined by the relation \n  \n    \n      \n        \n          \n            \u2211\n            \n              i\n            \n          \n          \n            \u03b1\n            \n              i\n            \n          \n          k\n          (\n          \n            x\n            \n              i\n            \n          \n          ,\n          x\n          )\n          =\n          \n            constant", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0016", "text": "=\n          \n            constant\n          \n          .\n        \n      \n    \n    {\\displaystyle \\textstyle \\sum _{i}\\alpha _{i}k(x_{i},x)={\\text{constant}}.}\n  \n  Note that if \n  \n    \n      \n        k\n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle k(x,y)}\n  \n becomes small as \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n grows further away from \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0017", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n, each term in the sum measures the degree of closeness of the test point \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n to the corresponding data base point \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0018", "text": "{\\displaystyle x_{i}}\n  \n. In this way, the sum of kernels above can be used to measure the relative nearness of each test point to the data points originating in one or the other of the sets to be discriminated. Note the fact that the set of points \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0019", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n mapped into any hyperplane can be quite convoluted as a result, allowing much more complex discrimination between sets that are not convex at all in the original space.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0020", "text": "Applications\nSVMs can be used to solve various real-world problems:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0021", "text": "SVMs are helpful in text and hypertext categorization, as their application can significantly reduce the need for labeled training instances in both the standard inductive and transductive settings. Some methods for shallow semantic parsing are based on support vector machines.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0022", "text": "Classification of images can also be performed using SVMs. Experimental results show that SVMs achieve significantly higher search accuracy than traditional query refinement schemes after just three to four rounds of relevance feedback. This is also true for image segmentation systems, including those using a modified version SVM that uses the privileged approach as suggested by Vapnik.\nClassification of satellite data like SAR data using supervised SVM.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0023", "text": "Hand-written characters can be recognized using SVM.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0024", "text": "The SVM algorithm has been widely applied in the biological and other sciences.  They have been used to classify proteins with up to 90% of the compounds classified correctly. Permutation tests based on SVM weights have been suggested as a mechanism for interpretation of SVM models. Support vector machine weights have also been used to interpret SVM models in the past", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0025", "text": ". Posthoc interpretation of support vector machine models in order to identify features used by the model to make predictions is a relatively new area of research with special significance in the biological sciences.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0026", "text": "History\nThe original SVM algorithm was invented by Vladimir N. Vapnik and Alexey Ya. Chervonenkis in 1964. In 1992, Bernhard Boser, Isabelle Guyon and Vladimir Vapnik suggested a way to create nonlinear classifiers by applying the kernel trick to maximum-margin hyperplanes. The \"soft margin\" incarnation, as is commonly used in software packages, was proposed by Corinna Cortes and Vapnik in 1993 and published in 1995.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0027", "text": "Linear SVM\nWe are given a training dataset of \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n points of the form", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0028", "text": "(\n        \n          \n            x\n          \n          \n            1\n          \n        \n        ,\n        \n          y\n          \n            1\n          \n        \n        )\n        ,\n        \u2026\n        ,\n        (\n        \n          \n            x\n          \n          \n            n\n          \n        \n        ,\n        \n          y\n          \n            n\n          \n        \n        )\n        ,", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0029", "text": ")\n        ,\n      \n    \n    {\\displaystyle (\\mathbf {x} _{1},y_{1}),\\ldots ,(\\mathbf {x} _{n},y_{n}),}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0030", "text": "where the \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}\n  \n are either 1 or \u22121, each indicating the class to which the point \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n belongs. Each \n  \n    \n      \n        \n          \n            x\n          \n          \n            i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0031", "text": "i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n is a \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n-dimensional real vector. We want to find the \"maximum-margin hyperplane\" that divides the group of points \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n for which \n  \n    \n      \n        \n          y\n          \n            i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0032", "text": "y\n          \n            i\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle y_{i}=1}\n  \n from the group of points for which \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n        =\n        \u2212\n        1\n      \n    \n    {\\displaystyle y_{i}=-1}\n  \n, which is defined so that the distance between the hyperplane and the nearest point \n  \n    \n      \n        \n          \n            x\n          \n          \n            i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0033", "text": "x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n from either group is maximized.\nAny hyperplane can be written as the set of points \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n satisfying", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0034", "text": "w\n          \n          \n            \n              T\n            \n          \n        \n        \n          x\n        \n        \u2212\n        b\n        =\n        0\n        ,\n      \n    \n    {\\displaystyle \\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} -b=0,}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0035", "text": "where \n  \n    \n      \n        \n          w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n is the (not necessarily normalized) normal vector to the hyperplane. This is much like Hesse normal form, except that \n  \n    \n      \n        \n          w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n is not necessarily a unit vector. The parameter \n  \n    \n      \n        \n          \n            \n              b\n              \n                \u2016\n                \n                  w", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0036", "text": "w\n                \n                \u2016\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\tfrac {b}{\\|\\mathbf {w} \\|}}}\n  \n determines the offset of the hyperplane from the origin along the normal vector \n  \n    \n      \n        \n          w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n.\nWarning: most of the literature on the subject defines the bias so that", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0037", "text": "w\n          \n          \n            \n              T\n            \n          \n        \n        \n          x\n        \n        +\n        b\n        =\n        0.\n      \n    \n    {\\displaystyle \\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} +b=0.}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0038", "text": "Hard-margin\nIf the training data is linearly separable, we can select two parallel hyperplanes that separate the two classes of data, so that the distance between them is as large as possible. The region bounded by these two hyperplanes is called the \"margin\", and the maximum-margin hyperplane is the hyperplane that lies halfway between them. With a normalized or standardized dataset, these hyperplanes can be described by the equations", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0039", "text": "w\n          \n          \n            \n              T\n            \n          \n        \n        \n          x\n        \n        \u2212\n        b\n        =\n        1\n      \n    \n    {\\displaystyle \\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} -b=1}\n  \n (anything on or above this boundary is of one class, with label 1)\nand", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0040", "text": "w\n          \n          \n            \n              T\n            \n          \n        \n        \n          x\n        \n        \u2212\n        b\n        =\n        \u2212\n        1\n      \n    \n    {\\displaystyle \\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} -b=-1}\n  \n (anything on or below this boundary is of the other class, with label \u22121).\nGeometrically, the distance between these two hyperplanes is \n  \n    \n      \n        \n          \n            \n              2", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0041", "text": "2\n              \n                \u2016\n                \n                  w\n                \n                \u2016\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\tfrac {2}{\\|\\mathbf {w} \\|}}}\n  \n, so to maximize the distance between the planes we want to minimize \n  \n    \n      \n        \u2016\n        \n          w\n        \n        \u2016\n      \n    \n    {\\displaystyle \\|\\mathbf {w} \\|}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0042", "text": "{\\displaystyle \\|\\mathbf {w} \\|}\n  \n. The distance is computed using the distance from a point to a plane equation. We also have to prevent data points from falling into the margin, we add the following constraint: for each \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n either", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0043", "text": "w\n          \n          \n            \n              T\n            \n          \n        \n        \n          \n            x\n          \n          \n            i\n          \n        \n        \u2212\n        b\n        \u2265\n        1\n        \n        ,\n        \n           if \n        \n        \n          y\n          \n            i\n          \n        \n        =\n        1\n        ,", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0044", "text": "=\n        1\n        ,\n      \n    \n    {\\displaystyle \\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-b\\geq 1\\,,{\\text{ if }}y_{i}=1,}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0045", "text": "or", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0046", "text": "w\n          \n          \n            \n              T\n            \n          \n        \n        \n          \n            x\n          \n          \n            i\n          \n        \n        \u2212\n        b\n        \u2264\n        \u2212\n        1\n        \n        ,\n        \n           if \n        \n        \n          y\n          \n            i\n          \n        \n        =\n        \u2212\n        1.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0047", "text": "=\n        \u2212\n        1.\n      \n    \n    {\\displaystyle \\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-b\\leq -1\\,,{\\text{ if }}y_{i}=-1.}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0048", "text": "These constraints state that each data point must lie on the correct side of the margin.\nThis can be rewritten as\n\nWe can put this together to get the optimization problem:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0049", "text": "minimize\n                    \n                      \n                        w\n                      \n                      ,\n                      \n                      b\n                    \n                  \n                \n              \n              \n              \n                \n                  \n                    1\n                    2", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0050", "text": "1\n                    2\n                  \n                \n                \u2016\n                \n                  w\n                \n                \n                  \u2016\n                  \n                    2\n                  \n                \n              \n            \n            \n              \n              \n                \n                  subject to\n                \n              \n              \n              \n                \n                  y", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0051", "text": "y\n                  \n                    i\n                  \n                \n                (\n                \n                  \n                    w\n                  \n                  \n                    \u22a4\n                  \n                \n                \n                  \n                    x\n                  \n                  \n                    i\n                  \n                \n                \u2212\n                b\n                )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0052", "text": "b\n                )\n                \u2265\n                1\n                \n                \u2200\n                i\n                \u2208\n                {\n                1\n                ,\n                \u2026\n                ,\n                n\n                }", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0053", "text": "{\\displaystyle {\\begin{aligned}&{\\underset {\\mathbf {w} ,\\;b}{\\operatorname {minimize} }}&&{\\frac {1}{2}}\\|\\mathbf {w} \\|^{2}\\\\&{\\text{subject to}}&&y_{i}(\\mathbf {w} ^{\\top }\\mathbf {x} _{i}-b)\\geq 1\\quad \\forall i\\in \\{1,\\dots ,n\\}\\end{aligned}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0054", "text": "The \n  \n    \n      \n        \n          w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n and \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}\n  \n that solve this problem determine the final classifier, \n  \n    \n      \n        \n          x\n        \n        \u21a6\n        sgn\n        \u2061\n        (\n        \n          \n            w\n          \n          \n            \n              T\n            \n          \n        \n        \n          x\n        \n        \u2212\n        b\n        )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0055", "text": "\u2212\n        b\n        )\n      \n    \n    {\\displaystyle \\mathbf {x} \\mapsto \\operatorname {sgn}(\\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} -b)}\n  \n, where \n  \n    \n      \n        sgn\n        \u2061\n        (\n        \u22c5\n        )\n      \n    \n    {\\displaystyle \\operatorname {sgn}(\\cdot )}\n  \n is the sign function.\nAn important consequence of this geometric description is that the max-margin hyperplane is completely determined by those \n  \n    \n      \n        \n          \n            x", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0056", "text": "x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n that lie nearest to it (explained below). These \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n are called support vectors.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0057", "text": "Soft-margin\nTo extend SVM to cases in which the data are not linearly separable, the hinge loss function is helpful", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0058", "text": "max\n        \n          (\n          \n            0\n            ,\n            1\n            \u2212\n            \n              y\n              \n                i\n              \n            \n            (\n            \n              \n                w\n              \n              \n                \n                  T\n                \n              \n            \n            \n              \n                x\n              \n              \n                i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0059", "text": "i\n              \n            \n            \u2212\n            b\n            )\n          \n          )\n        \n        .\n      \n    \n    {\\displaystyle \\max \\left(0,1-y_{i}(\\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-b)\\right).}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0060", "text": "Note that \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}\n  \n is the i-th target (i.e., in this case, 1 or \u22121), and \n  \n    \n      \n        \n          \n            w\n          \n          \n            \n              T\n            \n          \n        \n        \n          \n            x\n          \n          \n            i\n          \n        \n        \u2212\n        b", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0061", "text": "\u2212\n        b\n      \n    \n    {\\displaystyle \\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-b}\n  \n is the i-th output.\nThis function is zero if the constraint in (1) is satisfied, in other words, if \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0062", "text": "{\\displaystyle \\mathbf {x} _{i}}\n  \n lies on the correct side of the margin. For data on the wrong side of the margin, the function's value is proportional to the distance from the margin.\nThe goal of the optimization then is to minimize:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0063", "text": "\u2016\n        \n          w\n        \n        \n          \u2016\n          \n            2\n          \n        \n        +\n        C\n        \n          [\n          \n            \n              \n                1\n                n\n              \n            \n            \n              \u2211\n              \n                i\n                =\n                1\n              \n              \n                n\n              \n            \n            max\n            \n              (", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0064", "text": "(\n              \n                0\n                ,\n                1\n                \u2212\n                \n                  y\n                  \n                    i\n                  \n                \n                (\n                \n                  \n                    w\n                  \n                  \n                    \n                      T", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0065", "text": "x\n                  \n                  \n                    i\n                  \n                \n                \u2212\n                b\n                )\n              \n              )\n            \n          \n          ]\n        \n        ,\n      \n    \n    {\\displaystyle \\lVert \\mathbf {w} \\rVert ^{2}+C\\left[{\\frac {1}{n}}\\sum _{i=1}^{n}\\max \\left(0,1-y_{i}(\\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-b)\\right)\\right],}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0066", "text": "where the parameter \n  \n    \n      \n        C\n        >\n        0\n      \n    \n    {\\displaystyle C>0}\n  \n determines the trade-off between increasing the margin size and ensuring that the \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0067", "text": "{\\displaystyle \\mathbf {x} _{i}}\n  \n lie on the correct side of the margin (Note we can add a weight to either term in the equation above). By deconstructing the hinge loss, this optimization problem can be formulated into the following:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0068", "text": "minimize\n                    \n                      \n                        w\n                      \n                      ,\n                      \n                      b\n                      ,\n                      \n                      \n                        \u03b6", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0069", "text": "\u2016\n                \n                  w\n                \n                \n                  \u2016\n                  \n                    2\n                  \n                  \n                    2\n                  \n                \n                +\n                C\n                \n                  \u2211\n                  \n                    i\n                    =\n                    1", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0070", "text": "=\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  \u03b6\n                  \n                    i\n                  \n                \n              \n            \n            \n              \n              \n                \n                  subject to\n                \n              \n              \n              \n                \n                  y", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0071", "text": "y\n                  \n                    i\n                  \n                \n                (\n                \n                  \n                    w\n                  \n                  \n                    \u22a4\n                  \n                \n                \n                  \n                    x\n                  \n                  \n                    i\n                  \n                \n                \u2212\n                b\n                )\n                \u2265", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0072", "text": ")\n                \u2265\n                1\n                \u2212\n                \n                  \u03b6\n                  \n                    i\n                  \n                \n                ,\n                \n                \n                  \u03b6\n                  \n                    i\n                  \n                \n                \u2265\n                0\n                \n                \u2200\n                i\n                \u2208\n                {\n                1\n                ,", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0073", "text": "1\n                ,\n                \u2026\n                ,\n                n\n                }\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}&{\\underset {\\mathbf {w} ,\\;b,\\;\\mathbf {\\zeta } }{\\operatorname {minimize} }}&&\\|\\mathbf {w} \\|_{2}^{2}+C\\sum _{i=1}^{n}\\zeta _{i}\\\\&{\\text{subject to}}&&y_{i}(\\mathbf {w} ^{\\top }\\mathbf {x} _{i}-b)\\geq 1-\\zeta _{i},\\quad \\zeta _{i}\\geq 0\\quad \\forall i\\in \\{1,\\dots ,n\\}\\end{aligned}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0074", "text": "Thus, for large values of \n  \n    \n      \n        C\n      \n    \n    {\\displaystyle C}\n  \n, it will behave similar to the hard-margin SVM, if the input data are linearly classifiable, but will still learn if a classification rule is viable or not.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0075", "text": "Nonlinear kernels", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0076", "text": "The original maximum-margin hyperplane algorithm proposed by Vapnik in 1963 constructed a linear classifier. However, in 1992, Bernhard Boser, Isabelle Guyon and Vladimir Vapnik suggested a way to create nonlinear classifiers by applying the kernel trick (originally proposed by Aizerman et al.) to maximum-margin hyperplanes. The kernel trick, where dot products are replaced by kernels, is easily derived in the dual representation of the SVM problem", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0077", "text": ". This allows the algorithm to fit the maximum-margin hyperplane in a transformed feature space. The transformation may be nonlinear and the transformed space high-dimensional; although the classifier is a hyperplane in the transformed feature space, it may be nonlinear in the original input space.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0078", "text": "It is noteworthy that working in a higher-dimensional feature space increases the generalization error of support vector machines, although given enough samples the algorithm still performs well.\nSome common kernels include:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0079", "text": "Polynomial (homogeneous): \n  \n    \n      \n        k\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        ,\n        \n          \n            x\n          \n          \n            j\n          \n        \n        )\n        =\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        \u22c5\n        \n          \n            x\n          \n          \n            j\n          \n        \n        \n          )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0080", "text": ")\n          \n            d\n          \n        \n      \n    \n    {\\displaystyle k(\\mathbf {x} _{i},\\mathbf {x} _{j})=(\\mathbf {x} _{i}\\cdot \\mathbf {x} _{j})^{d}}\n  \n. Particularly, when \n  \n    \n      \n        d\n        =\n        1\n      \n    \n    {\\displaystyle d=1}\n  \n, this becomes the linear kernel.\nPolynomial (inhomogeneous): \n  \n    \n      \n        k\n        (\n        \n          \n            x\n          \n          \n            i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0081", "text": "i\n          \n        \n        ,\n        \n          \n            x\n          \n          \n            j\n          \n        \n        )\n        =\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        \u22c5\n        \n          \n            x\n          \n          \n            j\n          \n        \n        +\n        r\n        \n          )\n          \n            d", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0082", "text": "d\n          \n        \n      \n    \n    {\\displaystyle k(\\mathbf {x} _{i},\\mathbf {x} _{j})=(\\mathbf {x} _{i}\\cdot \\mathbf {x} _{j}+r)^{d}}\n  \n.\nGaussian radial basis function: \n  \n    \n      \n        k\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        ,\n        \n          \n            x\n          \n          \n            j\n          \n        \n        )\n        =\n        exp\n        \u2061\n        \n          (", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0083", "text": "\u2061\n        \n          (\n          \n            \u2212\n            \u03b3\n            \n              \n                \u2016\n                \n                  \n                    \n                      x\n                    \n                    \n                      i\n                    \n                  \n                  \u2212\n                  \n                    \n                      x\n                    \n                    \n                      j", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0084", "text": "\u2016\n              \n              \n                2\n              \n            \n          \n          )\n        \n      \n    \n    {\\displaystyle k(\\mathbf {x} _{i},\\mathbf {x} _{j})=\\exp \\left(-\\gamma \\left\\|\\mathbf {x} _{i}-\\mathbf {x} _{j}\\right\\|^{2}\\right)}\n  \n for \n  \n    \n      \n        \u03b3\n        >\n        0\n      \n    \n    {\\displaystyle \\gamma >0}\n  \n. Sometimes parametrized using \n  \n    \n      \n        \u03b3\n        =", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0085", "text": "\u03b3\n        =\n        1\n        \n          /\n        \n        (\n        2\n        \n          \u03c3\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle \\gamma =1/(2\\sigma ^{2})}\n  \n.\nSigmoid function (Hyperbolic tangent): \n  \n    \n      \n        k\n        (\n        \n          \n            x\n            \n              i\n            \n          \n        \n        ,\n        \n          \n            x\n            \n              j", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0086", "text": "j\n            \n          \n        \n        )\n        =\n        tanh\n        \u2061\n        (\n        \u03ba\n        \n          \n            x\n          \n          \n            i\n          \n        \n        \u22c5\n        \n          \n            x\n          \n          \n            j\n          \n        \n        +\n        c\n        )\n      \n    \n    {\\displaystyle k(\\mathbf {x_{i}} ,\\mathbf {x_{j}} )=\\tanh(\\kappa \\mathbf {x} _{i}\\cdot \\mathbf {x} _{j}+c)}\n  \n for some (not every)", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0087", "text": "for some (not every) \n  \n    \n      \n        \u03ba\n        >\n        0\n      \n    \n    {\\displaystyle \\kappa >0}\n  \n and \n  \n    \n      \n        c\n        <\n        0\n      \n    \n    {\\displaystyle c<0}\n  \n.\nThe kernel is related to the transform \n  \n    \n      \n        \u03c6\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle \\varphi (\\mathbf {x} _{i})}\n  \n by the equation \n  \n    \n      \n        k\n        (", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0088", "text": "k\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        ,\n        \n          \n            x\n          \n          \n            j\n          \n        \n        )\n        =\n        \u03c6\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        )\n        \u22c5\n        \u03c6\n        (\n        \n          \n            x\n          \n          \n            j\n          \n        \n        )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0089", "text": "j\n          \n        \n        )\n      \n    \n    {\\displaystyle k(\\mathbf {x} _{i},\\mathbf {x} _{j})=\\varphi (\\mathbf {x} _{i})\\cdot \\varphi (\\mathbf {x} _{j})}\n  \n. The value w is also in the transformed space, with \n  \n    \n      \n        \n          w\n        \n        =\n        \n          \u2211\n          \n            i\n          \n        \n        \n          \u03b1\n          \n            i\n          \n        \n        \n          y\n          \n            i\n          \n        \n        \u03c6", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0090", "text": "i\n          \n        \n        \u03c6\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        )\n      \n    \n    {\\textstyle \\mathbf {w} =\\sum _{i}\\alpha _{i}y_{i}\\varphi (\\mathbf {x} _{i})}\n  \n. Dot products with w for classification can again be computed by the kernel trick, i.e. \n  \n    \n      \n        \n          w\n        \n        \u22c5\n        \u03c6\n        (\n        \n          x\n        \n        )\n        =\n        \n          \u2211", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0091", "text": "=\n        \n          \u2211\n          \n            i\n          \n        \n        \n          \u03b1\n          \n            i\n          \n        \n        \n          y\n          \n            i\n          \n        \n        k\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        ,\n        \n          x\n        \n        )\n      \n    \n    {\\textstyle \\mathbf {w} \\cdot \\varphi (\\mathbf {x} )=\\sum _{i}\\alpha _{i}y_{i}k(\\mathbf {x} _{i},\\mathbf {x} )}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0092", "text": ".", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0093", "text": "Computing the SVM classifier\nComputing the (soft-margin) SVM classifier amounts to minimizing an expression of the form", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0094", "text": "We focus on the soft-margin classifier since, as noted above, choosing a sufficiently small value for \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n yields the hard-margin classifier for linearly classifiable input data. The classical approach, which involves reducing (2) to a quadratic programming problem, is detailed below. Then, more recent approaches such as sub-gradient descent and coordinate descent will be discussed.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0095", "text": "Primal\nMinimizing (2) can be rewritten as a constrained optimization problem with a differentiable objective function in the following way.\nFor each \n  \n    \n      \n        i\n        \u2208\n        {\n        1\n        ,\n        \n        \u2026\n        ,\n        \n        n\n        }\n      \n    \n    {\\displaystyle i\\in \\{1,\\,\\ldots ,\\,n\\}}\n  \n we introduce a variable \n  \n    \n      \n        \n          \u03b6\n          \n            i\n          \n        \n        =\n        max\n        \n          (", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0096", "text": "max\n        \n          (\n          \n            0\n            ,\n            1\n            \u2212\n            \n              y\n              \n                i\n              \n            \n            (\n            \n              \n                w\n              \n              \n                \n                  T\n                \n              \n            \n            \n              \n                x\n              \n              \n                i\n              \n            \n            \u2212", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0097", "text": "\u2212\n            b\n            )\n          \n          )\n        \n      \n    \n    {\\displaystyle \\zeta _{i}=\\max \\left(0,1-y_{i}(\\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-b)\\right)}\n  \n. Note that \n  \n    \n      \n        \n          \u03b6\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\zeta _{i}}\n  \n is the smallest nonnegative number satisfying \n  \n    \n      \n        \n          y\n          \n            i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0098", "text": "i\n          \n        \n        (\n        \n          \n            w\n          \n          \n            \n              T\n            \n          \n        \n        \n          \n            x\n          \n          \n            i\n          \n        \n        \u2212\n        b\n        )\n        \u2265\n        1\n        \u2212\n        \n          \u03b6\n          \n            i\n          \n        \n        .", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0099", "text": ".\n      \n    \n    {\\displaystyle y_{i}(\\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-b)\\geq 1-\\zeta _{i}.}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0100", "text": "Thus we can rewrite the optimization problem as follows", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0101", "text": "minimize \n                \n                \n                  \n                    1\n                    n\n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0102", "text": "\u03b6\n                  \n                    i\n                  \n                \n                +\n                \u03bb\n                \u2016\n                \n                  w\n                \n                \n                  \u2016\n                  \n                    2\n                  \n                \n              \n            \n            \n              \n              \n                \n                  subject to", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0103", "text": "subject to \n                \n                \n                  y\n                  \n                    i\n                  \n                \n                \n                  (\n                  \n                    \n                      \n                        w\n                      \n                      \n                        \n                          T", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0104", "text": "x\n                      \n                      \n                        i\n                      \n                    \n                    \u2212\n                    b\n                  \n                  )\n                \n                \u2265\n                1\n                \u2212\n                \n                  \u03b6\n                  \n                    i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0105", "text": "and \n                \n                \n                \n                  \u03b6\n                  \n                    i\n                  \n                \n                \u2265\n                0\n                ,\n                \n                \n                  for all \n                \n                i\n                .", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0106", "text": "{\\displaystyle {\\begin{aligned}&{\\text{minimize }}{\\frac {1}{n}}\\sum _{i=1}^{n}\\zeta _{i}+\\lambda \\|\\mathbf {w} \\|^{2}\\\\[0.5ex]&{\\text{subject to }}y_{i}\\left(\\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-b\\right)\\geq 1-\\zeta _{i}\\,{\\text{ and }}\\,\\zeta _{i}\\geq 0,\\,{\\text{for all }}i.\\end{aligned}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0107", "text": "This is called the primal problem.\n\nDual\nBy solving for the Lagrangian dual of the above problem, one obtains the simplified problem", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0108", "text": "maximize\n                \n                \n                \n                f\n                (\n                \n                  c\n                  \n                    1\n                  \n                \n                \u2026\n                \n                  c\n                  \n                    n\n                  \n                \n                )\n                =", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0109", "text": ")\n                =\n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  c\n                  \n                    i\n                  \n                \n                \u2212\n                \n                  \n                    1\n                    2", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0110", "text": "2\n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  \u2211\n                  \n                    j\n                    =\n                    1\n                  \n                  \n                    n", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0111", "text": "y\n                  \n                    i\n                  \n                \n                \n                  c\n                  \n                    i\n                  \n                \n                (\n                \n                  \n                    x\n                  \n                  \n                    i\n                  \n                  \n                    \n                      T", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0112", "text": "T\n                    \n                  \n                \n                \n                  \n                    x\n                  \n                  \n                    j\n                  \n                \n                )\n                \n                  y\n                  \n                    j\n                  \n                \n                \n                  c\n                  \n                    j", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0113", "text": "j\n                  \n                \n                ,\n              \n            \n            \n              \n              \n                \n                  subject to \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  c", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0114", "text": "c\n                  \n                    i\n                  \n                \n                \n                  y\n                  \n                    i\n                  \n                \n                =\n                0\n                ,\n                \n                \n                  and \n                \n                0\n                \u2264\n                \n                  c\n                  \n                    i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0115", "text": "\u2264\n                \n                  \n                    1\n                    \n                      2\n                      n\n                      \u03bb\n                    \n                  \n                \n                \n                \n                  for all \n                \n                i\n                .", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0116", "text": "{\\displaystyle {\\begin{aligned}&{\\text{maximize}}\\,\\,f(c_{1}\\ldots c_{n})=\\sum _{i=1}^{n}c_{i}-{\\frac {1}{2}}\\sum _{i=1}^{n}\\sum _{j=1}^{n}y_{i}c_{i}(\\mathbf {x} _{i}^{\\mathsf {T}}\\mathbf {x} _{j})y_{j}c_{j},\\\\&{\\text{subject to }}\\sum _{i=1}^{n}c_{i}y_{i}=0,\\,{\\text{and }}0\\leq c_{i}\\leq {\\frac {1}{2n\\lambda }}\\;{\\text{for all }}i.\\end{aligned}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0117", "text": "This is called the dual problem. Since the dual maximization problem is a quadratic function of the \n  \n    \n      \n        \n          c\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle c_{i}}\n  \n subject to linear constraints, it is efficiently solvable by quadratic programming algorithms.\nHere, the variables \n  \n    \n      \n        \n          c\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle c_{i}}\n  \n are defined such that", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0118", "text": "w\n        \n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          c\n          \n            i\n          \n        \n        \n          y\n          \n            i\n          \n        \n        \n          \n            x\n          \n          \n            i\n          \n        \n        .", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0119", "text": ".\n      \n    \n    {\\displaystyle \\mathbf {w} =\\sum _{i=1}^{n}c_{i}y_{i}\\mathbf {x} _{i}.}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0120", "text": "Moreover, \n  \n    \n      \n        \n          c\n          \n            i\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle c_{i}=0}\n  \n exactly when \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n lies on the correct side of the margin, and \n  \n    \n      \n        0\n        <\n        \n          c\n          \n            i\n          \n        \n        <\n        (", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0121", "text": "<\n        (\n        2\n        n\n        \u03bb\n        \n          )\n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle 0<c_{i}<(2n\\lambda )^{-1}}\n  \n  when \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n lies on the margin's boundary. It follows that \n  \n    \n      \n        \n          w", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0122", "text": "w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n can be written as a linear combination of the support vectors.\nThe offset, \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}\n  \n, can be recovered by finding an \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n on the margin's boundary and solving", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0123", "text": "y\n          \n            i\n          \n        \n        (\n        \n          \n            w\n          \n          \n            \n              T\n            \n          \n        \n        \n          \n            x\n          \n          \n            i\n          \n        \n        \u2212\n        b\n        )\n        =\n        1\n        \n        \u27fa\n        \n        b\n        =\n        \n          \n            w\n          \n          \n            \n              T", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0124", "text": "T\n            \n          \n        \n        \n          \n            x\n          \n          \n            i\n          \n        \n        \u2212\n        \n          y\n          \n            i\n          \n        \n        .\n      \n    \n    {\\displaystyle y_{i}(\\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-b)=1\\iff b=\\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-y_{i}.}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0125", "text": "(Note that \n  \n    \n      \n        \n          y\n          \n            i\n          \n          \n            \u2212\n            1\n          \n        \n        =\n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}^{-1}=y_{i}}\n  \n since \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n        =\n        \u00b1\n        1\n      \n    \n    {\\displaystyle y_{i}=\\pm 1}\n  \n.)", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0126", "text": "Kernel trick\nSuppose now that we would like to learn a nonlinear classification rule which corresponds to a linear classification rule for the transformed data points \n  \n    \n      \n        \u03c6\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        )\n        .\n      \n    \n    {\\displaystyle \\varphi (\\mathbf {x} _{i}).}\n  \n Moreover, we are given a kernel function \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n which satisfies", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0127", "text": "{\\displaystyle k}\n  \n which satisfies \n  \n    \n      \n        k\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        ,\n        \n          \n            x\n          \n          \n            j\n          \n        \n        )\n        =\n        \u03c6\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        )\n        \u22c5\n        \u03c6\n        (\n        \n          \n            x", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0128", "text": "x\n          \n          \n            j\n          \n        \n        )\n      \n    \n    {\\displaystyle k(\\mathbf {x} _{i},\\mathbf {x} _{j})=\\varphi (\\mathbf {x} _{i})\\cdot \\varphi (\\mathbf {x} _{j})}\n  \n.\nWe know the classification vector \n  \n    \n      \n        \n          w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n in the transformed space satisfies", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0129", "text": "w\n        \n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          c\n          \n            i\n          \n        \n        \n          y\n          \n            i\n          \n        \n        \u03c6\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        )\n        ,", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0130", "text": ")\n        ,\n      \n    \n    {\\displaystyle \\mathbf {w} =\\sum _{i=1}^{n}c_{i}y_{i}\\varphi (\\mathbf {x} _{i}),}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0131", "text": "where, the \n  \n    \n      \n        \n          c\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle c_{i}}\n  \n are obtained by solving the optimization problem", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0132", "text": "maximize\n                \n                \n                \n                f\n                (\n                \n                  c\n                  \n                    1\n                  \n                \n                \u2026\n                \n                  c\n                  \n                    n\n                  \n                \n                )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0133", "text": ")\n              \n              \n                \n                =\n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  c\n                  \n                    i\n                  \n                \n                \u2212\n                \n                  \n                    1", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0134", "text": "1\n                    2\n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  \u2211\n                  \n                    j\n                    =\n                    1\n                  \n                  \n                    n", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0135", "text": "n\n                  \n                \n                \n                  y\n                  \n                    i\n                  \n                \n                \n                  c\n                  \n                    i\n                  \n                \n                (\n                \u03c6\n                (\n                \n                  \n                    x\n                  \n                  \n                    i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0136", "text": "i\n                  \n                \n                )\n                \u22c5\n                \u03c6\n                (\n                \n                  \n                    x\n                  \n                  \n                    j\n                  \n                \n                )\n                )\n                \n                  y\n                  \n                    j\n                  \n                \n                \n                  c", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0137", "text": "c\n                  \n                    j\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  c\n                  \n                    i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0138", "text": "i\n                  \n                \n                \u2212\n                \n                  \n                    1\n                    2\n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  \u2211", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0139", "text": "\u2211\n                  \n                    j\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  y\n                  \n                    i\n                  \n                \n                \n                  c\n                  \n                    i\n                  \n                \n                k\n                (", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0140", "text": "(\n                \n                  \n                    x\n                  \n                  \n                    i\n                  \n                \n                ,\n                \n                  \n                    x\n                  \n                  \n                    j\n                  \n                \n                )\n                \n                  y\n                  \n                    j", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0141", "text": "c\n                  \n                    j\n                  \n                \n              \n            \n            \n              \n                \n                  subject to \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0142", "text": "c\n                  \n                    i\n                  \n                \n                \n                  y\n                  \n                    i\n                  \n                \n              \n              \n                \n                =\n                0\n                ,\n                \n                \n                  and \n                \n                0\n                \u2264\n                \n                  c", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0143", "text": "c\n                  \n                    i\n                  \n                \n                \u2264\n                \n                  \n                    1\n                    \n                      2\n                      n\n                      \u03bb\n                    \n                  \n                \n                \n                \n                  for all \n                \n                i\n                .", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0144", "text": "{\\displaystyle {\\begin{aligned}{\\text{maximize}}\\,\\,f(c_{1}\\ldots c_{n})&=\\sum _{i=1}^{n}c_{i}-{\\frac {1}{2}}\\sum _{i=1}^{n}\\sum _{j=1}^{n}y_{i}c_{i}(\\varphi (\\mathbf {x} _{i})\\cdot \\varphi (\\mathbf {x} _{j}))y_{j}c_{j}\\\\&=\\sum _{i=1}^{n}c_{i}-{\\frac {1}{2}}\\sum _{i=1}^{n}\\sum _{j=1}^{n}y_{i}c_{i}k(\\mathbf {x} _{i},\\mathbf {x} _{j})y_{j}c_{j}\\\\{\\text{subject to }}\\sum _{i=1}^{n}c_{i}y_{i}&=0,\\,{\\text{and }}0\\leq c_{i}\\leq {\\frac {1}{2n\\lambda }}\\;{\\text{for all }}i.\\end{aligned}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0145", "text": "The coefficients \n  \n    \n      \n        \n          c\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle c_{i}}\n  \n can be solved for using quadratic programming, as before. Again, we can find some index \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n such that \n  \n    \n      \n        0\n        <\n        \n          c\n          \n            i\n          \n        \n        <\n        (\n        2\n        n\n        \u03bb\n        \n          )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0146", "text": "\u03bb\n        \n          )\n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle 0<c_{i}<(2n\\lambda )^{-1}}\n  \n, so that \n  \n    \n      \n        \u03c6\n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle \\varphi (\\mathbf {x} _{i})}\n  \n lies on the boundary of the margin in the transformed space, and then solve", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0147", "text": "b\n                =\n                \n                  \n                    w\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \u03c6\n                (\n                \n                  \n                    x\n                  \n                  \n                    i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0148", "text": ")\n                \u2212\n                \n                  y\n                  \n                    i\n                  \n                \n              \n              \n                \n                =\n                \n                  [\n                  \n                    \n                      \u2211\n                      \n                        j\n                        =\n                        1", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0149", "text": "n\n                      \n                    \n                    \n                      c\n                      \n                        j\n                      \n                    \n                    \n                      y\n                      \n                        j\n                      \n                    \n                    \u03c6\n                    (", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0150", "text": "x\n                      \n                      \n                        j\n                      \n                    \n                    )\n                    \u22c5\n                    \u03c6\n                    (\n                    \n                      \n                        x\n                      \n                      \n                        i\n                      \n                    \n                    )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0151", "text": ")\n                  \n                  ]\n                \n                \u2212\n                \n                  y\n                  \n                    i\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  [\n                  \n                    \n                      \u2211\n                      \n                        j", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0152", "text": "j\n                        =\n                        1\n                      \n                      \n                        n\n                      \n                    \n                    \n                      c\n                      \n                        j\n                      \n                    \n                    \n                      y\n                      \n                        j", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0153", "text": "k\n                    (\n                    \n                      \n                        x\n                      \n                      \n                        j\n                      \n                    \n                    ,\n                    \n                      \n                        x\n                      \n                      \n                        i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0154", "text": ")\n                  \n                  ]\n                \n                \u2212\n                \n                  y\n                  \n                    i\n                  \n                \n                .", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0155", "text": "{\\displaystyle {\\begin{aligned}b=\\mathbf {w} ^{\\mathsf {T}}\\varphi (\\mathbf {x} _{i})-y_{i}&=\\left[\\sum _{j=1}^{n}c_{j}y_{j}\\varphi (\\mathbf {x} _{j})\\cdot \\varphi (\\mathbf {x} _{i})\\right]-y_{i}\\\\&=\\left[\\sum _{j=1}^{n}c_{j}y_{j}k(\\mathbf {x} _{j},\\mathbf {x} _{i})\\right]-y_{i}.\\end{aligned}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0156", "text": "Finally,", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0157", "text": "z\n        \n        \u21a6\n        sgn\n        \u2061\n        (\n        \n          \n            w\n          \n          \n            \n              T\n            \n          \n        \n        \u03c6\n        (\n        \n          z\n        \n        )\n        \u2212\n        b\n        )\n        =\n        sgn\n        \u2061\n        \n          (\n          \n            \n              [\n              \n                \n                  \u2211\n                  \n                    i", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0158", "text": "i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  c\n                  \n                    i\n                  \n                \n                \n                  y\n                  \n                    i\n                  \n                \n                k\n                (", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0159", "text": "x\n                  \n                  \n                    i\n                  \n                \n                ,\n                \n                  z\n                \n                )\n              \n              ]\n            \n            \u2212\n            b\n          \n          )\n        \n        .", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0160", "text": ")\n        \n        .\n      \n    \n    {\\displaystyle \\mathbf {z} \\mapsto \\operatorname {sgn}(\\mathbf {w} ^{\\mathsf {T}}\\varphi (\\mathbf {z} )-b)=\\operatorname {sgn} \\left(\\left[\\sum _{i=1}^{n}c_{i}y_{i}k(\\mathbf {x} _{i},\\mathbf {z} )\\right]-b\\right).}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0161", "text": "Modern methods\nRecent algorithms for finding the SVM classifier include sub-gradient descent and coordinate descent. Both techniques have proven to offer significant advantages over the traditional approach when dealing with large, sparse datasets\u2014sub-gradient methods are especially efficient when there are many training examples, and coordinate descent when the dimension of the feature space is high.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0162", "text": "Sub-gradient descent\nSub-gradient descent algorithms for the SVM work directly with the expression", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0163", "text": "f\n        (\n        \n          w\n        \n        ,\n        b\n        )\n        =\n        \n          [\n          \n            \n              \n                1\n                n\n              \n            \n            \n              \u2211\n              \n                i\n                =\n                1\n              \n              \n                n\n              \n            \n            max\n            \n              (\n              \n                0\n                ,", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0164", "text": "0\n                ,\n                1\n                \u2212\n                \n                  y\n                  \n                    i\n                  \n                \n                (\n                \n                  \n                    w\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  \n                    x", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0165", "text": "i\n                  \n                \n                \u2212\n                b\n                )\n              \n              )\n            \n          \n          ]\n        \n        +\n        \u03bb\n        \u2016\n        \n          w\n        \n        \n          \u2016\n          \n            2\n          \n        \n        .", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0166", "text": ".\n      \n    \n    {\\displaystyle f(\\mathbf {w} ,b)=\\left[{\\frac {1}{n}}\\sum _{i=1}^{n}\\max \\left(0,1-y_{i}(\\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} _{i}-b)\\right)\\right]+\\lambda \\|\\mathbf {w} \\|^{2}.}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0167", "text": "Note that \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is a convex function of \n  \n    \n      \n        \n          w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n and \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0168", "text": "b\n      \n    \n    {\\displaystyle b}\n  \n. As such, traditional gradient descent (or SGD) methods can be adapted, where instead of taking a step in the direction of the function's gradient, a step is taken in the direction of a vector selected from the function's sub-gradient. This approach has the advantage that, for certain implementations, the number of iterations does not scale with \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n, the number of data points.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0169", "text": "Coordinate descent\nCoordinate descent algorithms for the SVM work from the dual problem", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0170", "text": "maximize\n                \n                \n                \n                f\n                (\n                \n                  c\n                  \n                    1\n                  \n                \n                \u2026\n                \n                  c\n                  \n                    n\n                  \n                \n                )\n                =", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0171", "text": ")\n                =\n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  c\n                  \n                    i\n                  \n                \n                \u2212\n                \n                  \n                    1\n                    2", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0172", "text": "2\n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  \u2211\n                  \n                    j\n                    =\n                    1\n                  \n                  \n                    n", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0173", "text": "y\n                  \n                    i\n                  \n                \n                \n                  c\n                  \n                    i\n                  \n                \n                (\n                \n                  x\n                  \n                    i\n                  \n                \n                \u22c5\n                \n                  x\n                  \n                    j", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0174", "text": "j\n                  \n                \n                )\n                \n                  y\n                  \n                    j\n                  \n                \n                \n                  c\n                  \n                    j\n                  \n                \n                ,\n              \n            \n            \n              \n              \n                \n                  subject to", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0175", "text": "\u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  c\n                  \n                    i\n                  \n                \n                \n                  y\n                  \n                    i\n                  \n                \n                =\n                0", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0176", "text": "=\n                0\n                ,\n                \n                \n                  and \n                \n                0\n                \u2264\n                \n                  c\n                  \n                    i\n                  \n                \n                \u2264\n                \n                  \n                    1\n                    \n                      2\n                      n\n                      \u03bb", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0177", "text": "for all \n                \n                i\n                .", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0178", "text": "{\\displaystyle {\\begin{aligned}&{\\text{maximize}}\\,\\,f(c_{1}\\ldots c_{n})=\\sum _{i=1}^{n}c_{i}-{\\frac {1}{2}}\\sum _{i=1}^{n}\\sum _{j=1}^{n}y_{i}c_{i}(x_{i}\\cdot x_{j})y_{j}c_{j},\\\\&{\\text{subject to }}\\sum _{i=1}^{n}c_{i}y_{i}=0,\\,{\\text{and }}0\\leq c_{i}\\leq {\\frac {1}{2n\\lambda }}\\;{\\text{for all }}i.\\end{aligned}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0179", "text": "For each \n  \n    \n      \n        i\n        \u2208\n        {\n        1\n        ,\n        \n        \u2026\n        ,\n        \n        n\n        }\n      \n    \n    {\\displaystyle i\\in \\{1,\\,\\ldots ,\\,n\\}}\n  \n, iteratively, the coefficient \n  \n    \n      \n        \n          c\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle c_{i}}\n  \n is adjusted in the direction of \n  \n    \n      \n        \u2202\n        f\n        \n          /\n        \n        \u2202\n        \n          c", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0180", "text": "\u2202\n        \n          c\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\partial f/\\partial c_{i}}\n  \n. Then, the resulting vector of coefficients \n  \n    \n      \n        (\n        \n          c\n          \n            1\n          \n          \u2032\n        \n        ,\n        \n        \u2026\n        ,\n        \n        \n          c\n          \n            n\n          \n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle (c_{1}',\\,\\ldots ,\\,c_{n}')}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0181", "text": "is projected onto the nearest vector of coefficients that satisfies the given constraints. (Typically Euclidean distances are used.) The process is then repeated until a near-optimal vector of coefficients is obtained. The resulting algorithm is extremely fast in practice, although few performance guarantees have been proven.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0182", "text": "Empirical risk minimization\nThe soft-margin support vector machine described above is an example of an empirical risk minimization (ERM) algorithm for the hinge loss. Seen this way, support vector machines belong to a natural class of algorithms for statistical inference, and many of its unique features are due to the behavior of the hinge loss. This perspective can provide further insight into how and why SVMs work, and allow us to better analyze their statistical properties.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0183", "text": "Risk minimization\nIn supervised learning, one is given a set of training examples \n  \n    \n      \n        \n          X\n          \n            1\n          \n        \n        \u2026\n        \n          X\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle X_{1}\\ldots X_{n}}\n  \n with labels \n  \n    \n      \n        \n          y\n          \n            1\n          \n        \n        \u2026\n        \n          y\n          \n            n", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0184", "text": "n\n          \n        \n      \n    \n    {\\displaystyle y_{1}\\ldots y_{n}}\n  \n, and wishes to predict \n  \n    \n      \n        \n          y\n          \n            n\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle y_{n+1}}\n  \n given \n  \n    \n      \n        \n          X\n          \n            n\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle X_{n+1}}\n  \n. To do so one forms a hypothesis, \n  \n    \n      \n        f", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0185", "text": "f\n      \n    \n    {\\displaystyle f}\n  \n, such that \n  \n    \n      \n        f\n        (\n        \n          X\n          \n            n\n            +\n            1\n          \n        \n        )\n      \n    \n    {\\displaystyle f(X_{n+1})}\n  \n is a \"good\" approximation of \n  \n    \n      \n        \n          y\n          \n            n\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle y_{n+1}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0186", "text": "{\\displaystyle y_{n+1}}\n  \n. A \"good\" approximation is usually defined with the help of a loss function, \n  \n    \n      \n        \u2113\n        (\n        y\n        ,\n        z\n        )\n      \n    \n    {\\displaystyle \\ell (y,z)}\n  \n, which characterizes how bad \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n is as a prediction of \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n. We would then like to choose a hypothesis that minimizes the expected risk:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0187", "text": "\u03b5\n        (\n        f\n        )\n        =\n        \n          E\n        \n        \n          [\n          \n            \u2113\n            (\n            \n              y\n              \n                n\n                +\n                1\n              \n            \n            ,\n            f\n            (\n            \n              X\n              \n                n\n                +\n                1\n              \n            \n            )\n            )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0188", "text": ")\n            )\n          \n          ]\n        \n        .\n      \n    \n    {\\displaystyle \\varepsilon (f)=\\mathbb {E} \\left[\\ell (y_{n+1},f(X_{n+1}))\\right].}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0189", "text": "In most cases, we don't know the joint distribution of \n  \n    \n      \n        \n          X\n          \n            n\n            +\n            1\n          \n        \n        ,\n        \n        \n          y\n          \n            n\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle X_{n+1},\\,y_{n+1}}\n  \n outright. In these cases, a common strategy is to choose the hypothesis that minimizes the empirical risk:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0190", "text": "\u03b5\n              ^\n            \n          \n        \n        (\n        f\n        )\n        =\n        \n          \n            1\n            n\n          \n        \n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            n\n          \n        \n        \u2113\n        (\n        \n          y\n          \n            k\n          \n        \n        ,\n        f\n        (\n        \n          X", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0191", "text": "(\n        \n          X\n          \n            k\n          \n        \n        )\n        )\n        .\n      \n    \n    {\\displaystyle {\\hat {\\varepsilon }}(f)={\\frac {1}{n}}\\sum _{k=1}^{n}\\ell (y_{k},f(X_{k})).}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0192", "text": "Under certain assumptions about the sequence of random variables \n  \n    \n      \n        \n          X\n          \n            k\n          \n        \n        ,\n        \n        \n          y\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle X_{k},\\,y_{k}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0193", "text": "{\\displaystyle X_{k},\\,y_{k}}\n  \n (for example, that they are generated by a finite Markov process), if the set of hypotheses being considered is small enough, the minimizer of the empirical risk will closely approximate the minimizer of the expected risk as \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n grows large. This approach is called empirical risk minimization, or ERM.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0194", "text": "Regularization and stability\nIn order for the minimization problem to have a well-defined solution, we have to place constraints on the set \n  \n    \n      \n        \n          \n            H\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {H}}}\n  \n of hypotheses being considered. If \n  \n    \n      \n        \n          \n            H\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {H}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0195", "text": "{\\displaystyle {\\mathcal {H}}}\n  \n is a normed space (as is the case for SVM), a particularly effective technique is to consider only those hypotheses \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n for which \n  \n    \n      \n        \u2016\n        f\n        \n          \u2016\n          \n            \n              H\n            \n          \n        \n        <\n        k\n      \n    \n    {\\displaystyle \\lVert f\\rVert _{\\mathcal {H}}<k}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0196", "text": ". This is equivalent to imposing a regularization penalty \n  \n    \n      \n        \n          \n            R\n          \n        \n        (\n        f\n        )\n        =\n        \n          \u03bb\n          \n            k\n          \n        \n        \u2016\n        f\n        \n          \u2016\n          \n            \n              H\n            \n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {R}}(f)=\\lambda _{k}\\lVert f\\rVert _{\\mathcal {H}}}\n  \n, and solving the new optimization problem", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0197", "text": "f\n              ^\n            \n          \n        \n        =\n        \n          a\n          r\n          g\n        \n        \n          min\n          \n            f\n            \u2208\n            \n              \n                H\n              \n            \n          \n        \n        \n          \n            \n              \u03b5\n              ^\n            \n          \n        \n        (\n        f\n        )\n        +", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0198", "text": "f\n        )\n        +\n        \n          \n            R\n          \n        \n        (\n        f\n        )\n        .\n      \n    \n    {\\displaystyle {\\hat {f}}=\\mathrm {arg} \\min _{f\\in {\\mathcal {H}}}{\\hat {\\varepsilon }}(f)+{\\mathcal {R}}(f).}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0199", "text": "This approach is called Tikhonov regularization.\nMore generally, \n  \n    \n      \n        \n          \n            R\n          \n        \n        (\n        f\n        )\n      \n    \n    {\\displaystyle {\\mathcal {R}}(f)}\n  \n can be some measure of the complexity of the hypothesis \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n, so that simpler hypotheses are preferred.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0200", "text": "SVM and the hinge loss\nRecall that the (soft-margin) SVM classifier \n  \n    \n      \n        \n          \n            \n              \n                w\n              \n              ^\n            \n          \n        \n        ,\n        b\n        :\n        \n          x\n        \n        \u21a6\n        sgn\n        \u2061\n        (\n        \n          \n            \n              \n                \n                  w\n                \n                ^", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0201", "text": "T\n            \n          \n        \n        \n          x\n        \n        \u2212\n        b\n        )\n      \n    \n    {\\displaystyle {\\hat {\\mathbf {w} }},b:\\mathbf {x} \\mapsto \\operatorname {sgn}({\\hat {\\mathbf {w} }}^{\\mathsf {T}}\\mathbf {x} -b)}\n  \n is chosen to minimize the following expression:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0202", "text": "[\n          \n            \n              \n                1\n                n\n              \n            \n            \n              \u2211\n              \n                i\n                =\n                1\n              \n              \n                n\n              \n            \n            max\n            \n              (\n              \n                0\n                ,\n                1\n                \u2212\n                \n                  y", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0203", "text": "y\n                  \n                    i\n                  \n                \n                (\n                \n                  \n                    w\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  x\n                \n                \u2212\n                b\n                )\n              \n              )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0204", "text": ")\n            \n          \n          ]\n        \n        +\n        \u03bb\n        \u2016\n        \n          w\n        \n        \n          \u2016\n          \n            2\n          \n        \n        .\n      \n    \n    {\\displaystyle \\left[{\\frac {1}{n}}\\sum _{i=1}^{n}\\max \\left(0,1-y_{i}(\\mathbf {w} ^{\\mathsf {T}}\\mathbf {x} -b)\\right)\\right]+\\lambda \\|\\mathbf {w} \\|^{2}.}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0205", "text": "In light of the above discussion, we see that the SVM technique is equivalent to empirical risk minimization with Tikhonov regularization, where in this case the loss function is the hinge loss", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0206", "text": "\u2113\n        (\n        y\n        ,\n        z\n        )\n        =\n        max\n        \n          (\n          \n            0\n            ,\n            1\n            \u2212\n            y\n            z\n          \n          )\n        \n        .\n      \n    \n    {\\displaystyle \\ell (y,z)=\\max \\left(0,1-yz\\right).}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0207", "text": "From this perspective, SVM is closely related to other fundamental classification algorithms such as regularized least-squares and logistic regression. The difference between the three lies in the choice of loss function: regularized least-squares amounts to empirical risk minimization with the square-loss,  \n  \n    \n      \n        \n          \u2113\n          \n            s\n            q\n          \n        \n        (\n        y\n        ,\n        z\n        )\n        =\n        (\n        y\n        \u2212", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0208", "text": ")\n        =\n        (\n        y\n        \u2212\n        z\n        \n          )\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\ell _{sq}(y,z)=(y-z)^{2}}\n  \n; logistic regression employs the log-loss,", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0209", "text": "\u2113\n          \n            log\n          \n        \n        (\n        y\n        ,\n        z\n        )\n        =\n        ln\n        \u2061\n        (\n        1\n        +\n        \n          e\n          \n            \u2212\n            y\n            z\n          \n        \n        )\n        .\n      \n    \n    {\\displaystyle \\ell _{\\log }(y,z)=\\ln(1+e^{-yz}).}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0210", "text": "Target functions\nThe difference between the hinge loss and these other loss functions is best stated in terms of target functions - the function that minimizes expected risk for a given pair of random variables \n  \n    \n      \n        X\n        ,\n        \n        y\n      \n    \n    {\\displaystyle X,\\,y}\n  \n.\nIn particular, let \n  \n    \n      \n        \n          y\n          \n            x\n          \n        \n      \n    \n    {\\displaystyle y_{x}}\n  \n denote \n  \n    \n      \n        y", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0211", "text": "denote \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n conditional on the event that \n  \n    \n      \n        X\n        =\n        x\n      \n    \n    {\\displaystyle X=x}\n  \n.  In the classification setting, we have:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0212", "text": "y\n          \n            x\n          \n        \n        =\n        \n          \n            {\n            \n              \n                \n                  1\n                \n                \n                  \n                    with probability \n                  \n                  \n                    p\n                    \n                      x", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0213", "text": "\u2212\n                  1\n                \n                \n                  \n                    with probability \n                  \n                  1\n                  \u2212\n                  \n                    p\n                    \n                      x", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0214", "text": "{\\displaystyle y_{x}={\\begin{cases}1&{\\text{with probability }}p_{x}\\\\-1&{\\text{with probability }}1-p_{x}\\end{cases}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0215", "text": "The optimal classifier is therefore:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0216", "text": "f\n          \n            \u2217\n          \n        \n        (\n        x\n        )\n        =\n        \n          \n            {\n            \n              \n                \n                  1\n                \n                \n                  \n                    if \n                  \n                  \n                    p\n                    \n                      x\n                    \n                  \n                  \u2265\n                  1", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0217", "text": "\u2265\n                  1\n                  \n                    /\n                  \n                  2\n                \n              \n              \n                \n                  \u2212\n                  1\n                \n                \n                  \n                    otherwise", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0218", "text": "{\\displaystyle f^{*}(x)={\\begin{cases}1&{\\text{if }}p_{x}\\geq 1/2\\\\-1&{\\text{otherwise}}\\end{cases}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0219", "text": "For the square-loss, the target function is the conditional expectation function, \n  \n    \n      \n        \n          f\n          \n            s\n            q\n          \n        \n        (\n        x\n        )\n        =\n        \n          E\n        \n        \n          [\n          \n            y\n            \n              x\n            \n          \n          ]\n        \n      \n    \n    {\\displaystyle f_{sq}(x)=\\mathbb {E} \\left[y_{x}\\right]}\n  \n; For the logistic loss, it's the logit function,", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0220", "text": "f\n          \n            log\n          \n        \n        (\n        x\n        )\n        =\n        ln\n        \u2061\n        \n          (\n          \n            \n              p\n              \n                x\n              \n            \n            \n              /\n            \n            (\n            \n              1\n              \u2212\n              \n                p\n                \n                  x\n                \n              \n            \n            )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0221", "text": ")\n          \n          )\n        \n      \n    \n    {\\displaystyle f_{\\log }(x)=\\ln \\left(p_{x}/({1-p_{x}})\\right)}\n  \n. While both of these target functions yield the correct classifier, as \n  \n    \n      \n        sgn\n        \u2061\n        (\n        \n          f\n          \n            s\n            q\n          \n        \n        )\n        =\n        sgn\n        \u2061\n        (\n        \n          f\n          \n            log\n          \n        \n        )\n        =", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0222", "text": ")\n        =\n        \n          f\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle \\operatorname {sgn}(f_{sq})=\\operatorname {sgn}(f_{\\log })=f^{*}}\n  \n, they give us more information than we need. In fact, they give us enough information to completely describe the distribution of \n  \n    \n      \n        \n          y\n          \n            x\n          \n        \n      \n    \n    {\\displaystyle y_{x}}\n  \n.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0223", "text": "{\\displaystyle y_{x}}\n  \n.\nOn the other hand, one can check that the target function for the hinge loss is exactly \n  \n    \n      \n        \n          f\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle f^{*}}\n  \n. Thus, in a sufficiently rich hypothesis space\u2014or equivalently, for an appropriately chosen kernel\u2014the SVM classifier will converge to the simplest function (in terms of \n  \n    \n      \n        \n          \n            R", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0224", "text": "R\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {R}}}\n  \n) that correctly classifies the data. This extends the geometric interpretation of SVM\u2014for linear classification, the empirical risk is minimized by any function whose margins lie between the support vectors, and the simplest of these is the max-margin classifier.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0225", "text": "Properties\nSVMs belong to a family of generalized linear classifiers and can be interpreted as an extension of the perceptron. They can also be considered a special case of Tikhonov regularization. A special property is that they simultaneously minimize the empirical classification error and maximize the geometric margin; hence they are also known as maximum margin classifiers.\nA comparison of the SVM to other classifiers has been made by Meyer, Leisch and Hornik.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0226", "text": "Parameter selection\nThe effectiveness of SVM depends on the selection of kernel, the kernel's parameters, and soft margin parameter \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n.\nA common choice is a Gaussian kernel, which has a single parameter \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n. The best combination of \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n and \n  \n    \n      \n        \u03b3", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0227", "text": "and \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n is often selected by a grid search with exponentially growing sequences of \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n and \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n, for example, \n  \n    \n      \n        \u03bb\n        \u2208\n        {\n        \n          2\n          \n            \u2212\n            5\n          \n        \n        ,\n        \n          2\n          \n            \u2212", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0228", "text": "2\n          \n            \u2212\n            3\n          \n        \n        ,\n        \u2026\n        ,\n        \n          2\n          \n            13\n          \n        \n        ,\n        \n          2\n          \n            15\n          \n        \n        }\n      \n    \n    {\\displaystyle \\lambda \\in \\{2^{-5},2^{-3},\\dots ,2^{13},2^{15}\\}}\n  \n; \n  \n    \n      \n        \u03b3\n        \u2208\n        {\n        \n          2\n          \n            \u2212\n            15\n          \n        \n        ,", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0229", "text": ",\n        \n          2\n          \n            \u2212\n            13\n          \n        \n        ,\n        \u2026\n        ,\n        \n          2\n          \n            1\n          \n        \n        ,\n        \n          2\n          \n            3\n          \n        \n        }\n      \n    \n    {\\displaystyle \\gamma \\in \\{2^{-15},2^{-13},\\dots ,2^{1},2^{3}\\}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0230", "text": ". Typically, each combination of parameter choices is checked using cross validation, and the parameters with best cross-validation accuracy are picked. Alternatively, recent work in Bayesian optimization can be used to select \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n and \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0231", "text": "{\\displaystyle \\gamma }\n  \n , often requiring the evaluation of far fewer parameter combinations than grid search. The final model, which is used for testing and for classifying new data, is then trained on the whole training set using the selected parameters.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0232", "text": "Issues\nPotential drawbacks of the SVM include the following aspects:\n\nRequires full labeling of input data\nUncalibrated class membership probabilities\u2014SVM stems from Vapnik's theory which avoids estimating probabilities on finite data\nThe SVM is only directly applicable for two-class tasks. Therefore, algorithms that reduce the multi-class task to several binary problems have to be applied; see the multi-class SVM section.\nParameters of a solved model are difficult to interpret.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0233", "text": "Extensions\nMulticlass SVM\nMulticlass SVM aims to assign labels to instances by using support vector machines, where the labels are drawn from a finite set of several elements.\nThe dominant approach for doing so is to reduce the single multiclass problem into multiple binary classification problems. Common methods for such reduction include:", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0234", "text": "Building binary classifiers that distinguish between one of the labels and the rest (one-versus-all) or between every pair of classes (one-versus-one). Classification of new instances for the one-versus-all case is done by a winner-takes-all strategy, in which the classifier with the highest-output function assigns the class (it is important that the output functions be calibrated to produce comparable scores)", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0235", "text": ". For the one-versus-one approach, classification is done by a max-wins voting strategy, in which every classifier assigns the instance to one of the two classes, then the vote for the assigned class is increased by one vote, and finally the class with the most votes determines the instance classification.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0236", "text": "Directed acyclic graph SVM (DAGSVM)\nError-correcting output codes\nCrammer and Singer proposed a multiclass SVM method which casts the multiclass classification problem into a single optimization problem, rather than decomposing it into multiple binary classification problems. See also Lee, Lin and Wahba and Van den Burg and Groenen.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0237", "text": "Transductive support vector machines\nTransductive support vector machines extend SVMs in that they could also treat partially labeled data in semi-supervised learning by following the principles of transduction. Here, in addition to the training set \n  \n    \n      \n        \n          \n            D\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}}\n  \n, the learner is also given a set", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0238", "text": "D\n            \n          \n          \n            \u22c6\n          \n        \n        =\n        {\n        \n          \n            x\n          \n          \n            i\n          \n          \n            \u22c6\n          \n        \n        \u2223\n        \n          \n            x\n          \n          \n            i\n          \n          \n            \u22c6\n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            p", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0239", "text": "R\n          \n          \n            p\n          \n        \n        \n          }\n          \n            i\n            =\n            1\n          \n          \n            k\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}^{\\star }=\\{\\mathbf {x} _{i}^{\\star }\\mid \\mathbf {x} _{i}^{\\star }\\in \\mathbb {R} ^{p}\\}_{i=1}^{k}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0240", "text": "of test examples to be classified. Formally, a transductive support vector machine is defined by the following primal optimization problem:\nMinimize (in \n  \n    \n      \n        \n          w\n        \n        ,\n        b\n        ,\n        \n          \n            y\n          \n          \n            \u22c6\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {w} ,b,\\mathbf {y} ^{\\star }}\n  \n)", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0241", "text": "1\n            2\n          \n        \n        \u2016\n        \n          w\n        \n        \n          \u2016\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle {\\frac {1}{2}}\\|\\mathbf {w} \\|^{2}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0242", "text": "subject to (for any \n  \n    \n      \n        i\n        =\n        1\n        ,\n        \u2026\n        ,\n        n\n      \n    \n    {\\displaystyle i=1,\\dots ,n}\n  \n and any \n  \n    \n      \n        j\n        =\n        1\n        ,\n        \u2026\n        ,\n        k\n      \n    \n    {\\displaystyle j=1,\\dots ,k}\n  \n)", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0243", "text": "y\n                  \n                    i\n                  \n                \n                (\n                \n                  w\n                \n                \u22c5\n                \n                  \n                    x\n                  \n                  \n                    i\n                  \n                \n                \u2212\n                b\n                )", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0244", "text": "b\n                )\n                \u2265\n                1\n                ,\n              \n            \n            \n              \n              \n                \n                  y\n                  \n                    j\n                  \n                  \n                    \u22c6\n                  \n                \n                (\n                \n                  w\n                \n                \u22c5\n                \n                  \n                    x", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0245", "text": "x\n                  \n                  \n                    j\n                  \n                  \n                    \u22c6\n                  \n                \n                \u2212\n                b\n                )\n                \u2265\n                1\n                ,", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0246", "text": "{\\displaystyle {\\begin{aligned}&y_{i}(\\mathbf {w} \\cdot \\mathbf {x} _{i}-b)\\geq 1,\\\\&y_{j}^{\\star }(\\mathbf {w} \\cdot \\mathbf {x} _{j}^{\\star }-b)\\geq 1,\\end{aligned}}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0247", "text": "and\n\n  \n    \n      \n        \n          y\n          \n            j\n          \n          \n            \u22c6\n          \n        \n        \u2208\n        {\n        \u2212\n        1\n        ,\n        1\n        }\n        .\n      \n    \n    {\\displaystyle y_{j}^{\\star }\\in \\{-1,1\\}.}\n  \n\nTransductive support vector machines were introduced by Vladimir N. Vapnik in 1998.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0248", "text": "Structured SVM\nStructured support-vector machine is an extension of the traditional SVM model. While the SVM model is primarily designed for binary classification, multiclass classification, and regression tasks, structured SVM broadens its application to handle general structured output labels, for example parse trees, classification with taxonomies, sequence alignment and many more.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0249", "text": "Regression", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0250", "text": "A version of SVM for regression was proposed in 1996 by Vladimir N. Vapnik, Harris Drucker, Christopher J. C. Burges, Linda Kaufman and Alexander J. Smola. This method is called support vector regression (SVR). The model produced by support vector classification (as described above) depends only on a subset of the training data, because the cost function for building the model does not care about training points that lie beyond the margin", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0251", "text": ". Analogously, the model produced by SVR depends only on a subset of the training data, because the cost function for building the model ignores any training data close to the model prediction. Another SVM version known as least-squares support vector machine (LS-SVM) has been proposed by Suykens and Vandewalle.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0252", "text": "Training the original SVR means solving", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0253", "text": "minimize \n  \n    \n      \n        \n          \n            \n              1\n              2\n            \n          \n        \n        \u2016\n        w\n        \n          \u2016\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle {\\tfrac {1}{2}}\\|w\\|^{2}}", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0254", "text": "subject to \n  \n    \n      \n        \n          |\n        \n        \n          y\n          \n            i\n          \n        \n        \u2212\n        \u27e8\n        w\n        ,\n        \n          x\n          \n            i\n          \n        \n        \u27e9\n        \u2212\n        b\n        \n          |\n        \n        \u2264\n        \u03b5\n      \n    \n    {\\displaystyle |y_{i}-\\langle w,x_{i}\\rangle -b|\\leq \\varepsilon }", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0255", "text": "where \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n is a training sample with target value \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}\n  \n. The inner product plus intercept \n  \n    \n      \n        \u27e8\n        w\n        ,\n        \n          x\n          \n            i\n          \n        \n        \u27e9\n        +\n        b", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0256", "text": "\u27e9\n        +\n        b\n      \n    \n    {\\displaystyle \\langle w,x_{i}\\rangle +b}\n  \n is the prediction for that sample, and \n  \n    \n      \n        \u03b5\n      \n    \n    {\\displaystyle \\varepsilon }\n  \n is a free parameter that serves as a threshold: all predictions have to be within an \n  \n    \n      \n        \u03b5\n      \n    \n    {\\displaystyle \\varepsilon }", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0257", "text": "{\\displaystyle \\varepsilon }\n  \n range of the true predictions. Slack variables are usually added into the above to allow for errors and to allow approximation in the case the above problem is infeasible.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0258", "text": "Bayesian SVM", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0259", "text": "In 2011 it was shown by Polson and Scott that the SVM admits a Bayesian interpretation through the technique of data augmentation. In this approach the SVM is viewed as a graphical model (where the parameters are connected via probability distributions). This extended view allows the application of Bayesian techniques to SVMs, such as flexible feature modeling, automatic hyperparameter tuning, and predictive uncertainty quantification", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0260", "text": ". In 2017, a scalable version of the Bayesian SVM was developed by Florian Wenzel, enabling the application of Bayesian SVMs to big data. Florian Wenzel developed two different versions, a variational inference (VI) scheme for the Bayesian kernel support vector machine (SVM) and a stochastic version (SVI) for the linear Bayesian SVM.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0261", "text": "Implementation\nThe parameters of the maximum-margin hyperplane are derived by solving the optimization. There exist several specialized algorithms for quickly solving the quadratic programming (QP) problem that arises from SVMs, mostly relying on heuristics for breaking the problem down into smaller, more manageable chunks.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0262", "text": "Another approach is to use an interior-point method that uses Newton-like iterations to find a solution of the Karush\u2013Kuhn\u2013Tucker conditions of the primal and dual problems.\nInstead of solving a sequence of broken-down problems, this approach directly solves the problem altogether. To avoid solving a linear system involving the large kernel matrix, a low-rank approximation to the matrix is often used in the kernel trick.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0263", "text": "Another common method is Platt's sequential minimal optimization (SMO) algorithm, which breaks the problem down into 2-dimensional sub-problems that are solved analytically, eliminating the need for a numerical optimization algorithm and matrix storage. This algorithm is conceptually simple, easy to implement, generally faster, and has better scaling properties for difficult SVM problems.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0264", "text": "The special case of linear support vector machines can be solved more efficiently by the same kind of algorithms used to optimize its close cousin, logistic regression; this class of algorithms includes sub-gradient descent (e.g., PEGASOS) and coordinate descent (e.g., LIBLINEAR). LIBLINEAR has some attractive training-time properties", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0265", "text": ". Each convergence iteration takes time linear in the time taken to read the train data, and the iterations also have a Q-linear convergence property, making the algorithm extremely fast.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0266", "text": "The general kernel SVMs can also be solved more efficiently using sub-gradient descent (e.g. P-packSVM), especially when parallelization is allowed.\nKernel SVMs are available in many machine-learning toolkits, including LIBSVM, MATLAB, SAS, SVMlight, kernlab, scikit-learn, Shogun, Weka, Shark, JKernelMachines, OpenCV and others.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0267", "text": "Preprocessing of data (standardization) is highly recommended to enhance accuracy of classification. There are a few methods of standardization, such as min-max, normalization by decimal scaling, Z-score. Subtraction of mean and division by variance of each feature is usually used for SVM.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0268", "text": "See also\nIn situ adaptive tabulation\nKernel machines\nFisher kernel\nPlatt scaling\nPolynomial kernel\nPredictive analytics\nRegularization perspectives on support vector machines\nRelevance vector machine, a probabilistic sparse-kernel model identical in functional form to SVM\nSequential minimal optimization\nSpace mapping\nWinnow (algorithm)\nRadial basis function network", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0269", "text": "References\nFurther reading\nBennett, Kristin P.; Campbell, Colin (2000). \"Support Vector Machines: Hype or Hallelujah?\" (PDF). SIGKDD Explorations. 2 (2): 1\u201313. doi:10.1145/380995.380999. S2CID 207753020.\nCristianini, Nello; Shawe-Taylor, John (2000). An Introduction to Support Vector Machines and other kernel-based learning methods. Cambridge University Press. ISBN 0-521-78019-5.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0270", "text": "Fradkin, Dmitriy; Muchnik, Ilya (2006). \"Support Vector Machines for Classification\" (PDF). In Abello, J.; Carmode, G. (eds.). Discrete Methods in Epidemiology. DIMACS Series in Discrete Mathematics and Theoretical Computer Science. Vol. 70. pp. 13\u201320.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0271", "text": "Joachims, Thorsten (1998). \"Text categorization with Support Vector Machines: Learning with many relevant features\". In N\u00e9dellec, Claire; Rouveirol, C\u00e9line (eds.). Machine Learning: ECML-98. Lecture Notes in Computer Science. Vol. 1398. Berlin, Heidelberg: Springer. pp. 137\u2013142. doi:10.1007/BFb0026683. ISBN 978-3-540-64417-0. S2CID 2427083.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0272", "text": "Ivanciuc, Ovidiu (2007). \"Applications of Support Vector Machines in Chemistry\" (PDF). Reviews in Computational Chemistry. Vol. 23. pp. 291\u2013400. doi:10.1002/9780470116449.ch6. ISBN 9780470116449.\nJames, Gareth; Witten, Daniela; Hastie, Trevor; Tibshirani, Robert (2013). \"Support Vector Machines\" (PDF). An Introduction to Statistical Learning : with Applications in R. New York: Springer. pp. 337\u2013372. ISBN 978-1-4614-7137-0.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0273", "text": "Sch\u00f6lkopf, Bernhard; Smola, Alexander J. (2002). Learning with Kernels. Cambridge, MA: MIT Press. ISBN 0-262-19475-9.\nSteinwart, Ingo; Christmann, Andreas (2008). Support Vector Machines. New York: Springer. ISBN 978-0-387-77241-7.\nTheodoridis, Sergios; Koutroumbas, Konstantinos (2009). Pattern Recognition (4th ed.). Academic Press. ISBN 978-1-59749-272-0.", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Support_vector_machine_0274", "text": "External links\nlibsvm, LIBSVM is a popular library of SVM learners\nliblinear is a library for large linear classification including some SVMs\nSVM light is a collection of software tools for learning and classification using SVM\nSVMJS live demo Archived 2013-05-05 at the Wayback Machine is a GUI demo for JavaScript implementation of SVMs", "title": "Support vector machine", "url": "https://en.wikipedia.org/wiki/Support_vector_machine"}, {"id": "rec_Random_forest_0000", "text": "Random forests or random decision forests is an ensemble learning method for classification, regression and other tasks that works by creating a multitude of decision trees during training. For classification tasks, the output of the random forest is the class selected by most trees. For regression tasks, the output is the average of the predictions of the trees. Random forests correct for decision trees' habit of overfitting to their training set.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0001", "text": "The first algorithm for random decision forests was created in 1995 by Tin Kam Ho using the random subspace method, which, in Ho's formulation, is a way to implement the \"stochastic discrimination\" approach to classification proposed by Eugene Kleinberg.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0002", "text": "An extension of the algorithm was developed by Leo Breiman and Adele Cutler, who registered \"Random Forests\" as a trademark in 2006 (as of 2019, owned by Minitab, Inc.). The extension combines Breiman's \"bagging\" idea and random selection of features, introduced first by Ho and later independently by Amit and Geman in order to construct a collection of decision trees with controlled variance.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0003", "text": "History", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0004", "text": "The general method of random decision forests was first proposed by Salzberg and Heath in 1993, with a method that used a randomized decision tree algorithm to create multiple trees and then combine them using majority voting.  This idea was developed further by Ho in 1995", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0005", "text": ".  This idea was developed further by Ho in 1995.  Ho established that forests of trees splitting with oblique hyperplanes can gain accuracy as they grow without suffering from overtraining, as long as the forests are randomly restricted to be sensitive to only selected feature dimensions.  A subsequent work along the same lines concluded that other splitting methods behave similarly, as long as they are randomly forced to be insensitive to some feature dimensions", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0006", "text": ".  This observation that a more complex classifier (a larger forest) gets more accurate nearly monotonically is in sharp contrast to the common belief that the complexity of a classifier can only grow to a certain level of accuracy before being hurt by overfitting.  The explanation of the forest method's resistance to overtraining can be found in Kleinberg's theory of stochastic discrimination.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0007", "text": "The early development of Breiman's notion of random forests was influenced by the work of Amit and Geman who introduced the idea of searching over a random subset of the available decisions when splitting a node, in the context of growing a single tree.  The idea of random subspace selection from Ho was also influential in the design of random forests", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0008", "text": ".  This method grows a forest of trees, and introduces variation among the trees by projecting the training data into a randomly chosen subspace before fitting each tree or each node.  Finally, the idea of randomized node optimization, where the decision at each node is selected by a randomized procedure, rather than a deterministic optimization was first introduced by Thomas G. Dietterich.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0009", "text": "The proper introduction of random forests was made in a paper by Leo Breiman,\nthat has become one of the world's most cited papers.\nThis paper describes a method of building a forest of uncorrelated trees using a CART like procedure, combined with randomized node optimization and bagging.  In addition, this paper combines several ingredients, some previously known and some novel, which form the basis of the modern practice of random forests, in particular:", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0010", "text": "Using out-of-bag error as an estimate of the generalization error.\nMeasuring variable importance through permutation.\nThe report also offers the first theoretical result for random forests in the form of a bound on the generalization error which depends on the strength of the trees in the forest and their correlation.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0011", "text": "Algorithm\nPreliminaries: decision tree learning\nDecision trees are a popular method for various machine learning tasks. Tree learning is almost \"an off-the-shelf procedure for data mining\", say Hastie et al., \"because it is invariant under scaling and various other transformations of feature values, is robust to inclusion of irrelevant features, and produces inspectable models. However, they are seldom accurate\".", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0012", "text": "In particular, trees that are grown very deep tend to learn highly irregular patterns: they overfit their training sets, i.e. have low bias, but very high variance. Random forests are a way of averaging multiple deep decision trees, trained on different parts of the same training set, with the goal of reducing the variance. This comes at the expense of a small increase in the bias and some loss of interpretability, but generally greatly boosts the performance in the final model.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0013", "text": "Bagging\nThe training algorithm for random forests applies the general technique of bootstrap aggregating, or bagging, to tree learners. Given a training set X = x1, ..., xn with responses Y = y1, ..., yn, bagging repeatedly (B times) selects a random sample with replacement of the training set and fits trees to these samples:\n\nAfter training, predictions for unseen samples x' can be made by averaging the predictions from all the individual regression trees on x':", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0014", "text": "f\n              ^\n            \n          \n        \n        =\n        \n          \n            1\n            B\n          \n        \n        \n          \u2211\n          \n            b\n            =\n            1\n          \n          \n            B\n          \n        \n        \n          f\n          \n            b\n          \n        \n        (\n        \n          x\n          \u2032\n        \n        )", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0015", "text": "\u2032\n        \n        )\n      \n    \n    {\\displaystyle {\\hat {f}}={\\frac {1}{B}}\\sum _{b=1}^{B}f_{b}(x')}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0016", "text": "or by taking the plurality vote in the case of classification trees.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0017", "text": "This bootstrapping procedure leads to better model performance because it decreases the variance of the model, without increasing the bias. This means that while the predictions of a single tree are highly sensitive to noise in its training set, the average of many trees is not, as long as the trees are not correlated", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0018", "text": ". Simply training many trees on a single training set would give strongly correlated trees (or even the same tree many times, if the training algorithm is deterministic); bootstrap sampling is a way of de-correlating the trees by showing them different training sets.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0019", "text": "Additionally, an estimate of the uncertainty of the prediction can be made as the standard deviation of the predictions from all the individual regression trees on x\u2032:", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0020", "text": "\u03c3\n        =\n        \n          \n            \n              \n                \n                  \u2211\n                  \n                    b\n                    =\n                    1\n                  \n                  \n                    B\n                  \n                \n                (\n                \n                  f\n                  \n                    b\n                  \n                \n                (\n                \n                  x", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0021", "text": "x\n                  \u2032\n                \n                )\n                \u2212\n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                  )\n                  \n                    2\n                  \n                \n              \n              \n                B\n                \u2212\n                1", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0022", "text": "1\n              \n            \n          \n        \n        .\n      \n    \n    {\\displaystyle \\sigma ={\\sqrt {\\frac {\\sum _{b=1}^{B}(f_{b}(x')-{\\hat {f}})^{2}}{B-1}}}.}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0023", "text": "The number B of samples (equivalently, of trees) is a free parameter. Typically, a few hundred to several thousand trees are used, depending on the size and nature of the training set. B can be optimized using cross-validation, or by observing the out-of-bag error: the mean prediction error on each training sample xi, using only the trees that did not have xi in their bootstrap sample.\nThe training and test error tend to level off after some number of trees have been fit.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0024", "text": "From bagging to random forests", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0025", "text": "The above procedure describes the original bagging algorithm for trees. Random forests also include another type of bagging scheme: they use a modified tree learning algorithm that selects, at each candidate split in the learning process, a random subset of the features. This process is sometimes called \"feature bagging\"", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0026", "text": ". The reason for doing this is the correlation of the trees in an ordinary bootstrap sample: if one or a few features are very strong predictors for the response variable (target output), these features will be selected in many of the B trees, causing them to become correlated. An analysis of how bagging and random subspace projection contribute to accuracy gains under different conditions is given by Ho.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0027", "text": "Typically, for a classification problem with \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n features, \n  \n    \n      \n        \n          \n            p\n          \n        \n      \n    \n    {\\displaystyle {\\sqrt {p}}}\n  \n (rounded down) features are used in each split.  For regression problems the inventors recommend \n  \n    \n      \n        p\n        \n          /\n        \n        3\n      \n    \n    {\\displaystyle p/3}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0028", "text": "3\n      \n    \n    {\\displaystyle p/3}\n  \n (rounded down) with a minimum node size of 5 as the default. In practice, the best values for these parameters should be tuned on a case-to-case basis for every problem.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0029", "text": "ExtraTrees", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0030", "text": "Adding one further step of randomization yields extremely randomized trees, or ExtraTrees", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0031", "text": ". As with ordinary random forests, they are an ensemble of individual trees, but there are two main differences: (1) each tree is trained using the whole learning sample (rather than a bootstrap sample), and (2) the top-down splitting is randomized: for each feature under consideration, a number of random cut-points are selected, instead of computing the locally optimal cut-point (based on, e.g., information gain or the Gini impurity)", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0032", "text": ". The values are chosen from a uniform distribution within the feature's empirical range (in the tree's training set). Then, of all the randomly chosen splits, the split that yields the highest score is chosen to split the node.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0033", "text": "Similar to ordinary random forests, the number of randomly selected features to be considered at each node can be specified. Default values for this parameter are \n  \n    \n      \n        \n          \n            p\n          \n        \n      \n    \n    {\\displaystyle {\\sqrt {p}}}\n  \n for classification and \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n for regression, where \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n is the number of features in the model.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0034", "text": "Random forests for high-dimensional data\nThe basic random forest procedure may not work well in situations where there are a large number of features but only a small proportion of these features are informative with respect to sample classification. This can be addressed by encouraging the procedure to focus mainly on features and trees that are informative. Some methods for accomplishing this are:", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0035", "text": "Prefiltering: Eliminate features that are mostly just noise.\nEnriched Random Forest (ERF): Use weighted random sampling instead of simple random sampling at each node of each tree, giving greater weight to features that appear to be more informative.\nTree-weighted random forest (TWRF): Give more weight to more accurate trees.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0036", "text": "Properties\nVariable importance\nRandom forests can be used to rank the importance of variables in a regression or classification problem in a natural way.  The following technique was described in Breiman's original paper and is implemented in the R package randomForest.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0037", "text": "Permutation importance\nTo measure a feature's importance in a data set \n  \n    \n      \n        \n          \n            \n              D\n            \n          \n          \n            n\n          \n        \n        =\n        {\n        (\n        \n          X\n          \n            i\n          \n        \n        ,\n        \n          Y\n          \n            i\n          \n        \n        )\n        \n          }\n          \n            i\n            =\n            1\n          \n          \n            n", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0038", "text": "1\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}_{n}=\\{(X_{i},Y_{i})\\}_{i=1}^{n}}\n  \n, first a random forest is trained on the data.  During training, the out-of-bag error for each data point is recorded and averaged over the forest. (If bagging is not used during training, we can instead compute errors on an independent test set.)", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0039", "text": "After training, the values of the feature are permuted in the out-of-bag samples and the out-of-bag error is again computed on this perturbed data set. The importance for the feature is computed by averaging the difference in out-of-bag error before and after the permutation over all trees.  The score is normalized by the standard deviation of these differences.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0040", "text": "Features which produce large values for this score are ranked as more important than features which produce small values. The statistical definition of the variable importance measure was given and analyzed by Zhu et al.\nThis method of determining variable importance has some drawbacks:", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0041", "text": "When features have different numbers of values, random forests favor features with more values. Solutions to this problem include partial permutations and growing unbiased trees.\nIf the data contain groups of correlated features of similar relevance, then smaller groups are favored over large groups.\nIf there are collinear features, the procedure may fail to identify important features. A solution is to permute groups of correlated features together.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0042", "text": "Mean decrease in impurity feature importance\nThis approach to feature importance for random forests considers as important the variables which decrease a lot the impurity during splitting. It is described in the book Classification and Regression Trees by Leo Breiman and is the default implementation in sci-kit learn and R. The definition is:\n  \n    \n      \n        \n          unormalized average importance\n        \n        (\n        x\n        )\n        =\n        \n          \n            1", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0043", "text": "=\n        \n          \n            1\n            \n              n\n              \n                T\n              \n            \n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            \n              n\n              \n                T\n              \n            \n          \n        \n        \n          \u2211\n          \n            \n              node \n            \n            j\n            \u2208\n            \n              T", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0044", "text": "\u2208\n            \n              T\n              \n                i\n              \n            \n            \n              |\n            \n            \n              split variable\n            \n            (\n            j\n            )\n            =\n            x\n          \n        \n        \n          p\n          \n            \n              T\n              \n                i\n              \n            \n          \n        \n        (\n        j\n        )\n        \u0394\n        \n          i", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0045", "text": ")\n        \u0394\n        \n          i\n          \n            \n              T\n              \n                i\n              \n            \n          \n        \n        (\n        j\n        )\n        ,\n      \n    \n    {\\displaystyle {\\text{unormalized average importance}}(x)={\\frac {1}{n_{T}}}\\sum _{i=1}^{n_{T}}\\sum _{{\\text{node }}j\\in T_{i}|{\\text{split variable}}(j)=x}p_{T_{i}}(j)\\Delta i_{T_{i}}(j),}\n  \nwhere", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0046", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n is a feature\n\n  \n    \n      \n        \n          n\n          \n            T\n          \n        \n      \n    \n    {\\displaystyle n_{T}}\n  \n is the number of trees in the forest\n\n  \n    \n      \n        \n          T\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle T_{i}}\n  \n is tree \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0047", "text": "p\n          \n            \n              T\n              \n                i\n              \n            \n          \n        \n        (\n        j\n        )\n        =\n        \n          \n            \n              n\n              \n                j\n              \n            \n            n\n          \n        \n      \n    \n    {\\displaystyle p_{T_{i}}(j)={\\frac {n_{j}}{n}}}\n  \n is the fraction of samples reaching node \n  \n    \n      \n        j", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0048", "text": "j\n      \n    \n    {\\displaystyle j}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0049", "text": "\u0394\n        \n          i\n          \n            \n              T\n              \n                i\n              \n            \n          \n        \n        (\n        j\n        )\n      \n    \n    {\\displaystyle \\Delta i_{T_{i}}(j)}\n  \n is the change in impurity in tree \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n at node \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0050", "text": "j\n      \n    \n    {\\displaystyle j}\n  \n.\nAs impurity measure for samples falling in a node e.g. the following statistics can be used:", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0051", "text": "Entropy\nGini coefficient\nMean squared error\nThe normalized importance is then obtained by normalizing over all features, so that the sum of normalized feature importances is 1.\nThe sci-kit learn default implementation can report misleading feature importance:\n\nit favors high cardinality features\nit uses training statistics and so does not reflect a feature's usefulness for predictions on a test set", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0052", "text": "Relationship to nearest neighbors\nA relationship between random forests and the k-nearest neighbor algorithm (k-NN) was pointed out by Lin and Jeon in 2002. Both can be viewed as so-called weighted neighborhoods schemes. These are models built from a training set \n  \n    \n      \n        {\n        (\n        \n          x\n          \n            i\n          \n        \n        ,\n        \n          y\n          \n            i\n          \n        \n        )\n        \n          }\n          \n            i", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0053", "text": "}\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\{(x_{i},y_{i})\\}_{i=1}^{n}}\n  \n that make predictions \n  \n    \n      \n        \n          \n            \n              y\n              ^\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {y}}}\n  \n for new points x' by looking at the \"neighborhood\" of the point, formalized by a weight function W:", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0054", "text": "y\n              ^\n            \n          \n        \n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        W\n        (\n        \n          x\n          \n            i\n          \n        \n        ,\n        \n          x\n          \u2032\n        \n        )\n        \n        \n          y\n          \n            i\n          \n        \n        .", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0055", "text": ".\n      \n    \n    {\\displaystyle {\\hat {y}}=\\sum _{i=1}^{n}W(x_{i},x')\\,y_{i}.}\n  \nHere, \n  \n    \n      \n        W\n        (\n        \n          x\n          \n            i\n          \n        \n        ,\n        \n          x\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle W(x_{i},x')}\n  \n is the non-negative weight of the i'th training point relative to the new point x' in the same tree. For any x', the weights for points", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0056", "text": "x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n must sum to 1. Weight functions are as follows:", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0057", "text": "In k-NN, \n  \n    \n      \n        W\n        (\n        \n          x\n          \n            i\n          \n        \n        ,\n        \n          x\n          \u2032\n        \n        )\n        =\n        \n          \n            1\n            k\n          \n        \n      \n    \n    {\\displaystyle W(x_{i},x')={\\frac {1}{k}}}\n  \n if xi is one of the k points closest to x', and zero otherwise.\nIn a tree, \n  \n    \n      \n        W\n        (\n        \n          x\n          \n            i", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0058", "text": "i\n          \n        \n        ,\n        \n          x\n          \u2032\n        \n        )\n        =\n        \n          \n            1\n            \n              k\n              \u2032\n            \n          \n        \n      \n    \n    {\\displaystyle W(x_{i},x')={\\frac {1}{k'}}}\n  \n if xi is one of the k' points in the same leaf as x', and zero otherwise.\nSince a forest averages the predictions of a set of m trees with individual weight functions \n  \n    \n      \n        \n          W", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0059", "text": "W\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle W_{j}}\n  \n, its predictions are\n  \n    \n      \n        \n          \n            \n              y\n              ^\n            \n          \n        \n        =\n        \n          \n            1\n            m\n          \n        \n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            m\n          \n        \n        \n          \u2211", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0060", "text": "\u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          W\n          \n            j\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        ,\n        \n          x\n          \u2032\n        \n        )\n        \n        \n          y\n          \n            i\n          \n        \n        =\n        \n          \u2211\n          \n            i\n            =", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0061", "text": "i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          (\n          \n            \n              \n                1\n                m\n              \n            \n            \n              \u2211\n              \n                j\n                =\n                1\n              \n              \n                m\n              \n            \n            \n              W\n              \n                j", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0062", "text": "j\n              \n            \n            (\n            \n              x\n              \n                i\n              \n            \n            ,\n            \n              x\n              \u2032\n            \n            )\n          \n          )\n        \n        \n        \n          y\n          \n            i\n          \n        \n        .", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0063", "text": ".\n      \n    \n    {\\displaystyle {\\hat {y}}={\\frac {1}{m}}\\sum _{j=1}^{m}\\sum _{i=1}^{n}W_{j}(x_{i},x')\\,y_{i}=\\sum _{i=1}^{n}\\left({\\frac {1}{m}}\\sum _{j=1}^{m}W_{j}(x_{i},x')\\right)\\,y_{i}.}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0064", "text": "This shows that the whole forest is again a weighted neighborhood scheme, with weights that average those of the individual trees. The neighbors of x' in this interpretation are the points \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n sharing the same leaf in any tree \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0065", "text": "j\n      \n    \n    {\\displaystyle j}\n  \n. In this way, the neighborhood of x' depends in a complex way on the structure of the trees, and thus on the structure of the training set. Lin and Jeon show that the shape of the neighborhood used by a random forest adapts to the local importance of each feature.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0066", "text": "Unsupervised learning", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0067", "text": "As part of their construction, random forest predictors naturally lead to a dissimilarity measure among observations. One can analogously define dissimilarity between unlabeled data, by training a forest to distinguish original \"observed\" data from suitably generated synthetic data drawn from a reference distribution", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0068", "text": ". A random forest dissimilarity is attractive because it handles mixed variable types very well, is invariant to monotonic transformations of the input variables, and is robust to outlying observations. Random forest dissimilarity easily deals with a large number of semi-continuous variables due to its intrinsic variable selection; for example, the \"Addcl 1\" random forest dissimilarity weighs the contribution of each variable according to how dependent it is on other variables", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0069", "text": ". Random forest dissimilarity has been used in a variety of applications, e.g. to find clusters of patients based on tissue marker data.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0070", "text": "Variants\nInstead of decision trees, linear models have been proposed and evaluated as base estimators in random forests, in particular multinomial logistic regression and naive Bayes classifiers. In cases that the relationship between the predictors and the target variable is linear, the base learners may have an equally high accuracy as the ensemble learner.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0071", "text": "Kernel random forest\nIn machine learning, kernel random forests (KeRF) establish the connection between random forests and kernel methods. By slightly modifying their definition, random forests can be rewritten as kernel methods, which are more interpretable and easier to analyze.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0072", "text": "History", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0073", "text": "Leo Breiman was the first person to notice the link between random forest and kernel methods. He pointed out that random forests trained using i.i.d. random vectors in the tree construction are equivalent to a kernel acting on the true margin. Lin and Jeon established the connection between random forests and adaptive nearest neighbor, implying that random forests can be seen as adaptive kernel estimates", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0074", "text": ". Davies and Ghahramani proposed Kernel Random Forest (KeRF) and showed that it can empirically outperform state-of-art kernel methods. Scornet first defined KeRF estimates and gave the explicit link between KeRF estimates and random forest. He also gave explicit expressions for kernels based on centered random forest and uniform random forest, two simplified models of random forest. He named these two KeRFs Centered KeRF and Uniform KeRF, and proved upper bounds on their rates of consistency.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0075", "text": "Notations and definitions\nPreliminaries: Centered forests\nCentered forest is a simplified model for Breiman's original random forest, which uniformly selects an attribute among all attributes and performs splits at the center of the cell along the pre-chosen attribute. The algorithm stops when a fully binary tree of level \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n is built, where \n  \n    \n      \n        k\n        \u2208\n        \n          N", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0076", "text": "N\n        \n      \n    \n    {\\displaystyle k\\in \\mathbb {N} }\n  \n is a parameter of the algorithm.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0077", "text": "Uniform forest\nUniform forest is another simplified model for Breiman's original random forest, which uniformly selects a feature among all features and performs splits at a point uniformly drawn on the side of the cell, along the preselected feature.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0078", "text": "From random forest to KeRF\nGiven a training sample  \n  \n    \n      \n        \n          \n            \n              D\n            \n          \n          \n            n\n          \n        \n        =\n        {\n        (\n        \n          \n            X\n          \n          \n            i\n          \n        \n        ,\n        \n          Y\n          \n            i\n          \n        \n        )\n        \n          }\n          \n            i\n            =\n            1", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0079", "text": "=\n            1\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}_{n}=\\{(\\mathbf {X} _{i},Y_{i})\\}_{i=1}^{n}}\n  \n of \n  \n    \n      \n        [\n        0\n        ,\n        1\n        \n          ]\n          \n            p\n          \n        \n        \u00d7\n        \n          R\n        \n      \n    \n    {\\displaystyle [0,1]^{p}\\times \\mathbb {R} }\n  \n-valued independent random variables distributed as the independent prototype pair", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0080", "text": "(\n        \n          X\n        \n        ,\n        Y\n        )\n      \n    \n    {\\displaystyle (\\mathbf {X} ,Y)}\n  \n, where \n  \n    \n      \n        E\n        \u2061\n        [\n        \n          Y\n          \n            2\n          \n        \n        ]\n        <\n        \u221e\n      \n    \n    {\\displaystyle \\operatorname {E} [Y^{2}]<\\infty }\n  \n. We aim at predicting the response \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n, associated with the random variable", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0081", "text": ", associated with the random variable \n  \n    \n      \n        \n          X\n        \n      \n    \n    {\\displaystyle \\mathbf {X} }\n  \n, by estimating the regression function \n  \n    \n      \n        m\n        (\n        \n          x\n        \n        )\n        =\n        E\n        \u2061\n        [\n        Y\n        \u2223\n        \n          X\n        \n        =\n        \n          x\n        \n        ]\n      \n    \n    {\\displaystyle m(\\mathbf {x} )=\\operatorname {E} [Y\\mid \\mathbf {X} =\\mathbf {x} ]}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0082", "text": ". A random regression forest is an ensemble of \n  \n    \n      \n        M\n      \n    \n    {\\displaystyle M}\n  \n randomized regression trees. Denote \n  \n    \n      \n        \n          m\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          \n            \u0398\n          \n          \n            j\n          \n        \n        )\n      \n    \n    {\\displaystyle m_{n}(\\mathbf {x} ,\\mathbf {\\Theta } _{j})}\n  \n the predicted value at point", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0083", "text": "the predicted value at point \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n by the \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n-th tree, where \n  \n    \n      \n        \n          \n            \u0398\n          \n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \n            \u0398\n          \n          \n            M", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0084", "text": "M\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {\\Theta } _{1},\\ldots ,\\mathbf {\\Theta } _{M}}\n  \n are independent random variables, distributed as a generic random variable \n  \n    \n      \n        \n          \u0398\n        \n      \n    \n    {\\displaystyle \\mathbf {\\Theta } }\n  \n, independent of the sample \n  \n    \n      \n        \n          \n            \n              D\n            \n          \n          \n            n", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0085", "text": "n\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}_{n}}\n  \n. This random variable can be used to describe the randomness induced by node splitting and the sampling procedure for tree construction. The trees are combined to form the finite forest estimate \n  \n    \n      \n        \n          m\n          \n            M\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          \u0398\n          \n            1", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0086", "text": "\u0398\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \u0398\n          \n            M\n          \n        \n        )\n        =\n        \n          \n            1\n            M\n          \n        \n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            M\n          \n        \n        \n          m\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        ,", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0087", "text": "(\n        \n          x\n        \n        ,\n        \n          \u0398\n          \n            j\n          \n        \n        )\n      \n    \n    {\\displaystyle m_{M,n}(\\mathbf {x} ,\\Theta _{1},\\ldots ,\\Theta _{M})={\\frac {1}{M}}\\sum _{j=1}^{M}m_{n}(\\mathbf {x} ,\\Theta _{j})}\n  \n.\nFor regression trees, we have \n  \n    \n      \n        \n          m\n          \n            n\n          \n        \n        =\n        \n          \u2211\n          \n            i\n            =\n            1", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0088", "text": "=\n            1\n          \n          \n            n\n          \n        \n        \n          \n            \n              \n                Y\n                \n                  i\n                \n              \n              \n                \n                  1\n                \n                \n                  \n                    \n                      X\n                    \n                    \n                      i\n                    \n                  \n                  \u2208", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0089", "text": "\u2208\n                  \n                    A\n                    \n                      n\n                    \n                  \n                  (\n                  \n                    x\n                  \n                  ,\n                  \n                    \u0398\n                    \n                      j\n                    \n                  \n                  )", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0090", "text": "N\n                \n                  n\n                \n              \n              (\n              \n                x\n              \n              ,\n              \n                \u0398\n                \n                  j\n                \n              \n              )", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0091", "text": "{\\displaystyle m_{n}=\\sum _{i=1}^{n}{\\frac {Y_{i}\\mathbf {1} _{\\mathbf {X} _{i}\\in A_{n}(\\mathbf {x} ,\\Theta _{j})}}{N_{n}(\\mathbf {x} ,\\Theta _{j})}}}\n  \n, where \n  \n    \n      \n        \n          A\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          \u0398\n          \n            j\n          \n        \n        )\n      \n    \n    {\\displaystyle A_{n}(\\mathbf {x} ,\\Theta _{j})}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0092", "text": "is the cell containing \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n, designed with randomness \n  \n    \n      \n        \n          \u0398\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\Theta _{j}}\n  \n and dataset \n  \n    \n      \n        \n          \n            \n              D\n            \n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}_{n}}\n  \n, and", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0093", "text": ", and \n  \n    \n      \n        \n          N\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          \u0398\n          \n            j\n          \n        \n        )\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          \n            1\n          \n          \n            \n              \n                X", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0094", "text": "X\n              \n              \n                i\n              \n            \n            \u2208\n            \n              A\n              \n                n\n              \n            \n            (\n            \n              x\n            \n            ,\n            \n              \u0398\n              \n                j\n              \n            \n            )", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0095", "text": ")\n          \n        \n      \n    \n    {\\displaystyle N_{n}(\\mathbf {x} ,\\Theta _{j})=\\sum _{i=1}^{n}\\mathbf {1} _{\\mathbf {X} _{i}\\in A_{n}(\\mathbf {x} ,\\Theta _{j})}}\n  \n.\nThus random forest estimates satisfy, for all \n  \n    \n      \n        \n          x\n        \n        \u2208\n        [\n        0\n        ,\n        1\n        \n          ]\n          \n            d\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} \\in [0,1]^{d}}\n  \n, \n  \n    \n      \n        \n          m", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0096", "text": ", \n  \n    \n      \n        \n          m\n          \n            M\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          \u0398\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \u0398\n          \n            M\n          \n        \n        )\n        =\n        \n          \n            1\n            M\n          \n        \n        \n          \u2211\n          \n            j\n            =\n            1", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0097", "text": "=\n            1\n          \n          \n            M\n          \n        \n        \n          (\n          \n            \n              \u2211\n              \n                i\n                =\n                1\n              \n              \n                n\n              \n            \n            \n              \n                \n                  \n                    Y\n                    \n                      i", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0098", "text": "1\n                    \n                    \n                      \n                        \n                          X\n                        \n                        \n                          i\n                        \n                      \n                      \u2208\n                      \n                        A\n                        \n                          n", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0099", "text": "(\n                      \n                        x\n                      \n                      ,\n                      \n                        \u0398\n                        \n                          j\n                        \n                      \n                      )\n                    \n                  \n                \n                \n                  \n                    N", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0100", "text": "N\n                    \n                      n\n                    \n                  \n                  (\n                  \n                    x\n                  \n                  ,\n                  \n                    \u0398\n                    \n                      j\n                    \n                  \n                  )\n                \n              \n            \n          \n          )", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0101", "text": ")\n        \n      \n    \n    {\\displaystyle m_{M,n}(\\mathbf {x} ,\\Theta _{1},\\ldots ,\\Theta _{M})={\\frac {1}{M}}\\sum _{j=1}^{M}\\left(\\sum _{i=1}^{n}{\\frac {Y_{i}\\mathbf {1} _{\\mathbf {X} _{i}\\in A_{n}(\\mathbf {x} ,\\Theta _{j})}}{N_{n}(\\mathbf {x} ,\\Theta _{j})}}\\right)}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0102", "text": ". Random regression forest has two levels of averaging, first over the samples in the target cell of a tree, then over all trees. Thus the contributions of observations that are in cells with a high density of data points are smaller than that of observations which belong to less populated cells. In order to improve the random forest methods and compensate the misestimation, Scornet defined KeRF by", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0103", "text": "m\n                ~\n              \n            \n          \n          \n            M\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          \u0398\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \u0398\n          \n            M\n          \n        \n        )\n        =\n        \n          \n            1", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0104", "text": "1\n            \n              \n                \u2211\n                \n                  j\n                  =\n                  1\n                \n                \n                  M\n                \n              \n              \n                N\n                \n                  n\n                \n              \n              (\n              \n                x\n              \n              ,\n              \n                \u0398\n                \n                  j", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0105", "text": "j\n                \n              \n              )\n            \n          \n        \n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            M\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          Y\n          \n            i\n          \n        \n        \n          \n            1", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0106", "text": "1\n          \n          \n            \n              \n                X\n              \n              \n                i\n              \n            \n            \u2208\n            \n              A\n              \n                n\n              \n            \n            (\n            \n              x\n            \n            ,\n            \n              \u0398\n              \n                j\n              \n            \n            )\n          \n        \n        ,", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0107", "text": ",\n      \n    \n    {\\displaystyle {\\tilde {m}}_{M,n}(\\mathbf {x} ,\\Theta _{1},\\ldots ,\\Theta _{M})={\\frac {1}{\\sum _{j=1}^{M}N_{n}(\\mathbf {x} ,\\Theta _{j})}}\\sum _{j=1}^{M}\\sum _{i=1}^{n}Y_{i}\\mathbf {1} _{\\mathbf {X} _{i}\\in A_{n}(\\mathbf {x} ,\\Theta _{j})},}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0108", "text": "which is equal to the mean of the \n  \n    \n      \n        \n          Y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle Y_{i}}\n  \n's falling in the cells containing \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n in the forest. If we define the connection function of the \n  \n    \n      \n        M\n      \n    \n    {\\displaystyle M}\n  \n finite forest as \n  \n    \n      \n        \n          K\n          \n            M", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0109", "text": "K\n          \n            M\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          z\n        \n        )\n        =\n        \n          \n            1\n            M\n          \n        \n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            M\n          \n        \n        \n          \n            1\n          \n          \n            \n              z", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0110", "text": "z\n            \n            \u2208\n            \n              A\n              \n                n\n              \n            \n            (\n            \n              x\n            \n            ,\n            \n              \u0398\n              \n                j\n              \n            \n            )\n          \n        \n      \n    \n    {\\displaystyle K_{M,n}(\\mathbf {x} ,\\mathbf {z} )={\\frac {1}{M}}\\sum _{j=1}^{M}\\mathbf {1} _{\\mathbf {z} \\in A_{n}(\\mathbf {x} ,\\Theta _{j})}}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0111", "text": ", i.e. the proportion of cells shared between \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n and \n  \n    \n      \n        \n          z\n        \n      \n    \n    {\\displaystyle \\mathbf {z} }\n  \n, then almost surely we have \n  \n    \n      \n        \n          \n            \n              \n                m\n                ~\n              \n            \n          \n          \n            M\n            ,\n            n\n          \n        \n        (", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0112", "text": "n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          \u0398\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \u0398\n          \n            M\n          \n        \n        )\n        =\n        \n          \n            \n              \n                \u2211\n                \n                  i\n                  =\n                  1\n                \n                \n                  n", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0113", "text": "Y\n                \n                  i\n                \n              \n              \n                K\n                \n                  M\n                  ,\n                  n\n                \n              \n              (\n              \n                x\n              \n              ,\n              \n                \n                  x\n                \n                \n                  i", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0114", "text": ")\n            \n            \n              \n                \u2211\n                \n                  \u2113\n                  =\n                  1\n                \n                \n                  n\n                \n              \n              \n                K\n                \n                  M\n                  ,\n                  n\n                \n              \n              (\n              \n                x\n              \n              ,", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0115", "text": "x\n              \n              ,\n              \n                \n                  x\n                \n                \n                  \u2113\n                \n              \n              )\n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {m}}_{M,n}(\\mathbf {x} ,\\Theta _{1},\\ldots ,\\Theta _{M})={\\frac {\\sum _{i=1}^{n}Y_{i}K_{M,n}(\\mathbf {x} ,\\mathbf {x} _{i})}{\\sum _{\\ell =1}^{n}K_{M,n}(\\mathbf {x} ,\\mathbf {x} _{\\ell })}}}\n  \n, which defines the KeRF.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0116", "text": "Centered KeRF\nThe construction of Centered KeRF of level \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n is the same as for centered forest, except that predictions are made by \n  \n    \n      \n        \n          \n            \n              \n                m\n                ~\n              \n            \n          \n          \n            M\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          \u0398", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0117", "text": ",\n        \n          \u0398\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \u0398\n          \n            M\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\tilde {m}}_{M,n}(\\mathbf {x} ,\\Theta _{1},\\ldots ,\\Theta _{M})}\n  \n, the corresponding kernel function, or connection function is", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0118", "text": "K\n          \n            k\n          \n          \n            c\n            c\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          z\n        \n        )\n        =\n        \n          \u2211\n          \n            \n              k\n              \n                1\n              \n            \n            ,\n            \u2026\n            ,\n            \n              k\n              \n                d\n              \n            \n            ,", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0119", "text": ",\n            \n              \u2211\n              \n                j\n                =\n                1\n              \n              \n                d\n              \n            \n            \n              k\n              \n                j\n              \n            \n            =\n            k\n          \n        \n        \n          \n            \n              k\n              !\n            \n            \n              \n                k", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0120", "text": "k\n                \n                  1\n                \n              \n              !\n              \u22ef\n              \n                k\n                \n                  d\n                \n              \n              !\n            \n          \n        \n        \n          \n            (\n            \n              \n                1\n                d\n              \n            \n            )\n          \n          \n            k", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0121", "text": "k\n          \n        \n        \n          \u220f\n          \n            j\n            =\n            1\n          \n          \n            d\n          \n        \n        \n          \n            1\n          \n          \n            \u2308\n            \n              2\n              \n                \n                  k\n                  \n                    j\n                  \n                \n              \n            \n            \n              x\n              \n                j", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0122", "text": "j\n              \n            \n            \u2309\n            =\n            \u2308\n            \n              2\n              \n                \n                  k\n                  \n                    j\n                  \n                \n              \n            \n            \n              z\n              \n                j\n              \n            \n            \u2309\n          \n        \n        ,\n        \n        \n           for all \n        \n        \n          x", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0123", "text": "x\n        \n        ,\n        \n          z\n        \n        \u2208\n        [\n        0\n        ,\n        1\n        \n          ]\n          \n            d\n          \n        \n        .", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0124", "text": ".\n      \n    \n    {\\displaystyle K_{k}^{cc}(\\mathbf {x} ,\\mathbf {z} )=\\sum _{k_{1},\\ldots ,k_{d},\\sum _{j=1}^{d}k_{j}=k}{\\frac {k!}{k_{1}!\\cdots k_{d}!}}\\left({\\frac {1}{d}}\\right)^{k}\\prod _{j=1}^{d}\\mathbf {1} _{\\lceil 2^{k_{j}}x_{j}\\rceil =\\lceil 2^{k_{j}}z_{j}\\rceil },\\qquad {\\text{ for all }}\\mathbf {x} ,\\mathbf {z} \\in [0,1]^{d}.}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0125", "text": "Uniform KeRF\nUniform KeRF is built in the same way as uniform forest, except that predictions are made by \n  \n    \n      \n        \n          \n            \n              \n                m\n                ~\n              \n            \n          \n          \n            M\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \n          \u0398\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \u0398", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0126", "text": ",\n        \n          \u0398\n          \n            M\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\tilde {m}}_{M,n}(\\mathbf {x} ,\\Theta _{1},\\ldots ,\\Theta _{M})}\n  \n, the corresponding kernel function, or connection function is", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0127", "text": "K\n          \n            k\n          \n          \n            u\n            f\n          \n        \n        (\n        \n          0\n        \n        ,\n        \n          x\n        \n        )\n        =\n        \n          \u2211\n          \n            \n              k\n              \n                1\n              \n            \n            ,\n            \u2026\n            ,\n            \n              k\n              \n                d\n              \n            \n            ,", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0128", "text": ",\n            \n              \u2211\n              \n                j\n                =\n                1\n              \n              \n                d\n              \n            \n            \n              k\n              \n                j\n              \n            \n            =\n            k\n          \n        \n        \n          \n            \n              k\n              !\n            \n            \n              \n                k", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0129", "text": "k\n                \n                  1\n                \n              \n              !\n              \u2026\n              \n                k\n                \n                  d\n                \n              \n              !\n            \n          \n        \n        \n          \n            (\n            \n              \n                1\n                d\n              \n            \n            )\n          \n          \n            k", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0130", "text": "k\n          \n        \n        \n          \u220f\n          \n            m\n            =\n            1\n          \n          \n            d\n          \n        \n        \n          (\n          \n            1\n            \u2212\n            \n              |\n            \n            \n              x\n              \n                m\n              \n            \n            \n              |\n            \n            \n              \u2211\n              \n                j\n                =\n                0", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0131", "text": "=\n                0\n              \n              \n                \n                  k\n                  \n                    m\n                  \n                \n                \u2212\n                1\n              \n            \n            \n              \n                \n                  \n                    (\n                    \n                      \u2212\n                      ln\n                      \u2061\n                      \n                        |", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0132", "text": "|\n                      \n                      \n                        x\n                        \n                          m\n                        \n                      \n                      \n                        |\n                      \n                    \n                    )\n                  \n                  \n                    j\n                  \n                \n                \n                  j\n                  !", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0133", "text": ")\n        \n        \n           for all \n        \n        \n          x\n        \n        \u2208\n        [\n        0\n        ,\n        1\n        \n          ]\n          \n            d\n          \n        \n        .", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0134", "text": ".\n      \n    \n    {\\displaystyle K_{k}^{uf}(\\mathbf {0} ,\\mathbf {x} )=\\sum _{k_{1},\\ldots ,k_{d},\\sum _{j=1}^{d}k_{j}=k}{\\frac {k!}{k_{1}!\\ldots k_{d}!}}\\left({\\frac {1}{d}}\\right)^{k}\\prod _{m=1}^{d}\\left(1-|x_{m}|\\sum _{j=0}^{k_{m}-1}{\\frac {\\left(-\\ln |x_{m}|\\right)^{j}}{j!}}\\right){\\text{ for all }}\\mathbf {x} \\in [0,1]^{d}.}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0135", "text": "Properties\nRelation between KeRF and random forest\nPredictions given by KeRF and random forests are close if the number of points in each cell is controlled:\n\nAssume that there exist sequences \n  \n    \n      \n        (\n        \n          a\n          \n            n\n          \n        \n        )\n        ,\n        (\n        \n          b\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle (a_{n}),(b_{n})}\n  \n such that, almost surely,", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0136", "text": "a\n          \n            n\n          \n        \n        \u2264\n        \n          N\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \u0398\n        )\n        \u2264\n        \n          b\n          \n            n\n          \n        \n        \n           and \n        \n        \n          a\n          \n            n\n          \n        \n        \u2264\n        \n          \n            1\n            M", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0137", "text": "M\n          \n        \n        \n          \u2211\n          \n            m\n            =\n            1\n          \n          \n            M\n          \n        \n        \n          N\n          \n            n\n          \n        \n        \n          \n            x\n          \n          ,\n          \n            \u0398\n            \n              m\n            \n          \n        \n        \u2264\n        \n          b\n          \n            n\n          \n        \n        .", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0138", "text": ".\n      \n    \n    {\\displaystyle a_{n}\\leq N_{n}(\\mathbf {x} ,\\Theta )\\leq b_{n}{\\text{ and }}a_{n}\\leq {\\frac {1}{M}}\\sum _{m=1}^{M}N_{n}{\\mathbf {x} ,\\Theta _{m}}\\leq b_{n}.}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0139", "text": "Then almost surely,", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0140", "text": "|\n        \n        \n          m\n          \n            M\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        )\n        \u2212\n        \n          \n            \n              \n                m\n                ~\n              \n            \n          \n          \n            M\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        )\n        \n          |\n        \n        \u2264", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0141", "text": "|\n        \n        \u2264\n        \n          \n            \n              \n                b\n                \n                  n\n                \n              \n              \u2212\n              \n                a\n                \n                  n\n                \n              \n            \n            \n              a\n              \n                n\n              \n            \n          \n        \n        \n          \n            \n              \n                m\n                ~", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0142", "text": "m\n                ~\n              \n            \n          \n          \n            M\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        )\n        .\n      \n    \n    {\\displaystyle |m_{M,n}(\\mathbf {x} )-{\\tilde {m}}_{M,n}(\\mathbf {x} )|\\leq {\\frac {b_{n}-a_{n}}{a_{n}}}{\\tilde {m}}_{M,n}(\\mathbf {x} ).}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0143", "text": "Relation between infinite KeRF and infinite random forest\nWhen the number of trees \n  \n    \n      \n        M\n      \n    \n    {\\displaystyle M}\n  \n goes to infinity, then we have infinite random forest and infinite KeRF. Their estimates are close if the number of observations in each cell is bounded:", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0144", "text": "Assume that there exist sequences \n  \n    \n      \n        (\n        \n          \u03b5\n          \n            n\n          \n        \n        )\n        ,\n        (\n        \n          a\n          \n            n\n          \n        \n        )\n        ,\n        (\n        \n          b\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle (\\varepsilon _{n}),(a_{n}),(b_{n})}\n  \n such that, almost surely", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0145", "text": "E\n        \u2061\n        [\n        \n          N\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \u0398\n        )\n        ]\n        \u2265\n        1\n        ,\n      \n    \n    {\\displaystyle \\operatorname {E} [N_{n}(\\mathbf {x} ,\\Theta )]\\geq 1,}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0146", "text": "P\n        \u2061\n        [\n        \n          a\n          \n            n\n          \n        \n        \u2264\n        \n          N\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \u0398\n        )\n        \u2264\n        \n          b\n          \n            n\n          \n        \n        \u2223\n        \n          \n            \n              D\n            \n          \n          \n            n\n          \n        \n        ]\n        \u2265\n        1", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0147", "text": "]\n        \u2265\n        1\n        \u2212\n        \n          \u03b5\n          \n            n\n          \n        \n        \n          /\n        \n        2\n        ,\n      \n    \n    {\\displaystyle \\operatorname {P} [a_{n}\\leq N_{n}(\\mathbf {x} ,\\Theta )\\leq b_{n}\\mid {\\mathcal {D}}_{n}]\\geq 1-\\varepsilon _{n}/2,}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0148", "text": "P\n        \u2061\n        [\n        \n          a\n          \n            n\n          \n        \n        \u2264\n        \n          E\n          \n            \u0398\n          \n        \n        \u2061\n        [\n        \n          N\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        ,\n        \u0398\n        )\n        ]\n        \u2264\n        \n          b\n          \n            n\n          \n        \n        \u2223\n        \n          \n            \n              D", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0149", "text": "D\n            \n          \n          \n            n\n          \n        \n        ]\n        \u2265\n        1\n        \u2212\n        \n          \u03b5\n          \n            n\n          \n        \n        \n          /\n        \n        2\n        ,\n      \n    \n    {\\displaystyle \\operatorname {P} [a_{n}\\leq \\operatorname {E} _{\\Theta }[N_{n}(\\mathbf {x} ,\\Theta )]\\leq b_{n}\\mid {\\mathcal {D}}_{n}]\\geq 1-\\varepsilon _{n}/2,}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0150", "text": "Then almost surely,", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0151", "text": "|\n        \n        \n          m\n          \n            \u221e\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        )\n        \u2212\n        \n          \n            \n              \n                m\n                ~\n              \n            \n          \n          \n            \u221e\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        )\n        \n          |\n        \n        \u2264", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0152", "text": "|\n        \n        \u2264\n        \n          \n            \n              \n                b\n                \n                  n\n                \n              \n              \u2212\n              \n                a\n                \n                  n\n                \n              \n            \n            \n              a\n              \n                n\n              \n            \n          \n        \n        \n          \n            \n              \n                m\n                ~", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0153", "text": "m\n                ~\n              \n            \n          \n          \n            \u221e\n            ,\n            n\n          \n        \n        (\n        \n          x\n        \n        )\n        +\n        n\n        \n          \u03b5\n          \n            n\n          \n        \n        \n          (\n          \n            \n              max\n              \n                1\n                \u2264\n                i\n                \u2264\n                n", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0154", "text": "Y\n              \n                i\n              \n            \n          \n          )\n        \n        .\n      \n    \n    {\\displaystyle |m_{\\infty ,n}(\\mathbf {x} )-{\\tilde {m}}_{\\infty ,n}(\\mathbf {x} )|\\leq {\\frac {b_{n}-a_{n}}{a_{n}}}{\\tilde {m}}_{\\infty ,n}(\\mathbf {x} )+n\\varepsilon _{n}\\left(\\max _{1\\leq i\\leq n}Y_{i}\\right).}", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0155", "text": "Consistency results\nAssume that \n  \n    \n      \n        Y\n        =\n        m\n        (\n        \n          X\n        \n        )\n        +\n        \u03b5\n      \n    \n    {\\displaystyle Y=m(\\mathbf {X} )+\\varepsilon }\n  \n, where \n  \n    \n      \n        \u03b5\n      \n    \n    {\\displaystyle \\varepsilon }\n  \n is a centered Gaussian noise, independent of \n  \n    \n      \n        \n          X\n        \n      \n    \n    {\\displaystyle \\mathbf {X} }\n  \n, with finite variance \n  \n    \n      \n        \n          \u03c3", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0156", "text": "\u03c3\n          \n            2\n          \n        \n        <\n        \u221e\n      \n    \n    {\\displaystyle \\sigma ^{2}<\\infty }\n  \n. Moreover, \n  \n    \n      \n        \n          X\n        \n      \n    \n    {\\displaystyle \\mathbf {X} }\n  \n is uniformly distributed on \n  \n    \n      \n        [\n        0\n        ,\n        1\n        \n          ]\n          \n            d\n          \n        \n      \n    \n    {\\displaystyle [0,1]^{d}}\n  \n and \n  \n    \n      \n        m", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0157", "text": "and \n  \n    \n      \n        m\n      \n    \n    {\\displaystyle m}\n  \n is Lipschitz. Scornet proved upper bounds on the rates of consistency for centered KeRF and uniform KeRF.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0158", "text": "Consistency of centered KeRF\nProviding \n  \n    \n      \n        k\n        \u2192\n        \u221e\n      \n    \n    {\\displaystyle k\\rightarrow \\infty }\n  \n and \n  \n    \n      \n        n\n        \n          /\n        \n        \n          2\n          \n            k\n          \n        \n        \u2192\n        \u221e\n      \n    \n    {\\displaystyle n/2^{k}\\rightarrow \\infty }\n  \n, there exists a constant \n  \n    \n      \n        \n          C\n          \n            1\n          \n        \n        >\n        0", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0159", "text": ">\n        0\n      \n    \n    {\\displaystyle C_{1}>0}\n  \n such that, for all \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n,", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0160", "text": "E\n        \n        [\n        \n          \n            \n              \n                m\n                ~\n              \n            \n          \n          \n            n\n          \n          \n            c\n            c\n          \n        \n        (\n        \n          X\n        \n        )\n        \u2212\n        m\n        (\n        \n          X\n        \n        )\n        \n          ]\n          \n            2\n          \n        \n        \u2264\n        \n          C", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0161", "text": "\u2264\n        \n          C\n          \n            1\n          \n        \n        \n          n\n          \n            \u2212\n            1\n            \n              /\n            \n            (\n            3\n            +\n            d\n            log\n            \u2061\n            2\n            )\n          \n        \n        (\n        log\n        \u2061\n        n\n        \n          )\n          \n            2", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0162", "text": "2\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {E} [{\\tilde {m}}_{n}^{cc}(\\mathbf {X} )-m(\\mathbf {X} )]^{2}\\leq C_{1}n^{-1/(3+d\\log 2)}(\\log n)^{2}}\n  \n.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0163", "text": "Consistency of uniform KeRF\nProviding \n  \n    \n      \n        k\n        \u2192\n        \u221e\n      \n    \n    {\\displaystyle k\\rightarrow \\infty }\n  \n and \n  \n    \n      \n        n\n        \n          /\n        \n        \n          2\n          \n            k\n          \n        \n        \u2192\n        \u221e\n      \n    \n    {\\displaystyle n/2^{k}\\rightarrow \\infty }\n  \n, there exists a constant \n  \n    \n      \n        C\n        >\n        0\n      \n    \n    {\\displaystyle C>0}\n  \n such that,", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0164", "text": "E\n        \n        [\n        \n          \n            \n              \n                m\n                ~\n              \n            \n          \n          \n            n\n          \n          \n            u\n            f\n          \n        \n        (\n        \n          X\n        \n        )\n        \u2212\n        m\n        (\n        \n          X\n        \n        )\n        \n          ]\n          \n            2\n          \n        \n        \u2264\n        C", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0165", "text": "\u2264\n        C\n        \n          n\n          \n            \u2212\n            2\n            \n              /\n            \n            (\n            6\n            +\n            3\n            d\n            log\n            \u2061\n            2\n            )\n          \n        \n        (\n        log\n        \u2061\n        n\n        \n          )\n          \n            2", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0166", "text": "2\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {E} [{\\tilde {m}}_{n}^{uf}(\\mathbf {X} )-m(\\mathbf {X} )]^{2}\\leq Cn^{-2/(6+3d\\log 2)}(\\log n)^{2}}\n  \n.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0167", "text": "Disadvantages", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0168", "text": "While random forests often achieve higher accuracy than a single decision tree, they sacrifice the intrinsic interpretability of decision trees. Decision trees are among a fairly small family of machine learning models that are easily interpretable along with linear models, rule-based models, and attention-based models. This interpretability is one of the main advantages of decision trees", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0169", "text": ". It allows developers to confirm that the model has learned realistic information from the data and allows end-users to have trust and confidence in the decisions made by the model. For example, following the path that a decision tree takes to make its decision is quite trivial, but following the paths of tens or hundreds of trees is much harder", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0170", "text": ". To achieve both performance and interpretability, some model compression techniques allow transforming a random forest into a minimal \"born-again\" decision tree that faithfully reproduces the same decision function.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0171", "text": "Another limitation of random forests is that if features are linearly correlated with the target, random forest may not enhance the accuracy of the base learner. Likewise in problems with multiple categorical variables.", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0172", "text": "See also\nBoosting \u2013 Ensemble learning method\nDecision tree learning \u2013 Machine learning algorithm\nEnsemble learning \u2013 Statistics and machine learning technique\nGradient boosting \u2013 Machine learning technique\nNon-parametric statistics \u2013 Type of statistical analysisPages displaying short descriptions of redirect targets\nRandomized algorithm \u2013 Algorithm that employs a degree of randomness as part of its logic or procedure", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Random_forest_0173", "text": "References\nFurther reading\nExternal links\nRandom Forests classifier description (Leo Breiman's site)\nLiaw, Andy & Wiener, Matthew \"Classification and Regression by randomForest\" R News (2002) Vol. 2/3 p. 18 (Discussion of the use of the random forest package for R)", "title": "Random forest", "url": "https://en.wikipedia.org/wiki/Random_forest"}, {"id": "rec_Decision_tree_learning_0000", "text": "Decision tree learning is a supervised learning approach used in statistics, data mining and machine learning. In this formalism, a classification or regression decision tree is used as a predictive model to draw conclusions about a set of observations.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0001", "text": "Tree models where the target variable can take a discrete set of values are called classification trees; in these tree structures, leaves represent class labels and branches represent conjunctions of features that lead to those class labels. Decision trees where the target variable can take continuous values (typically real numbers) are called regression trees", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0002", "text": ". More generally, the concept of regression tree can be extended to any kind of object equipped with pairwise dissimilarities such as categorical sequences.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0003", "text": "Decision trees are among the most popular machine learning algorithms given their intelligibility and simplicity because they produce algorithms that are easy to interpret and visualize, even for users without a statistical background.\nIn decision analysis, a decision tree can be used to visually and explicitly represent decisions and decision making. In data mining, a decision tree describes data (but the resulting classification tree can be an input for decision making).", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0004", "text": "General\nDecision tree learning is a method commonly used in data mining. The goal is to create an algorithm that predicts the value of a target variable based on several input variables.\nA decision tree is a simple representation for classifying examples. For this section, assume that all of the input features have finite discrete domains, and there is a single target feature called the \"classification\". Each element of the domain of the classification is called a class.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0005", "text": "A decision tree or a classification tree is a tree in which each internal (non-leaf) node is labeled with an input feature. The arcs coming from a node labeled with an input feature are labeled with each of the possible values of the target feature or the arc leads to a subordinate decision node on a different input feature", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0006", "text": ". Each leaf of the tree is labeled with a class or a probability distribution over the classes, signifying that the data set has been classified by the tree into either a specific class, or into a particular probability distribution (which, if the decision tree is well-constructed, is skewed towards certain subsets of classes).", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0007", "text": "A tree is built by splitting the source set, constituting the root node of the tree, into subsets\u2014which constitute the successor children. The splitting is based on a set of splitting rules based on classification features.  This process is repeated on each derived subset in a recursive manner called recursive partitioning.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0008", "text": "The recursion is completed when the subset at a node has all the same values of the target variable, or when splitting no longer adds value to the predictions. This process of top-down induction of decision trees (TDIDT) is an example of a greedy algorithm, and it is by far the most common strategy for learning decision trees from data.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0009", "text": "In data mining, decision trees can be described also as the combination of mathematical and computational techniques to aid the description, categorization and generalization of a given set of data.\nData comes in records of the form:", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0010", "text": "(\n        \n          \n            x\n          \n        \n        ,\n        Y\n        )\n        =\n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \n          x\n          \n            3\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          x\n          \n            k\n          \n        \n        ,\n        Y\n        )", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0011", "text": ",\n        Y\n        )\n      \n    \n    {\\displaystyle ({\\textbf {x}},Y)=(x_{1},x_{2},x_{3},...,x_{k},Y)}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0012", "text": "The dependent variable, \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n, is the target variable that we are trying to understand, classify or generalize. The vector \n  \n    \n      \n        \n          \n            x\n          \n        \n      \n    \n    {\\displaystyle {\\textbf {x}}}\n  \n is composed of the features, \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0013", "text": "2\n          \n        \n        ,\n        \n          x\n          \n            3\n          \n        \n      \n    \n    {\\displaystyle x_{1},x_{2},x_{3}}\n  \n etc., that are used for that task.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0014", "text": "Decision tree types\nDecision trees used in data mining are of two main types:", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0015", "text": "Classification tree analysis is when the predicted outcome is the class (discrete) to which the data belongs.\nRegression tree analysis is when the predicted outcome can be considered a real number (e.g. the price of a house, or a patient's length of stay in a hospital).", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0016", "text": "The term classification and regression tree (CART) analysis is an umbrella term used to refer to either of the above procedures, first introduced by Breiman et al. (1984). Trees used for regression and trees used for classification have some similarities \u2013 but also some differences, such as the procedure used to determine where to split.\nSome techniques, often called ensemble methods, construct more than one decision tree:", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0017", "text": "Boosted trees Incrementally building an ensemble by training each new instance to emphasize the training instances previously mis-modeled. A typical example is AdaBoost. These can be used for regression-type and classification-type problems.\nCommittees of decision trees (also called k-DT), an early method that used randomized decision tree algorithms to generate multiple different trees from the training data, and then combine them using majority voting to generate output.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0018", "text": "Bootstrap aggregated (or bagged) decision trees, an early ensemble method, builds multiple decision trees by repeatedly resampling training data with replacement, and voting the trees for a consensus prediction.\nA random forest classifier is a specific type of bootstrap aggregating\nRotation forest \u2013 in which every decision tree is trained by first applying principal component analysis (PCA) on a random subset of the input features.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0019", "text": "A special case of a decision tree is a decision list, which is a one-sided decision tree, so that every internal node has exactly 1 leaf node and exactly 1 internal node as a child (except for the bottommost node, whose only child is a single leaf node).  While less expressive, decision lists are arguably easier to understand than general decision trees due to their added sparsity, permit non-greedy learning methods and monotonic constraints to be imposed.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0020", "text": "Notable decision tree algorithms include:", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0021", "text": "ID3 (Iterative Dichotomiser 3)\nC4.5 (successor of ID3)\nCART (Classification And Regression Tree)\nOC1 (Oblique classifier 1). First method that created multivariate splits at each node.\nChi-square automatic interaction detection (CHAID). Performs multi-level splits when computing classification trees.\nMARS: extends decision trees to handle numerical data better.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0022", "text": "Conditional Inference Trees. Statistics-based approach that uses non-parametric tests as splitting criteria, corrected for multiple testing to avoid overfitting. This approach results in unbiased predictor selection and does not require pruning.\nID3 and CART were invented independently at around the same time (between 1970 and 1980), yet follow a similar approach for learning a decision tree from training tuples.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0023", "text": "It has also been proposed to leverage concepts of fuzzy set theory for the definition of a special version of decision tree, known as Fuzzy Decision Tree (FDT).\nIn this type of fuzzy classification, generally, an input vector \n  \n    \n      \n        \n          \n            x\n          \n        \n      \n    \n    {\\displaystyle {\\textbf {x}}}\n  \n is associated with multiple classes, each with a different confidence value.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0024", "text": "Boosted ensembles of FDTs have been recently investigated as well, and they have shown performances comparable to those of other very efficient fuzzy classifiers.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0025", "text": "Metrics", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0026", "text": "Algorithms for constructing decision trees usually work top-down, by choosing a variable at each step that best splits the set of items. Different algorithms use different metrics for measuring \"best\".  These generally measure the homogeneity of the target variable within the subsets. Some examples are given below. These metrics are applied to each candidate subset, and the resulting values are combined (e.g., averaged) to provide a measure of the quality of the split", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0027", "text": ". Depending on the underlying metric, the performance of various heuristic algorithms for decision tree learning may vary significantly.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0028", "text": "Estimate of Positive Correctness\nA simple and effective metric can be used to identify the degree to which true positives outweigh false positives (see Confusion matrix). This metric, \"Estimate of Positive Correctness\" is defined below:\n\n  \n    \n      \n        \n          E\n          \n            P\n          \n        \n        =\n        T\n        P\n        \u2212\n        F\n        P\n      \n    \n    {\\displaystyle E_{P}=TP-FP}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0029", "text": "In this equation, the total false positives (FP) are subtracted from the total true positives (TP). The resulting number gives an estimate on how many positive examples the feature could correctly identify within the data, with higher numbers meaning that the feature could correctly classify more positive samples. Below is an example of how to use the metric when the full confusion matrix of a certain feature is given:\nFeature A Confusion Matrix", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0030", "text": "Here we can see that the TP value would be 8 and the FP value would be 2 (the underlined numbers in the table). When we plug these numbers in the equation we are able to calculate the estimate: \n  \n    \n      \n        \n          E\n          \n            p\n          \n        \n        =\n        T\n        P\n        \u2212\n        F\n        P\n        =\n        8\n        \u2212\n        2\n        =\n        6\n      \n    \n    {\\displaystyle E_{p}=TP-FP=8-2=6}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0031", "text": "{\\displaystyle E_{p}=TP-FP=8-2=6}\n  \n. This means that using the estimate on this feature would have it receive a score of 6.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0032", "text": "However, it should be worth noting that this number is only an estimate. For example, if two features both had a FP value of 2 while one of the features had a higher TP value, that feature would be ranked higher than the other because the resulting estimate when using the equation would give a higher value. This could lead to some inaccuracies when using the metric if some features have more positive samples than others", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0033", "text": ". To combat this, one could use a more powerful metric known as Sensitivity that takes into account the proportions of the values from the confusion matrix to give the actual true positive rate (TPR). The difference between these metrics is shown in the example below:", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0034", "text": "In this example, Feature A had an estimate of 6 and a TPR of approximately 0.73 while Feature B had an estimate of 4 and a TPR of 0.75. This shows that although the positive estimate for some feature may be higher, the more accurate TPR value for that feature may be lower when compared to other features that have a lower positive estimate", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0035", "text": ". Depending on the situation and knowledge of the data and decision trees, one may opt to use the positive estimate for a quick and easy solution to their problem. On the other hand, a more experienced user would most likely prefer to use the TPR value to rank the features because it takes into account the proportions of the data and all the samples that should have been classified as positive.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0036", "text": "Gini impurity\nGini impurity, Gini's diversity index, or Gini-Simpson Index in biodiversity research, is used by the CART (classification and regression tree) algorithm for classification trees. Gini impurity is the probability that a randomly chosen element of a set would be mislabeled if it were labeled randomly and independently according to the distribution of labels in the set. It reaches its minimum (zero) when all cases in the node fall into a single target category.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0037", "text": "For a set of items with \n  \n    \n      \n        J\n      \n    \n    {\\displaystyle J}\n  \n classes and relative frequencies \n  \n    \n      \n        \n          p\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle p_{i}}\n  \n, \n  \n    \n      \n        i\n        \u2208\n        {\n        1\n        ,\n        2\n        ,\n        .\n        .\n        .\n        ,\n        J\n        }\n      \n    \n    {\\displaystyle i\\in \\{1,2,...,J\\}}\n  \n, the probability of choosing an item with label", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0038", "text": "i\n      \n    \n    {\\displaystyle i}\n  \n is \n  \n    \n      \n        \n          p\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle p_{i}}\n  \n, and the probability of miscategorizing that item is \n  \n    \n      \n        \n          \u2211\n          \n            k\n            \u2260\n            i\n          \n        \n        \n          p\n          \n            k\n          \n        \n        =\n        1\n        \u2212\n        \n          p\n          \n            i", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0039", "text": "p\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\sum _{k\\neq i}p_{k}=1-p_{i}}\n  \n. The Gini impurity is computed by summing pairwise products of these probabilities for each class label:", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0040", "text": "I\n          \n            G\n          \n        \n        \u2061\n        (\n        p\n        )\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J\n          \n        \n        \n          (\n          \n            \n              p\n              \n                i\n              \n            \n            \n              \u2211\n              \n                k\n                \u2260\n                i", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0041", "text": "i\n              \n            \n            \n              p\n              \n                k\n              \n            \n          \n          )\n        \n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J\n          \n        \n        \n          p\n          \n            i\n          \n        \n        (\n        1\n        \u2212\n        \n          p\n          \n            i\n          \n        \n        )\n        =", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0042", "text": ")\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J\n          \n        \n        (\n        \n          p\n          \n            i\n          \n        \n        \u2212\n        \n          p\n          \n            i\n          \n          \n            2\n          \n        \n        )\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0043", "text": "J\n          \n        \n        \n          p\n          \n            i\n          \n        \n        \u2212\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J\n          \n        \n        \n          p\n          \n            i\n          \n          \n            2\n          \n        \n        =\n        1\n        \u2212\n        \n          \u2211\n          \n            i\n            =\n            1", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0044", "text": "=\n            1\n          \n          \n            J\n          \n        \n        \n          p\n          \n            i\n          \n          \n            2\n          \n        \n        .\n      \n    \n    {\\displaystyle \\operatorname {I} _{G}(p)=\\sum _{i=1}^{J}\\left(p_{i}\\sum _{k\\neq i}p_{k}\\right)=\\sum _{i=1}^{J}p_{i}(1-p_{i})=\\sum _{i=1}^{J}(p_{i}-p_{i}^{2})=\\sum _{i=1}^{J}p_{i}-\\sum _{i=1}^{J}p_{i}^{2}=1-\\sum _{i=1}^{J}p_{i}^{2}.}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0045", "text": "The Gini impurity is also an information theoretic measure and corresponds to Tsallis Entropy with deformation coefficient \n  \n    \n      \n        q\n        =\n        2\n      \n    \n    {\\displaystyle q=2}\n  \n, which in physics is associated with the lack of information in out-of-equilibrium, non-extensive, dissipative and quantum systems. For the limit \n  \n    \n      \n        q\n        \u2192\n        1\n      \n    \n    {\\displaystyle q\\to 1}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0046", "text": "{\\displaystyle q\\to 1}\n  \n one recovers the usual Boltzmann-Gibbs or Shannon entropy. In this sense, the Gini impurity is nothing but a variation of the usual entropy measure for decision trees.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0047", "text": "Information gain\nUsed by the ID3, C4.5 and C5.0 tree-generation algorithms. Information gain is based on the concept of entropy and information content from information theory.\nEntropy is defined as below", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0048", "text": "H\n        \n        (\n        T\n        )\n        =\n        \n          I\n          \n            E\n          \n        \n        \u2061\n        \n          (\n          \n            \n              p\n              \n                1\n              \n            \n            ,\n            \n              p\n              \n                2\n              \n            \n            ,\n            \u2026\n            ,\n            \n              p\n              \n                J", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0049", "text": "p\n              \n                J\n              \n            \n          \n          )\n        \n        =\n        \u2212\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J\n          \n        \n        \n          p\n          \n            i\n          \n        \n        \n          log\n          \n            2\n          \n        \n        \u2061\n        \n          p\n          \n            i", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0050", "text": "i\n          \n        \n      \n    \n    {\\displaystyle \\mathrm {H} (T)=\\operatorname {I} _{E}\\left(p_{1},p_{2},\\ldots ,p_{J}\\right)=-\\sum _{i=1}^{J}p_{i}\\log _{2}p_{i}}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0051", "text": "where \n  \n    \n      \n        \n          p\n          \n            1\n          \n        \n        ,\n        \n          p\n          \n            2\n          \n        \n        ,\n        \u2026\n      \n    \n    {\\displaystyle p_{1},p_{2},\\ldots }\n  \n are fractions that add up to 1 and represent the percentage of each class present in the child node that results from a split in the tree.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0052", "text": "I\n                G\n                (\n                T\n                ,\n                a\n                )\n              \n              \u23de\n            \n          \n          \n            information gain\n          \n        \n        =\n        \n          \n            \n              \n                \n                  H\n                \n                (\n                T\n                )", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0053", "text": ")\n              \n              \u23de\n            \n          \n          \n            entropy (parent)\n          \n        \n        \u2212\n        \n          \n            \n              \n                \n                  H\n                \n                (\n                T\n                \u2223\n                a\n                )\n              \n              \u23de\n            \n          \n          \n            sum of entropies (children)", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0054", "text": "{\\displaystyle \\overbrace {IG(T,a)} ^{\\text{information gain}}=\\overbrace {\\mathrm {H} (T)} ^{\\text{entropy (parent)}}-\\overbrace {\\mathrm {H} (T\\mid a)} ^{\\text{sum of entropies (children)}}}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0055", "text": "=\n        \u2212\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J\n          \n        \n        \n          p\n          \n            i\n          \n        \n        \n          log\n          \n            2\n          \n        \n        \u2061\n        \n          p\n          \n            i\n          \n        \n        \u2212\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0056", "text": "1\n          \n          \n            J\n          \n        \n        \u2212\n        Pr\n        (\n        i\n        \u2223\n        a\n        )\n        \n          log\n          \n            2\n          \n        \n        \u2061\n        Pr\n        (\n        i\n        \u2223\n        a\n        )\n      \n    \n    {\\displaystyle =-\\sum _{i=1}^{J}p_{i}\\log _{2}p_{i}-\\sum _{i=1}^{J}-\\Pr(i\\mid a)\\log _{2}\\Pr(i\\mid a)}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0057", "text": "Averaging over the possible values of \n  \n    \n      \n        A\n      \n    \n    {\\displaystyle A}\n  \n,", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0058", "text": "E\n                  \n                    A\n                  \n                \n                (\n                IG\n                \u2061\n                (\n                T\n                ,\n                a\n                )\n                )\n              \n              \u23de\n            \n          \n          \n            expected information gain\n          \n        \n        =", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0059", "text": "=\n        \n          \n            \n              \n                I\n                (\n                T\n                ;\n                A\n                )\n              \n              \u23de\n            \n          \n          \n            \n              mutual information between \n            \n            T\n            \n               and \n            \n            A\n          \n        \n        =", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0060", "text": "H\n                \n                (\n                T\n                )\n              \n              \u23de\n            \n          \n          \n            entropy (parent)\n          \n        \n        \u2212\n        \n          \n            \n              \n                \n                  H\n                \n                (\n                T\n                \u2223\n                A\n                )\n              \n              \u23de", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0061", "text": "\u23de\n            \n          \n          \n            weighted sum of entropies (children)\n          \n        \n      \n    \n    {\\displaystyle \\overbrace {E_{A}(\\operatorname {IG} (T,a))} ^{\\text{expected information gain}}=\\overbrace {I(T;A)} ^{{\\text{mutual information between }}T{\\text{ and }}A}=\\overbrace {\\mathrm {H} (T)} ^{\\text{entropy (parent)}}-\\overbrace {\\mathrm {H} (T\\mid A)} ^{\\text{weighted sum of entropies (children)}}}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0062", "text": "=\n        \u2212\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J\n          \n        \n        \n          p\n          \n            i\n          \n        \n        \n          log\n          \n            2\n          \n        \n        \u2061\n        \n          p\n          \n            i\n          \n        \n        \u2212\n        \n          \u2211\n          \n            a\n          \n        \n        p\n        (\n        a\n        )", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0063", "text": "p\n        (\n        a\n        )\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J\n          \n        \n        \u2212\n        Pr\n        (\n        i\n        \u2223\n        a\n        )\n        \n          log\n          \n            2\n          \n        \n        \u2061\n        Pr\n        (\n        i\n        \u2223\n        a\n        )", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0064", "text": "\u2223\n        a\n        )\n      \n    \n    {\\displaystyle =-\\sum _{i=1}^{J}p_{i}\\log _{2}p_{i}-\\sum _{a}p(a)\\sum _{i=1}^{J}-\\Pr(i\\mid a)\\log _{2}\\Pr(i\\mid a)}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0065", "text": "Where weighted sum of entropies is given by,", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0066", "text": "H\n          \n          (\n          T\n          \u2223\n          A\n          )\n        \n        =\n        \n          \u2211\n          \n            a\n          \n        \n        p\n        (\n        a\n        )\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            J\n          \n        \n        \u2212\n        Pr\n        (\n        i\n        \u2223\n        a\n        )\n        \n          log\n          \n            2", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0067", "text": "log\n          \n            2\n          \n        \n        \u2061\n        Pr\n        (\n        i\n        \u2223\n        a\n        )\n      \n    \n    {\\displaystyle {\\mathrm {H} (T\\mid A)}=\\sum _{a}p(a)\\sum _{i=1}^{J}-\\Pr(i\\mid a)\\log _{2}\\Pr(i\\mid a)}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0068", "text": "That is, the expected information gain is the mutual information, meaning that on average, the reduction in the entropy of T is the mutual information.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0069", "text": "Information gain is used to decide which feature to split on at each step in building the tree. Simplicity is best, so we want to keep our tree small. To do so, at each step we should choose the split that results in the most consistent child nodes. A commonly used measure of consistency is called information which is measured in bits", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0070", "text": ". For each node of the tree, the information value \"represents the expected amount of information that would be needed to specify whether a new instance should be classified yes or no, given that the example reached that node\".", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0071", "text": "Consider an example data set with four attributes: outlook (sunny, overcast, rainy), temperature (hot, mild, cool), humidity (high, normal), and windy (true, false), with a binary (yes or no) target variable, play, and 14 data points. To construct a decision tree on this data, we need to compare the information gain of each of four trees, each split on one of the four features", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0072", "text": ". The split with the highest information gain will be taken as the first split and the process will continue until all children nodes each have consistent data, or until the information gain is 0.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0073", "text": "To find the information gain of the split using windy, we must first calculate the information in the data before the split. The original data contained nine yes's and five no's.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0074", "text": "I\n          \n            E\n          \n        \n        (\n        [\n        9\n        ,\n        5\n        ]\n        )\n        =\n        \u2212\n        \n          \n            9\n            14\n          \n        \n        \n          log\n          \n            2\n          \n        \n        \u2061\n        \n          \n            9\n            14\n          \n        \n        \u2212\n        \n          \n            5\n            14\n          \n        \n        \n          log", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0075", "text": "log\n          \n            2\n          \n        \n        \u2061\n        \n          \n            5\n            14\n          \n        \n        =\n        0.94\n      \n    \n    {\\displaystyle I_{E}([9,5])=-{\\frac {9}{14}}\\log _{2}{\\frac {9}{14}}-{\\frac {5}{14}}\\log _{2}{\\frac {5}{14}}=0.94}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0076", "text": "The split using the feature windy results in two children nodes, one for a windy value of true and one for a windy value of false. In this data set, there are six data points with a true windy value, three of which have a play (where play is the target variable) value of yes and three with a play value of no. The eight remaining data points with a windy value of false contain two no's and six yes's. The information of the windy=true node is calculated using the entropy equation above", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0077", "text": ". Since there is an equal number of yes's and no's in this node, we have", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0078", "text": "I\n          \n            E\n          \n        \n        (\n        [\n        3\n        ,\n        3\n        ]\n        )\n        =\n        \u2212\n        \n          \n            3\n            6\n          \n        \n        \n          log\n          \n            2\n          \n        \n        \u2061\n        \n          \n            3\n            6\n          \n        \n        \u2212\n        \n          \n            3\n            6\n          \n        \n        \n          log", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0079", "text": "log\n          \n            2\n          \n        \n        \u2061\n        \n          \n            3\n            6\n          \n        \n        =\n        \u2212\n        \n          \n            1\n            2\n          \n        \n        \n          log\n          \n            2\n          \n        \n        \u2061\n        \n          \n            1\n            2\n          \n        \n        \u2212\n        \n          \n            1\n            2\n          \n        \n        \n          log", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0080", "text": "log\n          \n            2\n          \n        \n        \u2061\n        \n          \n            1\n            2\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle I_{E}([3,3])=-{\\frac {3}{6}}\\log _{2}{\\frac {3}{6}}-{\\frac {3}{6}}\\log _{2}{\\frac {3}{6}}=-{\\frac {1}{2}}\\log _{2}{\\frac {1}{2}}-{\\frac {1}{2}}\\log _{2}{\\frac {1}{2}}=1}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0081", "text": "For the node where windy=false there were eight data points, six yes's and two no's. Thus we have", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0082", "text": "I\n          \n            E\n          \n        \n        (\n        [\n        6\n        ,\n        2\n        ]\n        )\n        =\n        \u2212\n        \n          \n            6\n            8\n          \n        \n        \n          log\n          \n            2\n          \n        \n        \u2061\n        \n          \n            6\n            8\n          \n        \n        \u2212\n        \n          \n            2\n            8\n          \n        \n        \n          log", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0083", "text": "log\n          \n            2\n          \n        \n        \u2061\n        \n          \n            2\n            8\n          \n        \n        =\n        \u2212\n        \n          \n            3\n            4\n          \n        \n        \n          log\n          \n            2\n          \n        \n        \u2061\n        \n          \n            3\n            4\n          \n        \n        \u2212\n        \n          \n            1\n            4\n          \n        \n        \n          log", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0084", "text": "log\n          \n            2\n          \n        \n        \u2061\n        \n          \n            1\n            4\n          \n        \n        =\n        0.81\n      \n    \n    {\\displaystyle I_{E}([6,2])=-{\\frac {6}{8}}\\log _{2}{\\frac {6}{8}}-{\\frac {2}{8}}\\log _{2}{\\frac {2}{8}}=-{\\frac {3}{4}}\\log _{2}{\\frac {3}{4}}-{\\frac {1}{4}}\\log _{2}{\\frac {1}{4}}=0.81}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0085", "text": "To find the information of the split, we take the weighted average of these two numbers based on how many observations fell into which node.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0086", "text": "I\n          \n            E\n          \n        \n        (\n        [\n        3\n        ,\n        3\n        ]\n        ,\n        [\n        6\n        ,\n        2\n        ]\n        )\n        =\n        \n          I\n          \n            E\n          \n        \n        (\n        \n          windy or not\n        \n        )\n        =\n        \n          \n            6\n            14\n          \n        \n        \u22c5\n        1\n        +\n        \n          \n            8", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0087", "text": "+\n        \n          \n            8\n            14\n          \n        \n        \u22c5\n        0.81\n        =\n        0.89\n      \n    \n    {\\displaystyle I_{E}([3,3],[6,2])=I_{E}({\\text{windy or not}})={\\frac {6}{14}}\\cdot 1+{\\frac {8}{14}}\\cdot 0.81=0.89}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0088", "text": "Now we can calculate the information gain achieved by splitting on the windy feature.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0089", "text": "IG\n        \u2061\n        (\n        \n          windy\n        \n        )\n        =\n        \n          I\n          \n            E\n          \n        \n        (\n        [\n        9\n        ,\n        5\n        ]\n        )\n        \u2212\n        \n          I\n          \n            E\n          \n        \n        (\n        [\n        3\n        ,\n        3\n        ]\n        ,\n        [\n        6\n        ,\n        2\n        ]\n        )\n        =\n        0.94\n        \u2212\n        0.89\n        =", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0090", "text": "0.94\n        \u2212\n        0.89\n        =\n        0.05\n      \n    \n    {\\displaystyle \\operatorname {IG} ({\\text{windy}})=I_{E}([9,5])-I_{E}([3,3],[6,2])=0.94-0.89=0.05}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0091", "text": "To build the tree, the information gain of each possible first split would need to be calculated. The best first split is the one that provides the most information gain. This process is repeated for each impure node until the tree is complete. This example is adapted from the example appearing in Witten et al.\nInformation gain is also known as Shannon index in bio diversity research.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0092", "text": "Variance reduction\nIntroduced in CART, variance reduction is often employed in cases where the target variable is continuous (regression tree), meaning that use of many other metrics would first require discretization before being applied. The variance reduction of a node N is defined as the total reduction of the variance of the target variable Y due to the split at this node:", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0093", "text": "I\n          \n            V\n          \n        \n        (\n        N\n        )\n        =\n        \n          \n            1\n            \n              \n                |\n              \n              S\n              \n                \n                  |\n                \n                \n                  2\n                \n              \n            \n          \n        \n        \n          \u2211\n          \n            i\n            \u2208\n            S", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0094", "text": "\u2208\n            S\n          \n        \n        \n          \u2211\n          \n            j\n            \u2208\n            S\n          \n        \n        \n          \n            1\n            2\n          \n        \n        (\n        \n          y\n          \n            i\n          \n        \n        \u2212\n        \n          y\n          \n            j\n          \n        \n        \n          )\n          \n            2\n          \n        \n        \u2212\n        \n          (", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0095", "text": "|\n                  \n                  \n                    S\n                    \n                      t\n                    \n                  \n                  \n                    |\n                  \n                \n                \n                  \n                    |\n                  \n                  S\n                  \n                    |", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0096", "text": "1\n                \n                  \n                    |\n                  \n                  \n                    S\n                    \n                      t\n                    \n                  \n                  \n                    \n                      |\n                    \n                    \n                      2", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0097", "text": "\u2211\n              \n                i\n                \u2208\n                \n                  S\n                  \n                    t\n                  \n                \n              \n            \n            \n              \u2211\n              \n                j\n                \u2208\n                \n                  S\n                  \n                    t", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0098", "text": "1\n                2\n              \n            \n            (\n            \n              y\n              \n                i\n              \n            \n            \u2212\n            \n              y\n              \n                j\n              \n            \n            \n              )\n              \n                2\n              \n            \n            +", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0099", "text": "|\n                  \n                  \n                    S\n                    \n                      f\n                    \n                  \n                  \n                    |\n                  \n                \n                \n                  \n                    |\n                  \n                  S\n                  \n                    |", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0100", "text": "1\n                \n                  \n                    |\n                  \n                  \n                    S\n                    \n                      f\n                    \n                  \n                  \n                    \n                      |\n                    \n                    \n                      2", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0101", "text": "\u2211\n              \n                i\n                \u2208\n                \n                  S\n                  \n                    f\n                  \n                \n              \n            \n            \n              \u2211\n              \n                j\n                \u2208\n                \n                  S\n                  \n                    f", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0102", "text": "1\n                2\n              \n            \n            (\n            \n              y\n              \n                i\n              \n            \n            \u2212\n            \n              y\n              \n                j\n              \n            \n            \n              )\n              \n                2\n              \n            \n          \n          )", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0103", "text": ")\n        \n      \n    \n    {\\displaystyle I_{V}(N)={\\frac {1}{|S|^{2}}}\\sum _{i\\in S}\\sum _{j\\in S}{\\frac {1}{2}}(y_{i}-y_{j})^{2}-\\left({\\frac {|S_{t}|}{|S|}}{\\frac {1}{|S_{t}|^{2}}}\\sum _{i\\in S_{t}}\\sum _{j\\in S_{t}}{\\frac {1}{2}}(y_{i}-y_{j})^{2}+{\\frac {|S_{f}|}{|S|}}{\\frac {1}{|S_{f}|^{2}}}\\sum _{i\\in S_{f}}\\sum _{j\\in S_{f}}{\\frac {1}{2}}(y_{i}-y_{j})^{2}\\right)}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0104", "text": "where \n  \n    \n      \n        S\n      \n    \n    {\\displaystyle S}\n  \n, \n  \n    \n      \n        \n          S\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle S_{t}}\n  \n, and \n  \n    \n      \n        \n          S\n          \n            f\n          \n        \n      \n    \n    {\\displaystyle S_{f}}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0105", "text": "{\\displaystyle S_{f}}\n  \n are the set of presplit sample indices, set of sample indices for which the split test is true, and set of sample indices for which the split test is false, respectively. Each of the above summands are indeed variance estimates, though, written in a form without directly referring to the mean.\nBy replacing \n  \n    \n      \n        (\n        \n          y\n          \n            i\n          \n        \n        \u2212\n        \n          y", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0106", "text": "\u2212\n        \n          y\n          \n            j\n          \n        \n        \n          )\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle (y_{i}-y_{j})^{2}}\n  \n in the formula above with the dissimilarity \n  \n    \n      \n        \n          d\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle d_{ij}}\n  \n between two objects \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n and \n  \n    \n      \n        j", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0107", "text": "and \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n, the variance reduction criterion applies to any kind of object for which pairwise dissimilarities can be computed.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0108", "text": "Measure of \"goodness\"\nUsed by CART in 1984, the measure of \"goodness\" is a function that seeks to optimize the balance of a candidate split's capacity to create pure children with its capacity to create equally-sized children. This process is repeated for each impure node until the tree is complete. The function \n  \n    \n      \n        \u03c6\n        (\n        s\n        \u2223\n        t\n        )\n      \n    \n    {\\displaystyle \\varphi (s\\mid t)}\n  \n, where \n  \n    \n      \n        s", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0109", "text": ", where \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n is a candidate split at node \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n, is defined as below", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0110", "text": "\u03c6\n        (\n        s\n        \u2223\n        t\n        )\n        =\n        2\n        \n          P\n          \n            L\n          \n        \n        \n          P\n          \n            R\n          \n        \n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            class count\n          \n        \n        \n          |\n        \n        P\n        (\n        j\n        \u2223\n        \n          t\n          \n            L", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0111", "text": "t\n          \n            L\n          \n        \n        )\n        \u2212\n        P\n        (\n        j\n        \u2223\n        \n          t\n          \n            R\n          \n        \n        )\n        \n          |\n        \n      \n    \n    {\\displaystyle \\varphi (s\\mid t)=2P_{L}P_{R}\\sum _{j=1}^{\\text{class count}}|P(j\\mid t_{L})-P(j\\mid t_{R})|}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0112", "text": "where \n  \n    \n      \n        \n          t\n          \n            L\n          \n        \n      \n    \n    {\\displaystyle t_{L}}\n  \n and \n  \n    \n      \n        \n          t\n          \n            R\n          \n        \n      \n    \n    {\\displaystyle t_{R}}\n  \n are the left and right children of node \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n using split \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n, respectively; \n  \n    \n      \n        \n          P", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0113", "text": "P\n          \n            L\n          \n        \n      \n    \n    {\\displaystyle P_{L}}\n  \n and \n  \n    \n      \n        \n          P\n          \n            R\n          \n        \n      \n    \n    {\\displaystyle P_{R}}\n  \n are the proportions of records in \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n in \n  \n    \n      \n        \n          t\n          \n            L\n          \n        \n      \n    \n    {\\displaystyle t_{L}}\n  \n and", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0114", "text": "{\\displaystyle t_{L}}\n  \n and \n  \n    \n      \n        \n          t\n          \n            R\n          \n        \n      \n    \n    {\\displaystyle t_{R}}\n  \n, respectively; and \n  \n    \n      \n        P\n        (\n        j\n        \u2223\n        \n          t\n          \n            L\n          \n        \n        )\n      \n    \n    {\\displaystyle P(j\\mid t_{L})}\n  \n and \n  \n    \n      \n        P\n        (\n        j\n        \u2223\n        \n          t\n          \n            R\n          \n        \n        )", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0115", "text": "R\n          \n        \n        )\n      \n    \n    {\\displaystyle P(j\\mid t_{R})}\n  \n are the proportions of class \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n records in \n  \n    \n      \n        \n          t\n          \n            L\n          \n        \n      \n    \n    {\\displaystyle t_{L}}\n  \n and \n  \n    \n      \n        \n          t\n          \n            R\n          \n        \n      \n    \n    {\\displaystyle t_{R}}\n  \n, respectively.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0116", "text": "{\\displaystyle t_{R}}\n  \n, respectively.\nConsider an example data set with three attributes: savings(low, medium, high), assets(low, medium, high), income(numerical value), and a binary target variable credit risk(good, bad) and 8 data points. The full data is presented in the table below. To start a decision tree, we will calculate the maximum value of \n  \n    \n      \n        \u03c6\n        (\n        s\n        \u2223\n        t\n        )\n      \n    \n    {\\displaystyle \\varphi (s\\mid t)}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0117", "text": "{\\displaystyle \\varphi (s\\mid t)}\n  \n using each feature to find which one will split the root node. This process will continue until all children are pure or all \n  \n    \n      \n        \u03c6\n        (\n        s\n        \u2223\n        t\n        )\n      \n    \n    {\\displaystyle \\varphi (s\\mid t)}\n  \n values are below a set threshold.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0118", "text": "To find \n  \n    \n      \n        \u03c6\n        (\n        s\n        \u2223\n        t\n        )\n      \n    \n    {\\displaystyle \\varphi (s\\mid t)}\n  \n of the feature savings, we need to note the quantity of each value. The original data contained three low's, three medium's, and two high's. Out of the low's, one had a good credit risk while out of the medium's and high's, 4 had a good credit risk. Assume a candidate split \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0119", "text": "s\n      \n    \n    {\\displaystyle s}\n  \n such that records with a low savings will be put in the left child and all other records will be put into the right child.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0120", "text": "\u03c6\n        (\n        s\n        \u2223\n        \n          root\n        \n        )\n        =\n        2\n        \u22c5\n        \n          \n            3\n            8\n          \n        \n        \u22c5\n        \n          \n            5\n            8\n          \n        \n        \u22c5\n        \n          (\n          \n            \n              |\n              \n                (\n                \n                  \n                    \n                      1\n                      3", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0121", "text": "1\n                      3\n                    \n                  \n                  \u2212\n                  \n                    \n                      4\n                      5\n                    \n                  \n                \n                )\n              \n              |\n            \n            +\n            \n              |\n              \n                (\n                \n                  \n                    \n                      2\n                      3", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0122", "text": "2\n                      3\n                    \n                  \n                  \u2212\n                  \n                    \n                      1\n                      5\n                    \n                  \n                \n                )\n              \n              |\n            \n          \n          )\n        \n        =\n        0.44", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0123", "text": "=\n        0.44\n      \n    \n    {\\displaystyle \\varphi (s\\mid {\\text{root}})=2\\cdot {\\frac {3}{8}}\\cdot {\\frac {5}{8}}\\cdot \\left(\\left|\\left({\\frac {1}{3}}-{\\frac {4}{5}}\\right)\\right|+\\left|\\left({\\frac {2}{3}}-{\\frac {1}{5}}\\right)\\right|\\right)=0.44}", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0124", "text": "To build the tree, the \"goodness\" of all candidate splits for the root node need to be calculated. The candidate with the maximum value will split the root node, and the process will continue for each impure node until the tree is complete.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0125", "text": "Compared to other metrics such as information gain, the measure of \"goodness\" will attempt to create a more balanced tree, leading to more-consistent decision time. However, it sacrifices some priority for creating pure children which can lead to additional splits that are not present with other metrics.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0126", "text": "Uses\nAdvantages\nAmongst other data mining methods, decision trees have various advantages:", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0127", "text": "Simple to understand and interpret. People are able to understand decision tree models after a brief explanation. Trees can also be displayed graphically in a way that is easy for non-experts to interpret.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0128", "text": "Able to handle both numerical and categorical data. Other techniques are usually specialized in analyzing datasets that have only one type of variable. (For example, relation rules can be used only with nominal variables while neural networks can be used only with numerical variables or categoricals converted to 0-1 values.) Early decision trees were only capable of handling categorical variables, but more recent versions, such as C4.5, do not have this limitation.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0129", "text": "Requires little data preparation. Other techniques often require data normalization. Since trees can handle qualitative predictors, there is no need to create dummy variables.\nUses a white box or open-box model. If a given situation is observable in a model the explanation for the condition is easily explained by Boolean logic. By contrast, in a black box model, the explanation for the results is typically difficult to understand, for example with an artificial neural network.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0130", "text": "Possible to validate a model using statistical tests. That makes it possible to account for the reliability of the model.\nNon-parametric approach that makes no assumptions of the training data or prediction residuals; e.g., no distributional, independence, or constant variance assumptions\nPerforms well with large datasets. Large amounts of data can be analyzed using standard computing resources in reasonable time.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0131", "text": "Accuracy with flexible modeling. These methods may be applied to healthcare research with increased accuracy.\nMirrors human decision making more closely than other approaches. This could be useful when modeling human decisions/behavior.\nRobust against co-linearity, particularly boosting.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0132", "text": "In built feature selection. Additional irrelevant feature will be less used so that they can be removed on subsequent runs. The hierarchy of attributes in a decision tree reflects the importance of attributes. It means that the features on top are the most informative.\nDecision trees can approximate any Boolean function e.g. XOR.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0133", "text": "Limitations\nTrees can be very non-robust. A small change in the training data can result in a large change in the tree and consequently the final predictions.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0134", "text": "The problem of learning an optimal decision tree is known to be NP-complete under several aspects of optimality and even for simple concepts. Consequently, practical decision-tree learning algorithms are based on heuristics such as the greedy algorithm where locally optimal decisions are made at each node. Such algorithms cannot guarantee to return the globally optimal decision tree", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0135", "text": ". To reduce the greedy effect of local optimality, some methods such as the dual information distance (DID) tree were proposed.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0136", "text": "Decision-tree learners can create over-complex trees that do not generalize well from the training data.  (This is known as overfitting.)  Mechanisms such as pruning are necessary to avoid this problem (with the exception of some algorithms such as the Conditional Inference approach, that does not require pruning).\nThe average depth of the tree that is defined by the number of nodes or tests till classification is not guaranteed to be minimal or small under various splitting criteria.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0137", "text": "For data including categorical variables with different numbers of levels, information gain in decision trees is biased in favor of attributes with more levels. To counter this problem,  instead of choosing the attribute with highest information gain, one can choose the attribute with the highest information gain ratio among the attributes whose information gain is greater than the mean information gain", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0138", "text": ". This biases the decision tree against considering attributes with a large number of distinct values, while not giving an unfair advantage to attributes with very low information gain. Alternatively, the issue of biased predictor selection can be avoided by the Conditional Inference approach, a two-stage approach, or adaptive leave-one-out feature selection.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0139", "text": "Implementations\nMany data mining software packages provide implementations of one or more decision tree algorithms (e.g. random forest).\nOpen source examples include:", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0140", "text": "ALGLIB, a C++, C# and Java numerical analysis library with data analysis features (random forest)\nKNIME, a free and open-source data analytics, reporting and integration platform (decision trees, random forest)\nOrange, an open-source data visualization, machine learning and data mining toolkit (random forest)\nR (an open-source software environment for statistical computing, which includes several CART implementations such as rpart, party and randomForest packages),", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0141", "text": "scikit-learn (a free and open-source machine learning library for the Python programming language).\nWeka (a free and open-source data-mining suite, contains many decision tree algorithms),\nNotable commercial software:\n\nMATLAB\nMicrosoft SQL Server\nRapidMiner\nSAS Enterprise Miner\nIBM SPSS Modeler", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0142", "text": "Extensions\nDecision graphs", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0143", "text": "In a decision tree, all paths from the root node to the leaf node proceed by way of conjunction, or AND. In a decision graph, it is possible to use disjunctions (ORs) to join two more paths together using minimum message length (MML).  Decision graphs have been further extended to allow for previously unstated new attributes to be learnt dynamically and used at different places within the graph", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0144", "text": ".  The more general coding scheme results in better predictive accuracy and log-loss probabilistic scoring.  In general, decision graphs infer models with fewer leaves than decision trees.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0145", "text": "Alternative search methods\nEvolutionary algorithms have been used to avoid local optimal decisions and search the decision tree space with little a priori bias.\nIt is also possible for a tree to be sampled using MCMC.\nThe tree can be searched for in a bottom-up fashion. Or several trees can be constructed parallelly to reduce the expected number of tests till classification.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0146", "text": "See also\nReferences\nFurther reading\nJames, Gareth; Witten, Daniela; Hastie, Trevor; Tibshirani, Robert (2017). \"Tree-Based Methods\" (PDF). An Introduction to Statistical Learning: with Applications in R. New York: Springer. pp. 303\u2013336. ISBN 978-1-4614-7137-0.\nHajjem, Ahlem; Bellavance, Fran\u00e7ois; Larocque, Denis (2011). \"Mixed effects regression trees for clustered data\". Statistics & Probability Letters. 81 (4): 451\u2013459. doi:10.1016/j.spl.2010.12.003. ISSN 0167-7152.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0147", "text": "Hajjem, Ahlem; Larocque, Denis; Bellavance, Fran\u00e7ois (2017). \"Generalized mixed effects regression trees\". Statistics & Probability Letters. 126: 114\u2013118. doi:10.1016/j.spl.2017.02.033. ISSN 0167-7152.", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Decision_tree_learning_0148", "text": "External links\nEvolutionary Learning of Decision Trees in C++\nA very detailed explanation of information gain as splitting criterion", "title": "Decision tree learning", "url": "https://en.wikipedia.org/wiki/Decision_tree_learning"}, {"id": "rec_Knearest_neighbors_algorithm_0000", "text": "In statistics, the k-nearest neighbors algorithm (k-NN) is a non-parametric supervised learning method. It was first developed by Evelyn Fix and Joseph Hodges in 1951, and later expanded by Thomas Cover.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0001", "text": "Most often, it is used for classification, as a k-NN classifier, the output of which is a class membership. An object is classified by a plurality vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0002", "text": "The k-NN algorithm can also be generalized for regression. In k-NN regression, also known as nearest neighbor smoothing, the output is the property value for the object. This value is the average of the values of k nearest neighbors. If k = 1, then the output is simply assigned to the value of that single nearest neighbor, also known as nearest neighbor interpolation.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0003", "text": "For both classification and regression, a useful technique can be to assign weights to the contributions of the neighbors, so that nearer neighbors contribute more to the average than distant ones. For example, a common weighting scheme consists of giving each neighbor a weight of 1/d, where d is the distance to the neighbor.\nThe input consists of the k closest training examples in a data set.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0004", "text": "The neighbors are taken from a set of objects for which the class (for k-NN classification) or the object property value (for k-NN regression) is known. This can be thought of as the training set for the algorithm, though no explicit training step is required.\nA peculiarity (sometimes even a disadvantage) of the k-NN algorithm is its sensitivity to the local structure of the data.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0005", "text": "In k-NN classification the function is only approximated locally and all computation is deferred until function evaluation. Since this algorithm relies on distance, if the features represent different physical units or come in vastly different scales, then feature-wise normalizing of the training data can greatly improve its accuracy.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0006", "text": "Statistical setting\nSuppose we have pairs \n  \n    \n      \n        (\n        \n          X\n          \n            1\n          \n        \n        ,\n        \n          Y\n          \n            1\n          \n        \n        )\n        ,\n        (\n        \n          X\n          \n            2\n          \n        \n        ,\n        \n          Y\n          \n            2\n          \n        \n        )\n        ,\n        \u2026\n        ,\n        (\n        \n          X\n          \n            n", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0007", "text": "n\n          \n        \n        ,\n        \n          Y\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle (X_{1},Y_{1}),(X_{2},Y_{2}),\\dots ,(X_{n},Y_{n})}\n  \n taking values in \n  \n    \n      \n        \n          \n            R\n          \n          \n            d\n          \n        \n        \u00d7\n        {\n        1\n        ,\n        2\n        }\n      \n    \n    {\\displaystyle \\mathbb {R} ^{d}\\times \\{1,2\\}}", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0008", "text": ", where Y is the class label of X, so that \n  \n    \n      \n        X\n        \n          |\n        \n        Y\n        =\n        r\n        \u223c\n        \n          P\n          \n            r\n          \n        \n      \n    \n    {\\displaystyle X|Y=r\\sim P_{r}}\n  \n for \n  \n    \n      \n        r\n        =\n        1\n        ,\n        2\n      \n    \n    {\\displaystyle r=1,2}\n  \n (and probability distributions \n  \n    \n      \n        \n          P\n          \n            r", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0009", "text": "r\n          \n        \n      \n    \n    {\\displaystyle P_{r}}\n  \n). Given some norm \n  \n    \n      \n        \u2016\n        \u22c5\n        \u2016\n      \n    \n    {\\displaystyle \\|\\cdot \\|}\n  \n on \n  \n    \n      \n        \n          \n            R\n          \n          \n            d\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {R} ^{d}}\n  \n and a point \n  \n    \n      \n        x\n        \u2208\n        \n          \n            R\n          \n          \n            d", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0010", "text": "d\n          \n        \n      \n    \n    {\\displaystyle x\\in \\mathbb {R} ^{d}}\n  \n, let \n  \n    \n      \n        (\n        \n          X\n          \n            (\n            1\n            )\n          \n        \n        ,\n        \n          Y\n          \n            (\n            1\n            )\n          \n        \n        )\n        ,\n        \u2026\n        ,\n        (\n        \n          X\n          \n            (\n            n\n            )\n          \n        \n        ,\n        \n          Y", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0011", "text": ",\n        \n          Y\n          \n            (\n            n\n            )\n          \n        \n        )\n      \n    \n    {\\displaystyle (X_{(1)},Y_{(1)}),\\dots ,(X_{(n)},Y_{(n)})}\n  \n be a reordering of the training data such that \n  \n    \n      \n        \u2016\n        \n          X\n          \n            (\n            1\n            )\n          \n        \n        \u2212\n        x\n        \u2016\n        \u2264\n        \u22ef\n        \u2264\n        \u2016\n        \n          X\n          \n            (\n            n", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0012", "text": "(\n            n\n            )\n          \n        \n        \u2212\n        x\n        \u2016\n      \n    \n    {\\displaystyle \\|X_{(1)}-x\\|\\leq \\dots \\leq \\|X_{(n)}-x\\|}\n  \n.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0013", "text": "Algorithm\nThe training examples are vectors in a multidimensional feature space, each with a class label. The training phase of the algorithm consists only of storing the feature vectors and class labels of the training samples.\nIn the classification phase, k is a user-defined constant, and an unlabeled vector (a query or test point) is classified by assigning the label which is most frequent among the k training samples nearest to that query point.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0014", "text": "A commonly used distance metric for continuous variables is Euclidean distance. For discrete variables, such as for text classification, another metric can be used, such as the overlap metric (or Hamming distance). In the context of gene expression microarray data, for example, k-NN has been employed with correlation coefficients, such as Pearson and Spearman, as a metric", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0015", "text": ". Often, the classification accuracy of k-NN can be improved significantly if the distance metric is learned with specialized algorithms such as large margin nearest neighbor or neighborhood components analysis.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0016", "text": "A drawback of the basic \"majority voting\" classification occurs when the class distribution is skewed. That is, examples of a more frequent class tend to dominate the prediction of the new example, because they tend to be common among the k nearest neighbors due to their large number. One way to overcome this problem is to weight the classification, taking into account the distance from the test point to each of its k nearest neighbors", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0017", "text": ". The class (or value, in regression problems) of each of the k nearest points is multiplied by a weight proportional to the inverse of the distance from that point to the test point. Another way to overcome skew is by abstraction in data representation. For example, in a self-organizing map (SOM), each node is a representative (a center) of a cluster of similar points, regardless of their density in the original training data. k-NN can then be applied to the SOM.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0018", "text": "Parameter selection\nThe best choice of k depends upon the data; generally, larger values of k reduces effect of the noise on the classification, but make boundaries between classes less distinct. A good k can be selected by various heuristic techniques (see hyperparameter optimization). The special case where the class is predicted to be the class of the closest training sample (i.e. when k = 1) is called the nearest neighbor algorithm.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0019", "text": "The accuracy of the k-NN algorithm can be severely degraded by the presence of noisy or irrelevant features, or if the feature scales are not consistent with their importance. Much research effort has been put into selecting or scaling features to improve classification. A particularly popular approach is the use of evolutionary algorithms to optimize feature scaling. Another popular approach is to scale features by the mutual information of the training data with the training classes.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0020", "text": "In binary (two class) classification problems, it is helpful to choose k to be an odd number as this avoids tied votes. One popular way of choosing the empirically optimal k in this setting is via bootstrap method.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0021", "text": "The 1-nearest neighbor classifier\nThe most intuitive nearest neighbour type classifier is the one nearest neighbour classifier that assigns a point x to the class of its closest neighbour in the feature space, that is \n  \n    \n      \n        \n          C\n          \n            n\n          \n          \n            1\n            n\n            n\n          \n        \n        (\n        x\n        )\n        =\n        \n          Y\n          \n            (\n            1\n            )", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0022", "text": "1\n            )\n          \n        \n      \n    \n    {\\displaystyle C_{n}^{1nn}(x)=Y_{(1)}}\n  \n.\nAs the size of training data set approaches infinity, the one nearest neighbour classifier guarantees an error rate of no worse than twice the Bayes error rate (the minimum achievable error rate given the distribution of the data).", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0023", "text": "The weighted nearest neighbour classifier\nThe k-nearest neighbour classifier can be viewed as assigning the k nearest neighbours a weight \n  \n    \n      \n        1\n        \n          /\n        \n        k\n      \n    \n    {\\displaystyle 1/k}\n  \n and all others 0 weight. This can be generalised to weighted nearest neighbour classifiers. That is, where the ith nearest neighbour is assigned a weight \n  \n    \n      \n        \n          w\n          \n            n\n            i", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0024", "text": "n\n            i\n          \n        \n      \n    \n    {\\displaystyle w_{ni}}\n  \n, with \n  \n    \n      \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          w\n          \n            n\n            i\n          \n        \n        =\n        1\n      \n    \n    {\\textstyle \\sum _{i=1}^{n}w_{ni}=1}", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0025", "text": "{\\textstyle \\sum _{i=1}^{n}w_{ni}=1}\n  \n. An analogous result on the strong consistency of weighted nearest neighbour classifiers also holds.\nLet \n  \n    \n      \n        \n          C\n          \n            n\n          \n          \n            w\n            n\n            n\n          \n        \n      \n    \n    {\\displaystyle C_{n}^{wnn}}\n  \n denote the weighted nearest classifier with weights \n  \n    \n      \n        {\n        \n          w\n          \n            n\n            i", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0026", "text": "n\n            i\n          \n        \n        \n          }\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\{w_{ni}\\}_{i=1}^{n}}\n  \n. Subject to regularity conditions, which in asymptotic theory are conditional variables which require assumptions to differentiate among parameters with some criteria. On the class distributions the excess risk has the following asymptotic expansion", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0027", "text": "R\n            \n          \n          \n            \n              R\n            \n          \n        \n        (\n        \n          C\n          \n            n\n          \n          \n            w\n            n\n            n\n          \n        \n        )\n        \u2212\n        \n          \n            \n              R\n            \n          \n          \n            \n              R\n            \n          \n        \n        (\n        \n          C", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0028", "text": "(\n        \n          C\n          \n            Bayes\n          \n        \n        )\n        =\n        \n          (\n          \n            \n              B\n              \n                1\n              \n            \n            \n              s\n              \n                n\n              \n              \n                2\n              \n            \n            +\n            \n              B\n              \n                2", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0029", "text": "t\n              \n                n\n              \n              \n                2\n              \n            \n          \n          )\n        \n        {\n        1\n        +\n        o\n        (\n        1\n        )\n        }\n        ,\n      \n    \n    {\\displaystyle {\\mathcal {R}}_{\\mathcal {R}}(C_{n}^{wnn})-{\\mathcal {R}}_{\\mathcal {R}}(C^{\\text{Bayes}})=\\left(B_{1}s_{n}^{2}+B_{2}t_{n}^{2}\\right)\\{1+o(1)\\},}", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0030", "text": "for constants \n  \n    \n      \n        \n          B\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle B_{1}}\n  \n and \n  \n    \n      \n        \n          B\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle B_{2}}\n  \n where \n  \n    \n      \n        \n          s\n          \n            n\n          \n          \n            2\n          \n        \n        =\n        \n          \u2211\n          \n            i\n            =\n            1", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0031", "text": "=\n            1\n          \n          \n            n\n          \n        \n        \n          w\n          \n            n\n            i\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle s_{n}^{2}=\\sum _{i=1}^{n}w_{ni}^{2}}\n  \n and \n  \n    \n      \n        \n          t\n          \n            n\n          \n        \n        =\n        \n          n\n          \n            \u2212\n            2\n            \n              /\n            \n            d", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0032", "text": "d\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          w\n          \n            n\n            i\n          \n        \n        \n          {\n          \n            \n              i\n              \n                1\n                +\n                2\n                \n                  /\n                \n                d", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0033", "text": "d\n              \n            \n            \u2212\n            (\n            i\n            \u2212\n            1\n            \n              )\n              \n                1\n                +\n                2\n                \n                  /\n                \n                d\n              \n            \n          \n          }\n        \n      \n    \n    {\\displaystyle t_{n}=n^{-2/d}\\sum _{i=1}^{n}w_{ni}\\left\\{i^{1+2/d}-(i-1)^{1+2/d}\\right\\}}\n  \n.\nThe optimal weighting scheme", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0034", "text": ".\nThe optimal weighting scheme \n  \n    \n      \n        {\n        \n          w\n          \n            n\n            i\n          \n          \n            \u2217\n          \n        \n        \n          }\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\{w_{ni}^{*}\\}_{i=1}^{n}}\n  \n, that balances the two terms in the display above, is given as follows: set \n  \n    \n      \n        \n          k", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0035", "text": "k\n          \n            \u2217\n          \n        \n        =\n        \u230a\n        B\n        \n          n\n          \n            \n              4\n              \n                d\n                +\n                4\n              \n            \n          \n        \n        \u230b\n      \n    \n    {\\displaystyle k^{*}=\\lfloor Bn^{\\frac {4}{d+4}}\\rfloor }\n  \n,", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0036", "text": "w\n          \n            n\n            i\n          \n          \n            \u2217\n          \n        \n        =\n        \n          \n            1\n            \n              k\n              \n                \u2217\n              \n            \n          \n        \n        \n          [\n          \n            1\n            +\n            \n              \n                d\n                2\n              \n            \n            \u2212\n            \n              \n                d", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0037", "text": "d\n                \n                  2\n                  \n                    \n                      \n                        k\n                        \n                          \u2217\n                        \n                      \n                    \n                    \n                      2\n                      \n                        /\n                      \n                      d", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0038", "text": "{\n            \n              i\n              \n                1\n                +\n                2\n                \n                  /\n                \n                d\n              \n            \n            \u2212\n            (\n            i\n            \u2212\n            1\n            \n              )\n              \n                1\n                +\n                2\n                \n                  /", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0039", "text": "/\n                \n                d\n              \n            \n            }\n          \n          ]\n        \n      \n    \n    {\\displaystyle w_{ni}^{*}={\\frac {1}{k^{*}}}\\left[1+{\\frac {d}{2}}-{\\frac {d}{2{k^{*}}^{2/d}}}\\{i^{1+2/d}-(i-1)^{1+2/d}\\}\\right]}\n  \n for \n  \n    \n      \n        i\n        =\n        1\n        ,\n        2\n        ,\n        \u2026\n        ,\n        \n          k\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle i=1,2,\\dots ,k^{*}}", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0040", "text": "{\\displaystyle i=1,2,\\dots ,k^{*}}\n  \n and", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0041", "text": "w\n          \n            n\n            i\n          \n          \n            \u2217\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle w_{ni}^{*}=0}\n  \n for \n  \n    \n      \n        i\n        =\n        \n          k\n          \n            \u2217\n          \n        \n        +\n        1\n        ,\n        \u2026\n        ,\n        n\n      \n    \n    {\\displaystyle i=k^{*}+1,\\dots ,n}\n  \n.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0042", "text": "{\\displaystyle i=k^{*}+1,\\dots ,n}\n  \n.\nWith optimal weights the dominant term in the asymptotic expansion of the excess risk is \n  \n    \n      \n        \n          \n            O\n          \n        \n        (\n        \n          n\n          \n            \u2212\n            \n              \n                4\n                \n                  d\n                  +\n                  4\n                \n              \n            \n          \n        \n        )", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0043", "text": ")\n      \n    \n    {\\displaystyle {\\mathcal {O}}(n^{-{\\frac {4}{d+4}}})}\n  \n. Similar results are true when using a bagged nearest neighbour classifier.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0044", "text": "Furthest-neighbor variants", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0045", "text": "A variation of the nearest neighbor approach uses the k furthest neighbors instead of the nearest ones. In this setting, neighbors are selected based on maximal dissimilarity rather than similarity. This approach has been proposed in recommender systems as an \"inverted neighborhood\" model, where the most dissimilar users are identified and their preferences are used inversely to generate recommendations", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0046", "text": ". The goal is typically to increase diversity or novelty in recommended items, since traditional nearest-neighbor methods may favor popular or obvious items. Empirical evaluations have found that furthest-neighbor approaches generally achieve lower predictive accuracy than standard k-nearest neighbor methods, although user-perceived usefulness may be similar or higher in some cases.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0047", "text": "Properties\nk-NN is a special case of a variable-bandwidth, kernel density \"balloon\" estimator with a uniform kernel.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0048", "text": "The naive version of the algorithm is easy to implement by computing the distances from the test example to all stored examples, but it is computationally intensive for large training sets. Using an approximate nearest neighbor search algorithm makes k-NN computationally tractable even for large data sets. Many nearest neighbor search algorithms have been proposed over the years; these generally seek to reduce the number of distance evaluations actually performed.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0049", "text": "k-NN has some strong consistency results. As the amount of data approaches infinity, the two-class k-NN algorithm is guaranteed to yield an error rate no worse than twice the Bayes error rate (the minimum achievable error rate given the distribution of the data). Various improvements to the k-NN speed are possible by using proximity graphs.\nFor multi-class k-NN classification, Cover and Hart (1967) prove an upper bound error rate of", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0050", "text": "R\n          \n            \u2217\n          \n        \n         \n        \u2264\n         \n        \n          R\n          \n            k\n            \n              N\n              N\n            \n          \n        \n         \n        \u2264\n         \n        \n          R\n          \n            \u2217\n          \n        \n        \n          (\n          \n            2\n            \u2212\n            \n              \n                \n                  M\n                  \n                    R", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0051", "text": "R\n                    \n                      \u2217\n                    \n                  \n                \n                \n                  M\n                  \u2212\n                  1\n                \n              \n            \n          \n          )\n        \n      \n    \n    {\\displaystyle R^{*}\\ \\leq \\ R_{k\\mathrm {NN} }\\ \\leq \\ R^{*}\\left(2-{\\frac {MR^{*}}{M-1}}\\right)}", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0052", "text": "where \n  \n    \n      \n        \n          R\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle R^{*}}\n  \n is the Bayes error rate (which is the minimal error rate possible), \n  \n    \n      \n        \n          R\n          \n            k\n            N\n            N\n          \n        \n      \n    \n    {\\displaystyle R_{kNN}}", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0053", "text": "{\\displaystyle R_{kNN}}\n  \n is the asymptotic k-NN error rate, and M is the number of classes in the problem. This bound is tight in the sense that both the lower and upper bounds are achievable by some distribution. For \n  \n    \n      \n        M\n        =\n        2\n      \n    \n    {\\displaystyle M=2}\n  \n and as the Bayesian error rate \n  \n    \n      \n        \n          R\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle R^{*}}", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0054", "text": "{\\displaystyle R^{*}}\n  \n approaches zero, this limit reduces to \"not more than twice the Bayesian error rate\".", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0055", "text": "Error rates\nThere are many results on the error rate of the k nearest neighbour classifiers. The k-nearest neighbour classifier is strongly (that is for any joint distribution on \n  \n    \n      \n        (\n        X\n        ,\n        Y\n        )\n      \n    \n    {\\displaystyle (X,Y)}\n  \n) consistent provided \n  \n    \n      \n        k\n        :=\n        \n          k\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle k:=k_{n}}\n  \n diverges and", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0056", "text": "diverges and \n  \n    \n      \n        \n          k\n          \n            n\n          \n        \n        \n          /\n        \n        n\n      \n    \n    {\\displaystyle k_{n}/n}\n  \n converges to zero as \n  \n    \n      \n        n\n        \u2192\n        \u221e\n      \n    \n    {\\displaystyle n\\to \\infty }\n  \n.\nLet \n  \n    \n      \n        \n          C\n          \n            n\n          \n          \n            k\n            n\n            n\n          \n        \n      \n    \n    {\\displaystyle C_{n}^{knn}}", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0057", "text": "{\\displaystyle C_{n}^{knn}}\n  \n denote the k nearest neighbour classifier based on a training set of size n. Under certain regularity conditions, the excess risk yields the following asymptotic expansion", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0058", "text": "R\n            \n          \n          \n            \n              R\n            \n          \n        \n        (\n        \n          C\n          \n            n\n          \n          \n            k\n            n\n            n\n          \n        \n        )\n        \u2212\n        \n          \n            \n              R\n            \n          \n          \n            \n              R\n            \n          \n        \n        (\n        \n          C", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0059", "text": "(\n        \n          C\n          \n            Bayes\n          \n        \n        )\n        =\n        \n          {\n          \n            \n              B\n              \n                1\n              \n            \n            \n              \n                1\n                k\n              \n            \n            +\n            \n              B\n              \n                2\n              \n            \n            \n              \n                (", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0060", "text": "(\n                \n                  \n                    k\n                    n\n                  \n                \n                )\n              \n              \n                4\n                \n                  /\n                \n                d\n              \n            \n          \n          }\n        \n        {\n        1\n        +\n        o\n        (\n        1\n        )\n        }\n        ,", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0061", "text": ")\n        }\n        ,\n      \n    \n    {\\displaystyle {\\mathcal {R}}_{\\mathcal {R}}(C_{n}^{knn})-{\\mathcal {R}}_{\\mathcal {R}}(C^{\\text{Bayes}})=\\left\\{B_{1}{\\frac {1}{k}}+B_{2}\\left({\\frac {k}{n}}\\right)^{4/d}\\right\\}\\{1+o(1)\\},}", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0062", "text": "for some constants \n  \n    \n      \n        \n          B\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle B_{1}}\n  \n and \n  \n    \n      \n        \n          B\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle B_{2}}\n  \n.\nThe choice \n  \n    \n      \n        \n          k\n          \n            \u2217\n          \n        \n        =\n        \n          \u230a\n          \n            B\n            \n              n", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0063", "text": "n\n              \n                \n                  4\n                  \n                    d\n                    +\n                    4\n                  \n                \n              \n            \n          \n          \u230b\n        \n      \n    \n    {\\displaystyle k^{*}=\\left\\lfloor Bn^{\\frac {4}{d+4}}\\right\\rfloor }\n  \n offers a trade off between the two terms in the above display, for which the \n  \n    \n      \n        \n          k\n          \n            \u2217", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0064", "text": "\u2217\n          \n        \n      \n    \n    {\\displaystyle k^{*}}\n  \n-nearest neighbour error converges to the Bayes error at the optimal (minimax) rate \n  \n    \n      \n        \n          \n            O\n          \n        \n        \n          (\n          \n            n\n            \n              \u2212\n              \n                \n                  4\n                  \n                    d\n                    +\n                    4", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0065", "text": ")\n        \n      \n    \n    {\\displaystyle {\\mathcal {O}}\\left(n^{-{\\frac {4}{d+4}}}\\right)}\n  \n.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0066", "text": "Metric learning\nThe K-nearest neighbor classification performance can often be significantly improved through (supervised) metric learning. Popular algorithms are neighbourhood components analysis and large margin nearest neighbor. Supervised metric learning algorithms use the label information to learn a new metric or pseudo-metric.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0067", "text": "Feature extraction", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0068", "text": "When the input data to an algorithm is too large to be processed and it is suspected to be redundant (e.g. the same measurement in both feet and meters) then the input data will be transformed into a reduced representation set of features (also named features vector). Transforming the input data into the set of features is called feature extraction", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0069", "text": ". If the features extracted are carefully chosen it is expected that the features set will extract the relevant information from the input data in order to perform the desired task using this reduced representation instead of the full size input. Feature extraction is performed on raw data prior to applying k-NN algorithm on the transformed data in feature space.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0070", "text": "An example of a typical computer vision computation pipeline for face recognition using k-NN including feature extraction and dimension reduction pre-processing steps (usually implemented with OpenCV):", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0071", "text": "Haar face detection\nMean-shift tracking analysis\nPCA or Fisher LDA projection into feature space, followed by k-NN classification", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0072", "text": "Dimension reduction\nFor high-dimensional data (e.g., with number of dimensions more than 10) dimension reduction is usually performed prior to applying the k-NN algorithm in order to avoid the effects of the curse of dimensionality.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0073", "text": "The curse of dimensionality in the k-NN context basically means that Euclidean distance is unhelpful in high dimensions because all vectors are almost equidistant to the search query vector (imagine multiple points lying more or less on a circle with the query point at the center; the distance from the query to all data points in the search space is almost the same).", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0074", "text": "Feature extraction and dimension reduction can be combined in one step using principal component analysis (PCA),  linear discriminant analysis (LDA), or canonical correlation analysis (CCA) techniques as a pre-processing step, followed by clustering by k-NN on feature vectors in reduced-dimension space. This process is also called low-dimensional embedding.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0075", "text": "For very-high-dimensional datasets (e.g. when performing a similarity search on live video streams, DNA data or high-dimensional time series) running a fast approximate k-NN search using locality sensitive hashing, \"random projections\", \"sketches\" or other high-dimensional similarity search techniques from the VLDB toolbox might be the only feasible option.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0076", "text": "Decision boundary\nNearest neighbor rules in effect implicitly compute the decision boundary. It is also possible to compute the decision boundary explicitly, and to do so efficiently, so that the computational complexity is a function of the boundary complexity.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0077", "text": "Data reduction\nData reduction is one of the most important problems for work with huge data sets. Usually, only some of the data points are needed for accurate classification. Those data are called the prototypes and can be found as follows:", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0078", "text": "Select the class-outliers, that is, training data that are classified incorrectly by k-NN (for a given k)\nSeparate the rest of the data into two sets: (i) the prototypes that are used for the classification decisions and (ii) the absorbed points that can be correctly classified by k-NN using prototypes. The absorbed points can then be removed from the training set.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0079", "text": "Selection of class-outliers\nA training example surrounded by examples of other classes is called a class outlier. Causes of class outliers include:", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0080", "text": "random error\ninsufficient training examples of this class (an isolated example appears instead of a cluster)\nmissing important features (the classes are separated in other dimensions which we don't know)\ntoo many training examples of other classes (unbalanced classes) that create a \"hostile\" background for the given small class", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0081", "text": "Class outliers with k-NN produce noise. They can be detected and separated for future analysis. Given two natural numbers, k>r>0, a training example is called a (k,r)NN class-outlier if its k nearest neighbors include more than r examples of other classes.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0082", "text": "Condensed Nearest Neighbor for data reduction\nCondensed nearest neighbor (CNN, the Hart algorithm) is an algorithm designed to reduce the data set for k-NN classification. It selects the set of prototypes U from the training data, such that 1NN with U can classify the examples almost as accurately as 1NN does with the whole data set.\n\nGiven a training set X, CNN works iteratively:", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0083", "text": "Given a training set X, CNN works iteratively:\n\nScan all elements of X, looking for an element x whose nearest prototype from U has a different label than x.\nRemove x from X and add it to U\nRepeat the scan until no more prototypes are added to U.\nUse U instead of X for classification. The examples that are not prototypes are called \"absorbed\" points.\nIt is efficient to scan the training examples in order of decreasing border ratio. The border ratio of a training example x is defined as", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0084", "text": "where \u2016x-y\u2016 is the distance to the closest example y having a different color than x, and \u2016x'-y\u2016 is the distance from y to its closest example x'  with the same label as x.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0085", "text": "The border ratio is in the interval [0,1] because \u2016x'-y\u2016 never exceeds \u2016x-y\u2016. This ordering gives preference to the borders of the classes for inclusion in the set of prototypes U. A point of a different label than x is called external to x. The calculation of the border ratio is illustrated by the figure on the right. The data points are labeled by colors: the initial point is x and its label is red. External points are blue and green. The closest to x external point is y", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0086", "text": ". The closest to x external point is y. The closest to y red point is x' . The border ratio a(x) = \u2016x'-y\u2016 / \u2016x-y\u2016is the attribute of the initial point x.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0087", "text": "Below is an illustration of CNN in a series of figures. There are three classes (red, green and blue). Fig. 1: initially there are 60 points in each class. Fig. 2 shows the 1NN classification map: each pixel is classified by 1NN using all the data. Fig. 3 shows the 5NN classification map. White areas correspond to the unclassified regions, where 5NN voting is tied (for example, if there are two green, two red and one blue points among 5 nearest neighbors). Fig. 4 shows the reduced data set", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0088", "text": ". Fig. 4 shows the reduced data set. The crosses are the class-outliers selected by the (3,2)NN rule (all the three nearest neighbors of these instances belong to other classes); the squares are the prototypes, and the empty circles are the absorbed points. The left bottom corner shows the numbers of the class-outliers, prototypes and absorbed points for all three classes. The number of prototypes varies from 15% to 20% for different classes in this example. Fig", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0089", "text": ". Fig. 5 shows that the 1NN classification map with the prototypes is very similar to that with the initial data set. The figures were produced using the Mirkes applet.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0090", "text": "k-NN regression\nIn k-NN regression, also known as k-NN smoothing, the k-NN algorithm is used for estimating continuous variables. One such algorithm uses a weighted average of the k nearest neighbors, weighted by the inverse of their distance. This algorithm works as follows:", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0091", "text": "Compute the Euclidean or Mahalanobis distance from the query example to the labeled examples.\nOrder the labeled examples by increasing distance.\nFind a heuristically optimal number k of nearest neighbors, based on RMSE. This is done using cross validation.\nCalculate an inverse distance weighted average with the k-nearest multivariate neighbors.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0092", "text": "k-NN outlier", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0093", "text": "The distance to the kth nearest neighbor can also be seen as a local density estimate and thus is also a popular outlier score in anomaly detection. The larger the distance to the k-NN, the lower the local density, the more likely the query point is an outlier", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0094", "text": ". Although quite simple, this outlier model, along with another classic data mining method, local outlier factor, works quite well also in comparison to more recent and more complex approaches, according to a large scale experimental analysis.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0095", "text": "Validation of results\nA confusion matrix or \"matching matrix\" is often used as a tool to validate the accuracy of k-NN classification. More robust statistical methods such as likelihood-ratio test can also be applied.\n\nSee also\nNearest centroid classifier\nClosest pair of points problem\nNearest neighbor graph\nSegmentation-based object categorization", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Knearest_neighbors_algorithm_0096", "text": "References\nFurther reading\nDasarathy, Belur V., ed. (1991). Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques. IEEE Computer Society Press. ISBN 978-0818689307.\nShakhnarovich, Gregory; Darrell, Trevor; Indyk, Piotr, eds. (2005). Nearest-Neighbor Methods in Learning and Vision. MIT Press. ISBN 978-0262195478.", "title": "K-nearest neighbors algorithm", "url": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm"}, {"id": "rec_Naive_Bayes_classifier_0000", "text": "In statistics, naive (sometimes simple or idiot's) Bayes classifiers are a family of \"probabilistic classifiers\" which assumes that the features are conditionally independent, given the target class. In other words, a naive Bayes model assumes the information about the class provided by each variable is unrelated to the information from the others, with no information shared between the predictors", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0001", "text": ". The highly unrealistic nature of this assumption, called the naive independence assumption, is what gives the classifier its name. These classifiers are some of the simplest Bayesian network models.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0002", "text": "Naive Bayes classifiers generally perform worse than more advanced models like logistic regressions, especially at quantifying uncertainty (with naive Bayes models often producing wildly overconfident probabilities). However, they are highly scalable, requiring only one parameter for each feature or predictor in a learning problem", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0003", "text": ". Maximum-likelihood training can be done by evaluating a closed-form expression (simply by counting observations in each group), rather than the expensive iterative approximation algorithms required by most other models.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0004", "text": "Despite the use of Bayes' theorem in the classifier's decision rule, naive Bayes is not (necessarily) a Bayesian method, and naive Bayes models can be fit to data using either Bayesian or frequentist methods.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0005", "text": "Introduction", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0006", "text": "Naive Bayes is a simple technique for constructing classifiers: models that assign class labels to problem instances, represented as vectors of feature values, where the class labels are drawn from some finite set. There is not a single algorithm for training such classifiers, but a family of algorithms based on a common principle: all naive Bayes classifiers assume that the value of a particular feature is independent of the value of any other feature, given the class variable", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0007", "text": ". For example, a fruit may be considered to be an apple if it is red, round, and about 10 cm in diameter.  A naive Bayes classifier considers each of these features to contribute independently to the probability that this fruit is an apple, regardless of any possible correlations between the color, roundness, and diameter features.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0008", "text": "In many practical applications, parameter estimation for naive Bayes models uses the method of maximum likelihood; in other words, one can work with the naive Bayes model without accepting Bayesian probability or using any Bayesian methods.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0009", "text": "Despite their naive design and apparently oversimplified assumptions, naive Bayes classifiers have worked quite well in many complex real-world situations. In 2004, an analysis of the Bayesian classification problem showed that there are sound theoretical reasons for the apparently implausible efficacy of naive Bayes classifiers", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0010", "text": ". Still, a comprehensive comparison with other classification algorithms in 2006 showed that Bayes classification is outperformed by other approaches, such as boosted trees or random forests.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0011", "text": "An advantage of naive Bayes is that it only requires a small amount of training data to estimate the parameters necessary for classification.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0012", "text": "Probabilistic model\nAbstractly, naive Bayes is a conditional probability model: it assigns probabilities \n  \n    \n      \n        p\n        (\n        \n          C\n          \n            k\n          \n        \n        \u2223\n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle p(C_{k}\\mid x_{1},\\ldots ,x_{n})}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0013", "text": "for each of the K possible outcomes or classes \n  \n    \n      \n        \n          C\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle C_{k}}\n  \n given a problem instance to be classified, represented by a vector \n  \n    \n      \n        \n          x\n        \n        =\n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0014", "text": ")\n      \n    \n    {\\displaystyle \\mathbf {x} =(x_{1},\\ldots ,x_{n})}\n  \n encoding some n features (independent variables).\nThe problem with the above formulation is that if the number of features n is large or if a feature can take on a large number of values, then basing such a model on probability tables is infeasible. The model must therefore be reformulated to make it more tractable. Using Bayes' theorem, the conditional probability can be decomposed as:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0015", "text": "p\n        (\n        \n          C\n          \n            k\n          \n        \n        \u2223\n        \n          x\n        \n        )\n        =\n        \n          \n            \n              p\n              (\n              \n                C\n                \n                  k\n                \n              \n              )\n               \n              p\n              (\n              \n                x\n              \n              \u2223\n              \n                C", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0016", "text": "\u2223\n              \n                C\n                \n                  k\n                \n              \n              )\n            \n            \n              p\n              (\n              \n                x\n              \n              )\n            \n          \n        \n        \n      \n    \n    {\\displaystyle p(C_{k}\\mid \\mathbf {x} )={\\frac {p(C_{k})\\ p(\\mathbf {x} \\mid C_{k})}{p(\\mathbf {x} )}}\\,}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0017", "text": "In plain English, using Bayesian probability terminology, the above equation can be written as", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0018", "text": "posterior\n        \n        =\n        \n          \n            \n              \n                prior\n              \n              \u00d7\n              \n                likelihood\n              \n            \n            evidence\n          \n        \n        \n      \n    \n    {\\displaystyle {\\text{posterior}}={\\frac {{\\text{prior}}\\times {\\text{likelihood}}}{\\text{evidence}}}\\,}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0019", "text": "In practice, there is interest only in the numerator of that fraction, because the denominator does not depend on \n  \n    \n      \n        C\n      \n    \n    {\\displaystyle C}\n  \n and the values of the features \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n are given, so that the denominator is effectively constant.\nThe numerator is equivalent to the joint probability model", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0020", "text": "p\n        (\n        \n          C\n          \n            k\n          \n        \n        ,\n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        )\n        \n      \n    \n    {\\displaystyle p(C_{k},x_{1},\\ldots ,x_{n})\\,}\n  \n\nwhich can be rewritten as follows, using the chain rule for repeated applications of the definition of conditional probability:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0021", "text": "p\n                (\n                \n                  C\n                  \n                    k\n                  \n                \n                ,\n                \n                  x\n                  \n                    1\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    n", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0022", "text": "n\n                  \n                \n                )\n              \n              \n                \n                =\n                p\n                (\n                \n                  x\n                  \n                    1\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    n\n                  \n                \n                ,", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0023", "text": ",\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n              \n            \n            \n              \n              \n                \n                =\n                p\n                (\n                \n                  x\n                  \n                    1\n                  \n                \n                \u2223\n                \n                  x", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0024", "text": "x\n                  \n                    2\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    n\n                  \n                \n                ,\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n                 \n                p\n                (", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0025", "text": "(\n                \n                  x\n                  \n                    2\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    n\n                  \n                \n                ,\n                \n                  C\n                  \n                    k\n                  \n                \n                )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0026", "text": "=\n                p\n                (\n                \n                  x\n                  \n                    1\n                  \n                \n                \u2223\n                \n                  x\n                  \n                    2\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0027", "text": "x\n                  \n                    n\n                  \n                \n                ,\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n                 \n                p\n                (\n                \n                  x\n                  \n                    2\n                  \n                \n                \u2223\n                \n                  x", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0028", "text": "x\n                  \n                    3\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    n\n                  \n                \n                ,\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n                 \n                p\n                (", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0029", "text": "(\n                \n                  x\n                  \n                    3\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    n\n                  \n                \n                ,\n                \n                  C\n                  \n                    k\n                  \n                \n                )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0030", "text": "=\n                \u22ef\n              \n            \n            \n              \n              \n                \n                =\n                p\n                (\n                \n                  x\n                  \n                    1\n                  \n                \n                \u2223\n                \n                  x\n                  \n                    2", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0031", "text": "2\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    n\n                  \n                \n                ,\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n                 \n                p\n                (\n                \n                  x", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0032", "text": "x\n                  \n                    2\n                  \n                \n                \u2223\n                \n                  x\n                  \n                    3\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    n\n                  \n                \n                ,\n                \n                  C\n                  \n                    k", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0033", "text": "k\n                  \n                \n                )\n                \u22ef\n                p\n                (\n                \n                  x\n                  \n                    n\n                    \u2212\n                    1\n                  \n                \n                \u2223\n                \n                  x\n                  \n                    n\n                  \n                \n                ,\n                \n                  C", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0034", "text": "C\n                  \n                    k\n                  \n                \n                )\n                 \n                p\n                (\n                \n                  x\n                  \n                    n\n                  \n                \n                \u2223\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n                 \n                p", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0035", "text": "p\n                (\n                \n                  C\n                  \n                    k\n                  \n                \n                )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0036", "text": "{\\displaystyle {\\begin{aligned}p(C_{k},x_{1},\\ldots ,x_{n})&=p(x_{1},\\ldots ,x_{n},C_{k})\\\\&=p(x_{1}\\mid x_{2},\\ldots ,x_{n},C_{k})\\ p(x_{2},\\ldots ,x_{n},C_{k})\\\\&=p(x_{1}\\mid x_{2},\\ldots ,x_{n},C_{k})\\ p(x_{2}\\mid x_{3},\\ldots ,x_{n},C_{k})\\ p(x_{3},\\ldots ,x_{n},C_{k})\\\\&=\\cdots \\\\&=p(x_{1}\\mid x_{2},\\ldots ,x_{n},C_{k})\\ p(x_{2}\\mid x_{3},\\ldots ,x_{n},C_{k})\\cdots p(x_{n-1}\\mid x_{n},C_{k})\\ p(x_{n}\\mid C_{k})\\ p(C_{k})\\\\\\end{aligned}}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0037", "text": "Now the \"naive\" conditional independence assumptions come into play: assume that all features in \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n are mutually independent, conditional on the category \n  \n    \n      \n        \n          C\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle C_{k}}\n  \n. Under this assumption,", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0038", "text": "p\n        (\n        \n          x\n          \n            i\n          \n        \n        \u2223\n        \n          x\n          \n            i\n            +\n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        ,\n        \n          C\n          \n            k\n          \n        \n        )\n        =\n        p\n        (\n        \n          x\n          \n            i\n          \n        \n        \u2223", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0039", "text": "i\n          \n        \n        \u2223\n        \n          C\n          \n            k\n          \n        \n        )\n        \n        .\n      \n    \n    {\\displaystyle p(x_{i}\\mid x_{i+1},\\ldots ,x_{n},C_{k})=p(x_{i}\\mid C_{k})\\,.}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0040", "text": "Thus, the joint model can be expressed as", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0041", "text": "p\n                (\n                \n                  C\n                  \n                    k\n                  \n                \n                \u2223\n                \n                  x\n                  \n                    1\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    n", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0042", "text": "n\n                  \n                \n                )\n                \u221d\n                 \n              \n              \n                p\n                (\n                \n                  C\n                  \n                    k\n                  \n                \n                ,\n                \n                  x\n                  \n                    1\n                  \n                \n                ,\n                \u2026\n                ,", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0043", "text": ",\n                \n                  x\n                  \n                    n\n                  \n                \n                )\n              \n            \n            \n              \n              \n                \n                =\n                p\n                (\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n                 \n                p\n                (", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0044", "text": "(\n                \n                  x\n                  \n                    1\n                  \n                \n                \u2223\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n                 \n                p\n                (\n                \n                  x\n                  \n                    2\n                  \n                \n                \u2223", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0045", "text": "\u2223\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n                 \n                p\n                (\n                \n                  x\n                  \n                    3\n                  \n                \n                \u2223\n                \n                  C\n                  \n                    k\n                  \n                \n                )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0046", "text": ")\n                 \n                \u22ef\n              \n            \n            \n              \n              \n                \n                =\n                p\n                (\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n                \n                  \u220f\n                  \n                    i\n                    =\n                    1", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0047", "text": "n\n                  \n                \n                p\n                (\n                \n                  x\n                  \n                    i\n                  \n                \n                \u2223\n                \n                  C\n                  \n                    k\n                  \n                \n                )\n                \n                ,", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0048", "text": "{\\displaystyle {\\begin{aligned}p(C_{k}\\mid x_{1},\\ldots ,x_{n})\\varpropto \\ &p(C_{k},x_{1},\\ldots ,x_{n})\\\\&=p(C_{k})\\ p(x_{1}\\mid C_{k})\\ p(x_{2}\\mid C_{k})\\ p(x_{3}\\mid C_{k})\\ \\cdots \\\\&=p(C_{k})\\prod _{i=1}^{n}p(x_{i}\\mid C_{k})\\,,\\end{aligned}}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0049", "text": "where \n  \n    \n      \n        \u221d\n      \n    \n    {\\displaystyle \\varpropto }\n  \n denotes proportionality since the denominator \n  \n    \n      \n        p\n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p(\\mathbf {x} )}\n  \n is omitted.\nThis means that under the above independence assumptions, the conditional distribution over the class variable \n  \n    \n      \n        C\n      \n    \n    {\\displaystyle C}\n  \n is:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0050", "text": "p\n        (\n        \n          C\n          \n            k\n          \n        \n        \u2223\n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        )\n        =\n        \n          \n            1\n            Z\n          \n        \n         \n        p\n        (\n        \n          C\n          \n            k\n          \n        \n        )\n        \n          \u220f", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0051", "text": ")\n        \n          \u220f\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        p\n        (\n        \n          x\n          \n            i\n          \n        \n        \u2223\n        \n          C\n          \n            k\n          \n        \n        )\n      \n    \n    {\\displaystyle p(C_{k}\\mid x_{1},\\ldots ,x_{n})={\\frac {1}{Z}}\\ p(C_{k})\\prod _{i=1}^{n}p(x_{i}\\mid C_{k})}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0052", "text": "where the evidence \n  \n    \n      \n        Z\n        =\n        p\n        (\n        \n          x\n        \n        )\n        =\n        \n          \u2211\n          \n            k\n          \n        \n        p\n        (\n        \n          C\n          \n            k\n          \n        \n        )\n         \n        p\n        (\n        \n          x\n        \n        \u2223\n        \n          C\n          \n            k\n          \n        \n        )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0053", "text": ")\n      \n    \n    {\\displaystyle Z=p(\\mathbf {x} )=\\sum _{k}p(C_{k})\\ p(\\mathbf {x} \\mid C_{k})}\n  \n is a scaling factor dependent only on \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle x_{1},\\ldots ,x_{n}}\n  \n, that is, a constant if the values of the feature variables are known.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0054", "text": "Often, it is only necessary to discriminate between classes. In that case, the scaling factor is irrelevant, and it is sufficient to calculate the log-probability up to a factor:\n  \n    \n      \n        ln\n        \u2061\n        p\n        (\n        \n          C\n          \n            k\n          \n        \n        \u2223\n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        )\n        =", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0055", "text": ")\n        =\n        ln\n        \u2061\n        p\n        (\n        \n          C\n          \n            k\n          \n        \n        )\n        +\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        ln\n        \u2061\n        p\n        (\n        \n          x\n          \n            i\n          \n        \n        \u2223\n        \n          C\n          \n            k\n          \n        \n        )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0056", "text": ")\n        \n          \n            \n              \n                \u2212\n                ln\n                \u2061\n                Z\n              \n              \u23df\n            \n          \n          \n            irrelevant\n          \n        \n      \n    \n    {\\displaystyle \\ln p(C_{k}\\mid x_{1},\\ldots ,x_{n})=\\ln p(C_{k})+\\sum _{i=1}^{n}\\ln p(x_{i}\\mid C_{k})\\underbrace {-\\ln Z} _{\\text{irrelevant}}}\n  \nThe scaling factor is irrelevant, since discrimination subtracts it away:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0057", "text": "ln\n        \u2061\n        \n          \n            \n              p\n              (\n              \n                C\n                \n                  k\n                \n              \n              \u2223\n              \n                x\n                \n                  1\n                \n              \n              ,\n              \u2026\n              ,\n              \n                x\n                \n                  n\n                \n              \n              )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0058", "text": ")\n            \n            \n              p\n              (\n              \n                C\n                \n                  l\n                \n              \n              \u2223\n              \n                x\n                \n                  1\n                \n              \n              ,\n              \u2026\n              ,\n              \n                x\n                \n                  n\n                \n              \n              )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0059", "text": ")\n            \n          \n        \n        =\n        \n          (\n          \n            ln\n            \u2061\n            p\n            (\n            \n              C\n              \n                k\n              \n            \n            )\n            +\n            \n              \u2211\n              \n                i\n                =\n                1\n              \n              \n                n\n              \n            \n            ln\n            \u2061\n            p\n            (", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0060", "text": "\u2061\n            p\n            (\n            \n              x\n              \n                i\n              \n            \n            \u2223\n            \n              C\n              \n                k\n              \n            \n            )\n          \n          )\n        \n        \u2212\n        \n          (\n          \n            ln\n            \u2061\n            p\n            (\n            \n              C\n              \n                l\n              \n            \n            )\n            +", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0061", "text": ")\n            +\n            \n              \u2211\n              \n                i\n                =\n                1\n              \n              \n                n\n              \n            \n            ln\n            \u2061\n            p\n            (\n            \n              x\n              \n                i\n              \n            \n            \u2223\n            \n              C\n              \n                l\n              \n            \n            )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0062", "text": ")\n          \n          )\n        \n      \n    \n    {\\displaystyle \\ln {\\frac {p(C_{k}\\mid x_{1},\\ldots ,x_{n})}{p(C_{l}\\mid x_{1},\\ldots ,x_{n})}}=\\left(\\ln p(C_{k})+\\sum _{i=1}^{n}\\ln p(x_{i}\\mid C_{k})\\right)-\\left(\\ln p(C_{l})+\\sum _{i=1}^{n}\\ln p(x_{i}\\mid C_{l})\\right)}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0063", "text": "There are two benefits of using log-probability. One is that it allows an interpretation in information theory, where log-probabilities are units of information in nats. Another is that it avoids arithmetic underflow.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0064", "text": "Constructing a classifier from the probability model", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0065", "text": "The discussion so far has derived the independent feature model, that is, the naive Bayes probability model. The naive Bayes classifier combines this model with a decision rule. One common rule is to pick the hypothesis that is most probable so as to minimize the probability of misclassification; this is known as the maximum a posteriori or MAP decision rule. The corresponding classifier, a Bayes classifier, is the function that assigns a class label", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0066", "text": "y\n              ^\n            \n          \n        \n        =\n        \n          C\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle {\\hat {y}}=C_{k}}\n  \n for some k as follows:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0067", "text": "y\n              ^\n            \n          \n        \n        =\n        \n          \n            argmax\n            \n              k\n              \u2208\n              {\n              1\n              ,\n              \u2026\n              ,\n              K\n              }\n            \n          \n        \n         \n        p\n        (\n        \n          C\n          \n            k\n          \n        \n        )\n        \n          \n            \u220f", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0068", "text": ")\n        \n          \n            \u220f\n            \n              i\n              =\n              1\n            \n            \n              n\n            \n          \n          p\n          (\n          \n            x\n            \n              i\n            \n          \n          \u2223\n          \n            C\n            \n              k\n            \n          \n          )\n          .", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0069", "text": ")\n          .\n        \n      \n    \n    {\\displaystyle {\\hat {y}}={\\underset {k\\in \\{1,\\ldots ,K\\}}{\\operatorname {argmax} }}\\ p(C_{k})\\displaystyle \\prod _{i=1}^{n}p(x_{i}\\mid C_{k}).}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0070", "text": "Parameter estimation and event models\nA class's prior may be calculated by assuming equiprobable classes, i.e., \n  \n    \n      \n        p\n        (\n        \n          C\n          \n            k\n          \n        \n        )\n        =\n        \n          \n            1\n            K\n          \n        \n      \n    \n    {\\displaystyle p(C_{k})={\\frac {1}{K}}}\n  \n, or by calculating an estimate for the class probability from the training set:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0071", "text": "prior for a given class\n        \n        =\n        \n          \n            no. of samples in that class\n            total no. of samples\n          \n        \n        \n      \n    \n    {\\displaystyle {\\text{prior for a given class}}={\\frac {\\text{no. of samples in that class}}{\\text{total no. of samples}}}\\,}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0072", "text": "To estimate the parameters for a feature's distribution, one must assume a distribution or generate nonparametric models for the features from the training set.\nThe assumptions on distributions of features are called the \"event model\" of the naive Bayes classifier. For discrete features like the ones encountered in document classification (include spam filtering), multinomial and Bernoulli distributions are popular. These assumptions lead to two distinct models, which are often confused.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0073", "text": "Gaussian naive Bayes\nWhen dealing with continuous data, a typical assumption is that the continuous values associated with each class are distributed according to a normal (or Gaussian) distribution. For example, suppose the training data contains a continuous attribute, \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n. The data is first segmented by the class, and then the mean and variance of \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0074", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n is computed in each class. Let \n  \n    \n      \n        \n          \u03bc\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle \\mu _{k}}\n  \n be the mean of the values in \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n associated with class \n  \n    \n      \n        \n          C\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle C_{k}}\n  \n, and let \n  \n    \n      \n        \n          \u03c3", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0075", "text": ", and let \n  \n    \n      \n        \n          \u03c3\n          \n            k\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\sigma _{k}^{2}}\n  \n be the Bessel corrected variance of the values in \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n associated with class \n  \n    \n      \n        \n          C\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle C_{k}}\n  \n. Suppose one has collected some observation value", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0076", "text": "v\n      \n    \n    {\\displaystyle v}\n  \n. Then, the probability density of \n  \n    \n      \n        v\n      \n    \n    {\\displaystyle v}\n  \n given a class \n  \n    \n      \n        \n          C\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle C_{k}}\n  \n, i.e., \n  \n    \n      \n        p\n        (\n        x\n        =\n        v\n        \u2223\n        \n          C\n          \n            k\n          \n        \n        )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0077", "text": ")\n      \n    \n    {\\displaystyle p(x=v\\mid C_{k})}\n  \n, can be computed by plugging \n  \n    \n      \n        v\n      \n    \n    {\\displaystyle v}\n  \n into the equation for a normal distribution parameterized by \n  \n    \n      \n        \n          \u03bc\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle \\mu _{k}}\n  \n and \n  \n    \n      \n        \n          \u03c3\n          \n            k\n          \n          \n            2", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0078", "text": "2\n          \n        \n      \n    \n    {\\displaystyle \\sigma _{k}^{2}}\n  \n. Formally,", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0079", "text": "p\n        (\n        x\n        =\n        v\n        \u2223\n        \n          C\n          \n            k\n          \n        \n        )\n        =\n        \n          \n            1\n            \n              2\n              \u03c0\n              \n                \u03c3\n                \n                  k\n                \n                \n                  2\n                \n              \n            \n          \n        \n        \n        \n          e\n          \n            \u2212", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0080", "text": "e\n          \n            \u2212\n            \n              \n                \n                  (\n                  v\n                  \u2212\n                  \n                    \u03bc\n                    \n                      k\n                    \n                  \n                  \n                    )\n                    \n                      2\n                    \n                  \n                \n                \n                  2\n                  \n                    \u03c3", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0081", "text": "\u03c3\n                    \n                      k\n                    \n                    \n                      2\n                    \n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle p(x=v\\mid C_{k})={\\frac {1}{\\sqrt {2\\pi \\sigma _{k}^{2}}}}\\,e^{-{\\frac {(v-\\mu _{k})^{2}}{2\\sigma _{k}^{2}}}}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0082", "text": "Another common technique for handling continuous values is to use binning to discretize the feature values and obtain a new set of Bernoulli-distributed features. Some literature suggests that this is required in order to use naive Bayes, but it is not true, as the discretization may throw away discriminative information.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0083", "text": "Sometimes the distribution of class-conditional marginal densities is far from normal. In these cases, kernel density estimation can be used for a more realistic estimate of the marginal densities of each class. This method, which was introduced by John and Langley, can boost the accuracy of the classifier considerably.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0084", "text": "Multinomial naive Bayes\nWith a multinomial event model, samples (feature vectors) represent the frequencies with which certain events have been generated by a multinomial \n  \n    \n      \n        (\n        \n          p\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          p\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle (p_{1},\\dots ,p_{n})}\n  \n where \n  \n    \n      \n        \n          p\n          \n            i", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0085", "text": "p\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle p_{i}}\n  \n is the probability that event i occurs (or K such multinomials in the multiclass case). A feature vector \n  \n    \n      \n        \n          x\n        \n        =\n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0086", "text": ")\n      \n    \n    {\\displaystyle \\mathbf {x} =(x_{1},\\dots ,x_{n})}\n  \n is then a histogram, with \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0087", "text": "{\\displaystyle x_{i}}\n  \n counting the number of times event i was observed in a particular instance. This is the event model typically used for document classification, with events representing the occurrence of a word in a single document (see bag of words assumption). The likelihood of observing a histogram x is given by:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0088", "text": "p\n        (\n        \n          x\n        \n        \u2223\n        \n          C\n          \n            k\n          \n        \n        )\n        =\n        \n          \n            \n              (\n              \n                \u2211\n                \n                  i\n                  =\n                  1\n                \n                \n                  n\n                \n              \n              \n                x\n                \n                  i", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0089", "text": "i\n                \n              \n              )\n              !\n            \n            \n              \n                \u220f\n                \n                  i\n                  =\n                  1\n                \n                \n                  n\n                \n              \n              \n                x\n                \n                  i\n                \n              \n              !\n            \n          \n        \n        \n          \u220f", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0090", "text": "\u220f\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          \n            \n              p\n              \n                k\n                i\n              \n            \n          \n          \n            \n              x\n              \n                i", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0091", "text": "{\\displaystyle p(\\mathbf {x} \\mid C_{k})={\\frac {(\\sum _{i=1}^{n}x_{i})!}{\\prod _{i=1}^{n}x_{i}!}}\\prod _{i=1}^{n}{p_{ki}}^{x_{i}}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0092", "text": "where \n  \n    \n      \n        \n          p\n          \n            k\n            i\n          \n        \n        :=\n        p\n        (\n        i\n        \u2223\n        \n          C\n          \n            k\n          \n        \n        )\n      \n    \n    {\\displaystyle p_{ki}:=p(i\\mid C_{k})}\n  \n. \nThe multinomial naive Bayes classifier becomes a linear classifier when expressed in log-space:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0093", "text": "log\n                \u2061\n                p\n                (\n                \n                  C\n                  \n                    k\n                  \n                \n                \u2223\n                \n                  x\n                \n                )\n              \n              \n                \n                \u221d\n                log\n                \u2061\n                \n                  (", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0094", "text": "(\n                  \n                    p\n                    (\n                    \n                      C\n                      \n                        k\n                      \n                    \n                    )\n                    \n                      \u220f\n                      \n                        i\n                        =\n                        1\n                      \n                      \n                        n", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0095", "text": "n\n                      \n                    \n                    \n                      \n                        \n                          p\n                          \n                            k\n                            i\n                          \n                        \n                      \n                      \n                        \n                          x\n                          \n                            i", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0096", "text": "i\n                          \n                        \n                      \n                    \n                  \n                  )\n                \n              \n            \n            \n              \n              \n                \n                =\n                log\n                \u2061\n                p\n                (\n                \n                  C\n                  \n                    k\n                  \n                \n                )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0097", "text": ")\n                +\n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  x\n                  \n                    i\n                  \n                \n                \u22c5\n                log\n                \u2061\n                \n                  p", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0098", "text": "p\n                  \n                    k\n                    i\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                b\n                +\n                \n                  \n                    w\n                  \n                  \n                    k\n                  \n                  \n                    \u22a4", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0099", "text": "x\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}\\log p(C_{k}\\mid \\mathbf {x} )&\\varpropto \\log \\left(p(C_{k})\\prod _{i=1}^{n}{p_{ki}}^{x_{i}}\\right)\\\\&=\\log p(C_{k})+\\sum _{i=1}^{n}x_{i}\\cdot \\log p_{ki}\\\\&=b+\\mathbf {w} _{k}^{\\top }\\mathbf {x} \\end{aligned}}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0100", "text": "where \n  \n    \n      \n        b\n        =\n        log\n        \u2061\n        p\n        (\n        \n          C\n          \n            k\n          \n        \n        )\n      \n    \n    {\\displaystyle b=\\log p(C_{k})}\n  \n and \n  \n    \n      \n        \n          w\n          \n            k\n            i\n          \n        \n        =\n        log\n        \u2061\n        \n          p\n          \n            k\n            i\n          \n        \n      \n    \n    {\\displaystyle w_{ki}=\\log p_{ki}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0101", "text": "{\\displaystyle w_{ki}=\\log p_{ki}}\n  \n. Estimating the parameters in log space is advantageous since multiplying a large number of small values can lead to significant rounding error. Applying a log transform reduces the effect of this rounding error.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0102", "text": "If a given class and feature value never occur together in the training data, then the frequency-based probability estimate will be zero, because the probability estimate is directly proportional to the number of occurrences of a feature's value. This is problematic because it will wipe out all information in the other probabilities when they are multiplied", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0103", "text": ". Therefore, it is often desirable to incorporate a small-sample correction, called pseudocount, in all probability estimates such that no probability is ever set to be exactly zero. This way of regularizing naive Bayes is called Laplace smoothing when the pseudocount is one, and Lidstone smoothing in the general case.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0104", "text": "Rennie et al. discuss problems with the multinomial assumption in the context of document classification and possible ways to alleviate those problems, including the use of tf\u2013idf weights instead of raw term frequencies and document length normalization, to produce a naive Bayes classifier that is competitive with support vector machines.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0105", "text": "Bernoulli naive Bayes\nIn the multivariate Bernoulli event model, features are independent Boolean variables (binary variables) describing inputs. Like the multinomial model, this model is popular for document classification tasks, where binary term occurrence features are used rather than term frequencies. If \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0106", "text": "{\\displaystyle x_{i}}\n  \n is a Boolean expressing the occurrence or absence of the i'th term from the vocabulary, then the likelihood of a document given a class \n  \n    \n      \n        \n          C\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle C_{k}}\n  \n is given by:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0107", "text": "p\n        (\n        \n          x\n        \n        \u2223\n        \n          C\n          \n            k\n          \n        \n        )\n        =\n        \n          \u220f\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          p\n          \n            k\n            i\n          \n          \n            \n              x\n              \n                i\n              \n            \n          \n        \n        (", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0108", "text": "(\n        1\n        \u2212\n        \n          p\n          \n            k\n            i\n          \n        \n        \n          )\n          \n            (\n            1\n            \u2212\n            \n              x\n              \n                i\n              \n            \n            )\n          \n        \n      \n    \n    {\\displaystyle p(\\mathbf {x} \\mid C_{k})=\\prod _{i=1}^{n}p_{ki}^{x_{i}}(1-p_{ki})^{(1-x_{i})}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0109", "text": "where \n  \n    \n      \n        \n          p\n          \n            k\n            i\n          \n        \n      \n    \n    {\\displaystyle p_{ki}}\n  \n is the probability of class \n  \n    \n      \n        \n          C\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle C_{k}}\n  \n generating the term \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0110", "text": "{\\displaystyle x_{i}}\n  \n. This event model is especially popular for classifying short texts. It has the benefit of explicitly modelling the absence of terms. Note that a naive Bayes classifier with a Bernoulli event model is not the same as a multinomial NB classifier with frequency counts truncated to one.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0111", "text": "Semi-supervised parameter estimation\nGiven a way to train a naive Bayes classifier from labeled data, it's possible to construct a semi-supervised training algorithm that can learn from a combination of labeled and unlabeled data by running the supervised learning algorithm in a loop:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0112", "text": "Given a collection \n  \n    \n      \n        D\n        =\n        L\n        \u228e\n        U\n      \n    \n    {\\displaystyle D=L\\uplus U}\n  \n of labeled samples L and unlabeled samples U, start by training a naive Bayes classifier on L.\nUntil convergence, do:\nPredict class probabilities \n  \n    \n      \n        P\n        (\n        C\n        \u2223\n        x\n        )\n      \n    \n    {\\displaystyle P(C\\mid x)}\n  \n for all examples x in \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0113", "text": "D\n      \n    \n    {\\displaystyle D}\n  \n.\nRe-train the model based on the probabilities (not the labels) predicted in the previous step.\nConvergence is determined based on improvement to the model likelihood \n  \n    \n      \n        P\n        (\n        D\n        \u2223\n        \u03b8\n        )\n      \n    \n    {\\displaystyle P(D\\mid \\theta )}\n  \n, where \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n denotes the parameters of the naive Bayes model.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0114", "text": "denotes the parameters of the naive Bayes model.\nThis training algorithm is an instance of the more general expectation\u2013maximization algorithm (EM): the prediction step inside the loop is the E-step of EM, while the re-training of naive Bayes is the M-step. The algorithm is formally justified by the assumption that the data are generated by a mixture model, and the components of this mixture model are exactly the classes of the classification problem.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0115", "text": "Discussion", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0116", "text": "Despite the fact that the far-reaching independence assumptions are often inaccurate, the naive Bayes classifier has several properties that make it surprisingly useful in practice. In particular, the decoupling of the class conditional feature distributions means that each distribution can be independently estimated as a one-dimensional distribution", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0117", "text": ". This helps alleviate problems stemming from the curse of dimensionality, such as the need for data sets that scale exponentially with the number of features. While naive Bayes often fails to produce a good estimate for the correct class probabilities, this may not be a requirement for many applications. For example, the naive Bayes classifier will make the correct MAP decision rule classification so long as the correct class is predicted as more probable than any other class", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0118", "text": ". This is true regardless of whether the probability estimate is slightly, or even grossly inaccurate. In this manner, the overall classifier can be robust enough to ignore serious deficiencies in its underlying naive probability model. Other reasons for the observed success of the naive Bayes classifier are discussed in the literature cited below.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0119", "text": "Relation to logistic regression\nIn the case of discrete inputs (indicator or frequency features for discrete events), naive Bayes classifiers form a generative-discriminative pair with multinomial logistic regression classifiers: each naive Bayes classifier can be considered a way of fitting a probability model that optimizes the joint likelihood \n  \n    \n      \n        p\n        (\n        C\n        ,\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p(C,\\mathbf {x} )}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0120", "text": "{\\displaystyle p(C,\\mathbf {x} )}\n  \n, while logistic regression fits the same probability model to optimize the conditional \n  \n    \n      \n        p\n        (\n        C\n        \u2223\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p(C\\mid \\mathbf {x} )}\n  \n.\nMore formally, we have the following:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0121", "text": "The link between the two can be seen by observing that the decision function for naive Bayes (in the binary case) can be rewritten as \"predict class \n  \n    \n      \n        \n          C\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle C_{1}}\n  \n if the odds of \n  \n    \n      \n        p\n        (\n        \n          C\n          \n            1\n          \n        \n        \u2223\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p(C_{1}\\mid \\mathbf {x} )}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0122", "text": "{\\displaystyle p(C_{1}\\mid \\mathbf {x} )}\n  \n exceed those of \n  \n    \n      \n        p\n        (\n        \n          C\n          \n            2\n          \n        \n        \u2223\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p(C_{2}\\mid \\mathbf {x} )}\n  \n\". Expressing this in log-space gives:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0123", "text": "log\n        \u2061\n        \n          \n            \n              p\n              (\n              \n                C\n                \n                  1\n                \n              \n              \u2223\n              \n                x\n              \n              )\n            \n            \n              p\n              (\n              \n                C\n                \n                  2\n                \n              \n              \u2223\n              \n                x", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0124", "text": "\u2223\n              \n                x\n              \n              )\n            \n          \n        \n        =\n        log\n        \u2061\n        p\n        (\n        \n          C\n          \n            1\n          \n        \n        \u2223\n        \n          x\n        \n        )\n        \u2212\n        log\n        \u2061\n        p\n        (\n        \n          C\n          \n            2\n          \n        \n        \u2223\n        \n          x\n        \n        )\n        >\n        0", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0125", "text": ")\n        >\n        0\n      \n    \n    {\\displaystyle \\log {\\frac {p(C_{1}\\mid \\mathbf {x} )}{p(C_{2}\\mid \\mathbf {x} )}}=\\log p(C_{1}\\mid \\mathbf {x} )-\\log p(C_{2}\\mid \\mathbf {x} )>0}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0126", "text": "The left-hand side of this equation is the log-odds, or logit, the quantity predicted by the linear model that underlies logistic regression. Since naive Bayes is also a linear model for the two \"discrete\" event models, it can be reparametrised as a linear function \n  \n    \n      \n        b\n        +\n        \n          \n            w\n          \n          \n            \u22a4\n          \n        \n        x\n        >\n        0\n      \n    \n    {\\displaystyle b+\\mathbf {w} ^{\\top }x>0}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0127", "text": "{\\displaystyle b+\\mathbf {w} ^{\\top }x>0}\n  \n. Obtaining the probabilities is then a matter of applying the logistic function to \n  \n    \n      \n        b\n        +\n        \n          \n            w\n          \n          \n            \u22a4\n          \n        \n        x\n      \n    \n    {\\displaystyle b+\\mathbf {w} ^{\\top }x}\n  \n, or in the multiclass case, the softmax function.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0128", "text": "Discriminative classifiers have lower asymptotic error than generative ones; however, research by Ng and Jordan has shown that in some practical cases naive Bayes can outperform logistic regression because it reaches its asymptotic error faster.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0129", "text": "Examples\nPerson classification\nProblem: classify whether a given person is a male or a female based on the measured features.\nThe features include height, weight, and foot size. Although with NB classifier we treat them as independent, they are not in reality.\n\nTraining\nExample training set below.\n\nThe classifier created from the training set using a Gaussian distribution assumption would be (given variances are unbiased sample variances):", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0130", "text": "The following example assumes equiprobable classes so that P(male)= P(female) = 0.5. This prior probability distribution might be based on prior knowledge of frequencies in the larger population or in the training set.\n\nTesting\nBelow is a sample to be classified as male or female.\n\nIn order to classify the sample, one has to determine which posterior is greater, male or female. For the classification as male the posterior is given by", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0131", "text": "posterior (male)\n        \n        =\n        \n          \n            \n              P\n              (\n              \n                male\n              \n              )\n              \n              p\n              (\n              \n                height\n              \n              \u2223\n              \n                male\n              \n              )\n              \n              p\n              (\n              \n                weight", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0132", "text": "weight\n              \n              \u2223\n              \n                male\n              \n              )\n              \n              p\n              (\n              \n                foot size\n              \n              \u2223\n              \n                male\n              \n              )\n            \n            evidence", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0133", "text": "{\\displaystyle {\\text{posterior (male)}}={\\frac {P({\\text{male}})\\,p({\\text{height}}\\mid {\\text{male}})\\,p({\\text{weight}}\\mid {\\text{male}})\\,p({\\text{foot size}}\\mid {\\text{male}})}{\\text{evidence}}}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0134", "text": "For the classification as female the posterior is given by", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0135", "text": "posterior (female)\n        \n        =\n        \n          \n            \n              P\n              (\n              \n                female\n              \n              )\n              \n              p\n              (\n              \n                height\n              \n              \u2223\n              \n                female\n              \n              )\n              \n              p\n              (\n              \n                weight", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0136", "text": "weight\n              \n              \u2223\n              \n                female\n              \n              )\n              \n              p\n              (\n              \n                foot size\n              \n              \u2223\n              \n                female\n              \n              )\n            \n            evidence", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0137", "text": "{\\displaystyle {\\text{posterior (female)}}={\\frac {P({\\text{female}})\\,p({\\text{height}}\\mid {\\text{female}})\\,p({\\text{weight}}\\mid {\\text{female}})\\,p({\\text{foot size}}\\mid {\\text{female}})}{\\text{evidence}}}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0138", "text": "The evidence (also termed normalizing constant) may be calculated:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0139", "text": "evidence\n                \n                =\n                P\n                (\n                \n                  male\n                \n                )\n                \n                p\n                (\n                \n                  height\n                \n                \u2223\n                \n                  male\n                \n                )\n                \n                p", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0140", "text": "p\n                (\n                \n                  weight\n                \n                \u2223\n                \n                  male\n                \n                )\n                \n                p\n                (\n                \n                  foot size\n                \n                \u2223\n                \n                  male\n                \n                )\n              \n            \n            \n              \n                +", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0141", "text": "+\n                P\n                (\n                \n                  female\n                \n                )\n                \n                p\n                (\n                \n                  height\n                \n                \u2223\n                \n                  female\n                \n                )\n                \n                p\n                (\n                \n                  weight\n                \n                \u2223", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0142", "text": "\u2223\n                \n                  female\n                \n                )\n                \n                p\n                (\n                \n                  foot size\n                \n                \u2223\n                \n                  female\n                \n                )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0143", "text": "{\\displaystyle {\\begin{aligned}{\\text{evidence}}=P({\\text{male}})\\,p({\\text{height}}\\mid {\\text{male}})\\,p({\\text{weight}}\\mid {\\text{male}})\\,p({\\text{foot size}}\\mid {\\text{male}})\\\\+P({\\text{female}})\\,p({\\text{height}}\\mid {\\text{female}})\\,p({\\text{weight}}\\mid {\\text{female}})\\,p({\\text{foot size}}\\mid {\\text{female}})\\end{aligned}}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0144", "text": "However, given the sample, the evidence is a constant and thus scales both posteriors equally. It therefore does not affect classification and can be ignored.  The probability distribution for the sex of the sample can now be determined:\n\n  \n    \n      \n        P\n        (\n        \n          male\n        \n        )\n        =\n        0.5\n      \n    \n    {\\displaystyle P({\\text{male}})=0.5}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0145", "text": "p\n        (\n        \n          height\n        \n        \u2223\n        \n          male\n        \n        )\n        =\n        \n          \n            1\n            \n              2\n              \u03c0\n              \n                \u03c3\n                \n                  2\n                \n              \n            \n          \n        \n        exp\n        \u2061\n        \n          (\n          \n            \n              \n                \u2212\n                (\n                6", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0146", "text": "(\n                6\n                \u2212\n                \u03bc\n                \n                  )\n                  \n                    2\n                  \n                \n              \n              \n                2\n                \n                  \u03c3\n                  \n                    2\n                  \n                \n              \n            \n          \n          )\n        \n        \u2248\n        1.5789\n        ,", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0147", "text": "\u2248\n        1.5789\n        ,\n      \n    \n    {\\displaystyle p({\\text{height}}\\mid {\\text{male}})={\\frac {1}{\\sqrt {2\\pi \\sigma ^{2}}}}\\exp \\left({\\frac {-(6-\\mu )^{2}}{2\\sigma ^{2}}}\\right)\\approx 1.5789,}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0148", "text": "where \n  \n    \n      \n        \u03bc\n        =\n        5.855\n      \n    \n    {\\displaystyle \\mu =5.855}\n  \n and \n  \n    \n      \n        \n          \u03c3\n          \n            2\n          \n        \n        =\n        3.5033\n        \u22c5\n        \n          10\n          \n            \u2212\n            2\n          \n        \n      \n    \n    {\\displaystyle \\sigma ^{2}=3.5033\\cdot 10^{-2}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0149", "text": "are the parameters of normal distribution which have been previously determined from the training set. Note that a value greater than 1 is OK here \u2013 it is a probability density rather than a probability, because height is a continuous variable.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0150", "text": "p\n        (\n        \n          weight\n        \n        \u2223\n        \n          male\n        \n        )\n        =\n        \n          \n            1\n            \n              2\n              \u03c0\n              \n                \u03c3\n                \n                  2\n                \n              \n            \n          \n        \n        exp\n        \u2061\n        \n          (\n          \n            \n              \n                \u2212\n                (\n                130", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0151", "text": "(\n                130\n                \u2212\n                \u03bc\n                \n                  )\n                  \n                    2\n                  \n                \n              \n              \n                2\n                \n                  \u03c3\n                  \n                    2\n                  \n                \n              \n            \n          \n          )\n        \n        =\n        5.9881\n        \u22c5\n        \n          10\n          \n            \u2212", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0152", "text": "10\n          \n            \u2212\n            6\n          \n        \n      \n    \n    {\\displaystyle p({\\text{weight}}\\mid {\\text{male}})={\\frac {1}{\\sqrt {2\\pi \\sigma ^{2}}}}\\exp \\left({\\frac {-(130-\\mu )^{2}}{2\\sigma ^{2}}}\\right)=5.9881\\cdot 10^{-6}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0153", "text": "p\n        (\n        \n          foot size\n        \n        \u2223\n        \n          male\n        \n        )\n        =\n        \n          \n            1\n            \n              2\n              \u03c0\n              \n                \u03c3\n                \n                  2\n                \n              \n            \n          \n        \n        exp\n        \u2061\n        \n          (\n          \n            \n              \n                \u2212\n                (\n                8", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0154", "text": "(\n                8\n                \u2212\n                \u03bc\n                \n                  )\n                  \n                    2\n                  \n                \n              \n              \n                2\n                \n                  \u03c3\n                  \n                    2\n                  \n                \n              \n            \n          \n          )\n        \n        =\n        1.3112\n        \u22c5\n        \n          10\n          \n            \u2212", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0155", "text": "10\n          \n            \u2212\n            3\n          \n        \n      \n    \n    {\\displaystyle p({\\text{foot size}}\\mid {\\text{male}})={\\frac {1}{\\sqrt {2\\pi \\sigma ^{2}}}}\\exp \\left({\\frac {-(8-\\mu )^{2}}{2\\sigma ^{2}}}\\right)=1.3112\\cdot 10^{-3}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0156", "text": "posterior numerator (male)\n        \n        =\n        \n          their product\n        \n        =\n        6.1984\n        \u22c5\n        \n          10\n          \n            \u2212\n            9\n          \n        \n      \n    \n    {\\displaystyle {\\text{posterior numerator (male)}}={\\text{their product}}=6.1984\\cdot 10^{-9}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0157", "text": "P\n        (\n        \n          female\n        \n        )\n        =\n        0.5\n      \n    \n    {\\displaystyle P({\\text{female}})=0.5}\n  \n\n  \n    \n      \n        p\n        (\n        \n          height\n        \n        \u2223\n        \n          female\n        \n        )\n        =\n        2.23\n        \u22c5\n        \n          10\n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle p({\\text{height}}\\mid {\\text{female}})=2.23\\cdot 10^{-1}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0158", "text": "p\n        (\n        \n          weight\n        \n        \u2223\n        \n          female\n        \n        )\n        =\n        1.6789\n        \u22c5\n        \n          10\n          \n            \u2212\n            2\n          \n        \n      \n    \n    {\\displaystyle p({\\text{weight}}\\mid {\\text{female}})=1.6789\\cdot 10^{-2}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0159", "text": "p\n        (\n        \n          foot size\n        \n        \u2223\n        \n          female\n        \n        )\n        =\n        2.8669\n        \u22c5\n        \n          10\n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle p({\\text{foot size}}\\mid {\\text{female}})=2.8669\\cdot 10^{-1}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0160", "text": "posterior numerator (female)\n        \n        =\n        \n          their product\n        \n        =\n        5.3778\n        \u22c5\n        \n          10\n          \n            \u2212\n            4\n          \n        \n      \n    \n    {\\displaystyle {\\text{posterior numerator (female)}}={\\text{their product}}=5.3778\\cdot 10^{-4}}\n  \n\nSince posterior numerator is greater in the female case, the prediction is that the sample is female.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0161", "text": "Document classification\nHere is a worked example of naive Bayesian classification to the document classification problem.\nConsider the problem of classifying documents by their content, for example into spam and non-spam e-mails. Imagine that documents are drawn from a number of classes of documents which can be modeled as sets of words where the (independent) probability that the i-th word of a given document occurs in a document from class C can be written as", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0162", "text": "p\n        (\n        \n          w\n          \n            i\n          \n        \n        \u2223\n        C\n        )\n        \n      \n    \n    {\\displaystyle p(w_{i}\\mid C)\\,}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0163", "text": "(For this treatment, things are further simplified by assuming that words are randomly distributed in the document - that is, words are not dependent on the length of the document, position within the document with relation to other words, or other document-context.)\nThen the probability that a given document D contains all of the words \n  \n    \n      \n        \n          w\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle w_{i}}\n  \n, given a class C, is", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0164", "text": "p\n        (\n        D\n        \u2223\n        C\n        )\n        =\n        \n          \u220f\n          \n            i\n          \n        \n        p\n        (\n        \n          w\n          \n            i\n          \n        \n        \u2223\n        C\n        )\n        \n      \n    \n    {\\displaystyle p(D\\mid C)=\\prod _{i}p(w_{i}\\mid C)\\,}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0165", "text": "The question that has to be answered is: \"what is the probability that a given document D belongs to a given class C?\" In other words, what is \n  \n    \n      \n        p\n        (\n        C\n        \u2223\n        D\n        )\n        \n      \n    \n    {\\displaystyle p(C\\mid D)\\,}\n  \n?\nNow by definition", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0166", "text": "p\n        (\n        D\n        \u2223\n        C\n        )\n        =\n        \n          \n            \n              p\n              (\n              D\n              \u2229\n              C\n              )\n            \n            \n              p\n              (\n              C\n              )\n            \n          \n        \n      \n    \n    {\\displaystyle p(D\\mid C)={p(D\\cap C) \\over p(C)}}\n  \n\nand", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0167", "text": "and\n\n  \n    \n      \n        p\n        (\n        C\n        \u2223\n        D\n        )\n        =\n        \n          \n            \n              p\n              (\n              D\n              \u2229\n              C\n              )\n            \n            \n              p\n              (\n              D\n              )\n            \n          \n        \n      \n    \n    {\\displaystyle p(C\\mid D)={p(D\\cap C) \\over p(D)}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0168", "text": "Bayes' theorem manipulates these into a statement of probability in terms of likelihood.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0169", "text": "p\n        (\n        C\n        \u2223\n        D\n        )\n        =\n        \n          \n            \n              p\n              (\n              C\n              )\n              \n              p\n              (\n              D\n              \u2223\n              C\n              )\n            \n            \n              p\n              (\n              D\n              )\n            \n          \n        \n      \n    \n    {\\displaystyle p(C\\mid D)={\\frac {p(C)\\,p(D\\mid C)}{p(D)}}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0170", "text": "Assume for the moment that there are only two mutually exclusive classes, S and \u00acS (e.g. spam and not spam), such that every element (email) is in either one or the other;", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0171", "text": "p\n        (\n        D\n        \u2223\n        S\n        )\n        =\n        \n          \u220f\n          \n            i\n          \n        \n        p\n        (\n        \n          w\n          \n            i\n          \n        \n        \u2223\n        S\n        )\n        \n      \n    \n    {\\displaystyle p(D\\mid S)=\\prod _{i}p(w_{i}\\mid S)\\,}\n  \n\nand", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0172", "text": "and\n\n  \n    \n      \n        p\n        (\n        D\n        \u2223\n        \u00ac\n        S\n        )\n        =\n        \n          \u220f\n          \n            i\n          \n        \n        p\n        (\n        \n          w\n          \n            i\n          \n        \n        \u2223\n        \u00ac\n        S\n        )\n        \n      \n    \n    {\\displaystyle p(D\\mid \\neg S)=\\prod _{i}p(w_{i}\\mid \\neg S)\\,}\n  \n\nUsing the Bayesian result above, one can write:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0173", "text": "p\n        (\n        S\n        \u2223\n        D\n        )\n        =\n        \n          \n            \n              p\n              (\n              S\n              )\n            \n            \n              p\n              (\n              D\n              )\n            \n          \n        \n        \n        \n          \u220f\n          \n            i\n          \n        \n        p\n        (\n        \n          w\n          \n            i\n          \n        \n        \u2223\n        S\n        )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0174", "text": "\u2223\n        S\n        )\n      \n    \n    {\\displaystyle p(S\\mid D)={p(S) \\over p(D)}\\,\\prod _{i}p(w_{i}\\mid S)}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0175", "text": "p\n        (\n        \u00ac\n        S\n        \u2223\n        D\n        )\n        =\n        \n          \n            \n              p\n              (\n              \u00ac\n              S\n              )\n            \n            \n              p\n              (\n              D\n              )\n            \n          \n        \n        \n        \n          \u220f\n          \n            i\n          \n        \n        p\n        (\n        \n          w\n          \n            i", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0176", "text": "i\n          \n        \n        \u2223\n        \u00ac\n        S\n        )\n      \n    \n    {\\displaystyle p(\\neg S\\mid D)={p(\\neg S) \\over p(D)}\\,\\prod _{i}p(w_{i}\\mid \\neg S)}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0177", "text": "Dividing one by the other gives:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0178", "text": "p\n              (\n              S\n              \u2223\n              D\n              )\n            \n            \n              p\n              (\n              \u00ac\n              S\n              \u2223\n              D\n              )\n            \n          \n        \n        =\n        \n          \n            \n              p\n              (\n              S\n              )\n              \n              \n                \u220f", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0179", "text": "\u220f\n                \n                  i\n                \n              \n              p\n              (\n              \n                w\n                \n                  i\n                \n              \n              \u2223\n              S\n              )\n            \n            \n              p\n              (\n              \u00ac\n              S\n              )\n              \n              \n                \u220f\n                \n                  i", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0180", "text": "i\n                \n              \n              p\n              (\n              \n                w\n                \n                  i\n                \n              \n              \u2223\n              \u00ac\n              S\n              )\n            \n          \n        \n      \n    \n    {\\displaystyle {p(S\\mid D) \\over p(\\neg S\\mid D)}={p(S)\\,\\prod _{i}p(w_{i}\\mid S) \\over p(\\neg S)\\,\\prod _{i}p(w_{i}\\mid \\neg S)}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0181", "text": "Which can be re-factored as:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0182", "text": "p\n              (\n              S\n              \u2223\n              D\n              )\n            \n            \n              p\n              (\n              \u00ac\n              S\n              \u2223\n              D\n              )\n            \n          \n        \n        =\n        \n          \n            \n              p\n              (\n              S\n              )\n            \n            \n              p\n              (\n              \u00ac", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0183", "text": "p\n              (\n              \u00ac\n              S\n              )\n            \n          \n        \n        \n        \n          \u220f\n          \n            i\n          \n        \n        \n          \n            \n              p\n              (\n              \n                w\n                \n                  i\n                \n              \n              \u2223\n              S\n              )\n            \n            \n              p\n              (\n              \n                w", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0184", "text": "(\n              \n                w\n                \n                  i\n                \n              \n              \u2223\n              \u00ac\n              S\n              )\n            \n          \n        \n      \n    \n    {\\displaystyle {p(S\\mid D) \\over p(\\neg S\\mid D)}={p(S) \\over p(\\neg S)}\\,\\prod _{i}{p(w_{i}\\mid S) \\over p(w_{i}\\mid \\neg S)}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0185", "text": "Thus, the probability ratio p(S | D) / p(\u00acS | D) can be expressed in terms of a series of likelihood ratios.\nThe actual probability p(S | D) can be easily computed from log (p(S | D) / p(\u00acS | D)) based on the observation that p(S | D) + p(\u00acS | D) = 1.\nTaking the logarithm of all these ratios, one obtains:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0186", "text": "ln\n        \u2061\n        \n          \n            \n              p\n              (\n              S\n              \u2223\n              D\n              )\n            \n            \n              p\n              (\n              \u00ac\n              S\n              \u2223\n              D\n              )\n            \n          \n        \n        =\n        ln\n        \u2061\n        \n          \n            \n              p\n              (\n              S\n              )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0187", "text": ")\n            \n            \n              p\n              (\n              \u00ac\n              S\n              )\n            \n          \n        \n        +\n        \n          \u2211\n          \n            i\n          \n        \n        ln\n        \u2061\n        \n          \n            \n              p\n              (\n              \n                w\n                \n                  i\n                \n              \n              \u2223\n              S\n              )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0188", "text": ")\n            \n            \n              p\n              (\n              \n                w\n                \n                  i\n                \n              \n              \u2223\n              \u00ac\n              S\n              )\n            \n          \n        \n      \n    \n    {\\displaystyle \\ln {p(S\\mid D) \\over p(\\neg S\\mid D)}=\\ln {p(S) \\over p(\\neg S)}+\\sum _{i}\\ln {p(w_{i}\\mid S) \\over p(w_{i}\\mid \\neg S)}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0189", "text": "(This technique of \"log-likelihood ratios\" is a common technique in statistics.\nIn the case of two mutually exclusive alternatives (such as this example), the conversion of a log-likelihood ratio to a probability takes the form of a sigmoid curve: see logit for details.)\nFinally, the document can be classified as follows.  It is spam if \n  \n    \n      \n        p\n        (\n        S\n        \u2223\n        D\n        )\n        >\n        p\n        (\n        \u00ac\n        S\n        \u2223\n        D\n        )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0190", "text": "\u00ac\n        S\n        \u2223\n        D\n        )\n      \n    \n    {\\displaystyle p(S\\mid D)>p(\\neg S\\mid D)}\n  \n (i. e., \n  \n    \n      \n        ln\n        \u2061\n        \n          \n            \n              p\n              (\n              S\n              \u2223\n              D\n              )\n            \n            \n              p\n              (\n              \u00ac\n              S\n              \u2223\n              D\n              )\n            \n          \n        \n        >\n        0", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0191", "text": ">\n        0\n      \n    \n    {\\displaystyle \\ln {p(S\\mid D) \\over p(\\neg S\\mid D)}>0}\n  \n), otherwise it is not spam.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0192", "text": "Spam filtering\nNaive Bayes classifiers are a popular statistical technique of e-mail filtering. They typically use bag-of-words features to identify email spam, an approach commonly used in text classification. Naive Bayes classifiers work by correlating the use of tokens (typically words, or sometimes other things), with spam and non-spam e-mails and then using Bayes' theorem to calculate a probability that an email is or is not spam.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0193", "text": "Naive Bayes spam filtering is a baseline technique for dealing with spam that can tailor itself to the email needs of individual users and give low false positive spam detection rates that are generally acceptable to users. Bayesian algorithms were used for email filtering as early as 1996.  Although naive Bayesian filters did not become popular until later, multiple programs were released in 1998 to address the growing problem of unwanted email", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0194", "text": ".  The first scholarly publication using the naive Bayes classifier for spam filtering was by Sahami et al. in 1998.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0195", "text": "Variants of the basic technique have been implemented in a number of research works and commercial software products. Many modern mail clients implement Bayesian spam filtering. Users can also install separate email filtering programs. Server-side email filters, such as DSPAM, Rspamd, SpamAssassin, SpamBayes, Bogofilter, and ASSP, make use of Bayesian spam filtering techniques, and the functionality is sometimes embedded within mail server software itself", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0196", "text": ". CRM114, often cited as a Bayesian filter, is not intended to use a Bayes filter in production, but includes the \u2033unigram\u2033 feature for reference.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0197", "text": "Dealing with rare words\nIn the case a word has never been met during the learning phase, both the numerator and the denominator are equal to zero, both in the general formula and in the spamicity formula. The software can decide to discard such words for which there is no information available.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0198", "text": "More generally, the words that were encountered only a few times during the learning phase cause a problem, because it would be an error to trust blindly the information they provide. A simple solution is to simply avoid taking such unreliable words into account as well.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0199", "text": "Applying again Bayes' theorem, and assuming the classification between spam and ham of the emails containing a given word (\"replica\") is a random variable with beta distribution, some programs decide to use a corrected probability:", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0200", "text": "Pr\n          \u2032\n        \n        (\n        S\n        \n          |\n        \n        W\n        )\n        =\n        \n          \n            \n              s\n              \u22c5\n              Pr\n              (\n              S\n              )\n              +\n              n\n              \u22c5\n              Pr\n              (\n              S\n              \n                |\n              \n              W\n              )\n            \n            \n              s", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0201", "text": "s\n              +\n              n\n            \n          \n        \n      \n    \n    {\\displaystyle \\Pr '(S|W)={\\frac {s\\cdot \\Pr(S)+n\\cdot \\Pr(S|W)}{s+n}}}", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0202", "text": "where:\n\n  \n    \n      \n        \n          Pr\n          \u2032\n        \n        (\n        S\n        \n          |\n        \n        W\n        )\n      \n    \n    {\\displaystyle \\Pr '(S|W)}\n  \n is the corrected probability for the message to be spam, knowing that it contains a given word ;\n\n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n is the strength we give to background information about incoming spam ;", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0203", "text": "Pr\n        (\n        S\n        )\n      \n    \n    {\\displaystyle \\Pr(S)}\n  \n is the probability of any incoming message to be spam ;\n\n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n is the number of occurrences of this word during the learning phase ;", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0204", "text": "Pr\n        (\n        S\n        \n          |\n        \n        W\n        )\n      \n    \n    {\\displaystyle \\Pr(S|W)}\n  \n is the spamicity of this word.\n(Demonstration:)\nThis corrected probability is used instead of the spamicity in the combining formula.\nThis formula can be extended to the case where n is equal to zero (and where the spamicity is not defined), and evaluates in this case to \n  \n    \n      \n        P\n        r\n        (\n        S\n        )", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0205", "text": "(\n        S\n        )\n      \n    \n    {\\displaystyle Pr(S)}\n  \n.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0206", "text": "Other heuristics", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0207", "text": "\"Neutral\" words like \"the\", \"a\", \"some\", or \"is\" (in English), or their equivalents in other languages, can be ignored. These are also known as Stop words. More generally, some bayesian filtering filters simply ignore all the words which have a spamicity next to 0.5, as they contribute little to a good decision. The words taken into consideration are those whose spamicity is next to 0.0 (distinctive signs of legitimate messages), or next to 1.0 (distinctive signs of spam)", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0208", "text": ". A method can be for example to keep only those ten words, in the examined message, which have the greatest absolute value |0.5 \u2212 pI|.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0209", "text": "Some software products take into account the fact that a given word appears several times in the examined message, others don't.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0210", "text": "Some software products use patterns (sequences of words) instead of isolated natural languages words. For example, with a \"context window\" of four words, they compute the spamicity of \"Viagra is good for\", instead of computing the spamicities of \"Viagra\", \"is\", \"good\", and \"for\". This method gives more sensitivity to context and eliminates the Bayesian noise better, at the expense of a bigger database.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0211", "text": "Disadvantages", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0212", "text": "Depending on the implementation, Bayesian spam filtering may be susceptible to Bayesian poisoning, a technique used by spammers in an attempt to degrade the effectiveness of spam filters that rely on Bayesian filtering. A spammer practicing Bayesian poisoning will send out emails with large amounts of legitimate text (gathered from legitimate news or literary sources)", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0213", "text": ". Spammer tactics include insertion of random innocuous words that are not normally associated with spam, thereby decreasing the email's spam score, making it more likely to slip past a Bayesian spam filter. However, with (for example) Paul Graham's scheme only the most significant probabilities are used, so that padding the text out with non-spam-related words does not affect the detection probability significantly.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0214", "text": "Words that normally appear in large quantities in spam may also be transformed by spammers. For example, \u00abViagra\u00bb would be replaced with \u00abViaagra\u00bb or \u00abV!agra\u00bb in the spam message. The recipient of the message can still read the changed words, but each of these words is met more rarely by the Bayesian filter, which hinders its learning process. As a general rule, this spamming technique does not work very well, because the derived words end up recognized by the filter just like the normal ones.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0215", "text": "Another technique used to try to defeat Bayesian spam filters is to replace text with pictures, either directly included or linked. The whole text of the message, or some part of it, is replaced with a picture where the same text is \"drawn\". The spam filter is usually unable to analyze this picture, which would contain the sensitive words like \u00abViagra\u00bb", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0216", "text": ". However, since many mail clients disable the display of linked pictures for security reasons, the spammer sending links to distant pictures might reach fewer targets. Also, a picture's size in bytes is bigger than the equivalent text's size, so the spammer needs more bandwidth to send messages directly including pictures. Some filters are more inclined to decide that a message is spam if it has mostly graphical contents", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0217", "text": ". A solution used by Google in its Gmail email system is to perform an OCR (Optical Character Recognition) on every mid to large size image, analyzing the text inside.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0218", "text": "See also\nAODE\nAnti-spam techniques\nBayes classifier\nBayesian network\nBayesian poisoning\nEmail filtering\nLinear classifier\nLogistic regression\nMarkovian discrimination\nMozilla Thunderbird mail client with native implementation of Bayes filters\nPerceptron\nRandom naive Bayes\nTake-the-best heuristic", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0219", "text": "References\nFurther reading\nDomingos, Pedro; Pazzani, Michael (1997). \"On the optimality of the simple Bayesian classifier under zero-one loss\". Machine Learning. 29 (2/3): 103\u2013137. doi:10.1023/A:1007413511361.\nWebb, G. I.; Boughton, J.; Wang, Z. (2005). \"Not So Naive Bayes: Aggregating One-Dependence Estimators\". Machine Learning. 58 (1): 5\u201324. doi:10.1007/s10994-005-4258-6.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0220", "text": "Mozina, M.; Demsar, J.; Kattan, M.; Zupan, B. (2004). Nomograms for Visualization of Naive Bayesian Classifier (PDF). Proc. PKDD-2004. pp. 337\u2013348. Archived from the original (PDF) on 2023-11-29. Retrieved 2014-04-01.\nMaron, M. E. (1961). \"Automatic Indexing: An Experimental Inquiry\". Journal of the ACM. 8 (3): 404\u2013417. doi:10.1145/321075.321084. hdl:2027/uva.x030748531. S2CID 6692916.\nMinsky, M. (1961). Steps toward Artificial Intelligence. Proc. IRE. Vol. 49. pp. 8\u201330.", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Naive_Bayes_classifier_0221", "text": "External links\nBook Chapter: Naive Bayes text classification, Introduction to Information Retrieval\nNaive Bayes for Text Classification with Unbalanced Classes", "title": "Naive Bayes classifier", "url": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"}, {"id": "rec_Logistic_regression_0000", "text": "In statistics, a logistic model (or logit model) is a statistical model that models the log-odds of an event as a linear combination of one or more independent variables. In regression analysis, logistic regression (or logit regression) estimates the parameters of a logistic model (the coefficients in the linear or non linear combinations)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0001", "text": ". In binary logistic regression there is a single binary dependent variable, coded by an indicator variable, where the two values are labeled \"0\" and \"1\", while the independent variables can each be a binary variable (two classes, coded by an indicator variable) or a continuous variable (any real value)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0002", "text": ". The corresponding probability of the value labeled \"1\" can vary between 0 (certainly the value \"0\") and 1 (certainly the value \"1\"), hence the labeling; the function that converts log-odds to probability is the logistic function, hence the name. The unit of measurement for the log-odds scale is called a logit, from logistic unit, hence the alternative names. See \u00a7 Background and \u00a7 Definition for formal mathematics, and \u00a7 Example for a worked example.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0003", "text": "Binary variables are widely used in statistics to model the probability of a certain class or event taking place, such as the probability of a team winning, of a patient being healthy, etc. (see \u00a7 Applications), and the logistic model has been the most commonly used model for binary regression since about 1970. Binary variables can be generalized to categorical variables when there are more than two possible values (e.g", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0004", "text": ". whether an image is of a cat, dog, lion, etc.), and the binary logistic regression generalized to multinomial logistic regression. If the multiple categories are ordered, one can use the ordinal logistic regression (for example the proportional odds ordinal logistic model). See \u00a7 Extensions for further extensions", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0005", "text": ". See \u00a7 Extensions for further extensions. The logistic regression model itself simply models probability of output in terms of input and does not perform statistical classification (it is not a classifier), though it can be used to make a classifier, for instance by choosing a cutoff value and classifying inputs with probability greater than the cutoff as one class, below the cutoff as the other; this is a common way to make a binary classifier.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0006", "text": "Analogous linear models for binary variables with a different sigmoid function instead of the logistic function (to convert the linear combination to a probability) can also be used, most notably the probit model; see \u00a7 Alternatives", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0007", "text": ". The defining characteristic of the logistic model is that increasing one of the independent variables multiplicatively scales the odds of the given outcome at a constant rate, with each independent variable having its own parameter; for a binary dependent variable this generalizes the odds ratio. More abstractly, the logistic function is the natural parameter for the Bernoulli distribution, and in this sense is the \"simplest\" way to convert a real number to a probability.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0008", "text": "The parameters of a logistic regression are most commonly estimated by maximum-likelihood estimation (MLE). This does not have a closed-form expression, unlike linear least squares; see \u00a7 Model fitting. Logistic regression by MLE plays a similarly basic role for binary or categorical responses as linear regression by ordinary least squares (OLS) plays for scalar responses: it is a simple, well-analyzed baseline model; see \u00a7 Comparison with linear regression for discussion", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0009", "text": ". The logistic regression as a general statistical model was originally developed and popularized primarily by Joseph Berkson, beginning in Berkson (1944), where he coined \"logit\"; see \u00a7 History.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0010", "text": "Applications\nGeneral", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0011", "text": "Logistic regression is used in various fields, including machine learning, most medical fields, and social sciences. For example, the Trauma and Injury Severity Score (TRISS), which is widely used to predict mortality in injured patients, was originally developed by Boyd et al. using logistic regression.  Many other medical scales used to assess severity of a patient have been developed using logistic regression", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0012", "text": ". Logistic regression may be used to predict the risk of developing a given disease (e.g. diabetes; coronary heart disease), based on observed characteristics of the patient (age, sex, body mass index, results of various blood tests, etc.).  Another example might be to predict whether a Nepalese voter will vote Nepali Congress or Communist Party of Nepal or for any other party, based on age, income, sex, race, state of residence, votes in previous elections, etc", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0013", "text": ". The technique can also be used in engineering, especially for predicting the probability of failure of a given process, system or product. It is also used in marketing applications such as prediction of a customer's propensity to purchase a product or halt a subscription, etc. In economics, it can be used to predict the likelihood of a person ending up in the labor force, and a business application would be to predict the likelihood of a homeowner defaulting on a mortgage", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0014", "text": ". Conditional random fields, an extension of logistic regression to sequential data, are used in natural language processing. Disaster planners and engineers  rely on these models to predict decisions taken by householders or building occupants in small-scale and large-scales evacuations, such as building fires, wildfires, hurricanes among others. These models help in the development of reliable disaster managing plans and safer design for the built environment.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0015", "text": "Supervised machine learning", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0016", "text": "Logistic regression is a supervised machine learning algorithm widely used for binary classification tasks, such as identifying whether an email is spam or not and diagnosing diseases by assessing the presence or absence of specific conditions based on patient test results. This approach utilizes the logistic (or sigmoid) function to transform a linear combination of input features into a probability value ranging between 0 and 1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0017", "text": ". This probability indicates the likelihood that a given input corresponds to one of two predefined categories. The essential mechanism of logistic regression is grounded in the logistic function's ability to model the probability of binary outcomes accurately. With its distinctive S-shaped curve, the logistic function effectively maps any real-valued number to a value within the 0 to 1 interval", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0018", "text": ". This feature renders it particularly suitable for binary classification tasks, such as sorting emails into \"spam\" or \"not spam\". By calculating the probability that the dependent variable will be categorized into a specific group, logistic regression provides a probabilistic framework that supports informed decision-making.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0019", "text": "Example\nProblem\nAs a simple example, we can use a logistic regression with one explanatory variable and two categories to answer the following question:\n\nA group of 20 students spends between 0 and 6 hours studying for an exam. How does the number of hours spent studying affect the probability of the student passing the exam?", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0020", "text": "The reason for using logistic regression for this problem is that the values of the dependent variable, pass and fail, while represented by \"1\" and \"0\", are not cardinal numbers. If the problem was changed so that pass/fail was replaced with the grade 0\u2013100 (cardinal numbers), then simple regression analysis could be used.\nThe table shows the number of hours each student spent studying, and whether they passed (1) or failed (0).", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0021", "text": "We wish to fit a logistic function to the data consisting of the hours studied (xk) and the outcome of the test (yk =1 for pass, 0 for fail). The data points are indexed by the subscript k which runs from \n  \n    \n      \n        k\n        =\n        1\n      \n    \n    {\\displaystyle k=1}\n  \n to \n  \n    \n      \n        k\n        =\n        K\n        =\n        20\n      \n    \n    {\\displaystyle k=K=20}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0022", "text": "{\\displaystyle k=K=20}\n  \n. The x variable is called the \"explanatory variable\", and the y variable is called the \"categorical variable\" consisting of two categories: \"pass\" or \"fail\" corresponding to the categorical values 1 and 0 respectively.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0023", "text": "Model\nThe logistic function is of the form:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0024", "text": "p\n        (\n        x\n        )\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  (\n                  x\n                  \u2212\n                  \u03bc\n                  )\n                  \n                    /\n                  \n                  s", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0025", "text": "{\\displaystyle p(x)={\\frac {1}{1+e^{-(x-\\mu )/s}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0026", "text": "where \u03bc is a location parameter (the midpoint of the curve, where \n  \n    \n      \n        p\n        (\n        \u03bc\n        )\n        =\n        1\n        \n          /\n        \n        2\n      \n    \n    {\\displaystyle p(\\mu )=1/2}\n  \n) and s is a scale parameter. This expression may be rewritten as:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0027", "text": "p\n        (\n        x\n        )\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  (\n                  \n                    \u03b2\n                    \n                      0\n                    \n                  \n                  +\n                  \n                    \u03b2\n                    \n                      1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0028", "text": "1\n                    \n                  \n                  x\n                  )\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle p(x)={\\frac {1}{1+e^{-(\\beta _{0}+\\beta _{1}x)}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0029", "text": "where \n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n        =\n        \u2212\n        \u03bc\n        \n          /\n        \n        s\n      \n    \n    {\\displaystyle \\beta _{0}=-\\mu /s}\n  \n and is known as the intercept (it is the vertical intercept or y-intercept of the line \n  \n    \n      \n        y\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        x", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0030", "text": "x\n      \n    \n    {\\displaystyle y=\\beta _{0}+\\beta _{1}x}\n  \n), and \n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n        =\n        1\n        \n          /\n        \n        s\n      \n    \n    {\\displaystyle \\beta _{1}=1/s}\n  \n (inverse scale parameter or rate parameter): these are the y-intercept and slope of the log-odds as a function of x. Conversely, \n  \n    \n      \n        \u03bc\n        =\n        \u2212\n        \n          \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0031", "text": "\u2212\n        \n          \u03b2\n          \n            0\n          \n        \n        \n          /\n        \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\mu =-\\beta _{0}/\\beta _{1}}\n  \n and \n  \n    \n      \n        s\n        =\n        1\n        \n          /\n        \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle s=1/\\beta _{1}}\n  \n.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0032", "text": "{\\displaystyle s=1/\\beta _{1}}\n  \n.\nNote that this model is actually an oversimplification, as it implies that every student will pass if they study indefinitely (limit = 1).", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0033", "text": "Fit\nThe usual measure of goodness of fit for a logistic regression uses logistic loss (or log loss), the negative log-likelihood. For a given xk and yk, write \n  \n    \n      \n        \n          p\n          \n            k\n          \n        \n        =\n        p\n        (\n        \n          x\n          \n            k\n          \n        \n        )\n      \n    \n    {\\displaystyle p_{k}=p(x_{k})}\n  \n. The \u2060\n  \n    \n      \n        \n          p\n          \n            k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0034", "text": "k\n          \n        \n      \n    \n    {\\displaystyle p_{k}}\n  \n\u2060 are the probabilities that the corresponding \u2060\n  \n    \n      \n        \n          y\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle y_{k}}\n  \n\u2060 will equal one, and \u2060\n  \n    \n      \n        1\n        \u2212\n        \n          p\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle 1-p_{k}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0035", "text": "{\\displaystyle 1-p_{k}}\n  \n\u2060 are the probabilities that they will be zero (see Bernoulli distribution). We wish to find the values of \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n\u2060 and \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\beta _{1}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0036", "text": "{\\displaystyle \\beta _{1}}\n  \n\u2060 which give the \"best fit\" to the data. For comparison of a \"best fit\" to the data, see the case of linear regression. There, the sum of the squared deviations of the fit from the data points (yk), the squared error loss, is taken as a measure of the goodness of fit, and the best fit is obtained when this loss is minimized.\nThe log loss for the k-th point \u2060\n  \n    \n      \n        \n          \u2113\n          \n            k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0037", "text": "k\n          \n        \n      \n    \n    {\\displaystyle \\ell _{k}}\n  \n\u2060 is:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0038", "text": "\u2113\n          \n            k\n          \n        \n        =\n        \n          \n            {\n            \n              \n                \n                  \u2212\n                  ln\n                  \u2061\n                  \n                    p\n                    \n                      k\n                    \n                  \n                \n                \n                  \n                     if \n                  \n                  \n                    y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0039", "text": "y\n                    \n                      k\n                    \n                  \n                  =\n                  1\n                  ,\n                \n              \n              \n                \n                  \u2212\n                  ln\n                  \u2061\n                  (\n                  1\n                  \u2212\n                  \n                    p\n                    \n                      k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0040", "text": ")\n                \n                \n                  \n                     if \n                  \n                  \n                    y\n                    \n                      k\n                    \n                  \n                  =\n                  0.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0041", "text": "{\\displaystyle \\ell _{k}={\\begin{cases}-\\ln p_{k}&{\\text{ if }}y_{k}=1,\\\\-\\ln(1-p_{k})&{\\text{ if }}y_{k}=0.\\end{cases}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0042", "text": "The log loss can be interpreted as the \"surprisal\" of the actual outcome \u2060\n  \n    \n      \n        \n          y\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle y_{k}}\n  \n\u2060 relative to the prediction \u2060\n  \n    \n      \n        \n          p\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle p_{k}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0043", "text": "{\\displaystyle p_{k}}\n  \n\u2060, and is a measure of information content. Log loss is always greater than or equal to 0, equals 0 only in case of a perfect prediction (i.e., when \n  \n    \n      \n        \n          p\n          \n            k\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle p_{k}=1}\n  \n and \n  \n    \n      \n        \n          y\n          \n            k\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle y_{k}=1}\n  \n, or", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0044", "text": "{\\displaystyle y_{k}=1}\n  \n, or \n  \n    \n      \n        \n          p\n          \n            k\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle p_{k}=0}\n  \n and \n  \n    \n      \n        \n          y\n          \n            k\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle y_{k}=0}\n  \n), and approaches infinity as the prediction gets worse (i.e., when \n  \n    \n      \n        \n          y\n          \n            k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0045", "text": "k\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle y_{k}=1}\n  \n and \n  \n    \n      \n        \n          p\n          \n            k\n          \n        \n        \u2192\n        0\n      \n    \n    {\\displaystyle p_{k}\\to 0}\n  \n or \n  \n    \n      \n        \n          y\n          \n            k\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle y_{k}=0}\n  \n and \n  \n    \n      \n        \n          p\n          \n            k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0046", "text": "k\n          \n        \n        \u2192\n        1\n      \n    \n    {\\displaystyle p_{k}\\to 1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0047", "text": "{\\displaystyle p_{k}\\to 1}\n  \n), meaning the actual outcome is \"more surprising\". Since the value of the logistic function is always strictly between zero and one, the log loss is always greater than zero and less than infinity. Unlike in a linear regression, where the model can have zero loss at a point by passing through a data point (and zero loss overall if all points are on a line), in a logistic regression it is not possible to have zero loss at any points, since \u2060", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0048", "text": "y\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle y_{k}}\n  \n\u2060 is either 0 or 1, but \u2060\n  \n    \n      \n        0\n        <\n        \n          p\n          \n            k\n          \n        \n        <\n        1\n      \n    \n    {\\displaystyle 0<p_{k}<1}\n  \n\u2060.\nThese can be combined into a single expression:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0049", "text": "\u2113\n          \n            k\n          \n        \n        =\n        \u2212\n        \n          y\n          \n            k\n          \n        \n        ln\n        \u2061\n        \n          p\n          \n            k\n          \n        \n        \u2212\n        (\n        1\n        \u2212\n        \n          y\n          \n            k\n          \n        \n        )\n        ln\n        \u2061\n        (\n        1\n        \u2212\n        \n          p\n          \n            k\n          \n        \n        )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0050", "text": "k\n          \n        \n        )\n        .\n      \n    \n    {\\displaystyle \\ell _{k}=-y_{k}\\ln p_{k}-(1-y_{k})\\ln(1-p_{k}).}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0051", "text": "This expression is more formally known as the cross-entropy of the predicted distribution \n  \n    \n      \n        \n          \n            (\n          \n        \n        \n          p\n          \n            k\n          \n        \n        ,\n        (\n        1\n        \u2212\n        \n          p\n          \n            k\n          \n        \n        )\n        \n          \n            )\n          \n        \n      \n    \n    {\\displaystyle {\\big (}p_{k},(1-p_{k}){\\big )}}\n  \n from the actual distribution", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0052", "text": "from the actual distribution \n  \n    \n      \n        \n          \n            (\n          \n        \n        \n          y\n          \n            k\n          \n        \n        ,\n        (\n        1\n        \u2212\n        \n          y\n          \n            k\n          \n        \n        )\n        \n          \n            )\n          \n        \n      \n    \n    {\\displaystyle {\\big (}y_{k},(1-y_{k}){\\big )}}\n  \n, as probability distributions on the two-element space of (pass, fail).", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0053", "text": "The sum of these, the total loss, is the overall negative log-likelihood \u2060\n  \n    \n      \n        \u2212\n        \u2113\n      \n    \n    {\\displaystyle -\\ell }\n  \n\u2060, and the best fit is obtained for those choices of \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n\u2060 and \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\beta _{1}}\n  \n\u2060 for which \u2060", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0054", "text": "\u2060 for which \u2060\n  \n    \n      \n        \u2212\n        \u2113\n      \n    \n    {\\displaystyle -\\ell }\n  \n\u2060 is minimized.\nAlternatively, instead of minimizing the loss, one can maximize its inverse, the (positive) log-likelihood:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0055", "text": "\u2113\n        =\n        \n          \u2211\n          \n            k\n            :\n            \n              y\n              \n                k\n              \n            \n            =\n            1\n          \n        \n        ln\n        \u2061\n        (\n        \n          p\n          \n            k\n          \n        \n        )\n        +\n        \n          \u2211\n          \n            k\n            :\n            \n              y\n              \n                k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0056", "text": "k\n              \n            \n            =\n            0\n          \n        \n        ln\n        \u2061\n        (\n        1\n        \u2212\n        \n          p\n          \n            k\n          \n        \n        )\n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        \n          (\n          \n            \n            \n              y\n              \n                k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0057", "text": "k\n              \n            \n            ln\n            \u2061\n            (\n            \n              p\n              \n                k\n              \n            \n            )\n            +\n            (\n            1\n            \u2212\n            \n              y\n              \n                k\n              \n            \n            )\n            ln\n            \u2061\n            (\n            1\n            \u2212\n            \n              p", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0058", "text": "p\n              \n                k\n              \n            \n            )\n          \n          )\n        \n      \n    \n    {\\displaystyle \\ell =\\sum _{k:y_{k}=1}\\ln(p_{k})+\\sum _{k:y_{k}=0}\\ln(1-p_{k})=\\sum _{k=1}^{K}\\left(\\,y_{k}\\ln(p_{k})+(1-y_{k})\\ln(1-p_{k})\\right)}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0059", "text": "or equivalently maximize the likelihood function itself, which is the probability that the given data set is produced by a particular logistic function:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0060", "text": "L\n        =\n        \n          \u220f\n          \n            k\n            :\n            \n              y\n              \n                k\n              \n            \n            =\n            1\n          \n        \n        \n          p\n          \n            k\n          \n        \n        \n        \n          \u220f\n          \n            k\n            :\n            \n              y\n              \n                k\n              \n            \n            =\n            0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0061", "text": "=\n            0\n          \n        \n        (\n        1\n        \u2212\n        \n          p\n          \n            k\n          \n        \n        )\n      \n    \n    {\\displaystyle L=\\prod _{k:y_{k}=1}p_{k}\\,\\prod _{k:y_{k}=0}(1-p_{k})}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0062", "text": "This method is known as maximum likelihood estimation.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0063", "text": "Parameter estimation\nSince \u2113 is nonlinear in \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n\u2060 and \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\beta _{1}}\n  \n\u2060, determining their optimum values will require numerical methods. One method of maximizing  \u2113 is to require the derivatives of \u2113 with respect to \u2060\n  \n    \n      \n        \n          \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0064", "text": "\u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n\u2060 and \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\beta _{1}}\n  \n\u2060 to be zero:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0065", "text": "0\n        =\n        \n          \n            \n              \u2202\n              \u2113\n            \n            \n              \u2202\n              \n                \u03b2\n                \n                  0\n                \n              \n            \n          \n        \n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        (\n        \n          y\n          \n            k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0066", "text": "k\n          \n        \n        \u2212\n        \n          p\n          \n            k\n          \n        \n        )\n      \n    \n    {\\displaystyle 0={\\frac {\\partial \\ell }{\\partial \\beta _{0}}}=\\sum _{k=1}^{K}(y_{k}-p_{k})}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0067", "text": "0\n        =\n        \n          \n            \n              \u2202\n              \u2113\n            \n            \n              \u2202\n              \n                \u03b2\n                \n                  1\n                \n              \n            \n          \n        \n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        (\n        \n          y\n          \n            k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0068", "text": "k\n          \n        \n        \u2212\n        \n          p\n          \n            k\n          \n        \n        )\n        \n          x\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle 0={\\frac {\\partial \\ell }{\\partial \\beta _{1}}}=\\sum _{k=1}^{K}(y_{k}-p_{k})x_{k}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0069", "text": "and the maximization procedure can be accomplished by solving the above two equations for \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n\u2060 and \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\beta _{1}}\n  \n\u2060, which, again, will generally require the use of numerical methods.\nThe values of \u2060\n  \n    \n      \n        \n          \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0070", "text": "\u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n\u2060 and \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\beta _{1}}\n  \n\u2060 which maximize \u2113 and L using the above data are found to be:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0071", "text": "\u03b2\n          \n            0\n          \n        \n        \u2248\n        \u2212\n        4.1\n      \n    \n    {\\displaystyle \\beta _{0}\\approx -4.1}\n  \n\n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n        \u2248\n        1.5\n      \n    \n    {\\displaystyle \\beta _{1}\\approx 1.5}\n  \n\nwhich yields a value for \u03bc and s of:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0072", "text": "which yields a value for \u03bc and s of:\n\n  \n    \n      \n        \u03bc\n        =\n        \u2212\n        \n          \u03b2\n          \n            0\n          \n        \n        \n          /\n        \n        \n          \u03b2\n          \n            1\n          \n        \n        \u2248\n        2.7\n      \n    \n    {\\displaystyle \\mu =-\\beta _{0}/\\beta _{1}\\approx 2.7}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0073", "text": "s\n        =\n        1\n        \n          /\n        \n        \n          \u03b2\n          \n            1\n          \n        \n        \u2248\n        0.67\n      \n    \n    {\\displaystyle s=1/\\beta _{1}\\approx 0.67}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0074", "text": "Predictions\nThe \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n\u2060 and \u2060\n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\beta _{1}}\n  \n\u2060 coefficients may be entered into the logistic regression equation to estimate the probability of passing the exam.\nFor example, for a student who studies 2 hours, entering the value \n  \n    \n      \n        x", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0075", "text": "x\n        =\n        2\n      \n    \n    {\\displaystyle x=2}\n  \n into the equation gives the estimated probability of passing the exam of 0.25:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0076", "text": "t\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        2\n        \n          \u03b2\n          \n            1\n          \n        \n        \u2248\n        \u2212\n        4.1\n        +\n        2\n        \u22c5\n        1.5\n        =\n        \u2212\n        1.1\n      \n    \n    {\\displaystyle t=\\beta _{0}+2\\beta _{1}\\approx -4.1+2\\cdot 1.5=-1.1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0077", "text": "p\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  t\n                \n              \n            \n          \n        \n        \u2248\n        0.25\n        =\n        \n          Probability of passing exam\n        \n      \n    \n    {\\displaystyle p={\\frac {1}{1+e^{-t}}}\\approx 0.25={\\text{Probability of passing exam}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0078", "text": "Similarly, for a student who studies 4 hours, the estimated probability of passing the exam is 0.87:\n\n  \n    \n      \n        t\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        4\n        \n          \u03b2\n          \n            1\n          \n        \n        \u2248\n        \u2212\n        4.1\n        +\n        4\n        \u22c5\n        1.5\n        =\n        1.9\n      \n    \n    {\\displaystyle t=\\beta _{0}+4\\beta _{1}\\approx -4.1+4\\cdot 1.5=1.9}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0079", "text": "p\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  t\n                \n              \n            \n          \n        \n        \u2248\n        0.87\n        =\n        \n          Probability of passing exam\n        \n      \n    \n    {\\displaystyle p={\\frac {1}{1+e^{-t}}}\\approx 0.87={\\text{Probability of passing exam}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0080", "text": "This table shows the estimated probability of passing the exam for several values of hours studying.\n\nModel evaluation\nThe logistic regression analysis gives the following output.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0081", "text": "By the Wald test, the output indicates that hours studying is significantly associated with the probability of passing the exam (\n  \n    \n      \n        p\n        =\n        0.017\n      \n    \n    {\\displaystyle p=0.017}\n  \n). Rather than the Wald method, the recommended method to calculate the p-value for logistic regression is the likelihood-ratio test (LRT), which for these data give \n  \n    \n      \n        p\n        \u2248\n        0.00064\n      \n    \n    {\\displaystyle p\\approx 0.00064}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0082", "text": "{\\displaystyle p\\approx 0.00064}\n  \n (see \u00a7 Deviance and likelihood ratio tests below).", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0083", "text": "Generalizations\nThis simple model is an example of binary logistic regression, and has one explanatory variable and a binary categorical variable which can assume one of two categorical values. Multinomial logistic regression is the generalization of binary logistic regression to include any number of explanatory variables and any number of categories.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0084", "text": "Background\nDefinition of the logistic function\nAn explanation of logistic regression can begin with an explanation of the standard logistic function. The logistic function is a sigmoid function, which takes any real input \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n, and outputs a value between zero and one. For the logit, this is interpreted as taking input log-odds and having output probability. The standard logistic function \n  \n    \n      \n        \u03c3\n        :", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0085", "text": "\u03c3\n        :\n        \n          R\n        \n        \u2192\n        (\n        0\n        ,\n        1\n        )\n      \n    \n    {\\displaystyle \\sigma :\\mathbb {R} \\rightarrow (0,1)}\n  \n is defined as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0086", "text": "\u03c3\n        (\n        t\n        )\n        =\n        \n          \n            \n              e\n              \n                t\n              \n            \n            \n              \n                e\n                \n                  t\n                \n              \n              +\n              1\n            \n          \n        \n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0087", "text": "e\n                \n                  \u2212\n                  t\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\sigma (t)={\\frac {e^{t}}{e^{t}+1}}={\\frac {1}{1+e^{-t}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0088", "text": "A graph of the logistic function on the t-interval (\u22126,6) is shown in Figure 1.\nLet us assume that \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n is a linear function of a single explanatory variable \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n (the case where \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n is a linear combination of multiple explanatory variables is treated similarly). We can then express \n  \n    \n      \n        t", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0089", "text": "t\n      \n    \n    {\\displaystyle t}\n  \n as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0090", "text": "t\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        x\n      \n    \n    {\\displaystyle t=\\beta _{0}+\\beta _{1}x}\n  \n\nAnd the general logistic function \n  \n    \n      \n        p\n        :\n        \n          R\n        \n        \u2192\n        (\n        0\n        ,\n        1\n        )\n      \n    \n    {\\displaystyle p:\\mathbb {R} \\rightarrow (0,1)}\n  \n can now be written as:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0091", "text": "p\n        (\n        x\n        )\n        =\n        \u03c3\n        (\n        t\n        )\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  (\n                  \n                    \u03b2\n                    \n                      0\n                    \n                  \n                  +\n                  \n                    \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0092", "text": "\u03b2\n                    \n                      1\n                    \n                  \n                  x\n                  )\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle p(x)=\\sigma (t)={\\frac {1}{1+e^{-(\\beta _{0}+\\beta _{1}x)}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0093", "text": "In the logistic model, \n  \n    \n      \n        p\n        (\n        x\n        )\n      \n    \n    {\\displaystyle p(x)}\n  \n is interpreted as the probability of the dependent variable \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n equaling a success/case rather than a failure/non-case. It is clear that the response variables \n  \n    \n      \n        \n          Y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle Y_{i}}\n  \n are not identically distributed:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0094", "text": "are not identically distributed: \n  \n    \n      \n        P\n        (\n        \n          Y\n          \n            i\n          \n        \n        =\n        1\n        \u2223\n        X\n        )\n      \n    \n    {\\displaystyle P(Y_{i}=1\\mid X)}\n  \n differs from one data point \n  \n    \n      \n        \n          X\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle X_{i}}\n  \n to another, though they are independent given design matrix \n  \n    \n      \n        X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0095", "text": "X\n      \n    \n    {\\displaystyle X}\n  \n and shared parameters \n  \n    \n      \n        \u03b2\n      \n    \n    {\\displaystyle \\beta }\n  \n.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0096", "text": "Definition of the inverse of the logistic function\nWe can now define the logit (log odds) function as the inverse \n  \n    \n      \n        g\n        =\n        \n          \u03c3\n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle g=\\sigma ^{-1}}\n  \n of the standard logistic function. It is easy to see that it satisfies:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0097", "text": "g\n        (\n        p\n        (\n        x\n        )\n        )\n        =\n        \n          \u03c3\n          \n            \u2212\n            1\n          \n        \n        (\n        p\n        (\n        x\n        )\n        )\n        =\n        logit\n        \u2061\n        p\n        (\n        x\n        )\n        =\n        ln\n        \u2061\n        \n          (\n          \n            \n              \n                p\n                (\n                x\n                )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0098", "text": ")\n              \n              \n                1\n                \u2212\n                p\n                (\n                x\n                )\n              \n            \n          \n          )\n        \n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        x\n        ,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0099", "text": "x\n        ,\n      \n    \n    {\\displaystyle g(p(x))=\\sigma ^{-1}(p(x))=\\operatorname {logit} p(x)=\\ln \\left({\\frac {p(x)}{1-p(x)}}\\right)=\\beta _{0}+\\beta _{1}x,}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0100", "text": "and equivalently, after exponentiating both sides we have the odds:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0101", "text": "p\n              (\n              x\n              )\n            \n            \n              1\n              \u2212\n              p\n              (\n              x\n              )\n            \n          \n        \n        =\n        \n          e\n          \n            \n              \u03b2\n              \n                0\n              \n            \n            +\n            \n              \u03b2\n              \n                1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0102", "text": "1\n              \n            \n            x\n          \n        \n        .\n      \n    \n    {\\displaystyle {\\frac {p(x)}{1-p(x)}}=e^{\\beta _{0}+\\beta _{1}x}.}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0103", "text": "Interpretation of these terms\nIn the above equations, the terms are as follows:\n\n  \n    \n      \n        g\n      \n    \n    {\\displaystyle g}\n  \n is the logit function. The equation for \n  \n    \n      \n        g\n        (\n        p\n        (\n        x\n        )\n        )\n      \n    \n    {\\displaystyle g(p(x))}\n  \n illustrates that the logit (i.e., log-odds or natural logarithm of the odds) is equivalent to the linear regression expression.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0104", "text": "ln\n      \n    \n    {\\displaystyle \\ln }\n  \n denotes the natural logarithm.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0105", "text": "p\n        (\n        x\n        )\n      \n    \n    {\\displaystyle p(x)}\n  \n is the probability that the dependent variable equals a case, given some linear combination of the predictors. The formula for \n  \n    \n      \n        p\n        (\n        x\n        )\n      \n    \n    {\\displaystyle p(x)}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0106", "text": ")\n      \n    \n    {\\displaystyle p(x)}\n  \n illustrates that the probability of the dependent variable equaling a case is equal to the value of the logistic function of the linear regression expression. This is important in that it shows that the value of the linear regression expression can vary from negative to positive infinity and yet, after transformation, the resulting expression for the probability \n  \n    \n      \n        p\n        (\n        x\n        )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0107", "text": "(\n        x\n        )\n      \n    \n    {\\displaystyle p(x)}\n  \n ranges between 0 and 1.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0108", "text": "\u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n is the intercept from the linear regression equation (the value of the criterion when the predictor is equal to zero).", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0109", "text": "\u03b2\n          \n            1\n          \n        \n        x\n      \n    \n    {\\displaystyle \\beta _{1}x}\n  \n is the regression coefficient multiplied by some value of the predictor.\nbase \n  \n    \n      \n        e\n      \n    \n    {\\displaystyle e}\n  \n denotes the exponential function.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0110", "text": "Definition of the odds\nThe odds of the dependent variable equaling a case (given some linear combination \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0111", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n of the predictors) is equivalent to the exponential function of the linear regression expression. This illustrates how the logit serves as a link function between the probability and the linear regression expression. Given that the logit ranges between negative and positive infinity, it provides an adequate criterion upon which to conduct linear regression and the logit is easily converted back into the odds.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0112", "text": "So we define odds of the dependent variable equaling a case (given some linear combination \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n of the predictors) as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0113", "text": "odds\n        \n        =\n        \n          e\n          \n            \n              \u03b2\n              \n                0\n              \n            \n            +\n            \n              \u03b2\n              \n                1\n              \n            \n            x\n          \n        \n        .\n      \n    \n    {\\displaystyle {\\text{odds}}=e^{\\beta _{0}+\\beta _{1}x}.}\n\nThe odds ratio\nFor a continuous independent variable the odds ratio can be defined as:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0114", "text": "O\n          R\n        \n        =\n        \n          \n            \n              odds\n              \u2061\n              (\n              x\n              +\n              1\n              )\n            \n            \n              odds\n              \u2061\n              (\n              x\n              )\n            \n          \n        \n        =\n        \n          \n            \n              (\n              \n                \n                  \n                    p", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0115", "text": "p\n                    (\n                    x\n                    +\n                    1\n                    )\n                  \n                  \n                    1\n                    \u2212\n                    p\n                    (\n                    x\n                    +\n                    1\n                    )\n                  \n                \n              \n              )\n            \n            \n              (", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0116", "text": "(\n              \n                \n                  \n                    p\n                    (\n                    x\n                    )\n                  \n                  \n                    1\n                    \u2212\n                    p\n                    (\n                    x\n                    )\n                  \n                \n              \n              )\n            \n          \n        \n        =\n        \n          \n            \n              e", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0117", "text": "e\n              \n                \n                  \u03b2\n                  \n                    0\n                  \n                \n                +\n                \n                  \u03b2\n                  \n                    1\n                  \n                \n                (\n                x\n                +\n                1\n                )\n              \n            \n            \n              e", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0118", "text": "e\n              \n                \n                  \u03b2\n                  \n                    0\n                  \n                \n                +\n                \n                  \u03b2\n                  \n                    1\n                  \n                \n                x\n              \n            \n          \n        \n        =\n        \n          e\n          \n            \n              \u03b2\n              \n                1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0119", "text": "{\\displaystyle \\mathrm {OR} ={\\frac {\\operatorname {odds} (x+1)}{\\operatorname {odds} (x)}}={\\frac {\\left({\\frac {p(x+1)}{1-p(x+1)}}\\right)}{\\left({\\frac {p(x)}{1-p(x)}}\\right)}}={\\frac {e^{\\beta _{0}+\\beta _{1}(x+1)}}{e^{\\beta _{0}+\\beta _{1}x}}}=e^{\\beta _{1}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0120", "text": "This exponential relationship provides an interpretation for \n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\beta _{1}}\n  \n: The odds multiply by \n  \n    \n      \n        \n          e\n          \n            \n              \u03b2\n              \n                1\n              \n            \n          \n        \n      \n    \n    {\\displaystyle e^{\\beta _{1}}}\n  \n for every 1-unit increase in x.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0121", "text": "for every 1-unit increase in x.\nFor a binary independent variable the odds ratio is defined as \n  \n    \n      \n        \n          \n            \n              a\n              d\n            \n            \n              b\n              c\n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {ad}{bc}}}\n  \n where a, b, c and d are cells in a 2\u00d72 contingency table.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0122", "text": "Multiple explanatory variables\nIf there are multiple explanatory variables, the above expression \n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        x\n      \n    \n    {\\displaystyle \\beta _{0}+\\beta _{1}x}\n  \n can be revised to \n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0123", "text": "\u03b2\n          \n            1\n          \n        \n        \n          x\n          \n            1\n          \n        \n        +\n        \n          \u03b2\n          \n            2\n          \n        \n        \n          x\n          \n            2\n          \n        \n        +\n        \u22ef\n        +\n        \n          \u03b2\n          \n            m\n          \n        \n        \n          x\n          \n            m\n          \n        \n        =\n        \n          \u03b2\n          \n            0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0124", "text": "\u03b2\n          \n            0\n          \n        \n        +\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            m\n          \n        \n        \n          \u03b2\n          \n            i\n          \n        \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}+\\beta _{1}x_{1}+\\beta _{2}x_{2}+\\cdots +\\beta _{m}x_{m}=\\beta _{0}+\\sum _{i=1}^{m}\\beta _{i}x_{i}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0125", "text": ". Then when this is used in the equation relating the log odds of a success to the values of the predictors, the linear regression will be a multiple regression with m explanators; the parameters \n  \n    \n      \n        \n          \u03b2\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\beta _{i}}\n  \n for all \n  \n    \n      \n        i\n        =\n        0\n        ,\n        1\n        ,\n        2\n        ,\n        \u2026\n        ,\n        m", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0126", "text": "\u2026\n        ,\n        m\n      \n    \n    {\\displaystyle i=0,1,2,\\dots ,m}\n  \n are all estimated.\nAgain, the more traditional equations are:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0127", "text": "log\n        \u2061\n        \n          \n            p\n            \n              1\n              \u2212\n              p\n            \n          \n        \n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        \n          x\n          \n            1\n          \n        \n        +\n        \n          \u03b2\n          \n            2\n          \n        \n        \n          x", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0128", "text": "x\n          \n            2\n          \n        \n        +\n        \u22ef\n        +\n        \n          \u03b2\n          \n            m\n          \n        \n        \n          x\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle \\log {\\frac {p}{1-p}}=\\beta _{0}+\\beta _{1}x_{1}+\\beta _{2}x_{2}+\\cdots +\\beta _{m}x_{m}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0129", "text": "and", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0130", "text": "p\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                b\n                \n                  \u2212\n                  (\n                  \n                    \u03b2\n                    \n                      0\n                    \n                  \n                  +\n                  \n                    \u03b2\n                    \n                      1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0131", "text": "x\n                    \n                      1\n                    \n                  \n                  +\n                  \n                    \u03b2\n                    \n                      2\n                    \n                  \n                  \n                    x\n                    \n                      2\n                    \n                  \n                  +\n                  \u22ef\n                  +", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0132", "text": "+\n                  \n                    \u03b2\n                    \n                      m\n                    \n                  \n                  \n                    x\n                    \n                      m\n                    \n                  \n                  )\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle p={\\frac {1}{1+b^{-(\\beta _{0}+\\beta _{1}x_{1}+\\beta _{2}x_{2}+\\cdots +\\beta _{m}x_{m})}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0133", "text": "where usually \n  \n    \n      \n        b\n        =\n        e\n      \n    \n    {\\displaystyle b=e}\n  \n.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0134", "text": "Definition", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0135", "text": "A dataset contains N points. Each point i consists of a set of m input variables x1,i ... xm,i (also called independent variables, explanatory variables, predictor variables, features, or attributes), and a binary outcome variable Yi (also known as a dependent variable, response variable, output variable, or class), i.e. it can assume only the two possible values 0 (often meaning \"no\" or \"failure\") or 1 (often meaning \"yes\" or \"success\")", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0136", "text": ". The goal of logistic regression is to use the dataset to create a predictive model of the outcome variable.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0137", "text": "As in linear regression, the outcome variables Yi are assumed to depend on the explanatory variables x1,i ... xm,i.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0138", "text": "Explanatory variables\nThe explanatory variables may be of any type: real-valued, binary, categorical, etc.  The main distinction is between continuous variables and discrete variables.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0139", "text": "(Discrete variables referring to more than two possible choices are typically coded using dummy variables (or indicator variables), that is, separate explanatory variables taking the value 0 or 1 are created for each possible value of the discrete variable, with a 1 meaning \"variable does have the given value\" and a 0 meaning \"variable does not have that value\".)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0140", "text": "Outcome variables\nFormally, the outcomes Yi are described as being Bernoulli-distributed data, where each outcome is determined by an unobserved probability pi that is specific to the outcome at hand, but related to the explanatory variables.  This can be expressed in any of the following equivalent forms:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0141", "text": "Y\n                  \n                    i\n                  \n                \n                \u2223\n                \n                  x\n                  \n                    1\n                    ,\n                    i\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    m", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0142", "text": "m\n                    ,\n                    i\n                  \n                \n                 \n              \n              \n                \n                \u223c\n                Bernoulli\n                \u2061\n                (\n                \n                  p\n                  \n                    i\n                  \n                \n                )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0143", "text": "E\n                  \n                \n                \u2061\n                [\n                \n                  Y\n                  \n                    i\n                  \n                \n                \u2223\n                \n                  x\n                  \n                    1\n                    ,\n                    i\n                  \n                \n                ,\n                \u2026\n                ,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0144", "text": ",\n                \n                  x\n                  \n                    m\n                    ,\n                    i\n                  \n                \n                ]\n              \n              \n                \n                =\n                \n                  p\n                  \n                    i\n                  \n                \n              \n            \n            \n              \n                Pr\n                (", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0145", "text": "(\n                \n                  Y\n                  \n                    i\n                  \n                \n                =\n                y\n                \u2223\n                \n                  x\n                  \n                    1\n                    ,\n                    i\n                  \n                \n                ,\n                \u2026\n                ,\n                \n                  x\n                  \n                    m\n                    ,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0146", "text": "m\n                    ,\n                    i\n                  \n                \n                )\n              \n              \n                \n                =\n                \n                  \n                    {\n                    \n                      \n                        \n                          \n                            p\n                            \n                              i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0147", "text": "if \n                          \n                          y\n                          =\n                          1\n                        \n                      \n                      \n                        \n                          1\n                          \u2212\n                          \n                            p", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0148", "text": "i\n                            \n                          \n                        \n                        \n                          \n                            if \n                          \n                          y\n                          =\n                          0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0149", "text": "Pr\n                (\n                \n                  Y\n                  \n                    i\n                  \n                \n                =\n                y\n                \u2223\n                \n                  x\n                  \n                    1\n                    ,\n                    i\n                  \n                \n                ,\n                \u2026\n                ,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0150", "text": "\u2026\n                ,\n                \n                  x\n                  \n                    m\n                    ,\n                    i\n                  \n                \n                )\n              \n              \n                \n                =\n                \n                  p\n                  \n                    i\n                  \n                  \n                    y\n                  \n                \n                (\n                1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0151", "text": "(\n                1\n                \u2212\n                \n                  p\n                  \n                    i\n                  \n                \n                \n                  )\n                  \n                    (\n                    1\n                    \u2212\n                    y\n                    )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0152", "text": "{\\displaystyle {\\begin{aligned}Y_{i}\\mid x_{1,i},\\ldots ,x_{m,i}\\ &\\sim \\operatorname {Bernoulli} (p_{i})\\\\[5pt]\\operatorname {\\mathbb {E} } [Y_{i}\\mid x_{1,i},\\ldots ,x_{m,i}]&=p_{i}\\\\[5pt]\\Pr(Y_{i}=y\\mid x_{1,i},\\ldots ,x_{m,i})&={\\begin{cases}p_{i}&{\\text{if }}y=1\\\\1-p_{i}&{\\text{if }}y=0\\end{cases}}\\\\[5pt]\\Pr(Y_{i}=y\\mid x_{1,i},\\ldots ,x_{m,i})&=p_{i}^{y}(1-p_{i})^{(1-y)}\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0153", "text": "The meanings of these four lines are:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0154", "text": "The first line expresses the probability distribution of each Yi : conditioned on the explanatory variables, it follows a Bernoulli distribution with parameters pi, the probability of the outcome of 1 for trial i. As noted above, each separate trial has its own probability of success, just as each trial has its own explanatory variables.  The probability of success pi is not observed, only the outcome of an individual Bernoulli trial using that probability.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0155", "text": "The second line expresses the fact that the expected value of each Yi is equal to the probability of success pi, which is a general property of the Bernoulli distribution.  In other words, if we run a large number of Bernoulli trials using the same probability of success pi, then take the average of all the 1 and 0 outcomes, then the result would be close to pi", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0156", "text": ".  This is because doing an average this way simply computes the proportion of successes seen, which we expect to converge to the underlying probability of success.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0157", "text": "The third line writes out the probability mass function of the Bernoulli distribution, specifying the probability of seeing each of the two possible outcomes.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0158", "text": "The fourth line is another way of writing the probability mass function, which avoids having to write separate cases and is more convenient for certain types of calculations.  This relies on the fact that Yi can take only the value 0 or 1.  In each case, one of the exponents will be 1, \"choosing\" the value under it, while the other is 0, \"canceling out\" the value under it.  Hence, the outcome is either pi or 1 \u2212 pi, as in the previous line.\nLinear predictor function", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0159", "text": "Linear predictor function\nThe basic idea of logistic regression is to use the mechanism already developed for linear regression by modeling the probability pi using a linear predictor function, i.e. a linear combination of the explanatory variables and a set of regression coefficients that are specific to the model at hand but the same for all trials.  The linear predictor function \n  \n    \n      \n        f\n        (\n        i\n        )\n      \n    \n    {\\displaystyle f(i)}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0160", "text": ")\n      \n    \n    {\\displaystyle f(i)}\n  \n for a particular data point i is written as:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0161", "text": "f\n        (\n        i\n        )\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        \n          x\n          \n            1\n            ,\n            i\n          \n        \n        +\n        \u22ef\n        +\n        \n          \u03b2\n          \n            m\n          \n        \n        \n          x\n          \n            m\n            ,\n            i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0162", "text": ",\n            i\n          \n        \n        ,\n      \n    \n    {\\displaystyle f(i)=\\beta _{0}+\\beta _{1}x_{1,i}+\\cdots +\\beta _{m}x_{m,i},}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0163", "text": "where \n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \u03b2\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0},\\ldots ,\\beta _{m}}\n  \n are regression coefficients indicating the relative effect of a particular explanatory variable on the outcome.\nThe model is usually put into a more compact form as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0164", "text": "The regression coefficients \u03b20, \u03b21, ..., \u03b2m are grouped into a single vector \u03b2 of size m + 1.\nFor each data point i, an additional explanatory pseudo-variable x0,i is added, with a fixed value of 1, corresponding to the intercept coefficient \u03b20.\nThe resulting explanatory variables x0,i, x1,i, ..., xm,i are then grouped into a single vector Xi of size m + 1.\nThis makes it possible to write the linear predictor function as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0165", "text": "f\n        (\n        i\n        )\n        =\n        \n          \u03b2\n        \n        \u22c5\n        \n          \n            X\n          \n          \n            i\n          \n        \n        ,\n      \n    \n    {\\displaystyle f(i)={\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i},}\n  \n\nusing the notation for a dot product between two vectors.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0166", "text": "Many explanatory variables, two categories\nThe above example of binary logistic regression on one explanatory variable can be generalized to binary logistic regression on any number of explanatory variables x1, x2,... and any number of categorical values \n  \n    \n      \n        y\n        =\n        0\n        ,\n        1\n        ,\n        2\n        ,\n        \u2026\n      \n    \n    {\\displaystyle y=0,1,2,\\dots }\n  \n.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0167", "text": "{\\displaystyle y=0,1,2,\\dots }\n  \n.\nTo begin with, we may consider a logistic model with M explanatory variables, x1, x2 ... xM and, as in the example above, two categorical values (y = 0 and 1). For the simple binary logistic regression model, we assumed a linear relationship between the predictor variable and the log-odds (also called logit) of the event that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0168", "text": "1\n      \n    \n    {\\displaystyle y=1}\n  \n. This linear relationship may be extended to the case of M explanatory variables:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0169", "text": "t\n        =\n        \n          log\n          \n            b\n          \n        \n        \u2061\n        \n          \n            p\n            \n              1\n              \u2212\n              p\n            \n          \n        \n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        \n          x\n          \n            1\n          \n        \n        +\n        \n          \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0170", "text": "+\n        \n          \u03b2\n          \n            2\n          \n        \n        \n          x\n          \n            2\n          \n        \n        +\n        \u22ef\n        +\n        \n          \u03b2\n          \n            M\n          \n        \n        \n          x\n          \n            M\n          \n        \n      \n    \n    {\\displaystyle t=\\log _{b}{\\frac {p}{1-p}}=\\beta _{0}+\\beta _{1}x_{1}+\\beta _{2}x_{2}+\\cdots +\\beta _{M}x_{M}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0171", "text": "where t is the log-odds and \n  \n    \n      \n        \n          \u03b2\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\beta _{i}}\n  \n are parameters of the model. An additional generalization has been introduced in which the base of the model (b) is not restricted to Euler's number e. In most applications, the base \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0172", "text": "b\n      \n    \n    {\\displaystyle b}\n  \n of the logarithm is usually taken to be e. However, in some cases it can be easier to communicate results by working in base 2 or base 10.\nFor a more compact notation, we will specify the explanatory variables and the \u03b2 coefficients as \u2060\n  \n    \n      \n        (\n        M\n        +\n        1\n        )\n      \n    \n    {\\displaystyle (M+1)}\n  \n\u2060-dimensional vectors:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0173", "text": "x\n        \n        =\n        {\n        \n          x\n          \n            0\n          \n        \n        ,\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            M\n          \n        \n        }\n      \n    \n    {\\displaystyle {\\boldsymbol {x}}=\\{x_{0},x_{1},x_{2},\\dots ,x_{M}\\}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0174", "text": "\u03b2\n        \n        =\n        {\n        \n          \u03b2\n          \n            0\n          \n        \n        ,\n        \n          \u03b2\n          \n            1\n          \n        \n        ,\n        \n          \u03b2\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \u03b2\n          \n            M\n          \n        \n        }\n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}=\\{\\beta _{0},\\beta _{1},\\beta _{2},\\dots ,\\beta _{M}\\}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0175", "text": "with an added explanatory variable x0 =1. The logit may now be written as:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0176", "text": "t\n        =\n        \n          \u2211\n          \n            m\n            =\n            0\n          \n          \n            M\n          \n        \n        \n          \u03b2\n          \n            m\n          \n        \n        \n          x\n          \n            m\n          \n        \n        =\n        \n          \u03b2\n        \n        \u22c5\n        x\n      \n    \n    {\\displaystyle t=\\sum _{m=0}^{M}\\beta _{m}x_{m}={\\boldsymbol {\\beta }}\\cdot x}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0177", "text": "Solving for the probability p that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n yields:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0178", "text": "p\n        (\n        \n          x\n        \n        )\n        =\n        \n          \n            \n              b\n              \n                \n                  \u03b2\n                \n                \u22c5\n                \n                  x\n                \n              \n            \n            \n              1\n              +\n              \n                b\n                \n                  \n                    \u03b2\n                  \n                  \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0179", "text": "\u22c5\n                  \n                    x\n                  \n                \n              \n            \n          \n        \n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                b\n                \n                  \u2212\n                  \n                    \u03b2\n                  \n                  \u22c5\n                  \n                    x", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0180", "text": "=\n        \n          S\n          \n            b\n          \n        \n        (\n        t\n        )\n      \n    \n    {\\displaystyle p({\\boldsymbol {x}})={\\frac {b^{{\\boldsymbol {\\beta }}\\cdot {\\boldsymbol {x}}}}{1+b^{{\\boldsymbol {\\beta }}\\cdot {\\boldsymbol {x}}}}}={\\frac {1}{1+b^{-{\\boldsymbol {\\beta }}\\cdot {\\boldsymbol {x}}}}}=S_{b}(t)}\n  \n,\nwhere \n  \n    \n      \n        \n          S\n          \n            b", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0181", "text": "b\n          \n        \n      \n    \n    {\\displaystyle S_{b}}\n  \n is the sigmoid function with base \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}\n  \n. The above formula shows that once the \n  \n    \n      \n        \n          \u03b2\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle \\beta _{m}}\n  \n are fixed, we can easily compute either the log-odds that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0182", "text": "1\n      \n    \n    {\\displaystyle y=1}\n  \n for a given observation, or the probability that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n for a given observation. The main use-case of a logistic model is to be given an observation \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle {\\boldsymbol {x}}}\n  \n, and estimate the probability \n  \n    \n      \n        p\n        (\n        \n          x\n        \n        )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0183", "text": "x\n        \n        )\n      \n    \n    {\\displaystyle p({\\boldsymbol {x}})}\n  \n that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n. The optimum beta coefficients may again be found by maximizing the log-likelihood. For K measurements, defining \n  \n    \n      \n        \n          \n            x\n          \n          \n            k\n          \n        \n      \n    \n    {\\displaystyle {\\boldsymbol {x}}_{k}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0184", "text": "{\\displaystyle {\\boldsymbol {x}}_{k}}\n  \n as the explanatory vector of the k-th measurement, and \n  \n    \n      \n        \n          y\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle y_{k}}\n  \n as the categorical outcome of that measurement, the log likelihood may be written in a form very similar to the simple \n  \n    \n      \n        M\n        =\n        1\n      \n    \n    {\\displaystyle M=1}\n  \n case above:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0185", "text": "\u2113\n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        \n          y\n          \n            k\n          \n        \n        \n          log\n          \n            b\n          \n        \n        \u2061\n        (\n        p\n        (\n        \n          \n            x\n            \n              k\n            \n          \n        \n        )\n        )\n        +\n        \n          \u2211", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0186", "text": "+\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        (\n        1\n        \u2212\n        \n          y\n          \n            k\n          \n        \n        )\n        \n          log\n          \n            b\n          \n        \n        \u2061\n        (\n        1\n        \u2212\n        p\n        (\n        \n          \n            x\n            \n              k\n            \n          \n        \n        )\n        )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0187", "text": ")\n        )\n      \n    \n    {\\displaystyle \\ell =\\sum _{k=1}^{K}y_{k}\\log _{b}(p({\\boldsymbol {x_{k}}}))+\\sum _{k=1}^{K}(1-y_{k})\\log _{b}(1-p({\\boldsymbol {x_{k}}}))}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0188", "text": "As in the simple example above, finding the optimum \u03b2 parameters will require numerical methods. One useful technique is to equate the derivatives of the log likelihood with respect to each of the \u03b2 parameters to zero yielding a set of equations which will hold at the maximum of the log likelihood:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0189", "text": "\u2202\n              \u2113\n            \n            \n              \u2202\n              \n                \u03b2\n                \n                  m\n                \n              \n            \n          \n        \n        =\n        0\n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        \n          y\n          \n            k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0190", "text": "k\n          \n        \n        \n          x\n          \n            m\n            k\n          \n        \n        \u2212\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        p\n        (\n        \n          \n            x\n          \n          \n            k\n          \n        \n        )\n        \n          x\n          \n            m\n            k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0191", "text": "k\n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial \\ell }{\\partial \\beta _{m}}}=0=\\sum _{k=1}^{K}y_{k}x_{mk}-\\sum _{k=1}^{K}p({\\boldsymbol {x}}_{k})x_{mk}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0192", "text": "where xmk is the value of the xm explanatory variable from the k-th measurement.\nConsider an example with \n  \n    \n      \n        M\n        =\n        2\n      \n    \n    {\\displaystyle M=2}\n  \n explanatory variables, \n  \n    \n      \n        b\n        =\n        10\n      \n    \n    {\\displaystyle b=10}\n  \n, and coefficients \n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n        =\n        \u2212\n        3\n      \n    \n    {\\displaystyle \\beta _{0}=-3}\n  \n,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0193", "text": "{\\displaystyle \\beta _{0}=-3}\n  \n, \n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle \\beta _{1}=1}\n  \n, and \n  \n    \n      \n        \n          \u03b2\n          \n            2\n          \n        \n        =\n        2\n      \n    \n    {\\displaystyle \\beta _{2}=2}\n  \n which have been determined by the above method. To be concrete, the model is:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0194", "text": "t\n        =\n        \n          log\n          \n            10\n          \n        \n        \u2061\n        \n          \n            p\n            \n              1\n              \u2212\n              p\n            \n          \n        \n        =\n        \u2212\n        3\n        +\n        \n          x\n          \n            1\n          \n        \n        +\n        2\n        \n          x\n          \n            2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0195", "text": "2\n          \n        \n      \n    \n    {\\displaystyle t=\\log _{10}{\\frac {p}{1-p}}=-3+x_{1}+2x_{2}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0196", "text": "p\n        =\n        \n          \n            \n              b\n              \n                \n                  \u03b2\n                \n                \u22c5\n                \n                  x\n                \n              \n            \n            \n              1\n              +\n              \n                b\n                \n                  \n                    \u03b2\n                  \n                  \u22c5\n                  x", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0197", "text": "=\n        \n          \n            \n              b\n              \n                \n                  \u03b2\n                  \n                    0\n                  \n                \n                +\n                \n                  \u03b2\n                  \n                    1\n                  \n                \n                \n                  x\n                  \n                    1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0198", "text": "+\n                \n                  \u03b2\n                  \n                    2\n                  \n                \n                \n                  x\n                  \n                    2\n                  \n                \n              \n            \n            \n              1\n              +\n              \n                b\n                \n                  \n                    \u03b2\n                    \n                      0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0199", "text": "0\n                    \n                  \n                  +\n                  \n                    \u03b2\n                    \n                      1\n                    \n                  \n                  \n                    x\n                    \n                      1\n                    \n                  \n                  +\n                  \n                    \u03b2\n                    \n                      2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0200", "text": "2\n                    \n                  \n                  \n                    x\n                    \n                      2\n                    \n                  \n                \n              \n            \n          \n        \n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                b\n                \n                  \u2212\n                  (\n                  \n                    \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0201", "text": "\u03b2\n                    \n                      0\n                    \n                  \n                  +\n                  \n                    \u03b2\n                    \n                      1\n                    \n                  \n                  \n                    x\n                    \n                      1\n                    \n                  \n                  +\n                  \n                    \u03b2\n                    \n                      2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0202", "text": "2\n                    \n                  \n                  \n                    x\n                    \n                      2\n                    \n                  \n                  )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0203", "text": "{\\displaystyle p={\\frac {b^{{\\boldsymbol {\\beta }}\\cdot {\\boldsymbol {x}}}}{1+b^{{\\boldsymbol {\\beta }}\\cdot x}}}={\\frac {b^{\\beta _{0}+\\beta _{1}x_{1}+\\beta _{2}x_{2}}}{1+b^{\\beta _{0}+\\beta _{1}x_{1}+\\beta _{2}x_{2}}}}={\\frac {1}{1+b^{-(\\beta _{0}+\\beta _{1}x_{1}+\\beta _{2}x_{2})}}}}\n  \n,\nwhere p is the probability of the event that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0204", "text": "1\n      \n    \n    {\\displaystyle y=1}\n  \n. This can be interpreted as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0205", "text": "\u03b2\n          \n            0\n          \n        \n        =\n        \u2212\n        3\n      \n    \n    {\\displaystyle \\beta _{0}=-3}\n  \n is the y-intercept. It is the log-odds of the event that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n, when the predictors \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        =\n        \n          x\n          \n            2\n          \n        \n        =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0206", "text": "2\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle x_{1}=x_{2}=0}\n  \n. By exponentiating, we can see that when \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        =\n        \n          x\n          \n            2\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle x_{1}=x_{2}=0}\n  \n the odds of the event that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0207", "text": "1\n      \n    \n    {\\displaystyle y=1}\n  \n are 1-to-1000, or \n  \n    \n      \n        \n          10\n          \n            \u2212\n            3\n          \n        \n      \n    \n    {\\displaystyle 10^{-3}}\n  \n. Similarly, the probability of the event that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n when \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        =\n        \n          x\n          \n            2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0208", "text": "x\n          \n            2\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle x_{1}=x_{2}=0}\n  \n can be computed as \n  \n    \n      \n        1\n        \n          /\n        \n        (\n        1000\n        +\n        1\n        )\n        =\n        1\n        \n          /\n        \n        1001.\n      \n    \n    {\\displaystyle 1/(1000+1)=1/1001.}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0209", "text": "\u03b2\n          \n            1\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle \\beta _{1}=1}\n  \n means that increasing \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle x_{1}}\n  \n by 1 increases the log-odds by \n  \n    \n      \n        1\n      \n    \n    {\\displaystyle 1}\n  \n. So if \n  \n    \n      \n        \n          x\n          \n            1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0210", "text": "1\n          \n        \n      \n    \n    {\\displaystyle x_{1}}\n  \n increases by 1, the odds that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n increase by a factor of \n  \n    \n      \n        \n          10\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle 10^{1}}\n  \n. The probability of \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0211", "text": "1\n      \n    \n    {\\displaystyle y=1}\n  \n has also increased, but it has not increased by as much as the odds have increased.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0212", "text": "\u03b2\n          \n            2\n          \n        \n        =\n        2\n      \n    \n    {\\displaystyle \\beta _{2}=2}\n  \n means that increasing \n  \n    \n      \n        \n          x\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle x_{2}}\n  \n by 1 increases the log-odds by \n  \n    \n      \n        2\n      \n    \n    {\\displaystyle 2}\n  \n. So if \n  \n    \n      \n        \n          x\n          \n            2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0213", "text": "2\n          \n        \n      \n    \n    {\\displaystyle x_{2}}\n  \n increases by 1, the odds that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n increase by a factor of \n  \n    \n      \n        \n          10\n          \n            2\n          \n        \n        .\n      \n    \n    {\\displaystyle 10^{2}.}\n  \n Note how the effect of \n  \n    \n      \n        \n          x\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle x_{2}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0214", "text": "{\\displaystyle x_{2}}\n  \n on the log-odds is twice as great as the effect of \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle x_{1}}\n  \n, but the effect on the odds is 10 times greater. But the effect on the probability of \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n is not as much as 10 times greater, it's only the effect on the odds that is 10 times greater.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0215", "text": "Multinomial logistic regression: Many explanatory variables and many categories\nIn the above cases of two categories (binomial logistic regression), the categories were indexed by \"0\" and \"1\", and we had two probabilities: The probability that the outcome was in category 1 was given by \n  \n    \n      \n        p\n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p({\\boldsymbol {x}})}\n  \nand the probability that the outcome was in category 0 was given by", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0216", "text": "1\n        \u2212\n        p\n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle 1-p({\\boldsymbol {x}})}\n  \n. The sum of these probabilities equals 1, which must be true, since \"0\" and \"1\" are the only possible categories in this setup.\nIn general, if we have \u2060\n  \n    \n      \n        M\n        +\n        1\n      \n    \n    {\\displaystyle M+1}\n  \n\u2060 explanatory variables (including x0) and \u2060\n  \n    \n      \n        N\n        +\n        1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0217", "text": "N\n        +\n        1\n      \n    \n    {\\displaystyle N+1}\n  \n\u2060 categories, we will need \u2060\n  \n    \n      \n        N\n        +\n        1\n      \n    \n    {\\displaystyle N+1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0218", "text": "1\n      \n    \n    {\\displaystyle N+1}\n  \n\u2060 separate probabilities,  one for each category, indexed by n, which describe the probability that the categorical outcome y will be in category y=n, conditional on the vector of covariates x. The sum of these probabilities over all categories must equal 1. Using the mathematically convenient base e, these probabilities are:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0219", "text": "p\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        )\n        =\n        \n          \n            \n              e\n              \n                \n                  \n                    \u03b2\n                  \n                  \n                    n\n                  \n                \n                \u22c5\n                \n                  x\n                \n              \n            \n            \n              1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0220", "text": "1\n              +\n              \n                \u2211\n                \n                  u\n                  =\n                  1\n                \n                \n                  N\n                \n              \n              \n                e\n                \n                  \n                    \n                      \u03b2\n                    \n                    \n                      u\n                    \n                  \n                  \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0221", "text": "\u22c5\n                  \n                    x\n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle p_{n}({\\boldsymbol {x}})={\\frac {e^{{\\boldsymbol {\\beta }}_{n}\\cdot {\\boldsymbol {x}}}}{1+\\sum _{u=1}^{N}e^{{\\boldsymbol {\\beta }}_{u}\\cdot {\\boldsymbol {x}}}}}}\n  \n for \n  \n    \n      \n        n\n        =\n        1\n        ,\n        2\n        ,\n        \u2026\n        ,\n        N", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0222", "text": "\u2026\n        ,\n        N\n      \n    \n    {\\displaystyle n=1,2,\\dots ,N}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0223", "text": "p\n          \n            0\n          \n        \n        (\n        \n          x\n        \n        )\n        =\n        1\n        \u2212\n        \n          \u2211\n          \n            n\n            =\n            1\n          \n          \n            N\n          \n        \n        \n          p\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        )\n        =\n        \n          \n            1\n            \n              1\n              +", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0224", "text": "1\n              +\n              \n                \u2211\n                \n                  u\n                  =\n                  1\n                \n                \n                  N\n                \n              \n              \n                e\n                \n                  \n                    \n                      \u03b2\n                    \n                    \n                      u\n                    \n                  \n                  \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0225", "text": "\u22c5\n                  \n                    x\n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle p_{0}({\\boldsymbol {x}})=1-\\sum _{n=1}^{N}p_{n}({\\boldsymbol {x}})={\\frac {1}{1+\\sum _{u=1}^{N}e^{{\\boldsymbol {\\beta }}_{u}\\cdot {\\boldsymbol {x}}}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0226", "text": "Each of the probabilities except \n  \n    \n      \n        \n          p\n          \n            0\n          \n        \n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p_{0}({\\boldsymbol {x}})}\n  \n will have their own set of regression coefficients \n  \n    \n      \n        \n          \n            \u03b2\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}_{n}}\n  \n.  It can be seen that, as required, the sum of the", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0227", "text": "p\n          \n            n\n          \n        \n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p_{n}({\\boldsymbol {x}})}\n  \n over all categories n is 1. The selection of \n  \n    \n      \n        \n          p\n          \n            0\n          \n        \n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p_{0}({\\boldsymbol {x}})}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0228", "text": "{\\displaystyle p_{0}({\\boldsymbol {x}})}\n  \n to be defined in terms of the other probabilities is artificial. Any of the probabilities could have been selected to be so defined. This special value of n is termed the \"pivot index\", and the log-odds (tn) are expressed in terms of the pivot probability and are again expressed as a linear combination of the explanatory variables:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0229", "text": "t\n          \n            n\n          \n        \n        =\n        ln\n        \u2061\n        \n          (\n          \n            \n              \n                \n                  p\n                  \n                    n\n                  \n                \n                (\n                \n                  x\n                \n                )\n              \n              \n                \n                  p\n                  \n                    0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0230", "text": "0\n                  \n                \n                (\n                \n                  x\n                \n                )\n              \n            \n          \n          )\n        \n        =\n        \n          \n            \u03b2\n          \n          \n            n\n          \n        \n        \u22c5\n        \n          x", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0231", "text": "x\n        \n      \n    \n    {\\displaystyle t_{n}=\\ln \\left({\\frac {p_{n}({\\boldsymbol {x}})}{p_{0}({\\boldsymbol {x}})}}\\right)={\\boldsymbol {\\beta }}_{n}\\cdot {\\boldsymbol {x}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0232", "text": "Note also that for the simple case of \n  \n    \n      \n        N\n        =\n        1\n      \n    \n    {\\displaystyle N=1}\n  \n, the two-category case is recovered, with \n  \n    \n      \n        p\n        (\n        \n          x\n        \n        )\n        =\n        \n          p\n          \n            1\n          \n        \n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p({\\boldsymbol {x}})=p_{1}({\\boldsymbol {x}})}\n  \n and \n  \n    \n      \n        \n          p", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0233", "text": "and \n  \n    \n      \n        \n          p\n          \n            0\n          \n        \n        (\n        \n          x\n        \n        )\n        =\n        1\n        \u2212\n        \n          p\n          \n            1\n          \n        \n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle p_{0}({\\boldsymbol {x}})=1-p_{1}({\\boldsymbol {x}})}\n  \n.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0234", "text": ".\nThe log-likelihood that a particular set of K measurements or data points will be generated by the above probabilities can now be calculated. Indexing each measurement by k, let the k-th set of measured explanatory variables be denoted by \n  \n    \n      \n        \n          \n            x\n          \n          \n            k\n          \n        \n      \n    \n    {\\displaystyle {\\boldsymbol {x}}_{k}}\n  \n and their categorical outcomes be denoted by \n  \n    \n      \n        \n          y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0235", "text": "y\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle y_{k}}\n  \n which can be equal to any integer in [0,N]. The log-likelihood is then:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0236", "text": "\u2113\n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        \n          \u2211\n          \n            n\n            =\n            0\n          \n          \n            N\n          \n        \n        \u0394\n        (\n        n\n        ,\n        \n          y\n          \n            k\n          \n        \n        )\n        \n        ln\n        \u2061\n        (\n        \n          p\n          \n            n", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0237", "text": "p\n          \n            n\n          \n        \n        (\n        \n          \n            x\n          \n          \n            k\n          \n        \n        )\n        )\n      \n    \n    {\\displaystyle \\ell =\\sum _{k=1}^{K}\\sum _{n=0}^{N}\\Delta (n,y_{k})\\,\\ln(p_{n}({\\boldsymbol {x}}_{k}))}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0238", "text": "where \n  \n    \n      \n        \u0394\n        (\n        n\n        ,\n        \n          y\n          \n            k\n          \n        \n        )\n      \n    \n    {\\displaystyle \\Delta (n,y_{k})}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0239", "text": "is an indicator function which equals 1 if yk = n and zero otherwise. In the case of two explanatory variables, this indicator function was defined as yk when n = 1 and 1-yk when n = 0. This was convenient, but not necessary. Again, the optimum beta coefficients may be found by maximizing the log-likelihood function generally using numerical methods", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0240", "text": ". A possible method of solution is to set the derivatives of the log-likelihood with respect to each beta coefficient equal to zero and solve for the beta coefficients:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0241", "text": "\u2202\n              \u2113\n            \n            \n              \u2202\n              \n                \u03b2\n                \n                  n\n                  m\n                \n              \n            \n          \n        \n        =\n        0\n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        \u0394\n        (\n        n\n        ,\n        \n          y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0242", "text": "n\n        ,\n        \n          y\n          \n            k\n          \n        \n        )\n        \n          x\n          \n            m\n            k\n          \n        \n        \u2212\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        \n          p\n          \n            n\n          \n        \n        (\n        \n          \n            x\n          \n          \n            k\n          \n        \n        )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0243", "text": "k\n          \n        \n        )\n        \n          x\n          \n            m\n            k\n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial \\ell }{\\partial \\beta _{nm}}}=0=\\sum _{k=1}^{K}\\Delta (n,y_{k})x_{mk}-\\sum _{k=1}^{K}p_{n}({\\boldsymbol {x}}_{k})x_{mk}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0244", "text": "where \n  \n    \n      \n        \n          \u03b2\n          \n            n\n            m\n          \n        \n      \n    \n    {\\displaystyle \\beta _{nm}}\n  \n is the m-th coefficient of the \n  \n    \n      \n        \n          \n            \u03b2\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}_{n}}\n  \n vector and \n  \n    \n      \n        \n          x\n          \n            m\n            k\n          \n        \n      \n    \n    {\\displaystyle x_{mk}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0245", "text": "{\\displaystyle x_{mk}}\n  \n is the m-th explanatory variable of the k-th measurement. Once the beta coefficients have been estimated from the data, we will be able to estimate the probability that any subsequent set of explanatory variables will result in any of the possible outcome categories.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0246", "text": "Interpretations\nThere are various equivalent specifications and interpretations of logistic regression, which fit into different types of more general models, and allow different generalizations.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0247", "text": "As a generalized linear model\nThe particular model used by logistic regression, which distinguishes it from standard linear regression and from other types of regression analysis used for binary-valued outcomes, is the way the probability of a particular outcome is linked to the linear predictor function:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0248", "text": "logit\n        \u2061\n        (\n        \n          \n            E\n          \n        \n        \u2061\n        [\n        \n          Y\n          \n            i\n          \n        \n        \u2223\n        \n          x\n          \n            1\n            ,\n            i\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            m\n            ,\n            i\n          \n        \n        ]\n        )\n        =\n        logit\n        \u2061\n        (", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0249", "text": "logit\n        \u2061\n        (\n        \n          p\n          \n            i\n          \n        \n        )\n        =\n        ln\n        \u2061\n        \n          (\n          \n            \n              \n                p\n                \n                  i\n                \n              \n              \n                1\n                \u2212\n                \n                  p\n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0250", "text": ")\n        \n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        \n          x\n          \n            1\n            ,\n            i\n          \n        \n        +\n        \u22ef\n        +\n        \n          \u03b2\n          \n            m\n          \n        \n        \n          x\n          \n            m\n            ,\n            i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0251", "text": ",\n            i\n          \n        \n      \n    \n    {\\displaystyle \\operatorname {logit} (\\operatorname {\\mathbb {E} } [Y_{i}\\mid x_{1,i},\\ldots ,x_{m,i}])=\\operatorname {logit} (p_{i})=\\ln \\left({\\frac {p_{i}}{1-p_{i}}}\\right)=\\beta _{0}+\\beta _{1}x_{1,i}+\\cdots +\\beta _{m}x_{m,i}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0252", "text": "Written using the more compact notation described above, this is:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0253", "text": "logit\n        \u2061\n        (\n        \n          \n            E\n          \n        \n        \u2061\n        [\n        \n          Y\n          \n            i\n          \n        \n        \u2223\n        \n          \n            X\n          \n          \n            i\n          \n        \n        ]\n        )\n        =\n        logit\n        \u2061\n        (\n        \n          p\n          \n            i\n          \n        \n        )\n        =\n        ln\n        \u2061\n        \n          (", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0254", "text": "\u2061\n        \n          (\n          \n            \n              \n                p\n                \n                  i\n                \n              \n              \n                1\n                \u2212\n                \n                  p\n                  \n                    i\n                  \n                \n              \n            \n          \n          )\n        \n        =\n        \n          \u03b2\n        \n        \u22c5\n        \n          \n            X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0255", "text": "X\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\operatorname {logit} (\\operatorname {\\mathbb {E} } [Y_{i}\\mid \\mathbf {X} _{i}])=\\operatorname {logit} (p_{i})=\\ln \\left({\\frac {p_{i}}{1-p_{i}}}\\right)={\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0256", "text": "This formulation expresses logistic regression as a type of generalized linear model, which predicts variables with various types of probability distributions by fitting a linear predictor function of the above form to some sort of arbitrary transformation of the expected value of the variable.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0257", "text": "The intuition for transforming using the logit function (the natural log of the odds) was explained above.  It also has the practical effect of converting the probability (which is bounded to be between 0 and 1) to a variable that ranges over \n  \n    \n      \n        (\n        \u2212\n        \u221e\n        ,\n        +\n        \u221e\n        )\n      \n    \n    {\\displaystyle (-\\infty ,+\\infty )}\n  \n \u2014 thereby matching the potential range of the linear prediction function on the right side of the equation.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0258", "text": "Both the probabilities pi and the regression coefficients are unobserved, and the means of determining them is not part of the model itself.  They are typically determined by some sort of optimization procedure, e.g. maximum likelihood estimation, that finds values that best fit the observed data (i.e. that give the most accurate predictions for the data already observed), usually subject to regularization conditions that seek to exclude unlikely values, e.g", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0259", "text": ". extremely large values for any of the regression coefficients.  The use of a regularization condition is equivalent to doing maximum a posteriori (MAP) estimation, an extension of maximum likelihood", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0260", "text": ".  (Regularization is most commonly done using a squared regularizing function, which is equivalent to placing a zero-mean Gaussian prior distribution on the coefficients, but other regularizers are also possible.)  Whether or not regularization is used, it is usually not possible to find a closed-form solution; instead, an iterative numerical method must be used, such as iteratively reweighted least squares (IRLS) or, more commonly these days, a quasi-Newton method such as the L-BFGS method.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0261", "text": "The interpretation of the \u03b2j parameter estimates is as the additive effect on the log of the odds for a unit change in the j the explanatory variable.  In the case of a dichotomous explanatory variable, for instance, gender \n  \n    \n      \n        \n          e\n          \n            \u03b2\n          \n        \n      \n    \n    {\\displaystyle e^{\\beta }}\n  \n is the estimate of the odds of having the outcome for, say, males compared with females.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0262", "text": "An equivalent formula uses the inverse of the logit function, which is the logistic function, i.e.:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0263", "text": "E\n          \n        \n        \u2061\n        [\n        \n          Y\n          \n            i\n          \n        \n        \u2223\n        \n          \n            X\n          \n          \n            i\n          \n        \n        ]\n        =\n        \n          p\n          \n            i\n          \n        \n        =\n        \n          logit\n          \n            \u2212\n            1\n          \n        \n        \u2061\n        (\n        \n          \u03b2\n        \n        \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0264", "text": "(\n        \n          \u03b2\n        \n        \u22c5\n        \n          \n            X\n          \n          \n            i\n          \n        \n        )\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  \n                    \u03b2\n                  \n                  \u22c5\n                  \n                    \n                      X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0265", "text": "i\n                    \n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\operatorname {\\mathbb {E} } [Y_{i}\\mid \\mathbf {X} _{i}]=p_{i}=\\operatorname {logit} ^{-1}({\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i})={\\frac {1}{1+e^{-{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0266", "text": "The formula can also be written as a probability distribution (specifically, using a probability mass function):", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0267", "text": "Pr\n        (\n        \n          Y\n          \n            i\n          \n        \n        =\n        y\n        \u2223\n        \n          \n            X\n          \n          \n            i\n          \n        \n        )\n        =\n        \n          \n            \n              p\n              \n                i\n              \n            \n          \n          \n            y\n          \n        \n        (\n        1\n        \u2212\n        \n          p\n          \n            i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0268", "text": "p\n          \n            i\n          \n        \n        \n          )\n          \n            1\n            \u2212\n            y\n          \n        \n        =\n        \n          \n            (\n            \n              \n                \n                  e\n                  \n                    \n                      \u03b2\n                    \n                    \u22c5\n                    \n                      \n                        X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0269", "text": "i\n                      \n                    \n                  \n                \n                \n                  1\n                  +\n                  \n                    e\n                    \n                      \n                        \u03b2\n                      \n                      \u22c5\n                      \n                        \n                          X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0270", "text": "i\n                        \n                      \n                    \n                  \n                \n              \n            \n            )\n          \n          \n            y\n          \n        \n        \n          \n            (\n            \n              1\n              \u2212\n              \n                \n                  \n                    e", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0271", "text": "\u03b2\n                      \n                      \u22c5\n                      \n                        \n                          X\n                        \n                        \n                          i\n                        \n                      \n                    \n                  \n                  \n                    1\n                    +\n                    \n                      e", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0272", "text": "e\n                      \n                        \n                          \u03b2\n                        \n                        \u22c5\n                        \n                          \n                            X\n                          \n                          \n                            i\n                          \n                        \n                      \n                    \n                  \n                \n              \n            \n            )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0273", "text": ")\n          \n          \n            1\n            \u2212\n            y\n          \n        \n        =\n        \n          \n            \n              e\n              \n                \n                  \u03b2\n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                \u22c5\n                y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0274", "text": "y\n              \n            \n            \n              1\n              +\n              \n                e\n                \n                  \n                    \u03b2\n                  \n                  \u22c5\n                  \n                    \n                      X\n                    \n                    \n                      i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0275", "text": "{\\displaystyle \\Pr(Y_{i}=y\\mid \\mathbf {X} _{i})={p_{i}}^{y}(1-p_{i})^{1-y}=\\left({\\frac {e^{{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}}}{1+e^{{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}}}}\\right)^{y}\\left(1-{\\frac {e^{{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}}}{1+e^{{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}}}}\\right)^{1-y}={\\frac {e^{{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}\\cdot y}}{1+e^{{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0276", "text": "As a latent-variable model\nThe logistic model has an equivalent formulation as a latent-variable model.  This formulation is common in the theory of discrete choice models and makes it easier to extend to certain more complicated models with multiple, correlated choices, as well as to compare logistic regression to the closely related probit model.\nImagine that, for each trial i, there is a continuous latent variable Yi* (i.e. an unobserved random variable) that is distributed as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0277", "text": "Y\n          \n            i\n          \n          \n            \u2217\n          \n        \n        =\n        \n          \u03b2\n        \n        \u22c5\n        \n          \n            X\n          \n          \n            i\n          \n        \n        +\n        \n          \u03b5\n          \n            i\n          \n        \n        \n      \n    \n    {\\displaystyle Y_{i}^{\\ast }={\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}+\\varepsilon _{i}\\,}\n  \n\nwhere", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0278", "text": "where\n\n  \n    \n      \n        \n          \u03b5\n          \n            i\n          \n        \n        \u223c\n        Logistic\n        \u2061\n        (\n        0\n        ,\n        1\n        )\n        \n      \n    \n    {\\displaystyle \\varepsilon _{i}\\sim \\operatorname {Logistic} (0,1)\\,}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0279", "text": "i.e. the latent variable can be written directly in terms of the linear predictor function and an additive random error variable that is distributed according to a standard logistic distribution.\nThen Yi can be viewed as an indicator for whether this latent variable is positive:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0280", "text": "Y\n          \n            i\n          \n        \n        =\n        \n          \n            {\n            \n              \n                \n                  1\n                \n                \n                  \n                    if \n                  \n                  \n                    Y\n                    \n                      i\n                    \n                    \n                      \u2217\n                    \n                  \n                  >", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0281", "text": ">\n                  0\n                   \n                  \n                     i.e. \n                  \n                  \n                    \u2212\n                    \n                      \u03b5\n                      \n                        i\n                      \n                    \n                  \n                  <\n                  \n                    \u03b2\n                  \n                  \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0282", "text": "X\n                    \n                    \n                      i\n                    \n                  \n                  ,\n                \n              \n              \n                \n                  0\n                \n                \n                  \n                    otherwise.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0283", "text": "{\\displaystyle Y_{i}={\\begin{cases}1&{\\text{if }}Y_{i}^{\\ast }>0\\ {\\text{ i.e. }}{-\\varepsilon _{i}}<{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i},\\\\0&{\\text{otherwise.}}\\end{cases}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0284", "text": "The choice of modeling the error variable specifically with a standard logistic distribution, rather than a general logistic distribution with the location and scale set to arbitrary values, seems restrictive, but in fact, it is not.  It must be kept in mind that we can choose the regression coefficients ourselves, and very often can use them to offset changes in the parameters of the error variable's distribution", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0285", "text": ".  For example, a logistic error-variable distribution with a non-zero location parameter \u03bc (which sets the mean) is equivalent to a distribution with a zero location parameter, where \u03bc has been added to the intercept coefficient.  Both situations produce the same value for Yi* regardless of settings of explanatory variables.  Similarly, an arbitrary scale parameter s is equivalent to setting the scale parameter to 1 and then dividing all regression coefficients by s", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0286", "text": ".  In the latter case, the resulting value of Yi* will be smaller by a factor of s than in the former case, for all sets of explanatory variables \u2014 but critically, it will always remain on the same side of 0, and hence lead to the same Yi choice.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0287", "text": "(This predicts that the irrelevancy of the scale parameter may not carry over into more complex models where more than two choices are available.)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0288", "text": "It turns out that this formulation is exactly equivalent to the preceding one, phrased in terms of the generalized linear model and without any latent variables.  This can be shown as follows, using the fact that the cumulative distribution function (CDF) of the standard logistic distribution is the logistic function, which is the inverse of the logit function, i.e.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0289", "text": "Pr\n        (\n        \n          \u03b5\n          \n            i\n          \n        \n        <\n        x\n        )\n        =\n        \n          logit\n          \n            \u2212\n            1\n          \n        \n        \u2061\n        (\n        x\n        )\n      \n    \n    {\\displaystyle \\Pr(\\varepsilon _{i}<x)=\\operatorname {logit} ^{-1}(x)}\n  \n\nThen:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0290", "text": "Pr\n                (\n                \n                  Y\n                  \n                    i\n                  \n                \n                =\n                1\n                \u2223\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0291", "text": "=\n                Pr\n                (\n                \n                  Y\n                  \n                    i\n                  \n                  \n                    \u2217\n                  \n                \n                >\n                0\n                \u2223\n                \n                  \n                    X\n                  \n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0292", "text": ")\n              \n            \n            \n              \n              \n                \n                =\n                Pr\n                (\n                \n                  \u03b2\n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                +\n                \n                  \u03b5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0293", "text": "\u03b5\n                  \n                    i\n                  \n                \n                >\n                0\n                )\n              \n            \n            \n              \n              \n                \n                =\n                Pr\n                (\n                \n                  \u03b5\n                  \n                    i\n                  \n                \n                >\n                \u2212\n                \n                  \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0294", "text": "\u03b2\n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                )\n              \n            \n            \n              \n              \n                \n                =\n                Pr\n                (\n                \n                  \u03b5\n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0295", "text": "<\n                \n                  \u03b2\n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                )\n              \n              \n              \n                \n                  (because the logistic distribution is symmetric)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0296", "text": "=\n                \n                  logit\n                  \n                    \u2212\n                    1\n                  \n                \n                \u2061\n                (\n                \n                  \u03b2\n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0297", "text": ")\n              \n              \n            \n            \n              \n              \n                \n                =\n                \n                  p\n                  \n                    i\n                  \n                \n              \n              \n              \n                \n                  (see above)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0298", "text": "{\\displaystyle {\\begin{aligned}\\Pr(Y_{i}=1\\mid \\mathbf {X} _{i})&=\\Pr(Y_{i}^{\\ast }>0\\mid \\mathbf {X} _{i})\\\\[5pt]&=\\Pr({\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}+\\varepsilon _{i}>0)\\\\[5pt]&=\\Pr(\\varepsilon _{i}>-{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i})\\\\[5pt]&=\\Pr(\\varepsilon _{i}<{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i})&&{\\text{(because the logistic distribution is symmetric)}}\\\\[5pt]&=\\operatorname {logit} ^{-1}({\\boldsymbol {\\beta }}\\cdot \\mathbf {X}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0299", "text": "^{-1}({\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i})&\\\\[5pt]&=p_{i}&&{\\text{(see above)}}\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0300", "text": "This formulation\u2014which is standard in discrete choice models\u2014makes clear the relationship between logistic regression (the \"logit model\") and the probit model, which uses an error variable distributed according to a standard normal distribution instead of a standard logistic distribution.  Both the logistic and normal distributions are symmetric with a basic unimodal, \"bell curve\" shape", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0301", "text": ".  The only difference is that the logistic distribution has somewhat heavier tails, which means that it is less sensitive to outlying data (and hence somewhat more robust to model mis-specifications or erroneous data).", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0302", "text": "Two-way latent-variable model\nYet another formulation uses two separate latent variables:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0303", "text": "Y\n                  \n                    i\n                  \n                  \n                    0\n                    \u2217\n                  \n                \n              \n              \n                \n                =\n                \n                  \n                    \u03b2\n                  \n                  \n                    0\n                  \n                \n                \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0304", "text": "\u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                +\n                \n                  \u03b5\n                  \n                    0\n                  \n                \n                \n              \n            \n            \n              \n                \n                  Y\n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0305", "text": "i\n                  \n                  \n                    1\n                    \u2217\n                  \n                \n              \n              \n                \n                =\n                \n                  \n                    \u03b2\n                  \n                  \n                    1\n                  \n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0306", "text": "i\n                  \n                \n                +\n                \n                  \u03b5\n                  \n                    1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0307", "text": "{\\displaystyle {\\begin{aligned}Y_{i}^{0\\ast }&={\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}+\\varepsilon _{0}\\,\\\\Y_{i}^{1\\ast }&={\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}+\\varepsilon _{1}\\,\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0308", "text": "where", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0309", "text": "\u03b5\n                  \n                    0\n                  \n                \n              \n              \n                \n                \u223c\n                \n                  EV\n                  \n                    1\n                  \n                \n                \u2061\n                (\n                0\n                ,\n                1\n                )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0310", "text": ")\n              \n            \n            \n              \n                \n                  \u03b5\n                  \n                    1\n                  \n                \n              \n              \n                \n                \u223c\n                \n                  EV\n                  \n                    1\n                  \n                \n                \u2061\n                (\n                0\n                ,\n                1\n                )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0311", "text": ")\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}\\varepsilon _{0}&\\sim \\operatorname {EV} _{1}(0,1)\\\\\\varepsilon _{1}&\\sim \\operatorname {EV} _{1}(0,1)\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0312", "text": "where EV1(0,1) is a standard type-1 extreme value distribution: i.e.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0313", "text": "Pr\n        (\n        \n          \u03b5\n          \n            0\n          \n        \n        =\n        x\n        )\n        =\n        Pr\n        (\n        \n          \u03b5\n          \n            1\n          \n        \n        =\n        x\n        )\n        =\n        \n          e\n          \n            \u2212\n            x\n          \n        \n        \n          e\n          \n            \u2212\n            \n              e\n              \n                \u2212\n                x", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0314", "text": "x\n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\Pr(\\varepsilon _{0}=x)=\\Pr(\\varepsilon _{1}=x)=e^{-x}e^{-e^{-x}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0315", "text": "Then", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0316", "text": "Y\n          \n            i\n          \n        \n        =\n        \n          \n            {\n            \n              \n                \n                  1\n                \n                \n                  \n                    if \n                  \n                  \n                    Y\n                    \n                      i\n                    \n                    \n                      1\n                      \u2217", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0317", "text": "\u2217\n                    \n                  \n                  >\n                  \n                    Y\n                    \n                      i\n                    \n                    \n                      0\n                      \u2217\n                    \n                  \n                  ,\n                \n              \n              \n                \n                  0\n                \n                \n                  \n                    otherwise.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0318", "text": "otherwise.\n                  \n                \n              \n            \n            \n          \n        \n      \n    \n    {\\displaystyle Y_{i}={\\begin{cases}1&{\\text{if }}Y_{i}^{1\\ast }>Y_{i}^{0\\ast },\\\\0&{\\text{otherwise.}}\\end{cases}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0319", "text": "This model has a separate latent variable and a separate set of regression coefficients for each possible outcome of the dependent variable.  The reason for this separation is that it makes it easy to extend logistic regression to multi-outcome categorical variables, as in the multinomial logit model. In such a model, it is natural to model each possible outcome using a different set of regression coefficients", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0320", "text": ".  It is also possible to motivate each of the separate latent variables as the theoretical utility associated with making the associated choice, and thus motivate logistic regression in terms of utility theory", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0321", "text": ". (In terms of utility theory, a rational actor always chooses the choice with the greatest associated utility.) This is the approach taken by economists when formulating discrete choice models, because it both provides a theoretically strong foundation and facilitates intuitions about the model, which in turn makes it easy to consider various sorts of extensions. (See the example below.)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0322", "text": "The choice of the type-1 extreme value distribution seems fairly arbitrary, but it makes the mathematics work out, and it may be possible to justify its use through rational choice theory.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0323", "text": "It turns out that this model is equivalent to the previous model, although this seems non-obvious, since there are now two sets of regression coefficients and error variables, and the error variables have a different distribution.  In fact, this model reduces directly to the previous one with the following substitutions:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0324", "text": "\u03b2\n        \n        =\n        \n          \n            \u03b2\n          \n          \n            1\n          \n        \n        \u2212\n        \n          \n            \u03b2\n          \n          \n            0\n          \n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}={\\boldsymbol {\\beta }}_{1}-{\\boldsymbol {\\beta }}_{0}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0325", "text": "\u03b5\n        =\n        \n          \u03b5\n          \n            1\n          \n        \n        \u2212\n        \n          \u03b5\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\varepsilon =\\varepsilon _{1}-\\varepsilon _{0}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0326", "text": "An intuition for this comes from the fact that, since we choose based on the maximum of two values, only their difference matters, not the exact values \u2014 and this effectively removes one degree of freedom. Another critical fact is that the difference of two type-1 extreme-value-distributed variables is a logistic distribution, i.e. \n  \n    \n      \n        \u03b5\n        =\n        \n          \u03b5\n          \n            1\n          \n        \n        \u2212\n        \n          \u03b5\n          \n            0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0327", "text": "\u03b5\n          \n            0\n          \n        \n        \u223c\n        Logistic\n        \u2061\n        (\n        0\n        ,\n        1\n        )\n        .\n      \n    \n    {\\displaystyle \\varepsilon =\\varepsilon _{1}-\\varepsilon _{0}\\sim \\operatorname {Logistic} (0,1).}\n  \n We can demonstrate the equivalent as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0328", "text": "Pr\n                (\n                \n                  Y\n                  \n                    i\n                  \n                \n                =\n                1\n                \u2223\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                )\n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0329", "text": "Pr\n                \n                  (\n                  \n                    \n                      Y\n                      \n                        i\n                      \n                      \n                        1\n                        \u2217\n                      \n                    \n                    >\n                    \n                      Y\n                      \n                        i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0330", "text": "i\n                      \n                      \n                        0\n                        \u2217\n                      \n                    \n                    \u2223\n                    \n                      \n                        X\n                      \n                      \n                        i\n                      \n                    \n                  \n                  )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0331", "text": "=", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0332", "text": "Pr\n                \n                  (\n                  \n                    \n                      Y\n                      \n                        i\n                      \n                      \n                        1\n                        \u2217\n                      \n                    \n                    \u2212\n                    \n                      Y\n                      \n                        i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0333", "text": "i\n                      \n                      \n                        0\n                        \u2217\n                      \n                    \n                    >\n                    0\n                    \u2223\n                    \n                      \n                        X\n                      \n                      \n                        i\n                      \n                    \n                  \n                  )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0334", "text": ")\n                \n              \n              \n            \n            \n              \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0335", "text": "Pr\n                \n                  (\n                  \n                    \n                      \n                        \u03b2\n                      \n                      \n                        1\n                      \n                    \n                    \u22c5\n                    \n                      \n                        X\n                      \n                      \n                        i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0336", "text": "i\n                      \n                    \n                    +\n                    \n                      \u03b5\n                      \n                        1\n                      \n                    \n                    \u2212\n                    \n                      (\n                      \n                        \n                          \n                            \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0337", "text": "0\n                          \n                        \n                        \u22c5\n                        \n                          \n                            X\n                          \n                          \n                            i\n                          \n                        \n                        +\n                        \n                          \u03b5\n                          \n                            0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0338", "text": "0\n                          \n                        \n                      \n                      )\n                    \n                    >\n                    0\n                  \n                  )\n                \n              \n              \n            \n            \n              \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0339", "text": "Pr\n                \n                  (\n                  \n                    (\n                    \n                      \n                        \u03b2\n                      \n                      \n                        1\n                      \n                    \n                    \u22c5\n                    \n                      \n                        X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0340", "text": "i\n                      \n                    \n                    \u2212\n                    \n                      \n                        \u03b2\n                      \n                      \n                        0\n                      \n                    \n                    \u22c5\n                    \n                      \n                        X\n                      \n                      \n                        i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0341", "text": "i\n                      \n                    \n                    )\n                    +\n                    (\n                    \n                      \u03b5\n                      \n                        1\n                      \n                    \n                    \u2212\n                    \n                      \u03b5\n                      \n                        0\n                      \n                    \n                    )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0342", "text": ")\n                    >\n                    0\n                  \n                  )\n                \n              \n              \n            \n            \n              \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0343", "text": "Pr\n                (\n                (\n                \n                  \n                    \u03b2\n                  \n                  \n                    1\n                  \n                \n                \u2212\n                \n                  \n                    \u03b2\n                  \n                  \n                    0\n                  \n                \n                )\n                \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0344", "text": "\u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                +\n                (\n                \n                  \u03b5\n                  \n                    1\n                  \n                \n                \u2212\n                \n                  \u03b5\n                  \n                    0\n                  \n                \n                )\n                >", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0345", "text": ")\n                >\n                0\n                )\n              \n              \n            \n            \n              \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0346", "text": "Pr\n                (\n                (\n                \n                  \n                    \u03b2\n                  \n                  \n                    1\n                  \n                \n                \u2212\n                \n                  \n                    \u03b2\n                  \n                  \n                    0\n                  \n                \n                )\n                \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0347", "text": "\u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                +\n                \u03b5\n                >\n                0\n                )\n              \n              \n              \n                \n                  (substitute \n                \n                \u03b5\n                \n                   as above)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0348", "text": "=", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0349", "text": "Pr\n                (\n                \n                  \u03b2\n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                +\n                \u03b5\n                >\n                0\n                )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0350", "text": "(substitute \n                \n                \n                  \u03b2\n                \n                \n                   as above)\n                \n              \n            \n            \n              \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0351", "text": "Pr\n                (\n                \u03b5\n                >\n                \u2212\n                \n                  \u03b2\n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0352", "text": "(now, same as above model)\n                \n              \n            \n            \n              \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0353", "text": "Pr\n                (\n                \u03b5\n                <\n                \n                  \u03b2\n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                )\n              \n              \n            \n            \n              \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0354", "text": "logit\n                  \n                    \u2212\n                    1\n                  \n                \n                \u2061\n                (\n                \n                  \u03b2\n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0355", "text": ")\n              \n            \n            \n              \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0356", "text": "p\n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0357", "text": "{\\displaystyle {\\begin{aligned}\\Pr(Y_{i}=1\\mid \\mathbf {X} _{i})={}&\\Pr \\left(Y_{i}^{1\\ast }>Y_{i}^{0\\ast }\\mid \\mathbf {X} _{i}\\right)&\\\\[5pt]={}&\\Pr \\left(Y_{i}^{1\\ast }-Y_{i}^{0\\ast }>0\\mid \\mathbf {X} _{i}\\right)&\\\\[5pt]={}&\\Pr \\left({\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}+\\varepsilon _{1}-\\left({\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}+\\varepsilon _{0}\\right)>0\\right)&\\\\[5pt]={}&\\Pr \\left(({\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}-{\\boldsymbol {\\beta }}_{0}\\cdot", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0358", "text": "\\mathbf {X} _{i}-{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i})+(\\varepsilon _{1}-\\varepsilon _{0})>0\\right)&\\\\[5pt]={}&\\Pr(({\\boldsymbol {\\beta }}_{1}-{\\boldsymbol {\\beta }}_{0})\\cdot \\mathbf {X} _{i}+(\\varepsilon _{1}-\\varepsilon _{0})>0)&\\\\[5pt]={}&\\Pr(({\\boldsymbol {\\beta }}_{1}-{\\boldsymbol {\\beta }}_{0})\\cdot \\mathbf {X} _{i}+\\varepsilon >0)&&{\\text{(substitute }}\\varepsilon {\\text{ as above)}}\\\\[5pt]={}&\\Pr({\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}+\\varepsilon", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0359", "text": "{\\beta }}\\cdot \\mathbf {X} _{i}+\\varepsilon >0)&&{\\text{(substitute }}{\\boldsymbol {\\beta }}{\\text{ as above)}}\\\\[5pt]={}&\\Pr(\\varepsilon >-{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i})&&{\\text{(now, same as above model)}}\\\\[5pt]={}&\\Pr(\\varepsilon <{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i})&\\\\[5pt]={}&\\operatorname {logit} ^{-1}({\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i})\\\\[5pt]={}&p_{i}\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0360", "text": "As a \"log-linear\" model\nYet another formulation combines the two-way latent variable formulation above with the original formulation higher up without latent variables, and in the process provides a link to one of the standard formulations of the multinomial logit.\nHere, instead of writing the logit of the probabilities pi as a linear predictor, we separate the linear predictor into two, one for each of the two outcomes:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0361", "text": "ln\n                \u2061\n                Pr\n                (\n                \n                  Y\n                  \n                    i\n                  \n                \n                =\n                0\n                )\n              \n              \n                \n                =\n                \n                  \n                    \u03b2\n                  \n                  \n                    0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0362", "text": "0\n                  \n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i\n                  \n                \n                \u2212\n                ln\n                \u2061\n                Z\n              \n            \n            \n              \n                ln\n                \u2061\n                Pr\n                (\n                \n                  Y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0363", "text": "Y\n                  \n                    i\n                  \n                \n                =\n                1\n                )\n              \n              \n                \n                =\n                \n                  \n                    \u03b2\n                  \n                  \n                    1\n                  \n                \n                \u22c5\n                \n                  \n                    X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0364", "text": "X\n                  \n                  \n                    i\n                  \n                \n                \u2212\n                ln\n                \u2061\n                Z\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}\\ln \\Pr(Y_{i}=0)&={\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}-\\ln Z\\\\\\ln \\Pr(Y_{i}=1)&={\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}-\\ln Z\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0365", "text": "Two separate sets of regression coefficients have been introduced, just as in the two-way latent variable model, and the two equations appear a form that writes the logarithm of the associated probability as a linear predictor, with an extra term \n  \n    \n      \n        \u2212\n        ln\n        \u2061\n        Z\n      \n    \n    {\\displaystyle -\\ln Z}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0366", "text": "{\\displaystyle -\\ln Z}\n  \n at the end.  This term, as it turns out, serves as the normalizing factor ensuring that the result is a distribution.  This can be seen by exponentiating both sides:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0367", "text": "Pr\n                (\n                \n                  Y\n                  \n                    i\n                  \n                \n                =\n                0\n                )\n              \n              \n                \n                =\n                \n                  \n                    1\n                    Z\n                  \n                \n                \n                  e", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0368", "text": "e\n                  \n                    \n                      \n                        \u03b2\n                      \n                      \n                        0\n                      \n                    \n                    \u22c5\n                    \n                      \n                        X\n                      \n                      \n                        i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0369", "text": "Pr\n                (\n                \n                  Y\n                  \n                    i\n                  \n                \n                =\n                1\n                )\n              \n              \n                \n                =\n                \n                  \n                    1\n                    Z", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0370", "text": "e\n                  \n                    \n                      \n                        \u03b2\n                      \n                      \n                        1\n                      \n                    \n                    \u22c5\n                    \n                      \n                        X\n                      \n                      \n                        i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0371", "text": "{\\displaystyle {\\begin{aligned}\\Pr(Y_{i}=0)&={\\frac {1}{Z}}e^{{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}}\\\\[5pt]\\Pr(Y_{i}=1)&={\\frac {1}{Z}}e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0372", "text": "In this form it is clear that the purpose of Z is to ensure that the resulting distribution over Yi is in fact a probability distribution, i.e. it sums to 1.  This means that Z is simply the sum of all un-normalized probabilities, and by dividing each probability by Z, the probabilities become \"normalized\".  That is:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0373", "text": "Z\n        =\n        \n          e\n          \n            \n              \n                \u03b2\n              \n              \n                0\n              \n            \n            \u22c5\n            \n              \n                X\n              \n              \n                i\n              \n            \n          \n        \n        +\n        \n          e\n          \n            \n              \n                \u03b2\n              \n              \n                1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0374", "text": "1\n              \n            \n            \u22c5\n            \n              \n                X\n              \n              \n                i\n              \n            \n          \n        \n      \n    \n    {\\displaystyle Z=e^{{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}}+e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0375", "text": "and the resulting equations are", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0376", "text": "Pr\n                (\n                \n                  Y\n                  \n                    i\n                  \n                \n                =\n                0\n                )\n              \n              \n                \n                =\n                \n                  \n                    \n                      e", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0377", "text": "\u03b2\n                          \n                          \n                            0\n                          \n                        \n                        \u22c5\n                        \n                          \n                            X\n                          \n                          \n                            i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0378", "text": "e\n                        \n                          \n                            \n                              \u03b2\n                            \n                            \n                              0\n                            \n                          \n                          \u22c5\n                          \n                            \n                              X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0379", "text": "X\n                            \n                            \n                              i\n                            \n                          \n                        \n                      \n                      +\n                      \n                        e\n                        \n                          \n                            \n                              \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0380", "text": "1\n                            \n                          \n                          \u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0381", "text": "Pr\n                (\n                \n                  Y\n                  \n                    i\n                  \n                \n                =\n                1\n                )\n              \n              \n                \n                =\n                \n                  \n                    \n                      e", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0382", "text": "\u03b2\n                          \n                          \n                            1\n                          \n                        \n                        \u22c5\n                        \n                          \n                            X\n                          \n                          \n                            i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0383", "text": "e\n                        \n                          \n                            \n                              \u03b2\n                            \n                            \n                              0\n                            \n                          \n                          \u22c5\n                          \n                            \n                              X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0384", "text": "X\n                            \n                            \n                              i\n                            \n                          \n                        \n                      \n                      +\n                      \n                        e\n                        \n                          \n                            \n                              \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0385", "text": "1\n                            \n                          \n                          \u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0386", "text": ".", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0387", "text": "{\\displaystyle {\\begin{aligned}\\Pr(Y_{i}=0)&={\\frac {e^{{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}}}{e^{{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}}+e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}}}\\\\[5pt]\\Pr(Y_{i}=1)&={\\frac {e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}}{e^{{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}}+e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}}}.\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0388", "text": "Or generally:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0389", "text": "Pr\n        (\n        \n          Y\n          \n            i\n          \n        \n        =\n        c\n        )\n        =\n        \n          \n            \n              e\n              \n                \n                  \n                    \u03b2\n                  \n                  \n                    c\n                  \n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0390", "text": "i\n                  \n                \n              \n            \n            \n              \n                \u2211\n                \n                  h\n                \n              \n              \n                e\n                \n                  \n                    \n                      \u03b2\n                    \n                    \n                      h\n                    \n                  \n                  \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0391", "text": "X\n                    \n                    \n                      i\n                    \n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\Pr(Y_{i}=c)={\\frac {e^{{\\boldsymbol {\\beta }}_{c}\\cdot \\mathbf {X} _{i}}}{\\sum _{h}e^{{\\boldsymbol {\\beta }}_{h}\\cdot \\mathbf {X} _{i}}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0392", "text": "This shows clearly how to generalize this formulation to more than two outcomes, as in multinomial logit.\nThis general formulation is exactly the softmax function as in", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0393", "text": "Pr\n        (\n        \n          Y\n          \n            i\n          \n        \n        =\n        c\n        )\n        =\n        softmax\n        \u2061\n        (\n        c\n        ,\n        \n          \n            \u03b2\n          \n          \n            0\n          \n        \n        \u22c5\n        \n          \n            X\n          \n          \n            i\n          \n        \n        ,\n        \n          \n            \u03b2\n          \n          \n            1\n          \n        \n        \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0394", "text": "1\n          \n        \n        \u22c5\n        \n          \n            X\n          \n          \n            i\n          \n        \n        ,\n        \u2026\n        )\n        .\n      \n    \n    {\\displaystyle \\Pr(Y_{i}=c)=\\operatorname {softmax} (c,{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i},{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i},\\dots ).}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0395", "text": "To prove that this is equivalent to the previous model, we start by recognizing the above model is overspecified, in that \n  \n    \n      \n        Pr\n        (\n        \n          Y\n          \n            i\n          \n        \n        =\n        0\n        )\n      \n    \n    {\\displaystyle \\Pr(Y_{i}=0)}\n  \n and \n  \n    \n      \n        Pr\n        (\n        \n          Y\n          \n            i\n          \n        \n        =\n        1\n        )\n      \n    \n    {\\displaystyle \\Pr(Y_{i}=1)}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0396", "text": "{\\displaystyle \\Pr(Y_{i}=1)}\n  \n cannot be independently specified: rather \n  \n    \n      \n        Pr\n        (\n        \n          Y\n          \n            i\n          \n        \n        =\n        0\n        )\n        +\n        Pr\n        (\n        \n          Y\n          \n            i\n          \n        \n        =\n        1\n        )\n        =\n        1\n      \n    \n    {\\displaystyle \\Pr(Y_{i}=0)+\\Pr(Y_{i}=1)=1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0397", "text": "so knowing one automatically determines the other.  As a result, the model is nonidentifiable, in that multiple combinations of \n  \n    \n      \n        \n          \n            \u03b2\n          \n          \n            0\n          \n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}_{0}}\n  \n and \n  \n    \n      \n        \n          \n            \u03b2\n          \n          \n            1\n          \n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}_{1}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0398", "text": "{\\displaystyle {\\boldsymbol {\\beta }}_{1}}\n  \n will produce the same probabilities for all possible explanatory variables.  In fact, it can be seen that adding any constant vector to both of them will produce the same probabilities:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0399", "text": "Pr\n                (\n                \n                  Y\n                  \n                    i\n                  \n                \n                =\n                1\n                )\n              \n              \n                \n                =\n                \n                  \n                    \n                      e\n                      \n                        (", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0400", "text": "\u03b2\n                          \n                          \n                            1\n                          \n                        \n                        +\n                        \n                          C\n                        \n                        )\n                        \u22c5\n                        \n                          \n                            X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0401", "text": "i\n                          \n                        \n                      \n                    \n                    \n                      \n                        e\n                        \n                          (\n                          \n                            \n                              \u03b2\n                            \n                            \n                              0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0402", "text": "0\n                            \n                          \n                          +\n                          \n                            C\n                          \n                          )\n                          \u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0403", "text": "+\n                      \n                        e\n                        \n                          (\n                          \n                            \n                              \u03b2\n                            \n                            \n                              1\n                            \n                          \n                          +", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0404", "text": "C\n                          \n                          )\n                          \u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0405", "text": "=\n                \n                  \n                    \n                      \n                        e\n                        \n                          \n                            \n                              \u03b2\n                            \n                            \n                              1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0406", "text": "\u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i\n                            \n                          \n                        \n                      \n                      \n                        e\n                        \n                          \n                            C", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0407", "text": "\u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i\n                            \n                          \n                        \n                      \n                    \n                    \n                      \n                        e", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0408", "text": "\u03b2\n                            \n                            \n                              0\n                            \n                          \n                          \u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0409", "text": "e\n                        \n                          \n                            C\n                          \n                          \u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0410", "text": "+\n                      \n                        e\n                        \n                          \n                            \n                              \u03b2\n                            \n                            \n                              1\n                            \n                          \n                          \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0411", "text": "X\n                            \n                            \n                              i\n                            \n                          \n                        \n                      \n                      \n                        e\n                        \n                          \n                            C\n                          \n                          \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0412", "text": "X\n                            \n                            \n                              i\n                            \n                          \n                        \n                      \n                    \n                  \n                \n              \n            \n            \n              \n              \n                \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0413", "text": "e\n                        \n                          \n                            C\n                          \n                          \u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0414", "text": "e\n                        \n                          \n                            \n                              \u03b2\n                            \n                            \n                              1\n                            \n                          \n                          \u22c5\n                          \n                            \n                              X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0415", "text": "X\n                            \n                            \n                              i\n                            \n                          \n                        \n                      \n                    \n                    \n                      \n                        e\n                        \n                          \n                            C\n                          \n                          \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0416", "text": "X\n                            \n                            \n                              i\n                            \n                          \n                        \n                      \n                      (\n                      \n                        e\n                        \n                          \n                            \n                              \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0417", "text": "0\n                            \n                          \n                          \u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i\n                            \n                          \n                        \n                      \n                      +", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0418", "text": "+\n                      \n                        e\n                        \n                          \n                            \n                              \u03b2\n                            \n                            \n                              1\n                            \n                          \n                          \u22c5\n                          \n                            \n                              X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0419", "text": "i\n                            \n                          \n                        \n                      \n                      )\n                    \n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    \n                      e", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0420", "text": "e\n                      \n                        \n                          \n                            \u03b2\n                          \n                          \n                            1\n                          \n                        \n                        \u22c5\n                        \n                          \n                            X\n                          \n                          \n                            i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0421", "text": "e\n                        \n                          \n                            \n                              \u03b2\n                            \n                            \n                              0\n                            \n                          \n                          \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0422", "text": "X\n                            \n                            \n                              i\n                            \n                          \n                        \n                      \n                      +\n                      \n                        e\n                        \n                          \n                            \n                              \u03b2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0423", "text": "1\n                            \n                          \n                          \u22c5\n                          \n                            \n                              X\n                            \n                            \n                              i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0424", "text": ".", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0425", "text": "{\\displaystyle {\\begin{aligned}\\Pr(Y_{i}=1)&={\\frac {e^{({\\boldsymbol {\\beta }}_{1}+\\mathbf {C} )\\cdot \\mathbf {X} _{i}}}{e^{({\\boldsymbol {\\beta }}_{0}+\\mathbf {C} )\\cdot \\mathbf {X} _{i}}+e^{({\\boldsymbol {\\beta }}_{1}+\\mathbf {C} )\\cdot \\mathbf {X} _{i}}}}\\\\[5pt]&={\\frac {e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}e^{\\mathbf {C} \\cdot \\mathbf {X} _{i}}}{e^{{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}}e^{\\mathbf {C} \\cdot \\mathbf {X} _{i}}+e^{{\\boldsymbol {\\beta }}_{1}\\cdot", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0426", "text": "{X} _{i}}+e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}e^{\\mathbf {C} \\cdot \\mathbf {X} _{i}}}}\\\\[5pt]&={\\frac {e^{\\mathbf {C} \\cdot \\mathbf {X} _{i}}e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}}{e^{\\mathbf {C} \\cdot \\mathbf {X} _{i}}(e^{{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}}+e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}})}}\\\\[5pt]&={\\frac {e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}}{e^{{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}}+e^{{\\boldsymbol", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0427", "text": "}}_{0}\\cdot \\mathbf {X} _{i}}+e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}}}.\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0428", "text": "As a result, we can simplify matters, and restore identifiability, by picking an arbitrary value for one of the two vectors.  We choose to set \n  \n    \n      \n        \n          \n            \u03b2\n          \n          \n            0\n          \n        \n        =\n        \n          0\n        \n        .\n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}_{0}=\\mathbf {0} .}\n  \n  Then,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0429", "text": "e\n          \n            \n              \n                \u03b2\n              \n              \n                0\n              \n            \n            \u22c5\n            \n              \n                X\n              \n              \n                i\n              \n            \n          \n        \n        =\n        \n          e\n          \n            \n              0\n            \n            \u22c5\n            \n              \n                X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0430", "text": "X\n              \n              \n                i\n              \n            \n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle e^{{\\boldsymbol {\\beta }}_{0}\\cdot \\mathbf {X} _{i}}=e^{\\mathbf {0} \\cdot \\mathbf {X} _{i}}=1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0431", "text": "and so", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0432", "text": "Pr\n        (\n        \n          Y\n          \n            i\n          \n        \n        =\n        1\n        )\n        =\n        \n          \n            \n              e\n              \n                \n                  \n                    \u03b2\n                  \n                  \n                    1\n                  \n                \n                \u22c5\n                \n                  \n                    X\n                  \n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0433", "text": "i\n                  \n                \n              \n            \n            \n              1\n              +\n              \n                e\n                \n                  \n                    \n                      \u03b2\n                    \n                    \n                      1\n                    \n                  \n                  \u22c5\n                  \n                    \n                      X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0434", "text": "i\n                    \n                  \n                \n              \n            \n          \n        \n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  \n                    \n                      \u03b2\n                    \n                    \n                      1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0435", "text": "\u22c5\n                  \n                    \n                      X\n                    \n                    \n                      i\n                    \n                  \n                \n              \n            \n          \n        \n        =\n        \n          p\n          \n            i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0436", "text": "i\n          \n        \n      \n    \n    {\\displaystyle \\Pr(Y_{i}=1)={\\frac {e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}}{1+e^{{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}}}={\\frac {1}{1+e^{-{\\boldsymbol {\\beta }}_{1}\\cdot \\mathbf {X} _{i}}}}=p_{i}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0437", "text": "which shows that this formulation is indeed equivalent to the previous formulation. (As in the two-way latent variable formulation, any settings where \n  \n    \n      \n        \n          \u03b2\n        \n        =\n        \n          \n            \u03b2\n          \n          \n            1\n          \n        \n        \u2212\n        \n          \n            \u03b2\n          \n          \n            0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0438", "text": "0\n          \n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}={\\boldsymbol {\\beta }}_{1}-{\\boldsymbol {\\beta }}_{0}}\n  \n will produce equivalent results.)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0439", "text": "Most treatments of the multinomial logit model start out either by extending the \"log-linear\" formulation presented here or the two-way latent variable formulation presented above, since both clearly show the way that the model could be extended to multi-way outcomes", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0440", "text": ".  In general, the presentation with latent variables is more common in econometrics and political science, where discrete choice models and utility theory reign, while the \"log-linear\" formulation here is more common in computer science, e.g. machine learning and natural language processing.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0441", "text": "As a single-layer perceptron\nThe model has an equivalent formulation", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0442", "text": "p\n          \n            i\n          \n        \n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  (\n                  \n                    \u03b2\n                    \n                      0\n                    \n                  \n                  +\n                  \n                    \u03b2\n                    \n                      1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0443", "text": "1\n                    \n                  \n                  \n                    x\n                    \n                      1\n                      ,\n                      i\n                    \n                  \n                  +\n                  \u22ef\n                  +\n                  \n                    \u03b2\n                    \n                      k\n                    \n                  \n                  \n                    x", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0444", "text": "x\n                    \n                      k\n                      ,\n                      i\n                    \n                  \n                  )\n                \n              \n            \n          \n        \n        .\n        \n      \n    \n    {\\displaystyle p_{i}={\\frac {1}{1+e^{-(\\beta _{0}+\\beta _{1}x_{1,i}+\\cdots +\\beta _{k}x_{k,i})}}}.\\,}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0445", "text": "This functional form is commonly called a single-layer perceptron or single-layer artificial neural network. A single-layer neural network computes a continuous output instead of a step function. The derivative of pi with respect to  X = (x1, ..., xk) is computed from the general form:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0446", "text": "y\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  f\n                  (\n                  X\n                  )\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle y={\\frac {1}{1+e^{-f(X)}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0447", "text": "where f(X) is an analytic function in X. With this choice, the single-layer neural network is identical to the logistic regression model. This function has a continuous derivative, which allows it to be used in backpropagation. This function is also preferred because its derivative is easily calculated:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0448", "text": "d\n              \n              y\n            \n            \n              \n                d\n              \n              X\n            \n          \n        \n        =\n        y\n        (\n        1\n        \u2212\n        y\n        )\n        \n          \n            \n              \n                d\n              \n              f\n            \n            \n              \n                d\n              \n              X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0449", "text": "d\n              \n              X\n            \n          \n        \n        .\n        \n      \n    \n    {\\displaystyle {\\frac {\\mathrm {d} y}{\\mathrm {d} X}}=y(1-y){\\frac {\\mathrm {d} f}{\\mathrm {d} X}}.\\,}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0450", "text": "In terms of binomial data\nA closely related model assumes that each i is associated not with a single Bernoulli trial but with ni independent identically distributed trials, where the observation Yi is the number of successes observed (the sum of the individual Bernoulli-distributed random variables), and hence follows a binomial distribution:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0451", "text": "Y\n          \n            i\n          \n        \n        \n        \u223c\n        Bin\n        \u2061\n        (\n        \n          n\n          \n            i\n          \n        \n        ,\n        \n          p\n          \n            i\n          \n        \n        )\n        ,\n        \n           for \n        \n        i\n        =\n        1\n        ,\n        \u2026\n        ,\n        n", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0452", "text": "\u2026\n        ,\n        n\n      \n    \n    {\\displaystyle Y_{i}\\,\\sim \\operatorname {Bin} (n_{i},p_{i}),{\\text{ for }}i=1,\\dots ,n}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0453", "text": "An example of this distribution is the fraction of seeds (pi) that germinate after ni are planted.\nIn terms of expected values, this model is expressed as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0454", "text": "p\n          \n            i\n          \n        \n        =\n        \n          \n            E\n          \n        \n        \u2061\n        \n          [\n          \n            \n              \n              \n                \n                  \n                    \n                      Y\n                      \n                        i\n                      \n                    \n                    \n                      n\n                      \n                        i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0455", "text": "i\n                      \n                    \n                  \n                \n                \n              \n              |\n            \n            \n            \n              \n                X\n              \n              \n                i\n              \n            \n          \n          ]\n        \n        \n        ,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0456", "text": ",\n      \n    \n    {\\displaystyle p_{i}=\\operatorname {\\mathbb {E} } \\left[\\left.{\\frac {Y_{i}}{n_{i}}}\\,\\right|\\,\\mathbf {X} _{i}\\right]\\,,}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0457", "text": "so that", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0458", "text": "logit\n        \u2061\n        \n          (\n          \n            \n              \n                E\n              \n            \n            \u2061\n            \n              [\n              \n                \n                  \n                  \n                    \n                      \n                        \n                          Y\n                          \n                            i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0459", "text": "n\n                          \n                            i\n                          \n                        \n                      \n                    \n                    \n                  \n                  |\n                \n                \n                \n                  \n                    X\n                  \n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0460", "text": "]\n            \n          \n          )\n        \n        =\n        logit\n        \u2061\n        (\n        \n          p\n          \n            i\n          \n        \n        )\n        =\n        ln\n        \u2061\n        \n          (\n          \n            \n              \n                p\n                \n                  i\n                \n              \n              \n                1\n                \u2212\n                \n                  p", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0461", "text": "p\n                  \n                    i\n                  \n                \n              \n            \n          \n          )\n        \n        =\n        \n          \u03b2\n        \n        \u22c5\n        \n          \n            X\n          \n          \n            i\n          \n        \n        \n        ,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0462", "text": ",\n      \n    \n    {\\displaystyle \\operatorname {logit} \\left(\\operatorname {\\mathbb {E} } \\left[\\left.{\\frac {Y_{i}}{n_{i}}}\\,\\right|\\,\\mathbf {X} _{i}\\right]\\right)=\\operatorname {logit} (p_{i})=\\ln \\left({\\frac {p_{i}}{1-p_{i}}}\\right)={\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}\\,,}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0463", "text": "Or equivalently:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0464", "text": "Pr\n        (\n        \n          Y\n          \n            i\n          \n        \n        =\n        y\n        \u2223\n        \n          \n            X\n          \n          \n            i\n          \n        \n        )\n        =\n        \n          \n            \n              (\n            \n            \n              \n                n\n                \n                  i\n                \n              \n              y\n            \n            \n              )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0465", "text": ")\n            \n          \n        \n        \n          p\n          \n            i\n          \n          \n            y\n          \n        \n        (\n        1\n        \u2212\n        \n          p\n          \n            i\n          \n        \n        \n          )\n          \n            \n              n\n              \n                i\n              \n            \n            \u2212\n            y\n          \n        \n        =\n        \n          \n            \n              (", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0466", "text": "(\n            \n            \n              \n                n\n                \n                  i\n                \n              \n              y\n            \n            \n              )\n            \n          \n        \n        \n          \n            (\n            \n              \n                1\n                \n                  1\n                  +\n                  \n                    e\n                    \n                      \u2212", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0467", "text": "\u2212\n                      \n                        \u03b2\n                      \n                      \u22c5\n                      \n                        \n                          X\n                        \n                        \n                          i\n                        \n                      \n                    \n                  \n                \n              \n            \n            )\n          \n          \n            y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0468", "text": "y\n          \n        \n        \n          \n            (\n            \n              1\n              \u2212\n              \n                \n                  1\n                  \n                    1\n                    +\n                    \n                      e\n                      \n                        \u2212\n                        \n                          \u03b2\n                        \n                        \u22c5", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0469", "text": "X\n                          \n                          \n                            i\n                          \n                        \n                      \n                    \n                  \n                \n              \n            \n            )\n          \n          \n            \n              n\n              \n                i\n              \n            \n            \u2212\n            y\n          \n        \n        \n        .", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0470", "text": ".\n      \n    \n    {\\displaystyle \\Pr(Y_{i}=y\\mid \\mathbf {X} _{i})={n_{i} \\choose y}p_{i}^{y}(1-p_{i})^{n_{i}-y}={n_{i} \\choose y}\\left({\\frac {1}{1+e^{-{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}}}}\\right)^{y}\\left(1-{\\frac {1}{1+e^{-{\\boldsymbol {\\beta }}\\cdot \\mathbf {X} _{i}}}}\\right)^{n_{i}-y}\\,.}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0471", "text": "This model can be fit using the same sorts of methods as the above more basic model.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0472", "text": "Model fitting\nMaximum likelihood estimation (MLE)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0473", "text": "The regression coefficients are usually estimated using maximum likelihood estimation. Unlike linear regression with normally distributed residuals, it is not possible to find a closed-form expression for the coefficient values that maximize the likelihood function so an iterative process must be used instead; for example Newton's method", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0474", "text": ". This process begins with a tentative solution, revises it slightly to see if it can be improved, and repeats this revision until no more improvement is made, at which point the process is said to have converged.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0475", "text": "In some instances, the model may not reach convergence. Non-convergence of a model indicates that the coefficients are not meaningful because the iterative process was unable to find appropriate solutions. A failure to converge may occur for a number of reasons: having a large ratio of predictors to cases, multicollinearity, sparseness, or complete separation.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0476", "text": "Having a large ratio of variables to cases results in an overly conservative Wald statistic (discussed below) and can lead to non-convergence. Regularized logistic regression is specifically intended to be used in this situation.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0477", "text": "Multicollinearity refers to unacceptably high correlations between predictors. As multicollinearity increases, coefficients remain unbiased but standard errors increase and the likelihood of model convergence decreases. To detect multicollinearity amongst the predictors, one can conduct a linear regression analysis with the predictors of interest for the sole purpose of examining the tolerance statistic   used to assess whether multicollinearity is unacceptably high.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0478", "text": "Sparseness in the data refers to having a large proportion of empty cells (cells with zero counts). Zero cell counts are particularly problematic with categorical predictors. With continuous predictors, the model can infer values for the zero cell counts, but this is not the case with categorical predictors", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0479", "text": ". The model will not converge with zero cell counts for categorical predictors because the natural logarithm of zero is an undefined value so that the final solution to the model cannot be reached. To remedy this problem, researchers may collapse categories in a theoretically meaningful way or add a constant to all cells.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0480", "text": "Another numerical problem that may lead to a lack of convergence is complete separation, which refers to the instance in which the predictors perfectly predict the criterion \u2013 all cases are accurately classified and the likelihood maximized with infinite coefficients. In such instances, one should re-examine the data, as there may be some kind of error.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0481", "text": "One can also take semi-parametric or non-parametric approaches, e.g., via local-likelihood or nonparametric quasi-likelihood methods, which avoid assumptions of a parametric form for the index function and is robust to the choice of the link function (e.g., probit or logit).", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0482", "text": "Iteratively reweighted least squares (IRLS)\nBinary logistic regression (\n  \n    \n      \n        y\n        =\n        0\n      \n    \n    {\\displaystyle y=0}\n  \n or \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0483", "text": "1\n      \n    \n    {\\displaystyle y=1}\n  \n) can, for example, be calculated using iteratively reweighted least squares (IRLS), which is equivalent to maximizing the log-likelihood of a Bernoulli distributed  process using Newton's method. If the problem is written in vector matrix form, with parameters \n  \n    \n      \n        \n          \n            w\n          \n          \n            T\n          \n        \n        =\n        [\n        \n          \u03b2\n          \n            0", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0484", "text": "\u03b2\n          \n            0\n          \n        \n        ,\n        \n          \u03b2\n          \n            1\n          \n        \n        ,\n        \n          \u03b2\n          \n            2\n          \n        \n        ,\n        \u2026\n        ]\n      \n    \n    {\\displaystyle \\mathbf {w} ^{T}=[\\beta _{0},\\beta _{1},\\beta _{2},\\ldots ]}\n  \n, explanatory variables \n  \n    \n      \n        \n          x\n        \n        (\n        i\n        )\n        =\n        [\n        1\n        ,\n        \n          x", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0485", "text": "1\n        ,\n        \n          x\n          \n            1\n          \n        \n        (\n        i\n        )\n        ,\n        \n          x\n          \n            2\n          \n        \n        (\n        i\n        )\n        ,\n        \u2026\n        \n          ]\n          \n            T\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} (i)=[1,x_{1}(i),x_{2}(i),\\ldots ]^{T}}\n  \n and expected value of the Bernoulli distribution \n  \n    \n      \n        \u03bc\n        (\n        i\n        )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0486", "text": "\u03bc\n        (\n        i\n        )\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  \n                    \n                      w\n                    \n                    \n                      T\n                    \n                  \n                  \n                    x\n                  \n                  (\n                  i\n                  )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0487", "text": "i\n                  )\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\mu (i)={\\frac {1}{1+e^{-\\mathbf {w} ^{T}\\mathbf {x} (i)}}}}\n  \n, the parameters \n  \n    \n      \n        \n          w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n can be found using the following iterative algorithm:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0488", "text": "w\n          \n          \n            k\n            +\n            1\n          \n        \n        =\n        \n          \n            (\n            \n              \n                \n                  X\n                \n                \n                  T\n                \n              \n              \n                \n                  S\n                \n                \n                  k\n                \n              \n              \n                X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0489", "text": "X\n              \n            \n            )\n          \n          \n            \u2212\n            1\n          \n        \n        \n          \n            X\n          \n          \n            T\n          \n        \n        \n          (\n          \n            \n              \n                S\n              \n              \n                k\n              \n            \n            \n              X\n            \n            \n              \n                w", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0490", "text": "w\n              \n              \n                k\n              \n            \n            +\n            \n              y\n            \n            \u2212\n            \n              \n                \u03bc\n              \n              \n                k\n              \n            \n          \n          )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0491", "text": ")\n        \n      \n    \n    {\\displaystyle \\mathbf {w} _{k+1}=\\left(\\mathbf {X} ^{T}\\mathbf {S} _{k}\\mathbf {X} \\right)^{-1}\\mathbf {X} ^{T}\\left(\\mathbf {S} _{k}\\mathbf {X} \\mathbf {w} _{k}+\\mathbf {y} -\\mathbf {\\boldsymbol {\\mu }} _{k}\\right)}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0492", "text": "where \n  \n    \n      \n        \n          S\n        \n        =\n        diag\n        \u2061\n        (\n        \u03bc\n        (\n        i\n        )\n        (\n        1\n        \u2212\n        \u03bc\n        (\n        i\n        )\n        )\n        )\n      \n    \n    {\\displaystyle \\mathbf {S} =\\operatorname {diag} (\\mu (i)(1-\\mu (i)))}\n  \n is a diagonal weighting matrix, \n  \n    \n      \n        \n          \u03bc\n        \n        =\n        [\n        \u03bc\n        (\n        1\n        )\n        ,\n        \u03bc\n        (\n        2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0493", "text": ")\n        ,\n        \u03bc\n        (\n        2\n        )\n        ,\n        \u2026\n        ]\n      \n    \n    {\\displaystyle {\\boldsymbol {\\mu }}=[\\mu (1),\\mu (2),\\ldots ]}\n  \n the vector of expected values,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0494", "text": "X\n        \n        =\n        \n          \n            [\n            \n              \n                \n                  1\n                \n                \n                  \n                    x\n                    \n                      1\n                    \n                  \n                  (\n                  1\n                  )\n                \n                \n                  \n                    x\n                    \n                      2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0495", "text": "2\n                    \n                  \n                  (\n                  1\n                  )\n                \n                \n                  \u2026\n                \n              \n              \n                \n                  1\n                \n                \n                  \n                    x\n                    \n                      1\n                    \n                  \n                  (\n                  2", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0496", "text": "(\n                  2\n                  )\n                \n                \n                  \n                    x\n                    \n                      2\n                    \n                  \n                  (\n                  2\n                  )\n                \n                \n                  \u2026\n                \n              \n              \n                \n                  \u22ee\n                \n                \n                  \u22ee", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0497", "text": "\u22ee\n                \n                \n                  \u22ee\n                \n              \n            \n            ]\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {X} ={\\begin{bmatrix}1&x_{1}(1)&x_{2}(1)&\\ldots \\\\1&x_{1}(2)&x_{2}(2)&\\ldots \\\\\\vdots &\\vdots &\\vdots \\end{bmatrix}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0498", "text": "the regressor matrix and \n  \n    \n      \n        \n          y\n        \n        (\n        i\n        )\n        =\n        [\n        y\n        (\n        1\n        )\n        ,\n        y\n        (\n        2\n        )\n        ,\n        \u2026\n        \n          ]\n          \n            T\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {y} (i)=[y(1),y(2),\\ldots ]^{T}}\n  \n the vector of response variables. More details can be found in the literature.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0499", "text": "Bayesian", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0500", "text": "In a Bayesian statistics context, prior distributions are normally placed on the regression coefficients, for example in the form of Gaussian distributions.  There is no conjugate prior of the likelihood function in logistic regression.  When Bayesian inference was performed analytically, this made the posterior distribution difficult to calculate except in very low dimensions", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0501", "text": ".  Now, though, automatic software such as OpenBUGS, JAGS, PyMC, Stan or Turing.jl allows these posteriors to be computed using simulation, so lack of conjugacy is not a concern.  However, when the sample size or the number of parameters is large, full Bayesian simulation can be slow, and people often use approximate methods such as variational Bayesian methods and expectation propagation.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0502", "text": "\"Rule of ten\"\nWidely used, the \"one in ten rule\", states that logistic regression models give stable values for the explanatory variables if based on a minimum of about 10 events per explanatory variable (EPV); where event denotes the cases belonging to the less frequent category in the dependent variable. Thus a study designed to use \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0503", "text": "k\n      \n    \n    {\\displaystyle k}\n  \n explanatory variables for an event (e.g. myocardial infarction) expected to occur in a proportion \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n of participants in the study will require a total of \n  \n    \n      \n        10\n        k\n        \n          /\n        \n        p\n      \n    \n    {\\displaystyle 10k/p}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0504", "text": "participants. However, there is considerable debate about the reliability of this rule, which is based on simulation studies and lacks a secure theoretical underpinning", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0505", "text": ". According to some authors the rule is overly conservative in some circumstances, with the authors stating, \"If we (somewhat subjectively) regard confidence interval coverage less than 93 percent, type I error greater than 7 percent, or relative bias greater than 15 percent as problematic, our results indicate that problems are fairly frequent with 2\u20134 EPV, uncommon with 5\u20139 EPV, and still observed with 10\u201316 EPV", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0506", "text": ". The worst instances of each problem were not severe with 5\u20139 EPV and usually comparable to those with 10\u201316 EPV\".", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0507", "text": "Others have found results that are not consistent with the above, using different criteria.  A useful criterion is whether the fitted model will be expected to achieve the same predictive discrimination in a new sample as it appeared to achieve in the model development sample.  For that criterion, 20 events per candidate variable may be required", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0508", "text": ".  Also, one can argue that 96 observations are needed only to estimate the model's intercept precisely enough that the margin of error in predicted probabilities is \u00b10.1 with a 0.95 confidence level.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0509", "text": "Error and significance of fit\nDeviance and likelihood ratio test \u2500 a simple case", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0510", "text": "In any fitting procedure, the addition of another fitting parameter to a model (e.g. the beta parameters in a logistic regression model) will almost always improve the ability of the model to predict the measured outcomes. This will be true even if the additional term has no predictive value, since the model will simply be \"overfitting\" to the noise in the data", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0511", "text": ".  The question arises as to whether the improvement gained by the addition of another fitting parameter is significant enough to recommend the inclusion of the additional term, or whether the improvement is simply that which may be expected from overfitting.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0512", "text": "In short, for logistic regression, a statistic known as the deviance is defined which is a measure of the error between the logistic model fit and the outcome data. In the limit of a large number of data points, the deviance is chi-squared distributed, which allows a chi-squared test to be implemented in order to determine the significance of the explanatory variables.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0513", "text": "Linear regression and logistic regression have many similarities. For example, in simple linear regression, a set of K data points (xk, yk) are fitted to a proposed model function of the form \n  \n    \n      \n        y\n        =\n        \n          b\n          \n            0\n          \n        \n        +\n        \n          b\n          \n            1\n          \n        \n        x\n      \n    \n    {\\displaystyle y=b_{0}+b_{1}x}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0514", "text": "{\\displaystyle y=b_{0}+b_{1}x}\n  \n. The fit is obtained by choosing the b parameters which minimize the sum of the squares of the residuals (the squared error term) for each data point:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0515", "text": "\u03b5\n          \n            2\n          \n        \n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        (\n        \n          b\n          \n            0\n          \n        \n        +\n        \n          b\n          \n            1\n          \n        \n        \n          x\n          \n            k\n          \n        \n        \u2212\n        \n          y\n          \n            k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0516", "text": "y\n          \n            k\n          \n        \n        \n          )\n          \n            2\n          \n        \n        .\n      \n    \n    {\\displaystyle \\varepsilon ^{2}=\\sum _{k=1}^{K}(b_{0}+b_{1}x_{k}-y_{k})^{2}.}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0517", "text": "The minimum value which constitutes the fit will be denoted by \n  \n    \n      \n        \n          \n            \n              \n                \u03b5\n                ^\n              \n            \n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle {\\hat {\\varepsilon }}^{2}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0518", "text": "The idea of a null model may be introduced, in which it is assumed that the x variable is of no use in predicting the yk outcomes: The data points are fitted to a null model function of the form y = b0 with a squared error term:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0519", "text": "\u03b5\n          \n            2\n          \n        \n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        (\n        \n          b\n          \n            0\n          \n        \n        \u2212\n        \n          y\n          \n            k\n          \n        \n        \n          )\n          \n            2\n          \n        \n        .", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0520", "text": ".\n      \n    \n    {\\displaystyle \\varepsilon ^{2}=\\sum _{k=1}^{K}(b_{0}-y_{k})^{2}.}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0521", "text": "The fitting process consists of choosing a value of b0 which minimizes \n  \n    \n      \n        \n          \u03b5\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\varepsilon ^{2}}\n  \n of the fit to the null model, denoted by  \n  \n    \n      \n        \n          \u03b5\n          \n            \u03c6\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\varepsilon _{\\varphi }^{2}}\n  \n where the \n  \n    \n      \n        \u03c6", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0522", "text": "where the \n  \n    \n      \n        \u03c6\n      \n    \n    {\\displaystyle \\varphi }\n  \n subscript denotes the null model. It is seen that the null model is optimized by \n  \n    \n      \n        \n          b\n          \n            0\n          \n        \n        =\n        \n          \n            y\n            \u00af\n          \n        \n      \n    \n    {\\displaystyle b_{0}={\\overline {y}}}\n  \n where \n  \n    \n      \n        \n          \n            y\n            \u00af", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0523", "text": "\u00af\n          \n        \n      \n    \n    {\\displaystyle {\\overline {y}}}\n  \n is the mean of the yk values, and the optimized \n  \n    \n      \n        \n          \u03b5\n          \n            \u03c6\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\varepsilon _{\\varphi }^{2}}\n  \n is:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0524", "text": "\u03b5\n                ^\n              \n            \n          \n          \n            \u03c6\n          \n          \n            2\n          \n        \n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        (\n        \n          \n            y\n            \u00af\n          \n        \n        \u2212\n        \n          y\n          \n            k", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0525", "text": "y\n          \n            k\n          \n        \n        \n          )\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle {\\hat {\\varepsilon }}_{\\varphi }^{2}=\\sum _{k=1}^{K}({\\overline {y}}-y_{k})^{2}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0526", "text": "which is proportional to the square of the (uncorrected) sample standard deviation of the yk data points.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0527", "text": "We can imagine a case where the yk data points are randomly assigned to the various xk, and then fitted using the proposed model. Specifically, we can consider the fits of the proposed model to every permutation of the yk outcomes", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0528", "text": ". It can be shown that the optimized error of any of these fits will never be less than the optimum error of the null model, and that the difference between these minimum error will follow a chi-squared distribution, with degrees of freedom equal those of the proposed model minus those of the null model which, in this case, will be", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0529", "text": "2\n        \u2212\n        1\n        =\n        1\n      \n    \n    {\\displaystyle 2-1=1}\n  \n. Using the chi-squared test, we may then estimate how many of these permuted sets of yk will yield a minimum error less than or equal to the minimum  error using the original yk, and so we can estimate how significant an improvement is given by the inclusion of the x variable in the proposed model.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0530", "text": "For logistic regression, the measure of goodness-of-fit is the likelihood function L, or its logarithm, the log-likelihood \u2113. The likelihood function L is analogous to the \n  \n    \n      \n        \n          \u03b5\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\varepsilon ^{2}}\n  \n in the linear regression case, except that the likelihood is maximized rather than minimized. Denote the maximized log-likelihood of the proposed model by", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0531", "text": "\u2113\n              ^\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {\\ell }}}\n  \n.\nIn the case of simple binary logistic regression, the set of K data points are fitted in a probabilistic sense to a function of the form:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0532", "text": "p\n        (\n        x\n        )\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  t\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle p(x)={\\frac {1}{1+e^{-t}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0533", "text": "where \u2060\n  \n    \n      \n        p\n        (\n        x\n        )\n      \n    \n    {\\displaystyle p(x)}\n  \n\u2060 is the probability that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n. The log-odds are given by:\n\n  \n    \n      \n        t\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        x\n      \n    \n    {\\displaystyle t=\\beta _{0}+\\beta _{1}x}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0534", "text": "and the log-likelihood is:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0535", "text": "\u2113\n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        \n          (\n          \n            \n              y\n              \n                k\n              \n            \n            ln\n            \u2061\n            (\n            p\n            (\n            \n              x\n              \n                k\n              \n            \n            )\n            )\n            +", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0536", "text": ")\n            )\n            +\n            (\n            1\n            \u2212\n            \n              y\n              \n                k\n              \n            \n            )\n            ln\n            \u2061\n            (\n            1\n            \u2212\n            p\n            (\n            \n              x\n              \n                k\n              \n            \n            )\n            )\n          \n          )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0537", "text": ")\n        \n      \n    \n    {\\displaystyle \\ell =\\sum _{k=1}^{K}\\left(y_{k}\\ln(p(x_{k}))+(1-y_{k})\\ln(1-p(x_{k}))\\right)}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0538", "text": "For the null model, the probability that \n  \n    \n      \n        y\n        =\n        1\n      \n    \n    {\\displaystyle y=1}\n  \n is given by:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0539", "text": "p\n          \n            \u03c6\n          \n        \n        (\n        x\n        )\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  \n                    t\n                    \n                      \u03c6", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0540", "text": "{\\displaystyle p_{\\varphi }(x)={\\frac {1}{1+e^{-t_{\\varphi }}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0541", "text": "The log-odds for the null model are given by:\n\n  \n    \n      \n        \n          t\n          \n            \u03c6\n          \n        \n        =\n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle t_{\\varphi }=\\beta _{0}}\n  \n\nand the log-likelihood is:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0542", "text": "\u2113\n          \n            \u03c6\n          \n        \n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            K\n          \n        \n        \n          (\n          \n            \n              y\n              \n                k\n              \n            \n            ln\n            \u2061\n            (\n            \n              p\n              \n                \u03c6\n              \n            \n            )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0543", "text": ")\n            +\n            (\n            1\n            \u2212\n            \n              y\n              \n                k\n              \n            \n            )\n            ln\n            \u2061\n            (\n            1\n            \u2212\n            \n              p\n              \n                \u03c6\n              \n            \n            )\n          \n          )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0544", "text": ")\n        \n      \n    \n    {\\displaystyle \\ell _{\\varphi }=\\sum _{k=1}^{K}\\left(y_{k}\\ln(p_{\\varphi })+(1-y_{k})\\ln(1-p_{\\varphi })\\right)}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0545", "text": "Since we have \n  \n    \n      \n        \n          p\n          \n            \u03c6\n          \n        \n        =\n        \n          \n            y\n            \u00af\n          \n        \n      \n    \n    {\\displaystyle p_{\\varphi }={\\overline {y}}}\n  \n at the maximum of L, the maximum log-likelihood for the null model is", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0546", "text": "\u2113\n                ^\n              \n            \n          \n          \n            \u03c6\n          \n        \n        =\n        K\n        (\n        \n        \n          \n            y\n            \u00af\n          \n        \n        ln\n        \u2061\n        (\n        \n          \n            y\n            \u00af\n          \n        \n        )\n        +\n        (\n        1\n        \u2212\n        \n          \n            y\n            \u00af", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0547", "text": "y\n            \u00af\n          \n        \n        )\n        ln\n        \u2061\n        (\n        1\n        \u2212\n        \n          \n            y\n            \u00af\n          \n        \n        )\n        )\n      \n    \n    {\\displaystyle {\\hat {\\ell }}_{\\varphi }=K(\\,{\\overline {y}}\\ln({\\overline {y}})+(1-{\\overline {y}})\\ln(1-{\\overline {y}}))}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0548", "text": "The optimum \n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n is:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0549", "text": "\u03b2\n          \n            0\n          \n        \n        =\n        ln\n        \u2061\n        \n          (\n          \n            \n              \n                y\n                \u00af\n              \n              \n                1\n                \u2212\n                \n                  \n                    y\n                    \u00af\n                  \n                \n              \n            \n          \n          )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0550", "text": ")\n        \n      \n    \n    {\\displaystyle \\beta _{0}=\\ln \\left({\\frac {\\overline {y}}{1-{\\overline {y}}}}\\right)}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0551", "text": "where \n  \n    \n      \n        \n          \n            y\n            \u00af\n          \n        \n      \n    \n    {\\displaystyle {\\overline {y}}}\n  \n is again the mean of the yk values. Again, we can conceptually consider the fit of the proposed model to every permutation of the yk and it can be shown that the maximum log-likelihood of these permutation fits will never be smaller than that of the null model:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0552", "text": "\u2113\n              ^\n            \n          \n        \n        \u2265\n        \n          \n            \n              \n                \u2113\n                ^\n              \n            \n          \n          \n            \u03c6\n          \n        \n      \n    \n    {\\displaystyle {\\hat {\\ell }}\\geq {\\hat {\\ell }}_{\\varphi }}\n  \n\nAlso, as an analog to the error of the linear regression case, we may define the deviance of a logistic regression fit as:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0553", "text": "D\n        =\n        ln\n        \u2061\n        \n          (\n          \n            \n              \n                \n                  \n                    \n                      L\n                      ^\n                    \n                  \n                \n                \n                  2\n                \n              \n              \n                \n                  \n                    \n                      L\n                      ^", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0554", "text": "^\n                    \n                  \n                \n                \n                  \u03c6\n                \n                \n                  2\n                \n              \n            \n          \n          )\n        \n        =\n        2\n        (\n        \n          \n            \n              \u2113\n              ^\n            \n          \n        \n        \u2212\n        \n          \n            \n              \n                \u2113\n                ^", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0555", "text": "^\n              \n            \n          \n          \n            \u03c6\n          \n        \n        )\n      \n    \n    {\\displaystyle D=\\ln \\left({\\frac {{\\hat {L}}^{2}}{{\\hat {L}}_{\\varphi }^{2}}}\\right)=2({\\hat {\\ell }}-{\\hat {\\ell }}_{\\varphi })}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0556", "text": "which will always be positive or zero. The reason for this choice is that not only is the deviance a good measure of the goodness of fit, it is also approximately chi-squared distributed, with the approximation improving as the number of data points (K) increases, becoming exactly chi-square distributed in the limit of an infinite number of data points", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0557", "text": ". As in the case of linear regression, we may use this fact to estimate the probability that a random set of data points will give a better fit than the fit obtained by the proposed model, and so have an estimate how significantly the model is improved by including the xk data points in the proposed model.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0558", "text": "For the simple model of student test scores described above, the maximum value of the log-likelihood of the null model is \n  \n    \n      \n        \n          \n            \n              \n                \u2113\n                ^\n              \n            \n          \n          \n            \u03c6\n          \n        \n        =\n        \u2212\n        13.8629\n        \u2026\n      \n    \n    {\\displaystyle {\\hat {\\ell }}_{\\varphi }=-13.8629\\ldots }\n  \n The maximum value of the log-likelihood for the simple model is", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0559", "text": "\u2113\n              ^\n            \n          \n        \n        =\n        \u2212\n        8.02988\n        \u2026\n      \n    \n    {\\displaystyle {\\hat {\\ell }}=-8.02988\\ldots }\n  \n so that the deviance is \n  \n    \n      \n        D\n        =\n        2\n        (\n        \n          \n            \n              \u2113\n              ^\n            \n          \n        \n        \u2212\n        \n          \n            \n              \n                \u2113\n                ^", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0560", "text": "\u2113\n                ^\n              \n            \n          \n          \n            \u03c6\n          \n        \n        )\n        =\n        11.6661\n        \u2026\n      \n    \n    {\\displaystyle D=2({\\hat {\\ell }}-{\\hat {\\ell }}_{\\varphi })=11.6661\\ldots }", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0561", "text": "Using the chi-squared test of significance, the integral of the chi-squared distribution with one degree of freedom from 11.6661... to infinity is equal to 0.00063649...", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0562", "text": "This effectively means that about 6 out of a 10,000 fits to random yk can be expected to have a better fit (smaller deviance) than the given yk and so we can conclude that the inclusion of the x variable and data in the proposed model is a very significant improvement over the null model. In other words, we reject the null hypothesis with \n  \n    \n      \n        1\n        \u2212\n        D\n        \u2248\n        99.94\n        %\n      \n    \n    {\\displaystyle 1-D\\approx 99.94\\%}\n  \n confidence.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0563", "text": "Goodness of fit summary\nGoodness of fit in linear regression models is generally measured using R2. Since this has no direct analog in logistic regression, various methods including the following can be used instead.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0564", "text": "Deviance and likelihood ratio tests", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0565", "text": "In linear regression analysis, one is concerned with partitioning variance via the sum of squares calculations \u2013 variance in the criterion is essentially divided into variance accounted for by the predictors and residual variance. In logistic regression analysis, deviance is used in lieu of a sum of squares calculations. Deviance is analogous to the sum of squares calculations in linear regression  and is a measure of the lack of fit to the data in a logistic regression model", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0566", "text": ". When a \"saturated\" model is available (a model with a theoretically perfect fit), deviance is calculated by comparing a given model with the saturated model.  This computation gives the likelihood-ratio test:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0567", "text": "D\n        =\n        \u2212\n        2\n        ln\n        \u2061\n        \n          \n            likelihood of the fitted model\n            likelihood of the saturated model\n          \n        \n        .\n      \n    \n    {\\displaystyle D=-2\\ln {\\frac {\\text{likelihood of the fitted model}}{\\text{likelihood of the saturated model}}}.}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0568", "text": "In the above equation, D represents the deviance and ln represents the natural logarithm. The log of this likelihood ratio (the ratio of the fitted model to the saturated model) will produce a negative value, hence the need for a negative sign. D can be shown to follow an approximate chi-squared distribution.  Smaller values indicate better fit as the fitted model deviates less from the saturated model", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0569", "text": ". When assessed upon a chi-square distribution, nonsignificant chi-square values indicate very little unexplained variance and thus, good model fit. Conversely, a significant chi-square value indicates that a significant amount of the variance is unexplained.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0570", "text": "When the saturated model is not available (a common case), deviance is calculated simply as \u22122\u00b7(log likelihood of the fitted model), and the reference to the saturated model's log likelihood can be removed from all that follows without harm.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0571", "text": "Two measures of deviance are particularly important in logistic regression: null deviance and model deviance. The null deviance represents the difference between a model with only the intercept (which means \"no predictors\") and the saturated model. The model deviance represents the difference between a model with at least one predictor and the saturated model. In this respect, the null model provides a baseline upon which to compare predictor models", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0572", "text": ". Given that deviance is a measure of the difference between a given model and the saturated model, smaller values indicate better fit. Thus, to assess the contribution of a predictor or set of predictors, one can subtract the model deviance from the null deviance and assess the difference on a", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0573", "text": "\u03c7\n          \n            s\n            \u2212\n            p\n          \n          \n            2\n          \n        \n        ,\n      \n    \n    {\\displaystyle \\chi _{s-p}^{2},}\n  \n  chi-square distribution with degrees of freedom equal to the difference in the number of parameters estimated.\nLet", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0574", "text": "D\n                  \n                    null\n                  \n                \n              \n              \n                \n                =\n                \u2212\n                2\n                ln\n                \u2061\n                \n                  \n                    likelihood of null model\n                    likelihood of the saturated model", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0575", "text": "D\n                  \n                    fitted\n                  \n                \n              \n              \n                \n                =\n                \u2212\n                2\n                ln\n                \u2061\n                \n                  \n                    likelihood of fitted model\n                    likelihood of the saturated model", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0576", "text": ".\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}D_{\\text{null}}&=-2\\ln {\\frac {\\text{likelihood of null model}}{\\text{likelihood of the saturated model}}}\\\\[6pt]D_{\\text{fitted}}&=-2\\ln {\\frac {\\text{likelihood of fitted model}}{\\text{likelihood of the saturated model}}}.\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0577", "text": "Then the difference of both is:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0578", "text": "D\n                  \n                    null\n                  \n                \n                \u2212\n                \n                  D\n                  \n                    fitted\n                  \n                \n              \n              \n                \n                =\n                \u2212\n                2\n                \n                  (\n                  \n                    ln", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0579", "text": "ln\n                    \u2061\n                    \n                      \n                        likelihood of null model\n                        likelihood of the saturated model\n                      \n                    \n                    \u2212\n                    ln\n                    \u2061\n                    \n                      \n                        likelihood of fitted model\n                        likelihood of the saturated model", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0580", "text": ")\n                \n              \n            \n            \n              \n              \n                \n                =\n                \u2212\n                2\n                ln\n                \u2061\n                \n                  \n                    \n                      (\n                      \n                        \n                          \n                            likelihood of null model", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0581", "text": "likelihood of the saturated model\n                          \n                        \n                      \n                      )\n                    \n                    \n                      (\n                      \n                        \n                          \n                            likelihood of fitted model\n                            likelihood of the saturated model", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0582", "text": ")\n                    \n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \u2212\n                2\n                ln\n                \u2061\n                \n                  \n                    likelihood of the null model\n                    likelihood of fitted model\n                  \n                \n                .", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0583", "text": ".", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0584", "text": "{\\displaystyle {\\begin{aligned}D_{\\text{null}}-D_{\\text{fitted}}&=-2\\left(\\ln {\\frac {\\text{likelihood of null model}}{\\text{likelihood of the saturated model}}}-\\ln {\\frac {\\text{likelihood of fitted model}}{\\text{likelihood of the saturated model}}}\\right)\\\\[6pt]&=-2\\ln {\\frac {\\left({\\dfrac {\\text{likelihood of null model}}{\\text{likelihood of the saturated model}}}\\right)}{\\left({\\dfrac {\\text{likelihood of fitted model}}{\\text{likelihood of the saturated model}}}\\right)}}\\\\[6pt]&=-2\\ln", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0585", "text": "of the saturated model}}}\\right)}}\\\\[6pt]&=-2\\ln {\\frac {\\text{likelihood of the null model}}{\\text{likelihood of fitted model}}}.\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0586", "text": "If the model deviance is significantly smaller than the null deviance then one can conclude that the predictor or set of predictors significantly improve the model's fit. This is analogous to the F-test used in linear regression analysis to assess the significance of prediction.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0587", "text": "Pseudo-R-squared\nIn linear regression the squared multiple correlation, R2 is used to assess goodness of fit as it represents the proportion of variance in the criterion that is explained by the predictors. In logistic regression analysis, there is no agreed upon analogous measure, but there are several competing measures each with limitations.\nFour of the most commonly used indices and one less commonly used one are examined on this page:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0588", "text": "Likelihood ratio R2L\nCox and Snell R2CS\nNagelkerke R2N\nMcFadden R2McF\nTjur R2T", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0589", "text": "Hosmer\u2013Lemeshow test\nThe Hosmer\u2013Lemeshow test uses a test statistic that asymptotically follows a \n  \n    \n      \n        \n          \u03c7\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\chi ^{2}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0590", "text": "{\\displaystyle \\chi ^{2}}\n  \n distribution to assess whether or not the observed event rates match expected event rates in subgroups of the model population.  This test is considered to be obsolete by some statisticians because of its dependence on arbitrary binning of predicted probabilities and relative low power.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0591", "text": "Coefficient significance", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0592", "text": "After fitting the model, it is likely that researchers will want to examine the contribution of individual predictors. To do so, they will want to examine the regression coefficients. In linear regression, the regression coefficients represent the change in the criterion for each unit change in the predictor. In logistic regression, however, the regression coefficients represent the change in the logit for each unit change in the predictor", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0593", "text": ". Given that the logit is not intuitive, researchers are likely to focus on a predictor's effect on the exponential function of the regression coefficient \u2013 the odds ratio (see definition). In linear regression, the significance of a regression coefficient is assessed by computing a t test. In logistic regression, there are several different tests designed to assess the significance of an individual predictor, most notably the likelihood ratio test and the Wald statistic.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0594", "text": "Likelihood ratio test", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0595", "text": "The likelihood-ratio test discussed above to assess model fit is also the recommended procedure to assess the contribution of individual \"predictors\" to a given model. In the case of a single predictor model, one simply compares the deviance of the predictor model with that of the null model on a chi-square distribution with a single degree of freedom. If the predictor model has significantly smaller deviance (c.f", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0596", "text": ". chi-square using the difference in degrees of freedom of the two models), then one can conclude that there is a significant association between the \"predictor\" and the outcome. Although some common statistical packages (e.g. SPSS) do provide likelihood ratio test statistics, without this computationally intensive test it would be more difficult to assess the contribution of individual predictors in the multiple logistic regression case", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0597", "text": ". To assess the contribution of individual predictors one can enter the predictors hierarchically, comparing each new model with the previous to determine the contribution of each predictor. There is some debate among statisticians about the appropriateness of so-called \"stepwise\" procedures. The fear is that they may not preserve nominal statistical properties and may become misleading.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0598", "text": "Wald statistic\nAlternatively, when assessing the contribution of individual predictors in a given model, one may examine the significance of the Wald statistic. The Wald statistic, analogous to the t-test in linear regression, is used to assess the significance of coefficients. The Wald statistic is the ratio of the square of the regression coefficient to the square of the standard error of the coefficient and is asymptotically distributed as a chi-square distribution.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0599", "text": "W\n          \n            j\n          \n        \n        =\n        \n          \n            \n              \u03b2\n              \n                j\n              \n              \n                2\n              \n            \n            \n              S\n              \n                E\n                \n                  \n                    \u03b2\n                    \n                      j", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0600", "text": "2\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle W_{j}={\\frac {\\beta _{j}^{2}}{SE_{\\beta _{j}}^{2}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0601", "text": "Although several statistical packages (e.g., SPSS, SAS) report the Wald statistic to assess the contribution of individual predictors, the Wald statistic has limitations. When the regression coefficient is large, the standard error of the regression coefficient also tends to be larger increasing the probability of Type-II error. The Wald statistic also tends to be biased when data are sparse.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0602", "text": "Case-control sampling", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0603", "text": "Suppose cases are rare. Then we might wish to sample them more frequently than their prevalence in the population. For example, suppose there is a disease that affects 1 person in 10,000 and to collect our data we need to do a complete physical. It may be too expensive to do thousands of physicals of healthy people in order to obtain data for only a few diseased individuals. Thus, we may evaluate more diseased individuals, perhaps all of the rare outcomes", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0604", "text": ". This is also retrospective sampling, or equivalently it is called unbalanced data. As a rule of thumb, sampling controls at a rate of five times the number of cases will produce sufficient control data.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0605", "text": "Logistic regression is unique in that it may be estimated on unbalanced data, rather than randomly sampled data, and still yield correct coefficient estimates of the effects of each independent variable on the outcome.  That is to say, if we form a logistic model from such data, if the model is correct in the general population, the \n  \n    \n      \n        \n          \u03b2\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\beta _{j}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0606", "text": "{\\displaystyle \\beta _{j}}\n  \n parameters are all correct except for \n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n. We can correct \n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n if we know the true prevalence as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0607", "text": "\u03b2\n                ^\n              \n            \n          \n          \n            0\n          \n          \n            \u2217\n          \n        \n        =\n        \n          \n            \n              \n                \u03b2\n                ^\n              \n            \n          \n          \n            0\n          \n        \n        +\n        log\n        \u2061\n        \n          \n            \u03c0\n            \n              1", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0608", "text": "\u03c0\n            \n              1\n              \u2212\n              \u03c0\n            \n          \n        \n        \u2212\n        log\n        \u2061\n        \n          \n            \n              \n                \n                  \u03c0\n                  ~\n                \n              \n            \n            \n              1\n              \u2212\n              \n                \n                  \n                    \u03c0\n                    ~", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0609", "text": "{\\displaystyle {\\widehat {\\beta }}_{0}^{*}={\\widehat {\\beta }}_{0}+\\log {\\frac {\\pi }{1-\\pi }}-\\log {{\\tilde {\\pi }} \\over {1-{\\tilde {\\pi }}}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0610", "text": "where \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n is the true prevalence and \n  \n    \n      \n        \n          \n            \n              \u03c0\n              ~\n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {\\pi }}}\n  \n is the prevalence in the sample.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0611", "text": "Discussion", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0612", "text": "Like other forms of regression analysis, logistic regression makes use of one or more predictor variables that may be either continuous or categorical. Unlike ordinary linear regression, however, logistic regression is used for predicting dependent variables that take membership in one of a limited number of categories (treating the dependent variable in the binomial case as the outcome of a Bernoulli trial) rather than a continuous outcome", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0613", "text": ". Given this difference, the assumptions of linear regression are violated. In particular, the residuals cannot be normally distributed. In addition, linear regression may make nonsensical predictions for a binary dependent variable. What is needed is a way to convert a binary variable into a continuous one that can take on any real value (negative or positive)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0614", "text": ". To do that, binomial logistic regression first calculates the odds of the event happening for different levels of each independent variable, and then takes its logarithm to create a continuous criterion as a transformed version of the dependent variable. The logarithm of the odds is the logit of the probability, the logit is defined as follows:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0615", "text": "logit\n        \u2061\n        p\n        =\n        ln\n        \u2061\n        \n          \n            p\n            \n              1\n              \u2212\n              p\n            \n          \n        \n        \n        \n          for \n        \n        0\n        <\n        p\n        <\n        1\n        \n        .\n      \n    \n    {\\displaystyle \\operatorname {logit} p=\\ln {\\frac {p}{1-p}}\\quad {\\text{for }}0<p<1\\,.}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0616", "text": "Although the dependent variable in logistic regression is Bernoulli, the logit is on an unrestricted scale. The logit function is the link function in this kind of generalized linear model, i.e.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0617", "text": "logit\n        \u2061\n        \n          \n            E\n          \n        \n        \u2061\n        (\n        Y\n        )\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        x\n      \n    \n    {\\displaystyle \\operatorname {logit} \\operatorname {\\mathcal {E}} (Y)=\\beta _{0}+\\beta _{1}x}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0618", "text": "Y is the Bernoulli-distributed response variable and x is the predictor variable; the \u03b2 values are the linear parameters.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0619", "text": "The logit of the probability of success is then fitted to the predictors. The predicted value of the logit is converted back into predicted odds, via the inverse of the natural logarithm \u2013 the exponential function. Thus, although the observed dependent variable in binary logistic regression is a 0-or-1 variable, the logistic regression estimates the odds, as a continuous variable, that the dependent variable is a 'success'. In some applications, the odds are all that is needed", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0620", "text": ". In others, a specific yes-or-no prediction is needed for whether the dependent variable is or is not a 'success'; this categorical prediction can be based on the computed odds of success, with predicted odds above some chosen cutoff value being translated into a prediction of success.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0621", "text": "Machine learning and cross-entropy loss function\nIn machine learning applications where logistic regression is used for binary classification, the MLE minimises the cross-entropy loss function.\nLogistic regression is an important machine learning algorithm. The goal is to model the probability of a random variable \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n being 0 or 1 given experimental data.\nConsider a generalized linear model function parameterized by", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0622", "text": "\u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0623", "text": "h\n          \n            \u03b8\n          \n        \n        (\n        X\n        )\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  \n                    \u03b8\n                    \n                      T\n                    \n                  \n                  X\n                \n              \n            \n          \n        \n        =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0624", "text": "=\n        Pr\n        (\n        Y\n        =\n        1\n        \u2223\n        X\n        ;\n        \u03b8\n        )\n      \n    \n    {\\displaystyle h_{\\theta }(X)={\\frac {1}{1+e^{-\\theta ^{T}X}}}=\\Pr(Y=1\\mid X;\\theta )}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0625", "text": "Therefore,\n\n  \n    \n      \n        Pr\n        (\n        Y\n        =\n        0\n        \u2223\n        X\n        ;\n        \u03b8\n        )\n        =\n        1\n        \u2212\n        \n          h\n          \n            \u03b8\n          \n        \n        (\n        X\n        )\n      \n    \n    {\\displaystyle \\Pr(Y=0\\mid X;\\theta )=1-h_{\\theta }(X)}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0626", "text": "and since \n  \n    \n      \n        Y\n        \u2208\n        {\n        0\n        ,\n        1\n        }\n      \n    \n    {\\displaystyle Y\\in \\{0,1\\}}\n  \n, we see that \n  \n    \n      \n        Pr\n        (\n        y\n        \u2223\n        X\n        ;\n        \u03b8\n        )\n      \n    \n    {\\displaystyle \\Pr(y\\mid X;\\theta )}\n  \n is given by \n  \n    \n      \n        Pr\n        (\n        y\n        \u2223\n        X\n        ;\n        \u03b8\n        )\n        =\n        \n          h\n          \n            \u03b8", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0627", "text": "\u03b8\n          \n        \n        (\n        X\n        \n          )\n          \n            y\n          \n        \n        (\n        1\n        \u2212\n        \n          h\n          \n            \u03b8\n          \n        \n        (\n        X\n        )\n        \n          )\n          \n            (\n            1\n            \u2212\n            y\n            )\n          \n        \n        .\n      \n    \n    {\\displaystyle \\Pr(y\\mid X;\\theta )=h_{\\theta }(X)^{y}(1-h_{\\theta }(X))^{(1-y)}.}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0628", "text": "We now calculate the likelihood function assuming that all the observations in the sample are independently Bernoulli distributed,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0629", "text": "L\n                (\n                \u03b8\n                \u2223\n                y\n                ;\n                x\n                )\n              \n              \n                \n                =\n                Pr\n                (\n                Y\n                \u2223\n                X\n                ;\n                \u03b8\n                )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0630", "text": "=\n                \n                  \u220f\n                  \n                    i\n                  \n                \n                Pr\n                (\n                \n                  y\n                  \n                    i\n                  \n                \n                \u2223\n                \n                  x\n                  \n                    i\n                  \n                \n                ;", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0631", "text": ";\n                \u03b8\n                )\n              \n            \n            \n              \n              \n                \n                =\n                \n                  \u220f\n                  \n                    i\n                  \n                \n                \n                  h\n                  \n                    \u03b8\n                  \n                \n                (\n                \n                  x", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0632", "text": "x\n                  \n                    i\n                  \n                \n                \n                  )\n                  \n                    \n                      y\n                      \n                        i\n                      \n                    \n                  \n                \n                (\n                1\n                \u2212\n                \n                  h\n                  \n                    \u03b8", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0633", "text": "(\n                \n                  x\n                  \n                    i\n                  \n                \n                )\n                \n                  )\n                  \n                    (\n                    1\n                    \u2212\n                    \n                      y\n                      \n                        i\n                      \n                    \n                    )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0634", "text": ")\n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}L(\\theta \\mid y;x)&=\\Pr(Y\\mid X;\\theta )\\\\&=\\prod _{i}\\Pr(y_{i}\\mid x_{i};\\theta )\\\\&=\\prod _{i}h_{\\theta }(x_{i})^{y_{i}}(1-h_{\\theta }(x_{i}))^{(1-y_{i})}\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0635", "text": "Typically, the log likelihood is maximized,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0636", "text": "N\n          \n            \u2212\n            1\n          \n        \n        log\n        \u2061\n        L\n        (\n        \u03b8\n        \u2223\n        y\n        ;\n        x\n        )\n        =\n        \n          N\n          \n            \u2212\n            1\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            N\n          \n        \n        log\n        \u2061\n        Pr\n        (\n        \n          y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0637", "text": "(\n        \n          y\n          \n            i\n          \n        \n        \u2223\n        \n          x\n          \n            i\n          \n        \n        ;\n        \u03b8\n        )\n      \n    \n    {\\displaystyle N^{-1}\\log L(\\theta \\mid y;x)=N^{-1}\\sum _{i=1}^{N}\\log \\Pr(y_{i}\\mid x_{i};\\theta )}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0638", "text": "which is maximized using optimization techniques such as gradient descent.\nAssuming the \n  \n    \n      \n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle (x,y)}\n  \n pairs are drawn uniformly from the underlying distribution, then in the limit of large N,", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0639", "text": "lim\n                  \n                    N\n                    \u2192\n                    +\n                    \u221e\n                  \n                \n                \n                  N\n                  \n                    \u2212\n                    1\n                  \n                \n                \n                  \u2211\n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0640", "text": "i\n                    =\n                    1\n                  \n                  \n                    N\n                  \n                \n                log\n                \u2061\n                Pr\n                (\n                \n                  y\n                  \n                    i\n                  \n                \n                \u2223\n                \n                  x\n                  \n                    i", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0641", "text": "i\n                  \n                \n                ;\n                \u03b8\n                )\n                =\n                \n                  \u2211\n                  \n                    x\n                    \u2208\n                    \n                      \n                        X\n                      \n                    \n                  \n                \n                \n                  \u2211\n                  \n                    y\n                    \u2208", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0642", "text": "y\n                    \u2208\n                    \n                      \n                        Y\n                      \n                    \n                  \n                \n                Pr\n                (\n                X\n                =\n                x\n                ,\n                Y\n                =\n                y\n                )\n                log\n                \u2061\n                Pr\n                (\n                Y\n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0643", "text": "Y\n                =\n                y\n                \u2223\n                X\n                =\n                x\n                ;\n                \u03b8\n                )\n              \n            \n            \n              \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0644", "text": "\u2211\n                  \n                    x\n                    \u2208\n                    \n                      \n                        X\n                      \n                    \n                  \n                \n                \n                  \u2211\n                  \n                    y\n                    \u2208\n                    \n                      \n                        Y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0645", "text": "Y\n                      \n                    \n                  \n                \n                Pr\n                (\n                X\n                =\n                x\n                ,\n                Y\n                =\n                y\n                )\n                \n                  (\n                  \n                    \u2212\n                    log\n                    \u2061", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0646", "text": "Pr\n                          (\n                          Y\n                          =\n                          y\n                          \u2223\n                          X\n                          =\n                          x\n                          )\n                        \n                        \n                          Pr\n                          (\n                          Y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0647", "text": "Y\n                          =\n                          y\n                          \u2223\n                          X\n                          =\n                          x\n                          ;\n                          \u03b8\n                          )\n                        \n                      \n                    \n                    +\n                    log\n                    \u2061\n                    Pr\n                    (\n                    Y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0648", "text": "(\n                    Y\n                    =\n                    y\n                    \u2223\n                    X\n                    =\n                    x\n                    )\n                  \n                  )\n                \n              \n            \n            \n              \n                =", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0649", "text": "\u2212\n                \n                  D\n                  \n                    KL\n                  \n                \n                (\n                Y\n                \u2225\n                \n                  Y\n                  \n                    \u03b8\n                  \n                \n                )\n                \u2212\n                H\n                (\n                Y\n                \u2223\n                X", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0650", "text": "\u2223\n                X\n                )", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0651", "text": "{\\displaystyle {\\begin{aligned}&\\lim \\limits _{N\\rightarrow +\\infty }N^{-1}\\sum _{i=1}^{N}\\log \\Pr(y_{i}\\mid x_{i};\\theta )=\\sum _{x\\in {\\mathcal {X}}}\\sum _{y\\in {\\mathcal {Y}}}\\Pr(X=x,Y=y)\\log \\Pr(Y=y\\mid X=x;\\theta )\\\\[6pt]={}&\\sum _{x\\in {\\mathcal {X}}}\\sum _{y\\in {\\mathcal {Y}}}\\Pr(X=x,Y=y)\\left(-\\log {\\frac {\\Pr(Y=y\\mid X=x)}{\\Pr(Y=y\\mid X=x;\\theta )}}+\\log \\Pr(Y=y\\mid X=x)\\right)\\\\[6pt]={}&-D_{\\text{KL}}(Y\\parallel Y_{\\theta })-H(Y\\mid X)\\end{aligned}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0652", "text": "where \n  \n    \n      \n        H\n        (\n        Y\n        \u2223\n        X\n        )\n      \n    \n    {\\displaystyle H(Y\\mid X)}\n  \n is the conditional entropy and \n  \n    \n      \n        \n          D\n          \n            KL\n          \n        \n      \n    \n    {\\displaystyle D_{\\text{KL}}}", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0653", "text": "{\\displaystyle D_{\\text{KL}}}\n  \n is the Kullback\u2013Leibler divergence. This leads to the intuition that by maximizing the log-likelihood of a model, you are minimizing the KL divergence of your model from the maximal entropy distribution. Intuitively searching for the model that makes the fewest assumptions in its parameters.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0654", "text": "Comparison with linear regression", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0655", "text": "Logistic regression can be seen as a special case of the generalized linear model and thus analogous to linear regression. The model of logistic regression, however, is based on quite different assumptions (about the relationship between the dependent and independent variables) from those of linear regression. In particular, the key differences between these two models can be seen in the following two features of logistic regression. First, the conditional distribution \n  \n    \n      \n        y", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0656", "text": "y\n        \u2223\n        x\n      \n    \n    {\\displaystyle y\\mid x}\n  \n is a Bernoulli distribution rather than a Gaussian distribution, because the dependent variable is binary. Second, the predicted values are probabilities and are therefore restricted to (0,1) through the logistic distribution function because logistic regression predicts the probability of particular outcomes rather than the outcomes themselves.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0657", "text": "Alternatives", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0658", "text": "A common alternative to the logistic model (logit model) is the probit model, as the related names suggest. From the perspective of generalized linear models, these differ in the choice of link function: the logistic model uses the logit function (inverse logistic function), while the probit model uses the probit function (inverse error function)", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0659", "text": ". Equivalently, in the latent variable interpretations of these two methods, the first assumes a standard logistic distribution of errors and the second a standard normal distribution of errors. Other sigmoid functions or error distributions can be used instead.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0660", "text": "Logistic regression is an alternative to Fisher's 1936 method, linear discriminant analysis.  If the assumptions of linear discriminant analysis hold, the conditioning can be reversed to produce logistic regression.  The converse is not true, however, because logistic regression does not require the multivariate normal assumption of discriminant analysis.\nThe assumption of linear predictor effects can easily be relaxed using techniques such as spline functions.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0661", "text": "History", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0662", "text": "A detailed history of the logistic regression is given in Cramer (2002). The logistic function was developed as a model of population growth and named \"logistic\" by Pierre Fran\u00e7ois Verhulst in the 1830s and 1840s, under the guidance of Adolphe Quetelet; see Logistic function \u00a7 History for details. In his earliest paper (1838), Verhulst did not specify how he fit the curves to the data", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0663", "text": ". In his more detailed paper (1845), Verhulst determined the three parameters of the model by making the curve pass through three observed points, which yielded poor predictions.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0664", "text": "The logistic function was independently developed in chemistry as a model of autocatalysis (Wilhelm Ostwald, 1883). An autocatalytic reaction is one in which one of the products is itself a catalyst for the same reaction, while the supply of one of the reactants is fixed. This naturally gives rise to the logistic equation for the same reason as population growth: the reaction is self-reinforcing but constrained.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0665", "text": "The logistic function was independently rediscovered as a model of population growth in 1920 by Raymond Pearl and Lowell Reed, published as Pearl & Reed (1920), which led to its use in modern statistics. They were initially unaware of Verhulst's work and presumably learned about it from L. Gustave du Pasquier, but they gave him little credit and did not adopt his terminology. Verhulst's priority was acknowledged and the term \"logistic\" revived by Udny Yule in 1925 and has been followed since", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0666", "text": ". Pearl and Reed first applied the model to the population of the United States, and also initially fitted the curve by making it pass through three points; as with Verhulst, this again yielded poor results.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0667", "text": "In the 1930s, the probit model was developed and systematized by Chester Ittner Bliss, who coined the term \"probit\" in Bliss (1934), and by John Gaddum in Gaddum (1933), and the model fit by maximum likelihood estimation by Ronald A. Fisher in Fisher (1935), as an addendum to Bliss's work. The probit model was principally used in bioassay, and had been preceded by earlier work dating to 1860; see Probit model \u00a7 History", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0668", "text": ". The probit model influenced the subsequent development of the logit model and these models competed with each other.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0669", "text": "The logistic model was likely first used as an alternative to the probit model in bioassay by Edwin Bidwell Wilson and his student Jane Worcester in Wilson & Worcester (1943). However, the development of the logistic model as a general alternative to the probit model was principally due to the work of Joseph Berkson over many decades, beginning in Berkson (1944), where he coined \"logit\", by analogy with \"probit\", and continuing through Berkson (1951) and following years", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0670", "text": ". The logit model was initially dismissed as inferior to the probit model, but \"gradually achieved an equal footing with the probit\", particularly between 1960 and 1970. By 1970, the logit model achieved parity with the probit model in use in statistics journals and thereafter surpassed it", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0671", "text": ". This relative popularity was due to the adoption of the logit outside of bioassay, rather than displacing the probit within bioassay, and its informal use in practice; the logit's popularity is credited to the logit model's computational simplicity, mathematical properties, and generality, allowing its use in varied fields.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0672", "text": "Various refinements occurred during that time, notably by David Cox, as in Cox (1958).", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0673", "text": "The multinomial logit model was introduced independently in Cox (1966) and Theil (1969), which greatly increased the scope of application and the popularity of the logit model", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0674", "text": ". In 1973 Daniel McFadden linked the multinomial logit to the theory of discrete choice, specifically Luce's choice axiom, showing that the multinomial logit followed from the assumption of independence of irrelevant alternatives and interpreting odds of alternatives as relative preferences; this gave a theoretical foundation for the logistic regression.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0675", "text": "Extensions\nThere are large numbers of extensions:", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0676", "text": "Multinomial logistic regression (or multinomial logit) handles the case of a multi-way categorical dependent variable (with unordered values, also called \"classification\").  The general case of having dependent variables with more than two values is termed polytomous regression.\nOrdered logistic regression (or ordered logit) handles ordinal dependent variables (ordered values).", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0677", "text": "Mixed logit is an extension of multinomial logit that allows for correlations among the choices of the dependent variable.\nAn extension of the logistic model to sets of interdependent variables is the conditional random field.\nConditional logistic regression handles matched or stratified data when the strata are small. It is mostly used in the analysis of observational studies.", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0678", "text": "See also\nLogistic function\nDiscrete choice\nJarrow\u2013Turnbull model\nLimited dependent variable\nMultinomial logit model\nOrdered logit\nHosmer\u2013Lemeshow test\nBrier score\nmlpack - contains a C++ implementation of logistic regression\nLocal case-control sampling\nLogistic model tree\n\nReferences\nSources\nExternal links", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Logistic_regression_0679", "text": "References\nSources\nExternal links\n\n Media related to Logistic regression at Wikimedia Commons\nEconometrics Lecture (topic: Logit model) on YouTube by Mark Thoma\nLogistic Regression tutorial\nmlelr: software in C for teaching purposes", "title": "Logistic regression", "url": "https://en.wikipedia.org/wiki/Logistic_regression"}, {"id": "rec_Linear_regression_0000", "text": "In statistics, linear regression is a model that estimates the relationship between a scalar response (dependent variable) and one or more explanatory variables (regressor or independent variable). \n\"Linear regression is linear in its unknown parameters ( i.e., the parameters appear only to the first power) not necessarily in variables.\"", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0001", "text": "From a mathematical perspective, X and Y are treated as variables, and the parameters are considered fixed constants but from a statistics perspective, the focus is on the parameters. Once we substitute observed data for X and Y, the model becomes a function of the parameters, which then behave like variables that need to be estimated.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0002", "text": "Y\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        X\n        +\n        \u03f5\n      \n    \n    {\\displaystyle Y=\\beta _{0}+\\beta _{1}X+\\epsilon }", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0003", "text": "Y\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        \n          X\n          \n            2\n          \n        \n        +\n        \u03f5\n      \n    \n    {\\displaystyle Y=\\beta _{0}+\\beta _{1}X^{2}+\\epsilon }", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0004", "text": "Y\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        l\n        n\n        X\n        +\n        \u03f5\n      \n    \n    {\\displaystyle Y=\\beta _{0}+\\beta _{1}lnX+\\epsilon }\n  \n\nIn the above equations, the coefficients (parameters) are linear (i.e., raised to the power of 1).\nNon linear regression", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0005", "text": "Y\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n          \n            2\n          \n        \n        X\n        +\n        \u03f5\n      \n    \n    {\\displaystyle Y=\\beta _{0}+\\beta _{1}^{2}X+\\epsilon }", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0006", "text": "Y\n        =\n        l\n        n\n        \n          \u03b2\n          \n            1\n          \n        \n        X\n        +\n        \u03f5\n      \n    \n    {\\displaystyle Y=ln\\beta _{1}X+\\epsilon }\n  \n  \nIn the above equations, the coefficients (parameters) are non-linear.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0007", "text": "A model with exactly one explanatory variable is a simple linear regression; a model with two or more explanatory variables is a multiple linear regression. This term is distinct from multivariate linear regression, which predicts multiple correlated dependent variables rather than a single dependent variable.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0008", "text": "In linear regression, the relationships are modeled using linear predictor functions whose unknown model parameters are estimated from the data. Most commonly, the conditional mean of the response given the values of the explanatory variables (or predictors) is assumed to be an affine function of those values; less commonly, the conditional median or some other quantile is used", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0009", "text": ". Like all forms of regression analysis, linear regression focuses on the conditional probability distribution of the response given the values of the predictors, rather than on the joint probability distribution of all of these variables, which is the domain of multivariate analysis.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0010", "text": "Linear regression is also a type of machine learning algorithm, more specifically a supervised algorithm, that learns from the labelled datasets and maps the data points to the most optimized linear functions that can be used for prediction on new datasets.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0011", "text": "Linear regression was the first type of regression analysis to be studied rigorously, and to be used extensively in practical applications. This is because models which depend linearly on their unknown parameters are easier to fit than models which are non-linearly related to their parameters and because the statistical properties of the resulting estimators are easier to determine.\nLinear regression has many practical uses. Most applications fall into one of the following two broad categories:", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0012", "text": "If the goal is to reduce error, i.e. variance in prediction or forecasting, linear regression can be used to fit a predictive model to an observed data set of values of the response and explanatory variables. After developing such a model, if additional values of the explanatory variables are collected without an accompanying response value, the fitted model can be used to make a prediction of the response.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0013", "text": "If the goal is to explain variation in the response variable that can be attributed to variation in the explanatory variables, linear regression analysis can be applied to quantify the strength of the relationship between the response and the explanatory variables, and in particular to determine whether some explanatory variables may have no linear relationship with the response at all, or to identify which subsets of explanatory variables may contain redundant information about the response.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0014", "text": "Linear regression models are often fitted using the least squares approach, but they may also be fitted in other ways, such as by minimizing the \"lack of fit\" in some other norm (as with least absolute deviations regression), or by minimizing a penalized version of the least squares cost function as in ridge regression (L2-norm penalty) and lasso (L1-norm penalty)", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0015", "text": ". Use of the Mean Squared Error (MSE) as the cost on a dataset that has many large outliers, can result in a model that fits the outliers more than the true data due to the higher importance assigned by MSE to large errors. So, cost functions that are robust to outliers should be used if the dataset has many large outliers. Conversely, the least squares approach can be used to fit models that are not linear models", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0016", "text": ". Thus, although the terms \"least squares\" and \"linear model\" are closely linked, they are not synonymous.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0017", "text": "Formulation\nGiven a data set \n  \n    \n      \n        {\n        \n          y\n          \n            i\n          \n        \n        ,\n        \n        \n          x\n          \n            i\n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            i\n            p\n          \n        \n        \n          }\n          \n            i\n            =\n            1\n          \n          \n            n", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0018", "text": "n\n          \n        \n      \n    \n    {\\displaystyle \\{y_{i},\\,x_{i1},\\ldots ,x_{ip}\\}_{i=1}^{n}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0019", "text": "of n statistical units, a linear regression model assumes that the relationship between the dependent variable y and the vector of regressors x is linear. This relationship is modeled through a disturbance term or error variable \u03b5\u2014an unobserved random variable that adds extra \"noise\" to the linear relationship between the dependent variable and regressors. Thus the model takes its form\n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n        =", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0020", "text": "=\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        \n          x\n          \n            i\n            1\n          \n        \n        +\n        \u22ef\n        +\n        \n          \u03b2\n          \n            p\n          \n        \n        \n          x\n          \n            i\n            p\n          \n        \n        +\n        \n          \u03b5\n          \n            i", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0021", "text": "\u03b5\n          \n            i\n          \n        \n        =\n        \n          \n            x\n          \n          \n            i\n          \n          \n            \n              T\n            \n          \n        \n        \n          \u03b2\n        \n        +\n        \n          \u03b5\n          \n            i\n          \n        \n        ,\n        \n        i\n        =\n        1\n        ,\n        \u2026\n        ,\n        n\n        ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0022", "text": ",\n        n\n        ,\n      \n    \n    {\\displaystyle y_{i}=\\beta _{0}+\\beta _{1}x_{i1}+\\cdots +\\beta _{p}x_{ip}+\\varepsilon _{i}=\\mathbf {x} _{i}^{\\mathsf {T}}{\\boldsymbol {\\beta }}+\\varepsilon _{i},\\qquad i=1,\\ldots ,n,}\n  \nwhere T denotes the transpose, so that xiT\u03b2 is the inner product between vectors xi and \u03b2.\nOften these n equations are stacked together and written in matrix notation as", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0023", "text": "y\n        \n        =\n        \n          X\n        \n        \n          \u03b2\n        \n        +\n        \n          \u03b5\n        \n        ,\n        \n      \n    \n    {\\displaystyle \\mathbf {y} =\\mathbf {X} {\\boldsymbol {\\beta }}+{\\boldsymbol {\\varepsilon }},\\,}\n  \n\nwhere", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0024", "text": "y\n        \n        =\n        \n          \n            [\n            \n              \n                \n                  \n                    y\n                    \n                      1\n                    \n                  \n                \n              \n              \n                \n                  \n                    y\n                    \n                      2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0025", "text": "\u22ee\n                \n              \n              \n                \n                  \n                    y\n                    \n                      n\n                    \n                  \n                \n              \n            \n            ]\n          \n        \n        ,\n        \n      \n    \n    {\\displaystyle \\mathbf {y} ={\\begin{bmatrix}y_{1}\\\\y_{2}\\\\\\vdots \\\\y_{n}\\end{bmatrix}},\\quad }", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0026", "text": "X\n        \n        =\n        \n          \n            [\n            \n              \n                \n                  \n                    \n                      x\n                    \n                    \n                      1\n                    \n                    \n                      \n                        T", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0027", "text": "x\n                    \n                    \n                      2\n                    \n                    \n                      \n                        T\n                      \n                    \n                  \n                \n              \n              \n                \n                  \u22ee", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0028", "text": "x\n                    \n                    \n                      n\n                    \n                    \n                      \n                        T\n                      \n                    \n                  \n                \n              \n            \n            ]\n          \n        \n        =\n        \n          \n            [\n            \n              \n                \n                  1", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0029", "text": "1\n                \n                \n                  \n                    x\n                    \n                      11\n                    \n                  \n                \n                \n                  \u22ef\n                \n                \n                  \n                    x\n                    \n                      1\n                      p", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0030", "text": "1\n                \n                \n                  \n                    x\n                    \n                      21\n                    \n                  \n                \n                \n                  \u22ef\n                \n                \n                  \n                    x\n                    \n                      2\n                      p", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0031", "text": "\u22ee\n                \n                \n                  \u22ee\n                \n                \n                  \u22f1\n                \n                \n                  \u22ee\n                \n              \n              \n                \n                  1\n                \n                \n                  \n                    x\n                    \n                      n\n                      1", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0032", "text": "1\n                    \n                  \n                \n                \n                  \u22ef\n                \n                \n                  \n                    x\n                    \n                      n\n                      p\n                    \n                  \n                \n              \n            \n            ]\n          \n        \n        ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0033", "text": ",\n      \n    \n    {\\displaystyle \\mathbf {X} ={\\begin{bmatrix}\\mathbf {x} _{1}^{\\mathsf {T}}\\\\\\mathbf {x} _{2}^{\\mathsf {T}}\\\\\\vdots \\\\\\mathbf {x} _{n}^{\\mathsf {T}}\\end{bmatrix}}={\\begin{bmatrix}1&x_{11}&\\cdots &x_{1p}\\\\1&x_{21}&\\cdots &x_{2p}\\\\\\vdots &\\vdots &\\ddots &\\vdots \\\\1&x_{n1}&\\cdots &x_{np}\\end{bmatrix}},}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0034", "text": "\u03b2\n        \n        =\n        \n          \n            [\n            \n              \n                \n                  \n                    \u03b2\n                    \n                      0\n                    \n                  \n                \n              \n              \n                \n                  \n                    \u03b2\n                    \n                      1", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0035", "text": "\u03b2\n                    \n                      2\n                    \n                  \n                \n              \n              \n                \n                  \u22ee\n                \n              \n              \n                \n                  \n                    \u03b2\n                    \n                      p", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0036", "text": "]\n          \n        \n        ,\n        \n        \n          \u03b5\n        \n        =\n        \n          \n            [\n            \n              \n                \n                  \n                    \u03b5\n                    \n                      1\n                    \n                  \n                \n              \n              \n                \n                  \n                    \u03b5\n                    \n                      2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0037", "text": "2\n                    \n                  \n                \n              \n              \n                \n                  \u22ee\n                \n              \n              \n                \n                  \n                    \u03b5\n                    \n                      n\n                    \n                  \n                \n              \n            \n            ]\n          \n        \n        .", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0038", "text": ".\n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}={\\begin{bmatrix}\\beta _{0}\\\\\\beta _{1}\\\\\\beta _{2}\\\\\\vdots \\\\\\beta _{p}\\end{bmatrix}},\\quad {\\boldsymbol {\\varepsilon }}={\\begin{bmatrix}\\varepsilon _{1}\\\\\\varepsilon _{2}\\\\\\vdots \\\\\\varepsilon _{n}\\end{bmatrix}}.}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0039", "text": "Notation and terminology\ny\n        \n      \n    \n    {\\displaystyle \\mathbf {y} }\n  \n is a vector of observed values \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n         \n        (\n        i\n        =\n        1\n        ,\n        \u2026\n        ,\n        n\n        )\n      \n    \n    {\\displaystyle y_{i}\\ (i=1,\\ldots ,n)}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0040", "text": "{\\displaystyle y_{i}\\ (i=1,\\ldots ,n)}\n  \n of the variable called the regressand, endogenous variable, response variable, target variable, measured variable, criterion variable, or dependent variable. This variable is also sometimes known as the predicted variable, but this should not be confused with predicted values, which are denoted \n  \n    \n      \n        \n          \n            \n              y\n              ^\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {y}}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0041", "text": "{\\displaystyle {\\hat {y}}}\n  \n. The decision as to which variable in a data set is modeled as the dependent variable and which are modeled as the independent variables may be based on a presumption that the value of one of the variables is caused by, or directly influenced by the other variables. Alternatively, there may be an operational reason to model one of the variables in terms of the others, in which case there need be no presumption of causality.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0042", "text": "X\n        \n      \n    \n    {\\displaystyle \\mathbf {X} }\n  \n may be seen as a matrix of row-vectors \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n            \u22c5\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i\\cdot }}\n  \n or of n-dimensional column-vectors \n  \n    \n      \n        \n          \n            x\n          \n          \n            \u22c5\n            j", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0043", "text": "j\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{\\cdot j}}\n  \n, which are known as regressors, exogenous variables, explanatory variables, covariates, input variables, predictor variables, or independent variables (not to be confused with the concept of independent random variables). The matrix \n  \n    \n      \n        \n          X\n        \n      \n    \n    {\\displaystyle \\mathbf {X} }\n  \n is sometimes called the design matrix.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0044", "text": "is sometimes called the design matrix.\nUsually a constant is included as one of the regressors. In particular, \n  \n    \n      \n        \n          x\n          \n            i\n            0\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle x_{i0}=1}\n  \n for \n  \n    \n      \n        i\n        =\n        1\n        ,\n        \u2026\n        ,\n        n\n      \n    \n    {\\displaystyle i=1,\\ldots ,n}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0045", "text": "{\\displaystyle i=1,\\ldots ,n}\n  \n. The corresponding element of \u03b2 is called the intercept. Many statistical inference procedures for linear models require an intercept to be present, so it is often included even if theoretical considerations suggest that its value should be zero.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0046", "text": "Sometimes one of the regressors can be a non-linear function of another regressor or of the data values, as in polynomial regression and segmented regression. The model remains linear as long as it is linear in the parameter vector \u03b2.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0047", "text": "The values xij may be viewed as either observed values of random variables Xj or as fixed values chosen prior to observing the dependent variable. Both interpretations may be appropriate in different cases, and they generally lead to the same estimation procedures; however different approaches to asymptotic analysis are used in these two situations.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0048", "text": "\u03b2\n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}}\n  \n is a \n  \n    \n      \n        (\n        p\n        +\n        1\n        )\n      \n    \n    {\\displaystyle (p+1)}\n  \n-dimensional parameter vector, where \n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n is the intercept term (if one is included in the model\u2014otherwise \n  \n    \n      \n        \n          \u03b2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0049", "text": "\u03b2\n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0050", "text": "is p-dimensional). Its elements are known as effects or regression coefficients (although the latter term is sometimes reserved for the estimated effects). In simple linear regression, p=1, and the coefficient is known as regression slope. Statistical estimation and inference in linear regression focuses on \u03b2. The elements of this parameter vector are interpreted as the partial derivatives of the dependent variable with respect to the various independent variables.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0051", "text": "\u03b5\n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\varepsilon }}}\n  \n is a vector of values \n  \n    \n      \n        \n          \u03b5\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\varepsilon _{i}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0052", "text": ". This part of the model is called the error term, disturbance term, or sometimes noise (in contrast with the \"signal\" provided by the rest of the model). This variable captures all other factors which influence the dependent variable y other than the regressors x. The relationship between the error term and the regressors, for example their correlation, is a crucial consideration in formulating a linear regression model, as it will determine the appropriate estimation method.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0053", "text": "Fitting a linear model to a given data set usually requires estimating the regression coefficients \n  \n    \n      \n        \n          \u03b2\n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\beta }}}\n  \n such that the error term \n  \n    \n      \n        \n          \u03b5\n        \n        =\n        \n          y\n        \n        \u2212\n        \n          X\n        \n        \n          \u03b2\n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\varepsilon }}=\\mathbf {y} -\\mathbf {X} {\\boldsymbol {\\beta }}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0054", "text": "is minimized. For example, it is common to use the sum of squared errors \n  \n    \n      \n        \u2016\n        \n          \u03b5\n        \n        \n          \u2016\n          \n            2\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\|{\\boldsymbol {\\varepsilon }}\\|_{2}^{2}}\n  \n as a measure of \n  \n    \n      \n        \n          \u03b5\n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\varepsilon }}}\n  \n for minimization.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0055", "text": "Example\nConsider a situation where a small ball is being tossed up in the air and then we measure its heights of ascent hi at various moments in time ti. Physics tells us that, ignoring the drag, the relationship can be modeled as", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0056", "text": "h\n          \n            i\n          \n        \n        =\n        \n          \u03b2\n          \n            1\n          \n        \n        \n          t\n          \n            i\n          \n        \n        +\n        \n          \u03b2\n          \n            2\n          \n        \n        \n          t\n          \n            i\n          \n          \n            2\n          \n        \n        +\n        \n          \u03b5\n          \n            i\n          \n        \n        ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0057", "text": ",\n      \n    \n    {\\displaystyle h_{i}=\\beta _{1}t_{i}+\\beta _{2}t_{i}^{2}+\\varepsilon _{i},}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0058", "text": "where \u03b21 determines the initial velocity of the ball, \u03b22 is proportional to the standard gravity, and \u03b5i is due to measurement errors. Linear regression can be used to estimate the values of \u03b21 and \u03b22 from the measured data. This model is non-linear in the time variable, but it is linear in the parameters \u03b21 and \u03b22; if we take regressors xi = (xi1, xi2)  = (ti, ti2), the model takes on the standard form", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0059", "text": "h\n          \n            i\n          \n        \n        =\n        \n          \n            x\n          \n          \n            i\n          \n          \n            \n              T\n            \n          \n        \n        \n          \u03b2\n        \n        +\n        \n          \u03b5\n          \n            i\n          \n        \n        .\n      \n    \n    {\\displaystyle h_{i}=\\mathbf {x} _{i}^{\\mathsf {T}}{\\boldsymbol {\\beta }}+\\varepsilon _{i}.}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0060", "text": "Assumptions", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0061", "text": "When estimating the parameters of linear regression models with standard estimation techniques such as ordinary least squares, it is necessary to make a number of assumptions about the predictor variables, the response variable and their relationship, to get estimators that are unbiased in finite sample. Numerous extensions have been developed that allow each of these assumptions to be relaxed (reduced to a weaker form), and in some cases eliminated entirely", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0062", "text": ". Generally these extensions require more data or modelling assumptions to produce an equally precise model.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0063", "text": "Interpretation", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0064", "text": "A fitted linear regression model can be used to identify the relationship between a single predictor variable xj and the response variable y when all the other predictor variables in the model are \"held fixed\". Specifically, the interpretation of \u03b2j is the expected change in y for a one-unit change in xj when the other covariates are held fixed\u2014that is, the expected value of the partial derivative of y with respect to xj. This is sometimes called the unique effect of xj on y", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0065", "text": ". In contrast, the marginal effect of xj on y can be assessed using a correlation coefficient or simple linear regression model relating only xj to y; this effect is the total derivative of y with respect to xj.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0066", "text": "Care must be taken when interpreting regression results, as some of the regressors may not allow for marginal changes (such as dummy variables, or the intercept term), while others cannot be held fixed (recall the example from the introduction: it would be impossible to \"hold ti fixed\" and at the same time change the value of ti2).", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0067", "text": "It is possible that the unique effect be nearly zero even when the marginal effect is large. This may imply that some other covariate captures all the information in xj, so that once that variable is in the model, there is no contribution of xj to the variation in y. Conversely, the unique effect of xj can be large while its marginal effect is nearly zero", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0068", "text": ". This would happen if the other covariates explained a great deal of the variation of y, but they mainly explain variation in a way that is complementary to what is captured by xj. In this case, including the other variables in the model reduces the part of the variability of y that is unrelated to xj, thereby strengthening the apparent relationship with xj.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0069", "text": "The meaning of the expression \"held fixed\" may depend on how the values of the predictor variables arise. If the experimenter directly sets the values of the predictor variables according to a study design, the comparisons of interest may literally correspond to comparisons among units whose predictor variables have been \"held fixed\" by the experimenter. Alternatively, the expression \"held fixed\" can refer to a selection that takes place in the context of data analysis", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0070", "text": ". In this case, we \"hold a variable fixed\" by restricting our attention to the subsets of the data that happen to have a common value for the given predictor variable. This is the only interpretation of \"held fixed\" that can be used in an observational study.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0071", "text": "The notion of a \"unique effect\" is appealing when studying a complex system where multiple interrelated components influence the response variable. In some cases, it can literally be interpreted as the causal effect of an intervention that is linked to the value of a predictor variable", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0072", "text": ". However, it has been argued that in many cases multiple regression analysis fails to clarify the relationships between the predictor variables and the response variable when the predictors are correlated with each other and are not assigned following a study design.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0073", "text": "Extensions\nNumerous extensions of linear regression have been developed, which allow some or all of the assumptions underlying the basic model to be relaxed.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0074", "text": "Simple and multiple linear regression\nThe simplest case of a single scalar predictor variable x and a single scalar response variable y is known as simple linear regression. The extension to multiple and/or vector-valued predictor variables (denoted with a capital X) is known as multiple linear regression, also known as multivariable linear regression (not to be confused with multivariate linear regression).", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0075", "text": "Multiple linear regression is a generalization of simple linear regression to the case of more than one independent variable, and a special case of general linear models, restricted to one dependent variable. The basic model for multiple linear regression is", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0076", "text": "Y\n          \n            i\n          \n        \n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        \n          X\n          \n            i\n            1\n          \n        \n        +\n        \n          \u03b2\n          \n            2\n          \n        \n        \n          X\n          \n            i\n            2\n          \n        \n        +\n        \u2026\n        +", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0077", "text": "+\n        \u2026\n        +\n        \n          \u03b2\n          \n            p\n          \n        \n        \n          X\n          \n            i\n            p\n          \n        \n        +\n        \n          \u03f5\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle Y_{i}=\\beta _{0}+\\beta _{1}X_{i1}+\\beta _{2}X_{i2}+\\ldots +\\beta _{p}X_{ip}+\\epsilon _{i}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0078", "text": "for each observation \n  \n    \n      \n        i\n        =\n        1\n        ,\n        \u2026\n        ,\n        n\n      \n    \n    {\\textstyle i=1,\\ldots ,n}\n  \n.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0079", "text": "{\\textstyle i=1,\\ldots ,n}\n  \n.\nIn the formula above we consider n observations of one dependent variable and p independent variables. Thus, Yi is the ith observation of the dependent variable, Xij is ith observation of the jth independent variable, j = 1, 2, ..., p. The values \u03b2j represent parameters to be estimated, and \u03b5i is the ith independent identically distributed normal error.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0080", "text": "In the more general multivariate linear regression, there is one equation of the above form for each of m > 1 dependent variables that share the same set of explanatory variables and hence are estimated simultaneously with each other:", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0081", "text": "Y\n          \n            i\n            j\n          \n        \n        =\n        \n          \u03b2\n          \n            0\n            j\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n            j\n          \n        \n        \n          X\n          \n            i\n            1\n          \n        \n        +\n        \n          \u03b2\n          \n            2\n            j\n          \n        \n        \n          X\n          \n            i", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0082", "text": "X\n          \n            i\n            2\n          \n        \n        +\n        \u2026\n        +\n        \n          \u03b2\n          \n            p\n            j\n          \n        \n        \n          X\n          \n            i\n            p\n          \n        \n        +\n        \n          \u03f5\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle Y_{ij}=\\beta _{0j}+\\beta _{1j}X_{i1}+\\beta _{2j}X_{i2}+\\ldots +\\beta _{pj}X_{ip}+\\epsilon _{ij}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0083", "text": "for all observations indexed as i = 1, ... , n and for all dependent variables indexed as j = 1, ... , m.\nNearly all real-world regression models involve multiple predictors, and basic descriptions of linear regression are often phrased in terms of the multiple regression model. Note, however, that in these cases the response variable y is still a scalar. Another term, multivariate linear regression, refers to cases where y is a vector, i.e., the same as general linear regression.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0084", "text": "General linear models\nThe general linear model considers the situation when the response variable is not a scalar (for each observation) but a vector, yi. Conditional linearity of \n  \n    \n      \n        E\n        (\n        \n          y\n        \n        \u2223\n        \n          \n            x\n          \n          \n            i\n          \n        \n        )\n        =\n        \n          \n            x\n          \n          \n            i\n          \n          \n            \n              T", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0085", "text": "T\n            \n          \n        \n        B\n      \n    \n    {\\displaystyle E(\\mathbf {y} \\mid \\mathbf {x} _{i})=\\mathbf {x} _{i}^{\\mathsf {T}}B}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0086", "text": "is still assumed, with a matrix B replacing the vector \u03b2 of the classical linear regression model. Multivariate analogues of ordinary least squares (OLS) and generalized least squares (GLS) have been developed. \"General linear models\" are also called \"multivariate linear models\". These are not the same as multivariable linear models (also called \"multiple linear models\").", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0087", "text": "Heteroscedastic models", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0088", "text": "Various models have been created that allow for heteroscedasticity, i.e. the errors for different response variables may have different variances. For example, weighted least squares is a method for estimating linear regression models when the response variables may have different error variances, possibly with correlated errors", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0089", "text": ". (See also Weighted linear least squares, and Generalized least squares.) Heteroscedasticity-consistent standard errors is an improved method for use with uncorrelated but potentially heteroscedastic errors.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0090", "text": "Generalized linear models\nThe Generalized linear model (GLM) is a framework for modeling response variables that are bounded or discrete. This is used, for example:", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0091", "text": "when modeling positive quantities (e.g. prices or populations) that vary over a large scale\u2014which are better described using a skewed distribution such as the log-normal distribution or Poisson distribution (although GLMs are not used for log-normal data, instead the response variable is simply transformed using the logarithm function);", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0092", "text": "when modeling categorical data, such as the choice of a given candidate in an election (which is better described using a Bernoulli distribution/binomial distribution for binary choices, or a categorical distribution/multinomial distribution for multi-way choices), where there are a fixed number of choices that cannot be meaningfully ordered;", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0093", "text": "when modeling ordinal data, e.g. ratings on a scale from 0 to 5, where the different outcomes can be ordered but where the quantity itself may not have any absolute meaning (e.g. a rating of 4 may not be \"twice as good\" in any objective sense as a rating of 2, but simply indicates that it is better than 2 or 3 but not as good as 5).\nGeneralized linear models allow for an arbitrary link function, g, that relates the mean of the response variable(s) to the predictors: \n  \n    \n      \n        E", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0094", "text": "E\n        (\n        Y\n        )\n        =\n        \n          g\n          \n            \u2212\n            1\n          \n        \n        (\n        X\n        B\n        )\n      \n    \n    {\\displaystyle E(Y)=g^{-1}(XB)}\n  \n. The link function is often related to the distribution of the response, and in particular it typically has the effect of transforming between the \n  \n    \n      \n        (\n        \u2212\n        \u221e\n        ,\n        \u221e\n        )", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0095", "text": ",\n        \u221e\n        )\n      \n    \n    {\\displaystyle (-\\infty ,\\infty )}\n  \n range of the linear predictor and the range of the response variable.\nSome common examples of GLMs are:", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0096", "text": "Poisson regression for count data.\nLogistic regression and probit regression for binary data.\nMultinomial logistic regression and multinomial probit regression for categorical data.\nOrdered logit and ordered probit regression for ordinal data.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0097", "text": "Single index models allow some degree of nonlinearity in the relationship between x and y, while preserving the central role of the linear predictor \u03b2\u2032x as in the classical linear regression model. Under certain conditions, simply applying OLS to data from a single-index model will consistently estimate \u03b2 up to a proportionality constant.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0098", "text": "Hierarchical linear models", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0099", "text": "Hierarchical linear models (or multilevel regression) organizes the data into a hierarchy of regressions, for example where A is regressed on B, and B is regressed on C. It is often used where the variables of interest have a natural hierarchical structure such as in educational statistics, where students are nested in classrooms, classrooms are nested in schools, and schools are nested in some administrative grouping, such as a school district", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0100", "text": ". The response variable might be a measure of student achievement such as a test score, and different covariates would be collected at the classroom, school, and school district levels.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0101", "text": "Errors-in-variables\nErrors-in-variables models (or \"measurement error models\") extend the traditional linear regression model to allow the predictor variables X to be observed with error. This error causes standard estimators of \u03b2 to become biased. Generally, the form of bias is an attenuation, meaning that the effects are biased toward zero.\n\nGroup effects\nIn a multiple linear regression model", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0102", "text": "y\n        =\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u03b2\n          \n            1\n          \n        \n        \n          x\n          \n            1\n          \n        \n        +\n        \u22ef\n        +\n        \n          \u03b2\n          \n            p\n          \n        \n        \n          x\n          \n            p\n          \n        \n        +\n        \u03b5\n        ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0103", "text": "+\n        \u03b5\n        ,\n      \n    \n    {\\displaystyle y=\\beta _{0}+\\beta _{1}x_{1}+\\cdots +\\beta _{p}x_{p}+\\varepsilon ,}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0104", "text": "parameter \n  \n    \n      \n        \n          \u03b2\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\beta _{j}}\n  \n of predictor variable \n  \n    \n      \n        \n          x\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle x_{j}}\n  \n represents the individual effect of \n  \n    \n      \n        \n          x\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle x_{j}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0105", "text": "{\\displaystyle x_{j}}\n  \n. It has an interpretation as the expected change in the response variable \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n when \n  \n    \n      \n        \n          x\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle x_{j}}\n  \n increases by one unit with other predictor variables held constant. When \n  \n    \n      \n        \n          x\n          \n            j", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0106", "text": "j\n          \n        \n      \n    \n    {\\displaystyle x_{j}}\n  \n is strongly correlated with other predictor variables, it is improbable that \n  \n    \n      \n        \n          x\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle x_{j}}\n  \n can increase by one unit with other variables held constant. In this case, the interpretation of \n  \n    \n      \n        \n          \u03b2\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\beta _{j}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0107", "text": "{\\displaystyle \\beta _{j}}\n  \n becomes problematic as it is based on an improbable condition, and the effect of \n  \n    \n      \n        \n          x\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle x_{j}}\n  \n cannot be evaluated in isolation.\nFor a group of predictor variables, say, \n  \n    \n      \n        {\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0108", "text": "2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            q\n          \n        \n        }\n      \n    \n    {\\displaystyle \\{x_{1},x_{2},\\dots ,x_{q}\\}}\n  \n, a group effect \n  \n    \n      \n        \u03be\n        (\n        \n          w\n        \n        )\n      \n    \n    {\\displaystyle \\xi (\\mathbf {w} )}\n  \n is defined as a linear combination of their parameters", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0109", "text": "\u03be\n        (\n        \n          w\n        \n        )\n        =\n        \n          w\n          \n            1\n          \n        \n        \n          \u03b2\n          \n            1\n          \n        \n        +\n        \n          w\n          \n            2\n          \n        \n        \n          \u03b2\n          \n            2\n          \n        \n        +\n        \u22ef\n        +\n        \n          w\n          \n            q\n          \n        \n        \n          \u03b2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0110", "text": "\u03b2\n          \n            q\n          \n        \n        ,\n      \n    \n    {\\displaystyle \\xi (\\mathbf {w} )=w_{1}\\beta _{1}+w_{2}\\beta _{2}+\\dots +w_{q}\\beta _{q},}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0111", "text": "where \n  \n    \n      \n        \n          w\n        \n        =\n        (\n        \n          w\n          \n            1\n          \n        \n        ,\n        \n          w\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          w\n          \n            q\n          \n        \n        \n          )\n          \n            \u22ba\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {w} =(w_{1},w_{2},\\dots ,w_{q})^{\\intercal }}\n  \n is a weight vector satisfying", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0112", "text": "is a weight vector satisfying \n  \n    \n      \n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            q\n          \n        \n        \n          |\n        \n        \n          w\n          \n            j\n          \n        \n        \n          |\n        \n        =\n        1\n      \n    \n    {\\textstyle \\sum _{j=1}^{q}|w_{j}|=1}\n  \n. Because of the constraint on \n  \n    \n      \n        \n          \n            w\n            \n              j", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0113", "text": "w\n            \n              j\n            \n          \n        \n      \n    \n    {\\displaystyle {w_{j}}}\n  \n, \n  \n    \n      \n        \u03be\n        (\n        \n          w\n        \n        )\n      \n    \n    {\\displaystyle \\xi (\\mathbf {w} )}\n  \n is also referred to as a normalized group effect. A group effect \n  \n    \n      \n        \u03be\n        (\n        \n          w\n        \n        )\n      \n    \n    {\\displaystyle \\xi (\\mathbf {w} )}\n  \n has an interpretation as the expected change in", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0114", "text": "y\n      \n    \n    {\\displaystyle y}\n  \n when variables in the group \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            q\n          \n        \n      \n    \n    {\\displaystyle x_{1},x_{2},\\dots ,x_{q}}\n  \n change by the amount \n  \n    \n      \n        \n          w\n          \n            1", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0115", "text": "w\n          \n            1\n          \n        \n        ,\n        \n          w\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          w\n          \n            q\n          \n        \n      \n    \n    {\\displaystyle w_{1},w_{2},\\dots ,w_{q}}\n  \n, respectively, at the same time with other variables (not in the group) held constant. It generalizes the individual effect of a variable to a group of variables in that (\n  \n    \n      \n        i", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0116", "text": "i\n      \n    \n    {\\displaystyle i}\n  \n) if \n  \n    \n      \n        q\n        =\n        1\n      \n    \n    {\\displaystyle q=1}\n  \n, then the group effect reduces to an individual effect, and (\n  \n    \n      \n        i\n        i\n      \n    \n    {\\displaystyle ii}\n  \n) if \n  \n    \n      \n        \n          w\n          \n            i\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle w_{i}=1}\n  \n and \n  \n    \n      \n        \n          w", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0117", "text": "w\n          \n            j\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle w_{j}=0}\n  \n for \n  \n    \n      \n        j\n        \u2260\n        i\n      \n    \n    {\\displaystyle j\\neq i}\n  \n, then the group effect also reduces to an individual effect.\nA group effect \n  \n    \n      \n        \u03be\n        (\n        \n          w\n        \n        )\n      \n    \n    {\\displaystyle \\xi (\\mathbf {w} )}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0118", "text": "{\\displaystyle \\xi (\\mathbf {w} )}\n  \n is said to be meaningful if the underlying simultaneous changes of the \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n variables \n  \n    \n      \n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            q\n          \n        \n        \n          )", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0119", "text": ")\n          \n            \u22ba\n          \n        \n      \n    \n    {\\displaystyle (x_{1},x_{2},\\dots ,x_{q})^{\\intercal }}\n  \n is probable.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0120", "text": "Group effects provide a means to study the collective impact of strongly correlated predictor variables in linear regression models. Individual effects of such variables are not well-defined as their parameters do not have good interpretations. Furthermore, when the sample size is not large, none of their parameters can be accurately estimated by the least squares regression due to the multicollinearity problem", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0121", "text": ". Nevertheless, there are meaningful group effects that have good interpretations and can be accurately estimated by the least squares regression. A simple way to identify these meaningful group effects is to use an all positive correlations (APC) arrangement of the strongly correlated variables under which pairwise correlations among these variables are all positive, and standardize all", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0122", "text": "p\n      \n    \n    {\\displaystyle p}\n  \n predictor variables in the model so that they all have mean zero and length one. To illustrate this, suppose that \n  \n    \n      \n        {\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            q\n          \n        \n        }", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0123", "text": "}\n      \n    \n    {\\displaystyle \\{x_{1},x_{2},\\dots ,x_{q}\\}}\n  \n is a group of strongly correlated variables in an APC arrangement and that they are not strongly correlated with predictor variables outside the group. Let \n  \n    \n      \n        \n          y\n          \u2032\n        \n      \n    \n    {\\displaystyle y'}\n  \n be the centred \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n and \n  \n    \n      \n        \n          x\n          \n            j", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0124", "text": "x\n          \n            j\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle x_{j}'}\n  \n be the standardized \n  \n    \n      \n        \n          x\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle x_{j}}\n  \n. Then, the standardized linear regression model is", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0125", "text": "y\n          \u2032\n        \n        =\n        \n          \u03b2\n          \n            1\n          \n          \u2032\n        \n        \n          x\n          \n            1\n          \n          \u2032\n        \n        +\n        \u22ef\n        +\n        \n          \u03b2\n          \n            p\n          \n          \u2032\n        \n        \n          x\n          \n            p\n          \n          \u2032\n        \n        +\n        \u03b5\n        .", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0126", "text": "+\n        \u03b5\n        .\n      \n    \n    {\\displaystyle y'=\\beta _{1}'x_{1}'+\\cdots +\\beta _{p}'x_{p}'+\\varepsilon .}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0127", "text": "Parameters \n  \n    \n      \n        \n          \u03b2\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\beta _{j}}\n  \n in the original model, including \n  \n    \n      \n        \n          \u03b2\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\beta _{0}}\n  \n, are simple functions of \n  \n    \n      \n        \n          \u03b2\n          \n            j\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\beta _{j}'}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0128", "text": "{\\displaystyle \\beta _{j}'}\n  \n in the standardized model. The standardization of variables does not change their correlations, so \n  \n    \n      \n        {\n        \n          x\n          \n            1\n          \n          \u2032\n        \n        ,\n        \n          x\n          \n            2\n          \n          \u2032\n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            q\n          \n          \u2032\n        \n        }", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0129", "text": "\u2032\n        \n        }\n      \n    \n    {\\displaystyle \\{x_{1}',x_{2}',\\dots ,x_{q}'\\}}\n  \n is a group of strongly correlated variables in an APC arrangement and they are not strongly correlated with other predictor variables in the standardized model. A group effect of \n  \n    \n      \n        {\n        \n          x\n          \n            1\n          \n          \u2032\n        \n        ,\n        \n          x\n          \n            2\n          \n          \u2032\n        \n        ,\n        \u2026\n        ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0130", "text": ",\n        \u2026\n        ,\n        \n          x\n          \n            q\n          \n          \u2032\n        \n        }\n      \n    \n    {\\displaystyle \\{x_{1}',x_{2}',\\dots ,x_{q}'\\}}\n  \n is", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0131", "text": "\u03be\n          \u2032\n        \n        (\n        \n          w\n        \n        )\n        =\n        \n          w\n          \n            1\n          \n        \n        \n          \u03b2\n          \n            1\n          \n          \u2032\n        \n        +\n        \n          w\n          \n            2\n          \n        \n        \n          \u03b2\n          \n            2\n          \n          \u2032\n        \n        +\n        \u22ef\n        +\n        \n          w\n          \n            q", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0132", "text": "w\n          \n            q\n          \n        \n        \n          \u03b2\n          \n            q\n          \n          \u2032\n        \n        ,\n      \n    \n    {\\displaystyle \\xi '(\\mathbf {w} )=w_{1}\\beta _{1}'+w_{2}\\beta _{2}'+\\dots +w_{q}\\beta _{q}',}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0133", "text": "and its minimum-variance unbiased linear estimator is", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0134", "text": "\u03be\n                ^\n              \n            \n          \n          \u2032\n        \n        (\n        \n          w\n        \n        )\n        =\n        \n          w\n          \n            1\n          \n        \n        \n          \n            \n              \n                \u03b2\n                ^\n              \n            \n          \n          \n            1\n          \n          \u2032\n        \n        +\n        \n          w", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0135", "text": "+\n        \n          w\n          \n            2\n          \n        \n        \n          \n            \n              \n                \u03b2\n                ^\n              \n            \n          \n          \n            2\n          \n          \u2032\n        \n        +\n        \u22ef\n        +\n        \n          w\n          \n            q\n          \n        \n        \n          \n            \n              \n                \u03b2\n                ^", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0136", "text": "q\n          \n          \u2032\n        \n        ,\n      \n    \n    {\\displaystyle {\\hat {\\xi }}'(\\mathbf {w} )=w_{1}{\\hat {\\beta }}_{1}'+w_{2}{\\hat {\\beta }}_{2}'+\\dots +w_{q}{\\hat {\\beta }}_{q}',}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0137", "text": "where \n  \n    \n      \n        \n          \n            \n              \n                \u03b2\n                ^\n              \n            \n          \n          \n            j\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle {\\hat {\\beta }}_{j}'}\n  \n is the least squares estimator of \n  \n    \n      \n        \n          \u03b2\n          \n            j\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\beta _{j}'}\n  \n. In particular, the average group effect of the", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0138", "text": "q\n      \n    \n    {\\displaystyle q}\n  \n standardized variables is", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0139", "text": "\u03be\n          \n            A\n          \n        \n        =\n        \n          \n            1\n            q\n          \n        \n        (\n        \n          \u03b2\n          \n            1\n          \n          \u2032\n        \n        +\n        \n          \u03b2\n          \n            2\n          \n          \u2032\n        \n        +\n        \u22ef\n        +\n        \n          \u03b2\n          \n            q\n          \n          \u2032\n        \n        )\n        ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0140", "text": ")\n        ,\n      \n    \n    {\\displaystyle \\xi _{A}={\\frac {1}{q}}(\\beta _{1}'+\\beta _{2}'+\\dots +\\beta _{q}'),}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0141", "text": "which has an interpretation as the expected change in \n  \n    \n      \n        \n          y\n          \u2032\n        \n      \n    \n    {\\displaystyle y'}\n  \n when all \n  \n    \n      \n        \n          x\n          \n            j\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle x_{j}'}\n  \n in the strongly correlated group increase by \n  \n    \n      \n        (\n        1\n        \n          /\n        \n        q\n        )\n      \n    \n    {\\displaystyle (1/q)}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0142", "text": "{\\displaystyle (1/q)}\n  \nth of a unit at the same time with variables outside the group held constant. With strong positive correlations and in standardized units, variables in the group are approximately equal, so they are likely to increase at the same time and in similar amount. Thus, the average group effect \n  \n    \n      \n        \n          \u03be\n          \n            A\n          \n        \n      \n    \n    {\\displaystyle \\xi _{A}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0143", "text": "{\\displaystyle \\xi _{A}}\n  \n is a meaningful effect. It can be accurately estimated by its minimum-variance unbiased linear estimator \n  \n    \n      \n        \n          \n            \n              \n                \u03be\n                ^\n              \n            \n          \n          \n            A\n          \n        \n        =\n        \n          \n            1\n            q\n          \n        \n        (\n        \n          \n            \n              \n                \u03b2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0144", "text": "\u03b2\n                ^\n              \n            \n          \n          \n            1\n          \n          \u2032\n        \n        +\n        \n          \n            \n              \n                \u03b2\n                ^\n              \n            \n          \n          \n            2\n          \n          \u2032\n        \n        +\n        \u22ef\n        +\n        \n          \n            \n              \n                \u03b2\n                ^", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0145", "text": "^\n              \n            \n          \n          \n            q\n          \n          \u2032\n        \n        )\n      \n    \n    {\\textstyle {\\hat {\\xi }}_{A}={\\frac {1}{q}}({\\hat {\\beta }}_{1}'+{\\hat {\\beta }}_{2}'+\\dots +{\\hat {\\beta }}_{q}')}\n  \n, even when individually none of the \n  \n    \n      \n        \n          \u03b2\n          \n            j\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\beta _{j}'}\n  \n can be accurately estimated by", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0146", "text": "\u03b2\n                ^\n              \n            \n          \n          \n            j\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle {\\hat {\\beta }}_{j}'}\n  \n.\nNot all group effects are meaningful or can be accurately estimated. For example, \n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\beta _{1}'}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0147", "text": "{\\displaystyle \\beta _{1}'}\n  \n is a special group effect with weights \n  \n    \n      \n        \n          w\n          \n            1\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle w_{1}=1}\n  \n and \n  \n    \n      \n        \n          w\n          \n            j\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle w_{j}=0}\n  \n for \n  \n    \n      \n        j\n        \u2260\n        1\n      \n    \n    {\\displaystyle j\\neq 1}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0148", "text": "{\\displaystyle j\\neq 1}\n  \n, but it cannot be accurately estimated by \n  \n    \n      \n        \n          \n            \n              \n                \u03b2\n                ^\n              \n            \n          \n          \n            1\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle {\\hat {\\beta }}'_{1}}\n  \n. It is also not a meaningful effect. In general, for a group of \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0149", "text": "q\n      \n    \n    {\\displaystyle q}\n  \n strongly correlated predictor variables in an APC arrangement in the standardized model, group effects whose weight vectors \n  \n    \n      \n        \n          w\n        \n      \n    \n    {\\displaystyle \\mathbf {w} }\n  \n are at or near the centre of the simplex \n  \n    \n      \n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            q\n          \n        \n        \n          w\n          \n            j", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0150", "text": "w\n          \n            j\n          \n        \n        =\n        1\n      \n    \n    {\\textstyle \\sum _{j=1}^{q}w_{j}=1}\n  \n (\n  \n    \n      \n        \n          w\n          \n            j\n          \n        \n        \u2265\n        0\n      \n    \n    {\\displaystyle w_{j}\\geq 0}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0151", "text": "{\\displaystyle w_{j}\\geq 0}\n  \n) are meaningful and can be accurately estimated by their minimum-variance unbiased linear estimators. Effects with weight vectors far away from the centre are not meaningful as such weight vectors represent simultaneous changes of the variables that violate the strong positive correlations of the standardized variables in an APC arrangement. As such, they are not probable. These effects also cannot be accurately estimated.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0152", "text": "Applications of the group effects include (1) estimation and inference for meaningful group effects on the response variable, (2) testing for \"group significance\" of the \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n variables via testing \n  \n    \n      \n        \n          H\n          \n            0\n          \n        \n        :\n        \n          \u03be\n          \n            A\n          \n        \n        =\n        0\n      \n    \n    {\\displaystyle H_{0}:\\xi _{A}=0}\n  \n versus", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0153", "text": "versus \n  \n    \n      \n        \n          H\n          \n            1\n          \n        \n        :\n        \n          \u03be\n          \n            A\n          \n        \n        \u2260\n        0\n      \n    \n    {\\displaystyle H_{1}:\\xi _{A}\\neq 0}\n  \n, and (3) characterizing the region of the predictor variable space over which predictions by the least squares estimated model are accurate.\nA group effect of the original variables \n  \n    \n      \n        {\n        \n          x\n          \n            1", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0154", "text": "x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            q\n          \n        \n        }\n      \n    \n    {\\displaystyle \\{x_{1},x_{2},\\dots ,x_{q}\\}}\n  \n can be expressed as a constant times a group effect of the standardized variables \n  \n    \n      \n        {\n        \n          x\n          \n            1\n          \n          \u2032", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0155", "text": "1\n          \n          \u2032\n        \n        ,\n        \n          x\n          \n            2\n          \n          \u2032\n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            q\n          \n          \u2032\n        \n        }\n      \n    \n    {\\displaystyle \\{x_{1}',x_{2}',\\dots ,x_{q}'\\}}\n  \n. The former is meaningful when the latter is. Thus meaningful group effects of the original variables can be found through meaningful group effects of the standardized variables.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0156", "text": "Others\nIn Dempster\u2013Shafer theory, or a linear belief function in particular, a linear regression model may be represented as a partially swept matrix, which can be combined with similar matrices representing observations and other assumed normal distributions and state equations. The combination of swept or unswept matrices provides an alternative method for estimating linear regression models.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0157", "text": "Estimation methods\nA large number of procedures have been developed for parameter estimation and inference in linear regression. These methods differ in computational simplicity of algorithms, presence of a closed-form solution, robustness with respect to heavy-tailed distributions, and theoretical assumptions needed to validate desirable statistical properties such as consistency and asymptotic efficiency.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0158", "text": "Some of the more common estimation techniques for linear regression are summarized below.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0159", "text": "Least-squares estimation and related techniques\nAssuming that the independent variables are \n  \n    \n      \n        \n          \n            \n              \n                x\n                \n                  i\n                \n              \n              \u2192\n            \n          \n        \n        =\n        \n          [\n          \n            \n              x\n              \n                1\n              \n              \n                i\n              \n            \n            ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0160", "text": ",\n            \n              x\n              \n                2\n              \n              \n                i\n              \n            \n            ,\n            \u2026\n            ,\n            \n              x\n              \n                m\n              \n              \n                i\n              \n            \n          \n          ]\n        \n      \n    \n    {\\displaystyle {\\vec {x_{i}}}=\\left[x_{1}^{i},x_{2}^{i},\\ldots ,x_{m}^{i}\\right]}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0161", "text": "and the model's parameters are \n  \n    \n      \n        \n          \n            \n              \u03b2\n              \u2192\n            \n          \n        \n        =\n        \n          [\n          \n            \n              \u03b2\n              \n                0\n              \n            \n            ,\n            \n              \u03b2\n              \n                1\n              \n            \n            ,\n            \u2026\n            ,\n            \n              \u03b2\n              \n                m", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0162", "text": "\u03b2\n              \n                m\n              \n            \n          \n          ]\n        \n      \n    \n    {\\displaystyle {\\vec {\\beta }}=\\left[\\beta _{0},\\beta _{1},\\ldots ,\\beta _{m}\\right]}\n  \n, then the model's prediction would be", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0163", "text": "y\n          \n            i\n          \n        \n        \u2248\n        \n          \u03b2\n          \n            0\n          \n        \n        +\n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            m\n          \n        \n        \n          \u03b2\n          \n            j\n          \n        \n        \u00d7\n        \n          x\n          \n            j\n          \n          \n            i", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0164", "text": "i\n          \n        \n      \n    \n    {\\displaystyle y_{i}\\approx \\beta _{0}+\\sum _{j=1}^{m}\\beta _{j}\\times x_{j}^{i}}\n  \n.\nIf \n  \n    \n      \n        \n          \n            \n              \n                x\n                \n                  i\n                \n              \n              \u2192\n            \n          \n        \n      \n    \n    {\\displaystyle {\\vec {x_{i}}}}\n  \n is extended to \n  \n    \n      \n        \n          \n            \n              \n                x", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0165", "text": "x\n                \n                  i\n                \n              \n              \u2192\n            \n          \n        \n        =\n        \n          [\n          \n            1\n            ,\n            \n              x\n              \n                1\n              \n              \n                i\n              \n            \n            ,\n            \n              x\n              \n                2\n              \n              \n                i", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0166", "text": "i\n              \n            \n            ,\n            \u2026\n            ,\n            \n              x\n              \n                m\n              \n              \n                i\n              \n            \n          \n          ]\n        \n      \n    \n    {\\displaystyle {\\vec {x_{i}}}=\\left[1,x_{1}^{i},x_{2}^{i},\\ldots ,x_{m}^{i}\\right]}\n  \n then \n  \n    \n      \n        \n          y\n          \n            i", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0167", "text": "i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}\n  \n would become a dot product of the parameter and the independent vectors, i.e.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0168", "text": "y\n          \n            i\n          \n        \n        \u2248\n        \n          \u2211\n          \n            j\n            =\n            0\n          \n          \n            m\n          \n        \n        \n          \u03b2\n          \n            j\n          \n        \n        \u00d7\n        \n          x\n          \n            j\n          \n          \n            i\n          \n        \n        =\n        \n          \n            \n              \u03b2\n              \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0169", "text": "\u2192\n            \n          \n        \n        \u22c5\n        \n          \n            \n              \n                x\n                \n                  i\n                \n              \n              \u2192\n            \n          \n        \n      \n    \n    {\\displaystyle y_{i}\\approx \\sum _{j=0}^{m}\\beta _{j}\\times x_{j}^{i}={\\vec {\\beta }}\\cdot {\\vec {x_{i}}}}\n  \n.\nIn the least-squares setting, the optimum parameter vector is defined as such that minimizes the sum of mean squared loss:", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0170", "text": "\u03b2\n                  ^\n                \n              \n              \u2192\n            \n          \n        \n        =\n        \n          \n            \n              arg min\n            \n            \n              \n                \u03b2\n                \u2192\n              \n            \n          \n        \n        \n        L\n        \n          (\n          \n            D\n            ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0171", "text": ",\n            \n              \n                \n                  \u03b2\n                  \u2192\n                \n              \n            \n          \n          )\n        \n        =\n        \n          \n            \n              arg min\n            \n            \n              \n                \u03b2\n                \u2192\n              \n            \n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0172", "text": "n\n          \n        \n        \n          \n            (\n            \n              \n                \n                  \n                    \u03b2\n                    \u2192\n                  \n                \n              \n              \u22c5\n              \n                \n                  \n                    \n                      x\n                      \n                        i\n                      \n                    \n                    \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0173", "text": "\u2192\n                  \n                \n              \n              \u2212\n              \n                y\n                \n                  i\n                \n              \n            \n            )\n          \n          \n            2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0174", "text": "2\n          \n        \n      \n    \n    {\\displaystyle {\\vec {\\hat {\\beta }}}={\\underset {\\vec {\\beta }}{\\mbox{arg min}}}\\,L\\left(D,{\\vec {\\beta }}\\right)={\\underset {\\vec {\\beta }}{\\mbox{arg min}}}\\sum _{i=1}^{n}\\left({\\vec {\\beta }}\\cdot {\\vec {x_{i}}}-y_{i}\\right)^{2}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0175", "text": "Now putting the independent and dependent variables in matrices \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n and \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n respectively, the loss function can be rewritten as:", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0176", "text": "L\n                \n                  (\n                  \n                    D\n                    ,\n                    \n                      \n                        \n                          \u03b2\n                          \u2192\n                        \n                      \n                    \n                  \n                  )\n                \n              \n              \n                \n                =", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0177", "text": "=\n                \u2016\n                X\n                \n                  \n                    \n                      \u03b2\n                      \u2192\n                    \n                  \n                \n                \u2212\n                Y\n                \n                  \u2016\n                  \n                    2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0178", "text": "=\n                \n                  \n                    (\n                    \n                      X\n                      \n                        \n                          \n                            \u03b2\n                            \u2192\n                          \n                        \n                      \n                      \u2212\n                      Y\n                    \n                    )", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0179", "text": ")\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  (\n                  \n                    X\n                    \n                      \n                        \n                          \u03b2\n                          \u2192\n                        \n                      \n                    \n                    \u2212\n                    Y", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0180", "text": "Y\n                  \n                  )\n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  Y\n                  \n                    \n                      T\n                    \n                  \n                \n                Y\n                \u2212\n                \n                  Y\n                  \n                    \n                      T", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0181", "text": "T\n                    \n                  \n                \n                X\n                \n                  \n                    \n                      \u03b2\n                      \u2192\n                    \n                  \n                \n                \u2212\n                \n                  \n                    \n                      \n                        \u03b2\n                        \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0182", "text": "T\n                    \n                  \n                \n                \n                  X\n                  \n                    \n                      T\n                    \n                  \n                \n                Y\n                +\n                \n                  \n                    \n                      \n                        \u03b2\n                        \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0183", "text": "\u2192\n                      \n                    \n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  X\n                  \n                    \n                      T\n                    \n                  \n                \n                X\n                \n                  \n                    \n                      \u03b2\n                      \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0184", "text": "\u03b2\n                      \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0185", "text": "{\\displaystyle {\\begin{aligned}L\\left(D,{\\vec {\\beta }}\\right)&=\\|X{\\vec {\\beta }}-Y\\|^{2}\\\\&=\\left(X{\\vec {\\beta }}-Y\\right)^{\\textsf {T}}\\left(X{\\vec {\\beta }}-Y\\right)\\\\&=Y^{\\textsf {T}}Y-Y^{\\textsf {T}}X{\\vec {\\beta }}-{\\vec {\\beta }}^{\\textsf {T}}X^{\\textsf {T}}Y+{\\vec {\\beta }}^{\\textsf {T}}X^{\\textsf {T}}X{\\vec {\\beta }}\\end{aligned}}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0186", "text": "As the loss function is convex, the optimum solution lies at gradient zero. The gradient of the loss function is (using Denominator layout convention):", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0187", "text": "\u2202\n                      L\n                      \n                        (\n                        \n                          D\n                          ,\n                          \n                            \n                              \n                                \u03b2\n                                \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0188", "text": ")\n                      \n                    \n                    \n                      \u2202\n                      \n                        \n                          \n                            \u03b2\n                            \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0189", "text": "=\n                \n                  \n                    \n                      \u2202\n                      \n                        (\n                        \n                          \n                            Y\n                            \n                              \n                                T", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0190", "text": "Y\n                          \u2212\n                          \n                            Y\n                            \n                              \n                                T\n                              \n                            \n                          \n                          X\n                          \n                            \n                              \n                                \u03b2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0191", "text": "\u03b2\n                                \u2192\n                              \n                            \n                          \n                          \u2212\n                          \n                            \n                              \n                                \n                                  \u03b2\n                                  \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0192", "text": "T\n                              \n                            \n                          \n                          \n                            X\n                            \n                              \n                                T\n                              \n                            \n                          \n                          Y\n                          +", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0193", "text": "+\n                          \n                            \n                              \n                                \n                                  \u03b2\n                                  \u2192\n                                \n                              \n                            \n                            \n                              \n                                T", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0194", "text": "X\n                            \n                              \n                                T\n                              \n                            \n                          \n                          X\n                          \n                            \n                              \n                                \u03b2\n                                \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0195", "text": ")\n                      \n                    \n                    \n                      \u2202\n                      \n                        \n                          \n                            \u03b2\n                            \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0196", "text": "=\n                \u2212\n                2\n                \n                  X\n                  \n                    \n                      T\n                    \n                  \n                \n                Y\n                +\n                2\n                \n                  X\n                  \n                    \n                      T", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0197", "text": "T\n                    \n                  \n                \n                X\n                \n                  \n                    \n                      \u03b2\n                      \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0198", "text": "{\\displaystyle {\\begin{aligned}{\\frac {\\partial L\\left(D,{\\vec {\\beta }}\\right)}{\\partial {\\vec {\\beta }}}}&={\\frac {\\partial \\left(Y^{\\textsf {T}}Y-Y^{\\textsf {T}}X{\\vec {\\beta }}-{\\vec {\\beta }}^{\\textsf {T}}X^{\\textsf {T}}Y+{\\vec {\\beta }}^{\\textsf {T}}X^{\\textsf {T}}X{\\vec {\\beta }}\\right)}{\\partial {\\vec {\\beta }}}}\\\\&=-2X^{\\textsf {T}}Y+2X^{\\textsf {T}}X{\\vec {\\beta }}\\end{aligned}}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0199", "text": "Setting the gradient to zero produces the optimum parameter:", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0200", "text": "\u2212\n                2\n                \n                  X\n                  \n                    \n                      T\n                    \n                  \n                \n                Y\n                +\n                2\n                \n                  X\n                  \n                    \n                      T\n                    \n                  \n                \n                X", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0201", "text": "X\n                \n                  \n                    \n                      \u03b2\n                      \u2192\n                    \n                  \n                \n              \n              \n                \n                =\n                0\n              \n            \n            \n              \n                \u21d2\n                \n                  X\n                  \n                    \n                      T", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0202", "text": "X\n                \n                  \n                    \n                      \u03b2\n                      \u2192\n                    \n                  \n                \n              \n              \n                \n                =\n                \n                  X\n                  \n                    \n                      T\n                    \n                  \n                \n                Y", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0203", "text": "Y\n              \n            \n            \n              \n                \u21d2\n                \n                  \n                    \n                      \n                        \n                          \u03b2\n                          ^\n                        \n                      \n                      \u2192\n                    \n                  \n                \n              \n              \n                \n                =", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0204", "text": "(\n                    \n                      \n                        X\n                        \n                          \n                            T\n                          \n                        \n                      \n                      X\n                    \n                    )\n                  \n                  \n                    \u2212\n                    1", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0205", "text": "X\n                  \n                    \n                      T\n                    \n                  \n                \n                Y", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0206", "text": "{\\displaystyle {\\begin{aligned}-2X^{\\textsf {T}}Y+2X^{\\textsf {T}}X{\\vec {\\beta }}&=0\\\\\\Rightarrow X^{\\textsf {T}}X{\\vec {\\beta }}&=X^{\\textsf {T}}Y\\\\\\Rightarrow {\\vec {\\hat {\\beta }}}&=\\left(X^{\\textsf {T}}X\\right)^{-1}X^{\\textsf {T}}Y\\end{aligned}}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0207", "text": "Note: The \n  \n    \n      \n        \n          \n            \n              \u03b2\n              ^\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {\\beta }}}\n  \n obtained may indeed be the local minimum, one needs to differentiate once more to obtain the Hessian matrix and show that it is positive definite. This is provided by the Gauss\u2013Markov theorem.\nLinear least squares methods include mainly:", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0208", "text": "Ordinary least squares\nWeighted least squares\nGeneralized least squares\nLinear Template Fit", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0209", "text": "Maximum-likelihood estimation and related techniques\nMaximum likelihood estimation", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0210", "text": "Maximum likelihood estimation\nMaximum likelihood estimation can be performed when the distribution of the error terms is known to belong to a certain parametric family \u0192\u03b8 of probability distributions. When f\u03b8 is a normal distribution with zero mean and variance \u03b8, the resulting estimate is identical to the OLS estimate. GLS estimates are maximum likelihood estimates when \u03b5 follows a multivariate normal distribution with a known covariance matrix.\nLet's denote each data point by", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0211", "text": "Let's denote each data point by \n  \n    \n      \n        (\n        \n          \n            \n              \n                x\n                \n                  i\n                \n              \n              \u2192\n            \n          \n        \n        ,\n        \n          y\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle ({\\vec {x_{i}}},y_{i})}\n  \n and the regression parameters as \n  \n    \n      \n        \n          \n            \n              \u03b2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0212", "text": "\u03b2\n              \u2192\n            \n          \n        \n      \n    \n    {\\displaystyle {\\vec {\\beta }}}\n  \n, and the set of all data by \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n and the cost function by \n  \n    \n      \n        L\n        (\n        D\n        ,\n        \n          \n            \n              \u03b2\n              \u2192\n            \n          \n        \n        )\n        =\n        \n          \u2211\n          \n            i", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0213", "text": "\u2211\n          \n            i\n          \n        \n        (\n        \n          y\n          \n            i\n          \n        \n        \u2212\n        \n          \n            \n              \u03b2\n              \u2192\n            \n          \n        \n        \n        \u22c5\n        \n        \n          \n            \n              \n                x\n                \n                  i\n                \n              \n              \u2192\n            \n          \n        \n        \n          )\n          \n            2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0214", "text": ")\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle L(D,{\\vec {\\beta }})=\\sum _{i}(y_{i}-{\\vec {\\beta }}\\,\\cdot \\,{\\vec {x_{i}}})^{2}}\n  \n.\nAs shown below the same optimal parameter that minimizes \n  \n    \n      \n        L\n        (\n        D\n        ,\n        \n          \n            \n              \u03b2\n              \u2192\n            \n          \n        \n        )\n      \n    \n    {\\displaystyle L(D,{\\vec {\\beta }})}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0215", "text": "{\\displaystyle L(D,{\\vec {\\beta }})}\n  \n achieves maximum likelihood too. Here the assumption is that the dependent variable \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n is a random variable that follows a Gaussian distribution, where the standard deviation is fixed and the mean is a linear combination of \n  \n    \n      \n        \n          \n            \n              x\n              \u2192\n            \n          \n        \n      \n    \n    {\\displaystyle {\\vec {x}}}\n  \n:", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0216", "text": "{\\displaystyle {\\vec {x}}}\n  \n:\n  \n    \n      \n        \n          \n            \n              \n                H\n                (\n                D\n                ,\n                \n                  \n                    \n                      \u03b2\n                      \u2192\n                    \n                  \n                \n                )\n              \n              \n                \n                =\n                \n                  \u220f\n                  \n                    i", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0217", "text": "i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                P\n                r\n                (\n                \n                  y\n                  \n                    i\n                  \n                \n                \n                  |", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0218", "text": "x\n                        \n                          i\n                        \n                      \n                      \u2192\n                    \n                  \n                \n                \n                \n                ;\n                \n                  \n                    \n                      \u03b2\n                      \u2192\n                    \n                  \n                \n                ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0219", "text": ",\n                \u03c3\n                )\n              \n            \n            \n              \n              \n                \n                =\n                \n                  \u220f\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  \n                    1", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0220", "text": "2\n                          \u03c0\n                        \n                      \n                      \u03c3\n                    \n                  \n                \n                exp\n                \u2061\n                \n                  (\n                  \n                    \u2212\n                    \n                      \n                        \n                          \n                            (", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0221", "text": "(\n                            \n                              \n                                y\n                                \n                                  i\n                                \n                              \n                              \u2212\n                              \n                                \n                                  \n                                    \u03b2\n                                    \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0222", "text": "\u22c5\n                              \n                              \n                                \n                                  \n                                    \n                                      x\n                                      \n                                        i", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0223", "text": "\u2192\n                                  \n                                \n                              \n                            \n                            )\n                          \n                          \n                            2\n                          \n                        \n                        \n                          2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0224", "text": "\u03c3\n                            \n                              2\n                            \n                          \n                        \n                      \n                    \n                  \n                  )", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0225", "text": "{\\displaystyle {\\begin{aligned}H(D,{\\vec {\\beta }})&=\\prod _{i=1}^{n}Pr(y_{i}|{\\vec {x_{i}}}\\,\\,;{\\vec {\\beta }},\\sigma )\\\\&=\\prod _{i=1}^{n}{\\frac {1}{{\\sqrt {2\\pi }}\\sigma }}\\exp \\left(-{\\frac {\\left(y_{i}-{\\vec {\\beta }}\\,\\cdot \\,{\\vec {x_{i}}}\\right)^{2}}{2\\sigma ^{2}}}\\right)\\end{aligned}}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0226", "text": "Now, we need to look for a parameter that maximizes this likelihood function. Since the logarithmic function is strictly increasing, instead of maximizing this function, we can also maximize its logarithm and find the optimal parameter that way.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0227", "text": "I\n                (\n                D\n                ,\n                \n                  \n                    \n                      \u03b2\n                      \u2192\n                    \n                  \n                \n                )\n              \n              \n                \n                =\n                log\n                \u2061\n                \n                  \u220f\n                  \n                    i", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0228", "text": "i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                P\n                r\n                (\n                \n                  y\n                  \n                    i\n                  \n                \n                \n                  |", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0229", "text": "x\n                        \n                          i\n                        \n                      \n                      \u2192\n                    \n                  \n                \n                \n                \n                ;\n                \n                  \n                    \n                      \u03b2\n                      \u2192\n                    \n                  \n                \n                ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0230", "text": ",\n                \u03c3\n                )\n              \n            \n            \n              \n              \n                \n                =\n                log\n                \u2061\n                \n                  \u220f\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  \n                    1", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0231", "text": "1\n                    \n                      \n                        \n                          2\n                          \u03c0\n                        \n                      \n                      \u03c3\n                    \n                  \n                \n                exp\n                \u2061\n                \n                  (\n                  \n                    \u2212", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0232", "text": "(\n                            \n                              \n                                y\n                                \n                                  i\n                                \n                              \n                              \u2212", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0233", "text": "\u03b2\n                                    \u2192\n                                  \n                                \n                              \n                              \n                              \u22c5\n                              \n                              \n                                \n                                  \n                                    \n                                      x", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0234", "text": "x\n                                      \n                                        i\n                                      \n                                    \n                                    \u2192\n                                  \n                                \n                              \n                            \n                            )\n                          \n                          \n                            2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0235", "text": "2\n                          \n                        \n                        \n                          2\n                          \n                            \u03c3\n                            \n                              2\n                            \n                          \n                        \n                      \n                    \n                  \n                  )", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0236", "text": "=\n                n\n                log\n                \u2061\n                \n                  \n                    1\n                    \n                      \n                        \n                          2\n                          \u03c0\n                        \n                      \n                      \u03c3\n                    \n                  \n                \n                \u2212", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0237", "text": "\u2212\n                \n                  \n                    1\n                    \n                      2\n                      \n                        \u03c3\n                        \n                          2\n                        \n                      \n                    \n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0238", "text": "n\n                  \n                \n                \n                  \n                    (\n                    \n                      \n                        y\n                        \n                          i\n                        \n                      \n                      \u2212\n                      \n                        \n                          \n                            \u03b2\n                            \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0239", "text": "\u2192\n                          \n                        \n                      \n                      \n                      \u22c5\n                      \n                      \n                        \n                          \n                            \n                              x\n                              \n                                i\n                              \n                            \n                            \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0240", "text": ")\n                  \n                  \n                    2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0241", "text": "{\\displaystyle {\\begin{aligned}I(D,{\\vec {\\beta }})&=\\log \\prod _{i=1}^{n}Pr(y_{i}|{\\vec {x_{i}}}\\,\\,;{\\vec {\\beta }},\\sigma )\\\\&=\\log \\prod _{i=1}^{n}{\\frac {1}{{\\sqrt {2\\pi }}\\sigma }}\\exp \\left(-{\\frac {\\left(y_{i}-{\\vec {\\beta }}\\,\\cdot \\,{\\vec {x_{i}}}\\right)^{2}}{2\\sigma ^{2}}}\\right)\\\\&=n\\log {\\frac {1}{{\\sqrt {2\\pi }}\\sigma }}-{\\frac {1}{2\\sigma ^{2}}}\\sum _{i=1}^{n}\\left(y_{i}-{\\vec {\\beta }}\\,\\cdot \\,{\\vec {x_{i}}}\\right)^{2}\\end{aligned}}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0242", "text": "The optimal parameter is thus equal to:", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0243", "text": "arg max\n                    \n                    \n                      \n                        \u03b2\n                        \u2192\n                      \n                    \n                  \n                \n                \n                I\n                (\n                D\n                ,", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0244", "text": "\u03b2\n                      \u2192\n                    \n                  \n                \n                )\n              \n              \n                \n                =\n                \n                  \n                    \n                      arg max\n                    \n                    \n                      \n                        \u03b2\n                        \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0245", "text": "(\n                  \n                    n\n                    log\n                    \u2061\n                    \n                      \n                        1\n                        \n                          \n                            \n                              2\n                              \u03c0\n                            \n                          \n                          \u03c3", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0246", "text": "\u03c3\n                        \n                      \n                    \n                    \u2212\n                    \n                      \n                        1\n                        \n                          2\n                          \n                            \u03c3\n                            \n                              2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0247", "text": "\u2211\n                      \n                        i\n                        =\n                        1\n                      \n                      \n                        n\n                      \n                    \n                    \n                      \n                        (\n                        \n                          \n                            y", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0248", "text": "i\n                            \n                          \n                          \u2212\n                          \n                            \n                              \n                                \u03b2\n                                \u2192\n                              \n                            \n                          \n                          \n                          \u22c5", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0249", "text": "x\n                                  \n                                    i\n                                  \n                                \n                                \u2192\n                              \n                            \n                          \n                        \n                        )", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0250", "text": ")\n                      \n                      \n                        2\n                      \n                    \n                  \n                  )\n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    \n                      arg min\n                    \n                    \n                      \n                        \u03b2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0251", "text": "\u03b2\n                        \u2192\n                      \n                    \n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                \n                  \n                    (", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0252", "text": "y\n                        \n                          i\n                        \n                      \n                      \u2212\n                      \n                        \n                          \n                            \u03b2\n                            \u2192\n                          \n                        \n                      \n                      \n                      \u22c5", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0253", "text": "x\n                              \n                                i\n                              \n                            \n                            \u2192\n                          \n                        \n                      \n                    \n                    )\n                  \n                  \n                    2", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0254", "text": "2\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    \n                      arg min\n                    \n                    \n                      \n                        \u03b2\n                        \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0255", "text": "L\n                (\n                D\n                ,\n                \n                  \n                    \n                      \u03b2\n                      \u2192\n                    \n                  \n                \n                )\n              \n            \n            \n              \n              \n                \n                =", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0256", "text": "\u03b2\n                          ^\n                        \n                      \n                      \u2192", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0257", "text": "{\\displaystyle {\\begin{aligned}{\\underset {\\vec {\\beta }}{\\mbox{arg max}}}\\,I(D,{\\vec {\\beta }})&={\\underset {\\vec {\\beta }}{\\mbox{arg max}}}\\left(n\\log {\\frac {1}{{\\sqrt {2\\pi }}\\sigma }}-{\\frac {1}{2\\sigma ^{2}}}\\sum _{i=1}^{n}\\left(y_{i}-{\\vec {\\beta }}\\,\\cdot \\,{\\vec {x_{i}}}\\right)^{2}\\right)\\\\&={\\underset {\\vec {\\beta }}{\\mbox{arg min}}}\\sum _{i=1}^{n}\\left(y_{i}-{\\vec {\\beta }}\\,\\cdot \\,{\\vec {x_{i}}}\\right)^{2}\\\\&={\\underset {\\vec {\\beta }}{\\mbox{arg min}}}\\,L(D,{\\vec {\\beta", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0258", "text": "{\\beta }}{\\mbox{arg min}}}\\,L(D,{\\vec {\\beta }})\\\\&={\\vec {\\hat {\\beta }}}\\end{aligned}}}", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0259", "text": "In this way, the parameter that maximizes \n  \n    \n      \n        H\n        (\n        D\n        ,\n        \n          \n            \n              \u03b2\n              \u2192\n            \n          \n        \n        )\n      \n    \n    {\\displaystyle H(D,{\\vec {\\beta }})}\n  \n is the same as the one that minimizes \n  \n    \n      \n        L\n        (\n        D\n        ,\n        \n          \n            \n              \u03b2\n              \u2192\n            \n          \n        \n        )", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0260", "text": ")\n      \n    \n    {\\displaystyle L(D,{\\vec {\\beta }})}\n  \n. This means that in linear regression, the result of the least squares method is the same as the result of the maximum likelihood estimation method.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0261", "text": "Regularized Regression", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0262", "text": "Ridge regression and other forms of penalized estimation, such as Lasso regression, deliberately introduce bias into the estimation of \u03b2 in order to reduce the variability of the estimate. The resulting estimates generally have lower mean squared error than the OLS estimates, particularly when multicollinearity is present or when overfitting is a problem", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0263", "text": ". They are generally used when the goal is to predict the value of the response variable y for values of the predictors x that have not yet been observed. These methods are not as commonly used when the goal is inference, since it is difficult to account for the bias.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0264", "text": "Least Absolute Deviation\nLeast absolute deviation (LAD) regression is a robust estimation technique in that it is less sensitive to the presence of outliers than OLS (but is less efficient than OLS when no outliers are present). It is equivalent to maximum likelihood estimation under a Laplace distribution model for \u03b5.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0265", "text": "Adaptive Estimation\nIf we assume that error terms are independent of the regressors, \n  \n    \n      \n        \n          \u03b5\n          \n            i\n          \n        \n        \u22a5\n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\varepsilon _{i}\\perp \\mathbf {x} _{i}}\n  \n, then the optimal estimator is the 2-step MLE, where the first step is used to non-parametrically estimate the distribution of the error term.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0266", "text": "Other estimation techniques", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0267", "text": "Bayesian linear regression applies the framework of Bayesian statistics to linear regression. (See also Bayesian multivariate linear regression.) In particular, the regression coefficients \u03b2 are assumed to be random variables with a specified prior distribution. The prior distribution can bias the solutions for the regression coefficients, in a way similar to (but more general than) ridge regression or lasso regression", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0268", "text": ". In addition, the Bayesian estimation process produces not a single point estimate for the \"best\" values of the regression coefficients but an entire posterior distribution, completely describing the uncertainty surrounding the quantity. This can be used to estimate the \"best\" coefficients using the mean, mode, median, any quantile (see quantile regression), or any other function of the posterior distribution.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0269", "text": "Quantile regression focuses on the conditional quantiles of y given X rather than the conditional mean of y given X. Linear quantile regression models a particular conditional quantile, for example the conditional median, as a linear function \u03b2Tx of the predictors.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0270", "text": "Mixed models are widely used to analyze linear regression relationships involving dependent data when the dependencies have a known structure. Common applications of mixed models include analysis of data involving repeated measurements, such as longitudinal data, or data obtained from cluster sampling. They are generally fit as parametric models, using maximum likelihood or Bayesian estimation", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0271", "text": ". In the case where the errors are modeled as normal random variables, there is a close connection between mixed models and generalized least squares. Fixed effects estimation is an alternative approach to analyzing this type of data.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0272", "text": "Principal component regression (PCR) is used when the number of predictor variables is large, or when strong correlations exist among the predictor variables. This two-stage procedure first reduces the predictor variables using principal component analysis, and then uses the reduced variables in an OLS regression fit", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0273", "text": ". While it often works well in practice, there is no general theoretical reason that the most informative linear function of the predictor variables should lie among the dominant principal components of the multivariate distribution of the predictor variables. The partial least squares regression is the extension of the PCR method which does not suffer from the mentioned deficiency.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0274", "text": "Least-angle regression is an estimation procedure for linear regression models that was developed to handle high-dimensional covariate vectors, potentially with more covariates than observations.\nThe Theil\u2013Sen estimator is a simple robust estimation technique that chooses the slope of the fit line to be the median of the slopes of the lines through pairs of sample points. It has similar statistical efficiency properties to simple linear regression but is much less sensitive to outliers.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0275", "text": "Other robust estimation techniques, including the \u03b1-trimmed mean approach, and L-, M-, S-, and R-estimators have been introduced.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0276", "text": "Applications\nLinear regression is widely used in biological, behavioral and social sciences to describe possible relationships between variables. It ranks as one of the most important tools used in these disciplines.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0277", "text": "Trend line", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0278", "text": "A trend line represents a trend, the long-term movement in time series data after other components have been accounted for. It tells whether a particular data set (say GDP, oil prices or stock prices) have increased or decreased over the period of time. A trend line could simply be drawn by eye through a set of data points, but more properly their position and slope is calculated using statistical techniques like linear regression", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0279", "text": ". Trend lines typically are straight lines, although some variations use higher degree polynomials depending on the degree of curvature desired in the line.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0280", "text": "Trend lines are sometimes used in business analytics to show changes in data over time. This has the advantage of being simple. Trend lines are often used to argue that a particular action or event (such as training, or an advertising campaign) caused observed changes at a point in time. This is a simple technique, and does not require a control group, experimental design, or a sophisticated analysis technique", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0281", "text": ". However, it suffers from a lack of scientific validity in cases where other potential changes can affect the data.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0282", "text": "Epidemiology", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0283", "text": "Early evidence relating tobacco smoking to mortality and morbidity came from observational studies employing regression analysis. In order to reduce spurious correlations when analyzing observational data, researchers usually include several variables in their regression models in addition to the variable of primary interest", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0284", "text": ". For example, in a regression model in which cigarette smoking is the independent variable of primary interest and the dependent variable is lifespan measured in years, researchers might include education and income as additional independent variables, to ensure that any observed effect of smoking on lifespan is not due to those other socio-economic factors. However, it is never possible to include all possible confounding variables in an empirical analysis", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0285", "text": ". For example, a hypothetical gene might increase mortality and also cause people to smoke more. For this reason, randomized controlled trials are often able to generate more compelling evidence of causal relationships than can be obtained using regression analyses of observational data. When controlled experiments are not feasible, variants of regression analysis such as instrumental variables regression may be used to attempt to estimate causal relationships from observational data.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0286", "text": "Finance\nThe capital asset pricing model uses linear regression as well as the concept of beta for analyzing and quantifying the systematic risk of an investment. This comes directly from the beta coefficient of the linear regression model that relates the return on the investment to the return on all risky assets.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0287", "text": "Economics\nLinear regression is the predominant empirical tool in economics. For example, it is used to predict consumption spending, fixed investment spending, inventory investment, purchases of a country's exports, spending on imports, the demand to hold liquid assets, labor demand, and labor supply.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0288", "text": "Environmental science", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0289", "text": "Linear regression finds application in a wide range of environmental science applications such as land use, infectious diseases, and air pollution. For example, linear regression can be used to predict the changing effects of car pollution", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0290", "text": ". One notable example of this application in infectious diseases is the flattening the curve strategy emphasized early in the COVID-19 pandemic, where public health officials dealt with sparse data on infected individuals and sophisticated models of disease transmission to characterize the spread of COVID-19.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0291", "text": "Building science", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0292", "text": "Linear regression is commonly used in building science field studies to derive characteristics of building occupants. In a thermal comfort field study, building scientists usually ask occupants' thermal sensation votes, which range from -3 (feeling cold) to 0 (neutral) to +3 (feeling hot), and measure occupants' surrounding temperature data", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0293", "text": ". A neutral or comfort temperature can be calculated based on a linear regression between the thermal sensation vote and indoor temperature, and setting the thermal sensation vote as zero. However, there has been a debate on the regression direction: regressing thermal sensation votes (y-axis) against indoor temperature (x-axis) or the opposite: regressing indoor temperature (y-axis) against thermal sensation votes (x-axis).", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0294", "text": "Machine learning\nLinear regression plays an important role in the subfield of artificial intelligence known as machine learning. The linear regression algorithm is one of the fundamental supervised machine-learning algorithms due to its relative simplicity and well-known properties.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0295", "text": "History", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0296", "text": "Isaac Newton is credited with inventing \"a certain technique known today as linear regression analysis\" in his work on equinoxes in 1700, and wrote down the first of the two normal equations of the ordinary least squares method. The Least squares linear regression, as a means of finding a good rough linear fit to a set of points was performed by Legendre (1805) and Gauss (1809) for the prediction of planetary movement", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0297", "text": ". Quetelet was responsible for making the procedure well-known and for using it extensively in the social sciences.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0298", "text": "See also\nReferences\nCitations\nSources\nFurther reading\nPedhazur, Elazar J (1982). Multiple regression in behavioral research: Explanation and prediction (2nd ed.). New York: Holt, Rinehart and Winston. ISBN 978-0-03-041760-3.\nMathieu Rouaud, 2013: Probability, Statistics and Estimation Chapter 2: Linear Regression, Linear Regression with Error Bars and Nonlinear Regression.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0299", "text": "National Physical Laboratory (1961). \"Chapter 1: Linear Equations and Matrices: Direct Methods\". Modern Computing Methods. Notes on Applied Science. Vol. 16 (2nd ed.). Her Majesty's Stationery Office.", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Linear_regression_0300", "text": "External links\n\nLeast-Squares Regression, PhET Interactive simulations, University of Colorado at Boulder\nDIY Linear Fit", "title": "Linear regression", "url": "https://en.wikipedia.org/wiki/Linear_regression"}, {"id": "rec_Ensemble_learning_0000", "text": "In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone.\nUnlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0001", "text": "Overview\nSupervised learning algorithms search through a hypothesis space to find a suitable hypothesis that will make good predictions with a particular problem. Even if this space contains hypotheses that are very well-suited for a particular problem, it may be very difficult to find a good one. Ensembles combine multiple hypotheses to form one which should be theoretically better.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0002", "text": "Ensemble learning trains two or more machine learning algorithms on a specific classification or regression task. The algorithms within the ensemble model are generally referred as \"base models\", \"base learners\", or \"weak learners\" in literature. These base models can be constructed using a single modelling algorithm, or several different algorithms", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0003", "text": ". The idea is to train a diverse set of weak models on the same modelling task, such that the outputs of each weak learner have poor predictive ability (i.e., high bias), and among all weak learners, the outcome and error values exhibit high variance. Fundamentally, an ensemble learning model trains at least two high-bias (weak) and high-variance (diverse) models to be combined into a better-performing model", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0004", "text": ". The set of weak models \u2014 which would not produce satisfactory predictive results individually \u2014 are combined or averaged to produce a single, high performing, accurate, and low-variance model to fit the task as required.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0005", "text": "Ensemble learning typically refers to bagging (bootstrap aggregating), boosting or stacking/blending techniques to induce high variance among the base models. Bagging creates diversity by generating random samples from the training observations and fitting the same model to each different sample \u2014 also known as homogeneous parallel ensembles", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0006", "text": ". Boosting follows an iterative process by sequentially training each base model on the up-weighted errors of the previous base model, producing an additive model to reduce the final model errors \u2014 also known as sequential ensemble learning. Stacking or blending consists of different base models, each trained independently (i.e. diverse/high variance) to be combined into the ensemble model \u2014 producing a heterogeneous parallel ensemble", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0007", "text": ". Common applications of ensemble learning include random forests (an extension of bagging), Boosted Tree models, and Gradient Boosted Tree Models. Models in applications of stacking are generally more task-specific \u2014 such as combining clustering techniques with other parametric and/or non-parametric techniques.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0008", "text": "Evaluating the prediction of an ensemble typically requires more computation than evaluating the prediction of a single model. In one sense, ensemble learning may be thought of as a way to compensate for poor learning algorithms by performing a lot of extra computation. On the other hand, the alternative is to do a lot more learning with one non-ensemble model", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0009", "text": ". An ensemble may be more efficient at improving overall accuracy for the same increase in compute, storage, or communication resources by using that increase on two or more methods, than would have been improved by increasing resource use for a single method. Fast algorithms such as decision trees are commonly used in ensemble methods (e.g., random forests), although slower algorithms can benefit from ensemble techniques as well.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0010", "text": "By analogy, ensemble techniques have been used also in unsupervised learning scenarios, for example in consensus clustering or in anomaly detection.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0011", "text": "Ensemble theory", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0012", "text": "Empirically, ensembles tend to yield better results when there is a significant diversity among the models. Many ensemble methods, therefore, seek to promote diversity among the models they combine. Although perhaps non-intuitive, more random algorithms (like random decision trees) can be used to produce a stronger ensemble than very deliberate algorithms (like entropy-reducing decision trees)", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0013", "text": ". Using a variety of strong learning algorithms, however, has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity. It is possible to increase diversity in the training stage of the model using correlation for regression tasks  or using information measures such as cross entropy for classification tasks.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0014", "text": "Theoretically, one can justify the diversity concept because the lower bound of the error rate of an ensemble system can be decomposed into accuracy, diversity, and the other term.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0015", "text": "The geometric framework\nEnsemble learning, including both regression and classification tasks, can be explained using a geometric framework. Within this framework, the output of each individual classifier or regressor for the entire dataset can be viewed as a point in a multi-dimensional space. Additionally, the target result is also represented as a point in this space, referred to as the \"ideal point.\"", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0016", "text": "The Euclidean distance is used as the metric to measure both the performance of a single classifier or regressor (the distance between its point and the ideal point) and the dissimilarity between two classifiers or regressors (the distance between their respective points). This perspective transforms ensemble learning into a deterministic problem.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0017", "text": "For example, within this geometric framework, it can be proved that the averaging of the outputs (scores) of all base classifiers or regressors can lead to equal or better results than the average of all the individual models. It can also be proved that if the optimal weighting scheme is used, then a weighted averaging approach can outperform any of the individual classifiers or regressors that make up the ensemble or as good as the best performer at least.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0018", "text": "Ensemble size", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0019", "text": "While the number of component classifiers of an ensemble has a great impact on the accuracy of prediction, there is a limited number of studies addressing this problem. A priori determining of ensemble size and the volume and velocity of big data streams make this even more crucial for online ensemble classifiers. Mostly statistical tests were used for determining the proper number of components", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0020", "text": ". More recently, a theoretical framework suggested that there is an ideal number of component classifiers for an ensemble such that having more or less than this number of classifiers would deteriorate the accuracy. It is called \"the law of diminishing returns in ensemble construction.\" Their theoretical framework shows that using the same number of independent component classifiers as class labels gives the highest accuracy.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0021", "text": "Common types of ensembles\nBayes optimal classifier", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0022", "text": "The Bayes optimal classifier is a classification technique. It is an ensemble of all the hypotheses in the hypothesis space. On average, no other ensemble can outperform it. The Naive Bayes classifier is a version of this that assumes that the data is conditionally independent on the class and makes the computation more feasible. Each hypothesis is given a vote proportional to the likelihood that the training dataset would be sampled from a system if that hypothesis were true", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0023", "text": ". To facilitate training data of finite size, the vote of each hypothesis is also multiplied by the prior probability of that hypothesis. The Bayes optimal classifier can be expressed with the following equation:", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0024", "text": "y\n        =\n        \n          \n            \n              a\n              r\n              g\n              m\n              a\n              x\n            \n            \n              \n                c\n                \n                  j\n                \n              \n              \u2208\n              C\n            \n          \n        \n        \n          \u2211\n          \n            \n              h\n              \n                i\n              \n            \n            \u2208", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0025", "text": "\u2208\n            H\n          \n        \n        \n          P\n          (\n          \n            c\n            \n              j\n            \n          \n          \n            |\n          \n          \n            h\n            \n              i\n            \n          \n          )\n          P\n          (\n          T\n          \n            |\n          \n          \n            h\n            \n              i\n            \n          \n          )\n          P\n          (", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0026", "text": ")\n          P\n          (\n          \n            h\n            \n              i\n            \n          \n          )\n        \n      \n    \n    {\\displaystyle y={\\underset {c_{j}\\in C}{\\mathrm {argmax} }}\\sum _{h_{i}\\in H}{P(c_{j}|h_{i})P(T|h_{i})P(h_{i})}}", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0027", "text": "where \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n is the predicted class, \n  \n    \n      \n        C\n      \n    \n    {\\displaystyle C}\n  \n is the set of all possible classes, \n  \n    \n      \n        H\n      \n    \n    {\\displaystyle H}\n  \n is the hypothesis space, \n  \n    \n      \n        P\n      \n    \n    {\\displaystyle P}\n  \n refers to a probability, and \n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0028", "text": "T\n      \n    \n    {\\displaystyle T}\n  \n is the training data. As an ensemble, the Bayes optimal classifier represents a hypothesis that is not necessarily in \n  \n    \n      \n        H\n      \n    \n    {\\displaystyle H}\n  \n. The hypothesis represented by the Bayes optimal classifier, however, is the optimal hypothesis in ensemble space (the space of all possible ensembles consisting only of hypotheses in \n  \n    \n      \n        H\n      \n    \n    {\\displaystyle H}\n  \n).", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0029", "text": "H\n      \n    \n    {\\displaystyle H}\n  \n).\nThis formula can be restated using Bayes' theorem, which says that the posterior is proportional to the likelihood times the prior:", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0030", "text": "P\n        (\n        \n          h\n          \n            i\n          \n        \n        \n          |\n        \n        T\n        )\n        \u221d\n        P\n        (\n        T\n        \n          |\n        \n        \n          h\n          \n            i\n          \n        \n        )\n        P\n        (\n        \n          h\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle P(h_{i}|T)\\propto P(T|h_{i})P(h_{i})}\n  \n\nhence,", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0031", "text": "y\n        =\n        \n          \n            \n              a\n              r\n              g\n              m\n              a\n              x\n            \n            \n              \n                c\n                \n                  j\n                \n              \n              \u2208\n              C\n            \n          \n        \n        \n          \u2211\n          \n            \n              h\n              \n                i\n              \n            \n            \u2208", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0032", "text": "\u2208\n            H\n          \n        \n        \n          P\n          (\n          \n            c\n            \n              j\n            \n          \n          \n            |\n          \n          \n            h\n            \n              i\n            \n          \n          )\n          P\n          (\n          \n            h\n            \n              i\n            \n          \n          \n            |\n          \n          T\n          )", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0033", "text": "T\n          )\n        \n      \n    \n    {\\displaystyle y={\\underset {c_{j}\\in C}{\\mathrm {argmax} }}\\sum _{h_{i}\\in H}{P(c_{j}|h_{i})P(h_{i}|T)}}", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0034", "text": "Bootstrap aggregating (bagging)", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0035", "text": "Bootstrap aggregation (bagging) involves training an ensemble on bootstrapped data sets. A bootstrapped set is created by selecting from original training data set with replacement. Thus, a bootstrap set may contain a given example zero, one, or multiple times. Ensemble members can also have limits on the features (e.g., nodes of a decision tree), to encourage exploring of diverse features", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0036", "text": ". The variance of local information in the bootstrap sets and feature considerations promote diversity in the ensemble, and can strengthen the ensemble. To reduce overfitting, a member can be validated using the out-of-bag set (the examples that are not in its bootstrap set).", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0037", "text": "Inference is done by voting of predictions of ensemble members, called aggregation. It is illustrated below with an ensemble of four decision trees. The query example is classified by each tree. Because three of the four predict the positive class, the ensemble's overall classification is positive. Random forests like the one shown are a common application of bagging.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0038", "text": "Boosting\nBoosting involves training successive models by emphasizing training data mis-classified by previously learned models. Initially, all data (D1) has equal weight and is used to learn a base model M1. The examples mis-classified by M1 are assigned a weight greater than correctly classified examples. This boosted data (D2) is used to train a second base model M2, and so on. Inference is done by voting.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0039", "text": "In some cases, boosting has yielded better accuracy than bagging, but tends to over-fit more. The most common implementation of boosting is Adaboost, but some newer algorithms are reported to achieve better results.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0040", "text": "Bayesian model averaging\nBayesian model averaging (BMA) makes predictions by averaging the predictions of models weighted by their posterior probabilities given the data. BMA is known to generally give better answers than a single model, obtained, e.g., via stepwise regression, especially where very different models have nearly identical performance in the training set but may otherwise perform quite differently.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0041", "text": "The question with any use of Bayes' theorem is the prior, i.e., the probability (perhaps subjective) that each model is the best to use for a given purpose.  Conceptually, BMA can be used with any prior.  R packages ensembleBMA and BMA use the prior implied by the Bayesian information criterion, (BIC), following Raftery (1995)", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0042", "text": ". R package BAS supports the use of the priors implied by Akaike information criterion (AIC) and other criteria over the alternative models as well as priors over the coefficients.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0043", "text": "The difference between BIC and AIC is the strength of preference for parsimony.  BIC's penalty for model complexity is \n  \n    \n      \n        ln\n        \u2061\n        (\n        n\n        )\n        k\n      \n    \n    {\\displaystyle \\ln(n)k}\n  \n , while AIC's is \n  \n    \n      \n        2\n        k\n      \n    \n    {\\displaystyle 2k}", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0044", "text": "k\n      \n    \n    {\\displaystyle 2k}\n  \n. Large-sample asymptotic theory establishes that if there is a best model, then with increasing sample sizes, BIC is strongly consistent, i.e., will almost certainly find it, while AIC may not, because AIC may continue to place excessive posterior probability on models that are more complicated than they need to be. On the other hand, AIC and AICc are asymptotically \"efficient\" (i.e., minimum mean square prediction error), while BIC is not .", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0045", "text": "Haussler et al. (1994) showed that when BMA is used for classification, its expected error is at most twice the expected error of the Bayes optimal classifier. Burnham and Anderson (1998, 2002) contributed greatly to introducing a wider audience to the basic ideas of Bayesian model averaging and popularizing the methodology. The availability of software, including other free open-source packages for R beyond those mentioned above, helped make the methods accessible to a wider audience.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0046", "text": "Bayesian model combination", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0047", "text": "Bayesian model combination (BMC) is an algorithmic correction to Bayesian model averaging (BMA). Instead of sampling each model in the ensemble individually, it samples from the space of possible ensembles (with model weights drawn randomly from a Dirichlet distribution having uniform parameters). This modification overcomes the tendency of BMA to converge toward giving all the weight to a single model", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0048", "text": ". Although BMC is somewhat more computationally expensive than BMA, it tends to yield dramatically better results. BMC has been shown to be better on average (with statistical significance) than BMA and bagging.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0049", "text": "Use of Bayes' law to compute model weights requires computing the probability of the data given each model. Typically, none of the models in the ensemble are exactly the distribution from which the training data were generated, so all of them correctly receive a value close to zero for this term. This would work well if the ensemble were big enough to sample the entire model-space, but this is rarely possible", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0050", "text": ". Consequently, each pattern in the training data will cause the ensemble weight to shift toward the model in the ensemble that is closest to the distribution of the training data. It essentially reduces to an unnecessarily complex method for doing model selection.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0051", "text": "The possible weightings for an ensemble can be visualized as lying on a simplex. At each vertex of the simplex, all of the weight is given to a single model in the ensemble. BMA converges toward the vertex that is closest to the distribution of the training data. By contrast, BMC converges toward the point where this distribution projects onto the simplex", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0052", "text": ". In other words, instead of selecting the one model that is closest to the generating distribution, it seeks the combination of models that is closest to the generating distribution.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0053", "text": "The results from BMA can often be approximated by using cross-validation to select the best model from a bucket of models. Likewise, the results from BMC may be approximated by using cross-validation to select the best ensemble combination from a random sampling of possible weightings.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0054", "text": "Bucket of models\nA \"bucket of models\" is an ensemble technique in which a model selection algorithm is used to choose the best model for each problem. When tested with only one problem, a bucket of models can produce no better results than the best model in the set, but when evaluated across many problems, it will typically produce much better results, on average, than any model in the set.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0055", "text": "The most common approach used for model-selection is cross-validation selection (sometimes called a \"bake-off contest\"). It is described with the following pseudo-code:", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0056", "text": "For each model m in the bucket:\n    Do c times: (where 'c' is some constant)\n        Randomly divide the training dataset into two sets: A and B\n        Train m with A\n        Test m with B\nSelect the model that obtains the highest average score", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0057", "text": "Cross-Validation Selection can be summed up as: \"try them all with the training set, and pick the one that works best\".\nGating is a generalization of Cross-Validation Selection. It involves training another learning model to decide which of the models in the bucket is best-suited to solve the problem. Often, a perceptron is used for the gating model. It can be used to pick the \"best\" model, or it can be used to give a linear weight to the predictions from each model in the bucket.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0058", "text": "When a bucket of models is used with a large set of problems, it may be desirable to avoid training some of the models that take a long time to train. Landmark learning is a meta-learning approach that seeks to solve this problem. It involves training only the fast (but imprecise) algorithms in the bucket, and then using the performance of these algorithms to help determine which slow (but accurate) algorithm is most likely to do best.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0059", "text": "Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble\nThe most common approach for training classifier is using Cross-entropy cost function. However, one would like to train an ensemble of models that have diversity so when we combine them it would provide best results.\nAssuming we use a simple ensemble of averaging \n  \n    \n      \n        K\n      \n    \n    {\\displaystyle K}\n  \n classifiers. Then the Amended Cross-Entropy Cost is", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0060", "text": "e\n          \n            k\n          \n        \n        =\n        H\n        (\n        p\n        ,\n        \n          q\n          \n            k\n          \n        \n        )\n        \u2212\n        \n          \n            \u03bb\n            K\n          \n        \n        \n          \u2211\n          \n            j\n            \u2260\n            k\n          \n        \n        H\n        (\n        \n          q\n          \n            j\n          \n        \n        ,\n        \n          q", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0061", "text": ",\n        \n          q\n          \n            k\n          \n        \n        )\n      \n    \n    {\\displaystyle e^{k}=H(p,q^{k})-{\\frac {\\lambda }{K}}\\sum _{j\\neq k}H(q^{j},q^{k})}", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0062", "text": "where \n  \n    \n      \n        \n          e\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle e^{k}}\n  \n is the cost function of the \n  \n    \n      \n        \n          k\n          \n            t\n            h\n          \n        \n      \n    \n    {\\displaystyle k^{th}}\n  \n classifier, \n  \n    \n      \n        \n          q\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle q^{k}}\n  \n is the probability of the \n  \n    \n      \n        \n          k", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0063", "text": "k\n          \n            t\n            h\n          \n        \n      \n    \n    {\\displaystyle k^{th}}\n  \n classifier, \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n is the true probability that we need to estimate and \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n is a parameter between 0 and 1 that define the diversity that we would like to establish. When \n  \n    \n      \n        \u03bb\n        =\n        0", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0064", "text": "\u03bb\n        =\n        0\n      \n    \n    {\\displaystyle \\lambda =0}\n  \n we want each classifier to do its best regardless of the ensemble and when \n  \n    \n      \n        \u03bb\n        =\n        1\n      \n    \n    {\\displaystyle \\lambda =1}\n  \n we would like the classifier to be as diverse as possible.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0065", "text": "Stacking", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0066", "text": "Stacking (sometimes called stacked generalization) involves training a model to combine the predictions of several other learning algorithms. First, all of the other algorithms are trained using the available data, then a combiner algorithm (final estimator) is trained to make a final prediction using all the predictions of the other algorithms (base estimators) as additional inputs or using cross-validated predictions from the base estimators which can prevent overfitting", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0067", "text": ". If an arbitrary combiner algorithm is used, then stacking can theoretically represent any of the ensemble techniques described in this article, although, in practice, a logistic regression model is often used as the combiner.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0068", "text": "Stacking typically yields performance better than any single one of the trained models. It has been successfully used on both supervised learning tasks (regression, classification and distance learning ) and unsupervised learning (density estimation). It has also been used to estimate bagging's error rate. It has been reported to out-perform Bayesian model-averaging. The two top-performers in the Netflix competition utilized blending, which may be considered a form of stacking.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0069", "text": "Bayesian predictive stacking generalizes the idea of stacking from the statistical estimation literature to the combination of posterior predictive distributions. These ideas have also been developed and investigated for Gaussian process models, especially for spatial data analysis  and can be used to construct transfer learning frameworks for massive spatial data sets.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0070", "text": "Voting\nVoting is another form of ensembling. See e.g. Weighted majority algorithm (machine learning).", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0071", "text": "Implementations in statistics packages\nR: Several packages offer Bayesian model averaging tools, including the BMS (an acronym for Bayesian Model Selection) package, the BAS (an acronym for Bayesian Adaptive Sampling) package, and the BMA package. Other packages implementing predictive stacking for Gaussian process models include the spStack package.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0072", "text": "Python: scikit-learn, a package for machine learning in Python offers packages for ensemble learning including packages for bagging, voting and averaging methods.\nMATLAB: classification ensembles are implemented in Statistics and Machine Learning Toolbox.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0073", "text": "Ensemble learning applications\nIn recent years, due to growing computational power, which allows for training in large ensemble learning in a reasonable time frame, the number of ensemble learning applications has grown increasingly. Some of the applications of ensemble classifiers include:", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0074", "text": "Remote sensing\nLand cover mapping", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0075", "text": "Land cover mapping is one of the major applications of Earth observation satellite sensors, using remote sensing and geospatial data, to identify the materials and objects which are located on the surface of target areas. Generally, the classes of target materials include roads, buildings, rivers, lakes, and vegetation", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0076", "text": ". Some different ensemble learning approaches based on artificial neural networks, kernel principal component analysis (KPCA), decision trees with boosting, random forest and automatic design of multiple classifier systems, are proposed to efficiently identify land cover objects.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0077", "text": "Change detection\nChange detection is an image analysis problem, consisting of the identification of places where the land cover has changed over time. Change detection is widely used in fields such as urban growth, forest and vegetation dynamics, land use and disaster monitoring.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0078", "text": "The earliest applications of ensemble classifiers in change detection are designed with the majority voting, Bayesian model averaging, and the maximum posterior probability. Given the growth of satellite data over time, the past decade sees more use of time series methods for continuous change detection from image stacks. One example is a Bayesian ensemble changepoint detection method called BEAST, with the software available as a package Rbeast in R, Python, and Matlab.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0079", "text": "Computer security\nDistributed denial of service\nDistributed denial of service is one of the most threatening cyber-attacks that may happen to an internet service provider. By combining the output of single classifiers, ensemble classifiers reduce the total error of detecting and discriminating such attacks from legitimate flash crowds.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0080", "text": "Malware Detection\nClassification of malware codes such as computer viruses, computer worms, trojans, ransomware and spywares with the usage of machine learning techniques, is inspired by the document categorization problem. Ensemble learning systems have shown a proper efficacy in this area.\nModel Hardening", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0081", "text": "Model Hardening\nMalware detection models, like all machine learning models, are vulnerable to adversarial machine learning attacks where an attacker  pushes the boundary of what is classified as malware. By rotating through an ensemble of models using moving target defense, the attacker has a diminished knowledge advantage.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0082", "text": "Intrusion detection\nAn intrusion detection system monitors computer network or computer systems to identify intruder codes like an anomaly detection process. Ensemble learning successfully aids such monitoring systems to reduce their total error.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0083", "text": "Face recognition\nFace recognition, which recently has become one of the most popular research areas of pattern recognition, copes with identification or verification of a person by their digital images.\nHierarchical ensembles based on Gabor Fisher classifier and independent component analysis preprocessing techniques are some of the earliest ensembles employed in this field.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0084", "text": "Emotion recognition\nWhile speech recognition is mainly based on deep learning because most of the industry players in this field like Google, Microsoft and IBM reveal that the core technology of their speech recognition is based on this approach, speech-based emotion recognition can also have a satisfactory performance with ensemble learning.\nIt is also being successfully used in facial emotion recognition.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0085", "text": "Fraud detection\nFraud detection deals with the identification of bank fraud, such as money laundering, credit card fraud and telecommunication fraud, which have vast domains of research and applications of machine learning. Because ensemble learning improves the robustness of the normal behavior modelling, it has been proposed as an efficient technique to detect such fraudulent cases and activities in banking and credit card systems.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0086", "text": "Financial decision-making", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0087", "text": "Ensemble learning methods have been widely adopted in finance for tasks such as credit scoring, bankruptcy prediction, and risk management. By combining multiple base models, ensembles can exploit nonlinear relationships, handle high-dimensional and noisy data, and often deliver more stable out-of-sample performance than single models or traditional statistical baselines such as logistic regression and linear factor models", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0088", "text": ". This approach aligns with the broader trend in financial machine learning described by Marcos L\u00f3pez de Prado, who argues that combining diverse learners can improve robustness, mitigate overfitting, and extract persistent patterns from noisy financial time series.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0089", "text": "In retail and corporate credit scoring, ensemble classifiers such as random forests, gradient boosting machines and stacked models are frequently used to assess default risk. A recent systematic literature review of machine-learning credit-scoring studies published between 2018 and 2024 reports that tree-based ensembles and boosting methods are among the most commonly applied techniques and typically achieve higher predictive accuracy than traditional scorecards or single classifiers.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0090", "text": "Ensemble methods have also been applied to corporate failure and bankruptcy prediction. Studies comparing different ensemble constructions (such as bagging, boosting and heterogeneous classifier pools) report that well-tuned ensembles tend to achieve higher classification accuracy and more robust performance across industries than individual models. The accuracy of prediction of business failure is a very crucial issue in financial decision-making", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0091", "text": ". Therefore, different ensemble classifiers are proposed to predict financial crises and financial distress. Also, in the trade-based manipulation problem, where traders attempt to manipulate stock prices by buying and selling activities, ensemble classifiers are required to analyze the changes in the stock market data and detect suspicious symptom of stock price manipulation.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0092", "text": "Implementation of ensemble learning not only with machine learning models but also with mathematical frameworks such as hidden Markov models for applications in trading, regime detection, and asset pricing remains a rapidly developing area of research.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0093", "text": "Medicine\nEnsemble classifiers have been successfully applied in neuroscience, proteomics and medical diagnosis like in neuro-cognitive disorder (i.e. Alzheimer or myotonic dystrophy) detection based on MRI datasets, cervical cytology classification.\nBesides, ensembles have been successfully applied in medical segmentation tasks, for example brain tumor and hyperintensities segmentation.\n\nSee also\nEnsemble averaging (machine learning)\nBayesian structural time series (BSTS)\nMixture of experts", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0094", "text": "References\nFurther reading\nZhou Zhihua (2012). Ensemble Methods: Foundations and Algorithms. Chapman and Hall/CRC. ISBN 978-1-439-83003-1.\nRobert Schapire; Yoav Freund (2012). Boosting: Foundations and Algorithms. MIT. ISBN 978-0-262-01718-3.", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Ensemble_learning_0095", "text": "External links\nRobi Polikar (ed.). \"Ensemble learning\". Scholarpedia.\nThe Waffles (machine learning) toolkit contains implementations of Bagging, Boosting, Bayesian Model Averaging, Bayesian Model Combination, Bucket-of-models, and other ensemble techniques", "title": "Ensemble learning", "url": "https://en.wikipedia.org/wiki/Ensemble_learning"}, {"id": "rec_Boosting_machine_learning_0000", "text": "In machine learning (ML), boosting is an ensemble learning method that combines a set of less accurate models (called \"weak learners\") to create a single, highly accurate model (a \"strong learner\"). Unlike other ensemble methods that build models in parallel (such as bagging), boosting algorithms build models sequentially. Each new model in the sequence is trained to correct the errors made by its predecessors", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0001", "text": ". This iterative process allows the overall model to improve its accuracy, particularly by reducing bias. Boosting is a popular and effective technique used in supervised learning for both classification and regression tasks.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0002", "text": "The theoretical foundation for boosting came from a question posed by Kearns and Valiant (1988, 1989): \"Can a set of weak learners create a single strong learner?\" A weak learner is defined as a classifier that performs only slightly better than random guessing, whereas a strong learner is a classifier that is highly correlated with the true classification. Robert Schapire's affirmative answer to this question in a 1990 paper led to the development of practical boosting algorithms", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0003", "text": ". The first such algorithm was developed by Schapire, with Freund and Schapire later developing AdaBoost, which remains a foundational example of boosting.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0004", "text": "Algorithms", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0005", "text": "While boosting is not algorithmically constrained, most boosting algorithms consist of iteratively learning weak classifiers with respect to a distribution and adding them to a final strong classifier. When they are added, they are weighted in a way that is related to the weak learners' accuracy.  After a weak learner is added, the data weights are readjusted, known as \"re-weighting\". Misclassified input data gain a higher weight and examples that are classified correctly lose weight", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0006", "text": ". Thus, future weak learners focus more on the examples that previous weak learners misclassified.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0007", "text": "There are many boosting algorithms. The original ones, proposed by Robert Schapire (a recursive majority gate formulation), and Yoav Freund (boost by majority), were not adaptive and could not take full advantage of the weak learners. Schapire and Freund then developed AdaBoost, an adaptive boosting algorithm that won the prestigious G\u00f6del Prize.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0008", "text": "Only algorithms that are provable boosting algorithms in the probably approximately correct learning formulation can accurately be called boosting algorithms.  Other algorithms that are similar in spirit to boosting algorithms are sometimes called \"leveraging algorithms\", although they are also sometimes incorrectly called boosting algorithms.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0009", "text": "The main variation between many boosting algorithms is their method of weighting training data points and hypotheses. AdaBoost is very popular and the most significant historically as it was the first algorithm that could adapt to the weak learners. It is often the basis of introductory coverage of boosting in university machine learning courses. There are many more recent algorithms such as LPBoost, TotalBoost, BrownBoost, xgboost, MadaBoost, LogitBoost, CatBoost and others", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0010", "text": ". Many boosting algorithms fit into the AnyBoost framework, which shows that boosting performs gradient descent in a function space using a convex cost function.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0011", "text": "Object categorization in computer vision\nGiven images containing various known objects in the world, a classifier can be learned from them to automatically classify the objects in future images.  Simple classifiers built based on some image feature of the object tend to be weak in categorization performance. Using boosting methods for object categorization is a way to unify the weak classifiers in a special way to boost the overall ability of categorization.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0012", "text": "Problem of object categorization", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0013", "text": "Object categorization is a typical task of computer vision that involves determining whether or not an image contains some specific category of object.  The idea is closely related with recognition, identification, and detection.  Appearance based object categorization typically contains feature extraction, learning a classifier, and applying the classifier to new examples.  There are many ways to represent a category of objects, e.g", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0014", "text": ". from shape analysis, bag of words models, or local descriptors such as SIFT, etc.  Examples of supervised classifiers are Naive Bayes classifiers, support vector machines, mixtures of Gaussians, and neural networks.  However, research has shown that object categories and their locations in images can be discovered in an unsupervised manner as well.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0015", "text": "Status quo for object categorization", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0016", "text": "The recognition of object categories in images is a challenging problem in computer vision, especially when the number of categories is large.  This is due to high intra class variability and the need for generalization across variations of objects within the same category. Objects within one category may look quite different. Even the same object may appear unalike under different viewpoint, scale, and illumination", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0017", "text": ". Background clutter and partial occlusion add difficulties to recognition as well.  Humans are able to recognize thousands of object types, whereas most of the existing object recognition systems are trained to recognize only a few, e.g. human faces, cars, simple objects, etc", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0018", "text": ". human faces, cars, simple objects, etc.  Research has been very active on dealing with more categories and enabling incremental additions of new categories, and although the general problem remains unsolved, several multi-category objects detectors (for up to hundreds or thousands of categories) have been developed.  One means is by feature sharing and boosting.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0019", "text": "Boosting for binary categorization\nAdaBoost can be used for face detection as an example of binary categorization. The two categories are faces versus background. The general algorithm is as follows:", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0020", "text": "Form a large set of simple features\nInitialize weights for training images\nFor T rounds\nNormalize the weights\nFor available features from the set, train a classifier using a single feature and evaluate the training error\nChoose the classifier with the lowest error\nUpdate the weights of the training images: increase if classified wrongly by this classifier, decrease if correctly", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0021", "text": "Form the final strong classifier as the linear combination of the T classifiers (coefficient larger if training error is small)\nAfter boosting, a classifier constructed from 200 features could yield a 95% detection rate under a \n  \n    \n      \n        \n          10\n          \n            \u2212\n            5\n          \n        \n      \n    \n    {\\displaystyle 10^{-5}}\n  \n false positive rate.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0022", "text": "false positive rate.\nAnother application of boosting for binary categorization is a system that detects pedestrians using patterns of motion and appearance. This work is the first to combine both motion information and appearance information as features to detect a walking person. It takes a similar approach to the Viola-Jones object detection framework.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0023", "text": "Boosting for multi-class categorization\nCompared with binary categorization, multi-class categorization looks for common features that can be shared across the categories at the same time.  They turn to be more generic edge like features. During learning, the detectors for each category can be trained jointly. Compared with training separately, it generalizes better, needs less training data, and requires fewer features to achieve the same performance.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0024", "text": "The main flow of the algorithm is similar to the binary case. What is different is that a measure of the joint training error shall be defined in advance. During each iteration the algorithm chooses a classifier of a single feature (features that can be shared by more categories shall be encouraged)", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0025", "text": ". This can be done via converting multi-class classification into a binary one (a set of categories versus the rest), or by introducing a penalty error from the categories that do not have the feature of the classifier.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0026", "text": "In the paper \"Sharing visual features for multiclass and multiview object detection\", A. Torralba et al. used GentleBoost for boosting and showed that when training data is limited, learning via sharing features does a much better job than no sharing, given same boosting rounds", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0027", "text": ". Also, for a given performance level, the total number of features required (and therefore the run time cost of the classifier) for the feature sharing detectors, is observed to scale approximately logarithmically with the number of class, i.e., slower than linear growth in the non-sharing case. Similar results are shown in the paper \"Incremental learning of object detectors using a visual shape alphabet\", yet the authors used AdaBoost for boosting.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0028", "text": "Convex vs. non-convex boosting algorithms", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0029", "text": "Boosting algorithms can be based on convex or non-convex optimization algorithms.  Convex algorithms, such as AdaBoost and LogitBoost, can be \"defeated\" by random  noise such that they can't learn basic and learnable combinations of weak hypotheses. This limitation was pointed out by Long & Servedio in 2008", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0030", "text": ".  However, by 2009, multiple authors demonstrated that  boosting algorithms based on non-convex optimization, such as BrownBoost, can learn from noisy datasets and can specifically learn the underlying classifier of the Long\u2013Servedio dataset.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0031", "text": "See also\nImplementations\nscikit-learn, an open source machine learning library for Python\nOrange, a free data mining software suite, module Orange.ensemble\nWeka is a machine learning set of tools that offers variate implementations of boosting algorithms like AdaBoost and LogitBoost\nR package GBM (Generalized Boosted Regression Models) implements extensions to Freund and Schapire's AdaBoost algorithm and Friedman's gradient boosting machine.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0032", "text": "jboost; AdaBoost, LogitBoost, RobustBoost, Boostexter and alternating decision trees\nR package adabag: Applies Multiclass AdaBoost.M1, AdaBoost-SAMME and Bagging\nR package xgboost: An implementation of gradient boosting for linear and tree-based models.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0033", "text": "Notes\nReferences\nFurther reading\nFreund, Yoav; Schapire, Robert E. (1997). \"A Decision-Theoretic Generalization of On-line Learning and an Application to Boosting\" (PDF). Journal of Computer and System Sciences. 55 (1): 119\u2013139. doi:10.1006/jcss.1997.1504.\nSchapire, Robert E. (1990). \"The strength of weak learnability\". Machine Learning. 5 (2): 197\u2013227. doi:10.1007/BF00116037. S2CID 6207294.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0034", "text": "Schapire, Robert E.; Singer, Yoram (1999). \"Improved Boosting Algorithms Using Confidence-Rated Predictors\". Machine Learning. 37 (3): 297\u2013336. doi:10.1023/A:1007614523901. S2CID 2329907.\nZhou, Zhihua (2008). \"On the margin explanation of boosting algorithm\" (PDF). In: Proceedings of the 21st Annual Conference on Learning Theory (COLT'08): 479\u2013490.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0035", "text": "Zhou, Zhihua (2013). \"On the doubt about margin explanation of boosting\" (PDF). Artificial Intelligence. 203: 1\u201318. arXiv:1009.3613. doi:10.1016/j.artint.2013.07.002. S2CID 2828847.", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Boosting_machine_learning_0036", "text": "External links\nRobert E. Schapire (2003); The Boosting Approach to Machine Learning: An Overview, MSRI (Mathematical Sciences Research Institute) Workshop on Nonlinear Estimation and Classification\nBoosting: Foundations and Algorithms by Robert E. Schapire and Yoav Freund", "title": "Boosting (machine learning)", "url": "https://en.wikipedia.org/wiki/Boosting_(machine_learning)"}, {"id": "rec_Gradient_boosting_0000", "text": "Gradient boosting is a machine learning technique based on boosting in a functional space, where the target is pseudo-residuals instead of residuals as in traditional boosting. It gives a prediction model in the form of an ensemble of weak prediction models, i.e., models that make very few assumptions about the data, which are typically simple decision trees. When a decision tree is the weak learner, the resulting algorithm is called gradient-boosted trees; it usually outperforms random forest", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0001", "text": ". As with other boosting methods, a gradient-boosted trees model is built in stages, but it generalizes the other methods by allowing optimization of an arbitrary differentiable loss function.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0002", "text": "History\nThe idea of gradient boosting originated in the observation by Leo Breiman that boosting can be interpreted as an optimization algorithm on a suitable cost function. Explicit regression gradient boosting algorithms were subsequently developed, by Jerome H. Friedman, (in 1999 and later in 2001) simultaneously with the more general functional gradient boosting perspective of Llew Mason, Jonathan Baxter, Peter Bartlett and Marcus Frean.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0003", "text": "The latter two papers introduced the view of boosting algorithms as iterative functional gradient descent algorithms. That is, algorithms that optimize a cost function over function space by iteratively choosing a function (weak hypothesis) that points in the negative gradient direction. This functional gradient view of boosting has led to the development of boosting algorithms in many areas of machine learning and statistics beyond regression and classification.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0004", "text": "Informal introduction\n(This section follows the exposition by Cheng Li.)\nLike other boosting methods, gradient boosting combines weak \"learners\" into a single strong learner iteratively. It is easiest to explain in the least-squares regression setting, where the goal is to teach a model \n  \n    \n      \n        F\n      \n    \n    {\\displaystyle F}\n  \n to predict values of the form \n  \n    \n      \n        \n          \n            \n              y\n              ^", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0005", "text": "^\n            \n          \n        \n        =\n        F\n        (\n        x\n        )\n      \n    \n    {\\displaystyle {\\hat {y}}=F(x)}\n  \n by minimizing the mean squared error \n  \n    \n      \n        \n          \n            \n              1\n              n\n            \n          \n        \n        \n          \u2211\n          \n            i\n          \n        \n        (\n        \n          \n            \n              \n                y\n                ^", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0006", "text": "^\n              \n            \n          \n          \n            i\n          \n        \n        \u2212\n        \n          y\n          \n            i\n          \n        \n        \n          )\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle {\\tfrac {1}{n}}\\sum _{i}({\\hat {y}}_{i}-y_{i})^{2}}\n  \n, where \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n indexes over some training set of size \n  \n    \n      \n        n", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0007", "text": "n\n      \n    \n    {\\displaystyle n}\n  \n of actual values of the output variable \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0008", "text": "y\n                ^\n              \n            \n          \n          \n            i\n          \n        \n        =\n      \n    \n    {\\displaystyle {\\hat {y}}_{i}=}\n  \n the predicted value \n  \n    \n      \n        F\n        (\n        \n          x\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle F(x_{i})}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0009", "text": "y\n          \n            i\n          \n        \n        =\n      \n    \n    {\\displaystyle y_{i}=}\n  \n the observed value\n\n  \n    \n      \n        n\n        =\n      \n    \n    {\\displaystyle n=}\n  \n the size of the sample, i.e. the number of observations in \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0010", "text": "If the algorithm has \n  \n    \n      \n        M\n      \n    \n    {\\displaystyle M}\n  \n stages, at each stage \n  \n    \n      \n        m\n      \n    \n    {\\displaystyle m}\n  \n (\n  \n    \n      \n        1\n        \u2264\n        m\n        \u2264\n        M\n      \n    \n    {\\displaystyle 1\\leq m\\leq M}\n  \n), suppose some imperfect model \n  \n    \n      \n        \n          F\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle F_{m}}\n  \n (for low \n  \n    \n      \n        m", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0011", "text": "(for low \n  \n    \n      \n        m\n      \n    \n    {\\displaystyle m}\n  \n, this model may simply predict \n  \n    \n      \n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle {\\hat {y}}_{i}}\n  \n to be  \n  \n    \n      \n        \n          \n            \n              y\n              \u00af", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0012", "text": "{\\displaystyle {\\bar {y}}}\n  \n, the mean of \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n). In order to improve \n  \n    \n      \n        \n          F\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle F_{m}}\n  \n, our algorithm should add some new estimator, \n  \n    \n      \n        \n          h\n          \n            m\n          \n        \n        (\n        x\n        )", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0013", "text": "(\n        x\n        )\n      \n    \n    {\\displaystyle h_{m}(x)}\n  \n. Thus,", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0014", "text": "F\n          \n            m\n            +\n            1\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        =\n        \n          F\n          \n            m\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        +\n        \n          h\n          \n            m\n          \n        \n        (\n        \n          x\n          \n            i", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0015", "text": "x\n          \n            i\n          \n        \n        )\n        =\n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle F_{m+1}(x_{i})=F_{m}(x_{i})+h_{m}(x_{i})=y_{i}}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0016", "text": "or, equivalently,", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0017", "text": "h\n          \n            m\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        =\n        \n          y\n          \n            i\n          \n        \n        \u2212\n        \n          F\n          \n            m\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        .\n      \n    \n    {\\displaystyle h_{m}(x_{i})=y_{i}-F_{m}(x_{i}).}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0018", "text": "Therefore, gradient boosting will fit \n  \n    \n      \n        \n          h\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle h_{m}}\n  \n to the residual \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n        \u2212\n        \n          F\n          \n            m\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle y_{i}-F_{m}(x_{i})}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0019", "text": "{\\displaystyle y_{i}-F_{m}(x_{i})}\n  \n. As in other boosting variants, each \n  \n    \n      \n        \n          F\n          \n            m\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle F_{m+1}}\n  \n attempts to correct the errors of its predecessor \n  \n    \n      \n        \n          F\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle F_{m}}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0020", "text": "{\\displaystyle F_{m}}\n  \n. A generalization of this idea to loss functions other than squared error, and to classification and ranking problems, follows from the observation that residuals \n  \n    \n      \n        \n          h\n          \n            m\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle h_{m}(x_{i})}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0021", "text": "{\\displaystyle h_{m}(x_{i})}\n  \n for a given model are proportional to the negative gradients of the mean squared error (MSE) loss function (with respect to \n  \n    \n      \n        F\n        (\n        \n          x\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle F(x_{i})}\n  \n):", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0022", "text": "L\n          \n            \n              M\n              S\n              E\n            \n          \n        \n        =\n        \n          \n            1\n            n\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          \n            (\n            \n              \n                y\n                \n                  i", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0023", "text": "\u2212\n              F\n              (\n              \n                x\n                \n                  i\n                \n              \n              )\n            \n            )\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle L_{\\rm {MSE}}={\\frac {1}{n}}\\sum _{i=1}^{n}\\left(y_{i}-F(x_{i})\\right)^{2}}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0024", "text": "\u2212\n        \n          \n            \n              \u2202\n              \n                L\n                \n                  \n                    M\n                    S\n                    E\n                  \n                \n              \n            \n            \n              \u2202\n              F\n              (\n              \n                x\n                \n                  i\n                \n              \n              )\n            \n          \n        \n        =", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0025", "text": "=\n        \n          \n            2\n            n\n          \n        \n        (\n        \n          y\n          \n            i\n          \n        \n        \u2212\n        F\n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        )\n        =\n        \n          \n            2\n            n\n          \n        \n        \n          h\n          \n            m\n          \n        \n        (\n        \n          x", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0026", "text": "(\n        \n          x\n          \n            i\n          \n        \n        )\n        .\n      \n    \n    {\\displaystyle -{\\frac {\\partial L_{\\rm {MSE}}}{\\partial F(x_{i})}}={\\frac {2}{n}}(y_{i}-F(x_{i}))={\\frac {2}{n}}h_{m}(x_{i}).}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0027", "text": "So, gradient boosting could be generalized to a gradient descent algorithm by plugging in a different loss and its gradient.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0028", "text": "Algorithm\nMany supervised learning problems involve an output variable y and a vector of input variables x, related to each other with some probabilistic distribution. The goal is to find some function \n  \n    \n      \n        \n          \n            \n              F\n              ^\n            \n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle {\\hat {F}}(x)}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0029", "text": "{\\displaystyle {\\hat {F}}(x)}\n  \n that best approximates the output variable from the values of input variables. This is formalized by introducing some loss function \n  \n    \n      \n        L\n        (\n        y\n        ,\n        F\n        (\n        x\n        )\n        )\n      \n    \n    {\\displaystyle L(y,F(x))}\n  \n and minimizing it in expectation:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0030", "text": "F\n              ^\n            \n          \n        \n        =\n        \n          argmin\n          \n            F\n          \n        \n        \u2061\n        \n          \n            E\n          \n          \n            x\n            ,\n            y\n          \n        \n        [\n        L\n        (\n        y\n        ,\n        F\n        (\n        x\n        )\n        )\n        ]\n        .", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0031", "text": ")\n        ]\n        .\n      \n    \n    {\\displaystyle {\\hat {F}}=\\operatorname {argmin} \\limits _{F}\\mathbb {E} _{x,y}[L(y,F(x))].}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0032", "text": "The gradient boosting method assumes a real-valued y. It seeks an approximation \n  \n    \n      \n        \n          \n            \n              F\n              ^\n            \n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle {\\hat {F}}(x)}\n  \n in the form of a weighted sum of M functions \n  \n    \n      \n        \n          h\n          \n            m\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle h_{m}(x)}\n  \n from some class", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0033", "text": "from some class \n  \n    \n      \n        \n          \n            H\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {H}}}\n  \n, called base (or weak) learners:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0034", "text": "F\n              ^\n            \n          \n        \n        (\n        x\n        )\n        =\n        \n          \u2211\n          \n            m\n            =\n            1\n          \n          \n            M\n          \n        \n        \n          \u03b3\n          \n            m\n          \n        \n        \n          h\n          \n            m\n          \n        \n        (\n        x\n        )\n        +\n        \n          \n            const", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0035", "text": "+\n        \n          \n            const\n          \n        \n        ,\n      \n    \n    {\\displaystyle {\\hat {F}}(x)=\\sum _{m=1}^{M}\\gamma _{m}h_{m}(x)+{\\mbox{const}},}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0036", "text": "where \n  \n    \n      \n        \n          \u03b3\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle \\gamma _{m}}\n  \n is the weight at stage \n  \n    \n      \n        m\n      \n    \n    {\\displaystyle m}\n  \n. We are usually given a training set \n  \n    \n      \n        {\n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          y\n          \n            1\n          \n        \n        )\n        ,\n        \u2026\n        ,\n        (", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0037", "text": ",\n        \u2026\n        ,\n        (\n        \n          x\n          \n            n\n          \n        \n        ,\n        \n          y\n          \n            n\n          \n        \n        )\n        }\n      \n    \n    {\\displaystyle \\{(x_{1},y_{1}),\\dots ,(x_{n},y_{n})\\}}\n  \n of known values of x and corresponding values of y. In accordance with the empirical risk minimization principle, the method tries to find an approximation \n  \n    \n      \n        \n          \n            \n              F", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0038", "text": "F\n              ^\n            \n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle {\\hat {F}}(x)}\n  \n that minimizes the average value of the loss function on the training set, i.e., minimizes the empirical risk. It does so by starting with a model, consisting of a constant function \n  \n    \n      \n        \n          F\n          \n            0\n          \n        \n        (\n        x\n        )", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0039", "text": "(\n        x\n        )\n      \n    \n    {\\displaystyle F_{0}(x)}\n  \n, and incrementally expands it in a greedy fashion:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0040", "text": "F\n          \n            0\n          \n        \n        (\n        x\n        )\n        =\n        \n          \n            \n              arg\n              \u2061\n              min\n            \n            \n              \n                h\n                \n                  m\n                \n              \n              \u2208\n              \n                \n                  H\n                \n              \n            \n          \n        \n        \n          \u2211", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0041", "text": "\u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          L\n          (\n          \n            y\n            \n              i\n            \n          \n          ,\n          \n            h\n            \n              m\n            \n          \n          (\n          \n            x\n            \n              i\n            \n          \n          )\n          )\n        \n        ,", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0042", "text": ")\n        \n        ,\n      \n    \n    {\\displaystyle F_{0}(x)={\\underset {h_{m}\\in {\\mathcal {H}}}{\\arg \\min }}\\sum _{i=1}^{n}{L(y_{i},h_{m}(x_{i}))},}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0043", "text": "F\n          \n            m\n          \n        \n        (\n        x\n        )\n        =\n        \n          F\n          \n            m\n            \u2212\n            1\n          \n        \n        (\n        x\n        )\n        +\n        \n          (\n          \n            \n              \n                \n                  a\n                  r\n                  g\n                  \n                  m\n                  i\n                  n", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0044", "text": "n\n                \n                \n                  \n                    h\n                    \n                      m\n                    \n                  \n                  \u2208\n                  \n                    \n                      H\n                    \n                  \n                \n              \n            \n            \n              [\n              \n                \n                  \u2211\n                  \n                    i\n                    =", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0045", "text": "i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                L\n                (\n                \n                  y\n                  \n                    i\n                  \n                \n                ,\n                \n                  F\n                  \n                    m\n                    \u2212\n                    1", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0046", "text": "(\n                \n                  x\n                  \n                    i\n                  \n                \n                )\n                +\n                \n                  h\n                  \n                    m\n                  \n                \n                (\n                \n                  x\n                  \n                    i\n                  \n                \n                )\n                )", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0047", "text": ")\n                )\n              \n              ]\n            \n          \n          )\n        \n        (\n        x\n        )\n        ,\n      \n    \n    {\\displaystyle F_{m}(x)=F_{m-1}(x)+\\left({\\underset {h_{m}\\in {\\mathcal {H}}}{\\operatorname {arg\\,min} }}\\left[\\sum _{i=1}^{n}L(y_{i},F_{m-1}(x_{i})+h_{m}(x_{i}))\\right]\\right)(x),}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0048", "text": "for \n  \n    \n      \n        m\n        \u2265\n        1\n      \n    \n    {\\displaystyle m\\geq 1}\n  \n, where \n  \n    \n      \n        \n          h\n          \n            m\n          \n        \n        \u2208\n        \n          \n            H\n          \n        \n      \n    \n    {\\displaystyle h_{m}\\in {\\mathcal {H}}}\n  \n is a base learner function.\nUnfortunately, choosing the best function \n  \n    \n      \n        \n          h\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle h_{m}}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0049", "text": "{\\displaystyle h_{m}}\n  \n at each step for an arbitrary loss function L is a computationally infeasible optimization problem in general. Therefore, we restrict our approach to a simplified version of the problem. The idea is to apply a steepest descent step to this minimization problem (functional gradient descent). The basic idea is to find a local minimum of the loss function by iterating on \n  \n    \n      \n        \n          F\n          \n            m\n            \u2212", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0050", "text": "m\n            \u2212\n            1\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle F_{m-1}(x)}\n  \n. In fact, the local maximum-descent direction of the loss function is the negative gradient. Hence, moving a small amount \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n such that the linear approximation remains valid:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0051", "text": "F\n          \n            m\n          \n        \n        (\n        x\n        )\n        =\n        \n          F\n          \n            m\n            \u2212\n            1\n          \n        \n        (\n        x\n        )\n        \u2212\n        \u03b3\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          \u2207\n          \n            \n              F\n              \n                m", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0052", "text": "F\n              \n                m\n                \u2212\n                1\n              \n            \n          \n        \n        L\n        (\n        \n          y\n          \n            i\n          \n        \n        ,\n        \n          F\n          \n            m\n            \u2212\n            1\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        )", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0053", "text": ")\n        )\n      \n    \n    {\\displaystyle F_{m}(x)=F_{m-1}(x)-\\gamma \\sum _{i=1}^{n}\\nabla _{F_{m-1}}L(y_{i},F_{m-1}(x_{i}))}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0054", "text": "where \n  \n    \n      \n        \u03b3\n        >\n        0\n      \n    \n    {\\displaystyle \\gamma >0}\n  \n. For small \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n, this implies that \n  \n    \n      \n        L\n        (\n        \n          y\n          \n            i\n          \n        \n        ,\n        \n          F\n          \n            m\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        )\n        \u2264\n        L", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0055", "text": ")\n        )\n        \u2264\n        L\n        (\n        \n          y\n          \n            i\n          \n        \n        ,\n        \n          F\n          \n            m\n            \u2212\n            1\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        )\n      \n    \n    {\\displaystyle L(y_{i},F_{m}(x_{i}))\\leq L(y_{i},F_{m-1}(x_{i}))}\n  \n.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0056", "text": "Furthermore, we can optimize \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n by finding the \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n value for which the loss function has a minimum:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0057", "text": "\u03b3\n          \n            m\n          \n        \n        =\n        \n          \n            argmin\n            \u03b3\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        L\n        (\n        \n          y\n          \n            i\n          \n        \n        ,\n        \n          F\n          \n            m\n          \n        \n        (\n        \n          x", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0058", "text": "(\n        \n          x\n          \n            i\n          \n        \n        )\n        )\n        =\n        \n          \n            \n              arg\n              \u2061\n              min\n            \n            \u03b3\n          \n        \n        \n          \n            \u2211\n            \n              i\n              =\n              1\n            \n            \n              n\n            \n          \n          L\n          \n            (\n            \n              \n                y", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0059", "text": "y\n                \n                  i\n                \n              \n              ,\n              \n                F\n                \n                  m\n                  \u2212\n                  1\n                \n              \n              (\n              \n                x\n                \n                  i\n                \n              \n              )\n              \u2212\n              \u03b3\n              \n                \u2207", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0060", "text": "F\n                    \n                      m\n                      \u2212\n                      1\n                    \n                  \n                \n              \n              L\n              (\n              \n                y\n                \n                  i\n                \n              \n              ,\n              \n                F\n                \n                  m\n                  \u2212\n                  1", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0061", "text": "1\n                \n              \n              (\n              \n                x\n                \n                  i\n                \n              \n              )\n              )\n            \n            )\n          \n        \n        .", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0062", "text": ".\n      \n    \n    {\\displaystyle \\gamma _{m}={\\underset {\\gamma }{\\operatorname {argmin} }}\\sum _{i=1}^{n}L(y_{i},F_{m}(x_{i}))={\\underset {\\gamma }{\\arg \\min }}{\\sum _{i=1}^{n}L\\left(y_{i},F_{m-1}(x_{i})-\\gamma \\nabla _{F_{m-1}}L(y_{i},F_{m-1}(x_{i}))\\right)}.}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0063", "text": "If we considered the continuous case, i.e., where \n  \n    \n      \n        \n          \n            H\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {H}}}\n  \n is the set of arbitrary differentiable functions on \n  \n    \n      \n        \n          R\n        \n      \n    \n    {\\displaystyle \\mathbb {R} }\n  \n, we would update the model in accordance with the following equations", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0064", "text": "F\n          \n            m\n          \n        \n        (\n        x\n        )\n        =\n        \n          F\n          \n            m\n            \u2212\n            1\n          \n        \n        (\n        x\n        )\n        \u2212\n        \n          \u03b3\n          \n            m\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          \n            \u2207", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0065", "text": "\u2207\n            \n              \n                F\n                \n                  m\n                  \u2212\n                  1\n                \n              \n            \n          \n          L\n          (\n          \n            y\n            \n              i\n            \n          \n          ,\n          \n            F\n            \n              m\n              \u2212\n              1\n            \n          \n          (\n          \n            x", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0066", "text": "(\n          \n            x\n            \n              i\n            \n          \n          )\n          )\n        \n      \n    \n    {\\displaystyle F_{m}(x)=F_{m-1}(x)-\\gamma _{m}\\sum _{i=1}^{n}{\\nabla _{F_{m-1}}L(y_{i},F_{m-1}(x_{i}))}}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0067", "text": "where \n  \n    \n      \n        \n          \u03b3\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle \\gamma _{m}}\n  \n is the step length, defined as", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0068", "text": "\u03b3\n          \n            m\n          \n        \n        =\n        \n          \n            \n              arg\n              \u2061\n              min\n            \n            \u03b3\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        L\n        \n          (\n          \n            \n              y\n              \n                i\n              \n            \n            ,", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0069", "text": ",\n            \n              F\n              \n                m\n                \u2212\n                1\n              \n            \n            (\n            \n              x\n              \n                i\n              \n            \n            )\n            \u2212\n            \u03b3\n            \n              \u2207\n              \n                \n                  F\n                  \n                    m\n                    \u2212\n                    1", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0070", "text": "1\n                  \n                \n              \n            \n            L\n            (\n            \n              y\n              \n                i\n              \n            \n            ,\n            \n              F\n              \n                m\n                \u2212\n                1\n              \n            \n            (\n            \n              x\n              \n                i\n              \n            \n            )\n            )\n          \n          )", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0071", "text": ")\n          \n          )\n        \n        .\n      \n    \n    {\\displaystyle \\gamma _{m}={\\underset {\\gamma }{\\arg \\min }}\\sum _{i=1}^{n}L\\left(y_{i},F_{m-1}(x_{i})-\\gamma \\nabla _{F_{m-1}}L(y_{i},F_{m-1}(x_{i}))\\right).}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0072", "text": "In the discrete case however, i.e. when the set \n  \n    \n      \n        \n          \n            H\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {H}}}\n  \n is finite, we choose the candidate function h closest to the gradient of L for which the coefficient \u03b3 may then be calculated with the aid of line search on the above equations. Note that this approach is a heuristic and therefore doesn't yield an exact solution to the given problem, but rather an approximation.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0073", "text": "In pseudocode, the generic gradient boosting method is:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0074", "text": "Gradient tree boosting\nGradient boosting is typically used with decision trees (especially CARTs) of a fixed size as base learners. For this special case, Friedman proposes a modification to gradient boosting method which improves the quality of fit of each base learner.\nGeneric gradient boosting at the m-th step would fit a decision tree \n  \n    \n      \n        \n          h\n          \n            m\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle h_{m}(x)}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0075", "text": "{\\displaystyle h_{m}(x)}\n  \n to pseudo-residuals. Let \n  \n    \n      \n        \n          J\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle J_{m}}\n  \n be the number of its leaves. The tree partitions the input space into \n  \n    \n      \n        \n          J\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle J_{m}}\n  \n disjoint regions \n  \n    \n      \n        \n          R\n          \n            1\n            m", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0076", "text": "1\n            m\n          \n        \n        ,\n        \u2026\n        ,\n        \n          R\n          \n            \n              J\n              \n                m\n              \n            \n            m\n          \n        \n      \n    \n    {\\displaystyle R_{1m},\\ldots ,R_{J_{m}m}}\n  \n and predicts a constant value in each region. Using the indicator notation, the output of \n  \n    \n      \n        \n          h\n          \n            m\n          \n        \n        (\n        x", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0077", "text": "(\n        x\n        )\n      \n    \n    {\\displaystyle h_{m}(x)}\n  \n for input x can be written as the sum:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0078", "text": "h\n          \n            m\n          \n        \n        (\n        x\n        )\n        =\n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            \n              J\n              \n                m\n              \n            \n          \n        \n        \n          b\n          \n            j\n            m\n          \n        \n        \n          \n            1\n          \n          \n            \n              R", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0079", "text": "R\n              \n                j\n                m\n              \n            \n          \n        \n        (\n        x\n        )\n        ,\n      \n    \n    {\\displaystyle h_{m}(x)=\\sum _{j=1}^{J_{m}}b_{jm}\\mathbf {1} _{R_{jm}}(x),}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0080", "text": "where \n  \n    \n      \n        \n          b\n          \n            j\n            m\n          \n        \n      \n    \n    {\\displaystyle b_{jm}}\n  \n is the value predicted in the region \n  \n    \n      \n        \n          R\n          \n            j\n            m\n          \n        \n      \n    \n    {\\displaystyle R_{jm}}\n  \n.\nThen the coefficients \n  \n    \n      \n        \n          b\n          \n            j\n            m\n          \n        \n      \n    \n    {\\displaystyle b_{jm}}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0081", "text": "{\\displaystyle b_{jm}}\n  \n are multiplied by some value \n  \n    \n      \n        \n          \u03b3\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle \\gamma _{m}}\n  \n, chosen using line search so as to minimize the loss function, and the model is updated as follows:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0082", "text": "F\n          \n            m\n          \n        \n        (\n        x\n        )\n        =\n        \n          F\n          \n            m\n            \u2212\n            1\n          \n        \n        (\n        x\n        )\n        +\n        \n          \u03b3\n          \n            m\n          \n        \n        \n          h\n          \n            m\n          \n        \n        (\n        x\n        )\n        ,\n        \n        \n          \u03b3\n          \n            m", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0083", "text": "\u03b3\n          \n            m\n          \n        \n        =\n        \n          \n            \n              a\n              r\n              g\n              \n              m\n              i\n              n\n            \n            \u03b3\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        L\n        (\n        \n          y\n          \n            i\n          \n        \n        ,", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0084", "text": ",\n        \n          F\n          \n            m\n            \u2212\n            1\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        +\n        \u03b3\n        \n          h\n          \n            m\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        )\n        .", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0085", "text": ")\n        )\n        .\n      \n    \n    {\\displaystyle F_{m}(x)=F_{m-1}(x)+\\gamma _{m}h_{m}(x),\\quad \\gamma _{m}={\\underset {\\gamma }{\\operatorname {arg\\,min} }}\\sum _{i=1}^{n}L(y_{i},F_{m-1}(x_{i})+\\gamma h_{m}(x_{i})).}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0086", "text": "Friedman proposes to modify this algorithm so that it chooses a separate optimal value \n  \n    \n      \n        \n          \u03b3\n          \n            j\n            m\n          \n        \n      \n    \n    {\\displaystyle \\gamma _{jm}}\n  \n for each of the tree's regions, instead of a single \n  \n    \n      \n        \n          \u03b3\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle \\gamma _{m}}\n  \n for the whole tree. He calls the modified algorithm \"TreeBoost\". The coefficients", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0087", "text": "b\n          \n            j\n            m\n          \n        \n      \n    \n    {\\displaystyle b_{jm}}\n  \n from the tree-fitting procedure can be then simply discarded and the model update rule becomes:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0088", "text": "F\n          \n            m\n          \n        \n        (\n        x\n        )\n        =\n        \n          F\n          \n            m\n            \u2212\n            1\n          \n        \n        (\n        x\n        )\n        +\n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            \n              J\n              \n                m\n              \n            \n          \n        \n        \n          \u03b3", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0089", "text": "\u03b3\n          \n            j\n            m\n          \n        \n        \n          \n            1\n          \n          \n            \n              R\n              \n                j\n                m\n              \n            \n          \n        \n        (\n        x\n        )\n        ,\n        \n        \n          \u03b3\n          \n            j\n            m\n          \n        \n        =\n        \n          \n            \n              a\n              r\n              g", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0090", "text": "a\n              r\n              g\n              \n              m\n              i\n              n\n            \n            \u03b3\n          \n        \n        \n          \u2211\n          \n            \n              x\n              \n                i\n              \n            \n            \u2208\n            \n              R\n              \n                j\n                m\n              \n            \n          \n        \n        L\n        (\n        \n          y\n          \n            i", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0091", "text": "y\n          \n            i\n          \n        \n        ,\n        \n          F\n          \n            m\n            \u2212\n            1\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        +\n        \u03b3\n        )\n        .", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0092", "text": "\u03b3\n        )\n        .\n      \n    \n    {\\displaystyle F_{m}(x)=F_{m-1}(x)+\\sum _{j=1}^{J_{m}}\\gamma _{jm}\\mathbf {1} _{R_{jm}}(x),\\quad \\gamma _{jm}={\\underset {\\gamma }{\\operatorname {arg\\,min} }}\\sum _{x_{i}\\in R_{jm}}L(y_{i},F_{m-1}(x_{i})+\\gamma ).}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0093", "text": "When the loss \n  \n    \n      \n        L\n        (\n        \u22c5\n        ,\n        \u22c5\n        )\n      \n    \n    {\\displaystyle L(\\cdot ,\\cdot )}\n  \n is mean-squared error (MSE) the coefficients \n  \n    \n      \n        \n          \u03b3\n          \n            j\n            m\n          \n        \n      \n    \n    {\\displaystyle \\gamma _{jm}}\n  \n coincide with the coefficients of the tree-fitting procedure \n  \n    \n      \n        \n          b\n          \n            j\n            m", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0094", "text": "m\n          \n        \n      \n    \n    {\\displaystyle b_{jm}}\n  \n.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0095", "text": "Tree size\nThe number \n  \n    \n      \n        J\n      \n    \n    {\\displaystyle J}\n  \n of terminal nodes in the trees is a parameter which controls the maximum allowed level of interaction between variables in the model. With \n  \n    \n      \n        J\n        =\n        2\n      \n    \n    {\\displaystyle J=2}\n  \n (decision stumps), no interaction between variables is allowed. With \n  \n    \n      \n        J\n        =\n        3\n      \n    \n    {\\displaystyle J=3}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0096", "text": "3\n      \n    \n    {\\displaystyle J=3}\n  \n the model may include effects of the interaction between up to two variables, and so on. \n  \n    \n      \n        J\n      \n    \n    {\\displaystyle J}\n  \n can be adjusted for a data set at hand.\nHastie et al. comment that typically \n  \n    \n      \n        4\n        \u2264\n        J\n        \u2264\n        8\n      \n    \n    {\\displaystyle 4\\leq J\\leq 8}\n  \n work well for boosting and results are fairly insensitive to the choice of \n  \n    \n      \n        J", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0097", "text": "J\n      \n    \n    {\\displaystyle J}\n  \n in this range, \n  \n    \n      \n        J\n        =\n        2\n      \n    \n    {\\displaystyle J=2}\n  \n is insufficient for many applications, and \n  \n    \n      \n        J\n        >\n        10\n      \n    \n    {\\displaystyle J>10}\n  \n is unlikely to be required.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0098", "text": "Regularization\nFitting the training set too closely can lead to degradation of the model's generalization ability, that is, its performance on unseen examples. Several so-called regularization techniques reduce this overfitting effect by constraining the fitting procedure.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0099", "text": "One natural regularization parameter is the number of gradient boosting iterations M (i.e. the number of base models). Increasing M reduces the error on training set, but increases risk of overfitting. An optimal value of M is often selected by monitoring prediction error on a separate validation data set.\nAnother regularization parameter for tree boosting is tree depth. The higher this value the more likely the model will overfit the training data.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0100", "text": "Shrinkage\nAn important part of gradient boosting is regularization by shrinkage which uses a modified update rule:", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0101", "text": "F\n          \n            m\n          \n        \n        (\n        x\n        )\n        =\n        \n          F\n          \n            m\n            \u2212\n            1\n          \n        \n        (\n        x\n        )\n        +\n        \u03bd\n        \u22c5\n        \n          \u03b3\n          \n            m\n          \n        \n        \n          h\n          \n            m\n          \n        \n        (\n        x\n        )\n        ,\n        \n        0\n        <\n        \u03bd\n        \u2264", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0102", "text": "0\n        <\n        \u03bd\n        \u2264\n        1\n        ,\n      \n    \n    {\\displaystyle F_{m}(x)=F_{m-1}(x)+\\nu \\cdot \\gamma _{m}h_{m}(x),\\quad 0<\\nu \\leq 1,}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0103", "text": "where parameter \n  \n    \n      \n        \u03bd\n      \n    \n    {\\displaystyle \\nu }\n  \n is called the \"learning rate\".\nEmpirically, it has been found that using small learning rates (such as \n  \n    \n      \n        \u03bd\n        <\n        0.1\n      \n    \n    {\\displaystyle \\nu <0.1}\n  \n) yields dramatic improvements in models' generalization ability over gradient boosting without shrinking (\n  \n    \n      \n        \u03bd\n        =\n        1\n      \n    \n    {\\displaystyle \\nu =1}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0104", "text": "{\\displaystyle \\nu =1}\n  \n). However, it comes at the price of increasing computational time both during training and querying: lower learning rate requires more iterations.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0105", "text": "Stochastic gradient boosting\nSoon after the introduction of gradient boosting, Friedman proposed a minor modification to the algorithm, motivated by Breiman's bootstrap aggregation (\"bagging\") method. Specifically, he proposed that at each iteration of the algorithm, a base learner should be fit on a subsample of the training set drawn at random without replacement.  Friedman observed a substantial improvement in gradient boosting's accuracy with this modification.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0106", "text": "Subsample size is some constant fraction \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n of the size of the training set. When \n  \n    \n      \n        f\n        =\n        1\n      \n    \n    {\\displaystyle f=1}\n  \n, the algorithm is deterministic and identical to the one described above. Smaller values of \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0107", "text": "f\n      \n    \n    {\\displaystyle f}\n  \n introduce randomness into the algorithm and help prevent overfitting, acting as a kind of regularization. The algorithm also becomes faster, because regression trees have to be fit to smaller datasets at each iteration. Friedman obtained that \n  \n    \n      \n        0.5\n        \u2264\n        f\n        \u2264\n        0.8\n      \n    \n    {\\displaystyle 0.5\\leq f\\leq 0.8}\n  \n leads to good results for small and moderate sized training sets. Therefore,", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0108", "text": "f\n      \n    \n    {\\displaystyle f}\n  \n is typically set to 0.5, meaning that one half of the training set is used to build each base learner.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0109", "text": "Also, like in bagging, subsampling allows one to define an out-of-bag error of the prediction performance improvement by evaluating predictions on those observations which were not used in the building of the next base learner. Out-of-bag estimates help avoid the need for an independent validation dataset, but often underestimate actual performance improvement and the optimal number of iterations.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0110", "text": "Number of observations in leaves\nGradient tree boosting implementations often also use regularization by limiting the minimum number of observations in trees' terminal nodes. It is used in the tree building process by ignoring any splits that lead to nodes containing fewer than this number of training set instances.\nImposing this limit helps to reduce variance in predictions at leaves.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0111", "text": "Complexity penalty\nAnother useful regularization technique for gradient boosted model is to penalize its complexity. For gradient boosted trees, model complexity can be defined as the proportional number of leaves in the trees. The joint optimization of loss and model complexity corresponds to a post-pruning algorithm to remove branches that fail to reduce the loss by a threshold. \nOther kinds of regularization such as an \n  \n    \n      \n        \n          \u2113\n          \n            2", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0112", "text": "\u2113\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\ell _{2}}\n  \n penalty on the leaf values can also be used to avoid overfitting.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0113", "text": "Usage", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0114", "text": "Gradient boosting can be used in the field of learning to rank. The commercial web search engines Yahoo and Yandex use variants of gradient boosting in their machine-learned ranking engines. Gradient boosting is also utilized in High Energy Physics in data analysis. At the Large Hadron Collider (LHC), variants of gradient boosting Deep Neural Networks (DNN) were successful in reproducing the results of non-machine learning methods of analysis on datasets used to discover the Higgs boson", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0115", "text": ". Gradient boosting decision tree was also applied in earth and geological studies \u2013 for example quality evaluation of sandstone reservoir.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0116", "text": "Names\nThe method goes by a variety of names. Friedman introduced his regression technique as a \"Gradient Boosting Machine\" (GBM). Mason, Baxter et al. described the generalized abstract class of algorithms as \"functional gradient boosting\". Friedman et al. describe an advancement of  gradient boosted models as Multiple Additive Regression Trees (MART); Elith et al. describe that approach as \"Boosted Regression Trees\" (BRT).", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0117", "text": "A popular open-source implementation for R calls it a \"Generalized Boosting Model\", however packages expanding this work use BRT. Yet another name is TreeNet, after an early commercial implementation from Salford System's Dan Steinberg, one of researchers who pioneered the use of tree-based methods.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0118", "text": "Feature importance ranking\nGradient boosting can be used for feature importance ranking, which is usually based on aggregating importance function of the base learners. For example, if a gradient boosted trees algorithm is developed using entropy-based decision trees, the ensemble algorithm ranks the importance of features based on entropy as well with the caveat that it is averaged out over all base learners.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0119", "text": "Disadvantages", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0120", "text": "While boosting can increase the accuracy of a base learner, such as a decision tree or linear regression, it sacrifices intelligibility and interpretability. For example, following the path that a decision tree takes to make its decision is trivial and self-explained, but following the paths of hundreds or thousands of trees is much harder", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0121", "text": ". To achieve both performance and interpretability, some model compression techniques allow transforming an XGBoost into a single \"born-again\" decision tree that approximates the same decision function. Furthermore, its implementation may be more difficult due to the higher computational demand.", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_boosting_0122", "text": "See also\nAdaBoost\nRandom forest\nCatboost\nLightGBM\nXGBoost\nDecision tree learning\n\nReferences\nFurther reading\nBoehmke, Bradley; Greenwell, Brandon (2019). \"Gradient Boosting\". Hands-On Machine Learning with R. Chapman & Hall. pp. 221\u2013245. ISBN 978-1-138-49568-5.\n\nExternal links\nHow to explain gradient boosting\nGradient Boosted Regression Trees\nLightGBM", "title": "Gradient boosting", "url": "https://en.wikipedia.org/wiki/Gradient_boosting"}, {"id": "rec_Gradient_descent_0000", "text": "Gradient descent is a method for unconstrained mathematical optimization. It is a first-order iterative algorithm for minimizing a differentiable multivariate function.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0001", "text": "The idea is to take repeated steps in the opposite direction of the gradient (or approximate gradient) of the function at the current point, because this is the direction of steepest descent. Conversely, stepping in the direction of the gradient will lead to a trajectory that maximizes that function; the procedure is then known as gradient ascent.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0002", "text": "It is particularly useful in machine learning and artificial intelligence for minimizing the cost or loss function. Gradient descent should not be confused with local search algorithms, although both are iterative methods for optimization.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0003", "text": "Gradient descent is generally attributed to Augustin-Louis Cauchy, who first suggested it in 1847. Jacques Hadamard independently proposed a similar method in 1907. Its convergence properties for non-linear optimization problems were first studied by Haskell Curry in 1944, with the method becoming increasingly well-studied and used in the following decades.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0004", "text": "A simple extension of gradient descent, stochastic gradient descent, serves as the most basic algorithm used for training most deep networks today.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0005", "text": "Description\nGradient descent is based on the observation that if the multi-variable function \n  \n    \n      \n        f\n        (\n        \n          x\n        \n        )\n      \n    \n    {\\displaystyle f(\\mathbf {x} )}\n  \n is defined and differentiable in a neighborhood of a point \n  \n    \n      \n        \n          a\n        \n      \n    \n    {\\displaystyle \\mathbf {a} }\n  \n, then \n  \n    \n      \n        f\n        (\n        \n          x\n        \n        )", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0006", "text": "x\n        \n        )\n      \n    \n    {\\displaystyle f(\\mathbf {x} )}\n  \n decreases fastest if one goes from \n  \n    \n      \n        \n          a\n        \n      \n    \n    {\\displaystyle \\mathbf {a} }\n  \n in the direction of the negative gradient of \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n at \n  \n    \n      \n        \n          a\n        \n        ,\n        \u2212\n        \u2207\n        f\n        (\n        \n          a\n        \n        )", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0007", "text": "a\n        \n        )\n      \n    \n    {\\displaystyle \\mathbf {a} ,-\\nabla f(\\mathbf {a} )}\n  \n. It follows that, if", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0008", "text": "a\n          \n          \n            n\n            +\n            1\n          \n        \n        =\n        \n          \n            a\n          \n          \n            n\n          \n        \n        \u2212\n        \u03b7\n        \u2207\n        f\n        (\n        \n          \n            a\n          \n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle \\mathbf {a} _{n+1}=\\mathbf {a} _{n}-\\eta \\nabla f(\\mathbf {a} _{n})}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0009", "text": "for a small enough step size or learning rate \n  \n    \n      \n        \u03b7\n        \u2208\n        \n          \n            R\n          \n          \n            +\n          \n        \n      \n    \n    {\\displaystyle \\eta \\in \\mathbb {R} _{+}}\n  \n, then  \n  \n    \n      \n        f\n        (\n        \n          \n            a\n            \n              n\n            \n          \n        \n        )\n        \u2265\n        f\n        (\n        \n          \n            a\n            \n              n\n              +", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0010", "text": "n\n              +\n              1\n            \n          \n        \n        )\n      \n    \n    {\\displaystyle f(\\mathbf {a_{n}} )\\geq f(\\mathbf {a_{n+1}} )}\n  \n. In other words, the term \n  \n    \n      \n        \u03b7\n        \u2207\n        f\n        (\n        \n          a\n        \n        )\n      \n    \n    {\\displaystyle \\eta \\nabla f(\\mathbf {a} )}\n  \n is subtracted from \n  \n    \n      \n        \n          a\n        \n      \n    \n    {\\displaystyle \\mathbf {a} }", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0011", "text": "{\\displaystyle \\mathbf {a} }\n  \n because we want to move against the gradient, toward the local minimum. With this observation in mind, one starts with a guess \n  \n    \n      \n        \n          \n            x\n          \n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{0}}\n  \n for a local minimum of \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n, and considers the sequence \n  \n    \n      \n        \n          \n            x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0012", "text": "x\n          \n          \n            0\n          \n        \n        ,\n        \n          \n            x\n          \n          \n            1\n          \n        \n        ,\n        \n          \n            x\n          \n          \n            2\n          \n        \n        ,\n        \u2026\n      \n    \n    {\\displaystyle \\mathbf {x} _{0},\\mathbf {x} _{1},\\mathbf {x} _{2},\\ldots }\n  \n such that", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0013", "text": "x\n          \n          \n            n\n            +\n            1\n          \n        \n        =\n        \n          \n            x\n          \n          \n            n\n          \n        \n        \u2212\n        \n          \u03b7\n          \n            n\n          \n        \n        \u2207\n        f\n        (\n        \n          \n            x\n          \n          \n            n\n          \n        \n        )\n        ,\n         \n        n\n        \u2265\n        0.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0014", "text": "n\n        \u2265\n        0.\n      \n    \n    {\\displaystyle \\mathbf {x} _{n+1}=\\mathbf {x} _{n}-\\eta _{n}\\nabla f(\\mathbf {x} _{n}),\\ n\\geq 0.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0015", "text": "We have a monotonic sequence", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0016", "text": "f\n        (\n        \n          \n            x\n          \n          \n            0\n          \n        \n        )\n        \u2265\n        f\n        (\n        \n          \n            x\n          \n          \n            1\n          \n        \n        )\n        \u2265\n        f\n        (\n        \n          \n            x\n          \n          \n            2\n          \n        \n        )\n        \u2265\n        \u22ef\n        ,", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0017", "text": "\u2265\n        \u22ef\n        ,\n      \n    \n    {\\displaystyle f(\\mathbf {x} _{0})\\geq f(\\mathbf {x} _{1})\\geq f(\\mathbf {x} _{2})\\geq \\cdots ,}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0018", "text": "so the sequence \n  \n    \n      \n        (\n        \n          \n            x\n          \n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle (\\mathbf {x} _{n})}\n  \n converges to the desired local minimum. Note that the value of the step size \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n is allowed to change at every iteration. \nIt is possible to guarantee the convergence to a local minimum under certain assumptions on the function", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0019", "text": "f\n      \n    \n    {\\displaystyle f}\n  \n (for example, \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n convex and \n  \n    \n      \n        \u2207\n        f\n      \n    \n    {\\displaystyle \\nabla f}\n  \n Lipschitz) and particular choices of \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n. Those include the sequence", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0020", "text": "\u03b7\n          \n            n\n          \n        \n        =\n        \n          \n            \n              |\n              \n                \n                  \n                    (\n                    \n                      \n                        \n                          x\n                        \n                        \n                          n\n                        \n                      \n                      \u2212", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0021", "text": "\u2212\n                      \n                        \n                          x\n                        \n                        \n                          n\n                          \u2212\n                          1\n                        \n                      \n                    \n                    )\n                  \n                  \n                    \u22a4\n                  \n                \n                \n                  [\n                  \n                    \u2207", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0022", "text": "\u2207\n                    f\n                    (\n                    \n                      \n                        x\n                      \n                      \n                        n\n                      \n                    \n                    )\n                    \u2212\n                    \u2207\n                    f\n                    (\n                    \n                      \n                        x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0023", "text": "n\n                        \u2212\n                        1\n                      \n                    \n                    )\n                  \n                  ]\n                \n              \n              |\n            \n            \n              \n                \u2016\n                \n                  \u2207\n                  f\n                  (\n                  \n                    \n                      x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0024", "text": "x\n                    \n                    \n                      n\n                    \n                  \n                  )\n                  \u2212\n                  \u2207\n                  f\n                  (\n                  \n                    \n                      x\n                    \n                    \n                      n\n                      \u2212\n                      1\n                    \n                  \n                  )", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0025", "text": ")\n                \n                \u2016\n              \n              \n                2\n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\eta _{n}={\\frac {\\left|\\left(\\mathbf {x} _{n}-\\mathbf {x} _{n-1}\\right)^{\\top }\\left[\\nabla f(\\mathbf {x} _{n})-\\nabla f(\\mathbf {x} _{n-1})\\right]\\right|}{\\left\\|\\nabla f(\\mathbf {x} _{n})-\\nabla f(\\mathbf {x} _{n-1})\\right\\|^{2}}}}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0026", "text": "as in the Barzilai-Borwein method, or a sequence \n  \n    \n      \n        \n          \u03b7\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\eta _{n}}\n  \n satisfying the Wolfe conditions (which can be found by using line search). When the function \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is convex, all local minima are also global minima, so in this case gradient descent can converge to the global solution.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0027", "text": "This process is illustrated in the adjacent picture. Here, \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is assumed to be defined on the plane, and that its graph has a bowl shape.  The blue curves are the contour lines, that is, the regions on which the value of \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0028", "text": "f\n      \n    \n    {\\displaystyle f}\n  \n is constant. A red arrow originating at a point shows the direction of the negative gradient at that point. Note that the (negative) gradient at a point is orthogonal to the contour line going through that point. We see that gradient descent leads us to the bottom of the bowl, that is, to the point where the value of the function \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is minimal.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0029", "text": "An analogy for understanding gradient descent", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0030", "text": "The basic intuition behind gradient descent can be illustrated by a hypothetical scenario. People are stuck in the mountains and are trying to get down (i.e., trying to find the global minimum). There is heavy fog such that visibility is extremely low. Therefore, the path down the mountain is not visible, so they must use local information to find the minimum", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0031", "text": ". They can use the method of gradient descent, which involves looking at the steepness of the hill at their current position, then proceeding in the direction with the steepest descent (i.e., downhill). If they were trying to find the top of the mountain (i.e., the maximum), then they would proceed in the direction of steepest ascent (i.e., uphill)", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0032", "text": ". Using this method, they would eventually find their way down the mountain or possibly get stuck in some hole (i.e., local minimum or saddle point), like a mountain lake. However, assume also that the steepness of the hill is not immediately obvious with simple observation, but rather it requires a sophisticated instrument to measure, which the people happen to have at that moment. It takes quite some time to measure the steepness of the hill with the instrument", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0033", "text": ". Thus, they should minimize their use of the instrument if they want to get down the mountain before sunset. The difficulty then is choosing the frequency at which they should measure the steepness of the hill so as not to go off track.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0034", "text": "In this analogy, the people represent the algorithm, and the path taken down the mountain represents the sequence of parameter settings that the algorithm will explore. The steepness of the hill represents the slope of the function at that point. The instrument used to measure steepness is differentiation. The direction they choose to travel in aligns with the gradient of the function at that point. The amount of time they travel before taking another measurement is the step size.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0035", "text": "Choosing the step size and descent direction\nSince using a step size \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n that is too small would slow convergence, and a \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n too large would lead to overshoot and divergence, finding a good setting of \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0036", "text": "{\\displaystyle \\eta }\n  \n is an important practical problem. Philip Wolfe also advocated using \"clever choices of the [descent] direction\" in practice. While using a direction that deviates from the steepest descent direction may seem counter-intuitive, the idea is that the smaller slope may be compensated for by being sustained over a much longer distance.\nTo reason about this mathematically, consider a direction \n  \n    \n      \n        \n          \n            p", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0037", "text": "p\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {p} _{n}}\n  \n and step size \n  \n    \n      \n        \n          \u03b7\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\eta _{n}}\n  \n and consider the more general update:", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0038", "text": "a\n          \n          \n            n\n            +\n            1\n          \n        \n        =\n        \n          \n            a\n          \n          \n            n\n          \n        \n        \u2212\n        \n          \u03b7\n          \n            n\n          \n        \n        \n        \n          \n            p\n          \n          \n            n", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0039", "text": "n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {a} _{n+1}=\\mathbf {a} _{n}-\\eta _{n}\\,\\mathbf {p} _{n}}\n  \n.\nFinding good settings of \n  \n    \n      \n        \n          \n            p\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {p} _{n}}\n  \n and \n  \n    \n      \n        \n          \u03b7\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\eta _{n}}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0040", "text": "{\\displaystyle \\eta _{n}}\n  \n requires some thought. First of all, we would like the update direction to point downhill. Mathematically, letting \n  \n    \n      \n        \n          \u03b8\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\theta _{n}}\n  \n denote the angle between \n  \n    \n      \n        \u2212\n        \u2207\n        f\n        (\n        \n          \n            a\n            \n              n\n            \n          \n        \n        )", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0041", "text": ")\n      \n    \n    {\\displaystyle -\\nabla f(\\mathbf {a_{n}} )}\n  \n and \n  \n    \n      \n        \n          \n            p\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {p} _{n}}\n  \n, this requires that \n  \n    \n      \n        cos\n        \u2061\n        \n          \u03b8\n          \n            n\n          \n        \n        >\n        0.\n      \n    \n    {\\displaystyle \\cos \\theta _{n}>0.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0042", "text": "{\\displaystyle \\cos \\theta _{n}>0.}\n  \n To say more, we need more information about the objective function that we are optimising. Under the fairly weak assumption that \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is continuously differentiable, we may prove that:", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0043", "text": "This inequality implies that the amount by which we can be sure the function \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is decreased depends on a trade off between the two terms in square brackets. The first term in square brackets measures the angle between the descent direction and the negative gradient. The second term measures how quickly the gradient changes along the descent direction.\nIn principle inequality (1) could be optimized over", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0044", "text": "p\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {p} _{n}}\n  \n and \n  \n    \n      \n        \n          \u03b7\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\eta _{n}}\n  \n to choose an optimal step size and direction. The problem is that evaluating the second term in square brackets requires evaluating \n  \n    \n      \n        \u2207\n        f\n        (", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0045", "text": "\u2207\n        f\n        (\n        \n          \n            a\n          \n          \n            n\n          \n        \n        \u2212\n        t\n        \n          \u03b7\n          \n            n\n          \n        \n        \n          \n            p\n          \n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle \\nabla f(\\mathbf {a} _{n}-t\\eta _{n}\\mathbf {p} _{n})}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0046", "text": ", and extra gradient evaluations are generally expensive and undesirable. Some ways around this problem are:", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0047", "text": "Forgo the benefits of a clever descent direction by setting \n  \n    \n      \n        \n          \n            p\n          \n          \n            n\n          \n        \n        =\n        \u2207\n        f\n        (\n        \n          \n            a\n            \n              n\n            \n          \n        \n        )\n      \n    \n    {\\displaystyle \\mathbf {p} _{n}=\\nabla f(\\mathbf {a_{n}} )}\n  \n, and use line search to find a suitable step-size \n  \n    \n      \n        \n          \u03b3", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0048", "text": "\u03b3\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\gamma _{n}}\n  \n, such as one that satisfies the Wolfe conditions. A more economic way of choosing learning rates is backtracking line search, a method that has both good theoretical guarantees and experimental results. Note that one does not need to choose  \n  \n    \n      \n        \n          \n            p\n          \n          \n            n", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0049", "text": "n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {p} _{n}}\n  \n to be the gradient; any direction that has positive inner product with the gradient will result in a reduction of the function value (for a sufficiently small value of \n  \n    \n      \n        \n          \u03b7\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\eta _{n}}\n  \n).\nAssuming that \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0050", "text": "f\n      \n    \n    {\\displaystyle f}\n  \n is twice-differentiable, use its Hessian \n  \n    \n      \n        \n          \u2207\n          \n            2\n          \n        \n        f\n      \n    \n    {\\displaystyle \\nabla ^{2}f}\n  \n to estimate \n  \n    \n      \n        \u2016\n        \u2207\n        f\n        (\n        \n          \n            a\n          \n          \n            n\n          \n        \n        \u2212\n        t\n        \n          \u03b7\n          \n            n", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0051", "text": "p\n          \n          \n            n\n          \n        \n        )\n        \u2212\n        \u2207\n        f\n        (\n        \n          \n            a\n          \n          \n            n\n          \n        \n        )\n        \n          \u2016\n          \n            2\n          \n        \n        \u2248\n        \u2016\n        t\n        \n          \u03b7\n          \n            n\n          \n        \n        \n          \u2207\n          \n            2\n          \n        \n        f", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0052", "text": "2\n          \n        \n        f\n        (\n        \n          \n            a\n          \n          \n            n\n          \n        \n        )\n        \n          \n            p\n          \n          \n            n\n          \n        \n        \u2016\n        .\n      \n    \n    {\\displaystyle \\|\\nabla f(\\mathbf {a} _{n}-t\\eta _{n}\\mathbf {p} _{n})-\\nabla f(\\mathbf {a} _{n})\\|_{2}\\approx \\|t\\eta _{n}\\nabla ^{2}f(\\mathbf {a} _{n})\\mathbf {p} _{n}\\|.}\n  \nThen choose", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0053", "text": "Then choose \n  \n    \n      \n        \n          \n            p\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {p} _{n}}\n  \n and \n  \n    \n      \n        \n          \u03b7\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\eta _{n}}\n  \n by optimising inequality (1).\nAssuming that \n  \n    \n      \n        \u2207\n        f\n      \n    \n    {\\displaystyle \\nabla f}\n  \n is Lipschitz, use its Lipschitz constant \n  \n    \n      \n        L", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0054", "text": "L\n      \n    \n    {\\displaystyle L}\n  \n to bound \n  \n    \n      \n        \u2016\n        \u2207\n        f\n        (\n        \n          \n            a\n          \n          \n            n\n          \n        \n        \u2212\n        t\n        \n          \u03b7\n          \n            n\n          \n        \n        \n          \n            p\n          \n          \n            n\n          \n        \n        )\n        \u2212\n        \u2207\n        f\n        (\n        \n          \n            a", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0055", "text": "a\n          \n          \n            n\n          \n        \n        )\n        \n          \u2016\n          \n            2\n          \n        \n        \u2264\n        L\n        t\n        \n          \u03b7\n          \n            n\n          \n        \n        \u2016\n        \n          \n            p\n          \n          \n            n\n          \n        \n        \u2016\n        .", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0056", "text": "\u2016\n        .\n      \n    \n    {\\displaystyle \\|\\nabla f(\\mathbf {a} _{n}-t\\eta _{n}\\mathbf {p} _{n})-\\nabla f(\\mathbf {a} _{n})\\|_{2}\\leq Lt\\eta _{n}\\|\\mathbf {p} _{n}\\|.}\n  \n Then choose \n  \n    \n      \n        \n          \n            p\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {p} _{n}}\n  \n and \n  \n    \n      \n        \n          \u03b7\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\eta _{n}}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0057", "text": "{\\displaystyle \\eta _{n}}\n  \n by optimising inequality (1).\nBuild a custom model of \n  \n    \n      \n        \n          max\n          \n            t\n            \u2208\n            [\n            0\n            ,\n            1\n            ]\n          \n        \n        \n          \n            \n              \u2016\n              \u2207\n              f\n              (\n              \n                \n                  a\n                \n                \n                  n", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0058", "text": "n\n                \n              \n              \u2212\n              t\n              \n                \u03b7\n                \n                  n\n                \n              \n              \n                \n                  p\n                \n                \n                  n\n                \n              \n              )\n              \u2212\n              \u2207\n              f\n              (\n              \n                \n                  a", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0059", "text": "n\n                \n              \n              )\n              \n                \u2016\n                \n                  2\n                \n              \n            \n            \n              \u2016\n              \u2207\n              f\n              (\n              \n                \n                  a\n                \n                \n                  n\n                \n              \n              )\n              \n                \u2016", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0060", "text": "\u2016\n                \n                  2\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\max _{t\\in [0,1]}{\\frac {\\|\\nabla f(\\mathbf {a} _{n}-t\\eta _{n}\\mathbf {p} _{n})-\\nabla f(\\mathbf {a} _{n})\\|_{2}}{\\|\\nabla f(\\mathbf {a} _{n})\\|_{2}}}}\n  \n for \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n. Then choose \n  \n    \n      \n        \n          \n            p\n          \n          \n            n", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0061", "text": "p\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {p} _{n}}\n  \n and \n  \n    \n      \n        \n          \u03b7\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\eta _{n}}\n  \n by optimising inequality (1).\nUnder stronger assumptions on the function \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n such as convexity, more advanced techniques may be possible.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0062", "text": "Usually by following one of the recipes above, convergence to a local minimum can be guaranteed. When the function \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is convex, all local minima are also global minima, so in this case gradient descent can converge to the global solution.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0063", "text": "Solution of a linear system\nGradient descent can be used to solve a system of linear equations\n\n  \n    \n      \n        \n          A\n        \n        \n          x\n        \n        \u2212\n        \n          b\n        \n        =\n        0\n      \n    \n    {\\displaystyle \\mathbf {A} \\mathbf {x} -\\mathbf {b} =0}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0064", "text": "reformulated as a quadratic minimization problem.\nIf the system matrix \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }\n  \n is real symmetric and positive-definite, an objective function is defined as the quadratic function, with minimization of", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0065", "text": "f\n        (\n        \n          x\n        \n        )\n        =\n        \n          \n            x\n          \n          \n            \u22a4\n          \n        \n        \n          A\n        \n        \n          x\n        \n        \u2212\n        2\n        \n          \n            x\n          \n          \n            \u22a4\n          \n        \n        \n          b\n        \n        ,", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0066", "text": "b\n        \n        ,\n      \n    \n    {\\displaystyle f(\\mathbf {x} )=\\mathbf {x} ^{\\top }\\mathbf {A} \\mathbf {x} -2\\mathbf {x} ^{\\top }\\mathbf {b} ,}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0067", "text": "so that\n\n  \n    \n      \n        \u2207\n        f\n        (\n        \n          x\n        \n        )\n        =\n        2\n        (\n        \n          A\n        \n        \n          x\n        \n        \u2212\n        \n          b\n        \n        )\n        .\n      \n    \n    {\\displaystyle \\nabla f(\\mathbf {x} )=2(\\mathbf {A} \\mathbf {x} -\\mathbf {b} ).}\n  \n\nFor a general real matrix \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }\n  \n, linear least squares define", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0068", "text": "f\n        (\n        \n          x\n        \n        )\n        =\n        \n          \n            \u2016\n            \n              \n                A\n              \n              \n                x\n              \n              \u2212\n              \n                b\n              \n            \n            \u2016\n          \n          \n            2\n          \n        \n        .\n      \n    \n    {\\displaystyle f(\\mathbf {x} )=\\left\\|\\mathbf {A} \\mathbf {x} -\\mathbf {b} \\right\\|^{2}.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0069", "text": "In traditional linear least squares for real \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }\n  \n and \n  \n    \n      \n        \n          b\n        \n      \n    \n    {\\displaystyle \\mathbf {b} }\n  \n the Euclidean norm is used, in which case", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0070", "text": "\u2207\n        f\n        (\n        \n          x\n        \n        )\n        =\n        2\n        \n          \n            A\n          \n          \n            \u22a4\n          \n        \n        (\n        \n          A\n        \n        \n          x\n        \n        \u2212\n        \n          b\n        \n        )\n        .\n      \n    \n    {\\displaystyle \\nabla f(\\mathbf {x} )=2\\mathbf {A} ^{\\top }(\\mathbf {A} \\mathbf {x} -\\mathbf {b} ).}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0071", "text": "The line search minimization, finding the locally optimal step size \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n on every iteration, can be performed analytically for quadratic functions, and explicit formulas for the locally optimal \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n are known.\nFor example, for real symmetric and positive-definite matrix \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0072", "text": "{\\displaystyle \\mathbf {A} }\n  \n, a simple algorithm can be as follows,", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0073", "text": "repeat in the loop:\n                \n              \n            \n            \n              \n              \n                \n                \n                  r\n                \n                :=\n                \n                  b\n                \n                \u2212\n                \n                  A\n                  x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0074", "text": "\u03b7\n                :=\n                \n                  \n                    \n                      r\n                    \n                    \n                      \u22a4\n                    \n                  \n                  \n                    r\n                  \n                \n                \n                  /", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0075", "text": "r\n                    \n                    \n                      \u22a4\n                    \n                  \n                  \n                    A\n                    r\n                  \n                \n              \n            \n            \n              \n              \n                \n                \n                  x\n                \n                :=\n                \n                  x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0076", "text": "x\n                \n                +\n                \u03b7\n                \n                  r\n                \n              \n            \n            \n              \n              \n                \n                \n                  \n                    if \n                  \n                \n                \n                  \n                    r\n                  \n                  \n                    \u22a4", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0077", "text": "r\n                \n                \n                   is sufficiently small, then exit loop\n                \n              \n            \n            \n              \n              \n                \n                  end repeat loop\n                \n              \n            \n            \n              \n              \n                \n                  return \n                \n                \n                  x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0078", "text": "x\n                \n                \n                   as the result", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0079", "text": "{\\displaystyle {\\begin{aligned}&{\\text{repeat in the loop:}}\\\\&\\qquad \\mathbf {r} :=\\mathbf {b} -\\mathbf {Ax} \\\\&\\qquad \\eta :={\\mathbf {r} ^{\\top }\\mathbf {r} }/{\\mathbf {r} ^{\\top }\\mathbf {Ar} }\\\\&\\qquad \\mathbf {x} :=\\mathbf {x} +\\eta \\mathbf {r} \\\\&\\qquad {\\hbox{if }}\\mathbf {r} ^{\\top }\\mathbf {r} {\\text{ is sufficiently small, then exit loop}}\\\\&{\\text{end repeat loop}}\\\\&{\\text{return }}\\mathbf {x} {\\text{ as the result}}\\end{aligned}}}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0080", "text": "To avoid multiplying by \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }\n  \n twice per iteration,\nwe note that \n  \n    \n      \n        \n          x\n        \n        :=\n        \n          x\n        \n        +\n        \u03b7\n        \n          r\n        \n      \n    \n    {\\displaystyle \\mathbf {x} :=\\mathbf {x} +\\eta \\mathbf {r} }\n  \n implies \n  \n    \n      \n        \n          r\n        \n        :=\n        \n          r\n        \n        \u2212\n        \u03b7", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0081", "text": "r\n        \n        \u2212\n        \u03b7\n        \n          A\n          r\n        \n      \n    \n    {\\displaystyle \\mathbf {r} :=\\mathbf {r} -\\eta \\mathbf {Ar} }\n  \n, which gives the traditional algorithm,", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0082", "text": "r\n                \n                :=\n                \n                  b\n                \n                \u2212\n                \n                  A\n                  x\n                \n              \n            \n            \n              \n              \n                \n                  repeat in the loop:", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0083", "text": "\u03b7\n                :=\n                \n                  \n                    \n                      r\n                    \n                    \n                      \u22a4\n                    \n                  \n                  \n                    r\n                  \n                \n                \n                  /", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0084", "text": "r\n                    \n                    \n                      \u22a4\n                    \n                  \n                  \n                    A\n                    r\n                  \n                \n              \n            \n            \n              \n              \n                \n                \n                  x\n                \n                :=\n                \n                  x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0085", "text": "x\n                \n                +\n                \u03b7\n                \n                  r\n                \n              \n            \n            \n              \n              \n                \n                \n                  \n                    if \n                  \n                \n                \n                  \n                    r\n                  \n                  \n                    \u22a4", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0086", "text": "r\n                \n                \n                   is sufficiently small, then exit loop\n                \n              \n            \n            \n              \n              \n                \n                \n                  r\n                \n                :=\n                \n                  r\n                \n                \u2212\n                \u03b7\n                \n                  A\n                  r", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0087", "text": "r\n                \n              \n            \n            \n              \n              \n                \n                  end repeat loop\n                \n              \n            \n            \n              \n              \n                \n                  return \n                \n                \n                  x\n                \n                \n                   as the result", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0088", "text": "{\\displaystyle {\\begin{aligned}&\\mathbf {r} :=\\mathbf {b} -\\mathbf {Ax} \\\\&{\\text{repeat in the loop:}}\\\\&\\qquad \\eta :={\\mathbf {r} ^{\\top }\\mathbf {r} }/{\\mathbf {r} ^{\\top }\\mathbf {Ar} }\\\\&\\qquad \\mathbf {x} :=\\mathbf {x} +\\eta \\mathbf {r} \\\\&\\qquad {\\hbox{if }}\\mathbf {r} ^{\\top }\\mathbf {r} {\\text{ is sufficiently small, then exit loop}}\\\\&\\qquad \\mathbf {r} :=\\mathbf {r} -\\eta \\mathbf {Ar} \\\\&{\\text{end repeat loop}}\\\\&{\\text{return }}\\mathbf {x} {\\text{ as the", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0089", "text": "}}\\mathbf {x} {\\text{ as the result}}\\end{aligned}}}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0090", "text": "The method is rarely used for solving linear equations, with the conjugate gradient method being one of the most popular alternatives. The number of gradient descent iterations is commonly proportional to the spectral condition number \n  \n    \n      \n        \u03ba\n        (\n        \n          A\n        \n        )\n      \n    \n    {\\displaystyle \\kappa (\\mathbf {A} )}\n  \n of the system matrix \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0091", "text": "{\\displaystyle \\mathbf {A} }\n  \n (the ratio of the maximum to minimum eigenvalues of \n  \n    \n      \n        \n          \n            A\n          \n          \n            \u22a4\n          \n        \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} ^{\\top }\\mathbf {A} }", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0092", "text": "), while the convergence of conjugate gradient method is typically determined by a square root of the condition number, i.e., is much faster. Both methods can benefit from preconditioning, where gradient descent may require less assumptions on the preconditioner.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0093", "text": "Geometric behavior and residual orthogonality\nIn steepest descent applied to solving \n  \n    \n      \n        \n          A\n          x\n        \n        =\n        \n          b\n        \n      \n    \n    {\\displaystyle \\mathbf {Ax} =\\mathbf {b} }\n  \n, where \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }\n  \n is symmetric positive-definite, the residual vectors \n  \n    \n      \n        \n          \n            r\n          \n          \n            k", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0094", "text": "k\n          \n        \n        =\n        \n          b\n        \n        \u2212\n        \n          A\n        \n        \n          \n            x\n          \n          \n            k\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {r} _{k}=\\mathbf {b} -\\mathbf {A} \\mathbf {x} _{k}}\n  \n are orthogonal across iterations:", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0095", "text": "\u27e8\n        \n          \n            r\n          \n          \n            k\n            +\n            1\n          \n        \n        ,\n        \n          \n            r\n          \n          \n            k\n          \n        \n        \u27e9\n        =\n        0.\n      \n    \n    {\\displaystyle \\langle \\mathbf {r} _{k+1},\\mathbf {r} _{k}\\rangle =0.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0096", "text": "Because each step is taken in the steepest direction, steepest-descent steps alternate between directions aligned with the extreme axes of the elongated level sets.  When \n  \n    \n      \n        \u03ba\n        (\n        \n          A\n        \n        )\n      \n    \n    {\\displaystyle \\kappa (\\mathbf {A} )}\n  \n is large, this produces a characteristic zig\u2013zag path. The poor conditioning of \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0097", "text": "{\\displaystyle \\mathbf {A} }\n  \n is the primary cause of the slow convergence, and orthogonality of successive residuals reinforces this alternation.\nAs shown in the image on the right, steepest descent converges slowly due to the high condition number of \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0098", "text": "{\\displaystyle \\mathbf {A} }\n  \n, and the orthogonality of residuals forces each new direction to undo the overshoot from the previous step. The result is a path that zigzags toward the solution. This inefficiency is one reason conjugate gradient or preconditioning methods are preferred.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0099", "text": "Solution of a non-linear system\nGradient descent can also be used to solve a system of nonlinear equations. Below is an example that shows how to use the gradient descent to solve for three unknown variables, x1, x2, and x3. This example shows one iteration of the gradient descent.\nConsider the nonlinear system of equations", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0100", "text": "{\n            \n              \n                \n                  3\n                  \n                    x\n                    \n                      1\n                    \n                  \n                  \u2212\n                  cos\n                  \u2061\n                  (\n                  \n                    x\n                    \n                      2\n                    \n                  \n                  \n                    x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0101", "text": "x\n                    \n                      3\n                    \n                  \n                  )\n                  \u2212\n                  \n                    \n                      \n                        3\n                        2\n                      \n                    \n                  \n                  =\n                  0\n                \n              \n              \n                \n                  4", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0102", "text": "4\n                  \n                    x\n                    \n                      1\n                    \n                    \n                      2\n                    \n                  \n                  \u2212\n                  625\n                  \n                    x\n                    \n                      2\n                    \n                    \n                      2\n                    \n                  \n                  +\n                  2", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0103", "text": "+\n                  2\n                  \n                    x\n                    \n                      2\n                    \n                  \n                  \u2212\n                  1\n                  =\n                  0\n                \n              \n              \n                \n                  exp\n                  \u2061\n                  (\n                  \u2212\n                  \n                    x\n                    \n                      1", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0104", "text": "1\n                    \n                  \n                  \n                    x\n                    \n                      2\n                    \n                  \n                  )\n                  +\n                  20\n                  \n                    x\n                    \n                      3\n                    \n                  \n                  +", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0105", "text": "10\n                          \u03c0\n                          \u2212\n                          3\n                        \n                        3\n                      \n                    \n                  \n                  =\n                  0", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0106", "text": "{\\displaystyle {\\begin{cases}3x_{1}-\\cos(x_{2}x_{3})-{\\tfrac {3}{2}}=0\\\\4x_{1}^{2}-625x_{2}^{2}+2x_{2}-1=0\\\\\\exp(-x_{1}x_{2})+20x_{3}+{\\tfrac {10\\pi -3}{3}}=0\\end{cases}}}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0107", "text": "Let us introduce the associated function", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0108", "text": "G\n        (\n        \n          x\n        \n        )\n        =\n        \n          \n            [\n            \n              \n                \n                  3\n                  \n                    x\n                    \n                      1\n                    \n                  \n                  \u2212\n                  cos\n                  \u2061\n                  (\n                  \n                    x\n                    \n                      2", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0109", "text": "2\n                    \n                  \n                  \n                    x\n                    \n                      3\n                    \n                  \n                  )\n                  \u2212\n                  \n                    \n                      \n                        3\n                        2\n                      \n                    \n                  \n                \n              \n              \n                \n                  4", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0110", "text": "4\n                  \n                    x\n                    \n                      1\n                    \n                    \n                      2\n                    \n                  \n                  \u2212\n                  625\n                  \n                    x\n                    \n                      2\n                    \n                    \n                      2\n                    \n                  \n                  +", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0111", "text": "+\n                  2\n                  \n                    x\n                    \n                      2\n                    \n                  \n                  \u2212\n                  1\n                \n              \n              \n                \n                  exp\n                  \u2061\n                  (\n                  \u2212\n                  \n                    x\n                    \n                      1", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0112", "text": "x\n                    \n                      2\n                    \n                  \n                  )\n                  +\n                  20\n                  \n                    x\n                    \n                      3\n                    \n                  \n                  +\n                  \n                    \n                      \n                        \n                          10", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0113", "text": "10\n                          \u03c0\n                          \u2212\n                          3\n                        \n                        3\n                      \n                    \n                  \n                \n              \n            \n            ]\n          \n        \n        ,", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0114", "text": ",\n      \n    \n    {\\displaystyle G(\\mathbf {x} )={\\begin{bmatrix}3x_{1}-\\cos(x_{2}x_{3})-{\\tfrac {3}{2}}\\\\4x_{1}^{2}-625x_{2}^{2}+2x_{2}-1\\\\\\exp(-x_{1}x_{2})+20x_{3}+{\\tfrac {10\\pi -3}{3}}\\\\\\end{bmatrix}},}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0115", "text": "where", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0116", "text": "x\n        \n        =\n        \n          \n            [\n            \n              \n                \n                  \n                    x\n                    \n                      1\n                    \n                  \n                \n              \n              \n                \n                  \n                    x\n                    \n                      2", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0117", "text": "x\n                    \n                      3\n                    \n                  \n                \n              \n            \n            ]\n          \n        \n        .\n      \n    \n    {\\displaystyle \\mathbf {x} ={\\begin{bmatrix}x_{1}\\\\x_{2}\\\\x_{3}\\\\\\end{bmatrix}}.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0118", "text": "One might now define the objective function", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0119", "text": "f\n                (\n                \n                  x\n                \n                )\n              \n              \n                \n                =\n                \n                  \n                    1\n                    2\n                  \n                \n                \n                  G\n                  \n                    \u22a4\n                  \n                \n                (", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0120", "text": "(\n                \n                  x\n                \n                )\n                G\n                (\n                \n                  x\n                \n                )\n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    1\n                    2\n                  \n                \n                \n                  [", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0121", "text": "[\n                  \n                    \n                      \n                        (\n                        \n                          3\n                          \n                            x\n                            \n                              1\n                            \n                          \n                          \u2212\n                          cos\n                          \u2061\n                          (", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0122", "text": "x\n                            \n                              2\n                            \n                          \n                          \n                            x\n                            \n                              3\n                            \n                          \n                          )\n                          \u2212\n                          \n                            \n                              3", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0123", "text": "3\n                              2\n                            \n                          \n                        \n                        )\n                      \n                      \n                        2\n                      \n                    \n                    +\n                    \n                      \n                        (\n                        \n                          4\n                          \n                            x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0124", "text": "x\n                            \n                              1\n                            \n                            \n                              2\n                            \n                          \n                          \u2212\n                          625\n                          \n                            x\n                            \n                              2", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0125", "text": "2\n                            \n                          \n                          +\n                          2\n                          \n                            x\n                            \n                              2\n                            \n                          \n                          \u2212\n                          1\n                        \n                        )", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0126", "text": "2\n                      \n                    \n                    +", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0127", "text": "(\n                      \n                        exp\n                        \u2061\n                        (\n                        \u2212\n                        \n                          x\n                          \n                            1\n                          \n                        \n                        \n                          x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0128", "text": "x\n                          \n                            2\n                          \n                        \n                        )\n                        +\n                        20\n                        \n                          x\n                          \n                            3\n                          \n                        \n                        +", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0129", "text": "10\n                              \u03c0\n                              \u2212\n                              3\n                            \n                            3\n                          \n                        \n                      \n                      )\n                    \n                    \n                      2\n                    \n                  \n                  ]\n                \n                ,", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0130", "text": ",\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}f(\\mathbf {x} )&={\\frac {1}{2}}G^{\\top }(\\mathbf {x} )G(\\mathbf {x} )\\\\&={\\frac {1}{2}}\\left[\\left(3x_{1}-\\cos(x_{2}x_{3})-{\\frac {3}{2}}\\right)^{2}+\\left(4x_{1}^{2}-625x_{2}^{2}+2x_{2}-1\\right)^{2}+\\right.\\\\&{}\\qquad \\left.\\left(\\exp(-x_{1}x_{2})+20x_{3}+{\\frac {10\\pi -3}{3}}\\right)^{2}\\right],\\end{aligned}}}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0131", "text": "which we will attempt to minimize. As an initial guess, let us use", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0132", "text": "x\n          \n          \n            (\n            0\n            )\n          \n        \n        =\n        \n          0\n        \n        =\n        \n          \n            [\n            \n              \n                \n                  0\n                \n              \n              \n                \n                  0\n                \n              \n              \n                \n                  0", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0133", "text": "]\n          \n        \n        .\n      \n    \n    {\\displaystyle \\mathbf {x} ^{(0)}=\\mathbf {0} ={\\begin{bmatrix}0\\\\0\\\\0\\\\\\end{bmatrix}}.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0134", "text": "We know that", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0135", "text": "x\n          \n          \n            (\n            1\n            )\n          \n        \n        =\n        \n          0\n        \n        \u2212\n        \n          \u03b7\n          \n            0\n          \n        \n        \u2207\n        f\n        (\n        \n          0\n        \n        )\n        =\n        \n          0\n        \n        \u2212\n        \n          \u03b7\n          \n            0\n          \n        \n        \n          J\n          \n            G", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0136", "text": "J\n          \n            G\n          \n        \n        (\n        \n          0\n        \n        \n          )\n          \n            \u22a4\n          \n        \n        G\n        (\n        \n          0\n        \n        )\n        ,\n      \n    \n    {\\displaystyle \\mathbf {x} ^{(1)}=\\mathbf {0} -\\eta _{0}\\nabla f(\\mathbf {0} )=\\mathbf {0} -\\eta _{0}J_{G}(\\mathbf {0} )^{\\top }G(\\mathbf {0} ),}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0137", "text": "where the Jacobian matrix \n  \n    \n      \n        \n          J\n          \n            G\n          \n        \n      \n    \n    {\\displaystyle J_{G}}\n  \n is given by", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0138", "text": "J\n          \n            G\n          \n        \n        (\n        \n          x\n        \n        )\n        =\n        \n          \n            [\n            \n              \n                \n                  3\n                \n                \n                  sin\n                  \u2061\n                  (\n                  \n                    x\n                    \n                      2", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0139", "text": "x\n                    \n                      3\n                    \n                  \n                  )\n                  \n                    x\n                    \n                      3\n                    \n                  \n                \n                \n                  sin\n                  \u2061\n                  (\n                  \n                    x\n                    \n                      2", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0140", "text": "2\n                    \n                  \n                  \n                    x\n                    \n                      3\n                    \n                  \n                  )\n                  \n                    x\n                    \n                      2\n                    \n                  \n                \n              \n              \n                \n                  8\n                  \n                    x", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0141", "text": "x\n                    \n                      1\n                    \n                  \n                \n                \n                  \u2212\n                  1250\n                  \n                    x\n                    \n                      2\n                    \n                  \n                  +\n                  2\n                \n                \n                  0\n                \n              \n              \n                \n                  \u2212", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0142", "text": "\u2212\n                  \n                    x\n                    \n                      2\n                    \n                  \n                  exp\n                  \u2061\n                  \n                    (\n                    \u2212\n                    \n                      x\n                      \n                        1\n                      \n                    \n                    \n                      x\n                      \n                        2", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0143", "text": "2\n                      \n                    \n                    )\n                  \n                \n                \n                  \u2212\n                  \n                    x\n                    \n                      1\n                    \n                  \n                  exp\n                  \u2061\n                  (\n                  \u2212\n                  \n                    x\n                    \n                      1", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0144", "text": "1\n                    \n                  \n                  \n                    x\n                    \n                      2\n                    \n                  \n                  )\n                \n                \n                  20\n                \n              \n            \n            ]\n          \n        \n        .", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0145", "text": ".\n      \n    \n    {\\displaystyle J_{G}(\\mathbf {x} )={\\begin{bmatrix}3&\\sin(x_{2}x_{3})x_{3}&\\sin(x_{2}x_{3})x_{2}\\\\8x_{1}&-1250x_{2}+2&0\\\\-x_{2}\\exp {(-x_{1}x_{2})}&-x_{1}\\exp(-x_{1}x_{2})&20\\\\\\end{bmatrix}}.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0146", "text": "We calculate:", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0147", "text": "J\n          \n            G\n          \n        \n        (\n        \n          0\n        \n        )\n        =\n        \n          \n            [\n            \n              \n                \n                  3\n                \n                \n                  0\n                \n                \n                  0\n                \n              \n              \n                \n                  0\n                \n                \n                  2", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0148", "text": "2\n                \n                \n                  0\n                \n              \n              \n                \n                  0\n                \n                \n                  0\n                \n                \n                  20\n                \n              \n            \n            ]\n          \n        \n        ,\n        \n        G\n        (\n        \n          0\n        \n        )\n        =\n        \n          \n            [", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0149", "text": "[\n            \n              \n                \n                  \u2212\n                  2.5\n                \n              \n              \n                \n                  \u2212\n                  1\n                \n              \n              \n                \n                  10.472\n                \n              \n            \n            ]\n          \n        \n        .", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0150", "text": ".\n      \n    \n    {\\displaystyle J_{G}(\\mathbf {0} )={\\begin{bmatrix}3&0&0\\\\0&2&0\\\\0&0&20\\end{bmatrix}},\\qquad G(\\mathbf {0} )={\\begin{bmatrix}-2.5\\\\-1\\\\10.472\\end{bmatrix}}.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0151", "text": "Thus", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0152", "text": "x\n          \n          \n            (\n            1\n            )\n          \n        \n        =\n        \n          0\n        \n        \u2212\n        \n          \u03b7\n          \n            0\n          \n        \n        \n          \n            [\n            \n              \n                \n                  \u2212\n                  7.5\n                \n              \n              \n                \n                  \u2212\n                  2", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0153", "text": "2\n                \n              \n              \n                \n                  209.44\n                \n              \n            \n            ]\n          \n        \n        ,\n      \n    \n    {\\displaystyle \\mathbf {x} ^{(1)}=\\mathbf {0} -\\eta _{0}{\\begin{bmatrix}-7.5\\\\-2\\\\209.44\\end{bmatrix}},}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0154", "text": "and", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0155", "text": "f\n        (\n        \n          0\n        \n        )\n        =\n        0.5\n        \n          (\n          \n            (\n            \u2212\n            2.5\n            \n              )\n              \n                2\n              \n            \n            +\n            (\n            \u2212\n            1\n            \n              )\n              \n                2\n              \n            \n            +\n            (\n            10.472\n            \n              )", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0156", "text": "10.472\n            \n              )\n              \n                2\n              \n            \n          \n          )\n        \n        =\n        58.456.\n      \n    \n    {\\displaystyle f(\\mathbf {0} )=0.5\\left((-2.5)^{2}+(-1)^{2}+(10.472)^{2}\\right)=58.456.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0157", "text": "Now, a suitable \n  \n    \n      \n        \n          \u03b7\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\eta _{0}}\n  \n must be found such that", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0158", "text": "f\n        \n          (\n          \n            \n              x\n            \n            \n              (\n              1\n              )\n            \n          \n          )\n        \n        \u2264\n        f\n        \n          (\n          \n            \n              x\n            \n            \n              (\n              0\n              )\n            \n          \n          )\n        \n        =\n        f\n        (\n        \n          0\n        \n        )\n        .", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0159", "text": ")\n        .\n      \n    \n    {\\displaystyle f\\left(\\mathbf {x} ^{(1)}\\right)\\leq f\\left(\\mathbf {x} ^{(0)}\\right)=f(\\mathbf {0} ).}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0160", "text": "This can be done with any of a variety of line search algorithms. One might also simply guess \n  \n    \n      \n        \n          \u03b7\n          \n            0\n          \n        \n        =\n        0.001\n        ,\n      \n    \n    {\\displaystyle \\eta _{0}=0.001,}\n  \n which gives", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0161", "text": "x\n          \n          \n            (\n            1\n            )\n          \n        \n        =\n        \n          \n            [\n            \n              \n                \n                  0.0075\n                \n              \n              \n                \n                  0.002\n                \n              \n              \n                \n                  \u2212\n                  0.20944", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0162", "text": "]\n          \n        \n        .\n      \n    \n    {\\displaystyle \\mathbf {x} ^{(1)}={\\begin{bmatrix}0.0075\\\\0.002\\\\-0.20944\\\\\\end{bmatrix}}.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0163", "text": "Evaluating the objective function at this value, yields", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0164", "text": "f\n        \n          (\n          \n            \n              x\n            \n            \n              (\n              1\n              )\n            \n          \n          )\n        \n        =\n        0.5\n        \n          (\n          \n            (\n            \u2212\n            2.48\n            \n              )\n              \n                2\n              \n            \n            +\n            (\n            \u2212\n            1.00\n            \n              )", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0165", "text": ")\n              \n                2\n              \n            \n            +\n            (\n            6.28\n            \n              )\n              \n                2\n              \n            \n          \n          )\n        \n        =\n        23.306.\n      \n    \n    {\\displaystyle f\\left(\\mathbf {x} ^{(1)}\\right)=0.5\\left((-2.48)^{2}+(-1.00)^{2}+(6.28)^{2}\\right)=23.306.}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0166", "text": "The decrease from \n  \n    \n      \n        f\n        (\n        \n          0\n        \n        )\n        =\n        58.456\n      \n    \n    {\\displaystyle f(\\mathbf {0} )=58.456}\n  \n to the next step's value of", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0167", "text": "f\n        \n          (\n          \n            \n              x\n            \n            \n              (\n              1\n              )\n            \n          \n          )\n        \n        =\n        23.306\n      \n    \n    {\\displaystyle f\\left(\\mathbf {x} ^{(1)}\\right)=23.306}\n  \n\nis a sizable decrease in the objective function. Further steps would reduce its value further until an approximate solution to the system was found.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0168", "text": "Comments\nGradient descent works in spaces of any number of dimensions, even in infinite-dimensional ones. In the latter case, the search space is typically a function space, and one calculates the Fr\u00e9chet derivative of the functional to be minimized to determine the descent direction.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0169", "text": "That gradient descent works in any number of dimensions (finite number at least) can be seen as a consequence of the Cauchy\u2013Schwarz inequality, i.e. the magnitude of the inner (dot) product of two vectors of any dimension is maximized when they are colinear. In the case of gradient descent, that would be when the vector of independent variable adjustments is proportional to the gradient vector of partial derivatives.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0170", "text": "The gradient descent can take many iterations to compute a local minimum with a required accuracy, if the curvature in different directions is very different for the given function. For such functions, preconditioning, which changes the geometry of the space to shape the function level sets like concentric circles, cures the slow convergence. Constructing and applying preconditioning can be computationally expensive, however.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0171", "text": "The gradient descent can be modified via momentums (Nesterov, Polyak, and Frank\u2013Wolfe) and heavy-ball parameters (exponential moving averages and positive-negative momentum). The main examples of such optimizers are Adam, DiffGrad, Yogi, AdaBelief, etc.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0172", "text": "Methods based on Newton's method and inversion of the Hessian using conjugate gradient techniques can be better alternatives. Generally, such methods converge in fewer iterations, but the cost of each iteration is higher. An example is the BFGS method which consists in calculating on every step a matrix by which the gradient vector is multiplied to go into a \"better\" direction, combined with a more sophisticated line search algorithm, to find the \"best\" value of \n  \n    \n      \n        \u03b7", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0173", "text": "\u03b7\n        .\n      \n    \n    {\\displaystyle \\eta .}\n  \n For extremely large problems, where the computer-memory issues dominate, a limited-memory method such as L-BFGS should be used instead of BFGS or the steepest descent.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0174", "text": "While it is sometimes possible to substitute gradient descent for a local search algorithm, gradient descent is not in the same family: although it is an iterative method for local optimization, it relies on an objective function's gradient rather than an explicit exploration of a solution space.\nGradient descent can be viewed as applying Euler's method for solving ordinary differential equations \n  \n    \n      \n        \n          x\n          \u2032\n        \n        (\n        t\n        )\n        =", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0175", "text": "(\n        t\n        )\n        =\n        \u2212\n        \u2207\n        f\n        (\n        x\n        (\n        t\n        )\n        )\n      \n    \n    {\\displaystyle x'(t)=-\\nabla f(x(t))}\n  \n to a gradient flow.  In turn, this equation may be derived as an optimal controller for the control system \n  \n    \n      \n        \n          x\n          \u2032\n        \n        (\n        t\n        )\n        =\n        u\n        (\n        t\n        )\n      \n    \n    {\\displaystyle x'(t)=u(t)}\n  \n with", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0176", "text": "{\\displaystyle x'(t)=u(t)}\n  \n with \n  \n    \n      \n        u\n        (\n        t\n        )\n      \n    \n    {\\displaystyle u(t)}\n  \n given in feedback form \n  \n    \n      \n        u\n        (\n        t\n        )\n        =\n        \u2212\n        \u2207\n        f\n        (\n        x\n        (\n        t\n        )\n        )\n      \n    \n    {\\displaystyle u(t)=-\\nabla f(x(t))}\n  \n.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0177", "text": "Modifications\nGradient descent can converge to a local minimum and slow down in a neighborhood of a saddle point. Even for unconstrained quadratic minimization, gradient descent develops a zig\u2013zag pattern of subsequent iterates as iterations progress, resulting in slow convergence. Multiple modifications of gradient descent have been proposed to address these deficiencies.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0178", "text": "Fast gradient methods\nYurii Nesterov has proposed a simple modification that enables faster convergence for convex problems and has been since further generalized. For unconstrained smooth problems, the method is called the fast gradient method (FGM) or the accelerated gradient method (AGM). Specifically, if the differentiable function \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is convex and \n  \n    \n      \n        \u2207\n        f\n      \n    \n    {\\displaystyle \\nabla f}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0179", "text": "{\\displaystyle \\nabla f}\n  \n is Lipschitz, and it is not assumed that \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is strongly convex, then the error in the objective value generated at each step \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n by the gradient descent method will be bounded by \n  \n    \n      \n        \n          \n            O\n          \n        \n        \n          (\n          \n            \n              k", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0180", "text": "k\n              \n                \u2212\n                1\n              \n            \n          \n          )\n        \n      \n    \n    {\\textstyle {\\mathcal {O}}\\left({k^{-1}}\\right)}\n  \n. Using the Nesterov acceleration technique, the error decreases at \n  \n    \n      \n        \n          \n            O\n          \n        \n        \n          (\n          \n            \n              k\n              \n                \u2212\n                2", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0181", "text": ")\n        \n      \n    \n    {\\textstyle {\\mathcal {O}}\\left({k^{-2}}\\right)}\n  \n. It is known that the rate \n  \n    \n      \n        \n          \n            O\n          \n        \n        \n          (\n          \n            \n              k\n              \n                \u2212\n                2\n              \n            \n          \n          )\n        \n      \n    \n    {\\displaystyle {\\mathcal {O}}\\left({k^{-2}}\\right)}", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0182", "text": "for the decrease of the cost function is optimal for first-order optimization methods. Nevertheless, there is the opportunity to improve the algorithm by reducing the constant factor. The optimized gradient method (OGM) reduces that constant by a factor of two and is an optimal first-order method for large-scale problems.\nFor constrained or non-smooth problems, Nesterov's FGM is called the fast proximal gradient method (FPGM), an acceleration of the proximal gradient method.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0183", "text": "Momentum or heavy ball method", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0184", "text": "Trying to break the zig-zag pattern of gradient descent, the momentum or heavy ball method uses a momentum term in analogy to a heavy ball sliding on the surface of values of the function being minimized, or to mass movement in Newtonian dynamics through a viscous medium in a conservative force field. Gradient descent with momentum remembers the solution update at each iteration, and determines the next update as a linear combination of the gradient and the previous update", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0185", "text": ". For unconstrained quadratic minimization, a theoretical convergence rate bound of the heavy ball method is asymptotically the same as that for the optimal conjugate gradient method.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0186", "text": "This technique is used in stochastic gradient descent and as an extension to the backpropagation algorithms used to train artificial neural networks. In the direction of updating, stochastic gradient descent adds a stochastic property. The weights can be used to calculate the derivatives.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0187", "text": "Extensions\nGradient descent can be extended to handle constraints by including a projection onto the set of constraints. This method is only feasible when the projection is efficiently computable on a computer. Under suitable assumptions, this method converges.  This method is a specific case of the forward\u2013backward algorithm for monotone inclusions (which includes convex programming and variational inequalities).", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0188", "text": "Gradient descent is a special case of mirror descent using the squared Euclidean distance as the given Bregman divergence.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0189", "text": "Theoretical properties", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0190", "text": "The properties of gradient descent depend on the properties of the objective function and the variant of gradient descent used (for example, if a line search step is used). The assumptions made affect the convergence rate, and other properties, that can be proven for gradient descent. For example, if the objective is assumed to be strongly convex and lipschitz smooth, then gradient descent converges linearly with a fixed step size", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0191", "text": ". Looser assumptions lead to either weaker convergence guarantees or require a more sophisticated step size selection.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0192", "text": "Examples\nYang\u2013Mills flow\nYang\u2013Mills\u2013Higgs flow\nSeiberg\u2013Witten flow", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0193", "text": "See also\nReferences\nFurther reading\nBoyd, Stephen; Vandenberghe, Lieven (2004). \"Unconstrained Minimization\" (PDF). Convex Optimization. New York: Cambridge University Press. pp. 457\u2013520. ISBN 0-521-83378-7.\nChong, Edwin K. P.; \u017bak, Stanislaw H. (2013). \"Gradient Methods\". An Introduction to Optimization (Fourth ed.). Hoboken: Wiley. pp. 131\u2013160. ISBN 978-1-118-27901-4.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0194", "text": "Himmelblau, David M. (1972). \"Unconstrained Minimization Procedures Using Derivatives\". Applied Nonlinear Programming. New York: McGraw-Hill. pp. 63\u2013132. ISBN 0-07-028921-2.", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Gradient_descent_0195", "text": "External links\n\nUsing gradient descent in C++, Boost, Ublas for linear regression\nSeries of Khan Academy videos discusses gradient ascent\nOnline book teaching gradient descent in deep neural network context\nArchived at Ghostarchive and the Wayback Machine: \"Gradient Descent, How Neural Networks Learn\". 3Blue1Brown. October 16, 2017 \u2013 via YouTube.\nGarrigos, Guillaume; Gower, Robert M. (2023). \"Handbook of Convergence Theorems for (Stochastic) Gradient Methods\". arXiv:2301.11235 [math.OC].", "title": "Gradient descent", "url": "https://en.wikipedia.org/wiki/Gradient_descent"}, {"id": "rec_Backpropagation_0000", "text": "In machine learning, backpropagation is a gradient computation method commonly used for training a neural network in computing parameter updates.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0001", "text": "It is an efficient application of the chain rule to neural networks. Backpropagation computes the gradient of a loss function with respect to the weights of the network for a single input\u2013output example, and does so efficiently, computing the gradient one layer at a time, iterating backward from the last layer to avoid redundant calculations of intermediate terms in the chain rule; this can be derived through dynamic programming.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0002", "text": "Strictly speaking, the term backpropagation refers only to an algorithm for efficiently computing the gradient, not how the gradient is used; but the term is often used loosely to refer to the entire learning algorithm. This includes changing model parameters in the negative direction of the gradient, such as by stochastic gradient descent, or as an intermediate step in a more complicated optimizer, such as Adaptive Moment Estimation.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0003", "text": "Backpropagation had multiple discoveries and partial discoveries, with a tangled history and terminology. See the history section for details. Some other names for the technique include \"reverse mode of automatic differentiation\" or \"reverse accumulation\".", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0004", "text": "Overview\nBackpropagation computes the gradient in weight space of a feedforward neural network, with respect to a loss function. Denote:\n\n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n: input (vector of features)", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0005", "text": "y\n      \n    \n    {\\displaystyle y}\n  \n: target output\nFor classification, output will be a vector of class probabilities (e.g., \n  \n    \n      \n        (\n        0.1\n        ,\n        0.7\n        ,\n        0.2\n        )\n      \n    \n    {\\displaystyle (0.1,0.7,0.2)}\n  \n, and target output is a specific class, encoded by the one-hot/dummy variable (e.g., \n  \n    \n      \n        (\n        0\n        ,\n        1\n        ,\n        0\n        )", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0006", "text": ",\n        0\n        )\n      \n    \n    {\\displaystyle (0,1,0)}\n  \n).", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0007", "text": "C\n      \n    \n    {\\displaystyle C}\n  \n: loss function or \"cost function\"\nFor classification, this is usually cross-entropy (XC, log loss), while for regression it is usually squared error loss (SEL).\n\n  \n    \n      \n        L\n      \n    \n    {\\displaystyle L}\n  \n: the number of layers", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0008", "text": "W\n          \n            l\n          \n        \n        =\n        (\n        \n          w\n          \n            j\n            k\n          \n          \n            l\n          \n        \n        )\n      \n    \n    {\\displaystyle W^{l}=(w_{jk}^{l})}\n  \n: the weights between layer \n  \n    \n      \n        l\n        \u2212\n        1\n      \n    \n    {\\displaystyle l-1}\n  \n and \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n, where", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0009", "text": ", where \n  \n    \n      \n        \n          w\n          \n            j\n            k\n          \n          \n            l\n          \n        \n      \n    \n    {\\displaystyle w_{jk}^{l}}\n  \n is the weight between the \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n-th node in layer \n  \n    \n      \n        l\n        \u2212\n        1\n      \n    \n    {\\displaystyle l-1}\n  \n and the \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n-th node in layer \n  \n    \n      \n        l", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0010", "text": "-th node in layer \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0011", "text": "f\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle f^{l}}\n  \n: activation functions at layer \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0012", "text": "For classification the last layer is usually the logistic function for binary classification, and softmax (softargmax) for multi-class classification, while for the hidden layers this was traditionally a sigmoid function (logistic function or others) on each node (coordinate), but today is more varied, with rectifier (ramp, ReLU) being common.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0013", "text": "a\n          \n            j\n          \n          \n            l\n          \n        \n      \n    \n    {\\displaystyle a_{j}^{l}}\n  \n: activation of the \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n-th node in layer \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0014", "text": "{\\displaystyle l}\n  \n.\nIn the derivation of backpropagation, other intermediate quantities are used by introducing them as needed below. Bias terms are not treated specially since they correspond to a weight with a fixed input of 1. For backpropagation the specific loss function and activation functions do not matter as long as they and their derivatives can be evaluated efficiently. Traditional activation functions include sigmoid, tanh, and ReLU. Swish, Mish, and many others.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0015", "text": "The overall network is a combination of function composition and matrix multiplication:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0016", "text": "g\n        (\n        x\n        )\n        :=\n        \n          f\n          \n            L\n          \n        \n        (\n        \n          W\n          \n            L\n          \n        \n        \n          f\n          \n            L\n            \u2212\n            1\n          \n        \n        (\n        \n          W\n          \n            L\n            \u2212\n            1\n          \n        \n        \u22ef\n        \n          f\n          \n            1\n          \n        \n        (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0017", "text": "1\n          \n        \n        (\n        \n          W\n          \n            1\n          \n        \n        x\n        )\n        \u22ef\n        )\n        )\n      \n    \n    {\\displaystyle g(x):=f^{L}(W^{L}f^{L-1}(W^{L-1}\\cdots f^{1}(W^{1}x)\\cdots ))}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0018", "text": "For a training set there will be a set of input\u2013output pairs, \n  \n    \n      \n        \n          {\n          \n            (\n            \n              x\n              \n                i\n              \n            \n            ,\n            \n              y\n              \n                i\n              \n            \n            )\n          \n          }\n        \n      \n    \n    {\\displaystyle \\left\\{(x_{i},y_{i})\\right\\}}\n  \n. For each input\u2013output pair \n  \n    \n      \n        (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0019", "text": "(\n        \n          x\n          \n            i\n          \n        \n        ,\n        \n          y\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle (x_{i},y_{i})}\n  \n in the training set, the loss of the model on that pair is the cost of the difference between the predicted output \n  \n    \n      \n        g\n        (\n        \n          x\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle g(x_{i})}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0020", "text": "{\\displaystyle g(x_{i})}\n  \n and the target output \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}\n  \n:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0021", "text": "C\n        (\n        \n          y\n          \n            i\n          \n        \n        ,\n        g\n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        )\n      \n    \n    {\\displaystyle C(y_{i},g(x_{i}))}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0022", "text": "Note the distinction: during model evaluation the weights are fixed while the inputs vary (and the target output may be unknown), and the network ends with the output layer (it does not include the loss function). During model training the input\u2013output pair is fixed while the weights vary, and the network ends with the loss function.\nBackpropagation computes the gradient for a fixed input\u2013output pair \n  \n    \n      \n        (\n        \n          x\n          \n            i", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0023", "text": "i\n          \n        \n        ,\n        \n          y\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle (x_{i},y_{i})}\n  \n, where the weights \n  \n    \n      \n        \n          w\n          \n            j\n            k\n          \n          \n            l\n          \n        \n      \n    \n    {\\displaystyle w_{jk}^{l}}\n  \n can vary. Each individual component of the gradient, \n  \n    \n      \n        \u2202\n        C\n        \n          /", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0024", "text": "\u2202\n        C\n        \n          /\n        \n        \u2202\n        \n          w\n          \n            j\n            k\n          \n          \n            l\n          \n        \n        ,\n      \n    \n    {\\displaystyle \\partial C/\\partial w_{jk}^{l},}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0025", "text": "can be computed by the chain rule; but doing this separately for each weight is inefficient. Backpropagation efficiently computes the gradient by avoiding duplicate calculations and not computing unnecessary intermediate values, by computing the gradient of each layer \u2013 specifically the gradient of the weighted input of each layer, denoted by \n  \n    \n      \n        \n          \u03b4\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle \\delta ^{l}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0026", "text": "{\\displaystyle \\delta ^{l}}\n  \n \u2013 from back to front.\nInformally, the key point is that since the only way a weight in \n  \n    \n      \n        \n          W\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle W^{l}}\n  \n affects the loss is through its effect on the next layer, and it does so linearly, \n  \n    \n      \n        \n          \u03b4\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle \\delta ^{l}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0027", "text": "{\\displaystyle \\delta ^{l}}\n  \n are the only data you need to compute the gradients of the weights at layer \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n, and then the gradients of weights of previous layer can be computed by \n  \n    \n      \n        \n          \u03b4\n          \n            l\n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle \\delta ^{l-1}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0028", "text": "{\\displaystyle \\delta ^{l-1}}\n  \n and repeated recursively. This avoids inefficiency in two ways. First, it avoids duplication because when computing the gradient at layer \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n, it is unnecessary to recompute all derivatives on later layers \n  \n    \n      \n        l\n        +\n        1\n        ,\n        l\n        +\n        2\n        ,\n        \u2026\n      \n    \n    {\\displaystyle l+1,l+2,\\ldots }", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0029", "text": "{\\displaystyle l+1,l+2,\\ldots }\n  \n each time. Second, it avoids unnecessary intermediate calculations, because at each stage it directly computes the gradient of the weights with respect to the ultimate output (the loss), rather than unnecessarily computing the derivatives of the values of hidden layers with respect to changes in weights \n  \n    \n      \n        \u2202\n        \n          a\n          \n            \n              j\n              \u2032", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0030", "text": "l\n              \u2032\n            \n          \n        \n        \n          /\n        \n        \u2202\n        \n          w\n          \n            j\n            k\n          \n          \n            l\n          \n        \n      \n    \n    {\\displaystyle \\partial a_{j'}^{l'}/\\partial w_{jk}^{l}}\n  \n.\nBackpropagation can be expressed for simple feedforward networks in terms of matrix multiplication, or more generally in terms of the adjoint graph.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0031", "text": "Matrix multiplication", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0032", "text": "For the basic case of a feedforward network, where nodes in each layer are connected only to nodes in the immediate next layer (without skipping any layers), and there is a loss function that computes a scalar loss for the final output, backpropagation can be understood simply by matrix multiplication", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0033", "text": ". Essentially, backpropagation evaluates the expression for the derivative of the cost function as a product of derivatives between each layer from right to left \u2013 \"backwards\" \u2013 with the gradient of the weights between each layer being a simple modification of the partial products (the \"backwards propagated error\").", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0034", "text": "Given an input\u2013output pair \n  \n    \n      \n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle (x,y)}\n  \n, the loss is:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0035", "text": "C\n        (\n        y\n        ,\n        \n          f\n          \n            L\n          \n        \n        (\n        \n          W\n          \n            L\n          \n        \n        \n          f\n          \n            L\n            \u2212\n            1\n          \n        \n        (\n        \n          W\n          \n            L\n            \u2212\n            1\n          \n        \n        \u22ef\n        \n          f\n          \n            2\n          \n        \n        (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0036", "text": "(\n        \n          W\n          \n            2\n          \n        \n        \n          f\n          \n            1\n          \n        \n        (\n        \n          W\n          \n            1\n          \n        \n        x\n        )\n        )\n        \u22ef\n        )\n        )\n        )\n      \n    \n    {\\displaystyle C(y,f^{L}(W^{L}f^{L-1}(W^{L-1}\\cdots f^{2}(W^{2}f^{1}(W^{1}x))\\cdots )))}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0037", "text": "To compute this, one starts with the input \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n and works forward; denote the weighted input of each hidden layer as \n  \n    \n      \n        \n          z\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle z^{l}}\n  \n and the output of hidden layer \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n as the activation \n  \n    \n      \n        \n          a\n          \n            l", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0038", "text": "l\n          \n        \n      \n    \n    {\\displaystyle a^{l}}\n  \n. For backpropagation, the activation \n  \n    \n      \n        \n          a\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle a^{l}}\n  \n as well as the derivatives \n  \n    \n      \n        (\n        \n          f\n          \n            l\n          \n        \n        \n          )\n          \u2032\n        \n      \n    \n    {\\displaystyle (f^{l})'}\n  \n (evaluated at", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0039", "text": "(evaluated at \n  \n    \n      \n        \n          z\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle z^{l}}\n  \n) must be cached for use during the backwards pass.\nThe derivative of the loss in terms of the inputs is given by the chain rule; note that each term is a total derivative, evaluated at the value of the network (at each node) on the input \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0040", "text": "d\n              C\n            \n            \n              d\n              \n                a\n                \n                  L\n                \n              \n            \n          \n        \n        \u22c5\n        \n          \n            \n              d\n              \n                a\n                \n                  L\n                \n              \n            \n            \n              d\n              \n                z", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0041", "text": "d\n              \n                z\n                \n                  L\n                \n              \n            \n          \n        \n        \u22c5\n        \n          \n            \n              d\n              \n                z\n                \n                  L\n                \n              \n            \n            \n              d\n              \n                a\n                \n                  L\n                  \u2212\n                  1", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0042", "text": "\u22c5\n        \n          \n            \n              d\n              \n                a\n                \n                  L\n                  \u2212\n                  1\n                \n              \n            \n            \n              d\n              \n                z\n                \n                  L\n                  \u2212\n                  1\n                \n              \n            \n          \n        \n        \u22c5", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0043", "text": "\u22c5\n        \n          \n            \n              d\n              \n                z\n                \n                  L\n                  \u2212\n                  1\n                \n              \n            \n            \n              d\n              \n                a\n                \n                  L\n                  \u2212\n                  2\n                \n              \n            \n          \n        \n        \u22c5\n        \u2026\n        \u22c5", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0044", "text": "\u22c5\n        \u2026\n        \u22c5\n        \n          \n            \n              d\n              \n                a\n                \n                  1\n                \n              \n            \n            \n              d\n              \n                z\n                \n                  1\n                \n              \n            \n          \n        \n        \u22c5\n        \n          \n            \n              \u2202\n              \n                z\n                \n                  1", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0045", "text": "1\n                \n              \n            \n            \n              \u2202\n              x\n            \n          \n        \n        ,\n      \n    \n    {\\displaystyle {\\frac {dC}{da^{L}}}\\cdot {\\frac {da^{L}}{dz^{L}}}\\cdot {\\frac {dz^{L}}{da^{L-1}}}\\cdot {\\frac {da^{L-1}}{dz^{L-1}}}\\cdot {\\frac {dz^{L-1}}{da^{L-2}}}\\cdot \\ldots \\cdot {\\frac {da^{1}}{dz^{1}}}\\cdot {\\frac {\\partial z^{1}}{\\partial x}},}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0046", "text": "where \n  \n    \n      \n        \n          \n            \n              d\n              \n                a\n                \n                  L\n                \n              \n            \n            \n              d\n              \n                z\n                \n                  L\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {da^{L}}{dz^{L}}}}\n  \n is a diagonal matrix.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0047", "text": "is a diagonal matrix.\nThese terms are: the derivative of the loss function; the derivatives of the activation functions; and the matrices of weights:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0048", "text": "d\n              C\n            \n            \n              d\n              \n                a\n                \n                  L\n                \n              \n            \n          \n        \n        \u2218\n        (\n        \n          f\n          \n            L\n          \n        \n        \n          )\n          \u2032\n        \n        \u22c5\n        \n          W\n          \n            L\n          \n        \n        \u2218\n        (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0049", "text": "\u2218\n        (\n        \n          f\n          \n            L\n            \u2212\n            1\n          \n        \n        \n          )\n          \u2032\n        \n        \u22c5\n        \n          W\n          \n            L\n            \u2212\n            1\n          \n        \n        \u2218\n        \u22ef\n        \u2218\n        (\n        \n          f\n          \n            1\n          \n        \n        \n          )\n          \u2032\n        \n        \u22c5\n        \n          W\n          \n            1", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0050", "text": "W\n          \n            1\n          \n        \n        .\n      \n    \n    {\\displaystyle {\\frac {dC}{da^{L}}}\\circ (f^{L})'\\cdot W^{L}\\circ (f^{L-1})'\\cdot W^{L-1}\\circ \\cdots \\circ (f^{1})'\\cdot W^{1}.}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0051", "text": "The gradient \n  \n    \n      \n        \u2207\n      \n    \n    {\\displaystyle \\nabla }\n  \n is the transpose of the derivative of the output in terms of the input, so the matrices are transposed and the order of multiplication is reversed, but the entries are the same:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0052", "text": "\u2207\n          \n            x\n          \n        \n        C\n        =\n        (\n        \n          W\n          \n            1\n          \n        \n        \n          )\n          \n            T\n          \n        \n        \u22c5\n        (\n        \n          f\n          \n            1\n          \n        \n        \n          )\n          \u2032\n        \n        \u2218\n        \u2026\n        \u2218\n        (\n        \n          W\n          \n            L\n            \u2212\n            1", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0053", "text": "\u2212\n            1\n          \n        \n        \n          )\n          \n            T\n          \n        \n        \u22c5\n        (\n        \n          f\n          \n            L\n            \u2212\n            1\n          \n        \n        \n          )\n          \u2032\n        \n        \u2218\n        (\n        \n          W\n          \n            L\n          \n        \n        \n          )\n          \n            T\n          \n        \n        \u22c5\n        (\n        \n          f\n          \n            L", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0054", "text": "f\n          \n            L\n          \n        \n        \n          )\n          \u2032\n        \n        \u2218\n        \n          \u2207\n          \n            \n              a\n              \n                L\n              \n            \n          \n        \n        C\n        .\n      \n    \n    {\\displaystyle \\nabla _{x}C=(W^{1})^{T}\\cdot (f^{1})'\\circ \\ldots \\circ (W^{L-1})^{T}\\cdot (f^{L-1})'\\circ (W^{L})^{T}\\cdot (f^{L})'\\circ \\nabla _{a^{L}}C.}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0055", "text": "Backpropagation then consists essentially of evaluating this expression from right to left (equivalently, multiplying the previous expression for the derivative from left to right), computing the gradient at each layer on the way; there is an added step, because the gradient of the weights is not just a subexpression: there's an extra multiplication.\nIntroducing the auxiliary quantity \n  \n    \n      \n        \n          \u03b4\n          \n            l", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0056", "text": "l\n          \n        \n      \n    \n    {\\displaystyle \\delta ^{l}}\n  \n for the partial products (multiplying from right to left), interpreted as the \"error at level \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n\" and defined as the gradient of the input values at level \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0057", "text": "\u03b4\n          \n            l\n          \n        \n        :=\n        (\n        \n          f\n          \n            l\n          \n        \n        \n          )\n          \u2032\n        \n        \u2218\n        (\n        \n          W\n          \n            l\n            +\n            1\n          \n        \n        \n          )\n          \n            T\n          \n        \n        \u22c5\n        (\n        \n          f\n          \n            l\n            +\n            1", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0058", "text": "+\n            1\n          \n        \n        \n          )\n          \u2032\n        \n        \u2218\n        \u22ef\n        \u2218\n        (\n        \n          W\n          \n            L\n            \u2212\n            1\n          \n        \n        \n          )\n          \n            T\n          \n        \n        \u22c5\n        (\n        \n          f\n          \n            L\n            \u2212\n            1\n          \n        \n        \n          )\n          \u2032\n        \n        \u2218\n        (\n        \n          W", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0059", "text": "(\n        \n          W\n          \n            L\n          \n        \n        \n          )\n          \n            T\n          \n        \n        \u22c5\n        (\n        \n          f\n          \n            L\n          \n        \n        \n          )\n          \u2032\n        \n        \u2218\n        \n          \u2207\n          \n            \n              a\n              \n                L\n              \n            \n          \n        \n        C\n        .", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0060", "text": "C\n        .\n      \n    \n    {\\displaystyle \\delta ^{l}:=(f^{l})'\\circ (W^{l+1})^{T}\\cdot (f^{l+1})'\\circ \\cdots \\circ (W^{L-1})^{T}\\cdot (f^{L-1})'\\circ (W^{L})^{T}\\cdot (f^{L})'\\circ \\nabla _{a^{L}}C.}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0061", "text": "Note that \n  \n    \n      \n        \n          \u03b4\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle \\delta ^{l}}\n  \n is a vector, of length equal to the number of nodes in level \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n; each component is interpreted as the \"cost attributable to (the value of) that node\".\nThe gradient of the weights in layer \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n is then:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0062", "text": "\u2207\n          \n            \n              W\n              \n                l\n              \n            \n          \n        \n        C\n        =\n        \n          \u03b4\n          \n            l\n          \n        \n        (\n        \n          a\n          \n            l\n            \u2212\n            1\n          \n        \n        \n          )\n          \n            T\n          \n        \n        .\n      \n    \n    {\\displaystyle \\nabla _{W^{l}}C=\\delta ^{l}(a^{l-1})^{T}.}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0063", "text": "The factor of \n  \n    \n      \n        \n          a\n          \n            l\n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle a^{l-1}}\n  \n is because the weights \n  \n    \n      \n        \n          W\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle W^{l}}\n  \n between level \n  \n    \n      \n        l\n        \u2212\n        1\n      \n    \n    {\\displaystyle l-1}\n  \n and \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n affect level", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0064", "text": "{\\displaystyle l}\n  \n affect level \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n proportionally to the inputs (activations): the inputs are fixed, the weights vary.\nThe \n  \n    \n      \n        \n          \u03b4\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle \\delta ^{l}}\n  \n can easily be computed recursively, going from right to left, as:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0065", "text": "\u03b4\n          \n            l\n            \u2212\n            1\n          \n        \n        :=\n        (\n        \n          f\n          \n            l\n            \u2212\n            1\n          \n        \n        \n          )\n          \u2032\n        \n        \u2218\n        (\n        \n          W\n          \n            l\n          \n        \n        \n          )\n          \n            T\n          \n        \n        \u22c5\n        \n          \u03b4\n          \n            l", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0066", "text": "l\n          \n        \n        .\n      \n    \n    {\\displaystyle \\delta ^{l-1}:=(f^{l-1})'\\circ (W^{l})^{T}\\cdot \\delta ^{l}.}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0067", "text": "The gradients of the weights can thus be computed using a few matrix multiplications for each level; this is backpropagation.\nCompared with naively computing forwards (using the \n  \n    \n      \n        \n          \u03b4\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle \\delta ^{l}}\n  \n for illustration):", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0068", "text": "\u03b4\n                  \n                    1\n                  \n                \n              \n              \n                \n                =\n                (\n                \n                  f\n                  \n                    1\n                  \n                \n                \n                  )\n                  \u2032\n                \n                \u2218\n                (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0069", "text": "\u2218\n                (\n                \n                  W\n                  \n                    2\n                  \n                \n                \n                  )\n                  \n                    T\n                  \n                \n                \u22c5\n                (\n                \n                  f\n                  \n                    2\n                  \n                \n                \n                  )\n                  \u2032", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0070", "text": "\u2032\n                \n                \u2218\n                \u22ef\n                \u2218\n                (\n                \n                  W\n                  \n                    L\n                    \u2212\n                    1\n                  \n                \n                \n                  )\n                  \n                    T\n                  \n                \n                \u22c5\n                (\n                \n                  f\n                  \n                    L", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0071", "text": "L\n                    \u2212\n                    1\n                  \n                \n                \n                  )\n                  \u2032\n                \n                \u2218\n                (\n                \n                  W\n                  \n                    L\n                  \n                \n                \n                  )\n                  \n                    T\n                  \n                \n                \u22c5\n                (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0072", "text": "\u22c5\n                (\n                \n                  f\n                  \n                    L\n                  \n                \n                \n                  )\n                  \u2032\n                \n                \u2218\n                \n                  \u2207\n                  \n                    \n                      a\n                      \n                        L\n                      \n                    \n                  \n                \n                C", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0073", "text": "C\n              \n            \n            \n              \n                \n                  \u03b4\n                  \n                    2\n                  \n                \n              \n              \n                \n                =\n                (\n                \n                  f\n                  \n                    2\n                  \n                \n                \n                  )\n                  \u2032\n                \n                \u2218", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0074", "text": "\u2218\n                \u22ef\n                \u2218\n                (\n                \n                  W\n                  \n                    L\n                    \u2212\n                    1\n                  \n                \n                \n                  )\n                  \n                    T\n                  \n                \n                \u22c5\n                (\n                \n                  f\n                  \n                    L\n                    \u2212", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0075", "text": "L\n                    \u2212\n                    1\n                  \n                \n                \n                  )\n                  \u2032\n                \n                \u2218\n                (\n                \n                  W\n                  \n                    L\n                  \n                \n                \n                  )\n                  \n                    T\n                  \n                \n                \u22c5\n                (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0076", "text": "(\n                \n                  f\n                  \n                    L\n                  \n                \n                \n                  )\n                  \u2032\n                \n                \u2218\n                \n                  \u2207\n                  \n                    \n                      a\n                      \n                        L\n                      \n                    \n                  \n                \n                C", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0077", "text": "C\n              \n            \n            \n              \n              \n                \n                \u22ee\n              \n            \n            \n              \n                \n                  \u03b4\n                  \n                    L\n                    \u2212\n                    1\n                  \n                \n              \n              \n                \n                =\n                (\n                \n                  f\n                  \n                    L", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0078", "text": "L\n                    \u2212\n                    1\n                  \n                \n                \n                  )\n                  \u2032\n                \n                \u2218\n                (\n                \n                  W\n                  \n                    L\n                  \n                \n                \n                  )\n                  \n                    T\n                  \n                \n                \u22c5\n                (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0079", "text": "\u22c5\n                (\n                \n                  f\n                  \n                    L\n                  \n                \n                \n                  )\n                  \u2032\n                \n                \u2218\n                \n                  \u2207\n                  \n                    \n                      a\n                      \n                        L\n                      \n                    \n                  \n                \n                C", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0080", "text": "C\n              \n            \n            \n              \n                \n                  \u03b4\n                  \n                    L\n                  \n                \n              \n              \n                \n                =\n                (\n                \n                  f\n                  \n                    L\n                  \n                \n                \n                  )\n                  \u2032\n                \n                \u2218", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0081", "text": "\u2218\n                \n                  \u2207\n                  \n                    \n                      a\n                      \n                        L\n                      \n                    \n                  \n                \n                C\n                ,", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0082", "text": "{\\displaystyle {\\begin{aligned}\\delta ^{1}&=(f^{1})'\\circ (W^{2})^{T}\\cdot (f^{2})'\\circ \\cdots \\circ (W^{L-1})^{T}\\cdot (f^{L-1})'\\circ (W^{L})^{T}\\cdot (f^{L})'\\circ \\nabla _{a^{L}}C\\\\\\delta ^{2}&=(f^{2})'\\circ \\cdots \\circ (W^{L-1})^{T}\\cdot (f^{L-1})'\\circ (W^{L})^{T}\\cdot (f^{L})'\\circ \\nabla _{a^{L}}C\\\\&\\vdots \\\\\\delta ^{L-1}&=(f^{L-1})'\\circ (W^{L})^{T}\\cdot (f^{L})'\\circ \\nabla _{a^{L}}C\\\\\\delta ^{L}&=(f^{L})'\\circ \\nabla _{a^{L}}C,\\end{aligned}}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0083", "text": "There are two key differences with backpropagation:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0084", "text": "Computing \n  \n    \n      \n        \n          \u03b4\n          \n            l\n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle \\delta ^{l-1}}\n  \n in terms of \n  \n    \n      \n        \n          \u03b4\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle \\delta ^{l}}\n  \n avoids the obvious duplicate multiplication of layers \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n and beyond.\nMultiplying starting from", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0085", "text": "\u2207\n          \n            \n              a\n              \n                L\n              \n            \n          \n        \n        C\n      \n    \n    {\\displaystyle \\nabla _{a^{L}}C}\n  \n \u2013 propagating the error backwards \u2013 means that each step simply multiplies a vector (\n  \n    \n      \n        \n          \u03b4\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle \\delta ^{l}}\n  \n) by the matrices of weights \n  \n    \n      \n        (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0086", "text": "(\n        \n          W\n          \n            l\n          \n        \n        \n          )\n          \n            T\n          \n        \n      \n    \n    {\\displaystyle (W^{l})^{T}}\n  \n and derivatives of activations \n  \n    \n      \n        (\n        \n          f\n          \n            l\n            \u2212\n            1\n          \n        \n        \n          )\n          \u2032\n        \n      \n    \n    {\\displaystyle (f^{l-1})'}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0087", "text": "{\\displaystyle (f^{l-1})'}\n  \n. By contrast, multiplying forwards, starting from the changes at an earlier layer, means that each multiplication multiplies a matrix by a matrix. This is much more expensive, and corresponds to tracking every possible path of a change in one layer \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n forward to changes in the layer \n  \n    \n      \n        l\n        +\n        2\n      \n    \n    {\\displaystyle l+2}\n  \n (for multiplying", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0088", "text": "{\\displaystyle l+2}\n  \n (for multiplying \n  \n    \n      \n        \n          W\n          \n            l\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle W^{l+1}}\n  \n by \n  \n    \n      \n        \n          W\n          \n            l\n            +\n            2\n          \n        \n      \n    \n    {\\displaystyle W^{l+2}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0089", "text": "{\\displaystyle W^{l+2}}\n  \n, with additional multiplications for the derivatives of the activations), which unnecessarily computes the intermediate quantities of how weight changes affect the values of hidden nodes.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0090", "text": "Adjoint graph\nFor more general graphs, and other advanced variations, backpropagation can be understood in terms of automatic differentiation, where backpropagation is a special case of reverse accumulation (or \"reverse mode\").", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0091", "text": "Intuition\nMotivation\nThe goal of any supervised learning algorithm is to find a function that best maps a set of inputs to their correct output. The motivation for backpropagation is to train a multi-layered neural network such that it can learn the appropriate internal representations to allow it to learn any arbitrary mapping of input to output.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0092", "text": "Learning as an optimization problem", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0093", "text": "To understand the mathematical derivation of the backpropagation algorithm, it helps to first develop some intuition about the relationship between the actual output of a neuron and the correct output for a particular training example", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0094", "text": ". Consider a simple neural network with two input units, one output unit and no hidden units, and in which each neuron uses a linear output (unlike most work on neural networks, in which mapping from inputs to outputs is non-linear) that is the weighted sum of its input.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0095", "text": "Initially, before training, the weights will be set randomly. Then the neuron learns from training examples, which in this case consist of a set of tuples \n  \n    \n      \n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        t\n        )\n      \n    \n    {\\displaystyle (x_{1},x_{2},t)}\n  \n where \n  \n    \n      \n        \n          x\n          \n            1", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0096", "text": "1\n          \n        \n      \n    \n    {\\displaystyle x_{1}}\n  \n and \n  \n    \n      \n        \n          x\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle x_{2}}\n  \n are the inputs to the network and t is the correct output (the output the network should produce given those inputs, when it has been trained). The initial network, given \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle x_{1}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0097", "text": "{\\displaystyle x_{1}}\n  \n and \n  \n    \n      \n        \n          x\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle x_{2}}\n  \n, will compute an output y that likely differs from t (given random weights). A loss function \n  \n    \n      \n        L\n        (\n        t\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle L(t,y)}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0098", "text": "{\\displaystyle L(t,y)}\n  \n is used for measuring the discrepancy between the target output t and the computed output y. For regression analysis problems the squared error can be used as a loss function, for classification the categorical cross-entropy can be used.\nAs an example consider a regression problem using the square error as a loss:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0099", "text": "L\n        (\n        t\n        ,\n        y\n        )\n        =\n        (\n        t\n        \u2212\n        y\n        \n          )\n          \n            2\n          \n        \n        =\n        E\n        ,\n      \n    \n    {\\displaystyle L(t,y)=(t-y)^{2}=E,}\n  \n\nwhere E is the discrepancy or error.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0100", "text": "Consider the network on a single training case: \n  \n    \n      \n        (\n        1\n        ,\n        1\n        ,\n        0\n        )\n      \n    \n    {\\displaystyle (1,1,0)}\n  \n. Thus, the input \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle x_{1}}\n  \n and \n  \n    \n      \n        \n          x\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle x_{2}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0101", "text": "are 1 and 1 respectively and the correct output, t is 0. Now if the relation is plotted between the network's output y on the horizontal axis and the error E on the vertical axis, the result is a parabola. The minimum of the parabola corresponds to the output y which minimizes the error E. For a single training case, the minimum also touches the horizontal axis, which means the error will be zero and the network can produce an output y that exactly matches the target output t", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0102", "text": ". Therefore, the problem of mapping inputs to outputs can be reduced to an optimization problem of finding a function that will produce the minimal error.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0103", "text": "However, the output of a neuron depends on the weighted sum of all its inputs:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0104", "text": "y\n        =\n        \n          x\n          \n            1\n          \n        \n        \n          w\n          \n            1\n          \n        \n        +\n        \n          x\n          \n            2\n          \n        \n        \n          w\n          \n            2\n          \n        \n        ,\n      \n    \n    {\\displaystyle y=x_{1}w_{1}+x_{2}w_{2},}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0105", "text": "where \n  \n    \n      \n        \n          w\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle w_{1}}\n  \n and \n  \n    \n      \n        \n          w\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle w_{2}}\n  \n are the weights on the connection from the input units to the output unit. Therefore, the error also depends on the incoming weights to the neuron, which is ultimately what needs to be changed in the network to enable learning.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0106", "text": "In this example, upon injecting the training data \n  \n    \n      \n        (\n        1\n        ,\n        1\n        ,\n        0\n        )\n      \n    \n    {\\displaystyle (1,1,0)}\n  \n, the loss function becomes", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0107", "text": "E\n        =\n        (\n        t\n        \u2212\n        y\n        \n          )\n          \n            2\n          \n        \n        =\n        \n          y\n          \n            2\n          \n        \n        =\n        (\n        \n          x\n          \n            1\n          \n        \n        \n          w\n          \n            1\n          \n        \n        +\n        \n          x\n          \n            2\n          \n        \n        \n          w\n          \n            2", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0108", "text": "w\n          \n            2\n          \n        \n        \n          )\n          \n            2\n          \n        \n        =\n        (\n        \n          w\n          \n            1\n          \n        \n        +\n        \n          w\n          \n            2\n          \n        \n        \n          )\n          \n            2\n          \n        \n        .\n      \n    \n    {\\displaystyle E=(t-y)^{2}=y^{2}=(x_{1}w_{1}+x_{2}w_{2})^{2}=(w_{1}+w_{2})^{2}.}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0109", "text": "Then, the loss function \n  \n    \n      \n        E\n      \n    \n    {\\displaystyle E}\n  \n takes the form of a parabolic cylinder with its base directed along \n  \n    \n      \n        \n          w\n          \n            1\n          \n        \n        =\n        \u2212\n        \n          w\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle w_{1}=-w_{2}}\n  \n. Since all sets of weights that satisfy \n  \n    \n      \n        \n          w\n          \n            1", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0110", "text": "1\n          \n        \n        =\n        \u2212\n        \n          w\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle w_{1}=-w_{2}}\n  \n minimize the loss function, in this case additional constraints are required to converge to a unique solution. Additional constraints could either be generated by setting specific conditions to the weights, or by injecting additional training data.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0111", "text": "One commonly used algorithm to find the set of weights that minimizes the error is gradient descent. By backpropagation, the steepest descent direction is calculated of the loss function versus the present synaptic weights. Then, the weights can be modified along the steepest descent direction, and the error is minimized in an efficient way.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0112", "text": "Derivation\nThe gradient descent method involves calculating the derivative of the loss function with respect to the weights of the network. This is normally done using backpropagation. Assuming one output neuron, the squared error function is\n\n  \n    \n      \n        E\n        =\n        L\n        (\n        t\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle E=L(t,y)}\n  \n\nwhere", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0113", "text": "where\n\n  \n    \n      \n        L\n      \n    \n    {\\displaystyle L}\n  \n is the loss for the output \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n and target value \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n,\n\n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n is the target output for a training sample, and", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0114", "text": "y\n      \n    \n    {\\displaystyle y}\n  \n is the actual output of the output neuron.\nIn this section, the order of the weight indexes are reversed relative to the prior section: \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n is weight from the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \nth to the", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0115", "text": "j\n      \n    \n    {\\displaystyle j}\n  \nth unit.  For each neuron \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n, its output \n  \n    \n      \n        \n          o\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle o_{j}}\n  \n is defined as", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0116", "text": "o\n          \n            j\n          \n        \n        =\n        \u03c6\n        (\n        \n          \n            net\n          \n          \n            j\n          \n        \n        )\n        =\n        \u03c6\n        \n          (\n          \n            \n              \u2211\n              \n                k\n                =\n                1\n              \n              \n                n\n              \n            \n            \n              w", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0117", "text": "w\n              \n                k\n                j\n              \n            \n            \n              x\n              \n                k\n              \n            \n          \n          )\n        \n        ,\n      \n    \n    {\\displaystyle o_{j}=\\varphi ({\\text{net}}_{j})=\\varphi \\left(\\sum _{k=1}^{n}w_{kj}x_{k}\\right),}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0118", "text": "where the activation function \n  \n    \n      \n        \u03c6\n      \n    \n    {\\displaystyle \\varphi }\n  \n is non-linear and differentiable over the activation region (the ReLU is not differentiable at one point). A historically used activation function is the logistic function:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0119", "text": "\u03c6\n        (\n        z\n        )\n        =\n        \n          \n            1\n            \n              1\n              +\n              \n                e\n                \n                  \u2212\n                  z\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\varphi (z)={\\frac {1}{1+e^{-z}}}}\n  \n\nwhich has a convenient derivative of:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0120", "text": "which has a convenient derivative of:\n\n  \n    \n      \n        \n          \n            \n              d\n              \u03c6\n            \n            \n              d\n              z\n            \n          \n        \n        =\n        \u03c6\n        (\n        z\n        )\n        (\n        1\n        \u2212\n        \u03c6\n        (\n        z\n        )\n        )\n      \n    \n    {\\displaystyle {\\frac {d\\varphi }{dz}}=\\varphi (z)(1-\\varphi (z))}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0121", "text": "The input \n  \n    \n      \n        \n          \n            net\n          \n          \n            j\n          \n        \n      \n    \n    {\\displaystyle {\\text{net}}_{j}}\n  \n to a neuron is the weighted sum of outputs \n  \n    \n      \n        \n          o\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle o_{k}}\n  \n of previous neurons. If the neuron is in the first layer after the input layer, the \n  \n    \n      \n        \n          o\n          \n            k", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0122", "text": "o\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle o_{k}}\n  \n of the input layer are simply the inputs \n  \n    \n      \n        \n          x\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle x_{k}}\n  \n to the network. The number of input units to the neuron is \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n. The variable \n  \n    \n      \n        \n          w\n          \n            k\n            j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0123", "text": "k\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{kj}}\n  \n denotes the weight between neuron \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n of the previous layer and neuron \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n of the current layer.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0124", "text": "Finding the derivative of the error\nCalculating the partial derivative of the error with respect to a weight \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n is done using the chain rule twice:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0125", "text": "In the last factor of the right-hand side of the above, only one term in the sum \n  \n    \n      \n        \n          \n            net\n          \n          \n            j\n          \n        \n      \n    \n    {\\displaystyle {\\text{net}}_{j}}\n  \n depends on \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n, so that", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0126", "text": "If the neuron is in the first layer after the input layer, \n  \n    \n      \n        \n          o\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle o_{i}}\n  \n is just \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n.\nThe derivative of the output of neuron \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0127", "text": "j\n      \n    \n    {\\displaystyle j}\n  \n with respect to its input is simply the partial derivative of the activation function:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0128", "text": "which for the logistic activation function", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0129", "text": "\u2202\n              \n                o\n                \n                  j\n                \n              \n            \n            \n              \u2202\n              \n                \n                  net\n                \n                \n                  j\n                \n              \n            \n          \n        \n        =\n        \n          \n            \u2202\n            \n              \u2202", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0130", "text": "\u2202\n              \n                \n                  net\n                \n                \n                  j\n                \n              \n            \n          \n        \n        \u03c6\n        (\n        \n          \n            net\n          \n          \n            j\n          \n        \n        )\n        =\n        \u03c6\n        (\n        \n          \n            net\n          \n          \n            j\n          \n        \n        )\n        (\n        1\n        \u2212\n        \u03c6\n        (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0131", "text": "(\n        1\n        \u2212\n        \u03c6\n        (\n        \n          \n            net\n          \n          \n            j\n          \n        \n        )\n        )\n        =\n        \n          o\n          \n            j\n          \n        \n        (\n        1\n        \u2212\n        \n          o\n          \n            j\n          \n        \n        )", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0132", "text": ")\n      \n    \n    {\\displaystyle {\\frac {\\partial o_{j}}{\\partial {\\text{net}}_{j}}}={\\frac {\\partial }{\\partial {\\text{net}}_{j}}}\\varphi ({\\text{net}}_{j})=\\varphi ({\\text{net}}_{j})(1-\\varphi ({\\text{net}}_{j}))=o_{j}(1-o_{j})}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0133", "text": "This is the reason why backpropagation requires that the activation function be differentiable. (Nevertheless, the ReLU activation function, which is non-differentiable at 0, has become quite popular, e.g. in AlexNet)\nThe first factor is straightforward to evaluate if the neuron is in the output layer, because then \n  \n    \n      \n        \n          o\n          \n            j\n          \n        \n        =\n        y\n      \n    \n    {\\displaystyle o_{j}=y}\n  \n and", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0134", "text": "If half of the square error is used as loss function we can rewrite it as", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0135", "text": "\u2202\n              E\n            \n            \n              \u2202\n              \n                o\n                \n                  j\n                \n              \n            \n          \n        \n        =\n        \n          \n            \n              \u2202\n              E\n            \n            \n              \u2202\n              y\n            \n          \n        \n        =\n        \n          \n            \u2202\n            \n              \u2202", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0136", "text": "\u2202\n            \n              \u2202\n              y\n            \n          \n        \n        \n          \n            1\n            2\n          \n        \n        (\n        t\n        \u2212\n        y\n        \n          )\n          \n            2\n          \n        \n        =\n        y\n        \u2212\n        t\n      \n    \n    {\\displaystyle {\\frac {\\partial E}{\\partial o_{j}}}={\\frac {\\partial E}{\\partial y}}={\\frac {\\partial }{\\partial y}}{\\frac {1}{2}}(t-y)^{2}=y-t}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0137", "text": "However, if \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n is in an arbitrary inner layer of the network, finding the derivative \n  \n    \n      \n        E\n      \n    \n    {\\displaystyle E}\n  \n with respect to \n  \n    \n      \n        \n          o\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle o_{j}}\n  \n is less obvious.\nConsidering \n  \n    \n      \n        E\n      \n    \n    {\\displaystyle E}\n  \n as a function with the inputs being all neurons", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0138", "text": "L\n        =\n        {\n        u\n        ,\n        v\n        ,\n        \u2026\n        ,\n        w\n        }\n      \n    \n    {\\displaystyle L=\\{u,v,\\dots ,w\\}}\n  \n receiving input from neuron \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n,", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0139", "text": "\u2202\n              E\n              (\n              \n                o\n                \n                  j\n                \n              \n              )\n            \n            \n              \u2202\n              \n                o\n                \n                  j\n                \n              \n            \n          \n        \n        =\n        \n          \n            \n              \u2202\n              E\n              (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0140", "text": "E\n              (\n              \n                \n                  n\n                  e\n                  t\n                \n                \n                  u\n                \n              \n              ,\n              \n                \n                  net\n                \n                \n                  v\n                \n              \n              ,\n              \u2026\n              ,\n              \n                \n                  n\n                  e", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0141", "text": "n\n                  e\n                  t\n                \n                \n                  w\n                \n              \n              )\n            \n            \n              \u2202\n              \n                o\n                \n                  j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0142", "text": "{\\displaystyle {\\frac {\\partial E(o_{j})}{\\partial o_{j}}}={\\frac {\\partial E(\\mathrm {net} _{u},{\\text{net}}_{v},\\dots ,\\mathrm {net} _{w})}{\\partial o_{j}}}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0143", "text": "and taking the total derivative with respect to \n  \n    \n      \n        \n          o\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle o_{j}}\n  \n, a recursive expression for the derivative is obtained:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0144", "text": "Therefore, the derivative with respect to \n  \n    \n      \n        \n          o\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle o_{j}}\n  \n can be calculated if all the derivatives with respect to the outputs \n  \n    \n      \n        \n          o\n          \n            \u2113\n          \n        \n      \n    \n    {\\displaystyle o_{\\ell }}\n  \n of the next layer \u2013 the ones closer to the output neuron \u2013 are known. [Note, if any of the neurons in set \n  \n    \n      \n        L", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0145", "text": "L\n      \n    \n    {\\displaystyle L}\n  \n were not connected to neuron \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n, they would be independent of \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n and the corresponding partial derivative under the summation would vanish to 0.]\nSubstituting Eq. 2, Eq. 3 Eq.4 and Eq. 5 in Eq. 1 we obtain:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0146", "text": "\u2202\n              E\n            \n            \n              \u2202\n              \n                w\n                \n                  i\n                  j\n                \n              \n            \n          \n        \n        =\n        \n          \n            \n              \u2202\n              E\n            \n            \n              \u2202\n              \n                o\n                \n                  j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0147", "text": "\u2202\n              \n                o\n                \n                  j\n                \n              \n            \n            \n              \u2202\n              \n                \n                  net\n                \n                \n                  j\n                \n              \n            \n          \n        \n        \n          \n            \n              \u2202", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0148", "text": "\u2202\n              \n                \n                  net\n                \n                \n                  j\n                \n              \n            \n            \n              \u2202\n              \n                w\n                \n                  i\n                  j\n                \n              \n            \n          \n        \n        =\n        \n          \n            \n              \u2202\n              E\n            \n            \n              \u2202", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0149", "text": "\u2202\n              \n                o\n                \n                  j\n                \n              \n            \n          \n        \n        \n          \n            \n              \u2202\n              \n                o\n                \n                  j\n                \n              \n            \n            \n              \u2202\n              \n                \n                  net\n                \n                \n                  j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0150", "text": "o\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial E}{\\partial w_{ij}}}={\\frac {\\partial E}{\\partial o_{j}}}{\\frac {\\partial o_{j}}{\\partial {\\text{net}}_{j}}}{\\frac {\\partial {\\text{net}}_{j}}{\\partial w_{ij}}}={\\frac {\\partial E}{\\partial o_{j}}}{\\frac {\\partial o_{j}}{\\partial {\\text{net}}_{j}}}o_{i}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0151", "text": "\u2202\n              E\n            \n            \n              \u2202\n              \n                w\n                \n                  i\n                  j\n                \n              \n            \n          \n        \n        =\n        \n          o\n          \n            i\n          \n        \n        \n          \u03b4\n          \n            j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0152", "text": "j\n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial E}{\\partial w_{ij}}}=o_{i}\\delta _{j}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0153", "text": "with", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0154", "text": "\u03b4\n          \n            j\n          \n        \n        =\n        \n          \n            \n              \u2202\n              E\n            \n            \n              \u2202\n              \n                o\n                \n                  j\n                \n              \n            \n          \n        \n        \n          \n            \n              \u2202\n              \n                o\n                \n                  j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0155", "text": "\u2202\n              \n                \n                  net\n                \n                \n                  j\n                \n              \n            \n          \n        \n        =\n        \n          \n            {\n            \n              \n                \n                  \n                    \n                      \n                        \u2202\n                        L\n                        (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0156", "text": "(\n                        t\n                        ,\n                        \n                          o\n                          \n                            j\n                          \n                        \n                        )\n                      \n                      \n                        \u2202\n                        \n                          o\n                          \n                            j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0157", "text": "d\n                        \u03c6\n                        (\n                        \n                          \n                            net\n                          \n                          \n                            j\n                          \n                        \n                        )", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0158", "text": ")\n                      \n                      \n                        d\n                        \n                          \n                            net\n                          \n                          \n                            j\n                          \n                        \n                      \n                    \n                  \n                \n                \n                  \n                    if \n                  \n                  j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0159", "text": "j\n                  \n                     is an output neuron,\n                  \n                \n              \n              \n                \n                  (\n                  \n                    \u2211\n                    \n                      \u2113\n                      \u2208\n                      L\n                    \n                  \n                  \n                    w\n                    \n                      j\n                      \u2113", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0160", "text": "j\n                      \u2113\n                    \n                  \n                  \n                    \u03b4\n                    \n                      \u2113\n                    \n                  \n                  )\n                  \n                    \n                      \n                        d\n                        \u03c6\n                        (\n                        \n                          \n                            net", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0161", "text": "j\n                          \n                        \n                        )\n                      \n                      \n                        d\n                        \n                          \n                            net\n                          \n                          \n                            j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0162", "text": "if \n                  \n                  j\n                  \n                     is an inner neuron.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0163", "text": "{\\displaystyle \\delta _{j}={\\frac {\\partial E}{\\partial o_{j}}}{\\frac {\\partial o_{j}}{\\partial {\\text{net}}_{j}}}={\\begin{cases}{\\frac {\\partial L(t,o_{j})}{\\partial o_{j}}}{\\frac {d\\varphi ({\\text{net}}_{j})}{d{\\text{net}}_{j}}}&{\\text{if }}j{\\text{ is an output neuron,}}\\\\(\\sum _{\\ell \\in L}w_{j\\ell }\\delta _{\\ell }){\\frac {d\\varphi ({\\text{net}}_{j})}{d{\\text{net}}_{j}}}&{\\text{if }}j{\\text{ is an inner neuron.}}\\end{cases}}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0164", "text": "if \n  \n    \n      \n        \u03c6\n      \n    \n    {\\displaystyle \\varphi }\n  \n is the logistic function, and the error is the square error:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0165", "text": "\u03b4\n          \n            j\n          \n        \n        =\n        \n          \n            \n              \u2202\n              E\n            \n            \n              \u2202\n              \n                o\n                \n                  j\n                \n              \n            \n          \n        \n        \n          \n            \n              \u2202\n              \n                o\n                \n                  j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0166", "text": "\u2202\n              \n                \n                  net\n                \n                \n                  j\n                \n              \n            \n          \n        \n        =\n        \n          \n            {\n            \n              \n                \n                  (\n                  \n                    o\n                    \n                      j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0167", "text": "\u2212\n                  \n                    t\n                    \n                      j\n                    \n                  \n                  )\n                  \n                    o\n                    \n                      j\n                    \n                  \n                  (\n                  1\n                  \u2212\n                  \n                    o\n                    \n                      j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0168", "text": "j\n                    \n                  \n                  )\n                \n                \n                  \n                    if \n                  \n                  j\n                  \n                     is an output neuron,\n                  \n                \n              \n              \n                \n                  (\n                  \n                    \u2211\n                    \n                      \u2113\n                      \u2208\n                      L", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0169", "text": "\u2208\n                      L\n                    \n                  \n                  \n                    w\n                    \n                      j\n                      \u2113\n                    \n                  \n                  \n                    \u03b4\n                    \n                      \u2113\n                    \n                  \n                  )\n                  \n                    o\n                    \n                      j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0170", "text": "j\n                    \n                  \n                  (\n                  1\n                  \u2212\n                  \n                    o\n                    \n                      j\n                    \n                  \n                  )\n                \n                \n                  \n                    if \n                  \n                  j\n                  \n                     is an inner neuron.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0171", "text": "{\\displaystyle \\delta _{j}={\\frac {\\partial E}{\\partial o_{j}}}{\\frac {\\partial o_{j}}{\\partial {\\text{net}}_{j}}}={\\begin{cases}(o_{j}-t_{j})o_{j}(1-o_{j})&{\\text{if }}j{\\text{ is an output neuron,}}\\\\(\\sum _{\\ell \\in L}w_{j\\ell }\\delta _{\\ell })o_{j}(1-o_{j})&{\\text{if }}j{\\text{ is an inner neuron.}}\\end{cases}}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0172", "text": "To update the weight \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n using gradient descent, one must choose a learning rate, \n  \n    \n      \n        \u03b7\n        >\n        0\n      \n    \n    {\\displaystyle \\eta >0}\n  \n. The change in weight needs to reflect the impact on \n  \n    \n      \n        E\n      \n    \n    {\\displaystyle E}\n  \n of an increase or decrease in \n  \n    \n      \n        \n          w", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0173", "text": "w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n. If \n  \n    \n      \n        \n          \n            \n              \u2202\n              E\n            \n            \n              \u2202\n              \n                w\n                \n                  i\n                  j\n                \n              \n            \n          \n        \n        >\n        0", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0174", "text": ">\n        0\n      \n    \n    {\\displaystyle {\\frac {\\partial E}{\\partial w_{ij}}}>0}\n  \n, an increase in \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n increases \n  \n    \n      \n        E\n      \n    \n    {\\displaystyle E}\n  \n; conversely, if \n  \n    \n      \n        \n          \n            \n              \u2202\n              E\n            \n            \n              \u2202", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0175", "text": "\u2202\n              \n                w\n                \n                  i\n                  j\n                \n              \n            \n          \n        \n        <\n        0\n      \n    \n    {\\displaystyle {\\frac {\\partial E}{\\partial w_{ij}}}<0}\n  \n, an increase in \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n decreases \n  \n    \n      \n        E", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0176", "text": "decreases \n  \n    \n      \n        E\n      \n    \n    {\\displaystyle E}\n  \n. The new \n  \n    \n      \n        \u0394\n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle \\Delta w_{ij}}\n  \n is added to the old weight, and the product of the learning rate and the gradient, multiplied by \n  \n    \n      \n        \u2212\n        1\n      \n    \n    {\\displaystyle -1}\n  \n guarantees that \n  \n    \n      \n        \n          w\n          \n            i", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0177", "text": "w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n changes in a way that always decreases \n  \n    \n      \n        E\n      \n    \n    {\\displaystyle E}\n  \n. In other words, in the equation immediately below, \n  \n    \n      \n        \u2212\n        \u03b7\n        \n          \n            \n              \u2202\n              E\n            \n            \n              \u2202\n              \n                w\n                \n                  i", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0178", "text": "i\n                  j\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle -\\eta {\\frac {\\partial E}{\\partial w_{ij}}}}\n  \n always changes \n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n in such a way that \n  \n    \n      \n        E\n      \n    \n    {\\displaystyle E}\n  \n is decreased:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0179", "text": "\u0394\n        \n          w\n          \n            i\n            j\n          \n        \n        =\n        \u2212\n        \u03b7\n        \n          \n            \n              \u2202\n              E\n            \n            \n              \u2202\n              \n                w\n                \n                  i\n                  j\n                \n              \n            \n          \n        \n        =\n        \u2212\n        \u03b7\n        \n          o\n          \n            i", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0180", "text": "i\n          \n        \n        \n          \u03b4\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\Delta w_{ij}=-\\eta {\\frac {\\partial E}{\\partial w_{ij}}}=-\\eta o_{i}\\delta _{j}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0181", "text": "Second-order gradient descent\nUsing a Hessian matrix of second-order derivatives of the error function, the Levenberg\u2013Marquardt algorithm often converges faster than first-order gradient descent, especially when the topology of the error function is complicated. It may also find solutions in smaller node counts for which other methods might not converge. The Hessian can be approximated by the Fisher information matrix.\nAs an example, consider a simple feedforward network. At the", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0182", "text": "l\n      \n    \n    {\\displaystyle l}\n  \n-th layer, we have\n  \n    \n      \n        \n          x\n          \n            i\n          \n          \n            (\n            l\n            )\n          \n        \n        ,\n        \n        \n          a\n          \n            i\n          \n          \n            (\n            l\n            )\n          \n        \n        =\n        f\n        (\n        \n          x\n          \n            i\n          \n          \n            (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0183", "text": "i\n          \n          \n            (\n            l\n            )\n          \n        \n        )\n        ,\n        \n        \n          x\n          \n            i\n          \n          \n            (\n            l\n            +\n            1\n            )\n          \n        \n        =\n        \n          \u2211\n          \n            j\n          \n        \n        \n          W\n          \n            i\n            j\n          \n        \n        \n          a\n          \n            j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0184", "text": "a\n          \n            j\n          \n          \n            (\n            l\n            )\n          \n        \n      \n    \n    {\\displaystyle x_{i}^{(l)},\\quad a_{i}^{(l)}=f(x_{i}^{(l)}),\\quad x_{i}^{(l+1)}=\\sum _{j}W_{ij}a_{j}^{(l)}}\n  \nwhere \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n are the pre-activations, \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n are the activations, and \n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0185", "text": "W\n      \n    \n    {\\displaystyle W}\n  \n is the weight matrix. Given a loss function \n  \n    \n      \n        L\n      \n    \n    {\\displaystyle L}\n  \n, the first-order backpropagation states that\n  \n    \n      \n        \n          \n            \n              \u2202\n              L\n            \n            \n              \u2202\n              \n                a\n                \n                  j\n                \n                \n                  (\n                  l\n                  )", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0186", "text": "l\n                  )\n                \n              \n            \n          \n        \n        =\n        \n          \u2211\n          \n            j\n          \n        \n        \n          W\n          \n            i\n            j\n          \n        \n        \n          \n            \n              \u2202\n              L\n            \n            \n              \u2202\n              \n                x\n                \n                  i\n                \n                \n                  (", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0187", "text": "(\n                  l\n                  +\n                  1\n                  )\n                \n              \n            \n          \n        \n        ,\n        \n        \n          \n            \n              \u2202\n              L\n            \n            \n              \u2202\n              \n                x\n                \n                  j\n                \n                \n                  (\n                  l\n                  )", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0188", "text": ")\n                \n              \n            \n          \n        \n        =\n        \n          f\n          \u2032\n        \n        (\n        \n          x\n          \n            j\n          \n          \n            (\n            l\n            )\n          \n        \n        )\n        \n          \n            \n              \u2202\n              L\n            \n            \n              \u2202\n              \n                a\n                \n                  j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0189", "text": "(\n                  l\n                  )\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial L}{\\partial a_{j}^{(l)}}}=\\sum _{j}W_{ij}{\\frac {\\partial L}{\\partial x_{i}^{(l+1)}}},\\quad {\\frac {\\partial L}{\\partial x_{j}^{(l)}}}=f'(x_{j}^{(l)}){\\frac {\\partial L}{\\partial a_{j}^{(l)}}}}\n  \nand the second-order backpropagation states that", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0190", "text": "\u2202\n                \n                  2\n                \n              \n              L\n            \n            \n              \u2202\n              \n                a\n                \n                  \n                    j\n                    \n                      1\n                    \n                  \n                \n                \n                  (\n                  l\n                  )", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0191", "text": ")\n                \n              \n              \u2202\n              \n                a\n                \n                  \n                    j\n                    \n                      2\n                    \n                  \n                \n                \n                  (\n                  l\n                  )\n                \n              \n            \n          \n        \n        =\n        \n          \u2211\n          \n            \n              j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0192", "text": "j\n              \n                1\n              \n            \n            \n              j\n              \n                2\n              \n            \n          \n        \n        \n          W\n          \n            \n              i\n              \n                1\n              \n            \n            \n              j\n              \n                1\n              \n            \n          \n        \n        \n          W\n          \n            \n              i", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0193", "text": "i\n              \n                2\n              \n            \n            \n              j\n              \n                2\n              \n            \n          \n        \n        \n          \n            \n              \n                \u2202\n                \n                  2\n                \n              \n              L\n            \n            \n              \u2202\n              \n                x\n                \n                  \n                    i", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0194", "text": "i\n                    \n                      1\n                    \n                  \n                \n                \n                  (\n                  l\n                  +\n                  1\n                  )\n                \n              \n              \u2202\n              \n                x\n                \n                  \n                    i\n                    \n                      2", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0195", "text": "(\n                  l\n                  +\n                  1\n                  )\n                \n              \n            \n          \n        \n        ,\n        \n        \n          \n            \n              \n                \u2202\n                \n                  2\n                \n              \n              L\n            \n            \n              \u2202\n              \n                x", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0196", "text": "\u2202\n              \n                x\n                \n                  \n                    j\n                    \n                      1\n                    \n                  \n                \n                \n                  (\n                  l\n                  )\n                \n              \n              \u2202\n              \n                x\n                \n                  \n                    j\n                    \n                      2", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0197", "text": "2\n                    \n                  \n                \n                \n                  (\n                  l\n                  )\n                \n              \n            \n          \n        \n        =\n        \n          f\n          \u2032\n        \n        (\n        \n          x\n          \n            \n              j\n              \n                1\n              \n            \n          \n          \n            (\n            l\n            )", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0198", "text": "l\n            )\n          \n        \n        )\n        \n          f\n          \u2032\n        \n        (\n        \n          x\n          \n            \n              j\n              \n                2\n              \n            \n          \n          \n            (\n            l\n            )\n          \n        \n        )\n        \n          \n            \n              \n                \u2202\n                \n                  2\n                \n              \n              L", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0199", "text": "L\n            \n            \n              \u2202\n              \n                a\n                \n                  \n                    j\n                    \n                      1\n                    \n                  \n                \n                \n                  (\n                  l\n                  )\n                \n              \n              \u2202\n              \n                a\n                \n                  \n                    j", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0200", "text": "j\n                    \n                      2\n                    \n                  \n                \n                \n                  (\n                  l\n                  )\n                \n              \n            \n          \n        \n        +\n        \n          \u03b4\n          \n            \n              j\n              \n                1\n              \n            \n            \n              j\n              \n                2", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0201", "text": "2\n              \n            \n          \n        \n        \n          f\n          \u2033\n        \n        (\n        \n          x\n          \n            \n              j\n              \n                1\n              \n            \n          \n          \n            (\n            l\n            )\n          \n        \n        )\n        \n          \n            \n              \u2202\n              L\n            \n            \n              \u2202\n              \n                a", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0202", "text": "\u2202\n              \n                a\n                \n                  \n                    j\n                    \n                      1\n                    \n                  \n                \n                \n                  (\n                  l\n                  )", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0203", "text": "{\\displaystyle {\\frac {\\partial ^{2}L}{\\partial a_{j_{1}}^{(l)}\\partial a_{j_{2}}^{(l)}}}=\\sum _{j_{1}j_{2}}W_{i_{1}j_{1}}W_{i_{2}j_{2}}{\\frac {\\partial ^{2}L}{\\partial x_{i_{1}}^{(l+1)}\\partial x_{i_{2}}^{(l+1)}}},\\quad {\\frac {\\partial ^{2}L}{\\partial x_{j_{1}}^{(l)}\\partial x_{j_{2}}^{(l)}}}=f'(x_{j_{1}}^{(l)})f'(x_{j_{2}}^{(l)}){\\frac {\\partial ^{2}L}{\\partial a_{j_{1}}^{(l)}\\partial a_{j_{2}}^{(l)}}}+\\delta _{j_{1}j_{2}}f''(x_{j_{1}}^{(l)}){\\frac {\\partial L}{\\partial", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0204", "text": "{\\partial L}{\\partial a_{j_{1}}^{(l)}}}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0205", "text": "where \n  \n    \n      \n        \u03b4\n      \n    \n    {\\displaystyle \\delta }\n  \n is the Dirac delta symbol.\nArbitrary-order derivatives in arbitrary computational graphs can be computed with backpropagation, but with more complex expressions for higher orders.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0206", "text": "Loss function\nThe loss function is a function that maps values of one or more variables onto a real number intuitively representing some \"cost\" associated with those values. For backpropagation, the loss function calculates the difference between the network output and its expected output, after a training example has propagated through the network.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0207", "text": "Assumptions\nThe mathematical expression of the loss function must fulfill two conditions in order for it to be possibly used in backpropagation. The first is that it can be written as an average \n  \n    \n      \n        E\n        =\n        \n          \n            1\n            n\n          \n        \n        \n          \u2211\n          \n            x\n          \n        \n        \n          E\n          \n            x\n          \n        \n      \n    \n    {\\textstyle E={\\frac {1}{n}}\\sum _{x}E_{x}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0208", "text": "over error functions \n  \n    \n      \n        \n          E\n          \n            x\n          \n        \n      \n    \n    {\\textstyle E_{x}}\n  \n, for \n  \n    \n      \n        n\n      \n    \n    {\\textstyle n}\n  \n individual training examples, \n  \n    \n      \n        x\n      \n    \n    {\\textstyle x}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0209", "text": "x\n      \n    \n    {\\textstyle x}\n  \n. The reason for this assumption is that the backpropagation algorithm calculates the gradient of the error function for a single training example, which needs to be generalized to the overall error function. The second assumption is that it can be written as a function of the outputs from the neural network.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0210", "text": "Example loss function\nLet \n  \n    \n      \n        y\n        ,\n        \n          y\n          \u2032\n        \n      \n    \n    {\\displaystyle y,y'}\n  \n be vectors in \n  \n    \n      \n        \n          \n            R\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {R} ^{n}}\n  \n.\nSelect an error function \n  \n    \n      \n        E\n        (\n        y\n        ,\n        \n          y\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle E(y,y')}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0211", "text": "{\\displaystyle E(y,y')}\n  \n measuring the difference between two outputs. The standard choice is the square of the Euclidean distance between the vectors \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n and \n  \n    \n      \n        \n          y\n          \u2032\n        \n      \n    \n    {\\displaystyle y'}\n  \n:\n  \n    \n      \n        E\n        (\n        y\n        ,\n        \n          y\n          \u2032\n        \n        )\n        =", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0212", "text": "=\n        \n          \n            \n              1\n              2\n            \n          \n        \n        \u2016\n        y\n        \u2212\n        \n          y\n          \u2032\n        \n        \n          \u2016\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle E(y,y')={\\tfrac {1}{2}}\\lVert y-y'\\rVert ^{2}}\n  \nThe error function over \n  \n    \n      \n        n\n      \n    \n    {\\textstyle n}\n  \n training examples can then be written as an average of losses over individual examples:", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0213", "text": "E\n        =\n        \n          \n            1\n            \n              2\n              n\n            \n          \n        \n        \n          \u2211\n          \n            x\n          \n        \n        \u2016\n        (\n        y\n        (\n        x\n        )\n        \u2212\n        \n          y\n          \u2032\n        \n        (\n        x\n        )\n        )\n        \n          \u2016\n          \n            2", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0214", "text": "2\n          \n        \n      \n    \n    {\\displaystyle E={\\frac {1}{2n}}\\sum _{x}\\lVert (y(x)-y'(x))\\rVert ^{2}}", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0215", "text": "Limitations\nGradient descent with backpropagation is not guaranteed to find the global minimum of the error function, but only a local minimum; also, it has trouble crossing plateaus in the error function landscape. This issue, caused by the non-convexity of error functions in neural networks, was long thought to be a major drawback, but Yann LeCun et al. argue that in many practical problems, it is not.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0216", "text": "Backpropagation learning does not require normalization of input vectors; however, normalization could improve performance.\nBackpropagation requires the derivatives of activation functions to be known at network design time.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0217", "text": "History\nPrecursors\nBackpropagation had been derived repeatedly, as it is essentially an efficient application of the chain rule (first written down by Gottfried Wilhelm Leibniz in 1676) to neural networks.\nThe terminology \"back-propagating error correction\" was introduced in 1962 by Frank Rosenblatt, but he did not know how to implement this. In any case, he only studied neurons whose outputs were discrete levels, which only had zero derivatives, making backpropagation impossible.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0218", "text": "Precursors to backpropagation appeared in optimal control theory since 1950s. Yann LeCun et al credits 1950s work by Pontryagin and others in optimal control theory, especially the adjoint state method, for being a continuous-time version of backpropagation. Hecht-Nielsen credits the Robbins\u2013Monro algorithm (1951) and Arthur Bryson and Yu-Chi Ho's Applied Optimal Control (1969) as presages of backpropagation. Other precursors were Henry J. Kelley 1960, and Arthur E. Bryson (1961)", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0219", "text": ". Kelley 1960, and Arthur E. Bryson (1961). In 1962, Stuart Dreyfus published a simpler derivation based only on the chain rule. In 1973, he adapted parameters of controllers in proportion to error gradients. Unlike modern backpropagation, these precursors used standard Jacobian matrix calculations from one stage to the previous one, neither addressing direct links across several stages nor potential additional efficiency gains due to network sparsity.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0220", "text": "The ADALINE (1960) learning algorithm was gradient descent with a squared error loss for a single layer. The first multilayer perceptron (MLP) with more than one layer trained by stochastic gradient descent was published in 1967 by Shun'ichi Amari. The MLP had 5 layers, with 2 learnable layers, and it learned to classify patterns not linearly separable.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0221", "text": "Modern backpropagation\nModern backpropagation was first published by Seppo Linnainmaa as \"reverse mode of automatic differentiation\" (1970) for discrete connected networks of nested differentiable functions.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0222", "text": "In 1982, Paul Werbos applied backpropagation to MLPs in the way that has become standard. Werbos described how he developed backpropagation in an interview. In 1971, during his PhD work, he developed backpropagation to mathematicize Freud's \"flow of psychic energy\". He faced repeated difficulty in publishing the work, only managing in 1981", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0223", "text": ". He also claimed that \"the first practical application of back-propagation was for estimating a dynamic model to predict nationalism and social communications in 1974\" by him.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0224", "text": "Around 1982, David E. Rumelhart independently developed backpropagation and taught the algorithm to others in his research circle. He did not cite previous work as he was unaware of them. He published the algorithm first in a 1985 paper, then in a 1986 Nature paper an experimental analysis of the technique. These papers became highly cited, contributed to the popularization of backpropagation, and coincided with the resurging research interest in neural networks during the 1980s.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0225", "text": "In 1985, the method was also described by David Parker. Yann LeCun proposed an alternative form of backpropagation for neural networks in his PhD thesis in 1987.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0226", "text": "Gradient descent took a considerable amount of time to reach acceptance. Some early objections were: there were no guarantees that gradient descent could reach a global minimum, only local minimum; neurons were \"known\" by physiologists as making discrete signals (0/1), not continuous ones, and with discrete signals, there is no gradient to take. See the interview with Geoffrey Hinton, who was awarded the 2024 Nobel Prize in Physics for his contributions to the field.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0227", "text": "Early successes\nContributing to the acceptance were several applications in training neural networks via backpropagation, sometimes achieving popularity outside the research circles.\nIn 1987, NETtalk learned to convert English text into pronunciation. Sejnowski tried training it with both backpropagation and Boltzmann machine, but found the backpropagation significantly faster, so he used it for the final NETtalk. The NETtalk program became a popular success, appearing on the Today show.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0228", "text": "In 1989, Dean A. Pomerleau published ALVINN, a neural network trained to drive autonomously using backpropagation.\nThe LeNet was published in 1989 to recognize handwritten zip codes.\nIn 1992, TD-Gammon achieved top human level play in backgammon. It was a reinforcement learning agent with a neural network with two layers, trained by backpropagation.\nIn 1993, Eric Wan won an international pattern recognition contest through backpropagation.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0229", "text": "After backpropagation\nDuring the 2000s it fell out of favour, but returned in the 2010s, benefiting from cheap, powerful GPU-based computing systems. This has been especially so in speech recognition, machine vision, natural language processing, and language structure learning research (in which it has been used to explain a variety of phenomena related to first and second language learning.)", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0230", "text": "Error backpropagation has been suggested to explain human brain event-related potential (ERP) components like the N400 and P600.\nIn 2023, a backpropagation algorithm was implemented on a photonic processor by a team at Stanford University.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0231", "text": "See also\nArtificial neural network\nNeural circuit\nCatastrophic interference\nEnsemble learning\nAdaBoost\nOverfitting\nNeural backpropagation\nBackpropagation through time\nBackpropagation through structure\nThree-factor learning", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0232", "text": "Notes\nReferences\nFurther reading\nGoodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). \"6.5 Back-Propagation and Other Differentiation Algorithms\". Deep Learning. MIT Press. pp. 200\u2013220. ISBN 978-0-262-03561-3.\nNielsen, Michael A. (2015). \"How the backpropagation algorithm works\". Neural Networks and Deep Learning. Determination Press.\nMcCaffrey, James (October 2012). \"Neural Network Back-Propagation for Programmers\". MSDN Magazine.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0233", "text": "Rojas, Ra\u00fal (1996). \"The Backpropagation Algorithm\" (PDF). Neural Networks: A Systematic Introduction. Berlin: Springer. ISBN 3-540-60505-3.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0234", "text": "External links\nBackpropagation neural network tutorial at the Wikiversity\nBernacki, Mariusz; W\u0142odarczyk, Przemys\u0142aw (2004). \"Principles of training multi-layer neural network using backpropagation\".\nKarpathy, Andrej (2016). \"Lecture 4: Backpropagation, Neural Networks 1\". CS231n. Stanford University. Archived from the original on 2021-12-12 \u2013 via YouTube.\n\"What is Backpropagation Really Doing?\". 3Blue1Brown. November 3, 2017. Archived from the original on 2021-12-12 \u2013 via YouTube.", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Backpropagation_0235", "text": "Putta, Sudeep Raja (2022). \"Yet Another Derivation of Backpropagation in Matrix Form\".", "title": "Backpropagation", "url": "https://en.wikipedia.org/wiki/Backpropagation"}, {"id": "rec_Stochastic_gradient_descent_0000", "text": "Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data)", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0001", "text": ". Especially in high-dimensional optimization problems this reduces the very high computational burden, achieving faster iterations in exchange for a lower convergence rate.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0002", "text": "The basic idea behind stochastic approximation can be traced back to the Robbins\u2013Monro algorithm of the 1950s. Today, stochastic gradient descent has become an important optimization method in machine learning.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0003", "text": "Background\nBoth statistical estimation and machine learning consider the problem of minimizing an objective function that has the form of a sum:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0004", "text": "Q\n        (\n        w\n        )\n        =\n        \n          \n            1\n            n\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n        ,\n      \n    \n    {\\displaystyle Q(w)={\\frac {1}{n}}\\sum _{i=1}^{n}Q_{i}(w),}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0005", "text": "where the parameter \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n that minimizes \n  \n    \n      \n        Q\n        (\n        w\n        )\n      \n    \n    {\\displaystyle Q(w)}\n  \n is to be estimated. Each summand function \n  \n    \n      \n        \n          Q\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle Q_{i}}\n  \n is typically associated with the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0006", "text": "i\n      \n    \n    {\\displaystyle i}\n  \n-th observation in the data set (used for training).", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0007", "text": "In classical statistics, sum-minimization problems arise in least squares and in maximum-likelihood estimation (for independent observations).  The general class of estimators that arise as minimizers of sums are called M-estimators. However, in statistics, it has been long recognized that requiring even local minimization is too restrictive for some problems of maximum-likelihood estimation", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0008", "text": ". Therefore, contemporary statistical theorists often consider stationary points of the likelihood function (or zeros of its derivative, the score function, and other estimating equations).", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0009", "text": "The sum-minimization problem also arises for empirical risk minimization. There, \n  \n    \n      \n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n      \n    \n    {\\displaystyle Q_{i}(w)}\n  \n is the value of the loss function at \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n-th example, and \n  \n    \n      \n        Q\n        (\n        w\n        )\n      \n    \n    {\\displaystyle Q(w)}\n  \n is the empirical risk.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0010", "text": "is the empirical risk.\nWhen used to minimize the above function, a standard (or \"batch\") gradient descent method would perform the following iterations:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0011", "text": "w\n        :=\n        w\n        \u2212\n        \u03b7\n        \n        \u2207\n        Q\n        (\n        w\n        )\n        =\n        w\n        \u2212\n        \n          \n            \u03b7\n            n\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \u2207\n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n        .", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0012", "text": "w\n        )\n        .\n      \n    \n    {\\displaystyle w:=w-\\eta \\,\\nabla Q(w)=w-{\\frac {\\eta }{n}}\\sum _{i=1}^{n}\\nabla Q_{i}(w).}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0013", "text": "The step size is denoted by \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n (sometimes called the learning rate in machine learning) and here \"\n  \n    \n      \n        :=\n      \n    \n    {\\displaystyle :=}\n  \n\" denotes the update of a variable in the algorithm.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0014", "text": "In many cases, the summand functions have a simple form that enables inexpensive evaluations of the sum-function and the sum gradient. For example, in statistics, one-parameter exponential families allow economical function-evaluations and gradient-evaluations.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0015", "text": "However, in other cases, evaluating the sum-gradient may require expensive evaluations of the gradients from all summand functions. When the training set is enormous and no simple formulas exist, evaluating the sums of gradients becomes very expensive, because evaluating the gradient requires evaluating all the summand functions' gradients. To economize on the computational cost at every iteration, stochastic gradient descent samples a subset of summand functions at every step", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0016", "text": ". This is very effective in the case of large-scale machine learning problems.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0017", "text": "Iterative method\nIn stochastic (or \"on-line\") gradient descent, the true gradient of \n  \n    \n      \n        Q\n        (\n        w\n        )\n      \n    \n    {\\displaystyle Q(w)}\n  \n is approximated by a gradient at a single sample:\n\n  \n    \n      \n        w\n        :=\n        w\n        \u2212\n        \u03b7\n        \n        \u2207\n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n        .\n      \n    \n    {\\displaystyle w:=w-\\eta \\,\\nabla Q_{i}(w).}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0018", "text": "As the algorithm sweeps through the training set, it performs the above update for each training sample. Several passes can be made over the training set until the algorithm converges. If this is done, the data can be shuffled for each pass to prevent cycles. Typical implementations may use an adaptive learning rate so that the algorithm converges.\nIn pseudocode, stochastic gradient descent can be presented as :", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0019", "text": "A compromise between computing the true gradient and the gradient at a single sample is to compute the gradient against more than one training sample (called a \"mini-batch\") at each step.  This can perform significantly better than \"true\" stochastic gradient descent described, because the code can make use of vectorization libraries rather than computing each step separately as was first shown in  where it was called \"the bunch-mode back-propagation algorithm\"", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0020", "text": ".  It may also result in smoother convergence, as the gradient computed at each step is averaged over more training samples.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0021", "text": "The convergence of stochastic gradient descent has been analyzed using the theories of convex minimization and of stochastic approximation. Briefly, when the learning rates \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n decrease with an appropriate rate,\nand subject to relatively mild assumptions, stochastic gradient descent converges almost surely to a global minimum \nwhen the objective function is convex or pseudoconvex,", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0022", "text": "and otherwise converges almost surely to a local minimum.\nThis is in fact a consequence of the Robbins\u2013Siegmund theorem.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0023", "text": "Linear regression\nSuppose we want to fit a straight line \n  \n    \n      \n        \n          \n            \n              y\n              ^\n            \n          \n        \n        =\n        \n          w\n          \n            1\n          \n        \n        +\n        \n          w\n          \n            2\n          \n        \n        x\n      \n    \n    {\\displaystyle {\\hat {y}}=w_{1}+w_{2}x}\n  \n to a training set with observations \n  \n    \n      \n        (\n        (\n        \n          x", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0024", "text": "(\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          y\n          \n            1\n          \n        \n        )\n        ,\n        (\n        \n          x\n          \n            2\n          \n        \n        ,\n        \n          y\n          \n            2\n          \n        \n        )\n        \u2026\n        ,\n        (\n        \n          x\n          \n            n\n          \n        \n        ,\n        \n          y\n          \n            n", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0025", "text": "y\n          \n            n\n          \n        \n        )\n        )\n      \n    \n    {\\displaystyle ((x_{1},y_{1}),(x_{2},y_{2})\\ldots ,(x_{n},y_{n}))}\n  \n and corresponding estimated responses \n  \n    \n      \n        (\n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            1\n          \n        \n        ,\n        \n          \n            \n              \n                y\n                ^", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0026", "text": "y\n                ^\n              \n            \n          \n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle ({\\hat {y}}_{1},{\\hat {y}}_{2},\\ldots ,{\\hat {y}}_{n})}\n  \n using least squares. The objective function to be minimized is", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0027", "text": "Q\n        (\n        w\n        )\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          \n            (", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0028", "text": "(\n            \n              \n                \n                  \n                    \n                      y\n                      ^\n                    \n                  \n                \n                \n                  i\n                \n              \n              \u2212\n              \n                y\n                \n                  i\n                \n              \n            \n            )\n          \n          \n            2\n          \n        \n        =", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0029", "text": "=\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          \n            (\n            \n              \n                w\n                \n                  1\n                \n              \n              +\n              \n                w\n                \n                  2\n                \n              \n              \n                x\n                \n                  i", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0030", "text": "i\n                \n              \n              \u2212\n              \n                y\n                \n                  i\n                \n              \n            \n            )\n          \n          \n            2\n          \n        \n        .\n      \n    \n    {\\displaystyle Q(w)=\\sum _{i=1}^{n}Q_{i}(w)=\\sum _{i=1}^{n}\\left({\\hat {y}}_{i}-y_{i}\\right)^{2}=\\sum _{i=1}^{n}\\left(w_{1}+w_{2}x_{i}-y_{i}\\right)^{2}.}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0031", "text": "The last line in the above pseudocode for this specific problem will become:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0032", "text": "[\n            \n              \n                \n                  \n                    w\n                    \n                      1\n                    \n                  \n                \n              \n              \n                \n                  \n                    w\n                    \n                      2\n                    \n                  \n                \n              \n            \n            ]\n          \n        \n        \u2190", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0033", "text": "]\n          \n        \n        \u2190\n        \n          \n            [\n            \n              \n                \n                  \n                    w\n                    \n                      1\n                    \n                  \n                \n              \n              \n                \n                  \n                    w\n                    \n                      2\n                    \n                  \n                \n              \n            \n            ]", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0034", "text": "]\n          \n        \n        \u2212\n        \u03b7\n        \n          \n            [\n            \n              \n                \n                  \n                    \n                      \u2202\n                      \n                        \u2202\n                        \n                          w\n                          \n                            1", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0035", "text": "(\n                  \n                    w\n                    \n                      1\n                    \n                  \n                  +\n                  \n                    w\n                    \n                      2\n                    \n                  \n                  \n                    x\n                    \n                      i\n                    \n                  \n                  \u2212", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0036", "text": "\u2212\n                  \n                    y\n                    \n                      i\n                    \n                  \n                  \n                    )\n                    \n                      2\n                    \n                  \n                \n              \n              \n                \n                  \n                    \n                      \u2202\n                      \n                        \u2202", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0037", "text": "w\n                          \n                            2\n                          \n                        \n                      \n                    \n                  \n                  (\n                  \n                    w\n                    \n                      1\n                    \n                  \n                  +\n                  \n                    w\n                    \n                      2", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0038", "text": "2\n                    \n                  \n                  \n                    x\n                    \n                      i\n                    \n                  \n                  \u2212\n                  \n                    y\n                    \n                      i\n                    \n                  \n                  \n                    )\n                    \n                      2", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0039", "text": "]\n          \n        \n        =\n        \n          \n            [\n            \n              \n                \n                  \n                    w\n                    \n                      1\n                    \n                  \n                \n              \n              \n                \n                  \n                    w\n                    \n                      2", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0040", "text": "]\n          \n        \n        \u2212\n        \u03b7\n        \n          \n            [\n            \n              \n                \n                  2\n                  (\n                  \n                    w\n                    \n                      1\n                    \n                  \n                  +\n                  \n                    w\n                    \n                      2", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0041", "text": "2\n                    \n                  \n                  \n                    x\n                    \n                      i\n                    \n                  \n                  \u2212\n                  \n                    y\n                    \n                      i\n                    \n                  \n                  )\n                \n              \n              \n                \n                  2", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0042", "text": "2\n                  \n                    x\n                    \n                      i\n                    \n                  \n                  (\n                  \n                    w\n                    \n                      1\n                    \n                  \n                  +\n                  \n                    w\n                    \n                      2\n                    \n                  \n                  \n                    x", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0043", "text": "x\n                    \n                      i\n                    \n                  \n                  \u2212\n                  \n                    y\n                    \n                      i\n                    \n                  \n                  )\n                \n              \n            \n            ]\n          \n        \n        .", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0044", "text": ".\n      \n    \n    {\\displaystyle {\\begin{bmatrix}w_{1}\\\\w_{2}\\end{bmatrix}}\\leftarrow {\\begin{bmatrix}w_{1}\\\\w_{2}\\end{bmatrix}}-\\eta {\\begin{bmatrix}{\\frac {\\partial }{\\partial w_{1}}}(w_{1}+w_{2}x_{i}-y_{i})^{2}\\\\{\\frac {\\partial }{\\partial w_{2}}}(w_{1}+w_{2}x_{i}-y_{i})^{2}\\end{bmatrix}}={\\begin{bmatrix}w_{1}\\\\w_{2}\\end{bmatrix}}-\\eta {\\begin{bmatrix}2(w_{1}+w_{2}x_{i}-y_{i})\\\\2x_{i}(w_{1}+w_{2}x_{i}-y_{i})\\end{bmatrix}}.}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0045", "text": "Note that in each iteration or update step, the gradient is only evaluated at a single \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n. This is the key difference between stochastic gradient descent and batched gradient descent.\nIn general, given a linear regression \n  \n    \n      \n        \n          \n            \n              y\n              ^\n            \n          \n        \n        =\n        \n          \u2211", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0046", "text": "=\n        \n          \u2211\n          \n            k\n            \u2208\n            1\n            :\n            m\n          \n        \n        \n          w\n          \n            k\n          \n        \n        \n          x\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle {\\hat {y}}=\\sum _{k\\in 1:m}w_{k}x_{k}}\n  \n problem, stochastic gradient descent behaves differently when \n  \n    \n      \n        m\n        <\n        n\n      \n    \n    {\\displaystyle m<n}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0047", "text": "n\n      \n    \n    {\\displaystyle m<n}\n  \n (underparameterized) and \n  \n    \n      \n        m\n        \u2265\n        n\n      \n    \n    {\\displaystyle m\\geq n}\n  \n (overparameterized). In the overparameterized case, stochastic gradient descent converges to \n  \n    \n      \n        arg\n        \u2061\n        \n          min\n          \n            w\n            :\n            \n              w\n              \n                T\n              \n            \n            \n              x", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0048", "text": "x\n              \n                k\n              \n            \n            =\n            \n              y\n              \n                k\n              \n            \n            \u2200\n            k\n            \u2208\n            1\n            :\n            n\n          \n        \n        \u2016\n        w\n        \u2212\n        \n          w\n          \n            0\n          \n        \n        \u2016\n      \n    \n    {\\displaystyle \\arg \\min _{w:w^{T}x_{k}=y_{k}\\forall k\\in 1:n}\\|w-w_{0}\\|}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0049", "text": ". That is, SGD converges to the interpolation solution with minimum distance from the starting \n  \n    \n      \n        \n          w\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle w_{0}}\n  \n. This is true even when the learning rate remains constant. In the underparameterized case, SGD does not converge if learning rate remains constant.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0050", "text": "History", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0051", "text": "In 1951, Herbert Robbins and Sutton Monro introduced the earliest stochastic approximation methods, preceding stochastic gradient descent. Building on this work one year later, Jack Kiefer and Jacob Wolfowitz published an optimization algorithm very close to stochastic gradient descent, using central differences as an approximation of the gradient", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0052", "text": ". Later in the 1950s, Frank Rosenblatt used SGD to optimize his perceptron model, demonstrating the first applicability of stochastic gradient descent to neural networks.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0053", "text": "Backpropagation was first described in 1986, with stochastic gradient descent being used to efficiently optimize parameters across neural networks with multiple hidden layers. Soon after, another improvement was developed: mini-batch gradient descent, where small batches of data are substituted for single samples. In 1997, the practical performance benefits from vectorization achievable with such small batches were first explored, paving the way for efficient optimization in machine learning", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0054", "text": ". As of 2023, this mini-batch approach remains the norm for training neural networks, balancing the benefits of stochastic gradient descent with gradient descent.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0055", "text": "By the 1980s, momentum had already been introduced, and was added to SGD optimization techniques in 1986. However, these optimization techniques assumed constant hyperparameters, i.e. a fixed learning rate and momentum parameter. In the 2010s, adaptive approaches to applying SGD with a per-parameter learning rate were introduced with AdaGrad (for \"Adaptive Gradient\") in 2011 and RMSprop (for \"Root Mean Square Propagation\") in 2012", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0056", "text": ". In 2014, Adam (for \"Adaptive Moment Estimation\") was published, applying the adaptive approaches of RMSprop to momentum; many improvements and branches of Adam were then developed such as Adadelta, Adagrad, AdamW, and Adamax.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0057", "text": "Within machine learning, approaches to optimization in 2023 are dominated by Adam-derived optimizers, TensorFlow and PyTorch, by far the most popular machine learning libraries, as of 2023 largely only include Adam-derived optimizers, as well as predecessors to Adam such as RMSprop and classic SGD. PyTorch also partially supports limited-memory BFGS, a line-search method, but only for single-device setups without parameter groups.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0058", "text": "Notable applications\nStochastic gradient descent is a popular algorithm for training a wide range of models in machine learning, including (linear) support vector machines, logistic regression (see, e.g., Vowpal Wabbit) and graphical models. When combined with the backpropagation algorithm, it is the de facto standard algorithm for training artificial neural networks. Its use has been also reported in the Geophysics community, specifically to applications of Full Waveform Inversion (FWI).", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0059", "text": "Stochastic gradient descent competes with the L-BFGS algorithm, which is also widely used. Stochastic gradient descent has been used since at least 1960 for training linear regression models, originally under the name ADALINE.\nAnother stochastic gradient descent algorithm is the least mean squares (LMS) adaptive filter.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0060", "text": "Extensions and variants", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0061", "text": "Many improvements on the basic stochastic gradient descent algorithm have been proposed and used. In particular, in machine learning, the need to set a learning rate (step size) has been recognized as problematic. Setting this parameter too high can cause the algorithm to diverge; setting it too low makes it slow to converge", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0062", "text": ". A conceptually simple extension of stochastic gradient descent makes the learning rate a decreasing function \u03b7t of the iteration number t, giving a learning rate schedule, so that the first iterations cause large changes in the parameters, while the later ones do only fine-tuning. Such schedules have been known since the work of MacQueen on k-means clustering. Practical guidance on choosing the step size in several variants of SGD is given by Spall.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0063", "text": "Implicit updates (ISGD)\nAs mentioned earlier, classical stochastic gradient descent is generally sensitive to learning rate \u03b7. Fast convergence requires large learning rates but this may induce numerical instability. The problem can be largely solved by considering implicit updates whereby the stochastic gradient is evaluated at the next iterate rather than the current one:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0064", "text": "w\n          \n            new\n          \n        \n        :=\n        \n          w\n          \n            old\n          \n        \n        \u2212\n        \u03b7\n        \n        \u2207\n        \n          Q\n          \n            i\n          \n        \n        (\n        \n          w\n          \n            new\n          \n        \n        )\n        .\n      \n    \n    {\\displaystyle w^{\\text{new}}:=w^{\\text{old}}-\\eta \\,\\nabla Q_{i}(w^{\\text{new}}).}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0065", "text": "This equation is implicit since \n  \n    \n      \n        \n          w\n          \n            new\n          \n        \n      \n    \n    {\\displaystyle w^{\\text{new}}}\n  \n appears on both sides of the equation. It is a stochastic form of the proximal gradient method since the update\ncan also be written as:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0066", "text": "w\n          \n            new\n          \n        \n        :=\n        arg\n        \u2061\n        \n          min\n          \n            w\n          \n        \n        \n          {\n          \n            \n              Q\n              \n                i\n              \n            \n            (\n            w\n            )\n            +\n            \n              \n                1\n                \n                  2\n                  \u03b7", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0067", "text": "\u2016\n                \n                  w\n                  \u2212\n                  \n                    w\n                    \n                      old\n                    \n                  \n                \n                \u2016\n              \n              \n                2\n              \n            \n          \n          }\n        \n        .", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0068", "text": "}\n        \n        .\n      \n    \n    {\\displaystyle w^{\\text{new}}:=\\arg \\min _{w}\\left\\{Q_{i}(w)+{\\frac {1}{2\\eta }}\\left\\|w-w^{\\text{old}}\\right\\|^{2}\\right\\}.}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0069", "text": "As an example, \nconsider least squares with features \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            p\n          \n        \n      \n    \n    {\\displaystyle x_{1},\\ldots ,x_{n}\\in \\mathbb {R} ^{p}}\n  \n and observations", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0070", "text": "y\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          y\n          \n            n\n          \n        \n        \u2208\n        \n          R\n        \n      \n    \n    {\\displaystyle y_{1},\\ldots ,y_{n}\\in \\mathbb {R} }\n  \n. We wish to solve:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0071", "text": "min\n          \n            w\n          \n        \n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          \n            (\n            \n              \n                y\n                \n                  j\n                \n              \n              \u2212\n              \n                x\n                \n                  j\n                \n                \u2032", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0072", "text": "\u2032\n              \n              w\n            \n            )\n          \n          \n            2\n          \n        \n        ,\n      \n    \n    {\\displaystyle \\min _{w}\\sum _{j=1}^{n}\\left(y_{j}-x_{j}'w\\right)^{2},}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0073", "text": "where \n  \n    \n      \n        \n          x\n          \n            j\n          \n          \u2032\n        \n        w\n        =\n        \n          x\n          \n            j\n            1\n          \n        \n        \n          w\n          \n            1\n          \n        \n        +\n        \n          x\n          \n            j\n            ,\n            2\n          \n        \n        \n          w\n          \n            2\n          \n        \n        +\n        .\n        .\n        .\n        +", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0074", "text": ".\n        .\n        .\n        +\n        \n          x\n          \n            j\n            ,\n            p\n          \n        \n        \n          w\n          \n            p\n          \n        \n      \n    \n    {\\displaystyle x_{j}'w=x_{j1}w_{1}+x_{j,2}w_{2}+...+x_{j,p}w_{p}}\n  \n  indicates the inner product.\nNote that \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0075", "text": "x\n      \n    \n    {\\displaystyle x}\n  \n could have \"1\" as the first element to include an intercept. Classical stochastic gradient descent proceeds as follows:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0076", "text": "w\n          \n            new\n          \n        \n        =\n        \n          w\n          \n            old\n          \n        \n        +\n        \u03b7\n        \n          (\n          \n            \n              y\n              \n                i\n              \n            \n            \u2212\n            \n              x\n              \n                i\n              \n              \u2032\n            \n            \n              w\n              \n                old", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0077", "text": "old\n              \n            \n          \n          )\n        \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle w^{\\text{new}}=w^{\\text{old}}+\\eta \\left(y_{i}-x_{i}'w^{\\text{old}}\\right)x_{i}}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0078", "text": "where \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n is uniformly sampled between 1 and \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n. Although theoretical convergence of this procedure happens under relatively mild assumptions, in practice the procedure can be quite unstable. In particular, when \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n is misspecified so that \n  \n    \n      \n        I\n        \u2212\n        \u03b7\n        \n          x", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0079", "text": "\u2212\n        \u03b7\n        \n          x\n          \n            i\n          \n        \n        \n          x\n          \n            i\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle I-\\eta x_{i}x_{i}'}\n  \n has large absolute eigenvalues with high probability, the procedure may diverge numerically within a few iterations. In contrast, implicit stochastic gradient descent (shortened as ISGD) can be solved in closed-form as:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0080", "text": "w\n          \n            new\n          \n        \n        =\n        \n          w\n          \n            old\n          \n        \n        +\n        \n          \n            \u03b7\n            \n              1\n              +\n              \u03b7\n              \n                \n                  \u2016\n                  \n                    x\n                    \n                      i\n                    \n                  \n                  \u2016", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0081", "text": "2\n                \n              \n            \n          \n        \n        \n          (\n          \n            \n              y\n              \n                i\n              \n            \n            \u2212\n            \n              x\n              \n                i\n              \n              \u2032\n            \n            \n              w\n              \n                old\n              \n            \n          \n          )", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0082", "text": ")\n        \n        \n          x\n          \n            i\n          \n        \n        .\n      \n    \n    {\\displaystyle w^{\\text{new}}=w^{\\text{old}}+{\\frac {\\eta }{1+\\eta \\left\\|x_{i}\\right\\|^{2}}}\\left(y_{i}-x_{i}'w^{\\text{old}}\\right)x_{i}.}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0083", "text": "This procedure will remain numerically stable virtually for all \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n as the learning rate is now normalized. Such comparison between classical and implicit stochastic gradient descent in the least squares problem is very similar to the comparison between least mean squares (LMS) and \nnormalized least mean squares filter (NLMS).", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0084", "text": "normalized least mean squares filter (NLMS).\nEven though a closed-form solution for ISGD is only possible in least squares, the procedure can be efficiently implemented in a wide range of models. Specifically, suppose that \n  \n    \n      \n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n      \n    \n    {\\displaystyle Q_{i}(w)}\n  \n depends on \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0085", "text": "w\n      \n    \n    {\\displaystyle w}\n  \n only through a linear combination with features \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n, so that we can write \n  \n    \n      \n        \n          \u2207\n          \n            w\n          \n        \n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n        =\n        \u2212\n        q\n        (\n        \n          x", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0086", "text": "(\n        \n          x\n          \n            i\n          \n          \u2032\n        \n        w\n        )\n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\nabla _{w}Q_{i}(w)=-q(x_{i}'w)x_{i}}\n  \n, where \n  \n    \n      \n        q\n        (\n        )\n        \u2208\n        \n          R\n        \n      \n    \n    {\\displaystyle q()\\in \\mathbb {R} }\n  \n may depend on \n  \n    \n      \n        \n          x\n          \n            i", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0087", "text": "i\n          \n        \n        ,\n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i},y_{i}}\n  \n as well but not on \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n except through \n  \n    \n      \n        \n          x\n          \n            i\n          \n          \u2032\n        \n        w\n      \n    \n    {\\displaystyle x_{i}'w}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0088", "text": "{\\displaystyle x_{i}'w}\n  \n. Least squares obeys this rule, and so does logistic regression, and most generalized linear models. For instance, in least squares, \n  \n    \n      \n        q\n        (\n        \n          x\n          \n            i\n          \n          \u2032\n        \n        w\n        )\n        =\n        \n          y\n          \n            i\n          \n        \n        \u2212\n        \n          x\n          \n            i\n          \n          \u2032\n        \n        w", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0089", "text": "\u2032\n        \n        w\n      \n    \n    {\\displaystyle q(x_{i}'w)=y_{i}-x_{i}'w}\n  \n, and in logistic regression \n  \n    \n      \n        q\n        (\n        \n          x\n          \n            i\n          \n          \u2032\n        \n        w\n        )\n        =\n        \n          y\n          \n            i\n          \n        \n        \u2212\n        S\n        (\n        \n          x\n          \n            i\n          \n          \u2032\n        \n        w\n        )", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0090", "text": "w\n        )\n      \n    \n    {\\displaystyle q(x_{i}'w)=y_{i}-S(x_{i}'w)}\n  \n, where \n  \n    \n      \n        S\n        (\n        u\n        )\n        =\n        \n          e\n          \n            u\n          \n        \n        \n          /\n        \n        (\n        1\n        +\n        \n          e\n          \n            u\n          \n        \n        )\n      \n    \n    {\\displaystyle S(u)=e^{u}/(1+e^{u})}\n  \n is the logistic function. In Poisson regression, \n  \n    \n      \n        q", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0091", "text": "q\n        (\n        \n          x\n          \n            i\n          \n          \u2032\n        \n        w\n        )\n        =\n        \n          y\n          \n            i\n          \n        \n        \u2212\n        \n          e\n          \n            \n              x\n              \n                i\n              \n              \u2032\n            \n            w\n          \n        \n      \n    \n    {\\displaystyle q(x_{i}'w)=y_{i}-e^{x_{i}'w}}\n  \n, and so on.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0092", "text": ", and so on.\nIn such settings, ISGD is simply implemented as follows. Let \n  \n    \n      \n        f\n        (\n        \u03be\n        )\n        =\n        \u03b7\n        q\n        (\n        \n          x\n          \n            i\n          \n          \u2032\n        \n        \n          w\n          \n            old\n          \n        \n        +\n        \u03be\n        \u2016\n        \n          x\n          \n            i\n          \n        \n        \n          \u2016\n          \n            2\n          \n        \n        )", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0093", "text": ")\n      \n    \n    {\\displaystyle f(\\xi )=\\eta q(x_{i}'w^{\\text{old}}+\\xi \\|x_{i}\\|^{2})}\n  \n, where \n  \n    \n      \n        \u03be\n      \n    \n    {\\displaystyle \\xi }\n  \n is scalar.\nThen, ISGD is equivalent to:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0094", "text": "w\n          \n            new\n          \n        \n        =\n        \n          w\n          \n            old\n          \n        \n        +\n        \n          \u03be\n          \n            \u2217\n          \n        \n        \n          x\n          \n            i\n          \n        \n        ,\n         \n        \n          where\n        \n         \n        \n          \u03be\n          \n            \u2217\n          \n        \n        =\n        f\n        (\n        \n          \u03be", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0095", "text": "(\n        \n          \u03be\n          \n            \u2217\n          \n        \n        )\n        .\n      \n    \n    {\\displaystyle w^{\\text{new}}=w^{\\text{old}}+\\xi ^{\\ast }x_{i},~{\\text{where}}~\\xi ^{\\ast }=f(\\xi ^{\\ast }).}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0096", "text": "The scaling factor \n  \n    \n      \n        \n          \u03be\n          \n            \u2217\n          \n        \n        \u2208\n        \n          R\n        \n      \n    \n    {\\displaystyle \\xi ^{\\ast }\\in \\mathbb {R} }\n  \n can be found through the bisection method since in most regular models, such as the aforementioned generalized linear models, function \n  \n    \n      \n        q\n        (\n        )\n      \n    \n    {\\displaystyle q()}\n  \n is decreasing, and thus the search bounds for", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0097", "text": "\u03be\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle \\xi ^{\\ast }}\n  \n are \n  \n    \n      \n        [\n        min\n        (\n        0\n        ,\n        f\n        (\n        0\n        )\n        )\n        ,\n        max\n        (\n        0\n        ,\n        f\n        (\n        0\n        )\n        )\n        ]\n      \n    \n    {\\displaystyle [\\min(0,f(0)),\\max(0,f(0))]}\n  \n.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0098", "text": "Momentum\nFurther proposals include the momentum method or the heavy ball method, which in ML context appeared in Rumelhart, Hinton and Williams' paper on backpropagation learning and borrowed the idea from Soviet mathematician Boris Polyak's 1964 article on solving functional equations. Stochastic gradient descent with momentum remembers the update \u0394w at each iteration, and determines the next update as a linear combination of the gradient and the previous update:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0099", "text": "\u0394\n        w\n        :=\n        \u03b1\n        \u0394\n        w\n        \u2212\n        \u03b7\n        \n        \u2207\n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n      \n    \n    {\\displaystyle \\Delta w:=\\alpha \\Delta w-\\eta \\,\\nabla Q_{i}(w)}\n  \n\n  \n    \n      \n        w\n        :=\n        w\n        +\n        \u0394\n        w\n      \n    \n    {\\displaystyle w:=w+\\Delta w}\n  \n\nthat leads to:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0100", "text": "that leads to:\n\n  \n    \n      \n        w\n        :=\n        w\n        \u2212\n        \u03b7\n        \n        \u2207\n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n        +\n        \u03b1\n        \u0394\n        w\n      \n    \n    {\\displaystyle w:=w-\\eta \\,\\nabla Q_{i}(w)+\\alpha \\Delta w}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0101", "text": "where the parameter \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n which minimizes \n  \n    \n      \n        Q\n        (\n        w\n        )\n      \n    \n    {\\displaystyle Q(w)}\n  \n is to be estimated, \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }\n  \n is a step size (sometimes called the learning rate in machine learning) and \n  \n    \n      \n        \u03b1\n      \n    \n    {\\displaystyle \\alpha }", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0102", "text": "{\\displaystyle \\alpha }\n  \n is an exponential decay factor between 0 and 1 that determines the relative contribution of the current gradient and earlier gradients to the weight change.\nThe name momentum stems from an analogy to momentum in physics: the weight vector \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0103", "text": "w\n      \n    \n    {\\displaystyle w}\n  \n, thought of as a particle traveling through parameter space, incurs acceleration from the gradient of the loss (\"force\"). Unlike in classical stochastic gradient descent, it tends to keep traveling in the same direction, preventing oscillations. Momentum has been used successfully by computer scientists in the training of artificial neural networks for several decades.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0104", "text": "The momentum method  is closely related to underdamped Langevin dynamics, and may be combined with simulated annealing.\nIn mid-1980s the method was modified by Yurii Nesterov to use the gradient predicted at the next point, and the resulting so-called Nesterov Accelerated Gradient was sometimes used in ML in the 2010s.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0105", "text": "Averaging\nAveraged stochastic gradient descent, invented independently by Ruppert and Polyak in the late 1980s, is ordinary stochastic gradient descent that records an average of its parameter vector over time. That is, the update is the same as for ordinary stochastic gradient descent, but the algorithm also keeps track of", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0106", "text": "w\n              \u00af\n            \n          \n        \n        =\n        \n          \n            1\n            t\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            0\n          \n          \n            t\n            \u2212\n            1\n          \n        \n        \n          w\n          \n            i\n          \n        \n        .", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0107", "text": ".\n      \n    \n    {\\displaystyle {\\bar {w}}={\\frac {1}{t}}\\sum _{i=0}^{t-1}w_{i}.}\n  \nWhen optimization is done, this averaged parameter vector takes the place of w.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0108", "text": "AdaGrad", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0109", "text": "AdaGrad (for adaptive gradient algorithm) is a modified stochastic gradient descent algorithm with per-parameter learning rate, first published in 2011. Informally, this increases the learning rate for sparser parameters and decreases the learning rate for ones that are less sparse. This strategy often improves convergence performance over standard stochastic gradient descent in settings where data is sparse and sparse parameters are more informative", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0110", "text": ". Examples of such applications include natural language processing and image recognition.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0111", "text": "It still has a base learning rate \u03b7, but this is multiplied with the elements of a vector {Gj,j}  which is the diagonal of the outer product matrix", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0112", "text": "G\n        =\n        \n          \u2211\n          \n            \u03c4\n            =\n            1\n          \n          \n            t\n          \n        \n        \n          g\n          \n            \u03c4\n          \n        \n        \n          g\n          \n            \u03c4\n          \n          \n            \n              T\n            \n          \n        \n      \n    \n    {\\displaystyle G=\\sum _{\\tau =1}^{t}g_{\\tau }g_{\\tau }^{\\mathsf {T}}}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0113", "text": "where \n  \n    \n      \n        \n          g\n          \n            \u03c4\n          \n        \n        =\n        \u2207\n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n      \n    \n    {\\displaystyle g_{\\tau }=\\nabla Q_{i}(w)}\n  \n, the gradient, at iteration \u03c4. The diagonal is given by", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0114", "text": "G\n          \n            j\n            ,\n            j\n          \n        \n        =\n        \n          \u2211\n          \n            \u03c4\n            =\n            1\n          \n          \n            t\n          \n        \n        \n          g\n          \n            \u03c4\n            ,\n            j\n          \n          \n            2\n          \n        \n        .\n      \n    \n    {\\displaystyle G_{j,j}=\\sum _{\\tau =1}^{t}g_{\\tau ,j}^{2}.}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0115", "text": "This vector essentially stores a historical sum of gradient squares by dimension and is updated after every iteration. The formula for an update is now", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0116", "text": "w\n        :=\n        w\n        \u2212\n        \u03b7\n        \n        \n          d\n          i\n          a\n          g\n        \n        (\n        G\n        \n          )\n          \n            \u2212\n            \n              \n                1\n                2\n              \n            \n          \n        \n        \u2299\n        g\n      \n    \n    {\\displaystyle w:=w-\\eta \\,\\mathrm {diag} (G)^{-{\\frac {1}{2}}}\\odot g}\n  \n\nor, written as per-parameter updates,", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0117", "text": "w\n          \n            j\n          \n        \n        :=\n        \n          w\n          \n            j\n          \n        \n        \u2212\n        \n          \n            \u03b7\n            \n              \n                G\n                \n                  j\n                  ,\n                  j\n                \n              \n            \n          \n        \n        \n          g\n          \n            j\n          \n        \n        .", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0118", "text": ".\n      \n    \n    {\\displaystyle w_{j}:=w_{j}-{\\frac {\\eta }{\\sqrt {G_{j,j}}}}g_{j}.}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0119", "text": "Each {G(i,i)}  gives rise to a scaling factor for the learning rate that applies to a single parameter wi. Since the denominator in this factor, \n  \n    \n      \n        \n          \n            \n              G\n              \n                i\n              \n            \n          \n        \n        =\n        \n          \n            \n              \u2211\n              \n                \u03c4\n                =\n                1\n              \n              \n                t", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0120", "text": "t\n              \n            \n            \n              g\n              \n                \u03c4\n              \n              \n                2\n              \n            \n          \n        \n      \n    \n    {\\textstyle {\\sqrt {G_{i}}}={\\sqrt {\\sum _{\\tau =1}^{t}g_{\\tau }^{2}}}}\n  \n is the \u21132 norm of previous derivatives, extreme parameter updates get dampened, while parameters that get few or small updates receive higher learning rates.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0121", "text": "While designed for convex problems, AdaGrad has been successfully applied to non-convex optimization.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0122", "text": "RMSProp\nRMSProp (for Root Mean Square Propagation) is a method invented in 2012 by James Martens and Ilya Sutskever, at the time both PhD students in Geoffrey Hinton's group, in which the learning rate is, like in Adagrad, adapted for each of the parameters. The idea is to divide the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight. Unusually, it was not published in an article but merely described in a Coursera lecture.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0123", "text": "So, first the running average is calculated in terms of means square,", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0124", "text": "v\n        (\n        w\n        ,\n        t\n        )\n        :=\n        \u03b3\n        v\n        (\n        w\n        ,\n        t\n        \u2212\n        1\n        )\n        +\n        \n          (\n          \n            1\n            \u2212\n            \u03b3\n          \n          )\n        \n        \n          \n            (\n            \n              \u2207\n              \n                Q\n                \n                  i\n                \n              \n              (\n              w", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0125", "text": "(\n              w\n              )\n            \n            )\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle v(w,t):=\\gamma v(w,t-1)+\\left(1-\\gamma \\right)\\left(\\nabla Q_{i}(w)\\right)^{2}}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0126", "text": "where, \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n is the forgetting factor. The concept of storing the historical gradient as sum of squares is borrowed from Adagrad, but \"forgetting\" is introduced to solve Adagrad's diminishing learning rates in non-convex problems by gradually decreasing the influence of old data.\nAnd the parameters are updated as,", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0127", "text": "w\n        :=\n        w\n        \u2212\n        \n          \n            \u03b7\n            \n              v\n              (\n              w\n              ,\n              t\n              )\n            \n          \n        \n        \u2207\n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n      \n    \n    {\\displaystyle w:=w-{\\frac {\\eta }{\\sqrt {v(w,t)}}}\\nabla Q_{i}(w)}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0128", "text": "RMSProp has shown good adaptation of learning rate in different applications. RMSProp can be seen as a generalization of Rprop and is capable to work with mini-batches as well opposed to only full-batches.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0129", "text": "Adam\nAdam (short for Adaptive Moment Estimation) is a 2014 update to the RMSProp optimizer combining it with the main feature of the Momentum method. In this optimization algorithm, running averages with exponential forgetting of both the gradients and the second moments of the gradients are used. Given parameters \n  \n    \n      \n        \n          w\n          \n            (\n            t\n            )\n          \n        \n      \n    \n    {\\displaystyle w^{(t)}}\n  \n and a loss function", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0130", "text": "and a loss function \n  \n    \n      \n        \n          L\n          \n            (\n            t\n            )\n          \n        \n      \n    \n    {\\displaystyle L^{(t)}}\n  \n, where \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n indexes the current training iteration (indexed at \n  \n    \n      \n        1\n      \n    \n    {\\displaystyle 1}\n  \n), Adam's parameter update is given by:", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0131", "text": "m\n          \n            w\n          \n          \n            (\n            t\n            )\n          \n        \n        :=\n        \n          \u03b2\n          \n            1\n          \n        \n        \n          m\n          \n            w\n          \n          \n            (\n            t\n            \u2212\n            1\n            )\n          \n        \n        +\n        \n          (\n          \n            1\n            \u2212\n            \n              \u03b2", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0132", "text": "\u03b2\n              \n                1\n              \n            \n          \n          )\n        \n        \n          \u2207\n          \n            w\n          \n        \n        \n          L\n          \n            (\n            t\n            \u2212\n            1\n            )\n          \n        \n      \n    \n    {\\displaystyle m_{w}^{(t)}:=\\beta _{1}m_{w}^{(t-1)}+\\left(1-\\beta _{1}\\right)\\nabla _{w}L^{(t-1)}}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0133", "text": "v\n          \n            w\n          \n          \n            (\n            t\n            )\n          \n        \n        :=\n        \n          \u03b2\n          \n            2\n          \n        \n        \n          v\n          \n            w\n          \n          \n            (\n            t\n            \u2212\n            1\n            )\n          \n        \n        +\n        \n          (\n          \n            1\n            \u2212\n            \n              \u03b2", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0134", "text": "\u03b2\n              \n                2\n              \n            \n          \n          )\n        \n        \n          \n            (\n            \n              \n                \u2207\n                \n                  w\n                \n              \n              \n                L\n                \n                  (\n                  t\n                  \u2212\n                  1\n                  )\n                \n              \n            \n            )", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0135", "text": ")\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle v_{w}^{(t)}:=\\beta _{2}v_{w}^{(t-1)}+\\left(1-\\beta _{2}\\right)\\left(\\nabla _{w}L^{(t-1)}\\right)^{2}}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0136", "text": "m\n                ^\n              \n            \n          \n          \n            w\n          \n          \n            (\n            t\n            )\n          \n        \n        =\n        \n          \n            \n              m\n              \n                w\n              \n              \n                (\n                t\n                )\n              \n            \n            \n              1\n              \u2212", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0137", "text": "1\n              \u2212\n              \n                \u03b2\n                \n                  1\n                \n                \n                  t\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {m}}_{w}^{(t)}={\\frac {m_{w}^{(t)}}{1-\\beta _{1}^{t}}}}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0138", "text": "v\n                ^\n              \n            \n          \n          \n            w\n          \n          \n            (\n            t\n            )\n          \n        \n        =\n        \n          \n            \n              v\n              \n                w\n              \n              \n                (\n                t\n                )\n              \n            \n            \n              1\n              \u2212", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0139", "text": "1\n              \u2212\n              \n                \u03b2\n                \n                  2\n                \n                \n                  t\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {v}}_{w}^{(t)}={\\frac {v_{w}^{(t)}}{1-\\beta _{2}^{t}}}}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0140", "text": "w\n          \n            (\n            t\n            )\n          \n        \n        :=\n        \n          w\n          \n            (\n            t\n            \u2212\n            1\n            )\n          \n        \n        \u2212\n        \u03b7\n        \n          \n            \n              \n                \n                  \n                    m\n                    ^\n                  \n                \n              \n              \n                w", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0141", "text": "w\n              \n              \n                (\n                t\n                )\n              \n            \n            \n              \n                \n                  \n                    \n                      \n                        \n                          v\n                          ^\n                        \n                      \n                    \n                    \n                      w", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0142", "text": "(\n                      t\n                      )\n                    \n                  \n                \n              \n              +\n              \u03b5\n            \n          \n        \n      \n    \n    {\\displaystyle w^{(t)}:=w^{(t-1)}-\\eta {\\frac {{\\hat {m}}_{w}^{(t)}}{{\\sqrt {{\\hat {v}}_{w}^{(t)}}}+\\varepsilon }}}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0143", "text": "where \n  \n    \n      \n        \u03b5\n      \n    \n    {\\displaystyle \\varepsilon }\n  \n is a small scalar (e.g. \n  \n    \n      \n        \n          10\n          \n            \u2212\n            8\n          \n        \n      \n    \n    {\\displaystyle 10^{-8}}\n  \n) used to prevent division by 0, and \n  \n    \n      \n        \n          \u03b2\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\beta _{1}}\n  \n (e.g. 0.9) and \n  \n    \n      \n        \n          \u03b2\n          \n            2", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0144", "text": "\u03b2\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\beta _{2}}\n  \n (e.g. 0.999) are the forgetting factors for gradients and second moments of gradients, respectively. Squaring and square-rooting is done element-wise.\nAs the exponential moving averages of the gradient \n  \n    \n      \n        \n          m\n          \n            w\n          \n          \n            (\n            t\n            )\n          \n        \n      \n    \n    {\\displaystyle m_{w}^{(t)}}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0145", "text": "{\\displaystyle m_{w}^{(t)}}\n  \n and the squared gradient \n  \n    \n      \n        \n          v\n          \n            w\n          \n          \n            (\n            t\n            )\n          \n        \n      \n    \n    {\\displaystyle v_{w}^{(t)}}\n  \n are initialized with a vector of 0's, there would be a bias towards zero in the first training iterations. A factor \n  \n    \n      \n        \n          \n            \n              1\n              \n                1\n                \u2212", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0146", "text": "1\n                \u2212\n                \n                  \u03b2\n                  \n                    1\n                    \n                      /\n                    \n                    2\n                  \n                  \n                    t\n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\tfrac {1}{1-\\beta _{1/2}^{t}}}}\n  \n is introduced to compensate this bias and get better estimates", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0147", "text": "m\n                ^\n              \n            \n          \n          \n            w\n          \n          \n            (\n            t\n            )\n          \n        \n      \n    \n    {\\displaystyle {\\hat {m}}_{w}^{(t)}}\n  \n and \n  \n    \n      \n        \n          \n            \n              \n                v\n                ^\n              \n            \n          \n          \n            w", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0148", "text": "w\n          \n          \n            (\n            t\n            )\n          \n        \n      \n    \n    {\\displaystyle {\\hat {v}}_{w}^{(t)}}\n  \n.\nThe initial proof establishing the convergence of Adam was incomplete, and subsequent analysis has revealed that Adam does not converge for all convex objectives. Despite this, Adam continues to be used due to its strong performance in practice.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0149", "text": "Variants\nThe popularity of Adam inspired many variants and enhancements. Some examples include:\n\nNesterov-enhanced gradients: NAdam, FASFA\nvarying interpretations of second-order information: Powerpropagation and AdaSqrt.\nUsing infinity norm: AdaMax\nAMSGrad, which improves convergence over Adam by using maximum of past squared gradients instead of the exponential average. AdamX further improves convergence over AMSGrad.\nAdamW, which improves the weight decay.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0150", "text": "Sign-based stochastic gradient descent", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0151", "text": "Even though sign-based optimization goes back to the aforementioned Rprop, in 2018 researchers tried to simplify Adam by removing the magnitude of the stochastic gradient from being taken into account and only considering its sign. This results in a significantly lower communication cost of transferring gradients from workers to the parameter server. In this sense, it serves to better compress the gradient information, while having comparable convergence to standard SGD.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0152", "text": "Backtracking line search", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0153", "text": "Backtracking line search is another variant of gradient descent. All of the below are sourced from the mentioned link. It is based on a condition known as the Armijo\u2013Goldstein condition. Both methods allow learning rates to change at each iteration; however, the manner of the change is different. Backtracking line search uses function evaluations to check Armijo's condition, and in principle the loop in the algorithm for determining the learning rates can be long and unknown in advance", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0154", "text": ". Adaptive SGD does not need a loop in determining learning rates. On the other hand, adaptive SGD does not guarantee the \"descent property\" \u2013 which Backtracking line search enjoys \u2013 which is that", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0155", "text": "f\n        (\n        \n          x\n          \n            n\n            +\n            1\n          \n        \n        )\n        \u2264\n        f\n        (\n        \n          x\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle f(x_{n+1})\\leq f(x_{n})}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0156", "text": "{\\displaystyle f(x_{n+1})\\leq f(x_{n})}\n  \n for all n. If the gradient of the cost function is globally Lipschitz continuous, with Lipschitz constant L, and learning rate is chosen of the order 1/L, then the standard version of SGD is a special case of backtracking line search.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0157", "text": "Second-order methods", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0158", "text": "A stochastic analogue of the standard (deterministic) Newton\u2013Raphson algorithm (a \"second-order\" method) provides an asymptotically optimal or near-optimal form of iterative optimization in the setting of stochastic approximation. A method that uses direct measurements of the Hessian matrices of the summands in the empirical risk function was developed by Byrd, Hansen, Nocedal, and Singer", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0159", "text": ". However, directly determining the required Hessian matrices for optimization may not be possible in practice. Practical and theoretically sound methods for second-order versions of SGD that do not require direct Hessian information are given by Spall and others", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0160", "text": ". (A less efficient method based on finite differences, instead of simultaneous perturbations, is given by Ruppert.) Another approach to the approximation Hessian matrix is replacing it with the Fisher information matrix, which transforms usual gradient to natural. These methods not requiring direct Hessian information are based on either values of the summands in the above empirical risk function or values of the gradients of the summands (i.e., the SGD inputs)", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0161", "text": ". In particular, second-order optimality is asymptotically achievable without direct calculation of the Hessian matrices of the summands in the empirical risk function. When the objective is a  nonlinear least-squares loss", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0162", "text": "Q\n        (\n        w\n        )\n        =\n        \n          \n            1\n            n\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          Q\n          \n            i\n          \n        \n        (\n        w\n        )\n        =\n        \n          \n            1\n            n\n          \n        \n        \n          \u2211\n          \n            i\n            =", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0163", "text": "i\n            =\n            1\n          \n          \n            n\n          \n        \n        (\n        m\n        (\n        w\n        ;\n        \n          x\n          \n            i\n          \n        \n        )\n        \u2212\n        \n          y\n          \n            i\n          \n        \n        \n          )\n          \n            2\n          \n        \n        ,", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0164", "text": ",\n      \n    \n    {\\displaystyle Q(w)={\\frac {1}{n}}\\sum _{i=1}^{n}Q_{i}(w)={\\frac {1}{n}}\\sum _{i=1}^{n}(m(w;x_{i})-y_{i})^{2},}\n  \n \nwhere \n  \n    \n      \n        m\n        (\n        w\n        ;\n        \n          x\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle m(w;x_{i})}\n  \n is the predictive model (e.g., a deep neural network)", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0165", "text": "the objective's structure can be exploited to estimate 2nd order information using gradients only. The resulting \nmethods are simple and often effective", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0166", "text": "Approximations in continuous time\nFor small learning rate \n  \n    \n      \n        \u03b7\n      \n    \n    {\\textstyle \\eta }\n  \n stochastic gradient descent \n  \n    \n      \n        (\n        \n          w\n          \n            n\n          \n        \n        \n          )\n          \n            n\n            \u2208\n            \n              \n                N\n              \n              \n                0", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0167", "text": "{\\textstyle (w_{n})_{n\\in \\mathbb {N} _{0}}}\n  \n can be viewed as a discretization of the gradient flow ODE", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0168", "text": "d\n            \n              d\n              t\n            \n          \n        \n        \n          W\n          \n            t\n          \n        \n        =\n        \u2212\n        \u2207\n        Q\n        (\n        \n          W\n          \n            t\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\frac {d}{dt}}W_{t}=-\\nabla Q(W_{t})}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0169", "text": "subject to additional stochastic noise. This approximation is only valid on a finite time-horizon in the following sense: assume that all the coefficients  \n  \n    \n      \n        \n          Q\n          \n            i\n          \n        \n      \n    \n    {\\textstyle Q_{i}}\n  \n are sufficiently smooth. Let \n  \n    \n      \n        T\n        >\n        0\n      \n    \n    {\\textstyle T>0}\n  \n and \n  \n    \n      \n        g\n        :\n        \n          \n            R\n          \n          \n            d", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0170", "text": "R\n          \n          \n            d\n          \n        \n        \u2192\n        \n          R\n        \n      \n    \n    {\\textstyle g:\\mathbb {R} ^{d}\\to \\mathbb {R} }\n  \n be a sufficiently smooth test function. Then, there exists a constant \n  \n    \n      \n        C\n        >\n        0\n      \n    \n    {\\textstyle C>0}\n  \n such that for all \n  \n    \n      \n        \u03b7\n        >\n        0\n      \n    \n    {\\textstyle \\eta >0}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0171", "text": "max\n          \n            k\n            =\n            0\n            ,\n            \u2026\n            ,\n            \u230a\n            T\n            \n              /\n            \n            \u03b7\n            \u230b\n          \n        \n        \n          |\n          \n            \n              E\n            \n            [\n            g\n            (\n            \n              w\n              \n                k\n              \n            \n            )\n            ]\n            \u2212", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0172", "text": ")\n            ]\n            \u2212\n            g\n            (\n            \n              W\n              \n                k\n                \u03b7\n              \n            \n            )\n          \n          |\n        \n        \u2264\n        C\n        \u03b7\n        ,\n      \n    \n    {\\displaystyle \\max _{k=0,\\dots ,\\lfloor T/\\eta \\rfloor }\\left|\\mathbb {E} [g(w_{k})]-g(W_{k\\eta })\\right|\\leq C\\eta ,}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0173", "text": "where \n  \n    \n      \n        \n          E\n        \n      \n    \n    {\\textstyle \\mathbb {E} }\n  \n denotes taking the expectation with respect to the random choice of indices in the stochastic gradient descent scheme.\nSince this approximation does not capture the random fluctuations around the mean behavior of stochastic gradient descent solutions to stochastic differential equations (SDEs) have been proposed as limiting objects. More precisely, the solution to the SDE", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0174", "text": "d\n        \n          W\n          \n            t\n          \n        \n        =\n        \u2212\n        \u2207\n        \n          (\n          \n            Q\n            (\n            \n              W\n              \n                t\n              \n            \n            )\n            +\n            \n              \n                \n                  1\n                  4\n                \n              \n            \n            \u03b7\n            \n              |\n            \n            \u2207", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0175", "text": "|\n            \n            \u2207\n            Q\n            (\n            \n              W\n              \n                t\n              \n            \n            )\n            \n              \n                |\n              \n              \n                2\n              \n            \n          \n          )\n        \n        d\n        t\n        +\n        \n          \n            \u03b7\n          \n        \n        \u03a3\n        (\n        \n          W\n          \n            t", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0176", "text": "t\n          \n        \n        \n          )\n          \n            1\n            \n              /\n            \n            2\n          \n        \n        d\n        \n          B\n          \n            t\n          \n        \n        ,\n      \n    \n    {\\displaystyle dW_{t}=-\\nabla \\left(Q(W_{t})+{\\tfrac {1}{4}}\\eta |\\nabla Q(W_{t})|^{2}\\right)dt+{\\sqrt {\\eta }}\\Sigma (W_{t})^{1/2}dB_{t},}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0177", "text": "for \n  \n    \n      \n        \u03a3\n        (\n        w\n        )\n        =\n        \n          \n            1\n            \n              n\n              \n                2\n              \n            \n          \n        \n        \n          (\n          \n            \n              \u2211\n              \n                i\n                =\n                1\n              \n              \n                n\n              \n            \n            \n              Q\n              \n                i", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0178", "text": "i\n              \n            \n            (\n            w\n            )\n            \u2212\n            Q\n            (\n            w\n            )\n          \n          )\n        \n        \n          \n            (\n            \n              \n                \u2211\n                \n                  i\n                  =\n                  1\n                \n                \n                  n\n                \n              \n              \n                Q", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0179", "text": "Q\n                \n                  i\n                \n              \n              (\n              w\n              )\n              \u2212\n              Q\n              (\n              w\n              )\n            \n            )\n          \n          \n            T\n          \n        \n      \n    \n    {\\displaystyle \\Sigma (w)={\\frac {1}{n^{2}}}\\left(\\sum _{i=1}^{n}Q_{i}(w)-Q(w)\\right)\\left(\\sum _{i=1}^{n}Q_{i}(w)-Q(w)\\right)^{T}}\n  \n where \n  \n    \n      \n        d", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0180", "text": "where \n  \n    \n      \n        d\n        \n          B\n          \n            t\n          \n        \n      \n    \n    {\\textstyle dB_{t}}\n  \n denotes the Ito-integral with respect to a Brownian motion is a more precise approximation in the sense that there exists a constant \n  \n    \n      \n        C\n        >\n        0\n      \n    \n    {\\textstyle C>0}\n  \n such that", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0181", "text": "max\n          \n            k\n            =\n            0\n            ,\n            \u2026\n            ,\n            \u230a\n            T\n            \n              /\n            \n            \u03b7\n            \u230b\n          \n        \n        \n          |\n          \n            \n              E\n            \n            [\n            g\n            (\n            \n              w\n              \n                k\n              \n            \n            )\n            ]\n            \u2212", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0182", "text": ")\n            ]\n            \u2212\n            \n              E\n            \n            [\n            g\n            (\n            \n              W\n              \n                k\n                \u03b7\n              \n            \n            )\n            ]\n          \n          |\n        \n        \u2264\n        C\n        \n          \u03b7\n          \n            2\n          \n        \n        .", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0183", "text": ".\n      \n    \n    {\\displaystyle \\max _{k=0,\\dots ,\\lfloor T/\\eta \\rfloor }\\left|\\mathbb {E} [g(w_{k})]-\\mathbb {E} [g(W_{k\\eta })]\\right|\\leq C\\eta ^{2}.}", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0184", "text": "However this SDE only approximates the one-point motion of stochastic gradient descent. For an approximation of the stochastic flow one has to consider SDEs with infinite-dimensional noise.\n\nSee also\nBacktracking line search\nBroken Neural Scaling Law\nCoordinate descent \u2013 changes one coordinate at a time, rather than one example\nLinear classifier\nOnline machine learning\nStochastic hill climbing\nStochastic variance reduction", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0185", "text": "Notes\nReferences\nFurther reading\nBottou, L\u00e9on (2004), \"Stochastic Learning\", Advanced Lectures on Machine Learning, LNAI, vol. 3176, Springer, pp. 146\u2013168, ISBN 978-3-540-23122-6\nBuduma, Nikhil; Locascio, Nicholas (2017), \"Beyond Gradient Descent\", Fundamentals of Deep Learning : Designing Next-Generation Machine Intelligence Algorithms, O'Reilly, ISBN 9781491925584", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0186", "text": "LeCun, Yann A.; Bottou, L\u00e9on; Orr, Genevieve B.; M\u00fcller, Klaus-Robert (2012), \"Efficient BackProp\", Neural Networks: Tricks of the Trade, Springer, pp. 9\u201348, ISBN 978-3-642-35288-1\nSpall, James C. (2003), Introduction to Stochastic Search and Optimization, Wiley, ISBN 978-0-471-33052-3", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Stochastic_gradient_descent_0187", "text": "External links\n\"Gradient Descent, How Neural Networks Learn\". 3Blue1Brown. October 16, 2017. Archived from the original on 2021-12-22 \u2013 via YouTube.\nGoh (April 4, 2017). \"Why Momentum Really Works\". Distill. 2 (4). doi:10.23915/distill.00006. Interactive paper explaining momentum.", "title": "Stochastic gradient descent", "url": "https://en.wikipedia.org/wiki/Stochastic_gradient_descent"}, {"id": "rec_Activation_function_0000", "text": "In artificial neural networks, the activation function of a node is a function that calculates the output of the node based on its individual inputs and their weights. Nontrivial problems can be solved using only a few nodes if the activation function is nonlinear.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0001", "text": "Modern activation functions include the logistic (sigmoid) function used in the 2012 speech recognition model developed by Hinton et al; the ReLU used in the 2012 AlexNet computer vision model and in the 2015 ResNet model; and the smooth version of the ReLU, the GELU, which was used in the 2018 BERT model.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0002", "text": "Comparison of activation functions\nAside from their empirical performance, activation functions also have different mathematical properties:", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0003", "text": "Nonlinear\nWhen the activation function is non-linear, then a two-layer neural network can be proven to be a universal function approximator. This is known as the Universal Approximation Theorem. The identity activation function does not satisfy this property. When multiple layers use the identity activation function, the entire network is equivalent to a single-layer model.\nRange", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0004", "text": "Range\nWhen the range of the activation function is finite, gradient-based training methods tend to be more stable, because pattern presentations significantly affect only limited weights. When the range is infinite, training is generally more efficient because pattern presentations significantly affect most of the weights. In the latter case, smaller learning rates are typically necessary.\nContinuously differentiable", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0005", "text": "Continuously differentiable\nThis property is desirable for enabling gradient-based optimization methods (ReLU is not continuously differentiable and has some issues with gradient-based optimization, but it is still possible). The binary step activation function is not differentiable at 0, and it differentiates to 0 for all other values, so gradient-based methods can make no progress with it.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0006", "text": "These properties do not decisively influence performance, nor are they the only mathematical properties that may be useful. For instance, the strictly positive range of the softplus makes it suitable for predicting variances in variational autoencoders.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0007", "text": "Mathematical details\nThe most common activation functions can be divided into three categories: ridge functions, radial functions and fold functions.\nAn activation function \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is saturating if \n  \n    \n      \n        \n          lim\n          \n            \n              |\n            \n            v\n            \n              |\n            \n            \u2192\n            \u221e\n          \n        \n        \n          |\n        \n        \u2207", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0008", "text": "|\n        \n        \u2207\n        f\n        (\n        v\n        )\n        \n          |\n        \n        =\n        0\n      \n    \n    {\\displaystyle \\lim _{|v|\\to \\infty }|\\nabla f(v)|=0}\n  \n. It is nonsaturating if \n  \n    \n      \n        \n          lim\n          \n            \n              |\n            \n            v\n            \n              |\n            \n            \u2192\n            \u221e\n          \n        \n        \n          |\n        \n        \u2207\n        f\n        (", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0009", "text": "\u2207\n        f\n        (\n        v\n        )\n        \n          |\n        \n        \u2260\n        0\n      \n    \n    {\\displaystyle \\lim _{|v|\\to \\infty }|\\nabla f(v)|\\neq 0}\n  \n. Non-saturating activation functions, such as ReLU, may be better than saturating activation functions, because they are less likely to suffer from the vanishing gradient problem.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0010", "text": "Ridge activation functions\nRidge functions are multivariate functions acting on a linear combination of the input variables. Often used examples include:", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0011", "text": "Linear activation: \n  \n    \n      \n        \u03d5\n        (\n        \n          v\n        \n        )\n        =\n        a\n        +\n        \n          \n            v\n          \n          \u2032\n        \n        \n          b\n        \n      \n    \n    {\\displaystyle \\phi (\\mathbf {v} )=a+\\mathbf {v} '\\mathbf {b} }\n  \n,\nReLU activation: \n  \n    \n      \n        \u03d5\n        (\n        \n          v\n        \n        )\n        =\n        max\n        (\n        0\n        ,\n        a\n        +", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0012", "text": ",\n        a\n        +\n        \n          \n            v\n          \n          \u2032\n        \n        \n          b\n        \n        )\n      \n    \n    {\\displaystyle \\phi (\\mathbf {v} )=\\max(0,a+\\mathbf {v} '\\mathbf {b} )}\n  \n,\nHeaviside activation: \n  \n    \n      \n        \u03d5\n        (\n        \n          v\n        \n        )\n        =\n        \n          1\n          \n            a\n            +\n            \n              \n                v\n              \n              \u2032", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0013", "text": "\u2032\n            \n            \n              b\n            \n            >\n            0\n          \n        \n      \n    \n    {\\displaystyle \\phi (\\mathbf {v} )=1_{a+\\mathbf {v} '\\mathbf {b} >0}}\n  \n,\nLogistic activation: \n  \n    \n      \n        \u03d5\n        (\n        \n          v\n        \n        )\n        =\n        (\n        1\n        +\n        exp\n        \u2061\n        (\n        \u2212\n        a\n        \u2212\n        \n          \n            v\n          \n          \u2032\n        \n        \n          b", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0014", "text": "\u2032\n        \n        \n          b\n        \n        )\n        \n          )\n          \n            \u2212\n            1\n          \n        \n      \n    \n    {\\displaystyle \\phi (\\mathbf {v} )=(1+\\exp(-a-\\mathbf {v} '\\mathbf {b} ))^{-1}}\n  \n.\nIn biologically inspired neural networks, the activation function is usually an abstraction representing the rate of action potential firing in the cell. In its simplest form, this function is binary\u2014that is, either the neuron is firing or not.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0015", "text": "Neurons also cannot fire faster than a certain rate. This biological limitation motivates the use of sigmoid activation functions, whose output range is confined to a finite interval.\nThe function looks like \n  \n    \n      \n        \u03d5\n        (\n        \n          v\n        \n        )\n        =\n        U\n        (\n        a\n        +\n        \n          \n            v\n          \n          \u2032\n        \n        \n          b\n        \n        )", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0016", "text": "b\n        \n        )\n      \n    \n    {\\displaystyle \\phi (\\mathbf {v} )=U(a+\\mathbf {v} '\\mathbf {b} )}\n  \n, where \n  \n    \n      \n        U\n      \n    \n    {\\displaystyle U}\n  \n is the Heaviside step function. \nIf a line has a positive slope, on the other hand, it may reflect the increase in firing rate that occurs as input current increases. Such a function would be of the form \n  \n    \n      \n        \u03d5\n        (\n        \n          v\n        \n        )\n        =\n        a\n        +", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0017", "text": ")\n        =\n        a\n        +\n        \n          \n            v\n          \n          \u2032\n        \n        \n          b\n        \n      \n    \n    {\\displaystyle \\phi (\\mathbf {v} )=a+\\mathbf {v} '\\mathbf {b} }\n  \n.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0018", "text": "Radial activation functions\nA special class of activation functions known as radial basis functions (RBFs) are used in RBF networks. These activation functions can take many forms, but they are usually found as one of the following functions:", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0019", "text": "Gaussian: \n  \n    \n      \n        \n        \u03d5\n        (\n        \n          v\n        \n        )\n        =\n        exp\n        \u2061\n        \n          (\n          \n            \u2212\n            \n              \n                \n                  \u2016\n                  \n                    v\n                  \n                  \u2212\n                  \n                    c\n                  \n                  \n                    \u2016\n                    \n                      2", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0020", "text": "2\n                    \n                  \n                \n                \n                  2\n                  \n                    \u03c3\n                    \n                      2\n                    \n                  \n                \n              \n            \n          \n          )\n        \n      \n    \n    {\\displaystyle \\,\\phi (\\mathbf {v} )=\\exp \\left(-{\\frac {\\|\\mathbf {v} -\\mathbf {c} \\|^{2}}{2\\sigma ^{2}}}\\right)}", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0021", "text": "Multiquadratics: \n  \n    \n      \n        \n        \u03d5\n        (\n        \n          v\n        \n        )\n        =\n        \n          \n            \u2016\n            \n              v\n            \n            \u2212\n            \n              c\n            \n            \n              \u2016\n              \n                2\n              \n            \n            +\n            \n              a\n              \n                2", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0022", "text": "{\\displaystyle \\,\\phi (\\mathbf {v} )={\\sqrt {\\|\\mathbf {v} -\\mathbf {c} \\|^{2}+a^{2}}}}", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0023", "text": "Inverse multiquadratics: \n  \n    \n      \n        \n        \u03d5\n        (\n        \n          v\n        \n        )\n        =\n        \n          \n            (\n            \n              \u2016\n              \n                v\n              \n              \u2212\n              \n                c\n              \n              \n                \u2016\n                \n                  2\n                \n              \n              +\n              \n                a\n                \n                  2", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0024", "text": "2\n                \n              \n            \n            )\n          \n          \n            \u2212\n            \n              \n                1\n                2\n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\,\\phi (\\mathbf {v} )=\\left(\\|\\mathbf {v} -\\mathbf {c} \\|^{2}+a^{2}\\right)^{-{\\frac {1}{2}}}}", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0025", "text": "Polyharmonic splines\nwhere \n  \n    \n      \n        \n          c\n        \n      \n    \n    {\\displaystyle \\mathbf {c} }\n  \n is the vector representing the function center and \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n and \n  \n    \n      \n        \u03c3\n      \n    \n    {\\displaystyle \\sigma }\n  \n are parameters affecting the spread of the radius.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0026", "text": "Other examples\nPeriodic functions can serve as activation functions. Usually the sinusoid is used, as any periodic function is decomposable into sinusoids by the Fourier transform.\nQuadratic activation maps \n  \n    \n      \n        x\n        \u21a6\n        \n          x\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle x\\mapsto x^{2}}\n  \n.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0027", "text": "Folding activation functions\nFolding activation functions are extensively used in the pooling layers in convolutional neural networks, and in output layers of multiclass classification networks. These activations perform aggregation over the inputs, such as taking the mean, minimum or maximum. In multiclass classification the softmax activation is often used.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0028", "text": "Table of activation functions\nThe following table compares the properties of several activation functions that are functions of one fold x from the previous layer or layers:\n\nThe following table lists activation functions that are not functions of a single fold x from the previous layer or layers:", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0029", "text": "^  Here, \n  \n    \n      \n        \n          \u03b4\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle \\delta _{ij}}\n  \n is the Kronecker delta.\n^  For instance, \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n could be iterating through the number of kernels of the previous neural network layer while \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n iterates through the number of kernels of the current layer.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0030", "text": "Quantum activation functions", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0031", "text": "In quantum neural networks programmed on gate-model quantum computers, based on quantum perceptrons instead of variational quantum circuits, the non-linearity of the activation function can be implemented with no need of measuring the output of each perceptron at each layer", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0032", "text": ". The quantum properties loaded within the circuit such as superposition can be preserved by creating the Taylor series of the argument computed by the perceptron itself, with suitable quantum circuits computing the powers up to a wanted approximation degree. Because of the flexibility of such quantum circuits, they can be designed in order to approximate any arbitrary classical activation function.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0033", "text": "See also\nLogistic function\nRectifier (neural networks)\nStability (learning theory)\nSoftmax function", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0034", "text": "References\nFurther reading\nKunc, Vladim\u00edr; Kl\u00e9ma, Ji\u0159\u00ed (14 February 2024), Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks, arXiv:2402.09092\nNwankpa, Chigozie; Ijomah, Winifred; Gachagan, Anthony; Marshall, Stephen (8 November 2018). \"Activation Functions: Comparison of trends in Practice and Research for Deep Learning\". arXiv:1811.03378 [cs.LG].", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Activation_function_0035", "text": "Dubey, Shiv Ram; Singh, Satish Kumar; Chaudhuri, Bidyut Baran (2022). \"Activation functions in deep learning: A comprehensive survey and benchmark\". Neurocomputing. 503. Elsevier BV: 92\u2013108. arXiv:2109.14545. doi:10.1016/j.neucom.2022.06.111. ISSN 0925-2312.", "title": "Activation function", "url": "https://en.wikipedia.org/wiki/Activation_function"}, {"id": "rec_Batch_normalization_0000", "text": "In artificial neural networks, batch normalization (also known as batch norm) is a normalization technique used to make training faster and more stable by adjusting the inputs to each layer\u2014re-centering them around zero and re-scaling them to a standard size. It was introduced by Sergey Ioffe and Christian Szegedy in 2015.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0001", "text": "Experts still debate why batch normalization works so well. It was initially thought to tackle internal covariate shift, a problem where parameter initialization and changes in the distribution of the inputs of each layer affect the learning rate of the network. However, newer research suggests it doesn\u2019t fix this shift but instead smooths the objective function\u2014a mathematical guide the network follows to improve\u2014enhancing performance", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0002", "text": ". In very deep networks, batch normalization can initially cause a severe gradient explosion\u2014where updates to the network grow uncontrollably large\u2014but this is managed with shortcuts called skip connections in residual networks. Another theory is that batch normalization adjusts data by handling its size and path separately, speeding up training.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0003", "text": "Internal covariate shift", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0004", "text": "Each layer in a neural network has inputs that follow a specific distribution, which shifts during training due to two main factors: the random starting values of the network\u2019s settings (parameter initialization) and the natural variation in the input data. This shifting pattern affecting the inputs to the network\u2019s inner layers is called internal covariate shift", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0005", "text": ". While a strict definition isn\u2019t fully agreed upon, experiments show that it involves changes in the means and variances of these inputs during training.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0006", "text": "Batch normalization was first developed to address internal covariate shift. During training, as the parameters of preceding layers adjust, the distribution of inputs to the current layer changes accordingly, such that the current layer needs to constantly readjust to new distributions. This issue is particularly severe in deep networks, because small changes in shallower hidden layers will be amplified as they propagate within the network, resulting in significant shift in deeper hidden layers", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0007", "text": ". Batch normalization was proposed to reduced these unwanted shifts to speed up training and produce more reliable models.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0008", "text": "Beyond possibly tackling internal covariate shift, batch normalization offers several additional advantages. It allows the network to use a higher learning rate\u2014a setting that controls how quickly the network learns\u2014without causing problems like vanishing or exploding gradients, where updates become too small or too large", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0009", "text": ". It also appears to have a regularizing effect, improving the network\u2019s ability to generalize to new data, reducing the need for dropout, a technique used to prevent overfitting (when a model learns the training data too well and fails on new data). Additionally, networks using batch normalization are less sensitive to the choice of starting settings or learning rates, making them more robust and adaptable.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0010", "text": "Procedures\nTransformation\nIn a neural network, batch normalization is achieved through a normalization step that fixes the means and variances of each layer's inputs. Ideally, the normalization would be conducted over the entire training set, but to use this step jointly with stochastic optimization methods, it is impractical to use the global information. Thus, normalization is restrained to each mini-batch in the training process.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0011", "text": "Let us use B to denote a mini-batch of size m of the entire training set. The empirical mean and variance of B could thus be denoted as", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0012", "text": "\u03bc\n          \n            B\n          \n        \n        =\n        \n          \n            1\n            m\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            m\n          \n        \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mu _{B}={\\frac {1}{m}}\\sum _{i=1}^{m}x_{i}}\n  \n and \n  \n    \n      \n        \n          \u03c3", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0013", "text": "\u03c3\n          \n            B\n          \n          \n            2\n          \n        \n        =\n        \n          \n            1\n            m\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            m\n          \n        \n        (\n        \n          x\n          \n            i\n          \n        \n        \u2212\n        \n          \u03bc\n          \n            B\n          \n        \n        \n          )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0014", "text": ")\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\sigma _{B}^{2}={\\frac {1}{m}}\\sum _{i=1}^{m}(x_{i}-\\mu _{B})^{2}}\n  \n.\nFor a layer of the network with d-dimensional input, \n  \n    \n      \n        x\n        =\n        (\n        \n          x\n          \n            (\n            1\n            )\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          x\n          \n            (\n            d", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0015", "text": "(\n            d\n            )\n          \n        \n        )\n      \n    \n    {\\displaystyle x=(x^{(1)},...,x^{(d)})}\n  \n, each dimension of its input is then normalized (i.e. re-centered and re-scaled) separately,", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0016", "text": "x\n                ^\n              \n            \n          \n          \n            i\n          \n          \n            (\n            k\n            )\n          \n        \n        =\n        \n          \n            \n              \n                x\n                \n                  i\n                \n                \n                  (\n                  k\n                  )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0017", "text": "\u2212\n              \n                \u03bc\n                \n                  B\n                \n                \n                  (\n                  k\n                  )\n                \n              \n            \n            \n              \n                \n                  (\n                  \n                    \u03c3\n                    \n                      B\n                    \n                    \n                      (\n                      k", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0018", "text": "(\n                      k\n                      )\n                    \n                  \n                  )\n                \n                \n                  2\n                \n              \n              +\n              \u03f5\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {x}}_{i}^{(k)}={\\frac {x_{i}^{(k)}-\\mu _{B}^{(k)}}{\\sqrt {\\left(\\sigma _{B}^{(k)}\\right)^{2}+\\epsilon }}}}\n  \n, where \n  \n    \n      \n        k\n        \u2208\n        [\n        1", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0019", "text": "k\n        \u2208\n        [\n        1\n        ,\n        d\n        ]\n      \n    \n    {\\displaystyle k\\in [1,d]}\n  \n and  \n  \n    \n      \n        i\n        \u2208\n        [\n        1\n        ,\n        m\n        ]\n      \n    \n    {\\displaystyle i\\in [1,m]}\n  \n; \n  \n    \n      \n        \n          \u03bc\n          \n            B\n          \n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle \\mu _{B}^{(k)}}\n  \n and", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0020", "text": "and \n  \n    \n      \n        \n          \u03c3\n          \n            B\n          \n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle \\sigma _{B}^{(k)}}\n  \n are the per-dimension mean and standard deviation, respectively.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0021", "text": "\u03f5\n      \n    \n    {\\displaystyle \\epsilon }\n  \n is added in the denominator for numerical stability and is an arbitrarily small positive constant. The resulting normalized activation \n  \n    \n      \n        \n          \n            \n              \n                x\n                ^\n              \n            \n          \n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle {\\hat {x}}^{(k)}}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0022", "text": "{\\displaystyle {\\hat {x}}^{(k)}}\n  \n have zero mean and unit variance, if \n  \n    \n      \n        \u03f5\n      \n    \n    {\\displaystyle \\epsilon }\n  \n is not taken into account. To restore the representation power of the network, a transformation step then follows as", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0023", "text": "y\n          \n            i\n          \n          \n            (\n            k\n            )\n          \n        \n        =\n        \n          \u03b3\n          \n            (\n            k\n            )\n          \n        \n        \n          \n            \n              \n                x\n                ^\n              \n            \n          \n          \n            i\n          \n          \n            (\n            k\n            )\n          \n        \n        +", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0024", "text": ")\n          \n        \n        +\n        \n          \u03b2\n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle y_{i}^{(k)}=\\gamma ^{(k)}{\\hat {x}}_{i}^{(k)}+\\beta ^{(k)}}\n  \n,\nwhere the parameters \n  \n    \n      \n        \n          \u03b3\n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle \\gamma ^{(k)}}\n  \n and \n  \n    \n      \n        \n          \u03b2\n          \n            (\n            k", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0025", "text": "(\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle \\beta ^{(k)}}\n  \n are subsequently learned in the optimization process.\nFormally, the operation that implements batch normalization is a transform \n  \n    \n      \n        B\n        \n          N\n          \n            \n              \u03b3\n              \n                (\n                k\n                )\n              \n            \n            ,\n            \n              \u03b2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0026", "text": "\u03b2\n              \n                (\n                k\n                )\n              \n            \n          \n        \n        :\n        \n          x\n          \n            1...\n            m\n          \n          \n            (\n            k\n            )\n          \n        \n        \u2192\n        \n          y\n          \n            1...\n            m\n          \n          \n            (\n            k\n            )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0027", "text": ")\n          \n        \n      \n    \n    {\\displaystyle BN_{\\gamma ^{(k)},\\beta ^{(k)}}:x_{1...m}^{(k)}\\rightarrow y_{1...m}^{(k)}}\n  \n called the Batch Normalizing transform. The output of the BN transform \n  \n    \n      \n        \n          y\n          \n            (\n            k\n            )\n          \n        \n        =\n        B\n        \n          N\n          \n            \n              \u03b3\n              \n                (\n                k\n                )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0028", "text": ")\n              \n            \n            ,\n            \n              \u03b2\n              \n                (\n                k\n                )\n              \n            \n          \n        \n        (\n        \n          x\n          \n            (\n            k\n            )\n          \n        \n        )\n      \n    \n    {\\displaystyle y^{(k)}=BN_{\\gamma ^{(k)},\\beta ^{(k)}}(x^{(k)})}\n  \n is then passed to other network layers, while the normalized output", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0029", "text": "x\n                ^\n              \n            \n          \n          \n            i\n          \n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle {\\hat {x}}_{i}^{(k)}}\n  \n remains internal to the current layer.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0030", "text": "Backpropagation\nThe described BN transform is a differentiable operation, and the gradient of the loss \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n  with respect to the different parameters can be computed directly with the chain rule.\nSpecifically, \n  \n    \n      \n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202\n              \n                y\n                \n                  i", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0031", "text": "(\n                  k\n                  )\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial l}{\\partial y_{i}^{(k)}}}}\n  \n depends on the choice of activation function, and the gradient against other parameters could be expressed as a function of \n  \n    \n      \n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0032", "text": "\u2202\n              \n                y\n                \n                  i\n                \n                \n                  (\n                  k\n                  )\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial l}{\\partial y_{i}^{(k)}}}}\n  \n:", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0033", "text": "\u2202\n              l\n            \n            \n              \u2202\n              \n                \n                  \n                    \n                      x\n                      ^\n                    \n                  \n                \n                \n                  i\n                \n                \n                  (\n                  k\n                  )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0034", "text": "=\n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202\n              \n                y\n                \n                  i\n                \n                \n                  (\n                  k\n                  )\n                \n              \n            \n          \n        \n        \n          \u03b3\n          \n            (\n            k\n            )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0035", "text": ")\n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial l}{\\partial {\\hat {x}}_{i}^{(k)}}}={\\frac {\\partial l}{\\partial y_{i}^{(k)}}}\\gamma ^{(k)}}\n  \n,", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0036", "text": "\u2202\n              l\n            \n            \n              \u2202\n              \n                \u03b3\n                \n                  (\n                  k\n                  )\n                \n              \n            \n          \n        \n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            m\n          \n        \n        \n          \n            \n              \u2202", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0037", "text": "\u2202\n              l\n            \n            \n              \u2202\n              \n                y\n                \n                  i\n                \n                \n                  (\n                  k\n                  )\n                \n              \n            \n          \n        \n        \n          \n            \n              \n                x\n                ^\n              \n            \n          \n          \n            i", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0038", "text": "i\n          \n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial l}{\\partial \\gamma ^{(k)}}}=\\sum _{i=1}^{m}{\\frac {\\partial l}{\\partial y_{i}^{(k)}}}{\\hat {x}}_{i}^{(k)}}\n  \n, \n  \n    \n      \n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202\n              \n                \u03b2\n                \n                  (\n                  k", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0039", "text": "(\n                  k\n                  )\n                \n              \n            \n          \n        \n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            m\n          \n        \n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202\n              \n                y\n                \n                  i\n                \n                \n                  (", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0040", "text": "(\n                  k\n                  )\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial l}{\\partial \\beta ^{(k)}}}=\\sum _{i=1}^{m}{\\frac {\\partial l}{\\partial y_{i}^{(k)}}}}\n  \n,\n  \n    \n      \n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202\n              \n                \u03c3\n                \n                  B", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0041", "text": "B\n                \n                \n                  (\n                  k\n                  \n                    )\n                    \n                      2\n                    \n                  \n                \n              \n            \n          \n        \n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            m\n          \n        \n        \n          \n            \n              \u2202\n              l", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0042", "text": "\u2202\n              l\n            \n            \n              \u2202\n              \n                y\n                \n                  i\n                \n                \n                  (\n                  k\n                  )\n                \n              \n            \n          \n        \n        (\n        \n          x\n          \n            i\n          \n          \n            (\n            k\n            )\n          \n        \n        \u2212\n        \n          \u03bc", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0043", "text": "\u2212\n        \n          \u03bc\n          \n            B\n          \n          \n            (\n            k\n            )\n          \n        \n        )\n        \n          (\n          \n            \u2212\n            \n              \n                \n                  \u03b3\n                  \n                    (\n                    k\n                    )\n                  \n                \n                2\n              \n            \n            (\n            \n              \u03c3", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0044", "text": "\u03c3\n              \n                B\n              \n              \n                (\n                k\n                \n                  )\n                  \n                    2\n                  \n                \n              \n            \n            +\n            \u03f5\n            \n              )\n              \n                \u2212\n                3\n                \n                  /\n                \n                2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0045", "text": ")\n        \n      \n    \n    {\\displaystyle {\\frac {\\partial l}{\\partial \\sigma _{B}^{(k)^{2}}}}=\\sum _{i=1}^{m}{\\frac {\\partial l}{\\partial y_{i}^{(k)}}}(x_{i}^{(k)}-\\mu _{B}^{(k)})\\left(-{\\frac {\\gamma ^{(k)}}{2}}(\\sigma _{B}^{(k)^{2}}+\\epsilon )^{-3/2}\\right)}\n  \n, \n  \n    \n      \n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202\n              \n                \u03bc", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0046", "text": "\u03bc\n                \n                  B\n                \n                \n                  (\n                  k\n                  )\n                \n              \n            \n          \n        \n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            m\n          \n        \n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0047", "text": "\u2202\n              \n                y\n                \n                  i\n                \n                \n                  (\n                  k\n                  )\n                \n              \n            \n          \n        \n        \n          \n            \n              \u2212\n              \n                \u03b3\n                \n                  (\n                  k\n                  )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0048", "text": "\u03c3\n                \n                  B\n                \n                \n                  (\n                  k\n                  \n                    )\n                    \n                      2\n                    \n                  \n                \n              \n              +\n              \u03f5\n            \n          \n        \n        +\n        \n          \n            \n              \u2202\n              l", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0049", "text": "l\n            \n            \n              \u2202\n              \n                \u03c3\n                \n                  B\n                \n                \n                  (\n                  k\n                  \n                    )\n                    \n                      2\n                    \n                  \n                \n              \n            \n          \n        \n        \n          \n            1\n            m\n          \n        \n        \n          \u2211", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0050", "text": "\u2211\n          \n            i\n            =\n            1\n          \n          \n            m\n          \n        \n        (\n        \u2212\n        2\n        )\n        \u22c5\n        (\n        \n          x\n          \n            i\n          \n          \n            (\n            k\n            )\n          \n        \n        \u2212\n        \n          \u03bc\n          \n            B\n          \n          \n            (\n            k\n            )\n          \n        \n        )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0051", "text": ")\n      \n    \n    {\\displaystyle {\\frac {\\partial l}{\\partial \\mu _{B}^{(k)}}}=\\sum _{i=1}^{m}{\\frac {\\partial l}{\\partial y_{i}^{(k)}}}{\\frac {-\\gamma ^{(k)}}{\\sqrt {\\sigma _{B}^{(k)^{2}}+\\epsilon }}}+{\\frac {\\partial l}{\\partial \\sigma _{B}^{(k)^{2}}}}{\\frac {1}{m}}\\sum _{i=1}^{m}(-2)\\cdot (x_{i}^{(k)}-\\mu _{B}^{(k)})}\n  \n,\nand \n  \n    \n      \n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0052", "text": "\u2202\n              \n                x\n                \n                  i\n                \n                \n                  (\n                  k\n                  )\n                \n              \n            \n          \n        \n        =\n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202\n              \n                \n                  \n                    \n                      x\n                      ^", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0053", "text": "x\n                      ^\n                    \n                  \n                \n                \n                  i\n                \n                \n                  (\n                  k\n                  )\n                \n              \n            \n          \n        \n        \n          \n            1\n            \n              \n                \u03c3\n                \n                  B\n                \n                \n                  (\n                  k", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0054", "text": "(\n                  k\n                  \n                    )\n                    \n                      2\n                    \n                  \n                \n              \n              +\n              \u03f5\n            \n          \n        \n        +\n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202\n              \n                \u03c3\n                \n                  B", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0055", "text": "(\n                  k\n                  \n                    )\n                    \n                      2\n                    \n                  \n                \n              \n            \n          \n        \n        \n          \n            \n              2\n              (\n              \n                x\n                \n                  i\n                \n                \n                  (\n                  k\n                  )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0056", "text": "k\n                  )\n                \n              \n              \u2212\n              \n                \u03bc\n                \n                  B\n                \n                \n                  (\n                  k\n                  )\n                \n              \n              )\n            \n            m\n          \n        \n        +\n        \n          \n            \n              \u2202\n              l\n            \n            \n              \u2202\n              \n                \u03bc", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0057", "text": "\u2202\n              \n                \u03bc\n                \n                  B\n                \n                \n                  (\n                  k\n                  )\n                \n              \n            \n          \n        \n        \n          \n            1\n            m", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0058", "text": "m\n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\partial l}{\\partial x_{i}^{(k)}}}={\\frac {\\partial l}{\\partial {\\hat {x}}_{i}^{(k)}}}{\\frac {1}{\\sqrt {\\sigma _{B}^{(k)^{2}}+\\epsilon }}}+{\\frac {\\partial l}{\\partial \\sigma _{B}^{(k)^{2}}}}{\\frac {2(x_{i}^{(k)}-\\mu _{B}^{(k)})}{m}}+{\\frac {\\partial l}{\\partial \\mu _{B}^{(k)}}}{\\frac {1}{m}}}\n  \n.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0059", "text": "Inference\nDuring the training stage, the normalization steps depend on the mini-batches to ensure efficient and reliable training. However, in the inference stage, this dependence is not useful any more. Instead, the normalization step in this stage is computed with the population statistics such that the output could depend on the input in a deterministic manner. The population mean, \n  \n    \n      \n        E\n        [\n        \n          x\n          \n            (\n            k\n            )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0060", "text": "(\n            k\n            )\n          \n        \n        ]\n      \n    \n    {\\displaystyle E[x^{(k)}]}\n  \n, and variance, \n  \n    \n      \n        Var\n        \u2061\n        [\n        \n          x\n          \n            (\n            k\n            )\n          \n        \n        ]\n      \n    \n    {\\displaystyle \\operatorname {Var} [x^{(k)}]}\n  \n, are computed as:", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0061", "text": "E\n        [\n        \n          x\n          \n            (\n            k\n            )\n          \n        \n        ]\n        =\n        \n          E\n          \n            B\n          \n        \n        [\n        \n          \u03bc\n          \n            B\n          \n          \n            (\n            k\n            )\n          \n        \n        ]\n      \n    \n    {\\displaystyle E[x^{(k)}]=E_{B}[\\mu _{B}^{(k)}]}\n  \n, and \n  \n    \n      \n        Var\n        \u2061\n        [", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0062", "text": "Var\n        \u2061\n        [\n        \n          x\n          \n            (\n            k\n            )\n          \n        \n        ]\n        =\n        \n          \n            m\n            \n              m\n              \u2212\n              1\n            \n          \n        \n        \n          E\n          \n            B\n          \n        \n        [\n        \n          \n            (\n            \n              \u03c3\n              \n                B", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0063", "text": "B\n              \n              \n                (\n                k\n                )\n              \n            \n            )\n          \n          \n            2\n          \n        \n        ]\n      \n    \n    {\\displaystyle \\operatorname {Var} [x^{(k)}]={\\frac {m}{m-1}}E_{B}[\\left(\\sigma _{B}^{(k)}\\right)^{2}]}\n  \n.\nThe population statistics thus is a complete representation of the mini-batches.\nThe BN transform in the inference step thus becomes", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0064", "text": "y\n          \n            (\n            k\n            )\n          \n        \n        =\n        B\n        \n          N\n          \n            \n              \u03b3\n              \n                (\n                k\n                )\n              \n            \n            ,\n            \n              \u03b2\n              \n                (\n                k\n                )\n              \n            \n          \n          \n            inf\n          \n        \n        (", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0065", "text": "inf\n          \n        \n        (\n        \n          x\n          \n            (\n            k\n            )\n          \n        \n        )\n        =\n        \n          \u03b3\n          \n            (\n            k\n            )\n          \n        \n        \n          \n            \n              \n                x\n                \n                  (\n                  k\n                  )\n                \n              \n              \u2212\n              E\n              [", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0066", "text": "E\n              [\n              \n                x\n                \n                  (\n                  k\n                  )\n                \n              \n              ]\n            \n            \n              Var\n              \u2061\n              [\n              \n                x\n                \n                  (\n                  k\n                  )\n                \n              \n              ]\n              +\n              \u03f5\n            \n          \n        \n        +", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0067", "text": "+\n        \n          \u03b2\n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle y^{(k)}=BN_{\\gamma ^{(k)},\\beta ^{(k)}}^{\\text{inf}}(x^{(k)})=\\gamma ^{(k)}{\\frac {x^{(k)}-E[x^{(k)}]}{\\sqrt {\\operatorname {Var} [x^{(k)}]+\\epsilon }}}+\\beta ^{(k)}}\n  \n,\nwhere \n  \n    \n      \n        \n          y\n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle y^{(k)}}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0068", "text": "{\\displaystyle y^{(k)}}\n  \n is passed on to next layers instead of \n  \n    \n      \n        \n          x\n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle x^{(k)}}\n  \n. Since the parameters are fixed in this transformation, the batch normalization procedure is essentially applying a linear transform to the activation function.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0069", "text": "Theory", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0070", "text": "Although batch normalization has become popular due to its strong empirical performance, the working mechanism of the method is not yet well-understood. The explanation made in the original paper was that batch norm works by reducing internal covariate shift, but this has been challenged by more recent work. One experiment trained a VGG-16 network under 3 different training regimes: standard (no batch norm), batch norm, and batch norm with noise added to each layer during training", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0071", "text": ". In the third model, the noise has non-zero mean and non-unit variance, i.e. it explicitly introduces covariate shift. Despite this, it showed similar accuracy to the second model, and both performed better than the first, suggesting that covariate shift is not the reason that batch norm improves performance.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0072", "text": "Using batch normalization causes the items in a batch to no longer be iid, which can lead to difficulties in training due to lower quality gradient estimation.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0073", "text": "Smoothness\nOne alternative explanation is that the improvement with batch normalization is instead due to producing a smoother parameter space and smoother gradients, as formalized by a smaller Lipschitz constant. \nConsider two identical networks, one contains batch normalization layers and the other does not, the behaviors of these two networks are then compared. Denote the loss functions as \n  \n    \n      \n        \n          \n            \n              L\n              ^", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0074", "text": "L\n              ^\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {L}}}\n  \n and \n  \n    \n      \n        L\n      \n    \n    {\\displaystyle L}\n  \n, respectively. Let the input to both networks be \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n, and the output be \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n, for which \n  \n    \n      \n        y\n        =\n        W\n        x\n      \n    \n    {\\displaystyle y=Wx}\n  \n, where", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0075", "text": "{\\displaystyle y=Wx}\n  \n, where \n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}\n  \n is the layer weights. For the second network, \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n additionally goes through a batch normalization layer. Denote the normalized activation as \n  \n    \n      \n        \n          \n            \n              y\n              ^\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {y}}}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0076", "text": "{\\displaystyle {\\hat {y}}}\n  \n, which has zero mean and unit variance. Let the transformed activation be \n  \n    \n      \n        z\n        =\n        \u03b3\n        \n          \n            \n              y\n              ^\n            \n          \n        \n        +\n        \u03b2\n      \n    \n    {\\displaystyle z=\\gamma {\\hat {y}}+\\beta }\n  \n, and suppose \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n and \n  \n    \n      \n        \u03b2\n      \n    \n    {\\displaystyle \\beta }", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0077", "text": "\u03b2\n      \n    \n    {\\displaystyle \\beta }\n  \n are constants. Finally, denote the standard deviation over a mini-batch \n  \n    \n      \n        \n          \n            \n              \n                y\n                \n                  j\n                \n              \n              ^\n            \n          \n        \n        \u2208\n        \n          \n            R\n          \n          \n            m\n          \n        \n      \n    \n    {\\displaystyle {\\hat {y_{j}}}\\in \\mathbb {R} ^{m}}\n  \n as", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0078", "text": "as \n  \n    \n      \n        \n          \u03c3\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\sigma _{j}}\n  \n.\nFirst, it can be shown that the gradient magnitude of a batch normalized network, \n  \n    \n      \n        \n          |\n        \n        \n          |\n        \n        \n          \u25bd\n          \n            \n              y\n              \n                i\n              \n            \n          \n        \n        \n          \n            \n              L", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0079", "text": "L\n              ^\n            \n          \n        \n        \n          |\n        \n        \n          |\n        \n      \n    \n    {\\displaystyle ||\\triangledown _{y_{i}}{\\hat {L}}||}\n  \n, is bounded, with the bound expressed as", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0080", "text": "|\n        \n        \n          |\n        \n        \n          \u25bd\n          \n            \n              y\n              \n                i\n              \n            \n          \n        \n        \n          \n            \n              L\n              ^\n            \n          \n        \n        \n          |\n        \n        \n          \n            |\n          \n          \n            2\n          \n        \n        \u2264\n        \n          \n            \n              \u03b3", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0081", "text": "\u03b3\n              \n                2\n              \n            \n            \n              \u03c3\n              \n                j\n              \n              \n                2\n              \n            \n          \n        \n        \n          \n            (\n          \n        \n        \n          |\n        \n        \n          |\n        \n        \n          \u25bd\n          \n            \n              y\n              \n                i", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0082", "text": "i\n              \n            \n          \n        \n        L\n        \n          |\n        \n        \n          \n            |\n          \n          \n            2\n          \n        \n        \u2212\n        \n          \n            1\n            m\n          \n        \n        \u27e8\n        1\n        ,\n        \n          \u25bd\n          \n            \n              y\n              \n                i\n              \n            \n          \n        \n        L\n        \n          \u27e9", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0083", "text": "L\n        \n          \u27e9\n          \n            2\n          \n        \n        \u2212\n        \n          \n            1\n            m\n          \n        \n        \u27e8\n        \n          \u25bd\n          \n            \n              y\n              \n                i\n              \n            \n          \n        \n        L\n        ,\n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            j", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0084", "text": "j\n          \n        \n        \n          \u27e9\n          \n            2\n          \n        \n        \n          \n            )\n          \n        \n      \n    \n    {\\displaystyle ||\\triangledown _{y_{i}}{\\hat {L}}||^{2}\\leq {\\frac {\\gamma ^{2}}{\\sigma _{j}^{2}}}{\\Bigg (}||\\triangledown _{y_{i}}L||^{2}-{\\frac {1}{m}}\\langle 1,\\triangledown _{y_{i}}L\\rangle ^{2}-{\\frac {1}{m}}\\langle \\triangledown _{y_{i}}L,{\\hat {y}}_{j}\\rangle ^{2}{\\bigg )}}\n  \n.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0085", "text": ".\nSince the gradient magnitude represents the Lipschitzness of the loss, this relationship indicates that a batch normalized network could achieve greater Lipschitzness comparatively. Notice that the bound gets tighter when the gradient \n  \n    \n      \n        \n          \u25bd\n          \n            \n              y\n              \n                i\n              \n            \n          \n        \n        \n          \n            \n              L\n              ^", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0086", "text": "^\n            \n          \n        \n      \n    \n    {\\displaystyle \\triangledown _{y_{i}}{\\hat {L}}}\n  \n correlates with the activation \n  \n    \n      \n        \n          \n            \n              \n                y\n                \n                  i\n                \n              \n              ^\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {y_{i}}}}\n  \n, which is a common phenomena. The scaling of", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0087", "text": "\u03b3\n              \n                2\n              \n            \n            \n              \u03c3\n              \n                j\n              \n              \n                2\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\gamma ^{2}}{\\sigma _{j}^{2}}}}\n  \n is also significant, since the variance is often large.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0088", "text": "Secondly, the quadratic form of the loss Hessian with respect to activation in the gradient direction can be bounded as", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0089", "text": "(\n        \n          \u25bd\n          \n            \n              y\n              \n                j\n              \n            \n          \n        \n        \n          \n            \n              L\n              ^\n            \n          \n        \n        \n          )\n          \n            T\n          \n        \n        \n          \n            \n              \u2202\n              \n                \n                  \n                    L\n                    ^", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0090", "text": "^\n                  \n                \n              \n            \n            \n              \u2202\n              \n                y\n                \n                  j\n                \n              \n              \u2202\n              \n                y\n                \n                  j\n                \n              \n            \n          \n        \n        (\n        \n          \u25bd\n          \n            \n              y\n              \n                j", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0091", "text": "j\n              \n            \n          \n        \n        \n          \n            \n              L\n              ^\n            \n          \n        \n        )\n        \u2264\n        \n          \n            \n              \u03b3\n              \n                2\n              \n            \n            \n              \u03c3\n              \n                2\n              \n            \n          \n        \n        \n          \n            (", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0092", "text": "\u2202\n              \n                \n                  \n                    L\n                    ^\n                  \n                \n              \n            \n            \n              \u2202\n              \n                y\n                \n                  j\n                \n              \n            \n          \n        \n        \n          \n            \n              )\n            \n          \n          \n            T", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0093", "text": "T\n          \n        \n        \n          \n            (\n          \n        \n        \n          \n            \n              \u2202\n              L\n            \n            \n              \u2202\n              \n                y\n                \n                  j\n                \n              \n              \u2202\n              \n                y\n                \n                  j\n                \n              \n            \n          \n        \n        \n          \n            )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0094", "text": ")\n          \n        \n        \n          \n            (\n          \n        \n        \n          \n            \n              \u2202\n              \n                \n                  \n                    L\n                    ^\n                  \n                \n              \n            \n            \n              \u2202\n              \n                y\n                \n                  j", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0095", "text": ")\n          \n        \n        \u2212\n        \n          \n            \u03b3\n            \n              m\n              \n                \u03c3\n                \n                  2\n                \n              \n            \n          \n        \n        \u27e8\n        \n          \u25bd\n          \n            \n              y\n              \n                j\n              \n            \n          \n        \n        L\n        ,", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0096", "text": ",\n        \n          \n            \n              \n                y\n                \n                  j\n                \n              \n              ^\n            \n          \n        \n        \u27e9\n        \n          \n            |\n          \n        \n        \n          \n            |\n          \n        \n        \n          \n            \n              \u2202\n              \n                \n                  \n                    L\n                    ^", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0097", "text": "\u2202\n              \n                y\n                \n                  j\n                \n              \n            \n          \n        \n        \n          \n            |\n          \n        \n        \n          \n            \n              |\n            \n          \n          \n            2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0098", "text": "{\\displaystyle (\\triangledown _{y_{j}}{\\hat {L}})^{T}{\\frac {\\partial {\\hat {L}}}{\\partial y_{j}\\partial y_{j}}}(\\triangledown _{y_{j}}{\\hat {L}})\\leq {\\frac {\\gamma ^{2}}{\\sigma ^{2}}}{\\bigg (}{\\frac {\\partial {\\hat {L}}}{\\partial y_{j}}}{\\bigg )}^{T}{\\bigg (}{\\frac {\\partial L}{\\partial y_{j}\\partial y_{j}}}{\\bigg )}{\\bigg (}{\\frac {\\partial {\\hat {L}}}{\\partial y_{j}}}{\\bigg )}-{\\frac {\\gamma }{m\\sigma ^{2}}}\\langle \\triangledown _{y_{j}}L,{\\hat {y_{j}}}\\rangle {\\bigg |}{\\bigg |}{\\frac", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0099", "text": "{y_{j}}}\\rangle {\\bigg |}{\\bigg |}{\\frac {\\partial {\\hat {L}}}{\\partial y_{j}}}{\\bigg |}{\\bigg |}^{2}}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0100", "text": ".\nThe scaling of \n  \n    \n      \n        \n          \n            \n              \u03b3\n              \n                2\n              \n            \n            \n              \u03c3\n              \n                j\n              \n              \n                2\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\gamma ^{2}}{\\sigma _{j}^{2}}}}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0101", "text": "indicates that the loss Hessian is resilient to the mini-batch variance, whereas the second term on the right hand side suggests that it becomes smoother when the Hessian and the inner product are non-negative. If the loss is locally convex, then the Hessian is positive semi-definite, while the inner product is positive if \n  \n    \n      \n        \n          \n            \n              \n                g\n                \n                  j\n                \n              \n              ^", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0102", "text": "^\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {g_{j}}}}\n  \n is in the direction towards the minimum of the loss. It could thus be concluded from this inequality that the gradient generally becomes more predictive with the batch normalization layer.\nIt then follows to translate the bounds related to the loss with respect to the normalized activation to a bound on the loss with respect to the network weights:", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0103", "text": "g\n                \n                  j\n                \n              \n              ^\n            \n          \n        \n        \u2264\n        \n          \n            \n              \u03b3\n              \n                2\n              \n            \n            \n              \u03c3\n              \n                j\n              \n              \n                2\n              \n            \n          \n        \n        (", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0104", "text": "(\n        \n          g\n          \n            j\n          \n          \n            2\n          \n        \n        \u2212\n        m\n        \n          \u03bc\n          \n            \n              g\n              \n                j\n              \n            \n          \n          \n            2\n          \n        \n        \u2212\n        \n          \u03bb\n          \n            2\n          \n        \n        \u27e8\n        \n          \u25bd\n          \n            \n              y", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0105", "text": "y\n              \n                j\n              \n            \n          \n        \n        L\n        ,\n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            j\n          \n        \n        \n          \u27e9\n          \n            2\n          \n        \n        )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0106", "text": ")\n      \n    \n    {\\displaystyle {\\hat {g_{j}}}\\leq {\\frac {\\gamma ^{2}}{\\sigma _{j}^{2}}}(g_{j}^{2}-m\\mu _{g_{j}}^{2}-\\lambda ^{2}\\langle \\triangledown _{y_{j}}L,{\\hat {y}}_{j}\\rangle ^{2})}\n  \n, where \n  \n    \n      \n        \n          g\n          \n            j\n          \n        \n        =\n        m\n        a\n        \n          x\n          \n            \n              |\n            \n            \n              |\n            \n            X", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0107", "text": "X\n            \n              |\n            \n            \n              |\n            \n            \u2264\n            \u03bb\n          \n        \n        \n          |\n        \n        \n          |\n        \n        \n          \u25bd\n          \n            W\n          \n        \n        L\n        \n          |\n        \n        \n          \n            |\n          \n          \n            2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0108", "text": "2\n          \n        \n      \n    \n    {\\displaystyle g_{j}=max_{||X||\\leq \\lambda }||\\triangledown _{W}L||^{2}}\n  \n and \n  \n    \n      \n        \n          \n            \n              \n                g\n                ^\n              \n            \n          \n          \n            j\n          \n        \n        =\n        m\n        a\n        \n          x\n          \n            \n              |\n            \n            \n              |\n            \n            X", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0109", "text": "X\n            \n              |\n            \n            \n              |\n            \n            \u2264\n            \u03bb\n          \n        \n        \n          |\n        \n        \n          |\n        \n        \n          \u25bd\n          \n            W\n          \n        \n        \n          \n            \n              L\n              ^\n            \n          \n        \n        \n          |\n        \n        \n          \n            |\n          \n          \n            2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0110", "text": "2\n          \n        \n      \n    \n    {\\displaystyle {\\hat {g}}_{j}=max_{||X||\\leq \\lambda }||\\triangledown _{W}{\\hat {L}}||^{2}}\n  \n.\nIn addition to the smoother landscape, it is further shown that batch normalization could result in a better initialization with the following inequality:", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0111", "text": "|\n        \n        \n          |\n        \n        \n          W\n          \n            0\n          \n        \n        \u2212\n        \n          \n            \n              \n                W\n                ^\n              \n            \n          \n          \n            \u2217\n          \n        \n        \n          |\n        \n        \n          \n            |\n          \n          \n            2\n          \n        \n        \u2264\n        \n          |", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0112", "text": "\u2264\n        \n          |\n        \n        \n          |\n        \n        \n          W\n          \n            0\n          \n        \n        \u2212\n        \n          W\n          \n            \u2217\n          \n        \n        \n          |\n        \n        \n          \n            |\n          \n          \n            2\n          \n        \n        \u2212\n        \n          \n            1\n            \n              \n                |\n              \n              \n                |", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0113", "text": "|\n              \n              \n                W\n                \n                  \u2217\n                \n              \n              \n                |\n              \n              \n                \n                  |\n                \n                \n                  2\n                \n              \n            \n          \n        \n        (\n        \n          |\n        \n        \n          |\n        \n        \n          W\n          \n            \u2217", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0114", "text": "\u2217\n          \n        \n        \n          |\n        \n        \n          \n            |\n          \n          \n            2\n          \n        \n        \u2212\n        \u27e8\n        \n          W\n          \n            \u2217\n          \n        \n        ,\n        \n          W\n          \n            0\n          \n        \n        \u27e9\n        \n          )\n          \n            2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0115", "text": "2\n          \n        \n      \n    \n    {\\displaystyle ||W_{0}-{\\hat {W}}^{*}||^{2}\\leq ||W_{0}-W^{*}||^{2}-{\\frac {1}{||W^{*}||^{2}}}(||W^{*}||^{2}-\\langle W^{*},W_{0}\\rangle )^{2}}\n  \n, where \n  \n    \n      \n        \n          W\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle W^{*}}\n  \n and \n  \n    \n      \n        \n          \n            \n              \n                W\n                ^\n              \n            \n          \n          \n            \u2217", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0116", "text": "\u2217\n          \n        \n      \n    \n    {\\displaystyle {\\hat {W}}^{*}}\n  \n are the local optimal weights for the two networks, respectively.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0117", "text": "Some scholars argue that the above analysis cannot fully capture the performance of batch normalization, because the proof only concerns the largest eigenvalue, or equivalently, one direction in the landscape at all points. It is suggested that the complete eigenspectrum needs to be taken into account to make a conclusive analysis.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0118", "text": "Measure", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0119", "text": "Since it is hypothesized that batch normalization layers could reduce internal covariate shift, an experiment is set up to measure quantitatively how much covariate shift is reduced. First, the notion of internal covariate shift needs to be defined mathematically", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0120", "text": ". Specifically, to quantify the adjustment that a layer's parameters make in response to updates in previous layers, the correlation between the gradients of the loss before and after all previous layers are updated is measured, since gradients could capture the shifts from the first-order training method. If the shift introduced by the changes in previous layers is small, then the correlation between the gradients would be close to 1.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0121", "text": "The correlation between the gradients are computed for four models: a standard VGG network, a VGG network with batch normalization layers, a 25-layer deep linear network (DLN) trained with full-batch gradient descent, and a DLN network with batch normalization layers", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0122", "text": ". Interestingly, it is shown that the standard VGG and DLN models both have higher correlations of gradients compared with their counterparts, indicating that the additional batch normalization layers are not reducing internal covariate shift.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0123", "text": "Vanishing/exploding gradients\nEven though batch norm was originally introduced to alleviate gradient vanishing or explosion problems, a deep batch norm network in fact suffers from gradient explosion at initialization time, no matter what it uses for nonlinearity. Thus, the optimization landscape is very far from smooth for a randomly initialized, deep batch norm network.\nMore precisely, if the network has \n  \n    \n      \n        L\n      \n    \n    {\\displaystyle L}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0124", "text": "L\n      \n    \n    {\\displaystyle L}\n  \n layers, then the gradient of the first layer weights has norm \n  \n    \n      \n        >\n        c\n        \n          \u03bb\n          \n            L\n          \n        \n      \n    \n    {\\displaystyle >c\\lambda ^{L}}\n  \n for some \n  \n    \n      \n        \u03bb\n        >\n        1\n        ,\n        c\n        >\n        0\n      \n    \n    {\\displaystyle \\lambda >1,c>0}\n  \n depending only on the nonlinearity.\nFor any fixed nonlinearity, \n  \n    \n      \n        \u03bb", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0125", "text": "\u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n decreases as the batch size increases. For example, for ReLU, \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n decreases to \n  \n    \n      \n        \u03c0\n        \n          /\n        \n        (\n        \u03c0\n        \u2212\n        1\n        )\n        \u2248\n        1.467\n      \n    \n    {\\displaystyle \\pi /(\\pi -1)\\approx 1.467}\n  \n as the batch size tends to infinity.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0126", "text": "as the batch size tends to infinity.\nPractically, this means deep batch norm networks are untrainable.\nThis is only relieved by skip connections in the fashion of residual networks.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0127", "text": "This gradient explosion on the surface contradicts the smoothness property explained in the previous section, but in fact they are consistent. The previous section studies the effect of inserting a single batch norm in a network, while the gradient explosion depends on stacking batch norms typical of modern deep neural networks.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0128", "text": "Decoupling\nAnother possible reason for the success of batch normalization is that it decouples the length and direction of the weight vectors and thus facilitates better training.\nBy interpreting batch norm as a reparametrization of weight space, it can be shown that the length and the direction of the weights are separated and can thus be trained separately. For a particular neural network unit with input \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n and weight vector", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0129", "text": "{\\displaystyle x}\n  \n and weight vector \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n, denote its output as \n  \n    \n      \n        f\n        (\n        w\n        )\n        =\n        \n          E\n          \n            x\n          \n        \n        [\n        \u03d5\n        (\n        \n          x\n          \n            T\n          \n        \n        w\n        )\n        ]\n      \n    \n    {\\displaystyle f(w)=E_{x}[\\phi (x^{T}w)]}\n  \n, where \n  \n    \n      \n        \u03d5", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0130", "text": ", where \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n is the activation function, and denote \n  \n    \n      \n        S\n        =\n        E\n        [\n        x\n        \n          x\n          \n            T\n          \n        \n        ]\n      \n    \n    {\\displaystyle S=E[xx^{T}]}\n  \n. Assume that \n  \n    \n      \n        E\n        [\n        x\n        ]\n        =\n        0\n      \n    \n    {\\displaystyle E[x]=0}\n  \n, and that the spectrum of the matrix", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0131", "text": "S\n      \n    \n    {\\displaystyle S}\n  \n is bounded as \n  \n    \n      \n        0\n        <\n        \u03bc\n        =\n        \n          \u03bb\n          \n            m\n            i\n            n\n          \n        \n        (\n        S\n        )\n      \n    \n    {\\displaystyle 0<\\mu =\\lambda _{min}(S)}\n  \n, \n  \n    \n      \n        L\n        =\n        \n          \u03bb\n          \n            m\n            a\n            x\n          \n        \n        (\n        S\n        )\n        <\n        \u221e", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0132", "text": "(\n        S\n        )\n        <\n        \u221e\n      \n    \n    {\\displaystyle L=\\lambda _{max}(S)<\\infty }\n  \n, such that \n  \n    \n      \n        S\n      \n    \n    {\\displaystyle S}\n  \n is symmetric positive definite. Adding batch normalization to this unit thus results in", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0133", "text": "f\n          \n            B\n            N\n          \n        \n        (\n        w\n        ,\n        \u03b3\n        ,\n        \u03b2\n        )\n        =\n        \n          E\n          \n            x\n          \n        \n        [\n        \u03d5\n        (\n        B\n        N\n        (\n        \n          x\n          \n            T\n          \n        \n        w\n        )\n        )\n        ]\n        =\n        \n          E\n          \n            x", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0134", "text": "x\n          \n        \n        \n          \n            [\n          \n        \n        \u03d5\n        \n          \n            (\n          \n        \n        \u03b3\n        (\n        \n          \n            \n              \n                x\n                \n                  T\n                \n              \n              w\n              \u2212\n              \n                E\n                \n                  x\n                \n              \n              [\n              \n                x", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0135", "text": "[\n              \n                x\n                \n                  T\n                \n              \n              w\n              ]\n            \n            \n              v\n              a\n              \n                r\n                \n                  x\n                \n              \n              [\n              \n                x\n                \n                  T\n                \n              \n              w\n              \n                ]", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0136", "text": "]\n                \n                  1\n                  \n                    /\n                  \n                  2\n                \n              \n            \n          \n        \n        )\n        +\n        \u03b2\n        \n          \n            )\n          \n        \n        \n          \n            ]", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0137", "text": "]\n          \n        \n      \n    \n    {\\displaystyle f_{BN}(w,\\gamma ,\\beta )=E_{x}[\\phi (BN(x^{T}w))]=E_{x}{\\bigg [}\\phi {\\bigg (}\\gamma ({\\frac {x^{T}w-E_{x}[x^{T}w]}{var_{x}[x^{T}w]^{1/2}}})+\\beta {\\bigg )}{\\bigg ]}}\n  \n, by definition.\nThe variance term can be simplified such that \n  \n    \n      \n        v\n        a\n        \n          r\n          \n            x\n          \n        \n        [\n        \n          x\n          \n            T\n          \n        \n        w\n        ]", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0138", "text": "w\n        ]\n        =\n        \n          w\n          \n            T\n          \n        \n        S\n        w\n      \n    \n    {\\displaystyle var_{x}[x^{T}w]=w^{T}Sw}\n  \n. Assume that \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n has zero mean and \n  \n    \n      \n        \u03b2\n      \n    \n    {\\displaystyle \\beta }\n  \n can be omitted, then it follows that", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0139", "text": "f\n          \n            B\n            N\n          \n        \n        (\n        w\n        ,\n        \u03b3\n        )\n        =\n        \n          E\n          \n            x\n          \n        \n        \n          \n            [\n          \n        \n        \u03d5\n        \n          \n            (\n          \n        \n        \u03b3\n        \n          \n            \n              \n                x\n                \n                  T", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0140", "text": "w\n            \n            \n              (\n              \n                w\n                \n                  T\n                \n              \n              S\n              w\n              \n                )\n                \n                  1\n                  \n                    /\n                  \n                  2\n                \n              \n            \n          \n        \n        \n          \n            )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0141", "text": ")\n          \n        \n        \n          \n            ]\n          \n        \n      \n    \n    {\\displaystyle f_{BN}(w,\\gamma )=E_{x}{\\bigg [}\\phi {\\bigg (}\\gamma {\\frac {x^{T}w}{(w^{T}Sw)^{1/2}}}{\\bigg )}{\\bigg ]}}\n  \n, where \n  \n    \n      \n        (\n        \n          w\n          \n            T\n          \n        \n        S\n        w\n        \n          )\n          \n            \n              1\n              2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0142", "text": "{\\displaystyle (w^{T}Sw)^{\\frac {1}{2}}}\n  \n is the induced norm of \n  \n    \n      \n        S\n      \n    \n    {\\displaystyle S}\n  \n, \n  \n    \n      \n        \n          |\n        \n        \n          |\n        \n        w\n        \n          |\n        \n        \n          \n            |\n          \n          \n            s\n          \n        \n      \n    \n    {\\displaystyle ||w||_{s}}\n  \n.\nHence, it could be concluded that", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0143", "text": "f\n          \n            B\n            N\n          \n        \n        (\n        w\n        ,\n        \u03b3\n        )\n        =\n        \n          E\n          \n            x\n          \n        \n        [\n        \u03d5\n        (\n        \n          x\n          \n            T\n          \n        \n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )\n        ]", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0144", "text": ")\n        ]\n      \n    \n    {\\displaystyle f_{BN}(w,\\gamma )=E_{x}[\\phi (x^{T}{\\tilde {w}})]}\n  \n, where \n  \n    \n      \n        \n          \n            \n              w\n              ~\n            \n          \n        \n        =\n        \u03b3\n        \n          \n            w\n            \n              \n                |\n              \n              \n                |\n              \n              w\n              \n                |", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0145", "text": "|\n                \n                \n                  s\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {w}}=\\gamma {\\frac {w}{||w||_{s}}}}\n  \n, and \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n and \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0146", "text": "w\n      \n    \n    {\\displaystyle w}\n  \n account for its length and direction separately. This property could then be used to prove the faster convergence of problems with batch normalization.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0147", "text": "Linear convergence\nLeast-square problem\nWith the reparametrization interpretation, it could then be proved that applying batch normalization to the ordinary least squares problem achieves a linear convergence rate in gradient descent, which is faster than the regular gradient descent with only sub-linear convergence.\nDenote the objective of minimizing an ordinary least squares problem as", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0148", "text": "min\n          \n            \n              \n                \n                  w\n                  ~\n                \n              \n            \n            \u2208\n            \n              R\n              \n                d\n              \n            \n          \n        \n        \n          f\n          \n            O\n            L\n            S\n          \n        \n        (\n        \n          \n            \n              w\n              ~", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0149", "text": "~\n            \n          \n        \n        )\n        =\n        \n          min\n          \n            \n              \n                \n                  w\n                  ~\n                \n              \n            \n            \u2208\n            \n              R\n              \n                d\n              \n            \n          \n        \n        (\n        \n          E\n          \n            x\n            ,\n            y\n          \n        \n        [\n        (\n        y", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0150", "text": "[\n        (\n        y\n        \u2212\n        \n          x\n          \n            T\n          \n        \n        \n          \n            \n              w\n              ~\n            \n          \n        \n        \n          )\n          \n            2\n          \n        \n        ]\n        )\n        =\n        \n          min\n          \n            \n              \n                \n                  w\n                  ~\n                \n              \n            \n            \u2208", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0151", "text": "\u2208\n            \n              R\n              \n                d\n              \n            \n          \n        \n        (\n        2\n        \n          u\n          \n            T\n          \n        \n        \n          \n            \n              w\n              ~\n            \n          \n        \n        +\n        \n          \n            \n              \n                w\n                ~\n              \n            \n          \n          \n            T", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0152", "text": "T\n          \n        \n        S\n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )\n      \n    \n    {\\displaystyle \\min _{{\\tilde {w}}\\in R^{d}}f_{OLS}({\\tilde {w}})=\\min _{{\\tilde {w}}\\in R^{d}}(E_{x,y}[(y-x^{T}{\\tilde {w}})^{2}])=\\min _{{\\tilde {w}}\\in R^{d}}(2u^{T}{\\tilde {w}}+{\\tilde {w}}^{T}S{\\tilde {w}})}\n  \n, where \n  \n    \n      \n        u\n        =\n        E\n        [\n        \u2212", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0153", "text": "u\n        =\n        E\n        [\n        \u2212\n        y\n        x\n        ]\n      \n    \n    {\\displaystyle u=E[-yx]}\n  \n and \n  \n    \n      \n        S\n        =\n        E\n        [\n        x\n        \n          x\n          \n            T\n          \n        \n        ]\n      \n    \n    {\\displaystyle S=E[xx^{T}]}\n  \n.\nSince \n  \n    \n      \n        \n          \n            \n              w\n              ~\n            \n          \n        \n        =\n        \u03b3\n        \n          \n            w", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0154", "text": "\u03b3\n        \n          \n            w\n            \n              \n                |\n              \n              \n                |\n              \n              w\n              \n                |\n              \n              \n                \n                  |\n                \n                \n                  s\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {w}}=\\gamma {\\frac {w}{||w||_{s}}}}\n  \n, the objective thus becomes", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0155", "text": "min\n          \n            w\n            \u2208\n            \n              R\n              \n                d\n              \n            \n            \u2216\n            {\n            0\n            }\n            ,\n            \u03b3\n            \u2208\n            R\n          \n        \n        \n          f\n          \n            O\n            L\n            S\n          \n        \n        (\n        w\n        ,\n        \u03b3\n        )\n        =\n        \n          min", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0156", "text": "=\n        \n          min\n          \n            w\n            \u2208\n            \n              R\n              \n                d\n              \n            \n            \u2216\n            {\n            0\n            }\n            ,\n            \u03b3\n            \u2208\n            R\n          \n        \n        \n          \n            (\n          \n        \n        2\n        \u03b3\n        \n          \n            \n              \n                u\n                \n                  T", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0157", "text": "T\n                \n              \n              w\n            \n            \n              \n                |\n              \n              \n                |\n              \n              w\n              \n                |\n              \n              \n                \n                  |\n                \n                \n                  S\n                \n              \n              +\n              \n                \u03b3\n                \n                  2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0158", "text": "2\n                \n              \n            \n          \n        \n        \n          \n            )\n          \n        \n      \n    \n    {\\displaystyle \\min _{w\\in R^{d}\\backslash \\{0\\},\\gamma \\in R}f_{OLS}(w,\\gamma )=\\min _{w\\in R^{d}\\backslash \\{0\\},\\gamma \\in R}{\\bigg (}2\\gamma {\\frac {u^{T}w}{||w||_{S}+\\gamma ^{2}}}{\\bigg )}}\n  \n, where 0 is excluded to avoid 0 in the denominator.\nSince the objective is convex with respect to \n  \n    \n      \n        \u03b3", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0159", "text": "\u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n, its optimal value could be calculated by setting the partial derivative of the objective against \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n to 0. The objective could be further simplified to be", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0160", "text": "min\n          \n            w\n            \u2208\n            \n              R\n              \n                d\n              \n            \n            \u2216\n            {\n            0\n            }\n          \n        \n        \u03c1\n        (\n        w\n        )\n        =\n        \n          min\n          \n            w\n            \u2208\n            \n              R\n              \n                d\n              \n            \n            \u2216\n            {\n            0", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0161", "text": "\u2216\n            {\n            0\n            }\n          \n        \n        \n          \n            (\n          \n        \n        \u2212\n        \n          \n            \n              \n                w\n                \n                  T\n                \n              \n              u\n              \n                u\n                \n                  T\n                \n              \n              w\n            \n            \n              \n                w", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0162", "text": "w\n                \n                  T\n                \n              \n              S\n              w\n            \n          \n        \n        \n          \n            )\n          \n        \n      \n    \n    {\\displaystyle \\min _{w\\in R^{d}\\backslash \\{0\\}}\\rho (w)=\\min _{w\\in R^{d}\\backslash \\{0\\}}{\\bigg (}-{\\frac {w^{T}uu^{T}w}{w^{T}Sw}}{\\bigg )}}\n  \n.\nNote that this objective is a form of the generalized Rayleigh quotient", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0163", "text": "\u03c1\n              ~\n            \n          \n        \n        (\n        w\n        )\n        =\n        \n          \n            \n              \n                w\n                \n                  T\n                \n              \n              B\n              w\n            \n            \n              \n                w\n                \n                  T\n                \n              \n              A\n              w", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0164", "text": "A\n              w\n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {\\rho }}(w)={\\frac {w^{T}Bw}{w^{T}Aw}}}\n  \n, where \n  \n    \n      \n        B\n        \u2208\n        \n          R\n          \n            d\n            \u00d7\n            d\n          \n        \n      \n    \n    {\\displaystyle B\\in R^{d\\times d}}\n  \n is a symmetric matrix and \n  \n    \n      \n        A\n        \u2208\n        \n          R\n          \n            d\n            \u00d7\n            d", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0165", "text": "\u00d7\n            d\n          \n        \n      \n    \n    {\\displaystyle A\\in R^{d\\times d}}\n  \n is a symmetric positive definite matrix.\nIt is proven that the gradient descent convergence rate of the generalized Rayleigh quotient is", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0166", "text": "\u03bb\n                \n                  1\n                \n              \n              \u2212\n              \u03c1\n              (\n              \n                w\n                \n                  t\n                  +\n                  1\n                \n              \n              )\n            \n            \n              \u03c1\n              (\n              \n                w\n                \n                  t", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0167", "text": "t\n                  +\n                  1\n                \n              \n              \u2212\n              \n                \u03bb\n                \n                  2\n                \n              \n              )\n            \n          \n        \n        \u2264\n        \n          \n            (\n          \n        \n        1\n        \u2212\n        \n          \n            \n              \n                \u03bb\n                \n                  1", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0168", "text": "\u2212\n              \n                \u03bb\n                \n                  2\n                \n              \n            \n            \n              \n                \u03bb\n                \n                  1\n                \n              \n              \u2212\n              \n                \u03bb\n                \n                  m\n                  i\n                  n", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0169", "text": ")\n            \n          \n          \n            2\n            t\n          \n        \n        \n          \n            \n              \n                \u03bb\n                \n                  1\n                \n              \n              \u2212\n              \u03c1\n              (\n              \n                w\n                \n                  t\n                \n              \n              )\n            \n            \n              \u03c1", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0170", "text": "\u03c1\n              (\n              \n                w\n                \n                  t\n                \n              \n              )\n              \u2212\n              \n                \u03bb\n                \n                  2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0171", "text": "{\\displaystyle {\\frac {\\lambda _{1}-\\rho (w_{t+1})}{\\rho (w_{t+1}-\\lambda _{2})}}\\leq {\\bigg (}1-{\\frac {\\lambda _{1}-\\lambda _{2}}{\\lambda _{1}-\\lambda _{min}}}{\\bigg )}^{2t}{\\frac {\\lambda _{1}-\\rho (w_{t})}{\\rho (w_{t})-\\lambda _{2}}}}\n  \n, where \n  \n    \n      \n        \n          \u03bb\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{1}}\n  \n is the largest eigenvalue of \n  \n    \n      \n        B", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0172", "text": "B\n      \n    \n    {\\displaystyle B}\n  \n, \n  \n    \n      \n        \n          \u03bb\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{2}}\n  \n is the second largest eigenvalue of \n  \n    \n      \n        B\n      \n    \n    {\\displaystyle B}\n  \n, and \n  \n    \n      \n        \n          \u03bb\n          \n            m\n            i\n            n\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{min}}\n  \n is the smallest eigenvalue of", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0173", "text": "is the smallest eigenvalue of \n  \n    \n      \n        B\n      \n    \n    {\\displaystyle B}\n  \n.\nIn our case, \n  \n    \n      \n        B\n        =\n        u\n        \n          u\n          \n            T\n          \n        \n      \n    \n    {\\displaystyle B=uu^{T}}\n  \n is a rank one matrix, and the convergence result can be simplified accordingly. Specifically, consider gradient descent steps of the form \n  \n    \n      \n        \n          w\n          \n            t\n            +\n            1", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0174", "text": "t\n            +\n            1\n          \n        \n        =\n        \n          w\n          \n            t\n          \n        \n        \u2212\n        \n          \u03b7\n          \n            t\n          \n        \n        \u25bd\n        \u03c1\n        (\n        \n          w\n          \n            t\n          \n        \n        )\n      \n    \n    {\\displaystyle w_{t+1}=w_{t}-\\eta _{t}\\triangledown \\rho (w_{t})}\n  \n with step size \n  \n    \n      \n        \n          \u03b7\n          \n            t", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0175", "text": "\u03b7\n          \n            t\n          \n        \n        =\n        \n          \n            \n              \n                w\n                \n                  t\n                \n                \n                  T\n                \n              \n              S\n              \n                w\n                \n                  t\n                \n              \n            \n            \n              2\n              L\n              \n                |\n              \n              \u03c1", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0176", "text": "|\n              \n              \u03c1\n              (\n              \n                w\n                \n                  t\n                \n              \n              )\n              \n                |\n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\eta _{t}={\\frac {w_{t}^{T}Sw_{t}}{2L|\\rho (w_{t})|}}}\n  \n, and starting from \n  \n    \n      \n        \u03c1\n        (\n        \n          w\n          \n            0\n          \n        \n        )\n        \u2260", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0177", "text": ")\n        \u2260\n        0\n      \n    \n    {\\displaystyle \\rho (w_{0})\\neq 0}\n  \n, then", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0178", "text": "\u03c1\n        (\n        \n          w\n          \n            t\n          \n        \n        )\n        \u2212\n        \u03c1\n        (\n        \n          w\n          \n            \u2217\n          \n        \n        )\n        \u2264\n        \n          \n            (\n          \n        \n        1\n        \u2212\n        \n          \n            \u03bc\n            L\n          \n        \n        \n          \n            \n              )\n            \n          \n          \n            2\n            t", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0179", "text": "2\n            t\n          \n        \n        (\n        \u03c1\n        (\n        \n          w\n          \n            0\n          \n        \n        )\n        \u2212\n        \u03c1\n        (\n        \n          w\n          \n            \u2217\n          \n        \n        )\n        )\n      \n    \n    {\\displaystyle \\rho (w_{t})-\\rho (w^{*})\\leq {\\bigg (}1-{\\frac {\\mu }{L}}{\\bigg )}^{2t}(\\rho (w_{0})-\\rho (w^{*}))}\n  \n.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0180", "text": "Learning halfspace problem\nThe problem of learning halfspaces refers to the training of the Perceptron, which is the simplest form of neural network. The optimization problem in this case is", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0181", "text": "min\n          \n            \n              \n                \n                  w\n                  ~\n                \n              \n            \n            \u2208\n            \n              R\n              \n                d\n              \n            \n          \n        \n        \n          f\n          \n            L\n            H\n          \n        \n        (\n        \n          \n            \n              w\n              ~", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0182", "text": "~\n            \n          \n        \n        )\n        =\n        \n          E\n          \n            y\n            ,\n            x\n          \n        \n        [\n        \u03d5\n        (\n        \n          z\n          \n            T\n          \n        \n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )\n        ]\n      \n    \n    {\\displaystyle \\min _{{\\tilde {w}}\\in R^{d}}f_{LH}({\\tilde {w}})=E_{y,x}[\\phi (z^{T}{\\tilde {w}})]}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0183", "text": ", where \n  \n    \n      \n        z\n        =\n        \u2212\n        y\n        x\n      \n    \n    {\\displaystyle z=-yx}\n  \n and \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n is an arbitrary loss function.\nSuppose that \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n is infinitely differentiable and has a bounded derivative. Assume that the objective function \n  \n    \n      \n        \n          f\n          \n            L\n            H", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0184", "text": "H\n          \n        \n      \n    \n    {\\displaystyle f_{LH}}\n  \n is \n  \n    \n      \n        \u03b6\n      \n    \n    {\\displaystyle \\zeta }\n  \n-smooth, and that a solution \n  \n    \n      \n        \n          \u03b1\n          \n            \u2217\n          \n        \n        =\n        a\n        r\n        g\n        m\n        i\n        \n          n\n          \n            \u03b1\n          \n        \n        \n          |\n        \n        \n          |\n        \n        \u25bd\n        f\n        (\n        \u03b1\n        w", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0185", "text": "\u25bd\n        f\n        (\n        \u03b1\n        w\n        )\n        \n          |\n        \n        \n          \n            |\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\alpha ^{*}=argmin_{\\alpha }||\\triangledown f(\\alpha w)||^{2}}\n  \n exists and is bounded such that \n  \n    \n      \n        \u2212\n        \u221e\n        <\n        \n          \u03b1\n          \n            \u2217\n          \n        \n        <\n        \u221e", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0186", "text": "<\n        \u221e\n      \n    \n    {\\displaystyle -\\infty <\\alpha ^{*}<\\infty }\n  \n. Also assume \n  \n    \n      \n        z\n      \n    \n    {\\displaystyle z}\n  \n is a multivariate normal random variable. With the Gaussian assumption, it can be shown that all critical points lie on the same line, for any choice of loss function \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n. Specifically, the gradient of \n  \n    \n      \n        \n          f\n          \n            L", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0187", "text": "f\n          \n            L\n            H\n          \n        \n      \n    \n    {\\displaystyle f_{LH}}\n  \n could be represented as", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0188", "text": "\u25bd\n          \n            \n              \n                w\n                ~\n              \n            \n          \n        \n        \n          f\n          \n            L\n            H\n          \n        \n        (\n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )\n        =\n        \n          c\n          \n            1\n          \n        \n        (\n        \n          \n            \n              w", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0189", "text": "w\n              ~\n            \n          \n        \n        )\n        u\n        +\n        \n          c\n          \n            2\n          \n        \n        (\n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )\n        S\n        \n          \n            \n              w\n              ~", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0190", "text": "{\\displaystyle \\triangledown _{\\tilde {w}}f_{LH}({\\tilde {w}})=c_{1}({\\tilde {w}})u+c_{2}({\\tilde {w}})S{\\tilde {w}}}\n  \n, where  \n  \n    \n      \n        \n          c\n          \n            1\n          \n        \n        (\n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )\n        =\n        \n          E\n          \n            z\n          \n        \n        [\n        \n          \u03d5", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0191", "text": "[\n        \n          \u03d5\n          \n            (\n            1\n            )\n          \n        \n        (\n        \n          z\n          \n            T\n          \n        \n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )\n        ]\n        \u2212\n        \n          E\n          \n            z\n          \n        \n        [\n        \n          \u03d5\n          \n            (\n            2\n            )\n          \n        \n        (", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0192", "text": "(\n        \n          z\n          \n            T\n          \n        \n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )\n        ]\n        (\n        \n          u\n          \n            T\n          \n        \n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0193", "text": ")\n      \n    \n    {\\displaystyle c_{1}({\\tilde {w}})=E_{z}[\\phi ^{(1)}(z^{T}{\\tilde {w}})]-E_{z}[\\phi ^{(2)}(z^{T}{\\tilde {w}})](u^{T}{\\tilde {w}})}\n  \n, \n  \n    \n      \n        \n          c\n          \n            2\n          \n        \n        (\n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )\n        =\n        \n          E\n          \n            z\n          \n        \n        [\n        \n          \u03d5", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0194", "text": "[\n        \n          \u03d5\n          \n            (\n            2\n            )\n          \n        \n        (\n        \n          z\n          \n            T\n          \n        \n        \n          \n            \n              w\n              ~\n            \n          \n        \n        )\n        ]\n      \n    \n    {\\displaystyle c_{2}({\\tilde {w}})=E_{z}[\\phi ^{(2)}(z^{T}{\\tilde {w}})]}\n  \n, and \n  \n    \n      \n        \n          \u03d5\n          \n            (\n            i\n            )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0195", "text": "i\n            )\n          \n        \n      \n    \n    {\\displaystyle \\phi ^{(i)}}\n  \n is the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n-th derivative of \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n.\nBy setting the gradient to 0, it thus follows that the bounded critical points \n  \n    \n      \n        \n          \n            \n              \n                w\n                ~\n              \n            \n          \n          \n            \u2217", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0196", "text": "\u2217\n          \n        \n      \n    \n    {\\displaystyle {\\tilde {w}}_{*}}\n  \n can be expressed as \n  \n    \n      \n        \n          \n            \n              \n                w\n                ~\n              \n            \n          \n          \n            \u2217\n          \n        \n        =\n        \n          g\n          \n            \u2217\n          \n        \n        \n          S\n          \n            \u2212\n            1\n          \n        \n        u", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0197", "text": "u\n      \n    \n    {\\displaystyle {\\tilde {w}}_{*}=g_{*}S^{-1}u}\n  \n, where \n  \n    \n      \n        \n          g\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle g_{*}}\n  \n depends on \n  \n    \n      \n        \n          \n            \n              \n                w\n                ~\n              \n            \n          \n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle {\\tilde {w}}_{*}}\n  \n and", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0198", "text": "and \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n. Combining this global property with length-direction decoupling, it could thus be proved that this optimization problem converges linearly.\nFirst, a variation of gradient descent with batch normalization, Gradient Descent in Normalized Parameterization (GDNP), is designed for the objective function \n  \n    \n      \n        \n          min\n          \n            w\n            \u2208\n            \n              R", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0199", "text": "R\n              \n                d\n              \n            \n            \u2216\n            {\n            0\n            }\n            ,\n            \u03b3\n            \u2208\n            R\n          \n        \n        \n          f\n          \n            L\n            H\n          \n        \n        (\n        w\n        ,\n        \u03b3\n        )\n      \n    \n    {\\displaystyle \\min _{w\\in R^{d}\\backslash \\{0\\},\\gamma \\in R}f_{LH}(w,\\gamma )}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0200", "text": ", such that the direction and length of the weights are updated separately. Denote the stopping criterion of GDNP as", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0201", "text": "h\n        (\n        \n          w\n          \n            t\n          \n        \n        ,\n        \n          \u03b3\n          \n            t\n          \n        \n        )\n        =\n        \n          E\n          \n            z\n          \n        \n        [\n        \n          \u03d5\n          \u2032\n        \n        (\n        \n          z\n          \n            T\n          \n        \n        \n          \n            \n              \n                w\n                ~", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0202", "text": "~\n              \n            \n          \n          \n            t\n          \n        \n        )\n        ]\n        (\n        \n          u\n          \n            T\n          \n        \n        \n          w\n          \n            t\n          \n        \n        )\n        \u2212\n        \n          E\n          \n            z\n          \n        \n        [\n        \n          \u03d5\n          \u2033\n        \n        (\n        \n          z\n          \n            T", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0203", "text": "w\n                ~\n              \n            \n          \n          \n            t\n          \n        \n        )\n        ]\n        (\n        \n          u\n          \n            T\n          \n        \n        \n          w\n          \n            t\n          \n        \n        \n          )\n          \n            2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0204", "text": "2\n          \n        \n      \n    \n    {\\displaystyle h(w_{t},\\gamma _{t})=E_{z}[\\phi '(z^{T}{\\tilde {w}}_{t})](u^{T}w_{t})-E_{z}[\\phi ''(z^{T}{\\tilde {w}}_{t})](u^{T}w_{t})^{2}}\n  \n.\nLet the step size be", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0205", "text": "s\n          \n            t\n          \n        \n        =\n        s\n        (\n        \n          w\n          \n            t\n          \n        \n        ,\n        \n          \u03b3\n          \n            t\n          \n        \n        )\n        =\n        \u2212\n        \n          \n            \n              \n                |\n              \n              \n                |\n              \n              \n                w\n                \n                  t", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0206", "text": "t\n                \n              \n              \n                |\n              \n              \n                \n                  |\n                \n                \n                  S\n                \n                \n                  3\n                \n              \n            \n            \n              L\n              \n                g\n                \n                  t\n                \n              \n              h\n              (", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0207", "text": "h\n              (\n              \n                w\n                \n                  t\n                \n              \n              ,\n              \n                \u03b3\n                \n                  t\n                \n              \n              )\n            \n          \n        \n      \n    \n    {\\displaystyle s_{t}=s(w_{t},\\gamma _{t})=-{\\frac {||w_{t}||_{S}^{3}}{Lg_{t}h(w_{t},\\gamma _{t})}}}\n  \n.\nFor each step, if \n  \n    \n      \n        h\n        (\n        \n          w", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0208", "text": "h\n        (\n        \n          w\n          \n            t\n          \n        \n        ,\n        \n          \u03b3\n          \n            t\n          \n        \n        )\n        \u2260\n        0\n      \n    \n    {\\displaystyle h(w_{t},\\gamma _{t})\\neq 0}\n  \n, then update the direction as", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0209", "text": "w\n          \n            t\n            +\n            1\n          \n        \n        =\n        \n          w\n          \n            t\n          \n        \n        \u2212\n        \n          s\n          \n            t\n          \n        \n        \n          \u25bd\n          \n            w\n          \n        \n        f\n        (\n        \n          w\n          \n            t\n          \n        \n        ,\n        \n          \u03b3\n          \n            t\n          \n        \n        )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0210", "text": "t\n          \n        \n        )\n      \n    \n    {\\displaystyle w_{t+1}=w_{t}-s_{t}\\triangledown _{w}f(w_{t},\\gamma _{t})}\n  \n.\nThen update the length according to", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0211", "text": "\u03b3\n          \n            t\n          \n        \n        =\n        \n          Bisection\n        \n        (\n        \n          T\n          \n            s\n          \n        \n        ,\n        f\n        ,\n        \n          w\n          \n            t\n          \n        \n        )\n      \n    \n    {\\displaystyle \\gamma _{t}={\\text{Bisection}}(T_{s},f,w_{t})}\n  \n, where \n  \n    \n      \n        \n          Bisection()", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0212", "text": "Bisection()\n        \n      \n    \n    {\\displaystyle {\\text{Bisection()}}}\n  \n is the classical bisection algorithm, and \n  \n    \n      \n        \n          T\n          \n            s\n          \n        \n      \n    \n    {\\displaystyle T_{s}}\n  \n is the total iterations ran in the bisection step.\nDenote the total number of iterations as \n  \n    \n      \n        \n          T\n          \n            d\n          \n        \n      \n    \n    {\\displaystyle T_{d}}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0213", "text": "{\\displaystyle T_{d}}\n  \n, then the final output of GDNP is", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0214", "text": "w\n                ~\n              \n            \n          \n          \n            \n              T\n              \n                d\n              \n            \n          \n        \n        =\n        \n          \u03b3\n          \n            \n              T\n              \n                d\n              \n            \n          \n        \n        \n          \n            \n              w", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0215", "text": "w\n              \n                \n                  T\n                  \n                    d\n                  \n                \n              \n            \n            \n              \n                |\n              \n              \n                |\n              \n              \n                w\n                \n                  \n                    T\n                    \n                      d", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0216", "text": "|\n              \n              \n                \n                  |\n                \n                \n                  S\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {w}}_{T_{d}}=\\gamma _{T_{d}}{\\frac {w_{T_{d}}}{||w_{T_{d}}||_{S}}}}\n  \n.\nThe GDNP algorithm thus slightly modifies the batch normalization step for the ease of mathematical analysis.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0217", "text": "It can be shown that in GDNP, the partial derivative of \n  \n    \n      \n        \n          f\n          \n            L\n            H\n          \n        \n      \n    \n    {\\displaystyle f_{LH}}\n  \n against the length component converges to zero at a linear rate, such that", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0218", "text": "(\n        \n          \u2202\n          \n            \u03b3\n          \n        \n        \n          f\n          \n            L\n            H\n          \n        \n        (\n        \n          w\n          \n            t\n          \n        \n        ,\n        \n          a\n          \n            t\n          \n          \n            (\n            \n              T\n              \n                s\n              \n            \n            )\n          \n        \n        \n          )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0219", "text": ")\n          \n            2\n          \n        \n        \u2264\n        \n          \n            \n              \n                2\n                \n                  \u2212\n                  \n                    T\n                    \n                      s\n                    \n                  \n                \n              \n              \u03b6\n              \n                |\n              \n              \n                b\n                \n                  t", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0220", "text": "t\n                \n                \n                  (\n                  0\n                  )\n                \n              \n              \u2212\n              \n                a\n                \n                  t\n                \n                \n                  (\n                  0\n                  )\n                \n              \n              \n                |\n              \n            \n            \n              \u03bc\n              \n                2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0221", "text": "2\n              \n            \n          \n        \n      \n    \n    {\\displaystyle (\\partial _{\\gamma }f_{LH}(w_{t},a_{t}^{(T_{s})})^{2}\\leq {\\frac {2^{-T_{s}}\\zeta |b_{t}^{(0)}-a_{t}^{(0)}|}{\\mu ^{2}}}}\n  \n, where \n  \n    \n      \n        \n          a\n          \n            t\n          \n          \n            (\n            0\n            )\n          \n        \n      \n    \n    {\\displaystyle a_{t}^{(0)}}\n  \n and \n  \n    \n      \n        \n          b", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0222", "text": "b\n          \n            t\n          \n          \n            0\n          \n        \n      \n    \n    {\\displaystyle b_{t}^{0}}\n  \n are the two starting points of the bisection algorithm on the left and on the right, correspondingly.\nFurther, for each iteration, the norm of the gradient of \n  \n    \n      \n        \n          f\n          \n            L\n            H\n          \n        \n      \n    \n    {\\displaystyle f_{LH}}\n  \n with respect to", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0223", "text": "with respect to \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n converges linearly, such that", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0224", "text": "|\n        \n        \n          |\n        \n        \n          w\n          \n            t\n          \n        \n        \n          |\n        \n        \n          \n            |\n          \n          \n            S\n          \n          \n            2\n          \n        \n        \n          |\n        \n        \n          |\n        \n        \u25bd\n        \n          f\n          \n            L\n            H\n          \n        \n        (\n        \n          w", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0225", "text": "(\n        \n          w\n          \n            t\n          \n        \n        ,\n        \n          g\n          \n            t\n          \n        \n        )\n        \n          |\n        \n        \n          \n            |\n          \n          \n            \n              S\n              \n                \u2212\n                1\n              \n            \n          \n          \n            2\n          \n        \n        \u2264\n        \n          \n            (\n          \n        \n        1\n        \u2212", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0226", "text": "1\n        \u2212\n        \n          \n            \u03bc\n            L\n          \n        \n        \n          \n            \n              )\n            \n          \n          \n            2\n            t\n          \n        \n        \n          \u03a6\n          \n            2\n          \n        \n        \n          \u03b3\n          \n            t\n          \n          \n            2\n          \n        \n        (\n        \u03c1\n        (\n        \n          w\n          \n            0", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0227", "text": "w\n          \n            0\n          \n        \n        )\n        \u2212\n        \n          \u03c1\n          \n            \u2217\n          \n        \n        )\n      \n    \n    {\\displaystyle ||w_{t}||_{S}^{2}||\\triangledown f_{LH}(w_{t},g_{t})||_{S^{-1}}^{2}\\leq {\\bigg (}1-{\\frac {\\mu }{L}}{\\bigg )}^{2t}\\Phi ^{2}\\gamma _{t}^{2}(\\rho (w_{0})-\\rho ^{*})}\n  \n.\nCombining these two inequalities, a bound could thus be obtained for the gradient with respect to", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0228", "text": "w\n                ~\n              \n            \n          \n          \n            \n              T\n              \n                d\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {w}}_{T_{d}}}\n  \n:", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0229", "text": "|\n        \n        \n          |\n        \n        \n          \u25bd\n          \n            \n              \n                w\n                ~\n              \n            \n          \n        \n        f\n        (\n        \n          \n            \n              \n                w\n                ~\n              \n            \n          \n          \n            \n              T\n              \n                d\n              \n            \n          \n        \n        )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0230", "text": ")\n        \n          |\n        \n        \n          \n            |\n          \n          \n            2\n          \n        \n        \u2264\n        \n          \n            (\n          \n        \n        1\n        \u2212\n        \n          \n            \u03bc\n            L\n          \n        \n        \n          \n            \n              )\n            \n          \n          \n            2\n            \n              T\n              \n                d", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0231", "text": "d\n              \n            \n          \n        \n        \n          \u03a6\n          \n            2\n          \n        \n        (\n        \u03c1\n        (\n        \n          w\n          \n            0\n          \n        \n        )\n        \u2212\n        \n          \u03c1\n          \n            \u2217\n          \n        \n        )\n        +\n        \n          \n            \n              \n                2\n                \n                  \u2212\n                  \n                    T", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0232", "text": "T\n                    \n                      s\n                    \n                  \n                \n              \n              \u03b6\n              \n                |\n              \n              \n                b\n                \n                  t\n                \n                \n                  (\n                  0\n                  )\n                \n              \n              \u2212\n              \n                a", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0233", "text": "a\n                \n                  t\n                \n                \n                  (\n                  0\n                  )\n                \n              \n              \n                |\n              \n            \n            \n              \u03bc\n              \n                2", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0234", "text": "{\\displaystyle ||\\triangledown _{\\tilde {w}}f({\\tilde {w}}_{T_{d}})||^{2}\\leq {\\bigg (}1-{\\frac {\\mu }{L}}{\\bigg )}^{2T_{d}}\\Phi ^{2}(\\rho (w_{0})-\\rho ^{*})+{\\frac {2^{-T_{s}}\\zeta |b_{t}^{(0)}-a_{t}^{(0)}|}{\\mu ^{2}}}}\n  \n, such that the algorithm is guaranteed to converge linearly.\nAlthough the proof stands on the assumption of Gaussian input, it is also shown in experiments that GDNP could accelerate optimization without this constraint.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0235", "text": "Neural networks\nConsider a multilayer perceptron (MLP) with one hidden layer and \n  \n    \n      \n        m\n      \n    \n    {\\displaystyle m}\n  \n hidden units with mapping from input \n  \n    \n      \n        x\n        \u2208\n        \n          R\n          \n            d\n          \n        \n      \n    \n    {\\displaystyle x\\in R^{d}}\n  \n to a scalar output described as", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0236", "text": "F\n          \n            x\n          \n        \n        (\n        \n          \n            \n              W\n              ~\n            \n          \n        \n        ,\n        \u0398\n        )\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            m\n          \n        \n        \n          \u03b8\n          \n            i\n          \n        \n        \u03d5\n        (\n        \n          x\n          \n            T", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0237", "text": "x\n          \n            T\n          \n        \n        \n          \n            \n              \n                w\n                ~\n              \n            \n          \n          \n            (\n            i\n            )\n          \n        \n        )\n      \n    \n    {\\displaystyle F_{x}({\\tilde {W}},\\Theta )=\\sum _{i=1}^{m}\\theta _{i}\\phi (x^{T}{\\tilde {w}}^{(i)})}\n  \n, where \n  \n    \n      \n        \n          \n            \n              \n                w\n                ~", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0238", "text": "w\n                ~\n              \n            \n          \n          \n            (\n            i\n            )\n          \n        \n      \n    \n    {\\displaystyle {\\tilde {w}}^{(i)}}\n  \n and \n  \n    \n      \n        \n          \u03b8\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\theta _{i}}\n  \n are the input and output weights of unit \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n correspondingly, and \n  \n    \n      \n        \u03d5", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0239", "text": "\u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n is the activation function and is assumed to be a tanh function.\nThe input and output weights could then be optimized with", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0240", "text": "min\n          \n            \n              \n                \n                  W\n                  ~\n                \n              \n            \n            ,\n            \u0398\n          \n        \n        (\n        \n          f\n          \n            N\n            N\n          \n        \n        (\n        \n          \n            \n              W\n              ~\n            \n          \n        \n        ,\n        \u0398\n        )\n        =\n        \n          E", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0241", "text": "=\n        \n          E\n          \n            y\n            ,\n            x\n          \n        \n        [\n        l\n        (\n        \u2212\n        y\n        \n          F\n          \n            x\n          \n        \n        (\n        \n          \n            \n              W\n              ~\n            \n          \n        \n        ,\n        \u0398\n        )\n        )\n        ]\n        )", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0242", "text": ")\n        ]\n        )\n      \n    \n    {\\displaystyle \\min _{{\\tilde {W}},\\Theta }(f_{NN}({\\tilde {W}},\\Theta )=E_{y,x}[l(-yF_{x}({\\tilde {W}},\\Theta ))])}\n  \n, where \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n is a loss function, \n  \n    \n      \n        \n          \n            \n              W\n              ~\n            \n          \n        \n        =\n        {\n        \n          \n            \n              \n                w\n                ~", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0243", "text": "~\n              \n            \n          \n          \n            (\n            1\n            )\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          \n            \n              \n                w\n                ~\n              \n            \n          \n          \n            (\n            m\n            )\n          \n        \n        }\n      \n    \n    {\\displaystyle {\\tilde {W}}=\\{{\\tilde {w}}^{(1)},...,{\\tilde {w}}^{(m)}\\}}\n  \n, and", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0244", "text": ", and \n  \n    \n      \n        \u0398\n        =\n        {\n        \n          \u03b8\n          \n            (\n            1\n            )\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          \u03b8\n          \n            (\n            m\n            )\n          \n        \n        }\n      \n    \n    {\\displaystyle \\Theta =\\{\\theta ^{(1)},...,\\theta ^{(m)}\\}}\n  \n.\nConsider fixed \n  \n    \n      \n        \u0398\n      \n    \n    {\\displaystyle \\Theta }\n  \n and optimizing only", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0245", "text": "and optimizing only \n  \n    \n      \n        \n          \n            \n              W\n              ~\n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {W}}}\n  \n, it can be shown that the critical points of \n  \n    \n      \n        \n          f\n          \n            N\n            N\n          \n        \n        (\n        \n          \n            \n              W\n              ~\n            \n          \n        \n        )\n      \n    \n    {\\displaystyle f_{NN}({\\tilde {W}})}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0246", "text": "{\\displaystyle f_{NN}({\\tilde {W}})}\n  \n of a particular hidden unit \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n, \n  \n    \n      \n        \n          \n            \n              \n                w\n                ^\n              \n            \n          \n          \n            (\n            i\n            )\n          \n        \n      \n    \n    {\\displaystyle {\\hat {w}}^{(i)}}", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0247", "text": "{\\displaystyle {\\hat {w}}^{(i)}}\n  \n, all align along one line depending on incoming information into the hidden layer, such that", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0248", "text": "w\n                ^\n              \n            \n          \n          \n            (\n            i\n            )\n          \n        \n        =\n        \n          \n            \n              \n                c\n                ^\n              \n            \n          \n          \n            (\n            i\n            )\n          \n        \n        \n          S\n          \n            \u2212\n            1", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0249", "text": "\u2212\n            1\n          \n        \n        u\n      \n    \n    {\\displaystyle {\\hat {w}}^{(i)}={\\hat {c}}^{(i)}S^{-1}u}\n  \n, where \n  \n    \n      \n        \n          \n            \n              \n                c\n                ^\n              \n            \n          \n          \n            (\n            i\n            )\n          \n        \n        \u2208\n        R\n      \n    \n    {\\displaystyle {\\hat {c}}^{(i)}\\in R}\n  \n is a scalar, \n  \n    \n      \n        i\n        =\n        1", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0250", "text": "i\n        =\n        1\n        ,\n        .\n        .\n        .\n        ,\n        m\n      \n    \n    {\\displaystyle i=1,...,m}\n  \n.\nThis result could be proved by setting the gradient of \n  \n    \n      \n        \n          f\n          \n            N\n            N\n          \n        \n      \n    \n    {\\displaystyle f_{NN}}\n  \n to zero and solving the system of equations.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0251", "text": "to zero and solving the system of equations.\nApply the GDNP algorithm to this optimization problem by alternating optimization over the different hidden units. Specifically, for each hidden unit, run GDNP to find the optimal \n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}\n  \n and \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n. With the same choice of stopping criterion and stepsize, it follows that", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0252", "text": "|\n        \n        \n          |\n        \n        \n          \u25bd\n          \n            \n              \n                \n                  \n                    w\n                    ~\n                  \n                \n              \n              \n                (\n                i\n                )\n              \n            \n          \n        \n        f\n        (\n        \n          \n            \n              \n                w\n                ~", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0253", "text": "w\n                ~\n              \n            \n          \n          \n            t\n          \n          \n            (\n            i\n            )\n          \n        \n        )\n        \n          |\n        \n        \n          \n            |\n          \n          \n            \n              S\n              \n                \u2212\n                1\n              \n            \n          \n          \n            2\n          \n        \n        \u2264\n        \n          \n            (", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0254", "text": "(\n          \n        \n        1\n        \u2212\n        \n          \n            \u03bc\n            L\n          \n        \n        \n          \n            \n              )\n            \n          \n          \n            2\n            t\n          \n        \n        C\n        (\n        \u03c1\n        (\n        \n          w\n          \n            0\n          \n        \n        )\n        \u2212\n        \n          \u03c1\n          \n            \u2217\n          \n        \n        )\n        +", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0255", "text": ")\n        +\n        \n          \n            \n              \n                2\n                \n                  \u2212\n                  \n                    T\n                    \n                      s\n                    \n                    \n                      (\n                      i\n                      )\n                    \n                  \n                \n              \n              \u03b6\n              \n                |", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0256", "text": "|\n              \n              \n                b\n                \n                  t\n                \n                \n                  (\n                  0\n                  )\n                \n              \n              \u2212\n              \n                a\n                \n                  t\n                \n                \n                  (\n                  0\n                  )\n                \n              \n              \n                |", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0257", "text": "|\n              \n            \n            \n              \u03bc\n              \n                2\n              \n            \n          \n        \n      \n    \n    {\\displaystyle ||\\triangledown _{{\\tilde {w}}^{(i)}}f({\\tilde {w}}_{t}^{(i)})||_{S^{-1}}^{2}\\leq {\\bigg (}1-{\\frac {\\mu }{L}}{\\bigg )}^{2t}C(\\rho (w_{0})-\\rho ^{*})+{\\frac {2^{-T_{s}^{(i)}}\\zeta |b_{t}^{(0)}-a_{t}^{(0)}|}{\\mu ^{2}}}}\n  \n.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0258", "text": ".\nSince the parameters of each hidden unit converge linearly, the whole optimization problem has a linear rate of convergence.", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Batch_normalization_0259", "text": "References\nFurther reading\nIoffe, Sergey; Szegedy, Christian (2015). \"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift\", ICML'15: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, July 2015 Pages 448\u2013456\nSimonyan, Karen; Zisserman, Andrew (2014). \"Very Deep Convolutional Networks for Large-Scale Image Recognition\". arXiv:1409.1556 [cs.CV].", "title": "Batch normalization", "url": "https://en.wikipedia.org/wiki/Batch_normalization"}, {"id": "rec_Dilution_neural_networks_0000", "text": "Dropout and dilution (also called DropConnect) are regularization techniques for reducing overfitting in artificial neural networks by preventing complex co-adaptations on training data. They are an efficient way of performing model averaging with neural networks. Dilution refers to randomly decreasing weights towards zero, while dropout refers to randomly setting the outputs of hidden neurons to zero", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0001", "text": ". Both are usually performed during the training process of a neural network, not during inference.", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0002", "text": "Types and uses\nDilution is usually split in weak dilution and strong dilution. Weak dilution describes the process in which the finite fraction of removed connections is small, and strong dilution refers to when this fraction is large. There is no clear distinction on where the limit between strong and weak dilution is, and often the distinction is dependent on the precedent of a specific use-case and has implications for how to solve for exact solutions.", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0003", "text": "Sometimes dilution is used for adding damping noise to the inputs. In that case, weak dilution refers to adding a small amount of damping noise, while strong dilution refers to adding a greater amount of damping noise. Both can be rewritten as variants of weight dilution.", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0004", "text": "These techniques are also sometimes referred to as random pruning of weights, but this is usually a non-recurring one-way operation. The network is pruned, and then kept if it is an improvement over the previous model. Dilution and dropout both refer to an iterative process. The pruning of weights typically does not imply that the network continues learning, while in dilution/dropout, the network continues to learn after the technique is applied.", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0005", "text": "Generalized linear network\nOutput from a layer of linear nodes, in an artificial neural net can be described as\n\n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}\n  \n \u2013 output from node \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0006", "text": "w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n \u2013 real weight before dilution, also called the Hebb connection strength\n\n  \n    \n      \n        \n          x\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle x_{j}}\n  \n \u2013 input from node \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n\nThis can be written in vector notation as", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0007", "text": "This can be written in vector notation as\n\n  \n    \n      \n        \n          y\n        \n      \n    \n    {\\displaystyle \\mathbf {y} }\n  \n \u2013 output vector\n\n  \n    \n      \n        \n          W\n        \n      \n    \n    {\\displaystyle \\mathbf {W} }\n  \n \u2013 weight matrix\n\n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n \u2013 input vector\nEquations (1) and (2) are used in the subsequent sections.", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0008", "text": "Weak dilution\nDuring weak dilution, the finite fraction of removed connections (the weights) is small, giving rise to a tiny uncertainty. This edge-case can be solved exactly with mean field theory. In weak dilution the impact on the weights can be described as", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0009", "text": "w\n                ^\n              \n            \n          \n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle {\\hat {w}}_{ij}}\n  \n \u2013 diluted weight\n\n  \n    \n      \n        \n          w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n \u2013 real weight before dilution", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0010", "text": "P\n        (\n        c\n        )\n      \n    \n    {\\displaystyle P(c)}\n  \n \u2013 the probability of \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n, the probability of keeping a weight\nThe interpretation of probability \n  \n    \n      \n        P\n        (\n        c\n        )\n      \n    \n    {\\displaystyle P(c)}\n  \n can also be changed from keeping a weight into pruning a weight.\nIn vector notation this can be written as", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0011", "text": "where the function \n  \n    \n      \n        g\n        \u2061\n        (\n        \u22c5\n        )\n      \n    \n    {\\displaystyle \\operatorname {g} (\\cdot )}\n  \n imposes the previous dilution.\nIn weak dilution only a small and fixed fraction of the weights are diluted. When the number of terms in the sum goes to infinite (the weights for each node) it is still infinite (the fraction is fixed), thus mean field theory can be applied. In the notation from Hertz et al. this would be written as", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0012", "text": "\u27e8\n          \n            h\n            \n              i\n            \n          \n          \u27e9\n        \n      \n    \n    {\\displaystyle \\left\\langle h_{i}\\right\\rangle }\n  \n the mean field temperature\n\n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n \u2013 a scaling factor for the temperature from the probability of keeping the weight", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0013", "text": "w\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle w_{ij}}\n  \n \u2013 real weight before dilution, also called the Hebb connection strength", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0014", "text": "\u27e8\n          \n            S\n            \n              j\n            \n          \n          \u27e9\n        \n      \n    \n    {\\displaystyle \\left\\langle S_{j}\\right\\rangle }\n  \n \u2013 the mean stable equilibrium states\nThere are some assumptions for this to hold, which are not listed here.\n\nStrong dilution\nWhen the dilution is strong, the finite fraction of removed connections (the weights) is large, giving rise to a huge uncertainty.", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0015", "text": "Dropout\nDropout is a special case of the previous weight equation (3), where the aforementioned equation is adjusted to remove a whole row in the vector matrix, and not only random weights\n\n  \n    \n      \n        P\n        (\n        c\n        )\n      \n    \n    {\\displaystyle P(c)}\n  \n \u2013 the probability \n  \n    \n      \n        c\n      \n    \n    {\\displaystyle c}\n  \n to keep a row in the weight matrix", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0016", "text": "w\n          \n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {w} _{j}}\n  \n \u2013 real row in the weight matrix before dropout", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0017", "text": "w\n                \n                ^\n              \n            \n          \n          \n            j\n          \n        \n      \n    \n    {\\displaystyle {\\hat {\\mathbf {w} }}_{j}}\n  \n \u2013 diluted row in the weight matrix\nBecause dropout removes a whole row from the vector matrix, the previous (unlisted) assumptions for weak dilution and the use of mean field theory are not applicable.", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0018", "text": "The process by which the node is driven to zero, whether by setting the weights to zero, by \u201cremoving the node\u201d, or by some other means, does not impact the end result and does not create a new and unique case. If the neural net is processed by a high-performance digital array-multiplicator, then it is likely more effective to drive the value to zero late in the process graph", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0019", "text": ". If the net is processed by a constrained processor, perhaps even an analog neuromorphic processor, then it is likely a more power-efficient solution is to drive the value to zero early in the process graph.", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Dilution_neural_networks_0020", "text": "Google's patent\nAlthough there have been examples of randomly removing connections between neurons in a neural network to improve models, this technique was first introduced with the name dropout by Geoffrey Hinton, et al. in 2012. Google currently holds the patent for the dropout technique.\n\nSee also\nAlexNet\nConvolutional neural network \u00a7 Dropout\n\nNotes\n\n\n== References ==", "title": "Dilution (neural networks)", "url": "https://en.wikipedia.org/wiki/Dilution_(neural_networks)"}, {"id": "rec_Overfitting_0000", "text": "In mathematical modeling, overfitting is the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit to additional data or predict future observations reliably. An overfitted model is a mathematical model that contains more parameters than can be justified by the data. In the special case of a model that consists of a polynomial function, these parameters represent the degree of a polynomial", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0001", "text": ". The essence of overfitting is to unknowingly extract some of the residual variation (i.e., noise) as if that variation represents the underlying model structure.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0002", "text": "Underfitting occurs when a mathematical model cannot adequately capture the underlying structure of the data. An under-fitted model is a model that is missing some parameters or terms that would appear in a correctly specified model. Underfitting would occur, for example, when fitting a linear model to nonlinear data. Such a model will tend to have poor predictive performance.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0003", "text": "The possibility of over-fitting exists when the criterion used for selecting the model is not the same as the criterion used to judge the suitability of a model. For example, a model might be selected by maximizing its performance on some set of training data, yet its suitability might be determined by its ability to perform well on unseen data; overfitting occurs when a model begins to \"memorize\" training data rather than \"learning\" to generalize from a trend.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0004", "text": "As an extreme example, if the number of parameters is the same as or greater than the number of observations, then a model can perfectly predict the training data simply by memorizing the data in its entirety. (For an illustration, see Figure 2.) Such a model will typically fail severely when making predictions.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0005", "text": "Overfitting is directly related to approximation error of the selected function class and the optimization error of the optimization procedure. A function class that is too large, in a suitable sense, relative to the dataset size is likely to overfit", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0006", "text": ". Even when the fitted model does not have an excessive number of parameters, it is to be expected that the fitted relationship will appear to perform less well on a new dataset than on the dataset used for fitting (a phenomenon sometimes known as shrinkage). In particular, the value of the coefficient of determination will shrink relative to the original data.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0007", "text": "To lessen the chance or amount of overfitting, several techniques are available (e.g., model comparison, cross-validation, regularization, early stopping, pruning, Bayesian priors, or dropout). The basis of some techniques is to either (1) explicitly penalize overly complex models or (2) test the model's ability to generalize by evaluating its performance on a set of data not used for training, which is assumed to approximate the typical unseen data that a model will encounter.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0008", "text": "Statistical inference\nIn statistics, an inference is drawn from a statistical model, which has been selected via some procedure. Burnham & Anderson, in their much-cited text on model selection, argue that to avoid overfitting, one should adhere to the \"Principle of Parsimony\". The authors also state the following.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0009", "text": "Overfitted models ... are often free of bias in the parameter estimators, but have estimated (and actual) sampling variances that are needlessly large (the precision of the estimators is poor, relative to what could have been accomplished with a more parsimonious model). False treatment effects tend to be identified, and false variables are included with overfitted models. ... A best approximating model is achieved by properly balancing the errors of underfitting and overfitting.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0010", "text": "Overfitting is more likely to be a serious concern when there is little theory available to guide the analysis, in part because then there tend to be a large number of models to select from. The book Model Selection and Model Averaging (2008) puts it this way.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0011", "text": "Given a data set, you can fit thousands of models at the push of a button, but how do you choose the best? With so many candidate models, overfitting is a real danger. Is the monkey who typed Hamlet actually a good writer?", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0012", "text": "Regression", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0013", "text": "In regression analysis, overfitting occurs frequently. As an extreme example, if there are p variables in a linear regression with p data points, the fitted line can go exactly through every point. For logistic regression or Cox proportional hazards models, there are a variety of rules of thumb (e.g. 5\u20139, 10 and 10\u201315 \u2014 the guideline of 10 observations per independent variable is known as the \"one in ten rule\")", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0014", "text": ". In the process of regression model selection, the mean squared error of the random regression function can be split into random noise, approximation bias, and variance in the estimate of the regression function. The bias\u2013variance tradeoff is often used to overcome overfit models.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0015", "text": "With a large set of explanatory variables that actually have no relation to the dependent variable being predicted, some variables will in general be falsely found to be statistically significant and the researcher may thus retain them in the model, thereby overfitting the model. This is known as Freedman's paradox.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0016", "text": "Machine learning\nUsually, a learning algorithm is trained using some set of \"training data\": exemplary situations for which the desired output is known. The goal is that the algorithm will also perform well on predicting the output when fed \"validation data\" that was not encountered during its training.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0017", "text": "Overfitting is the use of models or procedures that violate Occam's razor, for example by including more adjustable parameters than are ultimately optimal, or by using a more complicated approach than is ultimately optimal. For an example where there are too many adjustable parameters, consider a dataset where training data for y can be adequately predicted by a linear function of two independent variables. Such a function requires only three parameters (the intercept and two slopes)", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0018", "text": ". Replacing this simple function with a new, more complex quadratic function, or with a new, more complex linear function on more than two independent variables, carries a risk: Occam's razor implies that any given complex function is a priori less probable than any given simple function", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0019", "text": ". If the new, more complicated function is selected instead of the simple function, and if there was not a large enough gain in training data fit to offset the complexity increase, then the new complex function \"overfits\" the data and the complex overfitted function will likely perform worse than the simpler function on validation data outside the training dataset, even though the complex function performed as well, or perhaps even better, on the training dataset.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0020", "text": "When comparing different types of models, complexity cannot be measured solely by counting how many parameters exist in each model; the expressivity of each parameter must be considered as well. For example, it is nontrivial to directly compare the complexity of a neural net (which can track curvilinear relationships) with m parameters to a regression model with n parameters.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0021", "text": "Overfitting is especially likely in cases where learning was performed too long or where training examples are rare, causing the learner to adjust to very specific random features of the training data that have no causal relation to the target function. In this process of overfitting, the performance on the training examples still increases while the performance on unseen data becomes worse.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0022", "text": "As a simple example, consider a database of retail purchases that includes the item bought, the purchaser, and the date and time of purchase. It's easy to construct a model that will fit the training set perfectly by using the date and time of purchase to predict the other attributes, but this model will not generalize at all to new data because those past times will never occur again.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0023", "text": "Generally, a learning algorithm is said to overfit relative to a simpler one if it is more accurate in fitting known data (hindsight) but less accurate in predicting new data (foresight). One can intuitively understand overfitting from the fact that information from all past experience can be divided into two groups: information that is relevant for the future, and irrelevant information (\"noise\")", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0024", "text": ". Everything else being equal, the more difficult a criterion is to predict (i.e., the higher its uncertainty), the more noise exists in past information that needs to be ignored. The problem is determining which part to ignore. A learning algorithm that can reduce the risk of fitting noise is called \"robust.\"", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0025", "text": "Consequences\nThe most obvious consequence of overfitting is poor performance on the validation dataset. Other negative consequences include:", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0026", "text": "A function that is overfitted is likely to request more information about each item in the validation dataset than does the optimal function; gathering this additional unneeded data can be expensive or error-prone, especially if each individual piece of information must be gathered by human observation and manual data entry.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0027", "text": "A more complex, overfitted function is likely to be less portable than a simple one. At one extreme, a one-variable linear regression is so portable that, if necessary, it could even be done by hand. At the other extreme are models that can be reproduced only by exactly duplicating the original modeler's entire setup, making reuse or scientific reproduction difficult.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0028", "text": "It may be possible to reconstruct details of individual training instances from an overfitted machine learning model's training set. This may be undesirable if, for example, the training data includes sensitive personally identifiable information (PII)", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0029", "text": ". This phenomenon also presents problems in the area of artificial intelligence and copyright, with the developers of some generative deep learning models such as Stable Diffusion and GitHub Copilot being sued for copyright infringement because these models have been found to be capable of reproducing certain copyrighted items from their training data.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0030", "text": "Remedy", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0031", "text": "The optimal function usually needs verification on bigger or completely new datasets.  There are, however, methods like minimum spanning tree or life-time of correlation that applies the dependence between correlation coefficients and time-series (window width). Whenever the window width is big enough, the correlation coefficients are stable and don't depend on the window width size anymore", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0032", "text": ". Therefore, a correlation matrix can be created by calculating a coefficient of correlation between investigated variables. This matrix can be represented topologically as a complex network where direct and indirect influences between variables are visualized.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0033", "text": "Dropout regularisation (random removal of training set data) can also improve robustness and therefore reduce over-fitting by probabilistically removing inputs to a layer. Pruning is another technique that mitigates overfitting and enhances generalization by identifying a sparse, optimal neural network structure, while simultaneously reducing the computational cost of both training and inference.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0034", "text": "Underfitting", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0035", "text": "Underfitting is the inverse of overfitting, meaning that the statistical model or machine learning algorithm is too simplistic to accurately capture the patterns in the data. A sign of underfitting is that there is a high bias and low variance detected in the current model  or algorithm used (the inverse of overfitting: low bias and high variance)", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0036", "text": ". This can be gathered from the Bias-variance tradeoff, which is the method of analyzing a model or algorithm for bias error, variance error, and irreducible error. With a high bias and low variance, the result of the model is that it will inaccurately represent the data points and thus insufficiently be able to predict future data results (see Generalization error)", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0037", "text": ". As shown in Figure 5, the linear line could not represent all the given data points due to the line not resembling the curvature of the points. One would expect to see a parabola-shaped line as shown in Figure 6 and Figure 1. If Figure 5 were to be used for analysis, false predictive results would be given contrary to the results if Figure 6 was analyzed.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0038", "text": "Burnham & Anderson state the following.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0039", "text": "... an underfitted model would ignore some important replicable (i.e., conceptually replicable in most other samples) structure in the data and thus fail to identify effects that were actually supported by the data. In this case, bias in the parameter estimators is often substantial, and the sampling variance is underestimated, both factors resulting in poor confidence interval coverage. Underfitted models tend to miss important treatment effects in experimental settings.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0040", "text": "Resolving underfitting\nThere are multiple ways to deal with underfitting:", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0041", "text": "Increase the complexity of the model: If the model is too simple, it may be necessary to increase its complexity by adding more features, increasing the number of parameters, or using a more flexible model. However, this should be done carefully to avoid overfitting.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0042", "text": "Use a different algorithm: If the current algorithm is not able to capture the patterns in the data, it may be necessary to try a different one. For example, a neural network may be more effective than a linear regression model for some types of data.\nIncrease the amount of training data: If the model is underfitting due to a lack of data, increasing the amount of training data may help. This will allow the model to better capture the underlying patterns in the data.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0043", "text": "Regularization: Regularization is a technique used to prevent overfitting by adding a penalty term to the loss function that discourages large parameter values. It can also be used to prevent underfitting by controlling the complexity of the model.\nEnsemble Methods: Ensemble methods combine multiple models to create a more accurate prediction. This can help reduce underfitting by allowing multiple models to work together to capture the underlying patterns in the data.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0044", "text": "Feature engineering: Feature engineering involves creating new model features from the existing ones that may be more relevant to the problem at hand. This can help improve the accuracy of the model and prevent underfitting.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0045", "text": "Benign overfitting", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0046", "text": "Benign overfitting describes the phenomenon of a statistical model that seems to generalize well to unseen data, even when it has been fit perfectly on noisy training data (i.e., obtains perfect predictive accuracy on the training set). The phenomenon is of particular interest in deep neural networks, but is studied from a theoretical perspective in the context of much simpler models, such as linear regression", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0047", "text": ". In particular, it has been shown that overparameterization is essential for benign overfitting in this setting. In other words, the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0048", "text": "See also\nNotes\nReferences\nLeinweber, D. J. (2007). \"Stupid data miner tricks\". The Journal of Investing. 16: 15\u201322. doi:10.3905/joi.2007.681820. S2CID 108627390.\nTetko, I. V.; Livingstone, D. J.; Luik, A. I. (1995). \"Neural network studies. 1. Comparison of Overfitting and Overtraining\" (PDF). Journal of Chemical Information and Modeling. 35 (5): 826\u2013833. doi:10.1021/ci00027a006.", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Overfitting_0049", "text": "Further reading\nChristian, Brian; Griffiths, Tom (April 2017), \"Chapter 7: Overfitting\", Algorithms To Live By: The computer science of human decisions, William Collins, pp. 149\u2013168, ISBN 978-0-00-754799-9\n\nExternal links\nThe Problem of Overfitting Data \u2013 Stony Brook University\nWhat is \"overfitting,\" exactly? \u2013 Andrew Gelman blog\nCSE546: Linear Regression Bias / Variance Tradeoff \u2013 University of Washington\nWhat is Underfitting \u2013 IBM", "title": "Overfitting", "url": "https://en.wikipedia.org/wiki/Overfitting"}, {"id": "rec_Regularization_mathematics_0000", "text": "In mathematics, statistics, finance, and computer science, particularly in machine learning and inverse problems, regularization is a process that converts the answer to a problem to a simpler one. It is often used in solving ill-posed problems or to prevent overfitting.\nAlthough regularization procedures can be divided in many ways, the following delineation is particularly helpful:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0001", "text": "Explicit regularization is regularization whenever one explicitly adds a term to the optimization problem.  These terms could be priors, penalties, or constraints. Explicit regularization is commonly employed with ill-posed optimization problems. The regularization term, or penalty, imposes a cost on the optimization function to make the optimal solution unique.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0002", "text": "Implicit regularization is all other forms of regularization.  This includes, for example, early stopping, using a robust loss function, and discarding outliers. Implicit regularization is essentially ubiquitous in modern machine learning approaches, including stochastic gradient descent for training deep neural networks, and ensemble methods (such as random forests and gradient boosted trees).", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0003", "text": "In explicit regularization, independent of the problem or model, there is always a data term, that corresponds to a likelihood of the measurement, and a regularization term that corresponds to a prior. By combining both using Bayesian statistics, one can compute a posterior, that includes both information sources and therefore stabilizes the estimation process. By trading off both objectives, one chooses to be more aligned to the data or to enforce regularization (to prevent overfitting)", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0004", "text": ". There is a whole research branch dealing with all possible regularizations. In practice, one usually tries a specific regularization and then figures out the probability density that corresponds to that regularization to justify the choice. It can also be physically motivated by common sense or intuition.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0005", "text": "In machine learning, the data term corresponds to the training data and the regularization is either the choice of the model or modifications to the algorithm. It is always intended to reduce the generalization error, i.e. the error score with the trained model on the evaluation set (testing data) and not the training data.\nOne of the earliest uses of regularization is Tikhonov regularization (ridge regression), related to the method of least squares.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0006", "text": "Regularization in machine learning", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0007", "text": "In machine learning, a key challenge is enabling models to accurately predict outcomes on unseen data, not just on familiar training data. Regularization is crucial for addressing overfitting\u2014where a model memorizes training data details but cannot generalize to new data. The goal of regularization is to encourage models to learn the broader patterns within the data rather than memorizing it", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0008", "text": ". Techniques like early stopping, L1 and L2 regularization, and dropout are designed to prevent overfitting and underfitting, thereby enhancing the model's ability to adapt to and perform well with new data, thus improving model generalization.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0009", "text": "Early stopping\nStops training when validation performance deteriorates, preventing overfitting by halting before the model memorizes training data.\n\nL1 and L2 regularization\nAdds penalty terms to the cost function to discourage complex models:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0010", "text": "L1 regularization (also called LASSO) leads to sparse models by adding a penalty based on the absolute value of coefficients.\nL2 regularization (also called ridge regression) encourages smaller, more evenly distributed weights by adding a penalty based on the square of the coefficients.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0011", "text": "Dropout\nIn the context of neural networks, the Dropout technique repeatedly ignores random subsets of neurons during training, which simulates the training of multiple neural network architectures at once to improve generalization.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0012", "text": "Classification\nEmpirical learning of classifiers (from a finite data set) is always an underdetermined problem, because it attempts to infer a function of any \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n given only examples \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0013", "text": "n\n          \n        \n      \n    \n    {\\displaystyle x_{1},x_{2},\\dots ,x_{n}}\n  \n.\nA regularization term (or regularizer) \n  \n    \n      \n        R\n        (\n        f\n        )\n      \n    \n    {\\displaystyle R(f)}\n  \n is added to a loss function:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0014", "text": "min\n          \n            f\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        V\n        (\n        f\n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        ,\n        \n          y\n          \n            i\n          \n        \n        )\n        +\n        \u03bb\n        R\n        (\n        f\n        )", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0015", "text": "(\n        f\n        )\n      \n    \n    {\\displaystyle \\min _{f}\\sum _{i=1}^{n}V(f(x_{i}),y_{i})+\\lambda R(f)}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0016", "text": "where \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n is an underlying loss function that describes the cost of predicting \n  \n    \n      \n        f\n        (\n        x\n        )\n      \n    \n    {\\displaystyle f(x)}\n  \n when the label is \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n, such as the square loss or hinge loss; and \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0017", "text": "{\\displaystyle \\lambda }\n  \n is a parameter which controls the importance of the regularization term. \n  \n    \n      \n        R\n        (\n        f\n        )\n      \n    \n    {\\displaystyle R(f)}\n  \n is typically chosen to impose a penalty on the complexity of \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n. Concrete notions of complexity used include restrictions for smoothness and bounds on the vector space norm.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0018", "text": "A theoretical justification for regularization is that it attempts to impose Occam's razor on the solution (as depicted in the figure above, where the green function, the simpler one, may be preferred). From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0019", "text": "Regularization can serve multiple purposes, including learning simpler models, inducing models to be sparse and introducing group structure into the learning problem.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0020", "text": "The same idea arose in many fields of science. A simple form of regularization applied to integral equations (Tikhonov regularization)  is essentially a trade-off between fitting the data and reducing a norm of the solution. More recently, non-linear regularization methods, including total variation regularization, have become popular.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0021", "text": "Generalization\nRegularization can be motivated as a technique to improve the generalizability of a learned model.\nThe goal of this learning problem is to find a function that fits or predicts the outcome (label) that minimizes the expected error over all possible inputs and labels. The expected error of a function \n  \n    \n      \n        \n          f\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle f_{n}}\n  \n is:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0022", "text": "I\n        [\n        \n          f\n          \n            n\n          \n        \n        ]\n        =\n        \n          \u222b\n          \n            X\n            \u00d7\n            Y\n          \n        \n        V\n        (\n        \n          f\n          \n            n\n          \n        \n        (\n        x\n        )\n        ,\n        y\n        )\n        \u03c1\n        (\n        x\n        ,\n        y\n        )\n        \n        d\n        x\n        \n        d\n        y", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0023", "text": "d\n        y\n      \n    \n    {\\displaystyle I[f_{n}]=\\int _{X\\times Y}V(f_{n}(x),y)\\rho (x,y)\\,dx\\,dy}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0024", "text": "where \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n and \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n are the domains of input data \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n and their labels \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n respectively.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0025", "text": "{\\displaystyle y}\n  \n respectively.\nTypically in learning problems, only a subset of input data and labels are available, measured with some noise. Therefore, the expected error is unmeasurable, and the best surrogate available is the empirical error over the \n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}\n  \n available samples:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0026", "text": "I\n          \n            S\n          \n        \n        [\n        \n          f\n          \n            n\n          \n        \n        ]\n        =\n        \n          \n            1\n            n\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            N\n          \n        \n        V\n        (\n        \n          f\n          \n            n\n          \n        \n        (", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0027", "text": "(\n        \n          \n            \n              \n                x\n                ^\n              \n            \n          \n          \n            i\n          \n        \n        )\n        ,\n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            i\n          \n        \n        )", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0028", "text": ")\n      \n    \n    {\\displaystyle I_{S}[f_{n}]={\\frac {1}{n}}\\sum _{i=1}^{N}V(f_{n}({\\hat {x}}_{i}),{\\hat {y}}_{i})}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0029", "text": "Without bounds on the complexity of the function space (formally, the reproducing kernel Hilbert space) available, a model will be learned that incurs zero loss on the surrogate empirical error. If measurements (e.g. of \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0030", "text": "{\\displaystyle x_{i}}\n  \n) were made with noise, this model may suffer from overfitting and display poor expected error. Regularization introduces a penalty for exploring certain regions of the function space used to build the model, which can improve generalization.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0031", "text": "Tikhonov regularization (ridge regression)\nThese techniques are named for Andrey Nikolayevich Tikhonov, who applied regularization to integral equations and made important contributions in many other areas.\nWhen learning a linear function \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n, characterized by an unknown vector \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n such that \n  \n    \n      \n        f\n        (\n        x\n        )\n        =\n        w\n        \u22c5", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0032", "text": "x\n        )\n        =\n        w\n        \u22c5\n        x\n      \n    \n    {\\displaystyle f(x)=w\\cdot x}\n  \n, one can add the \n  \n    \n      \n        \n          L\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle L_{2}}\n  \n-norm of the vector \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0033", "text": "w\n      \n    \n    {\\displaystyle w}\n  \n to the loss expression in order to prefer solutions with smaller norms. Tikhonov regularization is one of the most common forms. It is also known as ridge regression. It is expressed as:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0034", "text": "min\n          \n            w\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        V\n        (\n        \n          \n            \n              \n                x\n                ^\n              \n            \n          \n          \n            i\n          \n        \n        \u22c5\n        w\n        ,\n        \n          \n            \n              \n                y", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0035", "text": "y\n                ^\n              \n            \n          \n          \n            i\n          \n        \n        )\n        +\n        \u03bb\n        \n          \n            \u2016\n            w\n            \u2016\n          \n          \n            2\n          \n          \n            2\n          \n        \n        ,\n      \n    \n    {\\displaystyle \\min _{w}\\sum _{i=1}^{n}V({\\hat {x}}_{i}\\cdot w,{\\hat {y}}_{i})+\\lambda \\left\\|w\\right\\|_{2}^{2},}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0036", "text": "where \n  \n    \n      \n        (\n        \n          \n            \n              \n                x\n                ^\n              \n            \n          \n          \n            i\n          \n        \n        ,\n        \n          \n            \n              \n                y\n                ^\n              \n            \n          \n          \n            i\n          \n        \n        )\n        ,\n        \n        1\n        \u2264\n        i\n        \u2264\n        n\n        ,", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0037", "text": "\u2264\n        n\n        ,\n      \n    \n    {\\displaystyle ({\\hat {x}}_{i},{\\hat {y}}_{i}),\\,1\\leq i\\leq n,}\n  \n would represent samples used for training.\nIn the case of a general function, the norm of the function in its reproducing kernel Hilbert space is:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0038", "text": "min\n          \n            f\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        V\n        (\n        f\n        (\n        \n          \n            \n              \n                x\n                ^\n              \n            \n          \n          \n            i\n          \n        \n        )\n        ,", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0039", "text": "y\n                ^\n              \n            \n          \n          \n            i\n          \n        \n        )\n        +\n        \u03bb\n        \n          \n            \u2016\n            f\n            \u2016\n          \n          \n            \n              H\n            \n          \n          \n            2", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0040", "text": "2\n          \n        \n      \n    \n    {\\displaystyle \\min _{f}\\sum _{i=1}^{n}V(f({\\hat {x}}_{i}),{\\hat {y}}_{i})+\\lambda \\left\\|f\\right\\|_{\\mathcal {H}}^{2}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0041", "text": "As the \n  \n    \n      \n        \n          L\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle L_{2}}\n  \n norm is differentiable, learning can be advanced by gradient descent.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0042", "text": "Tikhonov-regularized least squares\nThe learning problem with the least squares loss function and Tikhonov regularization can be solved analytically. Written in matrix form, the optimal \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n is the one for which the gradient of the loss function with respect to \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n is 0.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0043", "text": "min\n          \n            w\n          \n        \n        \n          \n            1\n            n\n          \n        \n        \n          \n            (\n            \n              \n                \n                  \n                    X\n                    ^\n                  \n                \n              \n              w\n              \u2212\n              Y\n            \n            )\n          \n          \n            \n              T", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0044", "text": "T\n            \n          \n        \n        \n          (\n          \n            \n              \n                \n                  X\n                  ^\n                \n              \n            \n            w\n            \u2212\n            Y\n          \n          )\n        \n        +\n        \u03bb\n        \n          \n            \u2016\n            w\n            \u2016\n          \n          \n            2\n          \n          \n            2", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0045", "text": "2\n          \n        \n      \n    \n    {\\displaystyle \\min _{w}{\\frac {1}{n}}\\left({\\hat {X}}w-Y\\right)^{\\mathsf {T}}\\left({\\hat {X}}w-Y\\right)+\\lambda \\left\\|w\\right\\|_{2}^{2}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0046", "text": "\u2207\n          \n            w\n          \n        \n        =\n        \n          \n            2\n            n\n          \n        \n        \n          \n            \n              \n                X\n                ^\n              \n            \n          \n          \n            \n              T\n            \n          \n        \n        \n          (\n          \n            \n              \n                \n                  X\n                  ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0047", "text": "^\n                \n              \n            \n            w\n            \u2212\n            Y\n          \n          )\n        \n        +\n        2\n        \u03bb\n        w\n      \n    \n    {\\displaystyle \\nabla _{w}={\\frac {2}{n}}{\\hat {X}}^{\\mathsf {T}}\\left({\\hat {X}}w-Y\\right)+2\\lambda w}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0048", "text": "0\n        =\n        \n          \n            \n              \n                X\n                ^\n              \n            \n          \n          \n            \n              T\n            \n          \n        \n        \n          (\n          \n            \n              \n                \n                  X\n                  ^\n                \n              \n            \n            w\n            \u2212\n            Y\n          \n          )\n        \n        +\n        n\n        \u03bb", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0049", "text": "+\n        n\n        \u03bb\n        w\n      \n    \n    {\\displaystyle 0={\\hat {X}}^{\\mathsf {T}}\\left({\\hat {X}}w-Y\\right)+n\\lambda w}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0050", "text": "w\n        =\n        \n          \n            (\n            \n              \n                \n                  \n                    \n                      X\n                      ^\n                    \n                  \n                \n                \n                  \n                    T\n                  \n                \n              \n              \n                \n                  \n                    X\n                    ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0051", "text": "+\n              \u03bb\n              n\n              I\n            \n            )\n          \n          \n            \u2212\n            1\n          \n        \n        \n          (\n          \n            \n              \n                \n                  \n                    X\n                    ^\n                  \n                \n              \n              \n                \n                  T", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0052", "text": "Y\n          \n          )\n        \n      \n    \n    {\\displaystyle w=\\left({\\hat {X}}^{\\mathsf {T}}{\\hat {X}}+\\lambda nI\\right)^{-1}\\left({\\hat {X}}^{\\mathsf {T}}Y\\right)}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0053", "text": "where the third statement is a first-order condition.\nBy construction of the optimization problem, other values of \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n give larger values for the loss function. This can be verified by examining the second derivative \n  \n    \n      \n        \n          \u2207\n          \n            w\n            w\n          \n        \n      \n    \n    {\\displaystyle \\nabla _{ww}}\n  \n.\nDuring training, this algorithm takes \n  \n    \n      \n        O\n        (", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0054", "text": "O\n        (\n        \n          d\n          \n            3\n          \n        \n        +\n        n\n        \n          d\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle O(d^{3}+nd^{2})}\n  \n time. The terms correspond to the matrix inversion and calculating \n  \n    \n      \n        \n          X\n          \n            \n              T\n            \n          \n        \n        X\n      \n    \n    {\\displaystyle X^{\\mathsf {T}}X}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0055", "text": "{\\displaystyle X^{\\mathsf {T}}X}\n  \n, respectively. Testing takes \n  \n    \n      \n        O\n        (\n        n\n        d\n        )\n      \n    \n    {\\displaystyle O(nd)}\n  \n time.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0056", "text": "Early stopping\nEarly stopping can be viewed as regularization in time. Intuitively, a training procedure such as gradient descent tends to learn more and more complex functions with increasing iterations. By regularizing for time, model complexity can be controlled, improving generalization.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0057", "text": "Early stopping is implemented using one data set for training, one statistically independent data set for validation and another for testing. The model is trained until performance on the validation set no longer improves and then applied to the test set.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0058", "text": "Theoretical motivation in least squares\nConsider the finite approximation of Neumann series for an invertible matrix A where \n  \n    \n      \n        \n          \u2016\n          \n            I\n            \u2212\n            A\n          \n          \u2016\n        \n        <\n        1\n      \n    \n    {\\displaystyle \\left\\|I-A\\right\\|<1}\n  \n:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0059", "text": "\u2211\n          \n            i\n            =\n            0\n          \n          \n            T\n            \u2212\n            1\n          \n        \n        \n          \n            (\n            \n              I\n              \u2212\n              A\n            \n            )\n          \n          \n            i\n          \n        \n        \u2248\n        \n          A\n          \n            \u2212\n            1", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0060", "text": "1\n          \n        \n      \n    \n    {\\displaystyle \\sum _{i=0}^{T-1}\\left(I-A\\right)^{i}\\approx A^{-1}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0061", "text": "This can be used to approximate the analytical solution of unregularized least squares, if \u03b3 is introduced to ensure the norm is less than one.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0062", "text": "w\n          \n            T\n          \n        \n        =\n        \n          \n            \u03b3\n            n\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            0\n          \n          \n            T\n            \u2212\n            1\n          \n        \n        \n          \n            (\n            \n              I\n              \u2212\n              \n                \n                  \u03b3\n                  n", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0063", "text": "n\n                \n              \n              \n                \n                  \n                    \n                      X\n                      ^\n                    \n                  \n                \n                \n                  \n                    T\n                  \n                \n              \n              \n                \n                  \n                    X\n                    ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0064", "text": ")\n          \n          \n            i\n          \n        \n        \n          \n            \n              \n                X\n                ^\n              \n            \n          \n          \n            \n              T\n            \n          \n        \n        \n          \n            \n              Y\n              ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0065", "text": "{\\displaystyle w_{T}={\\frac {\\gamma }{n}}\\sum _{i=0}^{T-1}\\left(I-{\\frac {\\gamma }{n}}{\\hat {X}}^{\\mathsf {T}}{\\hat {X}}\\right)^{i}{\\hat {X}}^{\\mathsf {T}}{\\hat {Y}}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0066", "text": "The exact solution to the unregularized least squares learning problem minimizes the empirical error, but may fail. By limiting T, the only free parameter in the algorithm above, the problem is regularized for time, which may improve its generalization.\nThe algorithm above is equivalent to restricting the number of gradient descent iterations for the empirical risk", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0067", "text": "I\n          \n            s\n          \n        \n        [\n        w\n        ]\n        =\n        \n          \n            1\n            \n              2\n              n\n            \n          \n        \n        \n          \n            \u2016\n            \n              \n                \n                  \n                    X\n                    ^\n                  \n                \n              \n              w\n              \u2212", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0068", "text": "\u2212\n              \n                \n                  \n                    Y\n                    ^\n                  \n                \n              \n            \n            \u2016\n          \n          \n            \n              \n                R\n              \n              \n                n\n              \n            \n          \n          \n            2", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0069", "text": "2\n          \n        \n      \n    \n    {\\displaystyle I_{s}[w]={\\frac {1}{2n}}\\left\\|{\\hat {X}}w-{\\hat {Y}}\\right\\|_{\\mathbb {R} ^{n}}^{2}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0070", "text": "with the gradient descent update:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0071", "text": "w\n                  \n                    0\n                  \n                \n              \n              \n                \n                =\n                0\n              \n            \n            \n              \n                \n                  w\n                  \n                    t\n                    +\n                    1", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0072", "text": "=\n                \n                  (\n                  \n                    I\n                    \u2212\n                    \n                      \n                        \u03b3\n                        n\n                      \n                    \n                    \n                      \n                        \n                          \n                            X\n                            ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0073", "text": "^\n                          \n                        \n                      \n                      \n                        \n                          T\n                        \n                      \n                    \n                    \n                      \n                        \n                          X\n                          ^\n                        \n                      \n                    \n                  \n                  )", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0074", "text": ")\n                \n                \n                  w\n                  \n                    t\n                  \n                \n                +\n                \n                  \n                    \u03b3\n                    n\n                  \n                \n                \n                  \n                    \n                      \n                        X\n                        ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0075", "text": "T\n                    \n                  \n                \n                \n                  \n                    \n                      Y\n                      ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0076", "text": "{\\displaystyle {\\begin{aligned}w_{0}&=0\\\\[1ex]w_{t+1}&=\\left(I-{\\frac {\\gamma }{n}}{\\hat {X}}^{\\mathsf {T}}{\\hat {X}}\\right)w_{t}+{\\frac {\\gamma }{n}}{\\hat {X}}^{\\mathsf {T}}{\\hat {Y}}\\end{aligned}}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0077", "text": "The base case is trivial. The inductive case is proved as follows:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0078", "text": "w\n                  \n                    T\n                  \n                \n              \n              \n                \n                =\n                \n                  (\n                  \n                    I\n                    \u2212\n                    \n                      \n                        \u03b3\n                        n", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0079", "text": "X\n                            ^\n                          \n                        \n                      \n                      \n                        \n                          T", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0080", "text": "X\n                          ^\n                        \n                      \n                    \n                  \n                  )\n                \n                \n                  \n                    \u03b3\n                    n\n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    0", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0081", "text": "0\n                  \n                  \n                    T\n                    \u2212\n                    2\n                  \n                \n                \n                  \n                    (\n                    \n                      I\n                      \u2212\n                      \n                        \n                          \u03b3\n                          n", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0082", "text": "X\n                              ^\n                            \n                          \n                        \n                        \n                          \n                            T", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0083", "text": "X\n                            ^\n                          \n                        \n                      \n                    \n                    )\n                  \n                  \n                    i\n                  \n                \n                \n                  \n                    \n                      \n                        X\n                        ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0084", "text": "T\n                    \n                  \n                \n                \n                  \n                    \n                      Y\n                      ^\n                    \n                  \n                \n                +\n                \n                  \n                    \u03b3\n                    n", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0085", "text": "X\n                        ^\n                      \n                    \n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  \n                    \n                      Y\n                      ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0086", "text": "=\n                \n                  \n                    \u03b3\n                    n\n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    T\n                    \u2212\n                    1", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0087", "text": "1\n                  \n                \n                \n                  \n                    (\n                    \n                      I\n                      \u2212\n                      \n                        \n                          \u03b3\n                          n\n                        \n                      \n                      \n                        \n                          \n                            \n                              X", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0088", "text": "X\n                              ^\n                            \n                          \n                        \n                        \n                          \n                            T\n                          \n                        \n                      \n                      \n                        \n                          \n                            X\n                            ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0089", "text": ")\n                  \n                  \n                    i\n                  \n                \n                \n                  \n                    \n                      \n                        X\n                        ^\n                      \n                    \n                  \n                  \n                    \n                      T", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0090", "text": "Y\n                      ^\n                    \n                  \n                \n                +\n                \n                  \n                    \u03b3\n                    n\n                  \n                \n                \n                  \n                    \n                      \n                        X\n                        ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0091", "text": "^\n                      \n                    \n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  \n                    \n                      Y\n                      ^\n                    \n                  \n                \n              \n            \n            \n              \n              \n                \n                =", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0092", "text": "=\n                \n                  \n                    \u03b3\n                    n\n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    0\n                  \n                  \n                    T\n                    \u2212\n                    1\n                  \n                \n                \n                  \n                    (\n                    \n                      I", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0093", "text": "I\n                      \u2212\n                      \n                        \n                          \u03b3\n                          n\n                        \n                      \n                      \n                        \n                          \n                            \n                              X\n                              ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0094", "text": "T\n                          \n                        \n                      \n                      \n                        \n                          \n                            X\n                            ^\n                          \n                        \n                      \n                    \n                    )\n                  \n                  \n                    i", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0095", "text": "i\n                  \n                \n                \n                  \n                    \n                      \n                        X\n                        ^\n                      \n                    \n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  \n                    \n                      Y\n                      ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0096", "text": "Y\n                      ^", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0097", "text": "{\\displaystyle {\\begin{aligned}w_{T}&=\\left(I-{\\frac {\\gamma }{n}}{\\hat {X}}^{\\mathsf {T}}{\\hat {X}}\\right){\\frac {\\gamma }{n}}\\sum _{i=0}^{T-2}\\left(I-{\\frac {\\gamma }{n}}{\\hat {X}}^{\\mathsf {T}}{\\hat {X}}\\right)^{i}{\\hat {X}}^{\\mathsf {T}}{\\hat {Y}}+{\\frac {\\gamma }{n}}{\\hat {X}}^{\\mathsf {T}}{\\hat {Y}}\\\\[1ex]&={\\frac {\\gamma }{n}}\\sum _{i=1}^{T-1}\\left(I-{\\frac {\\gamma }{n}}{\\hat {X}}^{\\mathsf {T}}{\\hat {X}}\\right)^{i}{\\hat {X}}^{\\mathsf {T}}{\\hat {Y}}+{\\frac {\\gamma }{n}}{\\hat", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0098", "text": "{T}}{\\hat {Y}}+{\\frac {\\gamma }{n}}{\\hat {X}}^{\\mathsf {T}}{\\hat {Y}}\\\\[1ex]&={\\frac {\\gamma }{n}}\\sum _{i=0}^{T-1}\\left(I-{\\frac {\\gamma }{n}}{\\hat {X}}^{\\mathsf {T}}{\\hat {X}}\\right)^{i}{\\hat {X}}^{\\mathsf {T}}{\\hat {Y}}\\end{aligned}}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0099", "text": "Regularizers for sparsity\nAssume that a dictionary \n  \n    \n      \n        \n          \u03d5\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\phi _{j}}\n  \n with dimension \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n is given such that a function in the function space can be expressed as:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0100", "text": "f\n        (\n        x\n        )\n        =\n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            p\n          \n        \n        \n          \u03d5\n          \n            j\n          \n        \n        (\n        x\n        )\n        \n          w\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle f(x)=\\sum _{j=1}^{p}\\phi _{j}(x)w_{j}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0101", "text": "Enforcing a sparsity constraint on \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n can lead to simpler and more interpretable models. This is useful in many real-life applications such as computational biology. An example is developing a simple predictive test for a disease in order to minimize the cost of performing medical tests while maximizing predictive power.\nA sensible sparsity constraint is the \n  \n    \n      \n        \n          L\n          \n            0", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0102", "text": "L\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle L_{0}}\n  \n norm \n  \n    \n      \n        \u2016\n        w\n        \n          \u2016\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle \\|w\\|_{0}}\n  \n, defined as the number of non-zero elements in \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}\n  \n. Solving a \n  \n    \n      \n        \n          L\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle L_{0}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0103", "text": "{\\displaystyle L_{0}}\n  \n regularized learning problem, however, has been demonstrated to be NP-hard.\nThe \n  \n    \n      \n        \n          L\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle L_{1}}\n  \n norm (see also Norms) can be used to approximate the optimal \n  \n    \n      \n        \n          L\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle L_{0}}\n  \n norm via convex relaxation. It can be shown that the", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0104", "text": "L\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle L_{1}}\n  \n norm induces sparsity. In the case of least squares, this problem is known as LASSO in statistics and basis pursuit in signal processing.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0105", "text": "min\n          \n            w\n            \u2208\n            \n              \n                R\n              \n              \n                p\n              \n            \n          \n        \n        \n          \n            1\n            n\n          \n        \n        \n          \n            \u2016\n            \n              \n                \n                  \n                    X\n                    ^\n                  \n                \n              \n              w", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0106", "text": "w\n              \u2212\n              \n                \n                  \n                    Y\n                    ^\n                  \n                \n              \n            \n            \u2016\n          \n          \n            2\n          \n        \n        +\n        \u03bb\n        \n          \n            \u2016\n            w\n            \u2016\n          \n          \n            1", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0107", "text": "1\n          \n        \n      \n    \n    {\\displaystyle \\min _{w\\in \\mathbb {R} ^{p}}{\\frac {1}{n}}\\left\\|{\\hat {X}}w-{\\hat {Y}}\\right\\|^{2}+\\lambda \\left\\|w\\right\\|_{1}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0108", "text": "L\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle L_{1}}\n  \n regularization can occasionally produce non-unique solutions. A simple example is provided in the figure when the space of possible solutions lies on a 45 degree line. This can be problematic for certain applications, and is overcome by combining \n  \n    \n      \n        \n          L\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle L_{1}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0109", "text": "{\\displaystyle L_{1}}\n  \n with \n  \n    \n      \n        \n          L\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle L_{2}}\n  \n regularization in elastic net regularization, which takes the following form:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0110", "text": "min\n          \n            w\n            \u2208\n            \n              \n                R\n              \n              \n                p\n              \n            \n          \n        \n        \n          \n            1\n            n\n          \n        \n        \n          \n            \u2016\n            \n              \n                \n                  \n                    X\n                    ^\n                  \n                \n              \n              w", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0111", "text": "w\n              \u2212\n              \n                \n                  \n                    Y\n                    ^\n                  \n                \n              \n            \n            \u2016\n          \n          \n            2\n          \n        \n        +\n        \u03bb\n        \n          (\n          \n            \u03b1\n            \n              \n                \u2016\n                w\n                \u2016\n              \n              \n                1", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0112", "text": "1\n              \n            \n            +\n            (\n            1\n            \u2212\n            \u03b1\n            )\n            \n              \n                \u2016\n                w\n                \u2016\n              \n              \n                2\n              \n              \n                2\n              \n            \n          \n          )\n        \n        ,\n        \u03b1\n        \u2208\n        [\n        0\n        ,\n        1\n        ]", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0113", "text": ",\n        1\n        ]\n      \n    \n    {\\displaystyle \\min _{w\\in \\mathbb {R} ^{p}}{\\frac {1}{n}}\\left\\|{\\hat {X}}w-{\\hat {Y}}\\right\\|^{2}+\\lambda \\left(\\alpha \\left\\|w\\right\\|_{1}+(1-\\alpha )\\left\\|w\\right\\|_{2}^{2}\\right),\\alpha \\in [0,1]}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0114", "text": "Elastic net regularization tends to have a grouping effect, where correlated input features are assigned equal weights.\nElastic net regularization is commonly used in practice and is implemented in many machine learning libraries.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0115", "text": "Proximal methods\nWhile the \n  \n    \n      \n        \n          L\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle L_{1}}\n  \n norm does not result in an NP-hard problem, the \n  \n    \n      \n        \n          L\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle L_{1}}\n  \n norm is convex but is not strictly differentiable due to the kink at x = 0. Subgradient methods which rely on the subderivative can be used to solve", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0116", "text": "L\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle L_{1}}\n  \n regularized learning problems. However, faster convergence can be achieved through proximal methods.\nFor a problem \n  \n    \n      \n        \n          min\n          \n            w\n            \u2208\n            H\n          \n        \n        F\n        (\n        w\n        )\n        +\n        R\n        (\n        w\n        )\n      \n    \n    {\\displaystyle \\min _{w\\in H}F(w)+R(w)}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0117", "text": "{\\displaystyle \\min _{w\\in H}F(w)+R(w)}\n  \n such that \n  \n    \n      \n        F\n      \n    \n    {\\displaystyle F}\n  \n is convex, continuous, differentiable, with Lipschitz continuous gradient (such as the least squares loss function), and \n  \n    \n      \n        R\n      \n    \n    {\\displaystyle R}\n  \n is convex, continuous, and proper, then the proximal method to solve the problem is as follows. First define the proximal operator", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0118", "text": "prox\n          \n            R\n          \n        \n        \u2061\n        (\n        v\n        )\n        =\n        \n          \n            argmin\n          \n          \n            w\n            \u2208\n            \n              \n                R\n              \n              \n                D\n              \n            \n          \n        \n        \u2061\n        \n          {\n          \n            R\n            (\n            w\n            )\n            +", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0119", "text": ")\n            +\n            \n              \n                1\n                2\n              \n            \n            \n              \n                \u2016\n                \n                  w\n                  \u2212\n                  v\n                \n                \u2016\n              \n              \n                2\n              \n            \n          \n          }\n        \n        ,", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0120", "text": "}\n        \n        ,\n      \n    \n    {\\displaystyle \\operatorname {prox} _{R}(v)=\\mathop {\\operatorname {argmin} } _{w\\in \\mathbb {R} ^{D}}\\left\\{R(w)+{\\frac {1}{2}}\\left\\|w-v\\right\\|^{2}\\right\\},}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0121", "text": "and then iterate", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0122", "text": "w\n          \n            k\n            +\n            1\n          \n        \n        =\n        \n          \n            prox\n          \n          \n            \u03b3\n            ,\n            R\n          \n        \n        \u2061\n        \n          (\n          \n            \n              w\n              \n                k\n              \n            \n            \u2212\n            \u03b3\n            \u2207\n            F\n            (\n            \n              w", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0123", "text": "w\n              \n                k\n              \n            \n            )\n          \n          )\n        \n      \n    \n    {\\displaystyle w_{k+1}=\\mathop {\\operatorname {prox} } _{\\gamma ,R}\\left(w_{k}-\\gamma \\nabla F(w_{k})\\right)}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0124", "text": "The proximal method iteratively performs gradient descent and then projects the result back into the space permitted by \n  \n    \n      \n        R\n      \n    \n    {\\displaystyle R}\n  \n.\nWhen \n  \n    \n      \n        R\n      \n    \n    {\\displaystyle R}\n  \n is the L1 regularizer, the proximal operator is equivalent to the soft-thresholding operator,", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0125", "text": "S\n          \n            \u03bb\n          \n        \n        (\n        v\n        )\n        f\n        (\n        n\n        )\n        =\n        \n          \n            {\n            \n              \n                \n                  \n                    v\n                    \n                      i\n                    \n                  \n                  \u2212\n                  \u03bb\n                  ,", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0126", "text": "if \n                  \n                  \n                    v\n                    \n                      i\n                    \n                  \n                  >\n                  \u03bb\n                \n              \n              \n                \n                  0\n                  ,\n                \n                \n                  \n                    if \n                  \n                  \n                    v", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0127", "text": "v\n                    \n                      i\n                    \n                  \n                  \u2208\n                  [\n                  \u2212\n                  \u03bb\n                  ,\n                  \u03bb\n                  ]\n                \n              \n              \n                \n                  \n                    v\n                    \n                      i\n                    \n                  \n                  +\n                  \u03bb", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0128", "text": "+\n                  \u03bb\n                  ,\n                \n                \n                  \n                    if \n                  \n                  \n                    v\n                    \n                      i\n                    \n                  \n                  <\n                  \u2212\n                  \u03bb", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0129", "text": "{\\displaystyle S_{\\lambda }(v)f(n)={\\begin{cases}v_{i}-\\lambda ,&{\\text{if }}v_{i}>\\lambda \\\\0,&{\\text{if }}v_{i}\\in [-\\lambda ,\\lambda ]\\\\v_{i}+\\lambda ,&{\\text{if }}v_{i}<-\\lambda \\end{cases}}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0130", "text": "This allows for efficient computation.\n\nGroup sparsity without overlaps\nGroups of features can be regularized by a sparsity constraint, which can be useful for expressing certain prior knowledge into an optimization problem.\nIn the case of a linear model with non-overlapping known groups, a regularizer can be defined:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0131", "text": "R\n        (\n        w\n        )\n        =\n        \n          \u2211\n          \n            g\n            =\n            1\n          \n          \n            G\n          \n        \n        \n          \n            \u2016\n            \n              w\n              \n                g\n              \n            \n            \u2016\n          \n          \n            2\n          \n        \n        ,\n      \n    \n    {\\displaystyle R(w)=\\sum _{g=1}^{G}\\left\\|w_{g}\\right\\|_{2},}\n  \n where", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0132", "text": "where \n  \n    \n      \n        \u2016\n        \n          w\n          \n            g\n          \n        \n        \n          \u2016\n          \n            2\n          \n        \n        =\n        \n          \n            \n              \u2211\n              \n                j\n                =\n                1\n              \n              \n                \n                  |\n                \n                \n                  G\n                  \n                    g", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0133", "text": "|\n                \n              \n            \n            \n              \n                (\n                \n                  w\n                  \n                    g\n                  \n                  \n                    j\n                  \n                \n                )\n              \n              \n                2", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0134", "text": "{\\displaystyle \\|w_{g}\\|_{2}={\\sqrt {\\sum _{j=1}^{|G_{g}|}\\left(w_{g}^{j}\\right)^{2}}}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0135", "text": "This can be viewed as inducing a regularizer over the \n  \n    \n      \n        \n          L\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle L_{2}}\n  \n norm over members of each group followed by an \n  \n    \n      \n        \n          L\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle L_{1}}\n  \n norm over groups.\nThis can be solved by the proximal method, where the proximal operator is a block-wise soft-thresholding function:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0136", "text": "prox\n          \n            \u03bb\n            ,\n            R\n            ,\n            g\n          \n        \n        \u2061\n        (\n        \n          w\n          \n            g\n          \n        \n        )\n        =\n        \n          \n            {\n            \n              \n                \n                  \n                    (\n                    \n                      1\n                      \u2212", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0137", "text": "\u03bb\n                            \n                              \n                                \u2016\n                                \n                                  w\n                                  \n                                    g\n                                  \n                                \n                                \u2016", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0138", "text": "2\n                              \n                            \n                          \n                        \n                      \n                    \n                    )\n                  \n                  \n                    w\n                    \n                      g\n                    \n                  \n                  ,", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0139", "text": "if \n                  \n                  \n                    \n                      \u2016\n                      \n                        w\n                        \n                          g\n                        \n                      \n                      \u2016\n                    \n                    \n                      2\n                    \n                  \n                  >\n                  \u03bb", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0140", "text": "0\n                  ,\n                \n                \n                  \n                    if \n                  \n                  \u2016\n                  \n                    w\n                    \n                      g\n                    \n                  \n                  \n                    \u2016\n                    \n                      2\n                    \n                  \n                  \u2264", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0141", "text": "\u2264\n                  \u03bb\n                \n              \n            \n            \n          \n        \n      \n    \n    {\\displaystyle \\operatorname {prox} \\limits _{\\lambda ,R,g}(w_{g})={\\begin{cases}\\left(1-{\\dfrac {\\lambda }{\\left\\|w_{g}\\right\\|_{2}}}\\right)w_{g},&{\\text{if }}\\left\\|w_{g}\\right\\|_{2}>\\lambda \\\\[1ex]0,&{\\text{if }}\\|w_{g}\\|_{2}\\leq \\lambda \\end{cases}}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0142", "text": "Group sparsity with overlaps\nThe algorithm described for group sparsity without overlaps can be applied to the case where groups do overlap, in certain situations. This will likely result in some groups with all zero elements, and other groups with some non-zero and some zero elements.\nIf it is desired to preserve the group structure, a new regularizer can be defined:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0143", "text": "R\n        (\n        w\n        )\n        =\n        inf\n        \n          {\n          \n            \n              \u2211\n              \n                g\n                =\n                1\n              \n              \n                G\n              \n            \n            \u2016\n            \n              w\n              \n                g\n              \n            \n            \n              \u2016\n              \n                2\n              \n            \n            :", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0144", "text": ":\n            w\n            =\n            \n              \u2211\n              \n                g\n                =\n                1\n              \n              \n                G\n              \n            \n            \n              \n                \n                  \n                    w\n                    \u00af\n                  \n                \n              \n              \n                g\n              \n            \n          \n          }", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0145", "text": "}\n        \n      \n    \n    {\\displaystyle R(w)=\\inf \\left\\{\\sum _{g=1}^{G}\\|w_{g}\\|_{2}:w=\\sum _{g=1}^{G}{\\bar {w}}_{g}\\right\\}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0146", "text": "For each \n  \n    \n      \n        \n          w\n          \n            g\n          \n        \n      \n    \n    {\\displaystyle w_{g}}\n  \n, \n  \n    \n      \n        \n          \n            \n              \n                w\n                \u00af\n              \n            \n          \n          \n            g\n          \n        \n      \n    \n    {\\displaystyle {\\bar {w}}_{g}}\n  \n is defined as the vector such that the restriction of", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0147", "text": "w\n                \u00af\n              \n            \n          \n          \n            g\n          \n        \n      \n    \n    {\\displaystyle {\\bar {w}}_{g}}\n  \n to the group \n  \n    \n      \n        g\n      \n    \n    {\\displaystyle g}\n  \n equals \n  \n    \n      \n        \n          w\n          \n            g\n          \n        \n      \n    \n    {\\displaystyle w_{g}}\n  \n and all other entries of", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0148", "text": "w\n                \u00af\n              \n            \n          \n          \n            g\n          \n        \n      \n    \n    {\\displaystyle {\\bar {w}}_{g}}\n  \n are zero. The regularizer finds the optimal disintegration of \n  \n    \n      \n        w\n      \n    \n    {\\displaystyle w}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0149", "text": "w\n      \n    \n    {\\displaystyle w}\n  \n into parts. It can be viewed as duplicating all elements that exist in multiple groups. Learning problems with this regularizer can also be solved with the proximal method with a complication. The proximal operator cannot be computed in closed form, but can be effectively solved iteratively, inducing an inner iteration within the proximal method iteration.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0150", "text": "Regularizers for semi-supervised learning\nWhen labels are more expensive to gather than input examples, semi-supervised learning can be useful. Regularizers have been designed to guide learning algorithms to learn models that respect the structure of unsupervised training samples. If a symmetric weight matrix \n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}\n  \n is given, a regularizer can be defined:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0151", "text": "R\n        (\n        f\n        )\n        =\n        \n          \u2211\n          \n            i\n            ,\n            j\n          \n        \n        \n          w\n          \n            i\n            j\n          \n        \n        \n          \n            (\n            \n              f\n              (\n              \n                x\n                \n                  i\n                \n              \n              )\n              \u2212\n              f\n              (", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0152", "text": "f\n              (\n              \n                x\n                \n                  j\n                \n              \n              )\n            \n            )\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle R(f)=\\sum _{i,j}w_{ij}\\left(f(x_{i})-f(x_{j})\\right)^{2}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0153", "text": "If \n  \n    \n      \n        \n          W\n          \n            i\n            j\n          \n        \n      \n    \n    {\\displaystyle W_{ij}}\n  \n encodes the result of some distance metric for points \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n and \n  \n    \n      \n        \n          x\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle x_{j}}\n  \n, it is desirable that \n  \n    \n      \n        f", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0154", "text": ", it is desirable that \n  \n    \n      \n        f\n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        \u2248\n        f\n        (\n        \n          x\n          \n            j\n          \n        \n        )\n      \n    \n    {\\displaystyle f(x_{i})\\approx f(x_{j})}\n  \n. This regularizer captures this intuition, and is equivalent to:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0155", "text": "R\n        (\n        f\n        )\n        =\n        \n          \n            \n              \n                f\n                \u00af\n              \n            \n          \n          \n            \n              T\n            \n          \n        \n        L\n        \n          \n            \n              f\n              \u00af\n            \n          \n        \n      \n    \n    {\\displaystyle R(f)={\\bar {f}}^{\\mathsf {T}}L{\\bar {f}}}\n  \n where \n  \n    \n      \n        L\n        =\n        D", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0156", "text": "L\n        =\n        D\n        \u2212\n        W\n      \n    \n    {\\displaystyle L=D-W}\n  \n is the Laplacian matrix of the graph induced by \n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}\n  \n.\nThe optimization problem \n  \n    \n      \n        \n          min\n          \n            f\n            \u2208\n            \n              \n                R\n              \n              \n                m\n              \n            \n          \n        \n        R\n        (\n        f", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0157", "text": "R\n        (\n        f\n        )\n        ,\n        m\n        =\n        u\n        +\n        l\n      \n    \n    {\\displaystyle \\min _{f\\in \\mathbb {R} ^{m}}R(f),m=u+l}\n  \n can be solved analytically if the constraint \n  \n    \n      \n        f\n        (\n        \n          x\n          \n            i\n          \n        \n        )\n        =\n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle f(x_{i})=y_{i}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0158", "text": "{\\displaystyle f(x_{i})=y_{i}}\n  \n is applied for all supervised samples. The labeled part of the vector \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is therefore obvious. The unlabeled part of \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is solved for by:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0159", "text": "min\n          \n            \n              f\n              \n                u\n              \n            \n            \u2208\n            \n              \n                R\n              \n              \n                u\n              \n            \n          \n        \n        \n          f\n          \n            \n              T\n            \n          \n        \n        L\n        f\n        =\n        \n          min\n          \n            \n              f", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0160", "text": "f\n              \n                u\n              \n            \n            \u2208\n            \n              \n                R\n              \n              \n                u\n              \n            \n          \n        \n        \n          {\n          \n            \n              f\n              \n                u\n              \n              \n                \n                  T\n                \n              \n            \n            \n              L", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0161", "text": "L\n              \n                u\n                u\n              \n            \n            \n              f\n              \n                u\n              \n            \n            +\n            \n              f\n              \n                l\n              \n              \n                \n                  T\n                \n              \n            \n            \n              L\n              \n                l\n                u", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0162", "text": "u\n              \n            \n            \n              f\n              \n                u\n              \n            \n            +\n            \n              f\n              \n                u\n              \n              \n                \n                  T\n                \n              \n            \n            \n              L\n              \n                u\n                l\n              \n            \n            \n              f\n              \n                l", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0163", "text": "f\n              \n                l\n              \n            \n          \n          }\n        \n      \n    \n    {\\displaystyle \\min _{f_{u}\\in \\mathbb {R} ^{u}}f^{\\mathsf {T}}Lf=\\min _{f_{u}\\in \\mathbb {R} ^{u}}\\left\\{f_{u}^{\\mathsf {T}}L_{uu}f_{u}+f_{l}^{\\mathsf {T}}L_{lu}f_{u}+f_{u}^{\\mathsf {T}}L_{ul}f_{l}\\right\\}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0164", "text": "\u2207\n          \n            \n              f\n              \n                u\n              \n            \n          \n        \n        =\n        2\n        \n          L\n          \n            u\n            u\n          \n        \n        \n          f\n          \n            u\n          \n        \n        +\n        2\n        \n          L\n          \n            u\n            l\n          \n        \n        Y", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0165", "text": "Y\n      \n    \n    {\\displaystyle \\nabla _{f_{u}}=2L_{uu}f_{u}+2L_{ul}Y}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0166", "text": "f\n          \n            u\n          \n        \n        =\n        \n          L\n          \n            u\n            u\n          \n          \n            \u2020\n          \n        \n        \n          (\n          \n            \n              L\n              \n                u\n                l\n              \n            \n            Y\n          \n          )\n        \n      \n    \n    {\\displaystyle f_{u}=L_{uu}^{\\dagger }\\left(L_{ul}Y\\right)}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0167", "text": "The pseudo-inverse can be taken because \n  \n    \n      \n        \n          L\n          \n            u\n            l\n          \n        \n      \n    \n    {\\displaystyle L_{ul}}\n  \n has the same range as \n  \n    \n      \n        \n          L\n          \n            u\n            u\n          \n        \n      \n    \n    {\\displaystyle L_{uu}}\n  \n.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0168", "text": "Regularizers for multitask learning\nIn the case of multitask learning, \n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}\n  \n problems are considered simultaneously, each related in some way. The goal is to learn \n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}\n  \n functions, ideally borrowing strength from the relatedness of tasks, that have predictive power. This is equivalent to learning the matrix \n  \n    \n      \n        W\n        :\n        T\n        \u00d7\n        D", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0169", "text": ":\n        T\n        \u00d7\n        D\n      \n    \n    {\\displaystyle W:T\\times D}\n  \n .", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0170", "text": "Sparse regularizer on columns\nR\n        (\n        w\n        )\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            D\n          \n        \n        \n          \n            \u2016\n            W\n            \u2016\n          \n          \n            2\n            ,\n            1\n          \n        \n      \n    \n    {\\displaystyle R(w)=\\sum _{i=1}^{D}\\left\\|W\\right\\|_{2,1}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0171", "text": "This regularizer defines an L2 norm on each column and an L1 norm over all columns. It can be solved by proximal methods.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0172", "text": "Nuclear norm regularization\nR\n        (\n        w\n        )\n        =\n        \n          \n            \u2016\n            \n              \u03c3\n              (\n              W\n              )\n            \n            \u2016\n          \n          \n            1\n          \n        \n      \n    \n    {\\displaystyle R(w)=\\left\\|\\sigma (W)\\right\\|_{1}}\n  \n where \n  \n    \n      \n        \u03c3\n        (\n        W\n        )\n      \n    \n    {\\displaystyle \\sigma (W)}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0173", "text": "{\\displaystyle \\sigma (W)}\n  \n is the eigenvalues in the singular value decomposition of \n  \n    \n      \n        W\n      \n    \n    {\\displaystyle W}\n  \n.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0174", "text": "Mean-constrained regularization\nR\n        (\n        \n          f\n          \n            1\n          \n        \n        \u22ef\n        \n          f\n          \n            T\n          \n        \n        )\n        =\n        \n          \u2211\n          \n            t\n            =\n            1\n          \n          \n            T\n          \n        \n        \n          \n            \u2016\n            \n              \n                f\n                \n                  t", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0175", "text": "\u2212\n              \n                \n                  1\n                  T\n                \n              \n              \n                \u2211\n                \n                  s\n                  =\n                  1\n                \n                \n                  T\n                \n              \n              \n                f\n                \n                  s\n                \n              \n            \n            \u2016", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0176", "text": "\u2016\n          \n          \n            \n              H\n              \n                k\n              \n            \n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle R(f_{1}\\cdots f_{T})=\\sum _{t=1}^{T}\\left\\|f_{t}-{\\frac {1}{T}}\\sum _{s=1}^{T}f_{s}\\right\\|_{H_{k}}^{2}}", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0177", "text": "This regularizer constrains the functions learned for each task to be similar to the overall average of the functions across all tasks. This is useful for expressing prior information that each task is expected to share with each other task. An example is predicting blood iron levels measured at different times of the day, where each task represents an individual.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0178", "text": "Clustered mean-constrained regularization\nR\n        (\n        \n          f\n          \n            1\n          \n        \n        \u22ef\n        \n          f\n          \n            T\n          \n        \n        )\n        =\n        \n          \u2211\n          \n            r\n            =\n            1\n          \n          \n            C\n          \n        \n        \n          \u2211\n          \n            t\n            \u2208\n            I\n            (\n            r\n            )", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0179", "text": ")\n          \n        \n        \n          \n            \u2016\n            \n              \n                f\n                \n                  t\n                \n              \n              \u2212\n              \n                \n                  1\n                  \n                    I\n                    (\n                    r\n                    )\n                  \n                \n              \n              \n                \u2211\n                \n                  s\n                  \u2208", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0180", "text": "s\n                  \u2208\n                  I\n                  (\n                  r\n                  )\n                \n              \n              \n                f\n                \n                  s\n                \n              \n            \n            \u2016\n          \n          \n            \n              H\n              \n                k\n              \n            \n          \n          \n            2", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0181", "text": "2\n          \n        \n      \n    \n    {\\displaystyle R(f_{1}\\cdots f_{T})=\\sum _{r=1}^{C}\\sum _{t\\in I(r)}\\left\\|f_{t}-{\\frac {1}{I(r)}}\\sum _{s\\in I(r)}f_{s}\\right\\|_{H_{k}}^{2}}\n  \n where \n  \n    \n      \n        I\n        (\n        r\n        )\n      \n    \n    {\\displaystyle I(r)}\n  \n is a cluster of tasks.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0182", "text": "is a cluster of tasks.\nThis regularizer is similar to the mean-constrained regularizer, but instead enforces similarity between tasks within the same cluster. This can capture more complex prior information. This technique has been used to predict Netflix recommendations. A cluster would correspond to a group of people who share similar preferences.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0183", "text": "Graph-based similarity\nMore generally than above, similarity between tasks can be defined by a function. The regularizer encourages the model to learn similar functions for similar tasks.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0184", "text": "R\n        (\n        \n          f\n          \n            1\n          \n        \n        \u22ef\n        \n          f\n          \n            T\n          \n        \n        )\n        =\n        \n          \u2211\n          \n            t\n            ,\n            s\n            =\n            1\n            ,\n            t\n            \u2260\n            s\n          \n          \n            \n              T\n            \n          \n        \n        \n          \n            \u2016", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0185", "text": "\u2016\n            \n              \n                f\n                \n                  t\n                \n              \n              \u2212\n              \n                f\n                \n                  s\n                \n              \n            \n            \u2016\n          \n          \n            2\n          \n        \n        \n          M\n          \n            t\n            s", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0186", "text": "s\n          \n        \n      \n    \n    {\\displaystyle R(f_{1}\\cdots f_{T})=\\sum _{t,s=1,t\\neq s}^{\\mathsf {T}}\\left\\|f_{t}-f_{s}\\right\\|^{2}M_{ts}}\n  \n for a given symmetric similarity matrix \n  \n    \n      \n        M\n      \n    \n    {\\displaystyle M}\n  \n.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0187", "text": "Other uses of regularization in statistics and machine learning\nBayesian learning methods make use of a prior probability that (usually) gives lower probability to more complex models. Well-known model selection techniques include the Akaike information criterion (AIC), minimum description length (MDL), and the Bayesian information criterion (BIC). Alternative methods of controlling overfitting not involving regularization include cross-validation.", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0188", "text": "Examples of applications of different methods of regularization to the linear model are:", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0189", "text": "See also\nBayesian interpretation of regularization\nBias\u2013variance tradeoff\nMatrix regularization\nRegularization by spectral filtering\nRegularized least squares\nLagrange multiplier\nVariance reduction\nL-curve", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Regularization_mathematics_0190", "text": "Notes\nReferences\nNeumaier, A. (1998). \"Solving ill-conditioned and singular linear systems: A tutorial on regularization\" (PDF). SIAM Review. 40 (3): 636\u2013666. Bibcode:1998SIAMR..40..636N. doi:10.1137/S0036144597321909. Archived from the original (PDF) on 2007-06-30.\nKuka\u010dka, Jan; Golkov, Vladimir; Cremers, Daniel (2017). \"Regularization for Deep Learning: A Taxonomy\". arXiv:1710.10686 [cs.LG].", "title": "Regularization (mathematics)", "url": "https://en.wikipedia.org/wiki/Regularization_(mathematics)"}, {"id": "rec_Crossvalidation_statistics_0000", "text": "Cross-validation, sometimes called rotation estimation or out-of-sample testing, is any of various similar model validation techniques for assessing how the results of a statistical analysis will generalize to an independent data set.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0001", "text": "Cross-validation includes resampling and sample splitting methods that use different portions of the data to test and train a model on different iterations. It is often used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice. It can also be used to assess the quality of a fitted model and the stability of its parameters.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0002", "text": "In a prediction problem, a model is usually given a dataset of known data on which training is run (training dataset), and a dataset of unknown data (or first seen data) against which the model is tested (called the validation dataset or testing set)", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0003", "text": ". The goal of cross-validation is to test the model's ability to predict new data that was not used in estimating it, in order to flag problems like overfitting or selection bias and to give an insight on how the model will generalize to an independent dataset (i.e., an unknown dataset, for instance from a real problem).", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0004", "text": "One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, in most methods multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0005", "text": ". averaged) over the rounds to give an estimate of the model's predictive performance.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0006", "text": "In summary, cross-validation combines (averages) measures of fitness in prediction to derive a more accurate estimate of model prediction performance.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0007", "text": "Motivation", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0008", "text": "Assume a model with one or more unknown parameters, and a data set to which the model can be fit (the training data set). The fitting process optimizes the model parameters to make the model fit the training data as well as possible. If an independent sample of validation data is taken from the same population as the training data, it will generally turn out that the model does not fit the validation data as well as it fits the training data", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0009", "text": ". The size of this difference is likely to be large especially when the size of the training data set is small, or when the number of parameters in the model is large. Cross-validation is a way to estimate the size of this effect.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0010", "text": "Example: linear regression\nIn linear regression, there exist real response values \n  \n    \n      \n        \n          y\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          y\n          \n            n\n          \n        \n      \n    \n    {\\textstyle y_{1},\\ldots ,y_{n}}", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0011", "text": "{\\textstyle y_{1},\\ldots ,y_{n}}\n  \n, and n p-dimensional vector covariates x1, ..., xn.  The components of the vector xi are denoted xi1, ..., xip. If least squares is used to fit a function in the form of a hyperplane \u0177 = a + \u03b2Tx to the data (xi, yi) 1 \u2264 i \u2264 n, then the fit can be assessed using the mean squared error (MSE). The MSE for given estimated parameter values a and \u03b2 on the training set (xi, yi) 1 \u2264 i \u2264 n is defined as:", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0012", "text": "MSE\n                \n              \n              \n                \n                =\n                \n                  \n                    1\n                    n\n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0013", "text": "n\n                  \n                \n                (\n                \n                  y\n                  \n                    i\n                  \n                \n                \u2212\n                \n                  \n                    \n                      \n                        y\n                        ^\n                      \n                    \n                  \n                  \n                    i", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0014", "text": ")\n                  \n                    2\n                  \n                \n                =\n                \n                  \n                    1\n                    n\n                  \n                \n                \n                  \u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0015", "text": "n\n                  \n                \n                (\n                \n                  y\n                  \n                    i\n                  \n                \n                \u2212\n                a\n                \u2212\n                \n                  \n                    \u03b2\n                  \n                  \n                    T\n                  \n                \n                \n                  \n                    x", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0016", "text": "i\n                  \n                \n                \n                  )\n                  \n                    2\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    1\n                    n\n                  \n                \n                \n                  \u2211", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0017", "text": "\u2211\n                  \n                    i\n                    =\n                    1\n                  \n                  \n                    n\n                  \n                \n                (\n                \n                  y\n                  \n                    i\n                  \n                \n                \u2212\n                a\n                \u2212\n                \n                  \u03b2\n                  \n                    1", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0018", "text": "1\n                  \n                \n                \n                  x\n                  \n                    i\n                    1\n                  \n                \n                \u2212\n                \u22ef\n                \u2212\n                \n                  \u03b2\n                  \n                    p\n                  \n                \n                \n                  x\n                  \n                    i\n                    p", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0019", "text": ")\n                  \n                    2", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0020", "text": "{\\displaystyle {\\begin{aligned}{\\text{MSE}}&={\\frac {1}{n}}\\sum _{i=1}^{n}(y_{i}-{\\hat {y}}_{i})^{2}={\\frac {1}{n}}\\sum _{i=1}^{n}(y_{i}-a-{\\boldsymbol {\\beta }}^{T}\\mathbf {x} _{i})^{2}\\\\&={\\frac {1}{n}}\\sum _{i=1}^{n}(y_{i}-a-\\beta _{1}x_{i1}-\\dots -\\beta _{p}x_{ip})^{2}\\end{aligned}}}", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0021", "text": "If the model is correctly specified, it can be shown under mild assumptions that the expected value of the MSE for the training set is (n \u2212 p \u2212 1)/(n + p + 1) < 1 times the expected value of the MSE for the validation set (the expected value is taken over the distribution of training sets).  Thus, a fitted model and computed MSE on the training set will result in an optimistically biased assessment of how well the model will fit an independent data set", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0022", "text": ".  This biased estimate is called the in-sample estimate of the fit, whereas the cross-validation estimate is an out-of-sample estimate.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0023", "text": "Since in linear regression it is possible to directly compute the factor (n \u2212 p \u2212 1)/(n + p + 1) by which the training MSE underestimates the validation MSE under the assumption that the model specification is valid, cross-validation can be used for checking whether the model has been overfitted, in which case the MSE in the validation set will substantially exceed its anticipated value", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0024", "text": ". (Cross-validation in the context of linear regression is also useful in that it can be used to select an optimally regularized cost function.)", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0025", "text": "General case\nIn most other regression procedures (e.g. logistic regression), there is no simple formula to compute the expected out-of-sample fit. Cross-validation is, thus, a generally applicable way to predict the performance of a model on unavailable data using numerical computation in place of theoretical analysis.\n\nTypes\nTwo types of cross-validation can be distinguished: exhaustive and non-exhaustive cross-validation.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0026", "text": "Exhaustive cross-validation\nExhaustive cross-validation methods are cross-validation methods which learn and test on all possible ways to divide the original sample into a training and a validation set.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0027", "text": "Leave-p-out cross-validation\nLeave-p-out cross-validation (LpO CV) involves using p observations as the validation set and the remaining observations as the training set. This is repeated on all ways to cut the original sample on a validation set of p observations and a training set.\nLpO cross-validation require training and validating the model \n  \n    \n      \n        \n          C\n          \n            p\n          \n          \n            n", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0028", "text": "n\n          \n        \n      \n    \n    {\\displaystyle C_{p}^{n}}\n  \n times, where n is the number of observations in the original sample, and where \n  \n    \n      \n        \n          C\n          \n            p\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle C_{p}^{n}}\n  \n is the binomial coefficient.  For p > 1 and for even moderately large n, LpO CV can become computationally infeasible.  For example, with n = 100 and p = 30,", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0029", "text": "C\n          \n            30\n          \n          \n            100\n          \n        \n        \u2248\n        3\n        \u00d7\n        \n          10\n          \n            25\n          \n        \n        .\n      \n    \n    {\\displaystyle C_{30}^{100}\\approx 3\\times 10^{25}.}", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0030", "text": "A variant of LpO cross-validation with p=2 known as leave-pair-out cross-validation has been recommended as a nearly unbiased method for estimating the area under ROC curve of binary classifiers.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0031", "text": "Leave-one-out cross-validation\nLeave-one-out cross-validation (LOOCV) is a particular case of leave-p-out cross-validation with p = 1. The process looks similar to jackknife; however, with cross-validation one computes a statistic on the left-out sample(s), while with jackknifing one computes a statistic from the kept samples only.\nLOO cross-validation requires less computation time than LpO cross-validation because there are only \n  \n    \n      \n        \n          C\n          \n            1", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0032", "text": "C\n          \n            1\n          \n          \n            n\n          \n        \n        =\n        n\n      \n    \n    {\\displaystyle C_{1}^{n}=n}\n  \n passes rather than \n  \n    \n      \n        \n          C\n          \n            p\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle C_{p}^{n}}\n  \n. However, \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0033", "text": "n\n      \n    \n    {\\displaystyle n}\n  \n passes may still require quite a large computation time, in which case other approaches such as k-fold cross validation may be more appropriate.\nPseudo-code algorithm:\nInput:\nx, {vector of length N with x-values of incoming points}\ny, {vector of length N with y-values of the expected result}\ninterpolate( x_in, y_in, x_out ), { returns the estimation for point x_out after the model is trained with x_in-y_in pairs}\nOutput:", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0034", "text": "Output:\nerr, {estimate for the prediction error}\nSteps:", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0035", "text": "err \u2190 0\n for i \u2190 1, ..., N do\n   // define the cross-validation subsets\n   x_in \u2190 (x[1], ..., x[i \u2212 1], x[i + 1], ..., x[N])\n   y_in \u2190 (y[1], ..., y[i \u2212 1], y[i + 1], ..., y[N])\n   x_out \u2190 x[i]\n   y_out \u2190 interpolate(x_in, y_in, x_out)\n   err \u2190 err + (y[i] \u2212 y_out)^2\n end for\n err \u2190 err/N\n\nNon-exhaustive cross-validation\nNon-exhaustive cross validation methods do not compute all ways of splitting the original sample. These methods are approximations of leave-p-out cross-validation.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0036", "text": "k-fold cross-validation", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0037", "text": "In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples, often referred to as \"folds\". Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k \u2212 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data.  The k results can then be averaged to produce a single estimation", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0038", "text": ". The advantage of this method over repeated random sub-sampling  (see below) is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used, but in general k remains an unfixed parameter.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0039", "text": "For example, setting k = 2 results in 2-fold cross-validation. In 2-fold cross-validation, the dataset is randomly shuffled into two sets d0 and d1, so that both sets are equal size (this is usually implemented by shuffling the data array and then splitting it in two). We then train on d0 and validate on d1, followed by training on d1 and validating on d0.\nWhen k = n (the number of observations), k-fold cross-validation is equivalent to leave-one-out cross-validation.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0040", "text": "In stratified k-fold cross-validation, the partitions are selected so that the mean response value is approximately equal in all the partitions.  In the case of binary classification, this means that each partition contains roughly the same proportions of the two types of class labels.\nIn repeated cross-validation the data is randomly split into k partitions several times. The performance of the model can thereby be averaged over several runs, but this is rarely desirable in practice.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0041", "text": "When many different statistical or machine learning models are being considered, greedy k-fold cross-validation can be used to quickly identify the most promising candidate models.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0042", "text": "Holdout method\nIn the holdout method, we randomly assign data points to two sets d0 and d1, usually called the training set and the test set, respectively. The size of each of the sets is arbitrary although typically the test set is smaller than the training set. We then train (build a model) on d0 and test (evaluate its performance) on d1.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0043", "text": "In typical cross-validation, results of multiple runs of model-testing are averaged together; in contrast, the holdout method, in isolation, involves a single run. It should be used with caution because without such averaging of multiple runs, one may achieve highly misleading results. One's indicator of predictive accuracy (F*) will tend to be unstable since it will not be smoothed out by multiple iterations (see below)", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0044", "text": ".  Similarly, indicators of the specific role played by various predictor variables (e.g., values of regression coefficients) will tend to be unstable.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0045", "text": "While the holdout method can be framed as \"the simplest kind of cross-validation\", many sources instead classify holdout as a type of simple validation, rather than a simple or degenerate form of cross-validation.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0046", "text": "Repeated random sub-sampling validation", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0047", "text": "This method, also known as Monte Carlo cross-validation, creates multiple random splits of the dataset into training and validation data. For each such split, the model is fit to the training data, and predictive accuracy is assessed using the validation data. The results are then averaged over the splits", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0048", "text": ". The results are then averaged over the splits. The advantage of this method (over k-fold cross validation) is that the proportion of the training/validation split is not dependent on the number of iterations (i.e., the number of partitions). The disadvantage of this method is that some observations may never be selected in the validation subsample, whereas others may be selected more than once. In other words, validation subsets may overlap", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0049", "text": ". In other words, validation subsets may overlap. This method also exhibits Monte Carlo variation, meaning that the results will vary if the analysis is repeated with different random splits.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0050", "text": "As the number of random splits approaches infinity, the result of repeated random sub-sampling validation tends towards that of leave-p-out cross-validation.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0051", "text": "In a stratified variant of this approach, the random samples are generated in such a way that the mean response value (i.e. the dependent variable in the regression) is equal in the training and testing sets. This is particularly useful if the responses are dichotomous with an unbalanced representation of the two response values in the data.\nA method that applies repeated random sub-sampling is RANSAC.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0052", "text": "Nested cross-validation\nWhen cross-validation is used simultaneously for selection of the best set of hyperparameters and for error estimation (and assessment of generalization capacity), a nested cross-validation is required. Many variants exist. At least two variants can be distinguished:", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0053", "text": "k*l-fold cross-validation", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0054", "text": "This is a truly nested variant which contains an outer loop of k sets and an inner loop of l sets. The total data set is split into k sets. One by one, a set is selected as the (outer) test set and the k - 1 other sets are combined into the corresponding outer training set. This is repeated for each of the k sets. Each outer training set is further sub-divided into l sets", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0055", "text": ".  One by one, a set is selected as inner test (validation) set and the l - 1 other sets are combined into the corresponding inner training set. This is repeated for each of the l sets. The inner training sets are used to fit model parameters, while the inner test set is used as a validation set to provide an unbiased evaluation of the model fit", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0056", "text": ". Typically, this is repeated for many different hyperparameters (or even different model types) and the validation set is used to determine the best hyperparameter set (and model type) for this inner training set. After this, a new model is fit on the entire outer training set, using the best set of hyperparameters from the inner cross-validation. The performance of this model is then evaluated using the outer test set.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0057", "text": "k-fold cross-validation with validation and test set", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0058", "text": "This is a type of k*l-fold cross-validation when l = k - 1. A single k-fold cross-validation is used with both a validation and test set. The total data set is split into k sets. One by one, a set is selected as test set. Then, one by one, one of the remaining sets is used as a validation set and the other k - 2 sets are used as training sets until all possible combinations have been evaluated", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0059", "text": ". Similar to the k*l-fold cross validation, the training set is used for model fitting and the validation set is used for model evaluation for each of the hyperparameter sets. Finally, for the selected parameter set, the test set is used to evaluate the model with the best parameter set. Here, two variants are possible: either evaluating the model that was trained on the training set or evaluating a new model that was fit on the combination of the training and the validation set.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0060", "text": "Measures of fit", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0061", "text": "The goal of cross-validation is to estimate the expected level of fit of a model to a data set that is independent of the data that were used to train the model.  It can be used to estimate any quantitative measure of fit that is appropriate for the data and model.  For example, for binary classification problems, each case in the validation set is either predicted correctly or incorrectly", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0062", "text": ". In this situation the misclassification error rate can be used to summarize the fit, although other measures derived from information (e.g., counts, frequency) contained within a contingency table or confusion matrix could also be used.  When the value being predicted is continuously distributed, the mean squared error, root mean squared error or median absolute deviation could be used to summarize the errors.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0063", "text": "Using prior information\nWhen users apply cross-validation to select a good configuration \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0064", "text": ", then they might want to balance the cross-validated choice with their own estimate of the configuration. In this way, they can attempt to counter the volatility of cross-validation when the sample size is small and include relevant information from previous research. In a forecasting combination exercise, for instance, cross-validation can be applied to estimate the weights that are assigned to each forecast", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0065", "text": ". Since a simple equal-weighted forecast is difficult to beat, a penalty can be added for deviating from equal weights.  Or, if cross-validation is applied to assign individual weights to observations, then one can penalize deviations from equal weights to avoid wasting potentially relevant information.  Hoornweg (2018) shows how a tuning parameter", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0066", "text": "\u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n can be defined so that a user can intuitively balance between the accuracy of cross-validation and the simplicity of sticking to a reference parameter \n  \n    \n      \n        \n          \u03bb\n          \n            R\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{R}}\n  \n that is defined by the user.\nIf \n  \n    \n      \n        \n          \u03bb\n          \n            i", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0067", "text": "i\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{i}}\n  \n denotes the \n  \n    \n      \n        \n          i\n          \n            t\n            h\n          \n        \n      \n    \n    {\\displaystyle i^{th}}\n  \n candidate configuration that might be selected, then the loss function that is to be minimized can be defined as", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0068", "text": "L\n          \n            \n              \u03bb\n              \n                i\n              \n            \n          \n        \n        =\n        (\n        1\n        \u2212\n        \u03b3\n        )\n        \n          \n            \n               Relative Accuracy\n            \n          \n          \n            i\n          \n        \n        +\n        \u03b3\n        \n          \n            \n               Relative Simplicity\n            \n          \n          \n            i", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0069", "text": "i\n          \n        \n        .\n      \n    \n    {\\displaystyle L_{\\lambda _{i}}=(1-\\gamma ){\\mbox{ Relative Accuracy}}_{i}+\\gamma {\\mbox{ Relative Simplicity}}_{i}.}", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0070", "text": "Relative accuracy can be quantified as \n  \n    \n      \n        \n          \n            MSE\n          \n        \n        (\n        \n          \u03bb\n          \n            i\n          \n        \n        )\n        \n          /\n        \n        \n          \n            MSE\n          \n        \n        (\n        \n          \u03bb\n          \n            R\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\mbox{MSE}}(\\lambda _{i})/{\\mbox{MSE}}(\\lambda _{R})}", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0071", "text": ", so that the mean squared error of a candidate  \n  \n    \n      \n        \n          \u03bb\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{i}}\n  \n is made relative to that of a user-specified \n  \n    \n      \n        \n          \u03bb\n          \n            R\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{R}}\n  \n. The relative simplicity term measures the amount that \n  \n    \n      \n        \n          \u03bb\n          \n            i", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0072", "text": "i\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{i}}\n  \n deviates from \n  \n    \n      \n        \n          \u03bb\n          \n            R\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{R}}\n  \n relative to the maximum amount of deviation from \n  \n    \n      \n        \n          \u03bb\n          \n            R\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{R}}\n  \n. Accordingly, relative simplicity can be specified as", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0073", "text": "(\n              \n                \u03bb\n                \n                  i\n                \n              \n              \u2212\n              \n                \u03bb\n                \n                  R\n                \n              \n              \n                )\n                \n                  2\n                \n              \n            \n            \n              (\n              \n                \u03bb\n                \n                  max", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0074", "text": "max\n                \n              \n              \u2212\n              \n                \u03bb\n                \n                  R\n                \n              \n              \n                )\n                \n                  2\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {(\\lambda _{i}-\\lambda _{R})^{2}}{(\\lambda _{\\max }-\\lambda _{R})^{2}}}}\n  \n, where \n  \n    \n      \n        \n          \u03bb", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0075", "text": "\u03bb\n          \n            max\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{\\max }}\n  \n corresponds to the \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n value with the highest permissible deviation from \n  \n    \n      \n        \n          \u03bb\n          \n            R\n          \n        \n      \n    \n    {\\displaystyle \\lambda _{R}}\n  \n. With \n  \n    \n      \n        \u03b3\n        \u2208\n        [\n        0\n        ,\n        1\n        ]", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0076", "text": "[\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle \\gamma \\in [0,1]}\n  \n, the user determines how high the influence of the reference parameter is relative to cross-validation.\nOne can add relative simplicity terms for multiple configurations \n  \n    \n      \n        c\n        =\n        1\n        ,\n        2\n        ,\n        .\n        .\n        .\n        ,\n        C\n      \n    \n    {\\displaystyle c=1,2,...,C}\n  \n by specifying the loss function as", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0077", "text": "L\n          \n            \n              \u03bb\n              \n                i\n              \n            \n          \n        \n        =\n        \n          \n            \n               Relative Accuracy\n            \n          \n          \n            i\n          \n        \n        +\n        \n          \u2211\n          \n            c\n            =\n            1\n          \n          \n            C\n          \n        \n        \n          \n            \n              \u03b3", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0078", "text": "\u03b3\n              \n                c\n              \n            \n            \n              1\n              \u2212\n              \n                \u03b3\n                \n                  c\n                \n              \n            \n          \n        \n        \n          \n            \n               Relative Simplicity\n            \n          \n          \n            i\n            ,\n            c\n          \n        \n        .", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0079", "text": ".\n      \n    \n    {\\displaystyle L_{\\lambda _{i}}={\\mbox{ Relative Accuracy}}_{i}+\\sum _{c=1}^{C}{\\frac {\\gamma _{c}}{1-\\gamma _{c}}}{\\mbox{ Relative Simplicity}}_{i,c}.}", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0080", "text": "Hoornweg (2018) shows that a loss function with such an accuracy-simplicity tradeoff can also be used to intuitively define shrinkage estimators like the (adaptive) lasso and Bayesian / ridge regression. Click on the lasso for an example.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0081", "text": "Statistical properties\nSuppose we choose a measure of fit F, and use cross-validation to produce an estimate F* of the expected fit EF of a model to an independent data set drawn from the same population as the training data. If we imagine sampling multiple independent training sets following the same distribution, the resulting values for F* will vary. The statistical properties of F* result from this variation.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0082", "text": "The variance of F* can be large. For this reason, if two statistical procedures are compared based on the results of cross-validation, the procedure with the better estimated performance may not actually be the better of the two procedures (i.e. it may not have the better value of EF).  Some progress has been made on constructing confidence intervals around cross-validation estimates, but this is considered a difficult problem.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0083", "text": "Computational issues", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0084", "text": "Most forms of cross-validation are straightforward to implement as long as an implementation of the prediction method being studied is available.  In particular, the prediction method can be a \"black box\" \u2013 there is no need to have access to the internals of its implementation.  If the prediction method is expensive to train, cross-validation can be very slow since the training must be carried out repeatedly", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0085", "text": ".  In some cases such as least squares and kernel regression, cross-validation can be sped up significantly by pre-computing certain values that are needed repeatedly in the training, or by using fast \"updating rules\" such as the Sherman\u2013Morrison formula.  However one must be careful to preserve the \"total blinding\" of the validation set from the training procedure, otherwise bias may result", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0086", "text": ".  An extreme example of accelerating cross-validation occurs in linear regression, where the results of cross-validation have a closed-form expression known as the prediction residual error sum of squares (PRESS).", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0087", "text": "Limitations and misuse\nCross-validation only yields meaningful results if the validation set and training set are drawn from the same population and only if human biases are controlled.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0088", "text": "In many applications of predictive modeling, the structure of the system being studied evolves over time (i.e. it is \"non-stationary\").  Both of these can introduce systematic differences between the training and validation sets.  For example, if a model for prediction of trend changes in financial quotations is trained on data for a certain five-year period, it is unrealistic to treat the subsequent five-year period as a draw from the same population", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0089", "text": ".  As another example, suppose a model is developed to predict an individual's risk for being diagnosed with a particular disease within the next year.  If the model is trained using data from a study involving only a specific population group (e.g. young people or males), but is then applied to the general population, the cross-validation results from the training set could differ greatly from the actual predictive performance.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0090", "text": "In many applications, models also may be incorrectly specified and vary as a function of modeler biases and/or arbitrary choices. When this occurs, there may be an illusion that the system changes in external samples, whereas the reason is that the model has missed a critical predictor and/or included a confounded predictor", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0091", "text": ".   New evidence is that cross-validation by itself is not very predictive of external validity, whereas a form of experimental validation known as swap sampling that does control for human bias can be much more predictive of external validity.  As defined by this large MAQC-II study across 30,000 models, swap sampling incorporates cross-validation in the sense that predictions are tested across independent training and validation samples", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0092", "text": ". Yet, models are also developed across these independent samples and by modelers who are blinded to one another.  When there is a mismatch in these models developed across these swapped training and validation samples as happens quite frequently, MAQC-II shows that this will be much more predictive of poor external predictive validity than traditional cross-validation.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0093", "text": "The reason for the success of the swapped sampling is a built-in control for human biases in model building.  In addition to placing too much faith in predictions that may vary across modelers and lead to poor external validity due to these confounding modeler effects, these are some other ways that cross-validation can be misused:", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0094", "text": "By performing an initial analysis to identify the most informative features using the entire data set \u2013 if feature selection or model tuning is required by the modeling procedure, this must be repeated on every training set. Otherwise, predictions will certainly be upwardly biased.  If cross-validation is used to decide which features to use, an inner cross-validation to carry out the feature selection on every training set must be performed.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0095", "text": "Performing mean-centering, rescaling, dimensionality reduction, outlier removal or any other data-dependent preprocessing using the entire data set. While very common in practice, this has been shown to introduce biases into the cross-validation estimates.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0096", "text": "By allowing some of the training data to also be included in the test set \u2013 this can happen due to \"twinning\" in the data set, whereby some exactly identical or nearly identical samples are present in the data set, see pseudoreplication. To some extent twinning always takes place even in perfectly independent training and validation samples. This is because some of the training sample observations will have nearly identical values of predictors as validation sample observations", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0097", "text": ". And some of these will correlate with a target at better than chance levels in the same direction in both training and validation when they are actually driven by confounded predictors with poor external validity.  If such a cross-validated model is selected from a k-fold set, human confirmation bias will be at work and determine that such a model has been validated", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0098", "text": ". This is why traditional cross-validation needs to be supplemented with controls for human bias and confounded model specification like swap sampling and prospective studies.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0099", "text": "Cross validation for time-series, spatial and spatiotemporal models\nDue to correlations, cross-validation with random splits might be problematic for time-series models (if we are more interested in evaluating extrapolation, rather than interpolation). A more appropriate approach might be to use rolling cross-validation.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0100", "text": "However, if performance is described by a single summary statistic, it is possible that the approach described by Politis and Romano as a stationary bootstrap will work. The statistic of the bootstrap needs to accept an interval of the time series and return the summary statistic on it. The call to the stationary bootstrap needs to specify an appropriate mean interval length.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0101", "text": "Similar challenges occur with spatial and spatiotemporal data, where spatial autocorrelation can lead to overly optimistic error estimates when random splits are used.  Spatial blocking methods partition data into geographically distinct blocks, while buffered spatial cross-validation adds separation zones between training and test sets to reduce spatial leakage", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0102", "text": ". A related technique is to use a clustering algorithm for creating environmentally distinct clusters, to specifically test how well a spatial model can generalize from one environmental context to another. For spatiotemporal models, spatial blocking can be combined with rolling or forward-chaining temporal splits to account for both spatial and temporal dependence", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0103", "text": ". A recent review summarises cross-validation strategies for spatiotemporal statistics, outlining their theoretical foundations, computational challenges, and applications across environmental and econometric contexts.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0104", "text": "Applications", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0105", "text": "Cross-validation can be used to compare the performances of different predictive modeling procedures.  For example, suppose we are interested in optical character recognition, and we are considering using either a Support Vector Machine (SVM) or k-nearest neighbors (KNN) to predict the true character from an image of a handwritten character.  Using cross-validation, we can obtain empirical estimates comparing these two methods in terms of their respective fractions of misclassified characters", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0106", "text": ". In contrast, the in-sample estimate will not represent the quantity of interest (i.e. the generalization error).", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0107", "text": "Cross-validation can also be used in variable selection. Suppose we are using the expression levels of 20 proteins to predict whether a cancer patient will respond to a drug. A practical goal would be to determine which subset of the 20 features should be used to produce the best predictive model. For most modeling procedures, if we compare feature subsets using the in-sample error rates, the best performance will occur when all 20 features are used", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0108", "text": ". However under cross-validation, the model with the best fit will generally include only a subset of the features that are deemed truly informative.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0109", "text": "A recent development in medical statistics is its use in meta-analysis. It forms the basis of the validation statistic, Vn which is used to test the statistical validity of meta-analysis summary estimates.  It has also been used in a more conventional sense in meta-analysis to estimate the likely prediction error of meta-analysis results.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0110", "text": "See also\nNotes and references\nFurther reading\nBengio, Yoshua; Grandvalet, Yves (2004). \"No Unbiased Estimator of the Variance of K-Fold Cross-Validation\" (PDF). Journal of Machine Learning Research. 5: 1089\u20131105.\nKim, Ji-Hyun (September 2009). \"Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap\". Computational Statistics & Data Analysis. 53 (11): 3735\u20133745. doi:10.1016/j.csda.2009.04.009.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0111", "text": "Beleites, Claudia; Baumgartner, Richard; Bowman, Christopher; Somorjai, Ray; Steiner, Gerald; Salzer, Reiner; Sowa, Michael G. (October 2005). \"Variance reduction in estimating classification error using sparse datasets\". Chemometrics and Intelligent Laboratory Systems. 79 (1\u20132): 91\u2013100. doi:10.1016/j.chemolab.2005.04.008.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Crossvalidation_statistics_0112", "text": "Otto, P.; Fass\u00f2, A.; Maranzano, P. (2024). \"A review of regularised estimation methods and cross-validation in spatiotemporal statistics\". Statistics Surveys. 18. doi:10.1214/24-SS150. hdl:10281/527664.\nTrippa, Lorenzo; Waldron, Levi; Huttenhower, Curtis; Parmigiani, Giovanni (March 2015). \"Bayesian nonparametric cross-study validation of prediction methods\". The Annals of Applied Statistics. 9 (1). arXiv:1506.00474. doi:10.1214/14-AOAS798.", "title": "Cross-validation (statistics)", "url": "https://en.wikipedia.org/wiki/Cross-validation_(statistics)"}, {"id": "rec_Biasvariance_tradeoff_0000", "text": "In statistics and machine learning, the bias\u2013variance tradeoff describes the relationship between a model's complexity, the accuracy of its predictions, and how well it can make predictions on previously unseen data that were not used to train the model. In general, as the number of tunable parameters in a model increases, it becomes more flexible, and can better fit a training data set. That is, the model has lower error or lower bias", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0001", "text": ". That is, the model has lower error or lower bias. However, for more flexible models, there will tend to be greater variance to the model fit each time we take a set of samples to create a new training data set. It is said that there is greater variance in the model's estimated parameters.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0002", "text": "The bias\u2013variance dilemma or bias\u2013variance problem is the conflict in trying to simultaneously minimize these two sources of error that prevent supervised learning algorithms from generalizing beyond their training set:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0003", "text": "The bias error is an error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting).\nThe variance is an error from sensitivity to small fluctuations in the training set. High variance may result from an algorithm modeling the random noise in the training data (overfitting).", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0004", "text": "The bias\u2013variance decomposition is a way of analyzing a learning algorithm's expected generalization error with respect to a particular problem as a sum of three terms, the bias, variance, and a quantity called the irreducible error, resulting from noise in the problem itself.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0005", "text": "Motivation", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0006", "text": "The bias\u2013variance tradeoff is a central problem in supervised learning. Ideally, one wants to choose a model that both accurately captures the regularities in its training data, but also generalizes well to unseen data. Unfortunately, it is typically impossible to do both simultaneously. High-variance learning methods may be able to represent their training set well but are at risk of overfitting to noisy or unrepresentative training data", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0007", "text": ". In contrast, algorithms with high bias typically produce simpler models that may fail to capture important regularities (i.e. underfit) in the data.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0008", "text": "It is an often made fallacy to assume that complex models must have high variance. High variance models are \"complex\" in some sense, but the reverse need not be true. In addition, one has to be careful how to define complexity. In particular, the number of parameters used to describe the model is a poor measure of complexity. This is illustrated by an example adapted from: The model \n  \n    \n      \n        \n          f\n          \n            a\n            ,\n            b", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0009", "text": ",\n            b\n          \n        \n        (\n        x\n        )\n        =\n        a\n        sin\n        \u2061\n        (\n        b\n        x\n        )\n      \n    \n    {\\displaystyle f_{a,b}(x)=a\\sin(bx)}\n  \n has only two parameters (\n  \n    \n      \n        a\n        ,\n        b\n      \n    \n    {\\displaystyle a,b}\n  \n) but it can interpolate any number of points by oscillating with a high enough frequency, resulting in both a high bias and high variance.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0010", "text": "An analogy can be made to the relationship between accuracy and precision. Accuracy is one way of quantifying bias and can intuitively be improved by selecting from only local information. Consequently, a sample will appear accurate (i.e. have low bias) under the aforementioned selection conditions, but may result in underfitting. In other words, test data may not agree as closely with training data, which would indicate imprecision and therefore inflated variance", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0011", "text": ". A graphical example would be a straight line fit to data exhibiting quadratic behavior overall. Precision is a description of variance and generally can only be improved by selecting information from a comparatively larger space. The option to select many data points over a broad sample space is the ideal condition for any analysis. However, intrinsic constraints (whether physical, theoretical, computational, etc.) will always play a limiting role", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0012", "text": ". The limiting case where only a finite number of data points are selected over a broad sample space may result in improved precision and lower variance overall, but may also result in an overreliance on the training data (overfitting). This means that test data would also not agree as closely with the training data, but in this case the reason is inaccuracy or high bias", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0013", "text": ". To borrow from the previous example, the graphical representation would appear as a high-order polynomial fit to the same data exhibiting quadratic behavior. Note that error in each case is measured the same way, but the reason ascribed to the error is different depending on the balance between bias and variance. To mitigate how much information is used from neighboring observations, a model can be smoothed via explicit regularization, such as shrinkage.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0014", "text": "Bias\u2013variance decomposition of mean squared error\nSuppose that we have a training set consisting of a set of points \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle x_{1},\\dots ,x_{n}}\n  \n and real-valued labels \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0015", "text": "{\\displaystyle y_{i}}\n  \n associated with the points \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n. We assume that the data is generated by a function \n  \n    \n      \n        f\n        (\n        x\n        )\n      \n    \n    {\\displaystyle f(x)}\n  \n such as \n  \n    \n      \n        y\n        =\n        f\n        (\n        x\n        )\n        +\n        \u03b5", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0016", "text": ")\n        +\n        \u03b5\n      \n    \n    {\\displaystyle y=f(x)+\\varepsilon }\n  \n, where the noise, \n  \n    \n      \n        \u03b5\n      \n    \n    {\\displaystyle \\varepsilon }\n  \n, has zero mean and unit variance \n  \n    \n      \n        \n          \u03c3\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\sigma ^{2}}\n  \n. That is, \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n        =\n        f\n        (\n        \n          x", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0017", "text": "(\n        \n          x\n          \n            i\n          \n        \n        )\n        +\n        \n          \u03b5\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}=f(x_{i})+\\varepsilon _{i}}\n  \n, where \n  \n    \n      \n        \n          \u03b5\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\varepsilon _{i}}\n  \n is a noise sample.\nWe want to find a function \n  \n    \n      \n        \n          \n            \n              f\n              ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0018", "text": "f\n              ^\n            \n          \n        \n        \n        (\n        x\n        ;\n        D\n        )\n      \n    \n    {\\displaystyle {\\hat {f}}\\!(x;D)}\n  \n, that approximates the true function \n  \n    \n      \n        f\n        (\n        x\n        )\n      \n    \n    {\\displaystyle f(x)}\n  \n as well as possible, by means of some learning algorithm based on a training dataset (sample) \n  \n    \n      \n        D\n        =\n        {\n        (\n        \n          x", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0019", "text": "{\n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          y\n          \n            1\n          \n        \n        )\n        \u2026\n        ,\n        (\n        \n          x\n          \n            n\n          \n        \n        ,\n        \n          y\n          \n            n\n          \n        \n        )\n        }\n      \n    \n    {\\displaystyle D=\\{(x_{1},y_{1})\\dots ,(x_{n},y_{n})\\}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0020", "text": ". We make \"as well as possible\" precise by measuring the mean squared error between \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n and \n  \n    \n      \n        \n          \n            \n              f\n              ^\n            \n          \n        \n        \n        (\n        x\n        ;\n        D\n        )\n      \n    \n    {\\displaystyle {\\hat {f}}\\!(x;D)}\n  \n: we want \n  \n    \n      \n        (\n        y\n        \u2212\n        \n          \n            \n              f", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0021", "text": "f\n              ^\n            \n          \n        \n        \n        (\n        x\n        ;\n        D\n        )\n        \n          )\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle (y-{\\hat {f}}\\!(x;D))^{2}}\n  \n to be minimal, both for \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0022", "text": "n\n          \n        \n      \n    \n    {\\displaystyle x_{1},\\dots ,x_{n}}\n  \n and for points outside of our sample. Of course, we cannot hope to do so perfectly, since the \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle y_{i}}\n  \n contain noise \n  \n    \n      \n        \u03b5\n      \n    \n    {\\displaystyle \\varepsilon }\n  \n; this means we must be prepared to accept an irreducible error in any function we come up with.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0023", "text": "Finding an \n  \n    \n      \n        \n          \n            \n              f\n              ^\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {f}}}\n  \n that generalizes to points outside of the training set can be done with any of the countless algorithms used for supervised learning. It turns out that whichever function \n  \n    \n      \n        \n          \n            \n              f\n              ^\n            \n          \n        \n      \n    \n    {\\displaystyle {\\hat {f}}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0024", "text": "{\\displaystyle {\\hat {f}}}\n  \n we select, we can decompose its expected error on an unseen sample \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n (i.e. conditional on \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n) as follows:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0025", "text": "E\n          \n          \n            D\n            ,\n            \u03b5\n          \n        \n        \n          \n            [\n          \n        \n        \n          \n            (\n          \n        \n        y\n        \u2212\n        \n          \n            \n              f\n              ^\n            \n          \n        \n        \n        (\n        x\n        ;\n        D\n        )\n        \n          \n            \n              )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0026", "text": ")\n            \n          \n          \n            2\n          \n        \n        \n          \n            ]\n          \n        \n        =\n        \n          \n            (\n          \n        \n        \n          Bias\n          \n            D\n          \n        \n        \u2061\n        \n          \n            [\n          \n        \n        \n          \n            \n              f\n              ^\n            \n          \n        \n        \n        (\n        x\n        ;\n        D\n        )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0027", "text": "(\n        x\n        ;\n        D\n        )\n        \n          \n            ]\n          \n        \n        \n          \n            \n              )\n            \n          \n          \n            2\n          \n        \n        +\n        \n          Var\n          \n            D\n          \n        \n        \u2061\n        \n          \n            [\n          \n        \n        \n          \n            \n              f\n              ^\n            \n          \n        \n        \n        (\n        x", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0028", "text": "(\n        x\n        ;\n        D\n        )\n        \n          \n            ]\n          \n        \n        +\n        \n          \u03c3\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {E} _{D,\\varepsilon }{\\Big [}{\\big (}y-{\\hat {f}}\\!(x;D){\\big )}^{2}{\\Big ]}={\\Big (}\\operatorname {Bias} _{D}{\\big [}{\\hat {f}}\\!(x;D){\\big ]}{\\Big )}^{2}+\\operatorname {Var} _{D}{\\big [}{\\hat {f}}\\!(x;D){\\big ]}+\\sigma ^{2}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0029", "text": "where", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0030", "text": "Bias\n                  \n                    D\n                  \n                \n                \u2061\n                \n                  \n                    [\n                  \n                \n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0031", "text": "(\n                x\n                ;\n                D\n                )\n                \n                  \n                    ]\n                  \n                \n              \n              \n                \n                \u225c\n                \n                  \n                    E\n                  \n                  \n                    D\n                  \n                \n                \n                  \n                    [", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0032", "text": "[\n                  \n                \n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                ;\n                D\n                )\n                \u2212\n                f\n                (\n                x\n                )\n                \n                  \n                    ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0033", "text": "]\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    E\n                  \n                  \n                    D\n                  \n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    \n                      f", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0034", "text": "f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                ;\n                D\n                )\n                \n                  \n                    ]\n                  \n                \n                \n                \u2212\n                \n                f\n                (\n                x\n                )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0035", "text": "=\n                \n                  \n                    E\n                  \n                  \n                    D\n                  \n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    \n                      f\n                      ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0036", "text": "(\n                x\n                ;\n                D\n                )\n                \n                  \n                    ]\n                  \n                \n                \n                \u2212\n                \n                \n                  \n                    E\n                  \n                  \n                    y\n                    \n                      |", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0037", "text": "|\n                    \n                    x\n                  \n                \n                \n                  \n                    [\n                  \n                \n                y\n                (\n                x\n                )\n                \n                  \n                    ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0038", "text": "{\\displaystyle {\\begin{aligned}\\operatorname {Bias} _{D}{\\big [}{\\hat {f}}\\!(x;D){\\big ]}&\\triangleq \\mathbb {E} _{D}{\\big [}{\\hat {f}}\\!(x;D)-f(x){\\big ]}\\\\&=\\mathbb {E} _{D}{\\big [}{\\hat {f}}\\!(x;D){\\big ]}\\,-\\,f(x)\\\\&=\\mathbb {E} _{D}{\\big [}{\\hat {f}}\\!(x;D){\\big ]}\\,-\\,\\mathbb {E} _{y|x}{\\big [}y(x){\\big ]}\\end{aligned}}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0039", "text": "and", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0040", "text": "Var\n          \n            D\n          \n        \n        \u2061\n        \n          \n            [\n          \n        \n        \n          \n            \n              f\n              ^\n            \n          \n        \n        \n        (\n        x\n        ;\n        D\n        )\n        \n          \n            ]\n          \n        \n        \u225c\n        \n          \n            E\n          \n          \n            D\n          \n        \n        \n          [", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0041", "text": "[\n          \n            \n              (\n              \n                \n                  \n                    E\n                  \n                  \n                    D\n                  \n                \n                [\n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                ;", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0042", "text": "x\n                ;\n                D\n                )\n                ]\n                \u2212\n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                ;\n                D\n                )\n              \n              )\n            \n            \n              2\n            \n          \n          ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0043", "text": "]\n        \n      \n    \n    {\\displaystyle \\operatorname {Var} _{D}{\\big [}{\\hat {f}}\\!(x;D){\\big ]}\\triangleq \\mathbb {E} _{D}\\left[\\left(\\mathbb {E} _{D}[{\\hat {f}}\\!(x;D)]-{\\hat {f}}\\!(x;D)\\right)^{2}\\right]}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0044", "text": "and", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0045", "text": "\u03c3\n          \n            2\n          \n        \n        =\n        \n          E\n          \n            y\n          \n        \n        \u2061\n        \n          \n            [\n          \n        \n        \n          \n            (\n          \n        \n        y\n        \u2212\n        \n          \n            \n              \n                f\n                (\n                x\n                )\n              \n              \u23df", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0046", "text": "E\n              \n                y\n                \n                  |\n                \n                x\n              \n            \n            [\n            y\n            ]\n          \n        \n        \n          \n            \n              )\n            \n          \n          \n            2\n          \n        \n        \n          \n            ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0047", "text": "]\n          \n        \n      \n    \n    {\\displaystyle \\sigma ^{2}=\\operatorname {E} _{y}{\\Big [}{\\big (}y-\\underbrace {f(x)} _{E_{y|x}[y]}{\\big )}^{2}{\\Big ]}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0048", "text": "The expectation ranges over different choices of the training set \n  \n    \n      \n        D\n        =\n        {\n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \n          y\n          \n            1\n          \n        \n        )\n        \u2026\n        ,\n        (\n        \n          x\n          \n            n\n          \n        \n        ,\n        \n          y\n          \n            n\n          \n        \n        )\n        }", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0049", "text": ")\n        }\n      \n    \n    {\\displaystyle D=\\{(x_{1},y_{1})\\dots ,(x_{n},y_{n})\\}}\n  \n, all sampled from the same joint distribution \n  \n    \n      \n        P\n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle P(x,y)}\n  \n which can for example be done via bootstrapping.\nThe three terms represent:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0050", "text": "the square of the bias of the learning method, which can be thought of as the error caused by the simplifying assumptions built into the method. E.g., when approximating a non-linear function \n  \n    \n      \n        f\n        (\n        x\n        )\n      \n    \n    {\\displaystyle f(x)}\n  \n using a learning method for linear models, there will be error in the estimates \n  \n    \n      \n        \n          \n            \n              f\n              ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0051", "text": "(\n        x\n        )\n      \n    \n    {\\displaystyle {\\hat {f}}\\!(x)}\n  \n due to this assumption;\nthe variance of the learning method, or, intuitively, how much the learning method \n  \n    \n      \n        \n          \n            \n              f\n              ^\n            \n          \n        \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle {\\hat {f}}\\!(x)}\n  \n will move around its mean;\nthe irreducible error", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0052", "text": "the irreducible error \n  \n    \n      \n        \n          \u03c3\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\sigma ^{2}}\n  \n.\nSince all three terms are non-negative, the irreducible error forms a lower bound on the expected error on unseen samples.\nThe more complex the model \n  \n    \n      \n        \n          \n            \n              f\n              ^\n            \n          \n        \n        \n        (\n        x\n        )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0053", "text": "(\n        x\n        )\n      \n    \n    {\\displaystyle {\\hat {f}}\\!(x)}\n  \n is, the more data points it will capture, and the lower the bias will be. However, complexity will make the model \"move\" more to capture the data points, and hence its variance will be larger.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0054", "text": "Derivation\nThe derivation of the bias\u2013variance decomposition for squared error proceeds as follows. For convenience, we drop the \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n subscript in the following lines, such that \n  \n    \n      \n        \n          \n            \n              f\n              ^\n            \n          \n        \n        \n        (\n        x\n        ;\n        D\n        )\n        =\n        \n          \n            \n              f\n              ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0055", "text": "f\n              ^\n            \n          \n        \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle {\\hat {f}}\\!(x;D)={\\hat {f}}\\!(x)}\n  \n.\nLet us write the mean-squared error of our model:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0056", "text": "MSE\n                \n                (\n                x\n                )\n              \n              \n                \n                \u225c\n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    (\n                  \n                \n                y", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0057", "text": "y\n                \u2212\n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                )\n                \n                  \n                    \n                      )\n                    \n                  \n                  \n                    2", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0058", "text": "]\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    (\n                  \n                \n                f", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0059", "text": "f\n                (\n                x\n                )\n                +\n                \u03b5\n                \u2212\n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                )\n                \n                  \n                    \n                      )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0060", "text": "2\n                  \n                \n                \n                  \n                    ]\n                  \n                \n              \n              \n              \n                \n                  since \n                \n                y\n                \u225c\n                f\n                (\n                x\n                )\n                +\n                \u03b5", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0061", "text": "=\n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    (\n                  \n                \n                f\n                (\n                x\n                )\n                \u2212", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0062", "text": "f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                )\n                \n                  \n                    \n                      )\n                    \n                  \n                  \n                    2\n                  \n                \n                \n                  \n                    ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0063", "text": "]\n                  \n                \n                \n                +\n                \n                2\n                 \n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    (\n                  \n                \n                f\n                (\n                x\n                )\n                \u2212", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0064", "text": "\u2212\n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                )\n                \n                  \n                    )\n                  \n                \n                \u03b5\n                \n                  \n                    ]\n                  \n                \n                \n                +", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0065", "text": "+\n                \n                \n                  E\n                \n                [\n                \n                  \u03b5\n                  \n                    2\n                  \n                \n                ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0066", "text": "{\\displaystyle {\\begin{aligned}{\\text{MSE}}(x)&\\triangleq \\mathbb {E} {\\Big [}{\\big (}y-{\\hat {f}}\\!(x){\\big )}^{2}{\\Big ]}\\\\&=\\mathbb {E} {\\Big [}{\\big (}f(x)+\\varepsilon -{\\hat {f}}\\!(x){\\big )}^{2}{\\Big ]}&&{\\text{since }}y\\triangleq f(x)+\\varepsilon \\\\&=\\mathbb {E} {\\Big [}{\\big (}f(x)-{\\hat {f}}\\!(x){\\big )}^{2}{\\Big ]}\\,+\\,2\\ \\mathbb {E} {\\Big [}{\\big (}f(x)-{\\hat {f}}\\!(x){\\big )}\\varepsilon {\\Big ]}\\,+\\,\\mathbb {E} [\\varepsilon ^{2}]\\end{aligned}}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0067", "text": "We can show that the second term of this equation is null:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0068", "text": "E\n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    (\n                  \n                \n                f\n                (\n                x\n                )\n                \u2212\n                \n                  \n                    \n                      f", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0069", "text": "f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                )\n                \n                  \n                    )\n                  \n                \n                \u03b5\n                \n                  \n                    ]\n                  \n                \n              \n              \n                \n                =", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0070", "text": "=\n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                f\n                (\n                x\n                )\n                \u2212\n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0071", "text": "x\n                )\n                \n                  \n                    ]\n                  \n                \n                 \n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                \u03b5\n                \n                  \n                    ]\n                  \n                \n              \n              \n              \n                \n                  since", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0072", "text": "since \n                \n                \u03b5\n                \n                   is independent from \n                \n                x\n              \n            \n            \n              \n              \n                \n                =\n                0\n              \n              \n              \n                \n                  since \n                \n                \n                  E", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0073", "text": "[\n                  \n                \n                \u03b5\n                \n                  \n                    ]\n                  \n                \n                =\n                0", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0074", "text": "{\\displaystyle {\\begin{aligned}\\mathbb {E} {\\Big [}{\\big (}f(x)-{\\hat {f}}\\!(x){\\big )}\\varepsilon {\\Big ]}&=\\mathbb {E} {\\big [}f(x)-{\\hat {f}}\\!(x){\\big ]}\\ \\mathbb {E} {\\big [}\\varepsilon {\\big ]}&&{\\text{since }}\\varepsilon {\\text{ is independent from }}x\\\\&=0&&{\\text{since }}\\mathbb {E} {\\big [}\\varepsilon {\\big ]}=0\\end{aligned}}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0075", "text": "Moreover, the third term of this equation is nothing but \n  \n    \n      \n        \n          \u03c3\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\sigma ^{2}}\n  \n, the variance of \n  \n    \n      \n        \u03b5\n      \n    \n    {\\displaystyle \\varepsilon }\n  \n.\nLet us now expand the remaining term:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0076", "text": "E\n                  \n                \n                \u2061\n                \n                  [\n                  \n                    \n                      (\n                      \n                        f\n                        (\n                        x\n                        )\n                        \u2212", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0077", "text": "f\n                              ^\n                            \n                          \n                        \n                        \n                        (\n                        x\n                        )\n                      \n                      )\n                    \n                    \n                      2\n                    \n                  \n                  ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0078", "text": "]\n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    E\n                  \n                \n                \u2061\n                \n                  [\n                  \n                    \n                      (\n                      \n                        f\n                        (\n                        x\n                        )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0079", "text": ")\n                        \u2212\n                        \n                          \n                            E\n                          \n                        \n                        \u2061\n                        [\n                        \n                          \n                            \n                              f\n                              ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0080", "text": "(\n                        x\n                        )\n                        ]\n                        +\n                        \n                          \n                            E\n                          \n                        \n                        \u2061\n                        [\n                        \n                          \n                            \n                              f", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0081", "text": "f\n                              ^\n                            \n                          \n                        \n                        \n                        (\n                        x\n                        )\n                        ]\n                        \u2212\n                        \n                          \n                            \n                              f\n                              ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0082", "text": "(\n                        x\n                        )\n                      \n                      )\n                    \n                    \n                      2\n                    \n                  \n                  ]\n                \n              \n            \n            \n              \n              \n                \n                =", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0083", "text": "=\n                \n                  \n                    \n                      \n                        E\n                      \n                    \n                    \u2061\n                    \n                      [\n                      \n                        \n                          (\n                          \n                            f\n                            (\n                            x\n                            )\n                            \u2212", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0084", "text": "\u2212\n                            \n                              \n                                E\n                              \n                            \n                            \u2061\n                            [\n                            \n                              \n                                \n                                  f\n                                  ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0085", "text": "(\n                            x\n                            )\n                            ]\n                          \n                          )\n                        \n                        \n                          2\n                        \n                      \n                      ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0086", "text": "+\n                \n                \n                  \n                    E\n                  \n                \n                \u2061\n                \n                  [\n                  \n                    \n                      (\n                      \n                        \n                          \n                            E\n                          \n                        \n                        \u2061\n                        [", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0087", "text": "[\n                        \n                          \n                            \n                              f\n                              ^\n                            \n                          \n                        \n                        \n                        (\n                        x\n                        )\n                        ]\n                        \u2212", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0088", "text": "f\n                              ^\n                            \n                          \n                        \n                        \n                        (\n                        x\n                        )\n                      \n                      )\n                    \n                    \n                      2\n                    \n                  \n                  ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0089", "text": "+\n                \n                2\n                 \n                \n                  \n                    \n                      \n                        E\n                      \n                    \n                    \u2061\n                    \n                      [", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0090", "text": "(\n                          \n                            f\n                            (\n                            x\n                            )\n                            \u2212\n                            \n                              \n                                E\n                              \n                            \n                            \u2061\n                            [", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0091", "text": "f\n                                  ^\n                                \n                              \n                            \n                            \n                            (\n                            x\n                            )\n                            ]\n                          \n                          )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0092", "text": "(\n                          \n                            \n                              \n                                E\n                              \n                            \n                            \u2061\n                            [\n                            \n                              \n                                \n                                  f\n                                  ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0093", "text": "^\n                                \n                              \n                            \n                            \n                            (\n                            x\n                            )\n                            ]\n                            \u2212\n                            \n                              \n                                \n                                  f\n                                  ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0094", "text": "^\n                                \n                              \n                            \n                            \n                            (\n                            x\n                            )\n                          \n                          )\n                        \n                      \n                      ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0095", "text": "{\\displaystyle {\\begin{aligned}&\\operatorname {\\mathbb {E} } \\left[\\left(f(x)-{\\hat {f}}\\!(x)\\right)^{2}\\right]\\\\[1ex]&=\\operatorname {\\mathbb {E} } \\left[\\left(f(x)-\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]+\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]-{\\hat {f}}\\!(x)\\right)^{2}\\right]\\\\[1ex]&={\\color {Blue}\\operatorname {\\mathbb {E} } \\left[\\left(f(x)-\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]\\right)^{2}\\right]}\\,+\\,\\operatorname {\\mathbb {E} } \\left[\\left(\\operatorname {\\mathbb", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0096", "text": "{\\mathbb {E} } \\left[\\left(\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]-{\\hat {f}}\\!(x)\\right)^{2}\\right]\\\\&\\quad \\,+\\,2\\ {\\color {PineGreen}\\operatorname {\\mathbb {E} } \\left[\\left(f(x)-\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]\\right)\\left(\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]-{\\hat {f}}\\!(x)\\right)\\right]}\\end{aligned}}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0097", "text": "We show that:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0098", "text": "E\n                    \n                    \n                      \n                        [\n                      \n                    \n                    \n                      \n                        (\n                      \n                    \n                    f\n                    (\n                    x\n                    )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0099", "text": "x\n                    )\n                    \u2212\n                    \n                      E\n                    \n                    \n                      \n                        [\n                      \n                    \n                    \n                      \n                        \n                          f\n                          ^\n                        \n                      \n                    \n                    (\n                    x", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0100", "text": "(\n                    x\n                    )\n                    \n                      \n                        ]\n                      \n                    \n                    \n                      \n                        \n                          )\n                        \n                      \n                      \n                        2\n                      \n                    \n                    \n                      \n                        ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0101", "text": "]\n                      \n                    \n                  \n                \n              \n              \n                \n                =\n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                f\n                (\n                x\n                \n                  )\n                  \n                    2", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0102", "text": "2\n                  \n                \n                \n                  \n                    ]\n                  \n                \n                \n                \u2212\n                \n                2\n                 \n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                f\n                (\n                x\n                )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0103", "text": "E\n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                (\n                x\n                )\n                \n                  \n                    ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0104", "text": "]\n                  \n                \n                \n                +\n                \n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                \n                  E\n                \n                \n                  \n                    [", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0105", "text": "f\n                      ^\n                    \n                  \n                \n                (\n                x\n                )\n                \n                  \n                    \n                      ]\n                    \n                  \n                  \n                    2\n                  \n                \n                \n                  \n                    ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0106", "text": "]\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                f\n                (\n                x\n                \n                  )\n                  \n                    2\n                  \n                \n                \n                \u2212\n                \n                2\n                 \n                f\n                (\n                x\n                )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0107", "text": "x\n                )\n                 \n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                (\n                x\n                )\n                \n                  \n                    ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0108", "text": "]\n                  \n                \n                \n                +\n                \n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                (\n                x\n                )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0109", "text": ")\n                \n                  \n                    \n                      ]\n                    \n                  \n                  \n                    2\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    (\n                  \n                \n                f\n                (\n                x\n                )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0110", "text": "x\n                )\n                \u2212\n                \n                  E\n                \n                \n                  \n                    [\n                  \n                \n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                (\n                x\n                )\n                \n                  \n                    ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0111", "text": "]\n                  \n                \n                \n                  \n                    \n                      )\n                    \n                  \n                  \n                    2", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0112", "text": "{\\displaystyle {\\begin{aligned}{\\color {Blue}\\mathbb {E} {\\Big [}{\\big (}f(x)-\\mathbb {E} {\\big [}{\\hat {f}}(x){\\big ]}{\\big )}^{2}{\\Big ]}}&=\\mathbb {E} {\\big [}f(x)^{2}{\\big ]}\\,-\\,2\\ \\mathbb {E} {\\Big [}f(x)\\ \\mathbb {E} {\\big [}{\\hat {f}}(x){\\big ]}{\\Big ]}\\,+\\,\\mathbb {E} {\\Big [}\\mathbb {E} {\\big [}{\\hat {f}}(x){\\big ]}^{2}{\\Big ]}\\\\&=f(x)^{2}\\,-\\,2\\ f(x)\\ \\mathbb {E} {\\big [}{\\hat {f}}(x){\\big ]}\\,+\\,\\mathbb {E} {\\big [}{\\hat {f}}(x){\\big ]}^{2}\\\\&={\\Big (}f(x)-\\mathbb {E} {\\big", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0113", "text": "]}^{2}\\\\&={\\Big (}f(x)-\\mathbb {E} {\\big [}{\\hat {f}}(x){\\big ]}{\\Big )}^{2}\\end{aligned}}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0114", "text": "This last series of equalities comes from the fact that \n  \n    \n      \n        f\n        (\n        x\n        )\n      \n    \n    {\\displaystyle f(x)}\n  \n is not a random variable, but a fixed, deterministic function of \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n. Therefore, \n  \n    \n      \n        \n          \n            E\n          \n        \n        \u2061\n        \n          [\n          \n            f\n            (\n            x\n            )\n          \n          ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0115", "text": ")\n          \n          ]\n        \n        =\n        f\n        (\n        x\n        )\n      \n    \n    {\\displaystyle \\operatorname {\\mathbb {E} } \\left[f(x)\\right]=f(x)}\n  \n. Similarly \n  \n    \n      \n        \n          \n            E\n          \n        \n        \u2061\n        \n          [\n          \n            f\n            (\n            x\n            \n              )\n              \n                2\n              \n            \n          \n          ]\n        \n        =\n        f", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0116", "text": "]\n        \n        =\n        f\n        (\n        x\n        \n          )\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\operatorname {\\mathbb {E} } \\left[f(x)^{2}\\right]=f(x)^{2}}\n  \n, and \n  \n    \n      \n        \n          \n            E\n          \n        \n        \u2061\n        \n          [\n          \n            f\n            (\n            x\n            )\n            \n            \n              \n                E\n              \n            \n            \u2061", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0117", "text": "\u2061\n            [\n            \n              \n                \n                  f\n                  ^\n                \n              \n            \n            \n            (\n            x\n            )\n            ]\n          \n          ]\n        \n        =\n        f\n        (\n        x\n        )\n        \n        \n          \n            E\n          \n        \n        \u2061\n        \n          [\n          \n            \n              \n                E", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0118", "text": "E\n              \n            \n            \u2061\n            [\n            \n              \n                \n                  f\n                  ^\n                \n              \n            \n            \n            (\n            x\n            )\n            ]\n          \n          ]\n        \n        =\n        f\n        (\n        x\n        )\n        \n          \n            E\n          \n        \n        \u2061\n        [", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0119", "text": "[\n        \n          \n            \n              f\n              ^\n            \n          \n        \n        \n        (\n        x\n        )\n        ]\n      \n    \n    {\\displaystyle \\operatorname {\\mathbb {E} } \\left[f(x)\\,\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]\\right]=f(x)\\,\\operatorname {\\mathbb {E} } \\left[\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]\\right]=f(x)\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0120", "text": ". Using the same reasoning, we can expand the second term and show that it is null:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0121", "text": "E\n                      \n                    \n                    \u2061\n                    \n                      [\n                      \n                        \n                          (\n                          \n                            f\n                            (\n                            x", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0122", "text": "x\n                            )\n                            \u2212\n                            \n                              \n                                E\n                              \n                            \n                            \u2061\n                            [\n                            \n                              \n                                \n                                  f\n                                  ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0123", "text": "^\n                                \n                              \n                            \n                            \n                            (\n                            x\n                            )\n                            ]\n                          \n                          )\n                        \n                        \n                          (", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0124", "text": "E\n                              \n                            \n                            \u2061\n                            [\n                            \n                              \n                                \n                                  f\n                                  ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0125", "text": "(\n                            x\n                            )\n                            ]\n                            \u2212\n                            \n                              \n                                \n                                  f\n                                  ^\n                                \n                              \n                            \n                            \n                            (", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0126", "text": "(\n                            x\n                            )\n                          \n                          )\n                        \n                      \n                      ]\n                    \n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \n                    E\n                  \n                \n                \u2061", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0127", "text": "\u2061\n                \n                  [\n                  \n                    f\n                    (\n                    x\n                    )\n                    \n                    \n                      \n                        E\n                      \n                    \n                    \u2061\n                    [\n                    \n                      \n                        \n                          f\n                          ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0128", "text": "^\n                        \n                      \n                    \n                    \n                    (\n                    x\n                    )\n                    ]\n                    \n                    \u2212\n                    \n                    f\n                    (\n                    x\n                    )\n                    \n                      \n                        \n                          f\n                          ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0129", "text": "^\n                        \n                      \n                    \n                    \n                    (\n                    x\n                    )\n                    \n                    \u2212\n                    \n                    \n                      \n                        E\n                      \n                    \n                    \u2061\n                    [", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0130", "text": "f\n                          ^\n                        \n                      \n                    \n                    \n                    (\n                    x\n                    )\n                    \n                      ]\n                      \n                        2\n                      \n                    \n                    +\n                    \n                      \n                        E", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0131", "text": "E\n                      \n                    \n                    \u2061\n                    [\n                    \n                      \n                        \n                          f\n                          ^\n                        \n                      \n                    \n                    \n                    (\n                    x\n                    )\n                    ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0132", "text": "f\n                          ^\n                        \n                      \n                    \n                    \n                    (\n                    x\n                    )\n                  \n                  ]\n                \n              \n            \n            \n              \n              \n                \n                =\n                f\n                (", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0133", "text": "f\n                (\n                x\n                )\n                \n                \n                  \n                    E\n                  \n                \n                \u2061\n                [\n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (\n                x\n                )\n                ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0134", "text": "]\n                \n                \u2212\n                \n                f\n                (\n                x\n                )\n                \n                \n                  \n                    E\n                  \n                \n                \u2061\n                [\n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0135", "text": "(\n                x\n                )\n                ]\n                \n                \u2212\n                \n                \n                  \n                    E\n                  \n                \n                \u2061\n                [\n                \n                  \n                    \n                      f\n                      ^\n                    \n                  \n                \n                \n                (\n                x", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0136", "text": "(\n                x\n                )\n                \n                  ]\n                  \n                    2\n                  \n                \n                \n                +\n                \n                \n                  \n                    E\n                  \n                \n                \u2061\n                [\n                \n                  \n                    \n                      f\n                      ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0137", "text": "(\n                x\n                )\n                \n                  ]\n                  \n                    2\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                0", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0138", "text": "{\\displaystyle {\\begin{aligned}&{\\color {PineGreen}\\operatorname {\\mathbb {E} } \\left[\\left(f(x)-\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]\\right)\\left(\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]-{\\hat {f}}\\!(x)\\right)\\right]}\\\\&=\\operatorname {\\mathbb {E} } \\left[f(x)\\,\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]\\,-\\,f(x){\\hat {f}}\\!(x)\\,-\\,\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]^{2}+\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]\\,{\\hat", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0139", "text": "{\\mathbb {E} } [{\\hat {f}}\\!(x)]\\,{\\hat {f}}\\!(x)\\right]\\\\&=f(x)\\,\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]\\,-\\,f(x)\\,\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]\\,-\\,\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]^{2}\\,+\\,\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]^{2}\\\\&=0\\end{aligned}}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0140", "text": "Eventually, we plug our derivations back into the original equation, and identify each term:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0141", "text": "MSE\n                \n                (\n                x\n                )\n              \n              \n                \n                =\n                \n                  \n                    (\n                    \n                      f\n                      (\n                      x\n                      )\n                      \u2212", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0142", "text": "E\n                        \n                      \n                      \u2061\n                      [\n                      \n                        \n                          \n                            f\n                            ^\n                          \n                        \n                      \n                      \n                      (\n                      x\n                      )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0143", "text": "x\n                      )\n                      ]\n                    \n                    )\n                  \n                  \n                    2\n                  \n                \n                +\n                \n                  \n                    E\n                  \n                \n                \u2061\n                \n                  [\n                  \n                    \n                      (", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0144", "text": "E\n                          \n                        \n                        \u2061\n                        [\n                        \n                          \n                            \n                              f\n                              ^\n                            \n                          \n                        \n                        \n                        (", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0145", "text": "(\n                        x\n                        )\n                        ]\n                        \u2212\n                        \n                          \n                            \n                              f\n                              ^\n                            \n                          \n                        \n                        \n                        (\n                        x\n                        )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0146", "text": ")\n                      \n                      )\n                    \n                    \n                      2\n                    \n                  \n                  ]\n                \n                +\n                \n                  \u03c3\n                  \n                    2\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                Bias\n                \u2061", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0147", "text": "Bias\n                \u2061\n                \n                  \n                    [\n                    \n                      \n                        \n                          \n                            f\n                            ^\n                          \n                        \n                      \n                      \n                      (\n                      x\n                      )\n                    \n                    ]", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0148", "text": "]\n                  \n                  \n                    2\n                  \n                \n                +\n                \n                Var\n                \u2061\n                \n                  [\n                  \n                    \n                      \n                        \n                          f\n                          ^\n                        \n                      \n                    \n                    \n                    (", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0149", "text": "(\n                    x\n                    )\n                  \n                  ]\n                \n                \n                +\n                \n                \n                  \u03c3\n                  \n                    2", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0150", "text": "{\\displaystyle {\\begin{aligned}{\\text{MSE}}(x)&=\\left(f(x)-\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]\\right)^{2}+\\operatorname {\\mathbb {E} } \\left[\\left(\\operatorname {\\mathbb {E} } [{\\hat {f}}\\!(x)]-{\\hat {f}}\\!(x)\\right)^{2}\\right]+\\sigma ^{2}\\\\&=\\operatorname {Bias} \\left[{\\hat {f}}\\!(x)\\right]^{2}+\\,\\operatorname {Var} \\left[{\\hat {f}}\\!(x)\\right]\\,+\\,\\sigma ^{2}\\end{aligned}}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0151", "text": "Finally, the MSE loss function (or negative log-likelihood) is obtained by taking the expectation value over \n  \n    \n      \n        x\n        \u223c\n        P\n      \n    \n    {\\displaystyle x\\sim P}\n  \n:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0152", "text": "MSE\n        \n        =\n        \n          \n            \n              E\n            \n          \n          \n            x\n          \n        \n        \u2061\n        \n          [\n          \n            \n              MSE\n            \n            (\n            x\n            )\n          \n          ]\n        \n        =\n        \n          \n            \n              E\n            \n          \n          \n            x\n          \n        \n        \u2061\n        \n          {", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0153", "text": "\u2061\n        \n          {\n          \n            \n              Bias\n              \n                D\n              \n            \n            \n            \n              \n                [\n                \n                  \n                    \n                      \n                        f\n                        ^\n                      \n                    \n                  \n                  \n                  (\n                  x\n                  ;\n                  D", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0154", "text": ";\n                  D\n                  )\n                \n                ]\n              \n              \n                2\n              \n            \n            +\n            \n              Var\n              \n                D\n              \n            \n            \u2061\n            \n              [\n              \n                \n                  \n                    \n                      f\n                      ^", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0155", "text": "(\n                x\n                ;\n                D\n                )\n              \n              ]\n            \n          \n          }\n        \n        +\n        \n          \u03c3\n          \n            2\n          \n        \n        .", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0156", "text": ".\n      \n    \n    {\\displaystyle {\\text{MSE}}=\\operatorname {\\mathbb {E} } _{x}\\left[{\\text{MSE}}(x)\\right]=\\operatorname {\\mathbb {E} } _{x}\\left\\{\\operatorname {Bias} _{D}\\!\\left[{\\hat {f}}\\!(x;D)\\right]^{2}+\\operatorname {Var} _{D}\\left[{\\hat {f}}\\!(x;D)\\right]\\right\\}+\\sigma ^{2}.}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0157", "text": "Approaches\nDimensionality reduction and feature selection can decrease variance by simplifying models. Similarly, a larger training set tends to decrease variance. Adding features (predictors) tends to decrease bias, at the expense of introducing additional variance. Learning algorithms typically have some tunable parameters that control bias and variance; for example,", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0158", "text": "linear and Generalized linear models can be regularized to decrease their variance at the cost of increasing their bias.\nIn artificial neural networks, the variance increases and the bias decreases as the number of hidden units increase, although this classical assumption has been the subject of recent debate. Like in GLMs, regularization is typically applied.\nIn k-nearest neighbor models, a high value of k leads to high bias and low variance (see below).", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0159", "text": "In instance-based learning, regularization can be achieved varying the mixture of prototypes and exemplars.\nIn decision trees, the depth of the tree determines the variance. Decision trees are commonly pruned to control variance.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0160", "text": "One way of resolving the trade-off is to use mixture models and ensemble learning. For example, boosting combines many \"weak\" (high bias) models in an ensemble that has lower bias than the individual models, while bagging combines \"strong\" learners in a way that reduces their variance.\nModel validation methods such as cross-validation (statistics) can be used to tune models so as to optimize the trade-off.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0161", "text": "k-nearest neighbors\nIn the case of k-nearest neighbors regression, when the expectation is taken over the possible labeling of a fixed training set, a closed-form expression exists that relates the bias\u2013variance decomposition to the parameter k:", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0162", "text": "E\n          \n        \n        \u2061\n        \n          [\n          \n            \n              \n                (\n                \n                  y\n                  \u2212\n                  \n                    \n                      \n                        f\n                        ^\n                      \n                    \n                  \n                  \n                  (\n                  x\n                  )", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0163", "text": ")\n                \n                )\n              \n              \n                2\n              \n            \n            \u2223\n            X\n            =\n            x\n          \n          ]\n        \n        =\n        \n          \n            (\n            \n              f\n              (\n              x\n              )\n              \u2212\n              \n                \n                  1\n                  k\n                \n              \n              \n                \u2211", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0164", "text": "\u2211\n                \n                  i\n                  =\n                  1\n                \n                \n                  k\n                \n              \n              f\n              (\n              \n                N\n                \n                  i\n                \n              \n              (\n              x\n              )\n              )\n            \n            )\n          \n          \n            2", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0165", "text": "2\n          \n        \n        +\n        \n          \n            \n              \u03c3\n              \n                2\n              \n            \n            k\n          \n        \n        +\n        \n          \u03c3\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\operatorname {\\mathbb {E} } \\left[\\left(y-{\\hat {f}}\\!(x)\\right)^{2}\\mid X=x\\right]=\\left(f(x)-{\\frac {1}{k}}\\sum _{i=1}^{k}f(N_{i}(x))\\right)^{2}+{\\frac {\\sigma ^{2}}{k}}+\\sigma ^{2}}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0166", "text": "where \n  \n    \n      \n        \n          N\n          \n            1\n          \n        \n        (\n        x\n        )\n        ,\n        \u2026\n        ,\n        \n          N\n          \n            k\n          \n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle N_{1}(x),\\dots ,N_{k}(x)}", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0167", "text": "{\\displaystyle N_{1}(x),\\dots ,N_{k}(x)}\n  \n are the k nearest neighbors of x in the training set. The bias (first term) is a monotone rising function of k, while the variance (second term) drops off as k is increased. In fact, under \"reasonable assumptions\" the bias of the first-nearest neighbor (1-NN) estimator vanishes entirely as the size of the training set approaches infinity.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0168", "text": "Applications\nIn regression\nThe bias\u2013variance decomposition forms the conceptual basis for regression regularization methods such as LASSO and ridge regression. Regularization methods introduce bias into the regression solution that can reduce variance considerably relative to the ordinary least squares (OLS) solution. Although the OLS solution provides non-biased regression estimates, the lower variance solutions produced by regularization techniques provide superior MSE performance.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0169", "text": "In classification", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0170", "text": "The bias\u2013variance decomposition was originally formulated for least-squares regression. For the case of classification under the 0-1 loss (misclassification rate), it is possible to find a similar decomposition, with the caveat that the variance term becomes dependent on the target label", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0171", "text": ". Alternatively, if the classification problem can be phrased as probabilistic classification, then the expected cross-entropy can instead be decomposed to give bias and variance terms with the same semantics but taking a different form.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0172", "text": "It has been argued that as training data increases, the variance of learned models will tend to decrease, and hence that as training data quantity increases, error is minimised by methods that learn models with lesser bias, and that conversely, for smaller training data quantities it is ever more important to minimise variance.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0173", "text": "In reinforcement learning", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0174", "text": "Even though the bias\u2013variance decomposition does not directly apply in reinforcement learning, a similar tradeoff can also characterize generalization. When an agent has limited information on its environment, the suboptimality of an RL algorithm can be decomposed into the sum of two terms: a term related to an asymptotic bias and a term due to overfitting", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0175", "text": ". The asymptotic bias is directly related to the learning algorithm (independently of the quantity of data) while the overfitting term comes from the fact that the amount of data is limited.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0176", "text": "In Monte Carlo methods", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0177", "text": "While in traditional Monte Carlo methods the bias is typically zero, modern approaches, such as Markov chain Monte Carlo are only asymptotically unbiased, at best. Convergence diagnostics can be used to control bias via burn-in removal, but due to a limited computational budget, a bias\u2013variance trade-off arises, leading to a wide range of approaches, in which a controlled bias is accepted, if this allows to dramatically reduce the variance, and hence the overall estimation error.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0178", "text": "In human learning", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0179", "text": "While widely discussed in the context of machine learning, the bias\u2013variance dilemma has been examined in the context of human cognition, most notably by Gerd Gigerenzer and co-workers in the context of learned heuristics. They have argued (see references below) that the human brain resolves the dilemma in the case of the typically sparse, poorly-characterized training-sets provided by experience by adopting high-bias/low variance heuristics", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0180", "text": ". This reflects the fact that a zero-bias approach has poor generalizability to new situations, and also unreasonably presumes precise knowledge of the true state of the world. The resulting heuristics are relatively simple, but produce better inferences in a wider variety of situations.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0181", "text": "Geman et al. argue that the bias\u2013variance dilemma implies that abilities such as generic object recognition cannot be learned from scratch, but require a certain degree of \"hard wiring\" that is later tuned by experience. This is because model-free approaches to inference require impractically large training sets if they are to avoid high variance.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Biasvariance_tradeoff_0182", "text": "See also\nReferences\nExternal links\nMLU-Explain: The Bias Variance Tradeoff \u2014 An interactive visualization of the bias\u2013variance tradeoff in LOESS Regression and K-Nearest Neighbors.", "title": "Bias\u2013variance tradeoff", "url": "https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff"}, {"id": "rec_Curse_of_dimensionality_0000", "text": "The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces that do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience.  The expression was coined by Richard E. Bellman when considering problems in dynamic programming", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0001", "text": ". The curse generally refers to issues that arise when the number of datapoints is small (in a suitably defined sense) relative to the intrinsic dimension of the data.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0002", "text": "Dimensionally cursed phenomena occur in domains such as numerical analysis, sampling, combinatorics, machine learning, data mining and databases. The common theme of these problems is that when the dimensionality increases, the volume of the space increases so fast that the available data become sparse. In order to obtain a reliable result, the amount of data needed often grows exponentially with the dimensionality", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0003", "text": ". Also, organizing and searching data often relies on detecting areas where objects form groups with similar properties; in high dimensional data, however, all objects appear to be sparse and dissimilar in many ways, which prevents common data organization strategies from being efficient.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0004", "text": "Domains\nCombinatorics\nIn some problems, each variable can take one of several discrete values, or the range of possible values is divided to give a finite number of possibilities. Taking the variables together, a huge number of combinations of values must be considered. This effect is also known as the combinatorial explosion. Even in the simplest case of \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n binary variables, the number of possible combinations already is", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0005", "text": "2\n          \n            d\n          \n        \n      \n    \n    {\\displaystyle 2^{d}}\n  \n, exponential in the dimensionality. Naively, each additional dimension doubles the effort needed to try all combinations.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0006", "text": "Sampling", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0007", "text": "There is an exponential increase in volume associated with adding extra dimensions to a mathematical space. For example, 102 =\u2009100 evenly spaced sample points suffice to sample a unit interval (try to visualize a \"1-dimensional\" cube, i.e. a line) with no more than 10\u22122 = 0.01 distance between points; an equivalent sampling of a 10-dimensional unit hypercube with a lattice that has a spacing of 10\u22122 = 0.01 between adjacent points would require 1020 = [(102)10] sample points", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0008", "text": ". In general, with a spacing distance of 10\u2212n the 10-dimensional hypercube appears to be a factor of 10n(10\u22121) = [(10n)10/(10n)] \"larger\" than the 1-dimensional hypercube, which is the unit interval. In the above example n = 2: when using a sampling distance of 0.01 the 10-dimensional hypercube appears to be 1018 \"larger\" than the unit interval. This effect is a combination of the combinatorics problems above and the distance function problems explained below.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0009", "text": "Optimization\nWhen solving dynamic optimization problems by numerical backward induction, the objective function must be computed for each combination of values. This is a significant obstacle when the dimension of the \"state variable\" is large.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0010", "text": "Machine learning\nIn machine learning problems that involve learning a \"state-of-nature\" from a finite number of data samples in a high-dimensional feature space with each feature having a range of possible values, typically an enormous amount of training data is required to ensure that there are several samples with each combination of values. In an abstract sense, as the number of features or dimensions grows, the amount of data we need to generalize accurately grows exponentially.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0011", "text": "A typical rule of thumb is that there should be at least 5 training examples for each dimension in the representation. In machine learning and insofar as predictive performance is concerned, the curse of dimensionality is used interchangeably with the peaking phenomenon, which is also known as Hughes phenomenon", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0012", "text": ". This phenomenon states that with a fixed number of training samples, the average (expected) predictive power of a classifier or regressor first increases as the number of dimensions or features used is increased but beyond a certain dimensionality it starts deteriorating instead of improving steadily.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0013", "text": "Nevertheless, in the context of a simple classifier (e.g., linear discriminant analysis in the multivariate Gaussian model under the assumption of a common known covariance matrix), Zollanvari et al", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0014", "text": ". showed both analytically and empirically that as long as the relative cumulative efficacy of an additional feature set (with respect to features that are already part of the classifier) is greater (or less) than the size of this additional feature set, the expected error of the classifier constructed using these additional features will be less (or greater) than the expected error of the classifier constructed without them", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0015", "text": ". In other words, both the size of additional features and their (relative) cumulative discriminatory effect are important in observing a decrease or increase in the average predictive power.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0016", "text": "In metric learning, higher dimensions can sometimes allow a model to achieve better performance. After normalizing embeddings to the surface of a hypersphere, FaceNet achieves the best performance using 128 dimensions as opposed to 64, 256, or 512 dimensions in one ablation study. A loss function for unitary-invariant dissimilarity between word embeddings was found to be minimized in high dimensions.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0017", "text": "Data mining\nIn data mining, the curse of dimensionality refers to a data set with too many features. \nConsider the first table, which depicts 200 individuals and 2000 genes (features) with a 1 or 0 denoting whether or not they have a genetic mutation in that gene. A data mining application to this data set may be finding the correlation between specific genetic mutations and creating a classification algorithm such as a decision tree to determine whether an individual has cancer or not.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0018", "text": "A common practice of data mining in this domain would be to create association rules between genetic mutations that lead to the development of cancers. To do this, one would have to loop through each genetic mutation of each individual and find other genetic mutations that occur over a desired threshold and create pairs. They would start with pairs of two, then three, then four until they result in an empty set of pairs", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0019", "text": ". The complexity of this algorithm can lead to calculating all permutations of gene pairs for each individual or row. Given the formula for calculating the permutations of n items with a group size of r is:", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0020", "text": "n\n              !\n            \n            \n              (\n              n\n              \u2212\n              r\n              )\n              !\n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {n!}{(n-r)!}}}", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0021", "text": "{\\displaystyle {\\frac {n!}{(n-r)!}}}\n  \n, calculating the number of three pair permutations of any given individual would be 7988004000 different pairs of genes to evaluate for each individual. The number of pairs created will grow by an order of factorial as the size of the pairs increase. The growth is depicted in the permutation table (see right).", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0022", "text": "As we can see from the permutation table above, one of the major problems data miners face regarding the curse of dimensionality is that the space of possible parameter values grows exponentially or factorially as the number of features in the data set grows. This problem critically affects both computational time and space when searching for associations or optimal features to consider.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0023", "text": "Another problem data miners may face when dealing with too many features is that the number of false predictions or classifications tends to increase as the number of features grows in the data set. In terms of the classification problem discussed above, keeping every data point could lead to a higher number of false positives and false negatives in the model.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0024", "text": "This may seem counterintuitive, but consider the genetic mutation table from above, depicting all genetic mutations for each individual. Each genetic mutation, whether they correlate with cancer or not, will have some input or weight in the model that guides the decision-making process of the algorithm. There may be mutations that are outliers or ones that dominate the overall distribution of genetic mutations when in fact they do not correlate with cancer", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0025", "text": ". These features may be working against one's model, making it more difficult to obtain optimal results.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0026", "text": "This problem is up to the data miner to solve, and there is no universal solution. The first step any data miner should take is to explore the data, in an attempt to gain an understanding of how it can be used to solve the problem. One must first understand what the data means, and what they are trying to discover before they can decide if anything must be removed from the data set", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0027", "text": ". Then they can create or use a feature selection or dimensionality reduction algorithm to remove samples or features from the data set if they deem it necessary. One example of such methods is the interquartile range method, used to remove outliers in a data set by calculating the standard deviation of a feature or occurrence.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0028", "text": "Distance function\nWhen a measure such as a Euclidean distance is defined using many coordinates, there is little difference in the distances between different pairs of points.\nOne way to illustrate the \"vastness\" of high-dimensional Euclidean space is to compare the proportion of an inscribed hypersphere with radius \n  \n    \n      \n        r\n      \n    \n    {\\displaystyle r}\n  \n and dimension \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0029", "text": "d\n      \n    \n    {\\displaystyle d}\n  \n, to that of a hypercube with edges of length \n  \n    \n      \n        2\n        r\n        .\n      \n    \n    {\\displaystyle 2r.}", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0030", "text": "The volume of such a sphere is \n  \n    \n      \n        \n          \n            \n              2\n              \n                r\n                \n                  d\n                \n              \n              \n                \u03c0\n                \n                  d\n                  \n                    /\n                  \n                  2\n                \n              \n            \n            \n              d\n              \n              \u0393\n              (\n              d", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0031", "text": "\u0393\n              (\n              d\n              \n                /\n              \n              2\n              )\n            \n          \n        \n      \n    \n    {\\displaystyle {\\frac {2r^{d}\\pi ^{d/2}}{d\\;\\Gamma (d/2)}}}\n  \n, where \n  \n    \n      \n        \u0393\n      \n    \n    {\\displaystyle \\Gamma }\n  \n is the gamma function, while the volume of the cube is \n  \n    \n      \n        (\n        2\n        r\n        \n          )\n          \n            d", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0032", "text": "d\n          \n        \n      \n    \n    {\\displaystyle (2r)^{d}}\n  \n.\nAs the dimension \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n of the space increases, the hypersphere becomes an insignificant volume relative to that of the hypercube. This can clearly be seen by comparing the proportions as the dimension \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n goes to infinity:", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0033", "text": "V\n              \n                \n                  h\n                  y\n                  p\n                  e\n                  r\n                  s\n                  p\n                  h\n                  e\n                  r\n                  e\n                \n              \n            \n            \n              V\n              \n                \n                  h\n                  y\n                  p", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0034", "text": "y\n                  p\n                  e\n                  r\n                  c\n                  u\n                  b\n                  e\n                \n              \n            \n          \n        \n        =\n        \n          \n            \n              \u03c0\n              \n                d\n                \n                  /\n                \n                2\n              \n            \n            \n              d\n              \n                2", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0035", "text": "2\n                \n                  d\n                  \u2212\n                  1\n                \n              \n              \u0393\n              (\n              d\n              \n                /\n              \n              2\n              )\n            \n          \n        \n        \u2192\n        0\n      \n    \n    {\\displaystyle {\\frac {V_{\\mathrm {hypersphere} }}{V_{\\mathrm {hypercube} }}}={\\frac {\\pi ^{d/2}}{d2^{d-1}\\Gamma (d/2)}}\\rightarrow 0}\n  \n as", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0036", "text": "as \n  \n    \n      \n        d\n        \u2192\n        \u221e\n      \n    \n    {\\displaystyle d\\rightarrow \\infty }\n  \n.\nFurthermore, the distance between the center and the corners is \n  \n    \n      \n        r\n        \n          \n            d\n          \n        \n      \n    \n    {\\displaystyle r{\\sqrt {d}}}\n  \n, which increases without bound for fixed r.\nIn this sense when points are uniformly generated in a high-dimensional hypercube, almost all points are much farther than \n  \n    \n      \n        r", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0037", "text": "r\n      \n    \n    {\\displaystyle r}\n  \n units away from the center. In high dimensions, the volume of the d-dimensional unit hypercube (with coordinates of the vertices \n  \n    \n      \n        \u00b1\n        1\n      \n    \n    {\\displaystyle \\pm 1}\n  \n)  is concentrated near a sphere with the radius \n  \n    \n      \n        \n          \n            d\n          \n        \n        \n          /\n        \n        \n          \n            3", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0038", "text": "3\n          \n        \n      \n    \n    {\\displaystyle {\\sqrt {d}}/{\\sqrt {3}}}\n  \n for large dimension d. Indeed, for each coordinate \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle x_{i}}\n  \n the average value of  \n  \n    \n      \n        \n          x\n          \n            i\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle x_{i}^{2}}\n  \n in the cube is", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0039", "text": "\u27e8\n          \n            x\n            \n              i\n            \n            \n              2\n            \n          \n          \u27e9\n        \n        =\n        \n          \n            1\n            2\n          \n        \n        \n          \u222b\n          \n            \u2212\n            1\n          \n          \n            1\n          \n        \n        \n          x\n          \n            2\n          \n        \n        d\n        x\n        =", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0040", "text": "d\n        x\n        =\n        \n          \n            1\n            3\n          \n        \n      \n    \n    {\\displaystyle \\left\\langle x_{i}^{2}\\right\\rangle ={\\frac {1}{2}}\\int _{-1}^{1}x^{2}dx={\\frac {1}{3}}}\n  \n.\nThe variance of  \n  \n    \n      \n        \n          x\n          \n            i\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle x_{i}^{2}}\n  \n for uniform distribution in the cube is", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0041", "text": "1\n            2\n          \n        \n        \n          \u222b\n          \n            \u2212\n            1\n          \n          \n            1\n          \n        \n        \n          x\n          \n            4\n          \n        \n        d\n        x\n        \u2212\n        \n          \n            \u27e8\n            \n              x\n              \n                i\n              \n              \n                2\n              \n            \n            \u27e9", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0042", "text": "\u27e9\n          \n          \n            2\n          \n        \n        =\n        \n          \n            4\n            45\n          \n        \n      \n    \n    {\\displaystyle {\\frac {1}{2}}\\int _{-1}^{1}x^{4}dx-\\left\\langle x_{i}^{2}\\right\\rangle ^{2}={\\frac {4}{45}}}", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0043", "text": "Therefore, the squared distance from the origin, \n  \n    \n      \n        \n          r\n          \n            2\n          \n        \n        =\n        \n          \u2211\n          \n            i\n          \n        \n        \n          x\n          \n            i\n          \n          \n            2\n          \n        \n      \n    \n    {\\textstyle r^{2}=\\sum _{i}x_{i}^{2}}\n  \n has the average value d/3 and variance 4d/45. For large d, distribution of \n  \n    \n      \n        \n          r", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0044", "text": "r\n          \n            2\n          \n        \n        \n          /\n        \n        d\n      \n    \n    {\\displaystyle r^{2}/d}\n  \n is close to the normal distribution with the mean 1/3 and the standard deviation \n  \n    \n      \n        2\n        \n          /\n        \n        \n          \n            45\n            d\n          \n        \n      \n    \n    {\\displaystyle 2/{\\sqrt {45d}}}", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0045", "text": "{\\displaystyle 2/{\\sqrt {45d}}}\n  \n according to the central limit theorem. Thus, when uniformly generating points in high dimensions, both the \"middle\" of the hypercube, and the corners are empty, and all the volume is concentrated near the surface of a sphere of \"intermediate\" radius \n  \n    \n      \n        \n          \n            d\n            \n              /\n            \n            3\n          \n        \n      \n    \n    {\\textstyle {\\sqrt {d/3}}}\n  \n.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0046", "text": "This also helps to understand the chi-squared distribution. Indeed, the (non-central) chi-squared distribution associated to a random point in the interval [-1, 1] is the same as the distribution of the length-squared of a random point in the d-cube. By the law of large numbers, this distribution concentrates itself in a narrow band around d times the standard deviation squared (\u03c32) of the original derivation", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0047", "text": ". This illuminates the chi-squared distribution and also illustrates that most of the volume of the d-cube concentrates near the boundary of a sphere of radius", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0048", "text": "\u03c3\n        \n          \n            d\n          \n        \n      \n    \n    {\\displaystyle \\sigma {\\sqrt {d}}}\n  \n.\nA further development of this phenomenon is as follows. Any fixed distribution on the real numbers induces a product distribution on points in \n  \n    \n      \n        \n          \n            R\n          \n          \n            d\n          \n        \n      \n    \n    {\\displaystyle \\mathbb {R} ^{d}}", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0049", "text": "{\\displaystyle \\mathbb {R} ^{d}}\n  \n. For any fixed n, it turns out that the difference between the minimum and the maximum distance between a random reference point Q and a list of n random data points P1,...,Pn become indiscernible compared to the minimum distance:", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0050", "text": "lim\n          \n            d\n            \u2192\n            \u221e\n          \n        \n        E\n        \n          (\n          \n            \n              \n                \n                  dist\n                  \n                    max\n                  \n                \n                \u2061\n                (\n                d\n                )\n                \u2212\n                \n                  dist\n                  \n                    min", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0051", "text": "min\n                  \n                \n                \u2061\n                (\n                d\n                )\n              \n              \n                \n                  dist\n                  \n                    min\n                  \n                \n                \u2061\n                (\n                d\n                )\n              \n            \n          \n          )\n        \n        \u2192\n        0", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0052", "text": "\u2192\n        0\n      \n    \n    {\\displaystyle \\lim _{d\\to \\infty }E\\left({\\frac {\\operatorname {dist} _{\\max }(d)-\\operatorname {dist} _{\\min }(d)}{\\operatorname {dist} _{\\min }(d)}}\\right)\\to 0}\n  \n.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0053", "text": ".\nThis is often cited as distance functions losing their usefulness (for the nearest-neighbor criterion in feature-comparison algorithms, for example) in high dimensions. However, recent research has shown this to only hold in the artificial scenario when the one-dimensional distributions \n  \n    \n      \n        \n          R\n        \n      \n    \n    {\\displaystyle \\mathbb {R} }", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0054", "text": "{\\displaystyle \\mathbb {R} }\n  \n are independent and identically distributed. When attributes are correlated, data can become easier and provide higher distance contrast and the signal-to-noise ratio was found to play an important role, thus feature selection should be used.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0055", "text": "More recently, it has been suggested that there may be a conceptual flaw in the argument that contrast-loss creates a curse in high dimensions. Machine learning can be understood as the problem of assigning instances to their respective generative process of origin, with class labels acting as symbolic representations of individual generative processes.  The curse's derivation assumes all instances are independent, identical outcomes of a single high dimensional generative process", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0056", "text": ". If there is only one generative process, there would exist only one (naturally occurring) class and machine learning would be conceptually ill-defined in both high and low dimensions. Thus, the traditional argument that contrast-loss creates a curse, may be fundamentally inappropriate", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0057", "text": ". In addition, it has been shown that when the generative model is modified to accommodate multiple generative processes, contrast-loss can morph from a curse to a blessing, as it ensures that the nearest-neighbor of an instance is almost-surely its most closely related instance. From this perspective, contrast-loss makes high dimensional distances especially meaningful and not especially non-meaningful as is often argued.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0058", "text": "Nearest neighbor search\nThe effect complicates nearest neighbor search in high dimensional space. It is not possible to quickly reject candidates by using the difference in one coordinate as a lower bound for a distance based on all the dimensions.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0059", "text": "However, it has recently been observed that the mere number of dimensions does not necessarily result in difficulties, since relevant additional dimensions can also increase the contrast. In addition, for the resulting ranking it remains useful to discern close and far neighbors. Irrelevant (\"noise\") dimensions, however, reduce the contrast in the manner described above", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0060", "text": ". In time series analysis, where the data are inherently high-dimensional, distance functions also work reliably as long as the signal-to-noise ratio is high enough.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0061", "text": "k-nearest neighbor classification", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0062", "text": "Another effect of high dimensionality on distance functions concerns k-nearest neighbor (k-NN) graphs constructed from a data set using a distance function. As the dimension increases, the indegree distribution of the k-NN digraph becomes skewed with a peak on the right because of the emergence of a disproportionate number of hubs, that is, data-points that appear in many more k-NN lists of other data-points than the average", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0063", "text": ". This phenomenon can have a considerable impact on various techniques for classification (including the k-NN classifier), semi-supervised learning, and clustering, and it also affects information retrieval.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0064", "text": "Anomaly detection\nIn a 2012 survey, Zimek et al. identified the following problems when searching for anomalies in high-dimensional data:", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0065", "text": "Concentration of scores and distances: derived values such as distances become numerically similar\nIrrelevant attributes: in high dimensional data, a significant number of attributes may be irrelevant\nDefinition of reference sets: for local methods, reference sets are often nearest-neighbor based\nIncomparable scores for different dimensionalities: different subspaces produce incomparable scores\nInterpretability of scores: the scores often no longer convey a semantic meaning", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0066", "text": "Exponential search space: the search space can no longer be systematically scanned\nData snooping bias: given the large search space, for every desired significance a hypothesis can be found\nHubness: certain objects occur more frequently in neighbor lists than others.\nMany of the analyzed specialized methods tackle one or another of these problems, but there remain many open research questions.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0067", "text": "Blessing of dimensionality", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0068", "text": "Despite the expected  \"curse of dimensionality\" difficulties, common-sense heuristics based on the  most straightforward methods \"can yield results which are almost surely optimal\" for high-dimensional problems. The term \"blessing of dimensionality\" was introduced in the late 1990s. Donoho in his \"Millennium manifesto\" explained why he thinks the \"blessing of dimensionality\" will form a basis of future data mining", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0069", "text": ". The effects of the blessing of dimensionality were discovered in many applications and found their foundation in the concentration of measure phenomena. One example of the blessing of dimensionality phenomenon is linear separability of a random point from a large finite random set with high probability even if this set is exponentially large: the number of elements in this random set can grow exponentially with dimension", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0070", "text": ". Moreover, this linear functional can be selected in the form of the simplest linear Fisher discriminant. This separability theorem was proven for a wide class of probability distributions: general uniformly log-concave distributions, product distributions in a cube and many other families (reviewed recently in ).", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0071", "text": "\"The blessing of dimensionality and the curse of dimensionality are two sides of the same coin.\" For example, the typical property of essentially high-dimensional probability distributions in a high-dimensional space is: the squared distance of random points to a selected point is, with high probability, close to the average (or median) squared distance", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0072", "text": ". This property significantly simplifies the expected geometry of data and indexing of high-dimensional data (blessing), but, at the same time, it makes the similarity search in high dimensions difficult and even useless (curse).", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0073", "text": "Zimek et al. noted that while the typical formalizations of the curse of dimensionality affect i.i.d. data, having data that is separated in each attribute becomes easier even in high dimensions, and argued that the signal-to-noise ratio matters: data becomes easier with each attribute that adds signal, and harder with attributes that only add noise (irrelevant error) to the data. In particular for unsupervised data analysis this effect is known as swamping.", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Curse_of_dimensionality_0074", "text": "See also\n\n\n== References ==", "title": "Curse of dimensionality", "url": "https://en.wikipedia.org/wiki/Curse_of_dimensionality"}, {"id": "rec_Kmeans_clustering_0000", "text": "k-means clustering is a method of vector quantization, originally from signal processing, that aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean (cluster centers or cluster centroid). This results in a partitioning of the data space into Voronoi cells", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0001", "text": ". k-means clustering minimizes within-cluster variances (squared Euclidean distances), but not regular Euclidean distances, which would be the more difficult Weber problem: the mean optimizes squared errors, whereas only the geometric median minimizes Euclidean distances. For instance, better Euclidean solutions can be found using k-medians and k-medoids.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0002", "text": "The problem is computationally difficult (NP-hard); however, efficient heuristic algorithms converge quickly to a local optimum. These are usually similar to the expectation\u2013maximization algorithm for mixtures of Gaussian distributions via an iterative refinement approach employed by both k-means and Gaussian mixture modeling", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0003", "text": ". They both use cluster centers to model the data; however, k-means clustering tends to find clusters of comparable spatial extent, while the Gaussian mixture model allows clusters to have different shapes.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0004", "text": "The unsupervised k-means algorithm has a loose relationship to the k-nearest neighbor classifier, a popular supervised machine learning technique for classification that is often confused with k-means due to the name. Applying the 1-nearest neighbor classifier to the cluster centers obtained by k-means classifies new data into the existing clusters. This is known as nearest centroid classifier or Rocchio algorithm.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0005", "text": "Description\nGiven a set of observations (x1, x2, ..., xn), where each observation is a \n  \n    \n      \n        d\n      \n    \n    {\\displaystyle d}\n  \n-dimensional real vector, k-means clustering aims to partition the n observations into k (\u2264 n) sets S = {S1, S2, ..., Sk} so as to minimize the within-cluster sum of squares (WCSS) (i.e. variance). Formally, the objective is to find:", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0006", "text": "a\n              r\n              g\n              \n              m\n              i\n              n\n            \n          \n          \n            \n              S\n            \n          \n        \n        \u2061\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            k\n          \n        \n        \n          \u2211\n          \n            \n              x\n            \n            \u2208", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0007", "text": "\u2208\n            \n              S\n              \n                i\n              \n            \n          \n        \n        \n          \n            \u2016\n            \n              \n                x\n              \n              \u2212\n              \n                \n                  \u03bc\n                \n                \n                  i\n                \n              \n            \n            \u2016\n          \n          \n            2\n          \n        \n        =", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0008", "text": "=\n        \n          \n            \n              a\n              r\n              g\n              \n              m\n              i\n              n\n            \n          \n          \n            \n              S\n            \n          \n        \n        \u2061\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            k\n          \n        \n        \n          |\n        \n        \n          S\n          \n            i", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0009", "text": "S\n          \n            i\n          \n        \n        \n          |\n        \n        Var\n        \u2061\n        \n          S\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathop {\\operatorname {arg\\,min} } _{\\mathbf {S} }\\sum _{i=1}^{k}\\sum _{\\mathbf {x} \\in S_{i}}\\left\\|\\mathbf {x} -{\\boldsymbol {\\mu }}_{i}\\right\\|^{2}=\\mathop {\\operatorname {arg\\,min} } _{\\mathbf {S} }\\sum _{i=1}^{k}|S_{i}|\\operatorname {Var} S_{i}}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0010", "text": "where \u03bci is the mean (also called centroid) of points in \n  \n    \n      \n        \n          S\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle S_{i}}\n  \n, i.e.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0011", "text": "\u03bc\n            \n              i\n            \n          \n        \n        =\n        \n          \n            1\n            \n              \n                |\n              \n              \n                S\n                \n                  i\n                \n              \n              \n                |\n              \n            \n          \n        \n        \n          \u2211\n          \n            \n              x\n            \n            \u2208", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0012", "text": "x\n            \n            \u2208\n            \n              S\n              \n                i\n              \n            \n          \n        \n        \n          x\n        \n        ,\n      \n    \n    {\\displaystyle {\\boldsymbol {\\mu _{i}}}={\\frac {1}{|S_{i}|}}\\sum _{\\mathbf {x} \\in S_{i}}\\mathbf {x} ,}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0013", "text": "|\n        \n        \n          S\n          \n            i\n          \n        \n        \n          |\n        \n      \n    \n    {\\displaystyle |S_{i}|}\n  \n is the size of \n  \n    \n      \n        \n          S\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle S_{i}}\n  \n, and \n  \n    \n      \n        \u2016\n        \u22c5\n        \u2016\n      \n    \n    {\\displaystyle \\|\\cdot \\|}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0014", "text": "{\\displaystyle \\|\\cdot \\|}\n  \n is the usual L2 norm . This is equivalent to minimizing the pairwise squared deviations of points in the same cluster:", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0015", "text": "a\n              r\n              g\n              \n              m\n              i\n              n\n            \n          \n          \n            \n              S\n            \n          \n        \n        \u2061\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            k\n          \n        \n        \n        \n          \n            1\n            \n              \n                |", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0016", "text": "|\n              \n              \n                S\n                \n                  i\n                \n              \n              \n                |\n              \n            \n          \n        \n        \n        \n          \u2211\n          \n            \n              x\n            \n            ,\n            \n              y\n            \n            \u2208\n            \n              S\n              \n                i", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0017", "text": "\u2016\n            \n              \n                x\n              \n              \u2212\n              \n                y\n              \n            \n            \u2016\n          \n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\mathop {\\operatorname {arg\\,min} } _{\\mathbf {S} }\\sum _{i=1}^{k}\\,{\\frac {1}{|S_{i}|}}\\,\\sum _{\\mathbf {x} ,\\mathbf {y} \\in S_{i}}\\left\\|\\mathbf {x} -\\mathbf {y} \\right\\|^{2}}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0018", "text": "The equivalence can be deduced from the identity \n  \n    \n      \n        \n          |\n        \n        \n          S\n          \n            i\n          \n        \n        \n          |\n        \n        \n          \u2211\n          \n            \n              x\n            \n            \u2208\n            \n              S\n              \n                i\n              \n            \n          \n        \n        \n          \n            \u2016\n            \n              \n                x", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0019", "text": "x\n              \n              \u2212\n              \n                \n                  \u03bc\n                \n                \n                  i\n                \n              \n            \n            \u2016\n          \n          \n            2\n          \n        \n        =\n        \n          \n            1\n            2\n          \n        \n        \n          \u2211\n          \n            \n              x\n            \n            ,\n            \n              y", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0020", "text": "y\n            \n            \u2208\n            \n              S\n              \n                i\n              \n            \n          \n        \n        \n          \n            \u2016\n            \n              \n                x\n              \n              \u2212\n              \n                y\n              \n            \n            \u2016\n          \n          \n            2", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0021", "text": "2\n          \n        \n      \n    \n    {\\textstyle |S_{i}|\\sum _{\\mathbf {x} \\in S_{i}}\\left\\|\\mathbf {x} -{\\boldsymbol {\\mu }}_{i}\\right\\|^{2}={\\frac {1}{2}}\\sum _{\\mathbf {x} ,\\mathbf {y} \\in S_{i}}\\left\\|\\mathbf {x} -\\mathbf {y} \\right\\|^{2}}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0022", "text": ". Since the total variance is constant, this is equivalent to maximizing the sum of squared deviations between points in different clusters (between-cluster sum of squares, BCSS). This deterministic relationship is also related to the law of total variance in probability theory.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0023", "text": "History\nThe term \"k-means\" was first used by James MacQueen in 1967, though the idea goes back to Hugo Steinhaus in 1956. The standard algorithm was first proposed by Stuart Lloyd of Bell Labs in 1957 as a technique for pulse-code modulation, although it was not published as a journal article until 1982. In 1965, Edward W. Forgy published essentially the same method, which is why it is sometimes referred to as the Lloyd\u2013Forgy algorithm.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0024", "text": "Algorithms\nStandard algorithm (naive k-means)\nThe most common algorithm uses an iterative refinement technique. Due to its ubiquity, it is often called \"the k-means algorithm\"; it is also referred to as Lloyd's algorithm, particularly in the computer science community. It is sometimes also referred to as \"na\u00efve k-means\", because there exist much faster alternatives.\nGiven an initial set of k means m1(1), ..., mk(1) (see below), the algorithm proceeds by alternating between two steps:", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0025", "text": "Assignment step: Assign each observation to the cluster with the nearest mean (centroid): that with the least squared Euclidean distance. (Mathematically, this means partitioning the observations according to the Voronoi diagram generated by the means.) \n  \n    \n      \n        \n          S\n          \n            i\n          \n          \n            (\n            t\n            )\n          \n        \n        =\n        \n          {\n          \n            \n              x", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0026", "text": "x\n              \n                p\n              \n            \n            :\n            \n              \n                \u2016\n                \n                  \n                    x\n                    \n                      p\n                    \n                  \n                  \u2212\n                  \n                    m\n                    \n                      i\n                    \n                    \n                      (\n                      t", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0027", "text": "(\n                      t\n                      )\n                    \n                  \n                \n                \u2016\n              \n              \n                2\n              \n            \n            \u2264\n            \n              \n                \u2016\n                \n                  \n                    x\n                    \n                      p\n                    \n                  \n                  \u2212\n                  \n                    m", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0028", "text": "m\n                    \n                      j\n                    \n                    \n                      (\n                      t\n                      )\n                    \n                  \n                \n                \u2016\n              \n              \n                2\n              \n            \n             \n            \u2200\n            j\n            ,\n            1\n            \u2264\n            j\n            \u2264\n            k\n          \n          }", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0029", "text": "k\n          \n          }\n        \n        ,\n      \n    \n    {\\displaystyle S_{i}^{(t)}=\\left\\{x_{p}:\\left\\|x_{p}-m_{i}^{(t)}\\right\\|^{2}\\leq \\left\\|x_{p}-m_{j}^{(t)}\\right\\|^{2}\\ \\forall j,1\\leq j\\leq k\\right\\},}\n  \n where each \n  \n    \n      \n        \n          x\n          \n            p\n          \n        \n      \n    \n    {\\displaystyle x_{p}}\n  \n is assigned to exactly one \n  \n    \n      \n        \n          S\n          \n            (\n            t\n            )", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0030", "text": "t\n            )\n          \n        \n      \n    \n    {\\displaystyle S^{(t)}}\n  \n, even if it could be assigned to two or more of them.\nUpdate step: Recalculate means (centroids) for observations assigned to each cluster. This is also called refitting. \n  \n    \n      \n        \n          m\n          \n            i\n          \n          \n            (\n            t\n            +\n            1\n            )\n          \n        \n        =\n        \n          \n            1", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0031", "text": "1\n            \n              |\n              \n                S\n                \n                  i\n                \n                \n                  (\n                  t\n                  )\n                \n              \n              |\n            \n          \n        \n        \n          \u2211\n          \n            \n              x\n              \n                j\n              \n            \n            \u2208\n            \n              S", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0032", "text": "S\n              \n                i\n              \n              \n                (\n                t\n                )\n              \n            \n          \n        \n        \n          x\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle m_{i}^{(t+1)}={\\frac {1}{\\left|S_{i}^{(t)}\\right|}}\\sum _{x_{j}\\in S_{i}^{(t)}}x_{j}}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0033", "text": "The objective function in k-means is the WCSS (within cluster sum of squares). After each iteration, the WCSS monotonically decreases, giving a nonnegative monotonically decreasing sequence. This guarantees that the k-means always converges, but not necessarily to the global optimum.\nThe algorithm has converged when the assignments no longer change or equivalently, when the WCSS has become stable. The algorithm is not guaranteed to find the optimal cluster assignment.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0034", "text": "The algorithm is often presented as assigning objects to the nearest cluster by distance. Using a different distance function other than (squared) Euclidean distance may prevent the algorithm from converging. Various modifications of k-means such as spherical k-means and k-medoids have been proposed to allow using other distance measures.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0035", "text": "Pseudocode\nThe below pseudocode outlines the implementation of the standard k-means clustering algorithm. Initialization of centroids, distance metric between points and centroids, and the calculation of new centroids are design choices and will vary with different implementations. In this example pseudocode, distance() returns the distance between the specified points.\n\nfunction kmeans(k, points) is\n    // Initialize centroids\n    centroids \u2190 list of k starting centroids\n    converged \u2190 false", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0036", "text": "while converged == false do\n        // Create empty clusters\n        clusters \u2190 list of k empty lists", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0037", "text": "// Assign each point to the nearest centroid\n        for i \u2190 0 to length(points) - 1 do\n            point \u2190 points[i]\n            closestIndex \u2190 0\n            minDistance \u2190 distance(point, centroids[0])\n            for j \u2190 1 to k - 1 do\n                d \u2190 distance(point, centroids[j])\n                if d < minDistance THEN\n                    minDistance \u2190 d\n                    closestIndex \u2190 j\n            clusters[closestIndex].append(point)", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0038", "text": "// Recalculate centroids as the mean of each cluster\n        newCentroids \u2190 empty list\n        for i \u2190 0 to k - 1 do\n            newCentroid \u2190 calculateCentroid(clusters[i])\n            newCentroids.append(newCentroid)\n\n        // Check for convergence\n        if newCentroids == centroids THEN\n            converged \u2190 true\n        else\n            centroids \u2190 newCentroids\n\n    return clusters", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0039", "text": "Initialization methods", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0040", "text": "Commonly used initialization methods are Forgy and Random Partition. The Forgy method randomly chooses k observations from the dataset and uses these as the initial means. The Random Partition method first randomly assigns a cluster to each observation and then proceeds to the update step, thus computing the initial mean to be the centroid of the cluster's randomly assigned points", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0041", "text": ". The Forgy method tends to spread the initial means out, while Random Partition places all of them close to the center of the data set. According to Hamerly et al., the Random Partition method is generally preferable for algorithms such as the k-harmonic means and fuzzy k-means. For expectation maximization and standard k-means algorithms, the Forgy method of initialization is preferable", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0042", "text": ". A comprehensive study by Celebi et al., however, found that popular initialization methods such as Forgy, Random Partition, and Maximin often perform poorly, whereas Bradley and Fayyad's approach performs \"consistently\" in \"the best group\" and k-means++ performs \"generally well\".", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0043", "text": "The algorithm does not guarantee convergence to the global optimum. The result may depend on the initial clusters. As the algorithm is usually fast, it is common to run it multiple times with different starting conditions. However, worst-case performance can be slow: in particular certain point sets, even in two dimensions, converge in exponential time, that is 2\u03a9(n)", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0044", "text": ". These point sets do not seem to arise in practice: this is corroborated by the fact that the smoothed running time of k-means is polynomial.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0045", "text": "The \"assignment\" step is referred to as the \"expectation step\", while the \"update step\" is a maximization step, making this algorithm a variant of the generalized expectation\u2013maximization algorithm.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0046", "text": "Complexity\nFinding the optimal solution to the k-means clustering problem for observations in d dimensions is:", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0047", "text": "NP-hard in general Euclidean space (of d dimensions) even for two clusters,\nNP-hard for a general number of clusters k even in the plane,\nif k and d (the dimension) are fixed, the problem can be exactly solved in time \n  \n    \n      \n        O\n        (\n        \n          n\n          \n            d\n            k\n            +\n            1\n          \n        \n        )\n      \n    \n    {\\displaystyle O(n^{dk+1})}\n  \n, where n is the number of entities to be clustered.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0048", "text": "Thus, a variety of heuristic algorithms such as Lloyd's algorithm given above are generally used.\nThe running time of Lloyd's algorithm (and most variants) is \n  \n    \n      \n        O\n        (\n        n\n        k\n        d\n        i\n        )\n      \n    \n    {\\displaystyle O(nkdi)}\n  \n, where:", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0049", "text": "n is the number of d-dimensional vectors (to be clustered)\nk the number of clusters\ni the number of iterations needed until convergence.\nOn data that does have a clustering structure, the number of iterations until convergence is often small, and results only improve slightly after the first dozen iterations. Lloyd's algorithm is therefore often considered to be of \"linear\" complexity in practice, although it is in the worst case superpolynomial when performed until convergence.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0050", "text": "In the worst-case, Lloyd's algorithm needs \n  \n    \n      \n        i\n        =\n        \n          2\n          \n            \u03a9\n            (\n            \n              \n                n\n              \n            \n            )\n          \n        \n      \n    \n    {\\displaystyle i=2^{\\Omega ({\\sqrt {n}})}}\n  \n iterations, so that the worst-case complexity of Lloyd's algorithm is superpolynomial.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0051", "text": "Lloyd's k-means algorithm has polynomial smoothed running time. It is shown that for arbitrary set of n points in \n  \n    \n      \n        [\n        0\n        ,\n        1\n        \n          ]\n          \n            d\n          \n        \n      \n    \n    {\\displaystyle [0,1]^{d}}\n  \n, if each point is independently perturbed by a normal distribution with mean 0 and variance \n  \n    \n      \n        \n          \u03c3\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle \\sigma ^{2}}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0052", "text": "{\\displaystyle \\sigma ^{2}}\n  \n, then the expected running time of k-means algorithm is bounded by \n  \n    \n      \n        O\n        (\n        \n          n\n          \n            34\n          \n        \n        \n          k\n          \n            34\n          \n        \n        \n          d\n          \n            8\n          \n        \n        \n          log\n          \n            4\n          \n        \n        \u2061\n        (\n        n\n        )\n        \n          /", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0053", "text": ")\n        \n          /\n        \n        \n          \u03c3\n          \n            6\n          \n        \n        )\n      \n    \n    {\\displaystyle O(n^{34}k^{34}d^{8}\\log ^{4}(n)/\\sigma ^{6})}\n  \n, which is a polynomial in n, k, d and \n  \n    \n      \n        1\n        \n          /\n        \n        \u03c3\n      \n    \n    {\\displaystyle 1/\\sigma }\n  \n.\nBetter bounds are proven for simple cases. For example, it is shown that the running time of k-means algorithm is bounded by \n  \n    \n      \n        O", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0054", "text": "O\n        (\n        d\n        \n          n\n          \n            4\n          \n        \n        \n          M\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle O(dn^{4}M^{2})}\n  \n for n points in an integer lattice \n  \n    \n      \n        {\n        1\n        ,\n        \u2026\n        ,\n        M\n        \n          }\n          \n            d\n          \n        \n      \n    \n    {\\displaystyle \\{1,\\dots ,M\\}^{d}}\n  \n.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0055", "text": ".\nLloyd's algorithm is the standard approach for this problem. However, it spends a lot of processing time computing the distances between each of the k cluster centers and the n data points. Since points usually stay in the same clusters after a few iterations, much of this work is unnecessary, making the na\u00efve implementation very inefficient. Some implementations use caching and the triangle inequality in order to create bounds and accelerate Lloyd's algorithm.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0056", "text": "Optimal number of clusters\nFinding the optimal number of clusters (k) for k-means clustering is a crucial step to ensure that the clustering results are meaningful and useful. Several techniques are available to determine a suitable number of clusters. Here are some of commonly used methods:", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0057", "text": "Elbow method (clustering): This method involves plotting the explained variation as a function of the number of clusters, and picking the elbow of the curve as the number of clusters to use. However, the notion of an \"elbow\" is not well-defined and this is known to be unreliable.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0058", "text": "Silhouette (clustering): Silhouette analysis measures the quality of clustering and provides an insight into the separation distance between the resulting clusters. A higher silhouette score indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0059", "text": "Gap statistic: The Gap Statistic compares the total within intra-cluster variation for different values of k with their expected values under null reference distribution of the data. The optimal k is the value that yields the largest gap statistic.\nDavies\u2013Bouldin index: The Davies-Bouldin index is a measure of the how much separation there is between clusters. Lower values of the Davies-Bouldin index indicate a model with better separation.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0060", "text": "Calinski-Harabasz index: This Index evaluates clusters based on their compactness and separation. The index is calculated using the ratio of between-cluster variance to within-cluster variance, with higher values indicate better-defined clusters.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0061", "text": "Rand index: It calculates the proportion of agreement between the two clusters, considering both the pairs of elements that are correctly assigned to the same or different clusters. Higher values indicate greater similarity and better clustering quality. To provide a more accurate measure, the Adjusted Rand Index (ARI), introduced by Hubert and Arabie in 1985, corrects the Rand Index by adjusting for the expected similarity of all pairings due to chance.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0062", "text": "Variations\nJenks natural breaks optimization: k-means applied to univariate data\nk-medians clustering uses the median in each dimension instead of the mean, and this way minimizes \n  \n    \n      \n        \n          L\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle L_{1}}\n  \n norm (Taxicab geometry).\nk-medoids (also: Partitioning Around Medoids, PAM) uses the medoid instead of the mean, and this way minimizes the sum of distances for arbitrary distance functions.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0063", "text": "Fuzzy C-Means Clustering is a soft version of k-means, where each data point has a fuzzy degree of belonging to each cluster.\nGaussian mixture models trained with expectation\u2013maximization algorithm (EM algorithm) maintains probabilistic assignments to clusters, instead of deterministic assignments, and multivariate Gaussian distributions instead of means.\nk-means++ chooses initial centers in a way that gives a provable upper bound on the WCSS objective.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0064", "text": "The filtering algorithm uses k-d trees to speed up each k-means step.\nSome methods attempt to speed up each k-means step using the triangle inequality.\nEscape local optima by swapping points between clusters.\nThe Spherical k-means clustering algorithm is suitable for textual data.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0065", "text": "Hierarchical variants such as Bisecting k-means, X-means clustering and G-means clustering repeatedly split clusters to build a hierarchy, and can also try to automatically determine the optimal number of clusters in a dataset.\nInternal cluster evaluation measures such as cluster silhouette can be helpful at determining the number of clusters.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0066", "text": "Minkowski weighted k-means automatically calculates cluster specific feature weights, supporting the intuitive idea that a feature may have different degrees of relevance at different features. These weights can also be used to re-scale a given data set, increasing the likelihood of a cluster validity index to be optimized at the expected number of clusters.\nMini-batch k-means: k-means variation using \"mini batch\" samples for data sets that do not fit into memory.\nOtsu's method", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0067", "text": "Hartigan\u2013Wong method", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0068", "text": "Hartigan and Wong's method provides a variation of k-means algorithm which progresses towards a local minimum of the minimum sum-of-squares problem with different solution updates. The method is a local search that iteratively attempts to relocate a sample into a different cluster as long as this process improves the objective function. When no sample can be relocated into a different cluster with an improvement of the objective, the method stops (in a local minimum)", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0069", "text": ". In a similar way as the classical k-means, the approach remains a heuristic since it does not necessarily guarantee that the final solution is globally optimum.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0070", "text": "Let \n  \n    \n      \n        \u03c6\n        (\n        \n          S\n          \n            j\n          \n        \n        )\n      \n    \n    {\\displaystyle \\varphi (S_{j})}\n  \n be the individual cost of \n  \n    \n      \n        \n          S\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle S_{j}}\n  \n defined by \n  \n    \n      \n        \n          \u2211\n          \n            x\n            \u2208\n            \n              S\n              \n                j", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0071", "text": "j\n              \n            \n          \n        \n        (\n        x\n        \u2212\n        \n          \u03bc\n          \n            j\n          \n        \n        \n          )\n          \n            2\n          \n        \n      \n    \n    {\\textstyle \\sum _{x\\in S_{j}}(x-\\mu _{j})^{2}}\n  \n, with \n  \n    \n      \n        \n          \u03bc\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\mu _{j}}\n  \n the center of the cluster.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0072", "text": "Assignment step\nHartigan and Wong's method starts by partitioning the points into random clusters \n  \n    \n      \n        {\n        \n          S\n          \n            j\n          \n        \n        \n          }\n          \n            j\n            \u2208\n            {\n            1\n            ,\n            \u22ef\n            k\n            }\n          \n        \n      \n    \n    {\\displaystyle \\{S_{j}\\}_{j\\in \\{1,\\cdots k\\}}}\n  \n.\nUpdate step\nNext it determines the \n  \n    \n      \n        n\n        ,", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0073", "text": "n\n        ,\n        m\n        \u2208\n        {\n        1\n        ,\n        \u2026\n        ,\n        k\n        }\n      \n    \n    {\\displaystyle n,m\\in \\{1,\\ldots ,k\\}}\n  \n and \n  \n    \n      \n        x\n        \u2208\n        \n          S\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle x\\in S_{n}}\n  \n for which the following function reaches a maximum \n  \n    \n      \n        \u0394\n        (\n        m\n        ,\n        n\n        ,\n        x\n        )\n        =", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0074", "text": "n\n        ,\n        x\n        )\n        =\n        \u03c6\n        (\n        \n          S\n          \n            n\n          \n        \n        )\n        +\n        \u03c6\n        (\n        \n          S\n          \n            m\n          \n        \n        )\n        \u2212\n        \u03c6\n        (\n        \n          S\n          \n            n\n          \n        \n        \u2216\n        {\n        x\n        }\n        )\n        \u2212\n        \u03c6\n        (\n        \n          S\n          \n            m", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0075", "text": "m\n          \n        \n        \u222a\n        {\n        x\n        }\n        )\n        .\n      \n    \n    {\\displaystyle \\Delta (m,n,x)=\\varphi (S_{n})+\\varphi (S_{m})-\\varphi (S_{n}\\setminus \\{x\\})-\\varphi (S_{m}\\cup \\{x\\}).}\n  \n For the \n  \n    \n      \n        x\n        ,\n        n\n        ,\n        m\n      \n    \n    {\\displaystyle x,n,m}\n  \n that reach this maximum, \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n moves from the cluster", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0076", "text": "moves from the cluster \n  \n    \n      \n        \n          S\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle S_{n}}\n  \n to the cluster \n  \n    \n      \n        \n          S\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle S_{m}}\n  \n.\nTermination\nThe algorithm terminates once \n  \n    \n      \n        \u0394\n        (\n        m\n        ,\n        n\n        ,\n        x\n        )\n      \n    \n    {\\displaystyle \\Delta (m,n,x)}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0077", "text": "{\\displaystyle \\Delta (m,n,x)}\n  \n is less than zero for all \n  \n    \n      \n        x\n        ,\n        n\n        ,\n        m\n      \n    \n    {\\displaystyle x,n,m}\n  \n.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0078", "text": "{\\displaystyle x,n,m}\n  \n.\nDifferent move acceptance strategies can be used. In a first-improvement strategy, any improving relocation can be applied, whereas in a best-improvement strategy, all possible relocations are iteratively tested and only the best is applied at each iteration. The former approach favors speed, whether the latter approach generally favors solution quality at the expense of additional computational time. The function \n  \n    \n      \n        \u0394", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0079", "text": "\u0394\n      \n    \n    {\\displaystyle \\Delta }\n  \n used to calculate the result of a relocation can also be efficiently evaluated by using equality", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0080", "text": "\u0394\n        (\n        x\n        ,\n        n\n        ,\n        m\n        )\n        =\n        \n          \n            \n              \u2223\n              \n                S\n                \n                  n\n                \n              \n              \u2223\n            \n            \n              \u2223\n              \n                S\n                \n                  n\n                \n              \n              \u2223\n              \u2212\n              1", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0081", "text": "1\n            \n          \n        \n        \u22c5\n        \u2016\n        \n          \u03bc\n          \n            n\n          \n        \n        \u2212\n        x\n        \n          \u2016\n          \n            2\n          \n        \n        \u2212\n        \n          \n            \n              \u2223\n              \n                S\n                \n                  m\n                \n              \n              \u2223\n            \n            \n              \u2223\n              \n                S", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0082", "text": "S\n                \n                  m\n                \n              \n              \u2223\n              +\n              1\n            \n          \n        \n        \u22c5\n        \u2016\n        \n          \u03bc\n          \n            m\n          \n        \n        \u2212\n        x\n        \n          \u2016\n          \n            2\n          \n        \n        .", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0083", "text": ".\n      \n    \n    {\\displaystyle \\Delta (x,n,m)={\\frac {\\mid S_{n}\\mid }{\\mid S_{n}\\mid -1}}\\cdot \\lVert \\mu _{n}-x\\rVert ^{2}-{\\frac {\\mid S_{m}\\mid }{\\mid S_{m}\\mid +1}}\\cdot \\lVert \\mu _{m}-x\\rVert ^{2}.}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0084", "text": "Global optimization and meta-heuristics\nThe classical k-means algorithm and its variations are known to only converge to local minima of the minimum-sum-of-squares clustering problem defined as", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0085", "text": "a\n              r\n              g\n              \n              m\n              i\n              n\n            \n          \n          \n            \n              S\n            \n          \n        \n        \u2061\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            k\n          \n        \n        \n          \u2211\n          \n            \n              x\n            \n            \u2208", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0086", "text": "\u2208\n            \n              S\n              \n                i\n              \n            \n          \n        \n        \n          \n            \u2016\n            \n              \n                x\n              \n              \u2212\n              \n                \n                  \u03bc\n                \n                \n                  i\n                \n              \n            \n            \u2016\n          \n          \n            2\n          \n        \n        .", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0087", "text": ".\n      \n    \n    {\\displaystyle \\mathop {\\operatorname {arg\\,min} } _{\\mathbf {S} }\\sum _{i=1}^{k}\\sum _{\\mathbf {x} \\in S_{i}}\\left\\|\\mathbf {x} -{\\boldsymbol {\\mu }}_{i}\\right\\|^{2}.}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0088", "text": "Many studies have attempted to improve the convergence behavior of the algorithm and maximize the chances of attaining the global optimum (or at least, local minima of better quality). Initialization and restart techniques discussed in the previous sections are one alternative to find better solutions", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0089", "text": ". More recently, global optimization algorithms based on branch-and-bound and semidefinite programming have produced \u2018\u2019provenly optimal\u2019\u2019 solutions for datasets with up to 4,177 entities and 20,531 features. As expected, due to the NP-hardness of the subjacent optimization problem, the computational time of optimal algorithms for k-means quickly increases beyond this size", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0090", "text": ". Optimal solutions for small- and medium-scale still remain valuable as a benchmark tool, to evaluate the quality of other heuristics. To find high-quality local minima within a controlled computational time but without optimality guarantees, other works have explored metaheuristics and other global optimization techniques, e.g., based on incremental approaches and convex optimization, random swaps (i.e., iterated local search), variable neighborhood search and genetic algorithms", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0091", "text": ". It is indeed known that finding better local minima of the minimum sum-of-squares clustering problem can make the difference between failure and success to recover cluster structures in feature spaces of high dimension.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0092", "text": "Discussion\nThree key features of k-means that make it efficient are often regarded as its biggest drawbacks:", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0093", "text": "Euclidean distance is used as a metric and variance is used as a measure of cluster scatter.\nThe number of clusters k is an input parameter: an inappropriate choice of k may yield poor results. That is why, when performing k-means, it is important to run diagnostic checks for determining the number of clusters in the data set.\nConvergence to a local minimum may produce counterintuitive (\"wrong\") results (see example in Fig.).", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0094", "text": "A key limitation of k-means is its cluster model. The concept is based on spherical clusters that are separable so that the mean converges towards the cluster center. The clusters are expected to be of similar size, so that the assignment to the nearest cluster center is the correct assignment. When for example applying k-means with a value of \n  \n    \n      \n        k\n        =\n        3\n      \n    \n    {\\displaystyle k=3}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0095", "text": "3\n      \n    \n    {\\displaystyle k=3}\n  \n onto the well-known Iris flower data set, the result often fails to separate the three Iris species contained in the data set. With \n  \n    \n      \n        k\n        =\n        2\n      \n    \n    {\\displaystyle k=2}\n  \n, the two visible clusters (one containing two species) will be discovered, whereas with \n  \n    \n      \n        k\n        =\n        3\n      \n    \n    {\\displaystyle k=3}", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0096", "text": "3\n      \n    \n    {\\displaystyle k=3}\n  \n one of the two clusters will be split into two even parts. In fact, \n  \n    \n      \n        k\n        =\n        2\n      \n    \n    {\\displaystyle k=2}\n  \n is more appropriate for this data set, despite the data set's containing 3 classes. As with any other clustering algorithm, the k-means result makes assumptions that the data satisfy certain criteria. It works well on some data sets, and fails on others.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0097", "text": "The result of k-means can be seen as the Voronoi cells of the cluster means. Since data is split halfway between cluster means, this can lead to suboptimal splits as can be seen in the \"mouse\" example. The Gaussian models used by the expectation\u2013maximization algorithm (arguably a generalization of k-means) are more flexible by having both variances and covariances", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0098", "text": ". The EM result is thus able to accommodate clusters of variable size much better than k-means as well as correlated clusters (not in this example). In counterpart, EM requires the optimization of a larger number of free parameters and poses some methodological issues due to vanishing clusters or badly-conditioned covariance matrices. k-means is closely related to nonparametric Bayesian modeling.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0099", "text": "Applications\nk-means clustering is rather easy to apply to even large data sets, particularly when using heuristics such as Lloyd's algorithm. It has been successfully used in market segmentation, computer vision, and astronomy among many other domains. It often is used as a preprocessing step for other algorithms, for example to find a starting configuration.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0100", "text": "Vector quantization", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0101", "text": "Vector quantization, a technique commonly used in signal processing and computer graphics, involves reducing the color palette of an image to a fixed number of colors, known as k. One popular method for achieving vector quantization is through k-means clustering. In this process, k-means is applied to the color space of an image to partition it into k clusters, with each cluster representing a distinct color in the image", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0102", "text": ". This technique is particularly useful in image segmentation tasks, where it helps identify and group similar colors together.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0103", "text": "Example: In the field of computer graphics, k-means clustering is often employed for color quantization in image compression. By reducing the number of colors used to represent an image, file sizes can be significantly reduced without significant loss of visual quality. For instance, consider an image with millions of colors", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0104", "text": ". By applying k-means clustering with k set to a smaller number, the image can be represented using a more limited color palette, resulting in a compressed version that consumes less storage space and bandwidth. Other uses of vector quantization include non-random sampling, as k-means can easily be used to choose k different but prototypical objects from a large data set for further analysis.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0105", "text": "Cluster analysis\nCluster analysis, a fundamental task in data mining and machine learning, involves grouping a set of data points into clusters based on their similarity. k-means clustering is a popular algorithm used for partitioning data into k clusters, where each cluster is represented by its centroid.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0106", "text": "However, the pure k-means algorithm is not very flexible, and as such is of limited use (except for when vector quantization as above is actually the desired use case). In particular, the parameter k is known to be hard to choose (as discussed above) when not given by external constraints. Another limitation is that it cannot be used with arbitrary distance functions or on non-numerical data. For these use cases, many other algorithms are superior.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0107", "text": "Example: In marketing, k-means clustering is frequently employed for market segmentation, where customers with similar characteristics or behaviors are grouped together. For instance, a retail company may use k-means clustering to segment its customer base into distinct groups based on factors such as purchasing behavior, demographics, and geographic location", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0108", "text": ". These customer segments can then be targeted with tailored marketing strategies and product offerings to maximize sales and customer satisfaction.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0109", "text": "Feature learning", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0110", "text": "k-means clustering has been used as a feature learning (or dictionary learning) step, in either (semi-)supervised learning or unsupervised learning. The basic approach is first to train a k-means clustering representation, using the input training data (which need not be labelled)", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0111", "text": ". Then, to project any input datum into the new feature space, an \"encoding\" function, such as the thresholded matrix-product of the datum with the centroid locations, computes the distance from the datum to each centroid, or simply an indicator function for the nearest centroid, or some smooth transformation of the distance. Alternatively, transforming the sample-cluster distance through a Gaussian RBF, obtains the hidden layer of a radial basis function network.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0112", "text": "This use of k-means has been successfully combined with simple, linear classifiers for semi-supervised learning in NLP (specifically for named-entity recognition) and in computer vision. On an object recognition task, it was found to exhibit comparable performance with more sophisticated feature learning approaches such as autoencoders and restricted Boltzmann machines", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0113", "text": ". However, it generally requires more data, for equivalent performance, because each data point only contributes to one \"feature\".", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0114", "text": "Example: In natural language processing (NLP), k-means clustering has been integrated with simple linear classifiers for semi-supervised learning tasks such as named-entity recognition (NER). By first clustering unlabeled text data using k-means, meaningful features can be extracted to improve the performance of NER models", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0115", "text": ". For instance, k-means clustering can be applied to identify clusters of words or phrases that frequently co-occur in the input text, which can then be used as features for training the NER model. This approach has been shown to achieve comparable performance with more complex feature learning techniques such as autoencoders and restricted Boltzmann machines, albeit with a greater requirement for labeled data.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0116", "text": "Recent Developments", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0117", "text": "Recent advancements in the application of k-means clustering include improvements in initialization techniques, such as the use of k-means++ initialization to select initial cluster centroids in a more effective manner", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0118", "text": ". Additionally, researchers have explored the integration of k-means clustering with deep learning methods, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to enhance the performance of various tasks in computer vision, natural language processing, and other domains.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0119", "text": "Relation to other algorithms\nGaussian mixture model", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0120", "text": "The slow \"standard algorithm\" for k-means clustering, and its associated expectation\u2013maximization algorithm, is a special case of a Gaussian mixture model, specifically, the limiting case when fixing all covariances to be diagonal, equal and have infinitesimal small variance. Instead of small variances, a hard cluster assignment can also be used to show another equivalence of k-means clustering to a special case of \"hard\" Gaussian mixture modelling", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0121", "text": ". This does not mean that it is efficient to use Gaussian mixture modelling to compute k-means, but just that there is a theoretical relationship, and that Gaussian mixture modelling can be interpreted as a generalization of k-means; on the contrary, it has been suggested to use k-means clustering to find starting points for Gaussian mixture modelling on difficult data.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0122", "text": "k-SVD\nAnother generalization of the k-means algorithm is the k-SVD algorithm, which estimates data points as a sparse linear combination of \"codebook vectors\". k-means corresponds to the special case of using a single codebook vector, with a weight of 1.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0123", "text": "Principal component analysis", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0124", "text": "The relaxed solution of k-means clustering, specified by the cluster indicators, is given by principal component analysis (PCA).  The intuition is that k-means describe spherically shaped (ball-like) clusters. If the data has 2 clusters, the line connecting the two centroids is the best 1-dimensional projection direction, which is also the first PCA direction. Cutting the line at the center of mass separates the clusters (this is the continuous relaxation of the discrete cluster indicator)", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0125", "text": ". If the data have three clusters, the 2-dimensional plane spanned by three cluster centroids is the best 2-D projection. This plane is also defined by the first two PCA dimensions. Well-separated clusters are effectively modelled by ball-shaped clusters and thus discovered by k-means. Non-ball-shaped clusters are hard to separate when they are close. For example, two half-moon shaped clusters intertwined in space do not separate well when projected onto PCA subspace", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0126", "text": ". k-means should not be expected to do well on this data. It is straightforward to produce counterexamples to the statement that the cluster centroid subspace is spanned by the principal directions.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0127", "text": "Mean shift clustering", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0128", "text": "Basic mean shift clustering algorithms maintain a set of data points the same size as the input data set. Initially, this set is copied from the input set. All points are then iteratively moved towards the mean of the points surrounding them. By contrast, k-means restricts the set of clusters to k clusters, usually much less than the number of points in the input data set, using the mean of all points in the prior cluster that are closer to that point than any other for the centroid (e.g", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0129", "text": ". within the Voronoi partition of each updating point). A mean shift algorithm that is similar then to k-means, called likelihood mean shift, replaces the set of points undergoing replacement by the mean of all points in the input set that are within a given distance of the changing set. An advantage of mean shift clustering over k-means is the detection of an arbitrary number of clusters in the data set, as there is not a parameter determining the number of clusters", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0130", "text": ". Mean shift can be much slower than k-means, and still requires selection of a bandwidth parameter.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0131", "text": "Independent component analysis\nUnder sparsity assumptions and when input data is pre-processed with the whitening transformation, k-means produces the solution to the linear independent component analysis (ICA) task. This aids in explaining the successful application of k-means to feature learning.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0132", "text": "Bilateral filtering", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0133", "text": "k-means implicitly assumes that the ordering of the input data set does not matter. The bilateral filter is similar to k-means and mean shift in that it maintains a set of data points that are iteratively replaced by means. However, the bilateral filter restricts the calculation of the (kernel weighted) mean to include only points that are close in the ordering of the input data", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0134", "text": ". This makes it applicable to problems such as image denoising, where the spatial arrangement of pixels in an image is of critical importance.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0135", "text": "Similar problems\nThe set of squared error minimizing cluster functions also includes the k-medoids algorithm, an approach which forces the center point of each cluster to be one of the actual points, i.e., it uses medoids in place of centroids.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0136", "text": "Software implementations\nDifferent implementations of the algorithm exhibit performance differences, with the fastest on a test data set finishing in 10 seconds, the slowest taking 25,988 seconds (~7 hours). The differences can be attributed to implementation quality, language and compiler differences, different termination criteria and precision levels, and the use of indexes for acceleration.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0137", "text": "Free Software/Open Source\nThe following implementations are available under Free/Open Source Software licenses, with publicly available source code.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0138", "text": "Accord.NET contains C# implementations for k-means, k-means++ and k-modes.\nALGLIB contains parallelized C++ and C# implementations for k-means and k-means++.\nAOSP contains a Java implementation for k-means.\nCrimeStat implements two spatial k-means algorithms, one of which allows the user to define the starting locations.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0139", "text": "ELKI contains k-means (with Lloyd and MacQueen iteration, along with different initializations such as k-means++ initialization) and various more advanced clustering algorithms.\nSmile contains k-means and various more other algorithms and results visualization (for java, kotlin and scala).\nJulia contains a k-means implementation in the JuliaStats Clustering package.\nKNIME contains nodes for k-means and k-medoids.\nMahout contains a MapReduce based k-means.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0140", "text": "Mahout contains a MapReduce based k-means.\nmlpack contains a C++ implementation of k-means.\nOctave contains k-means.\nOpenCV contains a k-means implementation.\nOrange includes a component for k-means clustering with automatic selection of k and cluster silhouette scoring.\nPSPP contains k-means, The QUICK CLUSTER command performs k-means clustering on the dataset.\nR contains three k-means variations.\nSciPy and scikit-learn contain multiple k-means implementations.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0141", "text": "Spark MLlib implements a distributed k-means algorithm.\nTorch contains an unsup package that provides k-means clustering.\nWeka contains k-means and x-means.", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Kmeans_clustering_0142", "text": "Proprietary\nThe following implementations are available under proprietary license terms, and may not have publicly available source code.\n\nSee also\nK-medoids\nBFR algorithm\nCentroidal Voronoi tessellation\nCluster analysis\nDBSCAN\nHead/tail breaks\nk q-flats\nk-means++\nLinde\u2013Buzo\u2013Gray algorithm\nSelf-organizing map\n\n\n== References ==", "title": "K-means clustering", "url": "https://en.wikipedia.org/wiki/K-means_clustering"}, {"id": "rec_Hierarchical_clustering_0000", "text": "In data mining and statistics, hierarchical clustering (also called hierarchical cluster analysis or HCA) is a method of cluster analysis that seeks to build a hierarchy of clusters. Strategies for hierarchical clustering generally fall into two categories:", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0001", "text": "Agglomerative: Agglomerative clustering, often referred to as a \"bottom-up\" approach, begins with each data point as an individual cluster. At each step, the algorithm merges the two most similar clusters based on a chosen distance metric (e.g., Euclidean distance) and linkage criterion (e.g., single-linkage, complete-linkage). This process continues until all data points are combined into a single cluster or a stopping criterion is met", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0002", "text": ". Agglomerative methods are more commonly used due to their simplicity and computational efficiency for small to medium-sized datasets.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0003", "text": "Divisive: Divisive clustering, known as a \"top-down\" approach, starts with all data points in a single cluster and recursively splits the cluster into smaller ones. At each step, the algorithm selects a cluster and divides it into two or more subsets, often using a criterion such as maximizing the distance between resulting clusters. Divisive methods are less common but can be useful when the goal is to identify large, distinct clusters first.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0004", "text": "In general, the merges and splits are determined in a greedy manner. The results of hierarchical clustering are usually presented in a dendrogram. \nHierarchical clustering has the distinct advantage that any valid measure of distance can be used. In fact, the observations themselves are not required: all that is used is a matrix of distances. On the other hand, except for the special case of single-linkage distance, none of the algorithms (except exhaustive search in", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0005", "text": "O\n          \n        \n        (\n        \n          2\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\mathcal {O}}(2^{n})}\n  \n) can be guaranteed to find the optimum solution.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0006", "text": "Complexity\nThe standard algorithm for hierarchical agglomerative clustering (HAC) has a time complexity of \n  \n    \n      \n        \n          \n            O\n          \n        \n        (\n        \n          n\n          \n            3\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\mathcal {O}}(n^{3})}\n  \n and requires \n  \n    \n      \n        \u03a9\n        (\n        \n          n\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle \\Omega (n^{2})}", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0007", "text": "{\\displaystyle \\Omega (n^{2})}\n  \n memory, which makes it too slow for even medium data sets. However, for some special cases, optimal efficient agglomerative methods (of complexity \n  \n    \n      \n        \n          \n            O\n          \n        \n        (\n        \n          n\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\mathcal {O}}(n^{2})}", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0008", "text": "{\\displaystyle {\\mathcal {O}}(n^{2})}\n  \n) are known: SLINK for single-linkage and CLINK for complete-linkage clustering. With a heap, the runtime of the general case can be reduced to \n  \n    \n      \n        \n          \n            O\n          \n        \n        (\n        \n          n\n          \n            2\n          \n        \n        log\n        \u2061\n        n\n        )\n      \n    \n    {\\displaystyle {\\mathcal {O}}(n^{2}\\log n)}\n  \n instead of", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0009", "text": "instead of \n  \n    \n      \n        \n          \n            O\n          \n        \n        (\n        \n          n\n          \n            3\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\mathcal {O}}(n^{3})}\n  \n, at the cost of additional the memory requirements. In many cases, the memory overheads of this approach are too large to make it practically usable. Methods exist which use quadtrees that demonstrate \n  \n    \n      \n        \n          \n            O", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0010", "text": "O\n          \n        \n        (\n        \n          n\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\mathcal {O}}(n^{2})}\n  \n total running time with \n  \n    \n      \n        \n          \n            O\n          \n        \n        (\n        n\n        )\n      \n    \n    {\\displaystyle {\\mathcal {O}}(n)}\n  \n space.\nDivisive clustering with an exhaustive search is \n  \n    \n      \n        \n          \n            O", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0011", "text": "O\n          \n        \n        (\n        \n          2\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\mathcal {O}}(2^{n})}\n  \n, but it is common to use faster heuristics to choose splits, such as k-means.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0012", "text": "Cluster linkage", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0013", "text": "In order to decide which clusters should be combined (for agglomerative), or where a cluster should be split (for divisive), a measure of dissimilarity between sets of observations is required", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0014", "text": ". In most methods of hierarchical clustering, this is achieved by use of an appropriate distance d, such as the Euclidean distance, between single observations of the data set, and a linkage criterion, which specifies the dissimilarity of sets as a function of the pairwise distances of observations in the sets", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0015", "text": ". The choice of metric as well as linkage can have a major impact on the result of the clustering, where the lower level metric determines which objects are most similar, whereas the linkage criterion influences the shape of the clusters. For example, complete-linkage tends to produce more spherical clusters than single-linkage.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0016", "text": "The linkage criterion determines the distance between sets of observations as a function of the pairwise distances between observations.\nSome commonly used linkage criteria between two sets of observations A and B and a distance d are:", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0017", "text": "Some of these can only be recomputed recursively (WPGMA, WPGMC), for many a recursive computation with Lance-Williams-equations is more efficient, while for other (Hausdorff, Medoid) the distances have to be computed with the slower full formula. Other linkage criteria include:", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0018", "text": "The probability that candidate clusters spawn from the same distribution function (V-linkage).\nThe product of in-degree and out-degree on a k-nearest-neighbour graph (graph degree linkage).\nThe increment of some cluster descriptor (i.e., a quantity defined for measuring the quality of a cluster) after merging two clusters.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0019", "text": "Agglomerative clustering example\nFor example, suppose this data is to be clustered, and the Euclidean distance is the distance metric.\nThe hierarchical clustering dendrogram would be:", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0020", "text": "Cutting the tree at a given height will give a partitioning clustering at a selected precision. In this example, cutting after the second row (from the top) of the dendrogram will yield clusters {a} {b c} {d e} {f}. Cutting after the third row will yield clusters {a} {b c} {d e f}, which is a coarser clustering, with a smaller number but larger clusters.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0021", "text": "This method builds the hierarchy from the individual elements by progressively merging clusters. In our example, we have six elements {a} {b} {c} {d} {e} and {f}. The first step is to determine which elements to merge in a cluster. Usually, we want to take the two closest elements, according to the chosen distance.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0022", "text": "Optionally, one can also construct a distance matrix at this stage, where the number in the i-th row j-th column is the distance between the i-th and j-th elements. Then, as clustering progresses, rows and columns are merged as the clusters are merged and the distances updated. This is a common way to implement this type of clustering, and has the benefit of caching distances between clusters", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0023", "text": ". A simple agglomerative clustering algorithm is described in the single-linkage clustering page; it can easily be adapted to different types of linkage (see below).", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0024", "text": "Suppose we have merged the two closest elements b and c, we now have the following clusters {a}, {b, c}, {d}, {e} and {f}, and want to merge them further. To do that, we need to take the distance between {a} and {b c}, and therefore define the distance between two clusters.\nUsually the distance between two clusters \n  \n    \n      \n        \n          \n            A\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {A}}}\n  \n and \n  \n    \n      \n        \n          \n            B", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0025", "text": "B\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {B}}}\n  \n is one of the following:", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0026", "text": "The maximum distance between elements of each cluster (also called complete-linkage clustering):", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0027", "text": "max\n        {\n        \n        d\n        (\n        x\n        ,\n        y\n        )\n        :\n        x\n        \u2208\n        \n          \n            A\n          \n        \n        ,\n        \n        y\n        \u2208\n        \n          \n            B\n          \n        \n        \n        }\n        .\n      \n    \n    {\\displaystyle \\max\\{\\,d(x,y):x\\in {\\mathcal {A}},\\,y\\in {\\mathcal {B}}\\,\\}.}", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0028", "text": "The minimum distance between elements of each cluster (also called single-linkage clustering):", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0029", "text": "min\n        {\n        \n        d\n        (\n        x\n        ,\n        y\n        )\n        :\n        x\n        \u2208\n        \n          \n            A\n          \n        \n        ,\n        \n        y\n        \u2208\n        \n          \n            B\n          \n        \n        \n        }\n        .\n      \n    \n    {\\displaystyle \\min\\{\\,d(x,y):x\\in {\\mathcal {A}},\\,y\\in {\\mathcal {B}}\\,\\}.}", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0030", "text": "The mean distance between elements of each cluster (also called average linkage clustering, used e.g. in UPGMA):", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0031", "text": "1\n            \n              \n                |\n              \n              \n                \n                  A\n                \n              \n              \n                |\n              \n              \u22c5\n              \n                |\n              \n              \n                \n                  B\n                \n              \n              \n                |\n              \n            \n          \n        \n        \n          \u2211", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0032", "text": "\u2211\n          \n            x\n            \u2208\n            \n              \n                A\n              \n            \n          \n        \n        \n          \u2211\n          \n            y\n            \u2208\n            \n              \n                B\n              \n            \n          \n        \n        d\n        (\n        x\n        ,\n        y\n        )\n        .", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0033", "text": "y\n        )\n        .\n      \n    \n    {\\displaystyle {1 \\over {|{\\mathcal {A}}|\\cdot |{\\mathcal {B}}|}}\\sum _{x\\in {\\mathcal {A}}}\\sum _{y\\in {\\mathcal {B}}}d(x,y).}", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0034", "text": "The sum of all intra-cluster variance.\nThe increase in variance for the cluster being merged (Ward's method)\nThe probability that candidate clusters spawn from the same distribution function (V-linkage).\nIn case of tied minimum distances, a pair is randomly chosen, thus being able to generate several structurally different dendrograms. Alternatively, all tied pairs may be joined at the same time, generating a unique dendrogram.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0035", "text": "One can always decide to stop clustering when there is a sufficiently small number of clusters (number criterion). Some linkages may also guarantee that agglomeration occurs at a greater distance between clusters than the previous agglomeration, and then one can stop clustering when the clusters are too far apart to be merged (distance criterion). However, this is not the case of, e.g., the centroid linkage where the so-called reversals (inversions, departures from ultrametricity) may occur.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0036", "text": "Divisive clustering\nThe basic principle of divisive clustering was published as the DIANA (DIvisive ANAlysis clustering) algorithm. Initially, all data is in the same cluster, and the largest cluster is split until every object is separate.\nBecause there exist \n  \n    \n      \n        O\n        (\n        \n          2\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle O(2^{n})}", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0037", "text": "{\\displaystyle O(2^{n})}\n  \n ways of splitting each cluster, heuristics are needed. DIANA chooses the object with the maximum average dissimilarity and then moves all objects to this cluster that are more similar to the new cluster than to the remainder.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0038", "text": "Informally, DIANA is not so much a process of \"dividing\" as it is of \"hollowing out\": each iteration, an existing cluster (e.g. the initial cluster of the entire dataset) is chosen to form a new cluster inside of it. Objects progressively move to this nested cluster, and hollow out the existing cluster. Eventually, all that's left inside a cluster is nested clusters that grew there, without it owning any loose objects by itself.\nFormally, DIANA operates in the following steps:", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0039", "text": "Let \n  \n    \n      \n        \n          C\n          \n            0\n          \n        \n        =\n        {\n        1\n        \u2026\n        n\n        }\n      \n    \n    {\\displaystyle C_{0}=\\{1\\dots n\\}}\n  \n be the set of all \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n object indices and \n  \n    \n      \n        \n          \n            C\n          \n        \n        =\n        {\n        \n          C\n          \n            0\n          \n        \n        }", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0040", "text": "}\n      \n    \n    {\\displaystyle {\\mathcal {C}}=\\{C_{0}\\}}\n  \n the set of all formed clusters so far.\nIterate the following until \n  \n    \n      \n        \n          |\n        \n        \n          \n            C\n          \n        \n        \n          |\n        \n        =\n        n\n      \n    \n    {\\displaystyle |{\\mathcal {C}}|=n}\n  \n:\nFind the current cluster with 2 or more objects that has the largest diameter: \n  \n    \n      \n        \n          C", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0041", "text": "C\n          \n            \u2217\n          \n        \n        =\n        arg\n        \u2061\n        \n          max\n          \n            C\n            \u2208\n            \n              \n                C\n              \n            \n          \n        \n        \n          max\n          \n            \n              i\n              \n                1\n              \n            \n            ,\n            \n              i\n              \n                2", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0042", "text": "2\n              \n            \n            \u2208\n            C\n          \n        \n        \u03b4\n        (\n        \n          i\n          \n            1\n          \n        \n        ,\n        \n          i\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle C_{*}=\\arg \\max _{C\\in {\\mathcal {C}}}\\max _{i_{1},i_{2}\\in C}\\delta (i_{1},i_{2})}", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0043", "text": "Find the object in this cluster with the most dissimilarity to the rest of the cluster: \n  \n    \n      \n        \n          i\n          \n            \u2217\n          \n        \n        =\n        arg\n        \u2061\n        \n          max\n          \n            i\n            \u2208\n            \n              C\n              \n                \u2217\n              \n            \n          \n        \n        \n          \n            1\n            \n              \n                |", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0044", "text": "|\n              \n              \n                C\n                \n                  \u2217\n                \n              \n              \n                |\n              \n              \u2212\n              1\n            \n          \n        \n        \n          \u2211\n          \n            j\n            \u2208\n            \n              C\n              \n                \u2217\n              \n            \n            \u2216\n            {\n            i\n            }\n          \n        \n        \u03b4\n        (", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0045", "text": "\u03b4\n        (\n        i\n        ,\n        j\n        )\n      \n    \n    {\\displaystyle i^{*}=\\arg \\max _{i\\in C_{*}}{\\frac {1}{|C_{*}|-1}}\\sum _{j\\in C_{*}\\setminus \\{i\\}}\\delta (i,j)}", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0046", "text": "Pop \n  \n    \n      \n        \n          i\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle i^{*}}\n  \n from its old cluster \n  \n    \n      \n        \n          C\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle C_{*}}\n  \n and put it into a new splinter group \n  \n    \n      \n        \n          C\n          \n            \n              new\n            \n          \n        \n        =\n        {\n        \n          i\n          \n            \u2217", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0047", "text": "i\n          \n            \u2217\n          \n        \n        }\n      \n    \n    {\\displaystyle C_{\\textrm {new}}=\\{i^{*}\\}}\n  \n.\nAs long as \n  \n    \n      \n        \n          C\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle C_{*}}\n  \n isn't empty, keep migrating objects from \n  \n    \n      \n        \n          C\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle C_{*}}\n  \n to add them to \n  \n    \n      \n        \n          C", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0048", "text": "C\n          \n            \n              new\n            \n          \n        \n      \n    \n    {\\displaystyle C_{\\textrm {new}}}\n  \n. To choose which objects to migrate, don't just consider dissimilarity to \n  \n    \n      \n        \n          C\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle C_{*}}\n  \n, but also adjust for dissimilarity to the splinter group: let \n  \n    \n      \n        \n          i\n          \n            \u2217", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0049", "text": "i\n          \n            \u2217\n          \n        \n        =\n        arg\n        \u2061\n        \n          max\n          \n            i\n            \u2208\n            C\n          \n        \n        D\n        (\n        i\n        )\n      \n    \n    {\\displaystyle i^{*}=\\arg \\max _{i\\in C}D(i)}\n  \n where we define \n  \n    \n      \n        D\n        (\n        i\n        )\n        =\n        \n          \n            1\n            \n              \n                |", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0050", "text": "|\n              \n              \n                C\n                \n                  \u2217\n                \n              \n              \n                |\n              \n              \u2212\n              1\n            \n          \n        \n        \n          \u2211\n          \n            j\n            \u2208\n            \n              C\n              \n                \u2217\n              \n            \n            \u2216\n            {\n            i\n            }\n          \n        \n        \u03b4\n        (", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0051", "text": "\u03b4\n        (\n        i\n        ,\n        j\n        )\n        \u2212\n        \n          \n            1\n            \n              \n                |\n              \n              \n                C\n                \n                  \n                    new\n                  \n                \n              \n              \n                |\n              \n            \n          \n        \n        \n          \u2211\n          \n            j\n            \u2208\n            \n              C", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0052", "text": "\u2208\n            \n              C\n              \n                \n                  new\n                \n              \n            \n          \n        \n        \u03b4\n        (\n        i\n        ,\n        j\n        )\n      \n    \n    {\\displaystyle D(i)={\\frac {1}{|C_{*}|-1}}\\sum _{j\\in C_{*}\\setminus \\{i\\}}\\delta (i,j)-{\\frac {1}{|C_{\\textrm {new}}|}}\\sum _{j\\in C_{\\textrm {new}}}\\delta (i,j)}\n  \n, then either stop iterating when \n  \n    \n      \n        D\n        (\n        \n          i", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0053", "text": "D\n        (\n        \n          i\n          \n            \u2217\n          \n        \n        )\n        <\n        0\n      \n    \n    {\\displaystyle D(i^{*})<0}\n  \n, or migrate \n  \n    \n      \n        \n          i\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle i^{*}}\n  \n.\nAdd \n  \n    \n      \n        \n          C\n          \n            \n              new\n            \n          \n        \n      \n    \n    {\\displaystyle C_{\\textrm {new}}}\n  \n to", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0054", "text": "to \n  \n    \n      \n        \n          \n            C\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {C}}}\n  \n.\nIntuitively, \n  \n    \n      \n        D\n        (\n        i\n        )\n      \n    \n    {\\displaystyle D(i)}\n  \n above measures how strongly an object wants to leave its current cluster, but it is attenuated when the object wouldn't fit in the splinter group either. Such objects will likely start their own splinter group eventually.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0055", "text": "The dendrogram of DIANA can be constructed by letting the splinter group \n  \n    \n      \n        \n          C\n          \n            \n              new\n            \n          \n        \n      \n    \n    {\\displaystyle C_{\\textrm {new}}}\n  \n be a child of the hollowed-out cluster \n  \n    \n      \n        \n          C\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle C_{*}}\n  \n each time. This constructs a tree with \n  \n    \n      \n        \n          C", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0056", "text": "C\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle C_{0}}\n  \n as its root and \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n unique single-object clusters as its leaves.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0057", "text": "Software\nOpen source implementations\nALGLIB implements several hierarchical clustering algorithms (single-link, complete-link, Ward) in C++ and C# with O(n\u00b2) memory and O(n\u00b3) run time.\nELKI includes multiple hierarchical clustering algorithms, various linkage strategies and also includes the efficient SLINK, CLINK and Anderberg algorithms, flexible cluster extraction from dendrograms and various other cluster analysis algorithms.\nJulia has an implementation inside the Clustering.jl package.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0058", "text": "Octave, the GNU analog to MATLAB implements hierarchical clustering in function \"linkage\".\nOrange, a data mining software suite, includes hierarchical clustering with interactive dendrogram visualisation.\nR has built-in functions and packages that provide functions for hierarchical clustering.\nSciPy implements hierarchical clustering in Python, including the efficient SLINK algorithm.\nscikit-learn also implements hierarchical clustering in Python.\nWeka includes hierarchical cluster analysis.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0059", "text": "Commercial implementations\nMATLAB includes hierarchical cluster analysis.\nSAS includes hierarchical cluster analysis in PROC CLUSTER.\nMathematica includes a Hierarchical Clustering Package.\nNCSS includes hierarchical cluster analysis.\nSPSS includes hierarchical cluster analysis.\nQlucore Omics Explorer includes hierarchical cluster analysis.\nStata includes hierarchical cluster analysis.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0060", "text": "Stata includes hierarchical cluster analysis.\nCrimeStat includes a nearest neighbor hierarchical cluster algorithm with a graphical output for a Geographic Information System.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Hierarchical_clustering_0061", "text": "See also\nReferences\nFurther reading\nKaufman, L.; Rousseeuw, P.J. (1990). Finding Groups in Data: An Introduction to Cluster Analysis (1 ed.). New York: John Wiley. ISBN 0-471-87876-6.\nHastie, Trevor; Tibshirani, Robert; Friedman, Jerome (2009). \"14.3.12 Hierarchical clustering\". The Elements of Statistical Learning (2nd ed.). New York: Springer. pp. 520\u20138. ISBN 978-0-387-84857-0. Archived from the original (PDF) on 2009-11-10. Retrieved 2009-10-20.", "title": "Hierarchical clustering", "url": "https://en.wikipedia.org/wiki/Hierarchical_clustering"}, {"id": "rec_Principal_component_analysis_0000", "text": "Principal component analysis (PCA) is a linear dimensionality reduction technique with applications in exploratory data analysis, visualization and data preprocessing.\nThe data are linearly transformed onto a new coordinate system such that the directions (principal components) capturing the largest variation in the data can be easily identified.\nThe principal components of a collection of points in a real coordinate space are a sequence of \n  \n    \n      \n        p", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0001", "text": "p\n      \n    \n    {\\displaystyle p}\n  \n unit vectors, where the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n-th vector is the direction of a line that best fits the data while being orthogonal to the first \n  \n    \n      \n        i\n        \u2212\n        1\n      \n    \n    {\\displaystyle i-1}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0002", "text": "{\\displaystyle i-1}\n  \n vectors. Here, a best-fitting line is defined as one that minimizes the average squared perpendicular distance from the points to the line. These directions (i.e., principal components) constitute an orthonormal basis in which different individual dimensions of the data are linearly uncorrelated. Many studies use the first two principal components in order to plot the data in two dimensions and to visually identify clusters of closely related data points.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0003", "text": "Principal component analysis has applications in many fields such as population genetics, microbiome studies, and atmospheric science.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0004", "text": "Overview\nWhen performing PCA, the first principal component of a set of \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n variables is the derived variable formed as a linear combination of the original variables that explains the most variance. The second principal component explains the most variance in what is left once the effect of the first component is removed, and we may proceed through \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0005", "text": "p\n      \n    \n    {\\displaystyle p}\n  \n iterations until all the variance is explained. PCA is most commonly used when many of the variables are highly correlated with each other and it is desirable to reduce their number to an independent set.\nThe first principal component can equivalently be defined as a direction that maximizes the variance of the projected data. The \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0006", "text": "i\n      \n    \n    {\\displaystyle i}\n  \n-th principal component can be taken as a direction orthogonal to the first \n  \n    \n      \n        i\n        \u2212\n        1\n      \n    \n    {\\displaystyle i-1}\n  \n principal components that maximizes the variance of the projected data.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0007", "text": "For either objective, it can be shown that the principal components are eigenvectors of the data's covariance matrix. Thus, the principal components are often computed by eigendecomposition of the data covariance matrix or singular value decomposition of the data matrix. PCA is the simplest of the true eigenvector-based multivariate analyses and is closely related to factor analysis", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0008", "text": ". Factor analysis typically incorporates more domain-specific assumptions about the underlying structure and solves eigenvectors of a slightly different matrix. PCA is also related to canonical correlation analysis (CCA). CCA defines coordinate systems that optimally describe the cross-covariance between two datasets while PCA defines a new orthogonal coordinate system that optimally describes variance in a single dataset", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0009", "text": ". Robust and L1-norm-based variants of standard PCA have also been proposed.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0010", "text": "History", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0011", "text": "PCA was invented in 1901 by Karl Pearson, as an analogue of the principal axis theorem in mechanics; it was later independently developed and named by Harold Hotelling in the 1930s", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0012", "text": ". Depending on the field of application, it is also named the discrete Karhunen\u2013Lo\u00e8ve transform (KLT) in signal processing, the Hotelling transform in multivariate quality control, proper orthogonal decomposition (POD) in mechanical engineering, singular value decomposition (SVD) of X (invented in the last quarter of the 19th century), eigenvalue decomposition (EVD) of XTX in linear algebra, factor analysis (for a discussion of the differences between PCA and factor analysis see Ch", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0013", "text": ". 7 of Jolliffe's Principal Component Analysis), Eckart\u2013Young theorem (Harman, 1960), or empirical orthogonal functions (EOF) in meteorological science (Lorenz, 1956), empirical eigenfunction decomposition (Sirovich, 1987), quasiharmonic modes (Brooks et al., 1988), spectral decomposition in noise and vibration, and empirical modal analysis in structural dynamics.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0014", "text": "Intuition\nPCA can be thought of as fitting a p-dimensional ellipsoid to the data, where each axis of the ellipsoid represents a principal component. If some axis of the ellipsoid is small, then the variance along that axis is also small.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0015", "text": "To find the axes of the ellipsoid, we must first center the values of each variable in the dataset on 0 by subtracting the mean of the variable's observed values from each of those values. These transformed values are used instead of the original observed values for each of the variables. Then, we compute the covariance matrix of the data and calculate the eigenvalues and corresponding eigenvectors of this covariance matrix", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0016", "text": ". Then we must normalize each of the orthogonal eigenvectors to turn them into unit vectors. Once this is done, each of the mutually-orthogonal unit eigenvectors can be interpreted as an axis of the ellipsoid fitted to the data. This choice of basis will transform the covariance matrix into a diagonalized form, in which the diagonal elements represent the variance of each axis", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0017", "text": ". The proportion of the variance that each eigenvector represents can be calculated by dividing the eigenvalue corresponding to that eigenvector by the sum of all eigenvalues.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0018", "text": "Biplots and scree plots (degree of explained variance) are used to interpret findings of the PCA.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0019", "text": "Details\nPCA is defined as an orthogonal linear transformation on a real inner product space that transforms the data to a new coordinate system such that the greatest variance by some scalar projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on.\nConsider an \n  \n    \n      \n        n\n        \u00d7\n        p\n      \n    \n    {\\displaystyle n\\times p}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0020", "text": "{\\displaystyle n\\times p}\n  \n data matrix, X, with column-wise zero empirical mean (the sample mean of each column has been shifted to zero), where each of the n rows represents a different repetition of the experiment, and each of the p columns gives a particular kind of feature (say, the results from a particular sensor).\nMathematically, the transformation is defined by a set of size \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n (where", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0021", "text": "{\\displaystyle l}\n  \n (where \n  \n    \n      \n        l\n      \n    \n    {\\displaystyle l}\n  \n is usually selected to be strictly less than \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n to reduce dimensionality) of \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n-dimensional vectors of weights or coefficients \n  \n    \n      \n        \n          \n            w\n          \n          \n            (\n            k\n            )\n          \n        \n        =\n        (", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0022", "text": "=\n        (\n        \n          w\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          w\n          \n            p\n          \n        \n        \n          )\n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {w} _{(k)}=(w_{1},\\dots ,w_{p})_{(k)}}\n  \n that map each row vector \n  \n    \n      \n        \n          \n            x\n          \n          \n            (", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0023", "text": "x\n          \n          \n            (\n            i\n            )\n          \n        \n        =\n        (\n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            p\n          \n        \n        \n          )\n          \n            (\n            i\n            )\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{(i)}=(x_{1},\\dots ,x_{p})_{(i)}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0024", "text": "of X to a new vector of principal component scores \n  \n    \n      \n        \n          \n            t\n          \n          \n            (\n            i\n            )\n          \n        \n        =\n        (\n        \n          t\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          t\n          \n            l\n          \n        \n        \n          )\n          \n            (\n            i\n            )", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0025", "text": ")\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {t} _{(i)}=(t_{1},\\dots ,t_{l})_{(i)}}\n  \n, given by", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0026", "text": "t\n              \n                k\n              \n            \n          \n          \n            (\n            i\n            )\n          \n        \n        =\n        \n          \n            x\n          \n          \n            (\n            i\n            )\n          \n        \n        \u22c5\n        \n          \n            w\n          \n          \n            (\n            k\n            )\n          \n        \n        \n        \n          f", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0027", "text": "f\n          o\n          r\n        \n        \n        i\n        =\n        1\n        ,\n        \u2026\n        ,\n        n\n        \n        k\n        =\n        1\n        ,\n        \u2026\n        ,\n        l\n      \n    \n    {\\displaystyle {t_{k}}_{(i)}=\\mathbf {x} _{(i)}\\cdot \\mathbf {w} _{(k)}\\qquad \\mathrm {for} \\qquad i=1,\\dots ,n\\qquad k=1,\\dots ,l}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0028", "text": "in such a way that the individual variables \n  \n    \n      \n        \n          t\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          t\n          \n            l\n          \n        \n      \n    \n    {\\displaystyle t_{1},\\dots ,t_{l}}\n  \n of t considered over the data set successively inherit the maximum possible variance from X, with each coefficient vector w constrained to be a unit vector.\nThe above may equivalently be written in matrix form as", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0029", "text": "T\n        \n        =\n        \n          X\n        \n        \n          W\n        \n      \n    \n    {\\displaystyle \\mathbf {T} =\\mathbf {X} \\mathbf {W} }\n  \n\nwhere", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0030", "text": "where\n\n  \n    \n      \n        \n          \n            \n              T\n            \n          \n          \n            i\n            k\n          \n        \n        =\n        \n          \n            \n              t\n              \n                k\n              \n            \n          \n          \n            (\n            i\n            )\n          \n        \n      \n    \n    {\\displaystyle {\\mathbf {T} }_{ik}={t_{k}}_{(i)}}\n  \n,", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0031", "text": "X\n            \n          \n          \n            i\n            j\n          \n        \n        =\n        \n          \n            \n              x\n              \n                j\n              \n            \n          \n          \n            (\n            i\n            )\n          \n        \n      \n    \n    {\\displaystyle {\\mathbf {X} }_{ij}={x_{j}}_{(i)}}\n  \n, and", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0032", "text": "W\n            \n          \n          \n            j\n            k\n          \n        \n        =\n        \n          \n            \n              w\n              \n                j\n              \n            \n          \n          \n            (\n            k\n            )\n          \n        \n      \n    \n    {\\displaystyle {\\mathbf {W} }_{jk}={w_{j}}_{(k)}}\n  \n.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0033", "text": "First component\nIn order to maximize variance, the first weight vector w(1) thus has to satisfy", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0034", "text": "w\n          \n          \n            (\n            1\n            )\n          \n        \n        =\n        arg\n        \u2061\n        \n          max\n          \n            \u2016\n            \n              w\n            \n            \u2016\n            =\n            1\n          \n        \n        \n        \n          {\n          \n            \n              \u2211\n              \n                i\n              \n            \n            (\n            \n              t", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0035", "text": "(\n            \n              t\n              \n                1\n              \n            \n            \n              )\n              \n                (\n                i\n                )\n              \n              \n                2\n              \n            \n          \n          }\n        \n        =\n        arg\n        \u2061\n        \n          max\n          \n            \u2016\n            \n              w\n            \n            \u2016\n            =\n            1", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0036", "text": "=\n            1\n          \n        \n        \n        \n          {\n          \n            \n              \u2211\n              \n                i\n              \n            \n            \n              \n                (\n                \n                  \n                    \n                      x\n                    \n                    \n                      (\n                      i\n                      )\n                    \n                  \n                  \u22c5", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0037", "text": "\u22c5\n                  \n                    w\n                  \n                \n                )\n              \n              \n                2\n              \n            \n          \n          }\n        \n      \n    \n    {\\displaystyle \\mathbf {w} _{(1)}=\\arg \\max _{\\Vert \\mathbf {w} \\Vert =1}\\,\\left\\{\\sum _{i}(t_{1})_{(i)}^{2}\\right\\}=\\arg \\max _{\\Vert \\mathbf {w} \\Vert =1}\\,\\left\\{\\sum _{i}\\left(\\mathbf {x} _{(i)}\\cdot \\mathbf {w} \\right)^{2}\\right\\}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0038", "text": "Equivalently, writing this in matrix form gives", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0039", "text": "w\n          \n          \n            (\n            1\n            )\n          \n        \n        =\n        arg\n        \u2061\n        \n          max\n          \n            \n              \u2016\n              \n                w\n              \n              \u2016\n            \n            =\n            1\n          \n        \n        \n          {\n          \n            \n              \u2016\n              \n                X\n                w\n              \n              \u2016", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0040", "text": "w\n              \n              \u2016\n            \n            \n              2\n            \n          \n          }\n        \n        =\n        arg\n        \u2061\n        \n          max\n          \n            \n              \u2016\n              \n                w\n              \n              \u2016\n            \n            =\n            1\n          \n        \n        \n          {\n          \n            \n              \n                w", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0041", "text": "T\n                \n              \n            \n            \n              \n                X\n              \n              \n                \n                  T\n                \n              \n            \n            \n              X\n              w\n            \n          \n          }", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0042", "text": "}\n        \n      \n    \n    {\\displaystyle \\mathbf {w} _{(1)}=\\arg \\max _{\\left\\|\\mathbf {w} \\right\\|=1}\\left\\{\\left\\|\\mathbf {Xw} \\right\\|^{2}\\right\\}=\\arg \\max _{\\left\\|\\mathbf {w} \\right\\|=1}\\left\\{\\mathbf {w} ^{\\mathsf {T}}\\mathbf {X} ^{\\mathsf {T}}\\mathbf {Xw} \\right\\}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0043", "text": "Since w(1) has been defined to be a unit vector, it equivalently also satisfies", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0044", "text": "w\n          \n          \n            (\n            1\n            )\n          \n        \n        =\n        arg\n        \u2061\n        max\n        \n          {\n          \n            \n              \n                \n                  \n                    w\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  \n                    X", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0045", "text": "X\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  X\n                  w\n                \n              \n              \n                \n                  \n                    w\n                  \n                  \n                    \n                      T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0046", "text": "w\n                \n              \n            \n          \n          }\n        \n      \n    \n    {\\displaystyle \\mathbf {w} _{(1)}=\\arg \\max \\left\\{{\\frac {\\mathbf {w} ^{\\mathsf {T}}\\mathbf {X} ^{\\mathsf {T}}\\mathbf {Xw} }{\\mathbf {w} ^{\\mathsf {T}}\\mathbf {w} }}\\right\\}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0047", "text": "The quantity to be maximised can be recognised as a Rayleigh quotient. A standard result for a positive semidefinite matrix such as XTX is that the quotient's maximum possible value is the largest eigenvalue of the matrix, which occurs when w is the corresponding eigenvector.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0048", "text": "With w(1) found, the first principal component of a data vector x(i) can then be given as a score t1(i) = x(i) \u22c5 w(1) in the transformed co-ordinates, or as the corresponding vector in the original variables, {x(i) \u22c5 w(1)} w(1).", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0049", "text": "Further components\nThe k-th component can be found by subtracting the first k \u2212 1 principal components from X:", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0050", "text": "X\n                ^\n              \n            \n          \n          \n            k\n          \n        \n        =\n        \n          X\n        \n        \u2212\n        \n          \u2211\n          \n            s\n            =\n            1\n          \n          \n            k\n            \u2212\n            1\n          \n        \n        \n          X\n        \n        \n          \n            w\n          \n          \n            (", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0051", "text": "w\n          \n          \n            (\n            s\n            )\n          \n        \n        \n          \n            w\n          \n          \n            (\n            s\n            )\n          \n          \n            \n              T\n            \n          \n        \n      \n    \n    {\\displaystyle \\mathbf {\\hat {X}} _{k}=\\mathbf {X} -\\sum _{s=1}^{k-1}\\mathbf {X} \\mathbf {w} _{(s)}\\mathbf {w} _{(s)}^{\\mathsf {T}}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0052", "text": "and then finding the weight vector which extracts the maximum variance from this new data matrix", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0053", "text": "w\n          \n          \n            (\n            k\n            )\n          \n        \n        =\n        \n          \n            \n              a\n              r\n              g\n              \n              m\n              a\n              x\n            \n          \n          \n            \n              \u2016\n              \n                w\n              \n              \u2016\n            \n            =\n            1\n          \n        \n        \u2061", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0054", "text": "\u2061\n        \n          {\n          \n            \n              \u2016\n              \n                \n                  \n                    \n                      \n                        X\n                        ^\n                      \n                    \n                  \n                  \n                    k\n                  \n                \n                \n                  w\n                \n              \n              \u2016", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0055", "text": "\u2016\n            \n            \n              2\n            \n          \n          }\n        \n        =\n        arg\n        \u2061\n        max\n        \n          {\n          \n            \n              \n                \n                  \n                    \n                      w\n                    \n                    \n                      \n                        T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0056", "text": "X\n                          ^\n                        \n                      \n                    \n                    \n                      k\n                    \n                    \n                      \n                        T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0057", "text": "X\n                          ^\n                        \n                      \n                    \n                    \n                      k\n                    \n                  \n                  \n                    w\n                  \n                \n                \n                  \n                    \n                      w\n                    \n                    \n                      T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0058", "text": "T\n                    \n                  \n                  \n                    w\n                  \n                \n              \n            \n          \n          }", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0059", "text": "}\n        \n      \n    \n    {\\displaystyle \\mathbf {w} _{(k)}=\\mathop {\\operatorname {arg\\,max} } _{\\left\\|\\mathbf {w} \\right\\|=1}\\left\\{\\left\\|\\mathbf {\\hat {X}} _{k}\\mathbf {w} \\right\\|^{2}\\right\\}=\\arg \\max \\left\\{{\\tfrac {\\mathbf {w} ^{\\mathsf {T}}\\mathbf {\\hat {X}} _{k}^{\\mathsf {T}}\\mathbf {\\hat {X}} _{k}\\mathbf {w} }{\\mathbf {w} ^{T}\\mathbf {w} }}\\right\\}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0060", "text": "It turns out that this gives the remaining eigenvectors of XTX, with the maximum values for the quantity in brackets given by their corresponding eigenvalues. Thus the weight vectors are eigenvectors of XTX.\nThe k-th principal component of a data vector x(i) can therefore be given as a score tk(i) = x(i) \u22c5 w(k) in the transformed coordinates, or as the corresponding vector in the space of the original variables, {x(i) \u22c5 w(k)} w(k), where w(k) is the kth eigenvector of XTX.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0061", "text": "The full principal components decomposition of X can therefore be given as", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0062", "text": "T\n        \n        =\n        \n          X\n        \n        \n          W\n        \n      \n    \n    {\\displaystyle \\mathbf {T} =\\mathbf {X} \\mathbf {W} }", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0063", "text": "where W is a p-by-p matrix of weights whose columns are the eigenvectors of XTX. The transpose of W is sometimes called the whitening or sphering transformation. Columns of W multiplied by the square root of corresponding eigenvalues, that is, eigenvectors scaled up by the variances, are called loadings in PCA or in Factor analysis.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0064", "text": "Covariances\nXTX itself can be recognized as proportional to the empirical sample covariance matrix of the dataset XT.\nThe sample covariance Q between two of the different principal components over the dataset is given by:", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0065", "text": "Q\n                (\n                \n                  \n                    P\n                    C\n                  \n                  \n                    (\n                    j\n                    )\n                  \n                \n                ,\n                \n                  \n                    P\n                    C\n                  \n                  \n                    (\n                    k", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0066", "text": "(\n                    k\n                    )\n                  \n                \n                )\n              \n              \n                \n                \u221d\n                (\n                \n                  X\n                \n                \n                  \n                    w\n                  \n                  \n                    (\n                    j\n                    )\n                  \n                \n                \n                  )", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0067", "text": ")\n                  \n                    \n                      T\n                    \n                  \n                \n                (\n                \n                  X\n                \n                \n                  \n                    w\n                  \n                  \n                    (\n                    k\n                    )\n                  \n                \n                )", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0068", "text": "=\n                \n                  \n                    w\n                  \n                  \n                    (\n                    j\n                    )\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  \n                    X", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0069", "text": "X\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  X\n                \n                \n                  \n                    w\n                  \n                  \n                    (\n                    k\n                    )", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0070", "text": "=\n                \n                  \n                    w\n                  \n                  \n                    (\n                    j\n                    )\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  \u03bb\n                  \n                    (\n                    k", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0071", "text": "(\n                    k\n                    )\n                  \n                \n                \n                  \n                    w\n                  \n                  \n                    (\n                    k\n                    )\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  \u03bb\n                  \n                    (", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0072", "text": "(\n                    k\n                    )\n                  \n                \n                \n                  \n                    w\n                  \n                  \n                    (\n                    j\n                    )\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  \n                    w", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0073", "text": "w\n                  \n                  \n                    (\n                    k\n                    )", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0074", "text": "{\\displaystyle {\\begin{aligned}Q(\\mathrm {PC} _{(j)},\\mathrm {PC} _{(k)})&\\propto (\\mathbf {X} \\mathbf {w} _{(j)})^{\\mathsf {T}}(\\mathbf {X} \\mathbf {w} _{(k)})\\\\&=\\mathbf {w} _{(j)}^{\\mathsf {T}}\\mathbf {X} ^{\\mathsf {T}}\\mathbf {X} \\mathbf {w} _{(k)}\\\\&=\\mathbf {w} _{(j)}^{\\mathsf {T}}\\lambda _{(k)}\\mathbf {w} _{(k)}\\\\&=\\lambda _{(k)}\\mathbf {w} _{(j)}^{\\mathsf {T}}\\mathbf {w} _{(k)}\\end{aligned}}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0075", "text": "where the eigenvalue property of w(k) has been used to move from line 2 to line 3. However eigenvectors w(j) and w(k) corresponding to eigenvalues of a symmetric matrix are orthogonal (if the eigenvalues are different), or can be orthogonalised (if the vectors happen to share an equal repeated value). The product in the final line is therefore zero; there is no sample covariance between different principal components over the dataset.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0076", "text": "Another way to characterise the principal components transformation is therefore as the transformation to coordinates which diagonalise the empirical sample covariance matrix.\nIn matrix form, the empirical covariance matrix for the original variables can be written", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0077", "text": "Q\n        \n        \u221d\n        \n          \n            X\n          \n          \n            \n              T\n            \n          \n        \n        \n          X\n        \n        =\n        \n          W\n        \n        \n          \u039b\n        \n        \n          \n            W\n          \n          \n            \n              T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0078", "text": "{\\displaystyle \\mathbf {Q} \\propto \\mathbf {X} ^{\\mathsf {T}}\\mathbf {X} =\\mathbf {W} \\mathbf {\\Lambda } \\mathbf {W} ^{\\mathsf {T}}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0079", "text": "The empirical covariance matrix between the principal components becomes", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0080", "text": "W\n          \n          \n            \n              T\n            \n          \n        \n        \n          Q\n        \n        \n          W\n        \n        \u221d\n        \n          \n            W\n          \n          \n            \n              T\n            \n          \n        \n        \n          W\n        \n        \n        \n          \u039b\n        \n        \n        \n          \n            W\n          \n          \n            \n              T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0081", "text": "T\n            \n          \n        \n        \n          W\n        \n        =\n        \n          \u039b\n        \n      \n    \n    {\\displaystyle \\mathbf {W} ^{\\mathsf {T}}\\mathbf {Q} \\mathbf {W} \\propto \\mathbf {W} ^{\\mathsf {T}}\\mathbf {W} \\,\\mathbf {\\Lambda } \\,\\mathbf {W} ^{\\mathsf {T}}\\mathbf {W} =\\mathbf {\\Lambda } }", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0082", "text": "where \u039b is the diagonal matrix of eigenvalues \u03bb(k) of XTX. \u03bb(k) is equal to the sum of the squares over the dataset associated with each component k, that is, \u03bb(k) = \u03a3i tk2(i) = \u03a3i (x(i) \u22c5 w(k))2.\n\nDimensionality reduction\nThe transformation P = X W maps a data vector x(i) from an original space of x variables to a new space of p variables which are uncorrelated over the dataset.\nTo non-dimensionalize the centered data, let Xc represent the characteristic values of data vectors Xi, given by:", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0083", "text": "\u2016\n        X\n        \n          \u2016\n          \n            \u221e\n          \n        \n      \n    \n    {\\displaystyle \\|X\\|_{\\infty }}\n  \n (maximum norm),\n\n  \n    \n      \n        \n          \n            1\n            n\n          \n        \n        \u2016\n        X\n        \n          \u2016\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle {\\frac {1}{n}}\\|X\\|_{1}}\n  \n (mean absolute value), or", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0084", "text": "1\n            \n              n\n            \n          \n        \n        \u2016\n        X\n        \n          \u2016\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle {\\frac {1}{\\sqrt {n}}}\\|X\\|_{2}}\n  \n (normalized Euclidean norm),\nfor a dataset of size n. These norms are used to transform the original space of variables x, y to a new space of uncorrelated variables p, q (given Yc with same meaning), such that", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0085", "text": "p\n          \n            i\n          \n        \n        =\n        \n          \n            \n              X\n              \n                i\n              \n            \n            \n              X\n              \n                c\n              \n            \n          \n        \n        ,\n        \n        \n          q\n          \n            i\n          \n        \n        =\n        \n          \n            \n              Y\n              \n                i", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0086", "text": "Y\n              \n                i\n              \n            \n            \n              Y\n              \n                c\n              \n            \n          \n        \n      \n    \n    {\\displaystyle p_{i}={\\frac {X_{i}}{X_{c}}},\\quad q_{i}={\\frac {Y_{i}}{Y_{c}}}}\n  \n;\nand the new variables are linearly related as: \n  \n    \n      \n        q\n        =\n        \u03b1\n        p\n      \n    \n    {\\displaystyle q=\\alpha p}\n  \n.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0087", "text": "{\\displaystyle q=\\alpha p}\n  \n.\nTo find the optimal linear relationship, we minimize the total squared reconstruction error:", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0088", "text": "E\n        (\n        \u03b1\n        )\n        =\n        \n          \n            1\n            \n              1\n              \u2212\n              \n                \u03b1\n                \n                  2\n                \n              \n            \n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        (\n        \u03b1\n        \n          p\n          \n            i", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0089", "text": "i\n          \n        \n        \u2212\n        \n          q\n          \n            i\n          \n        \n        \n          )\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle E(\\alpha )={\\frac {1}{1-\\alpha ^{2}}}\\sum _{i=1}^{n}(\\alpha p_{i}-q_{i})^{2}}\n  \n; such that setting the derivative of the error function to zero \n  \n    \n      \n        (\n        \n          E\n          \u2032\n        \n        (\n        \u03b1\n        )\n        =\n        0\n        )", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0090", "text": ")\n        =\n        0\n        )\n      \n    \n    {\\displaystyle (E'(\\alpha )=0)}\n  \n yields:\n  \n    \n      \n        \u03b1\n        =\n        \n          \n            1\n            2\n          \n        \n        \n          (\n          \n            \u2212\n            \u03bb\n            \u00b1\n            \n              \n                \n                  \u03bb\n                  \n                    2\n                  \n                \n                +\n                4", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0091", "text": ")\n        \n      \n    \n    {\\displaystyle \\alpha ={\\frac {1}{2}}\\left(-\\lambda \\pm {\\sqrt {\\lambda ^{2}+4}}\\right)}\n  \n where\n  \n    \n      \n        \u03bb\n        =\n        \n          \n            \n              p\n              \u22c5\n              p\n              \u2212\n              q\n              \u22c5\n              q\n            \n            \n              p\n              \u22c5\n              q", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0092", "text": "{\\displaystyle \\lambda ={\\frac {p\\cdot p-q\\cdot q}{p\\cdot q}}}\n  \n.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0093", "text": "Such dimensionality reduction can be a very useful step for visualising and processing high-dimensional datasets, while still retaining as much of the variance in the dataset as possible", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0094", "text": ". For example, selecting L = 2 and keeping only the first two principal components finds the two-dimensional plane through the high-dimensional dataset in which the data are most spread out, so if the data contains clusters these too may be most spread out, and therefore most visible to be plotted out in a two-dimensional diagram; whereas if two directions through the data (or two of the original variables) are chosen at random, the clusters may be much less spread apart from each other, and may", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0095", "text": "much less spread apart from each other, and may in fact be much more likely to substantially overlay each other, making them indistinguishable.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0096", "text": "Similarly, in regression analysis, the larger the number of explanatory variables allowed, the greater is the chance of overfitting the model, producing conclusions that fail to generalise to other datasets. One approach, especially when there are strong correlations between different possible explanatory variables, is to reduce them to a few principal components and then run the regression against them, a method called principal component regression.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0097", "text": "Dimensionality reduction may also be appropriate when the variables in a dataset are noisy. If each column of the dataset contains independent identically distributed Gaussian noise, then the columns of T will also contain similarly identically distributed Gaussian noise (such a distribution is invariant under the effects of the matrix W, which can be thought of as a high-dimensional rotation of the co-ordinate axes)", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0098", "text": ". However, with more of the total variance concentrated in the first few principal components compared to the same noise variance, the proportionate effect of the noise is less\u2014the first few components achieve a higher signal-to-noise ratio", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0099", "text": ". PCA thus can have the effect of concentrating much of the signal into the first few principal components, which can usefully be captured by dimensionality reduction; while the later principal components may be dominated by noise, and so disposed of without great loss. If the dataset is not too large, the significance of the principal components can be tested using parametric bootstrap, as an aid in determining how many principal components to retain.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0100", "text": "Singular value decomposition\nThe principal components transformation can also be associated with another matrix factorization, the singular value decomposition (SVD) of X,\n\n  \n    \n      \n        \n          X\n        \n        =\n        \n          U\n        \n        \n          \u03a3\n        \n        \n          \n            W\n          \n          \n            T\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {X} =\\mathbf {U} \\mathbf {\\Sigma } \\mathbf {W} ^{T}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0101", "text": "Here \u03a3 is an n-by-p rectangular diagonal matrix of positive numbers \u03c3(k), called the singular values of X; U is an n-by-n matrix, the columns of which are orthogonal unit vectors of length n called the left singular vectors of X; and W is a p-by-p matrix whose columns are orthogonal unit vectors of length p and called the right singular vectors of X.\nIn terms of this factorization, the matrix XTX can be written", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0102", "text": "X\n                  \n                  \n                    T\n                  \n                \n                \n                  X\n                \n              \n              \n                \n                =\n                \n                  W\n                \n                \n                  \n                    \u03a3", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0103", "text": "T\n                    \n                  \n                \n                \n                  \n                    U\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  U\n                \n                \n                  \u03a3", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0104", "text": "W\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  W\n                \n                \n                  \n                    \u03a3\n                  \n                  \n                    \n                      T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0105", "text": "T\n                    \n                  \n                \n                \n                  \u03a3\n                \n                \n                  \n                    W\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  W", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0106", "text": "W\n                \n                \n                  \n                    \n                      \n                        \u03a3\n                        ^\n                      \n                    \n                  \n                  \n                    2\n                  \n                \n                \n                  \n                    W\n                  \n                  \n                    \n                      T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0107", "text": "T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0108", "text": "{\\displaystyle {\\begin{aligned}\\mathbf {X} ^{T}\\mathbf {X} &=\\mathbf {W} \\mathbf {\\Sigma } ^{\\mathsf {T}}\\mathbf {U} ^{\\mathsf {T}}\\mathbf {U} \\mathbf {\\Sigma } \\mathbf {W} ^{\\mathsf {T}}\\\\&=\\mathbf {W} \\mathbf {\\Sigma } ^{\\mathsf {T}}\\mathbf {\\Sigma } \\mathbf {W} ^{\\mathsf {T}}\\\\&=\\mathbf {W} \\mathbf {\\hat {\\Sigma }} ^{2}\\mathbf {W} ^{\\mathsf {T}}\\end{aligned}}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0109", "text": "where  \n  \n    \n      \n        \n          \n            \n              \u03a3\n              ^\n            \n          \n        \n      \n    \n    {\\displaystyle \\mathbf {\\hat {\\Sigma }} }\n  \n is the square diagonal matrix with the singular values of X and the excess zeros chopped off that satisfies \n  \n    \n      \n        \n          \n            \n              \n                \n                  \u03a3\n                  ^\n                \n              \n            \n            \n              2", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0110", "text": "2\n            \n          \n        \n        =\n        \n          \n            \u03a3\n          \n          \n            \n              T\n            \n          \n        \n        \n          \u03a3\n        \n      \n    \n    {\\displaystyle \\mathbf {{\\hat {\\Sigma }}^{2}} =\\mathbf {\\Sigma } ^{\\mathsf {T}}\\mathbf {\\Sigma } }", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0111", "text": ". Comparison with the eigenvector factorization of XTX establishes that the right singular vectors W of X are equivalent to the eigenvectors of XTX, while the singular values \u03c3(k) of  \n  \n    \n      \n        \n          X\n        \n      \n    \n    {\\displaystyle \\mathbf {X} }\n  \n are equal to the square-root of the eigenvalues \u03bb(k) of XTX.\nUsing the singular value decomposition the score matrix T can be written", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0112", "text": "T\n                \n              \n              \n                \n                =\n                \n                  X\n                \n                \n                  W\n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  U\n                \n                \n                  \u03a3", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0113", "text": "\u03a3\n                \n                \n                  \n                    W\n                  \n                  \n                    \n                      T\n                    \n                  \n                \n                \n                  W\n                \n              \n            \n            \n              \n              \n                \n                =\n                \n                  U", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0114", "text": "\u03a3\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}\\mathbf {T} &=\\mathbf {X} \\mathbf {W} \\\\&=\\mathbf {U} \\mathbf {\\Sigma } \\mathbf {W} ^{\\mathsf {T}}\\mathbf {W} \\\\&=\\mathbf {U} \\mathbf {\\Sigma } \\end{aligned}}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0115", "text": "so each column of T is given by one of the left singular vectors of X multiplied by the corresponding singular value. This form is also the polar decomposition of T.\nEfficient algorithms exist to calculate the SVD of X without having to form the matrix XTX, so computing the SVD is now the standard way to calculate a principal components analysis from a data matrix, unless only a handful of components are required.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0116", "text": "As with the eigen-decomposition, a truncated n \u00d7 L score matrix TL can be obtained by considering only the first L largest singular values and their singular vectors:", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0117", "text": "T\n          \n          \n            L\n          \n        \n        =\n        \n          \n            U\n          \n          \n            L\n          \n        \n        \n          \n            \u03a3\n          \n          \n            L\n          \n        \n        =\n        \n          X\n        \n        \n          \n            W\n          \n          \n            L", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0118", "text": "L\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {T} _{L}=\\mathbf {U} _{L}\\mathbf {\\Sigma } _{L}=\\mathbf {X} \\mathbf {W} _{L}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0119", "text": "The truncation of a matrix M or T using a truncated singular value decomposition in this way produces a truncated matrix that is the nearest possible matrix of rank L to the original matrix, in the sense of the difference between the two having the smallest possible Frobenius norm, a result known as the Eckart\u2013Young theorem [1936].\n\nTheorem (Optimal k\u2011dimensional fit).  \nLet P be an n\u00d7m data matrix whose columns have been mean\u2011centered and scaled, and let", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0120", "text": "P\n        =\n        U\n        \n        \u03a3\n        \n        \n          V\n          \n            T\n          \n        \n      \n    \n    {\\displaystyle P=U\\,\\Sigma \\,V^{T}}\n  \n  \nbe its singular value decomposition. Then the best rank\u2011k approximation to P in the least\u2011squares (Frobenius\u2011norm) sense is", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0121", "text": "P\n          \n            k\n          \n        \n        =\n        \n          U\n          \n            k\n          \n        \n        \n        \n          \u03a3\n          \n            k\n          \n        \n        \n        \n          V\n          \n            k\n          \n          \n            T\n          \n        \n      \n    \n    {\\displaystyle P_{k}=U_{k}\\,\\Sigma _{k}\\,V_{k}^{T}}\n  \n,", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0122", "text": ",  \nwhere Vk consists of the first k columns of V. Moreover, the relative residual variance is", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0123", "text": "R\n        (\n        k\n        )\n        =\n        \n          \n            \n              \n                \u2211\n                \n                  j\n                  =\n                  k\n                  +\n                  1\n                \n                \n                  m\n                \n              \n              \n                \u03c3\n                \n                  j\n                \n                \n                  2", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0124", "text": "\u2211\n                \n                  j\n                  =\n                  1\n                \n                \n                  m\n                \n              \n              \n                \u03c3\n                \n                  j\n                \n                \n                  2", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0125", "text": "{\\displaystyle R(k)={\\frac {\\sum _{j=k+1}^{m}\\sigma _{j}^{2}}{\\sum _{j=1}^{m}\\sigma _{j}^{2}}}}\n  \n.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0126", "text": "Further considerations", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0127", "text": "The singular values (in \u03a3) are the square roots of the eigenvalues of the matrix XTX. Each eigenvalue is proportional to the portion of the \"variance\" (more correctly of the sum of the squared distances of the points from their multidimensional mean) that is associated with each eigenvector. The sum of all the eigenvalues is equal to the sum of the squared distances of the points from their multidimensional mean", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0128", "text": ". PCA essentially rotates the set of points around their mean in order to align with the principal components. This moves as much of the variance as possible (using an orthogonal transformation) into the first few dimensions. The values in the remaining dimensions, therefore, tend to be small and may be dropped with minimal loss of information (see below). PCA is often used in this manner for dimensionality reduction", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0129", "text": ". PCA has the distinction of being the optimal orthogonal transformation for keeping the subspace that has largest \"variance\" (as defined above). This advantage, however, comes at the price of greater computational requirements if compared, for example, and when applicable, to the discrete cosine transform, and in particular to the DCT-II which is simply known as the \"DCT\". Nonlinear dimensionality reduction techniques tend to be more computationally demanding than PCA.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0130", "text": "PCA is sensitive to the scaling of the variables. Mathematically this sensitivity comes from the way a rescaling changes the sample\u2011covariance matrix that PCA diagonalises.\nLet \n  \n    \n      \n        \n          \n            X\n          \n          \n            c\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {X} _{\\text{c}}}\n  \n be the *centered* data matrix (n rows, p columns) and define the covariance", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0131", "text": "\u03a3\n        =\n        \n          \n            1\n            n\n          \n        \n        \n        \n          \n            X\n          \n          \n            c\n          \n          \n            \n              T\n            \n          \n        \n        \n          \n            X\n          \n          \n            c\n          \n        \n        .\n      \n    \n    {\\displaystyle \\Sigma ={\\frac {1}{n}}\\,\\mathbf {X} _{\\text{c}}^{\\mathsf {T}}\\mathbf {X} _{\\text{c}}.}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0132", "text": "If the \n  \n    \n      \n        j\n      \n    \n    {\\displaystyle j}\n  \n\u2011th variable is multiplied by a factor \n  \n    \n      \n        \n          \u03b1\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle \\alpha _{j}}\n  \n we obtain", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0133", "text": "X\n          \n          \n            c\n          \n          \n            (\n            \u03b1\n            )\n          \n        \n        =\n        \n          \n            X\n          \n          \n            c\n          \n        \n        D\n        ,\n        \n        D\n        =\n        diag\n        \u2061\n        (\n        \n          \u03b1\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \u03b1\n          \n            p", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0134", "text": "\u03b1\n          \n            p\n          \n        \n        )\n        .\n      \n    \n    {\\displaystyle \\mathbf {X} _{\\text{c}}^{(\\alpha )}=\\mathbf {X} _{\\text{c}}D,\\qquad D=\\operatorname {diag} (\\alpha _{1},\\ldots ,\\alpha _{p}).}\n  \n  \nHence the new covariance is", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0135", "text": "\u03a3\n          \n            (\n            \u03b1\n            )\n          \n        \n        =\n        \n          D\n          \n            \n              T\n            \n          \n        \n        \n        \u03a3\n        \n        D\n        .\n      \n    \n    {\\displaystyle \\Sigma ^{(\\alpha )}=D^{\\mathsf {T}}\\,\\Sigma \\,D.}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0136", "text": "Because the eigenvalues and eigenvectors of \n  \n    \n      \n        \n          \u03a3\n          \n            (\n            \u03b1\n            )\n          \n        \n      \n    \n    {\\displaystyle \\Sigma ^{(\\alpha )}}\n  \n are those of \n  \n    \n      \n        \u03a3\n      \n    \n    {\\displaystyle \\Sigma }\n  \n scaled by \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n, the principal axes rotate toward any column whose variance has been inflated, exactly as the 2\u2011D example below illustrates.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0137", "text": "If we have just two variables and they have the same sample variance and are completely correlated, then the PCA will entail a rotation by 45\u00b0 and the \"weights\" (they are the cosines of rotation) for the two variables with respect to the principal component will be equal", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0138", "text": ". But if we multiply all values of the first variable by 100, then the first principal component will be almost the same as that variable, with a small contribution from the other variable, whereas the second component will be almost aligned with the second original variable. This means that whenever the different variables have different units (like temperature and mass), PCA is a somewhat arbitrary method of analysis", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0139", "text": ". (Different results would be obtained if one used Fahrenheit rather than Celsius for example.) Pearson's original paper was entitled \"On Lines and Planes of Closest Fit to Systems of Points in Space\" \u2013 \"in space\" implies physical Euclidean space where such concerns do not arise. One way of making the PCA less arbitrary is to use variables scaled so as to have unit variance, by standardizing the data and hence use the autocorrelation matrix instead of the autocovariance matrix as a basis for PCA", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0140", "text": ". However, this compresses (or expands) the fluctuations in all dimensions of the signal space to unit variance.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0141", "text": "Classical PCA assumes the cloud of points has already been translated so its centroid is at the origin.\nWrite each observation as", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0142", "text": "q\n          \n          \n            i\n          \n        \n        =\n        \n          \u03bc\n        \n        +\n        \n          \n            z\n          \n          \n            i\n          \n        \n        ,\n        \n        \n          \u03bc\n        \n        =\n        \n          \n            \n              1\n              n\n            \n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0143", "text": "=\n            1\n          \n          \n            n\n          \n        \n        \n          \n            q\n          \n          \n            i\n          \n        \n        .\n      \n    \n    {\\displaystyle \\mathbf {q} _{i}={\\boldsymbol {\\mu }}+\\mathbf {z} _{i},\\qquad {\\boldsymbol {\\mu }}={\\tfrac {1}{n}}\\sum _{i=1}^{n}\\mathbf {q} _{i}.}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0144", "text": "Without subtracting \n  \n    \n      \n        \n          \u03bc\n        \n      \n    \n    {\\displaystyle {\\boldsymbol {\\mu }}}\n  \n we are in effect diagonalising", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0145", "text": "\u03a3\n          \n            unc\n          \n        \n        \n        =\n        \n        n\n        \n        \n          \u03bc\n        \n        \n          \n            \u03bc\n          \n          \n            \n              T\n            \n          \n        \n        \n        +\n        \n        \n          \n            \n              1\n              n\n            \n          \n        \n        \n        \n          \n            Z\n          \n          \n            \n              T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0146", "text": "T\n            \n          \n        \n        \n          Z\n        \n        ,\n      \n    \n    {\\displaystyle \\Sigma _{\\text{unc}}\\;=\\;n\\,{\\boldsymbol {\\mu }}{\\boldsymbol {\\mu }}^{\\mathsf {T}}\\;+\\;{\\tfrac {1}{n}}\\,\\mathbf {Z} ^{\\mathsf {T}}\\mathbf {Z} ,}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0147", "text": "where \n  \n    \n      \n        \n          Z\n        \n      \n    \n    {\\displaystyle \\mathbf {Z} }\n  \n is the centered matrix.  \nThe rank\u2011one term \n  \n    \n      \n        n\n        \n        \n          \u03bc\n        \n        \n          \n            \u03bc\n          \n          \n            \n              T\n            \n          \n        \n      \n    \n    {\\displaystyle n\\,{\\boldsymbol {\\mu }}{\\boldsymbol {\\mu }}^{\\mathsf {T}}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0148", "text": "often dominates, forcing the leading eigenvector to point almost exactly toward the mean and obliterating any structure in the centred part \n  \n    \n      \n        \n          Z\n        \n      \n    \n    {\\displaystyle \\mathbf {Z} }\n  \n.  \nAfter mean subtraction that term vanishes and the principal axes align with the true directions of maximal variance.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0149", "text": "Mean-centering is unnecessary if performing a principal components analysis on a correlation matrix, as the data are already centered after calculating correlations. Correlations are derived from the cross-product of two standard scores (Z-scores) or statistical moments (hence the name: Pearson Product-Moment Correlation). Also see the article by Kromrey & Foster-Johnson (1998) on \"Mean-centering in Moderated Regression: Much Ado About Nothing\"", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0150", "text": ". Since covariances are correlations of normalized variables (Z- or standard-scores) a PCA based on the correlation matrix of X is equal to a PCA based on the covariance matrix of Z, the standardized version of X.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0151", "text": "PCA is a popular primary technique in pattern recognition. It is not, however, optimized for class separability. However, it has been used to quantify the distance between two or more classes by calculating center of mass for each class in principal component space and reporting Euclidean distance between center of mass of two or more classes. The linear discriminant analysis is an alternative which is optimized for class separability.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0152", "text": "Table of symbols and abbreviations\nProperties and limitations\nProperties\nSome properties of PCA include:\n\nProperty 1: For any integer q, 1 \u2264 q \u2264 p, consider the orthogonal linear transformation\n\n  \n    \n      \n        y\n        =\n        \n          \n            B\n            \u2032\n          \n        \n        x\n      \n    \n    {\\displaystyle y=\\mathbf {B'} x}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0153", "text": "where \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n is a q-element vector and \n  \n    \n      \n        \n          \n            B\n            \u2032\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {B'} }\n  \n is a (q \u00d7 p) matrix, and let \n  \n    \n      \n        \n          \n            \u03a3\n          \n          \n            y\n          \n        \n        =\n        \n          \n            B\n            \u2032\n          \n        \n        \n          \u03a3\n        \n        \n          B", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0154", "text": "\u03a3\n        \n        \n          B\n        \n      \n    \n    {\\displaystyle \\mathbf {\\Sigma } _{y}=\\mathbf {B'} \\mathbf {\\Sigma } \\mathbf {B} }\n  \n be the variance-covariance matrix for \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n. Then the trace of \n  \n    \n      \n        \n          \n            \u03a3\n          \n          \n            y\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {\\Sigma } _{y}}\n  \n, denoted \n  \n    \n      \n        tr\n        \u2061\n        (", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0155", "text": "tr\n        \u2061\n        (\n        \n          \n            \u03a3\n          \n          \n            y\n          \n        \n        )\n      \n    \n    {\\displaystyle \\operatorname {tr} (\\mathbf {\\Sigma } _{y})}\n  \n, is maximized by taking \n  \n    \n      \n        \n          B\n        \n        =\n        \n          \n            A\n          \n          \n            q\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {B} =\\mathbf {A} _{q}}\n  \n, where", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0156", "text": ", where \n  \n    \n      \n        \n          \n            A\n          \n          \n            q\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {A} _{q}}\n  \n consists of the first q columns of \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }\n  \n \n  \n    \n      \n        (\n        \n          \n            B\n            \u2032\n          \n        \n      \n    \n    {\\displaystyle (\\mathbf {B'} }\n  \n is the transpose of", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0157", "text": "is the transpose of \n  \n    \n      \n        \n          B\n        \n        )\n      \n    \n    {\\displaystyle \\mathbf {B} )}\n  \n. (\n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }\n  \n is not defined here)\nProperty 2: Consider again the orthonormal transformation", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0158", "text": "y\n        =\n        \n          \n            B\n            \u2032\n          \n        \n        x\n      \n    \n    {\\displaystyle y=\\mathbf {B'} x}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0159", "text": "with \n  \n    \n      \n        x\n        ,\n        \n          B\n        \n        ,\n        \n          A\n        \n      \n    \n    {\\displaystyle x,\\mathbf {B} ,\\mathbf {A} }\n  \n and \n  \n    \n      \n        \n          \n            \u03a3\n          \n          \n            y\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {\\Sigma } _{y}}\n  \n defined as before. Then \n  \n    \n      \n        tr\n        \u2061\n        (\n        \n          \n            \u03a3\n          \n          \n            y", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0160", "text": "y\n          \n        \n        )\n      \n    \n    {\\displaystyle \\operatorname {tr} (\\mathbf {\\Sigma } _{y})}\n  \n is minimized by taking \n  \n    \n      \n        \n          B\n        \n        =\n        \n          \n            A\n          \n          \n            q\n          \n          \n            \u2217\n          \n        \n        ,\n      \n    \n    {\\displaystyle \\mathbf {B} =\\mathbf {A} _{q}^{*},}\n  \n where \n  \n    \n      \n        \n          \n            A", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0161", "text": "A\n          \n          \n            q\n          \n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {A} _{q}^{*}}\n  \n consists of the last q columns of \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }\n  \n.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0162", "text": "{\\displaystyle \\mathbf {A} }\n  \n.\nThe statistical implication of this property is that the last few PCs are not simply unstructured left-overs after removing the important PCs. Because these last PCs have variances as small as possible they are useful in their own right. They can help to detect unsuspected near-constant linear relationships between the elements of x, and they may also be useful in regression, in selecting a subset of variables from x, and in outlier detection.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0163", "text": "Property 3: (Spectral decomposition of \u03a3)", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0164", "text": "\u03a3\n        \n        =\n        \n          \u03bb\n          \n            1\n          \n        \n        \n          \u03b1\n          \n            1\n          \n        \n        \n          \u03b1\n          \n            1\n          \n          \u2032\n        \n        +\n        \u22ef\n        +\n        \n          \u03bb\n          \n            p\n          \n        \n        \n          \u03b1\n          \n            p\n          \n        \n        \n          \u03b1\n          \n            p\n          \n          \u2032", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0165", "text": "p\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\mathbf {\\Sigma } =\\lambda _{1}\\alpha _{1}\\alpha _{1}'+\\cdots +\\lambda _{p}\\alpha _{p}\\alpha _{p}'}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0166", "text": "Before we look at its usage, we first look at diagonal elements,", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0167", "text": "Var\n        \u2061\n        (\n        \n          x\n          \n            j\n          \n        \n        )\n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            P\n          \n        \n        \n          \u03bb\n          \n            k\n          \n        \n        \n          \u03b1\n          \n            k\n            j\n          \n          \n            2", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0168", "text": "2\n          \n        \n      \n    \n    {\\displaystyle \\operatorname {Var} (x_{j})=\\sum _{k=1}^{P}\\lambda _{k}\\alpha _{kj}^{2}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0169", "text": "Then, perhaps the main statistical implication of the result is that not only can we decompose the combined variances of all the elements of x into decreasing contributions due to each PC, but we can also decompose the whole covariance matrix into contributions \n  \n    \n      \n        \n          \u03bb\n          \n            k\n          \n        \n        \n          \u03b1\n          \n            k\n          \n        \n        \n          \u03b1\n          \n            k\n          \n          \u2032", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0170", "text": "\u2032\n        \n      \n    \n    {\\displaystyle \\lambda _{k}\\alpha _{k}\\alpha _{k}'}\n  \n from each PC. Although not strictly decreasing, the elements of \n  \n    \n      \n        \n          \u03bb\n          \n            k\n          \n        \n        \n          \u03b1\n          \n            k\n          \n        \n        \n          \u03b1\n          \n            k\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\lambda _{k}\\alpha _{k}\\alpha _{k}'}\n  \n will tend to become smaller as", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0171", "text": "will tend to become smaller as \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n increases, as \n  \n    \n      \n        \n          \u03bb\n          \n            k\n          \n        \n        \n          \u03b1\n          \n            k\n          \n        \n        \n          \u03b1\n          \n            k\n          \n          \u2032\n        \n      \n    \n    {\\displaystyle \\lambda _{k}\\alpha _{k}\\alpha _{k}'}\n  \n is nonincreasing for increasing \n  \n    \n      \n        k", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0172", "text": "k\n      \n    \n    {\\displaystyle k}\n  \n, whereas the elements of \n  \n    \n      \n        \n          \u03b1\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle \\alpha _{k}}\n  \n tend to stay about the same size because of the normalization constraints: \n  \n    \n      \n        \n          \u03b1\n          \n            k\n          \n          \u2032\n        \n        \n          \u03b1\n          \n            k\n          \n        \n        =\n        1\n        ,\n        k", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0173", "text": "=\n        1\n        ,\n        k\n        =\n        1\n        ,\n        \u2026\n        ,\n        p\n      \n    \n    {\\displaystyle \\alpha _{k}'\\alpha _{k}=1,k=1,\\dots ,p}\n  \n.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0174", "text": "Limitations\nAs noted above, the results of PCA depend on the scaling of the variables. This can be cured by scaling each feature by its standard deviation, so that one ends up with dimensionless features with unital variance.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0175", "text": "The applicability of PCA as described above is limited by certain (tacit) assumptions made in its derivation. In particular, PCA can capture linear correlations between the features but fails when this assumption is violated (see Figure 6a in the reference). In some cases, coordinate transformations can restore the linearity assumption and PCA can then be applied (see kernel PCA).", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0176", "text": "Another limitation is the mean-removal process before constructing the covariance matrix for PCA. In fields such as astronomy, all the signals are non-negative, and the mean-removal process will force the mean of some astrophysical exposures to be zero, which consequently creates unphysical negative fluxes, and forward modeling has to be performed to recover the true magnitude of the signals", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0177", "text": ". As an alternative method, non-negative matrix factorization focusing only on the non-negative elements in the matrices is well-suited for astrophysical observations. See more at the relation between PCA and non-negative matrix factorization.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0178", "text": "PCA is at a disadvantage if the data has not been standardized before applying the algorithm to it. PCA transforms the original data into data that is relevant to the principal components of that data, which means that the new data variables cannot be interpreted in the same ways that the originals were. They are linear interpretations of the original variables. Also, if PCA is not performed properly, there is a high likelihood of information loss.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0179", "text": "PCA relies on a linear model. If a dataset has a pattern hidden inside it that is nonlinear, then PCA can actually steer the analysis in the complete opposite direction of progress. Researchers at Kansas State University discovered that the sampling error in their experiments impacted the bias of PCA results", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0180", "text": ". \"If the number of subjects or blocks is smaller than 30, and/or the researcher is interested in PC's beyond the first, it may be better to first correct for the serial correlation, before PCA is conducted\". The researchers at Kansas State also found that PCA could be \"seriously biased if the autocorrelation structure of the data are not correctly handled\".", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0181", "text": "PCA and information theory\nDimensionality reduction results in a loss of information, in general. PCA-based dimensionality reduction tends to minimize that information loss, under certain signal and noise models.\nUnder the assumption that\n\n  \n    \n      \n        \n          x\n        \n        =\n        \n          s\n        \n        +\n        \n          n\n        \n        ,\n      \n    \n    {\\displaystyle \\mathbf {x} =\\mathbf {s} +\\mathbf {n} ,}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0182", "text": "that is, that the data vector \n  \n    \n      \n        \n          x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n is the sum of the desired information-bearing signal \n  \n    \n      \n        \n          s\n        \n      \n    \n    {\\displaystyle \\mathbf {s} }\n  \n and a noise signal \n  \n    \n      \n        \n          n\n        \n      \n    \n    {\\displaystyle \\mathbf {n} }\n  \n one can show that PCA can be optimal for dimensionality reduction, from an information-theoretic point-of-view.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0183", "text": "In particular, Linsker showed that if \n  \n    \n      \n        \n          s\n        \n      \n    \n    {\\displaystyle \\mathbf {s} }\n  \n is Gaussian and \n  \n    \n      \n        \n          n\n        \n      \n    \n    {\\displaystyle \\mathbf {n} }\n  \n is Gaussian noise with a covariance matrix proportional to the identity matrix, the PCA maximizes the mutual information \n  \n    \n      \n        I\n        (\n        \n          y\n        \n        ;\n        \n          s\n        \n        )", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0184", "text": "s\n        \n        )\n      \n    \n    {\\displaystyle I(\\mathbf {y} ;\\mathbf {s} )}\n  \n between the desired information \n  \n    \n      \n        \n          s\n        \n      \n    \n    {\\displaystyle \\mathbf {s} }\n  \n and the dimensionality-reduced output \n  \n    \n      \n        \n          y\n        \n        =\n        \n          \n            W\n          \n          \n            L\n          \n          \n            T\n          \n        \n        \n          x", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0185", "text": "x\n        \n      \n    \n    {\\displaystyle \\mathbf {y} =\\mathbf {W} _{L}^{T}\\mathbf {x} }\n  \n.\nIf the noise is still Gaussian and has a covariance matrix proportional to the identity matrix (that is, the components of the vector \n  \n    \n      \n        \n          n\n        \n      \n    \n    {\\displaystyle \\mathbf {n} }\n  \n are iid), but the information-bearing signal \n  \n    \n      \n        \n          s\n        \n      \n    \n    {\\displaystyle \\mathbf {s} }", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0186", "text": "{\\displaystyle \\mathbf {s} }\n  \n is non-Gaussian (which is a common scenario), PCA at least minimizes an upper bound on the information loss, which is defined as", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0187", "text": "I\n        (\n        \n          x\n        \n        ;\n        \n          s\n        \n        )\n        \u2212\n        I\n        (\n        \n          y\n        \n        ;\n        \n          s\n        \n        )\n        .\n      \n    \n    {\\displaystyle I(\\mathbf {x} ;\\mathbf {s} )-I(\\mathbf {y} ;\\mathbf {s} ).}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0188", "text": "The optimality of PCA is also preserved if the noise \n  \n    \n      \n        \n          n\n        \n      \n    \n    {\\displaystyle \\mathbf {n} }\n  \n is iid and at least more Gaussian (in terms of the Kullback\u2013Leibler divergence) than the information-bearing signal \n  \n    \n      \n        \n          s\n        \n      \n    \n    {\\displaystyle \\mathbf {s} }\n  \n. In general, even if the above signal model holds, PCA loses its information-theoretic optimality as soon as the noise", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0189", "text": "n\n        \n      \n    \n    {\\displaystyle \\mathbf {n} }\n  \n becomes dependent.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0190", "text": "Computation using the covariance method\nThe following is a detailed description of PCA using the covariance method as opposed to the correlation method.\nThe goal is to transform a given data set X of dimension p to an alternative data set Y of smaller dimension L. Equivalently, we are seeking to find the matrix Y, where Y is the Karhunen\u2013Lo\u00e8ve transform (KLT) of matrix X:", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0191", "text": "Y\n        \n        =\n        \n          K\n          L\n          T\n        \n        {\n        \n          X\n        \n        }\n      \n    \n    {\\displaystyle \\mathbf {Y} =\\mathbb {KLT} \\{\\mathbf {X} \\}}\n  \n\nOrganize the data set", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0192", "text": "Suppose you have data comprising a set of observations of p variables, and you want to reduce the data so that each observation can be described with only L variables, L < p. Suppose further, that the data are arranged as a set of n data vectors \n  \n    \n      \n        \n          \n            x\n          \n          \n            1\n          \n        \n        \u2026\n        \n          \n            x\n          \n          \n            n", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0193", "text": "n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{1}\\ldots \\mathbf {x} _{n}}\n  \n with each \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n representing a single grouped observation of the p variables.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0194", "text": "Write \n  \n    \n      \n        \n          \n            x\n          \n          \n            1\n          \n        \n        \u2026\n        \n          \n            x\n          \n          \n            n\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{1}\\ldots \\mathbf {x} _{n}}\n  \n as row vectors, each with p elements.\nPlace the row vectors into a single matrix X of dimensions n \u00d7 p.\n\nCalculate the empirical mean", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0195", "text": "Find the empirical mean along each column j = 1, ..., p.\nPlace the calculated mean values into an empirical mean vector u of dimensions p \u00d7 1. \n  \n    \n      \n        \n          u\n          \n            j\n          \n        \n        =\n        \n          \n            1\n            n\n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          X\n          \n            i\n            j", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0196", "text": "i\n            j\n          \n        \n      \n    \n    {\\displaystyle u_{j}={\\frac {1}{n}}\\sum _{i=1}^{n}X_{ij}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0197", "text": "Calculate the deviations from the mean\n\nMean subtraction is an integral part of the solution towards finding a principal component basis that minimizes the mean square error of approximating the data. Hence we proceed by centering the data as follows:", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0198", "text": "Subtract the empirical mean vector \n  \n    \n      \n        \n          \n            u\n          \n          \n            T\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {u} ^{T}}\n  \n from each row of the data matrix X.\nStore mean-subtracted data in the n \u00d7 p matrix B. \n  \n    \n      \n        \n          B\n        \n        =\n        \n          X\n        \n        \u2212\n        \n          h\n        \n        \n          \n            u\n          \n          \n            T", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0199", "text": "T\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {B} =\\mathbf {X} -\\mathbf {h} \\mathbf {u} ^{T}}\n  \n where h is an n \u00d7 1 column vector of all 1s: \n  \n    \n      \n        \n          h\n          \n            i\n          \n        \n        =\n        1\n        \n        \n        \n        \n          for \n        \n        i\n        =\n        1\n        ,\n        \u2026\n        ,\n        n\n      \n    \n    {\\displaystyle h_{i}=1\\,\\qquad \\qquad {\\text{for }}i=1,\\ldots ,n}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0200", "text": "In some applications, each variable (column of B) may also be scaled to have a variance equal to 1 (see Z-score). This step affects the calculated principal components, but makes them independent of the units used to measure the different variables.\n\nFind the covariance matrix", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0201", "text": "Find the p \u00d7 p empirical covariance matrix C from matrix B: \n  \n    \n      \n        \n          C\n        \n        =\n        \n          \n            1\n            \n              n\n              \u2212\n              1\n            \n          \n        \n        \n          \n            B\n          \n          \n            \u2217\n          \n        \n        \n          B\n        \n      \n    \n    {\\displaystyle \\mathbf {C} ={1 \\over {n-1}}\\mathbf {B} ^{*}\\mathbf {B} }\n  \n where \n  \n    \n      \n        \u2217", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0202", "text": "where \n  \n    \n      \n        \u2217\n      \n    \n    {\\displaystyle *}\n  \n is the conjugate transpose operator. If B consists entirely of real numbers, which is the case in many applications, the \"conjugate transpose\" is the same as the regular transpose.\nThe reasoning behind using n \u2212 1 instead of n to calculate the covariance is Bessel's correction.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0203", "text": "Find the eigenvectors and eigenvalues of the covariance matrix", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0204", "text": "Compute the matrix V of eigenvectors which diagonalizes the covariance matrix C: \n  \n    \n      \n        \n          \n            V\n          \n          \n            \u2212\n            1\n          \n        \n        \n          C\n        \n        \n          V\n        \n        =\n        \n          D\n        \n      \n    \n    {\\displaystyle \\mathbf {V} ^{-1}\\mathbf {C} \\mathbf {V} =\\mathbf {D} }", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0205", "text": "where D is the diagonal matrix of eigenvalues of C. This step will typically involve the use of a computer-based algorithm for computing eigenvectors and eigenvalues. These algorithms are readily available as sub-components of most matrix algebra systems, such as SAS, R, MATLAB, Mathematica, SciPy, IDL (Interactive Data Language), or GNU Octave as well as OpenCV.\nMatrix D will take the form of an p \u00d7 p diagonal matrix, where \n  \n    \n      \n        \n          D\n          \n            k", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0206", "text": "D\n          \n            k\n            \u2113\n          \n        \n        =\n        \n          \u03bb\n          \n            k\n          \n        \n        \n        \n          for \n        \n        k\n        =\n        \u2113\n      \n    \n    {\\displaystyle D_{k\\ell }=\\lambda _{k}\\qquad {\\text{for }}k=\\ell }\n  \n is the jth eigenvalue of the covariance matrix C, and \n  \n    \n      \n        \n          D\n          \n            k\n            \u2113\n          \n        \n        =\n        0", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0207", "text": "=\n        0\n        \n        \n          for \n        \n        k\n        \u2260\n        \u2113\n        .\n      \n    \n    {\\displaystyle D_{k\\ell }=0\\qquad {\\text{for }}k\\neq \\ell .}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0208", "text": "Matrix V, also of dimension p \u00d7 p, contains p column vectors, each of length p, which represent the p eigenvectors of the covariance matrix C.\nThe eigenvalues and eigenvectors are ordered and paired. The jth eigenvalue corresponds to the jth eigenvector.\nMatrix V denotes the matrix of right eigenvectors (as opposed to left eigenvectors). In general, the matrix of right eigenvectors need not be the (conjugate) transpose of the matrix of left eigenvectors.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0209", "text": "Rearrange the eigenvectors and eigenvalues\n\nSort the columns of the eigenvector matrix V and eigenvalue matrix D in order of decreasing eigenvalue.\nMake sure to maintain the correct pairings between the columns in each matrix.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0210", "text": "Compute the cumulative energy content for each eigenvector\nThe eigenvalues represent the distribution of the source data's energy among each of the eigenvectors, where the eigenvectors form a basis for the data. The cumulative energy content g for the jth eigenvector is the sum of the energy content across all of the eigenvalues from 1 through j divided by the sum of energy content across all eigenvalues (shown in step 8): \n  \n    \n      \n        \n          g\n          \n            j", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0211", "text": "g\n          \n            j\n          \n        \n        =\n        \n          \u2211\n          \n            k\n            =\n            1\n          \n          \n            j\n          \n        \n        \n          D\n          \n            k\n            k\n          \n        \n        \n        \n          for \n        \n        j\n        =\n        1\n        ,\n        \u2026\n        ,\n        p\n      \n    \n    {\\displaystyle g_{j}=\\sum _{k=1}^{j}D_{kk}\\qquad {\\text{for }}j=1,\\dots ,p}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0212", "text": "Select a subset of the eigenvectors as basis vectors", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0213", "text": "Save the first L columns of V as the p \u00d7 L matrix W: \n  \n    \n      \n        \n          W\n          \n            k\n            l\n          \n        \n        =\n        \n          V\n          \n            k\n            \u2113\n          \n        \n        \n        \n          for \n        \n        k\n        =\n        1\n        ,\n        \u2026\n        ,\n        p\n        \n        \u2113\n        =\n        1\n        ,\n        \u2026\n        ,\n        L", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0214", "text": "\u2026\n        ,\n        L\n      \n    \n    {\\displaystyle W_{kl}=V_{k\\ell }\\qquad {\\text{for }}k=1,\\dots ,p\\qquad \\ell =1,\\dots ,L}\n  \n where \n  \n    \n      \n        1\n        \u2264\n        L\n        \u2264\n        p\n        .\n      \n    \n    {\\displaystyle 1\\leq L\\leq p.}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0215", "text": "Use the vector g as a guide in choosing an appropriate value for L. The goal is to choose a value of L as small as possible while achieving a reasonably high value of g on a percentage basis. For example, you may want to choose L so that the cumulative energy g is above a certain threshold, like 90 percent. In this case, choose the smallest value of L such that \n  \n    \n      \n        \n          \n            \n              g\n              \n                L", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0216", "text": "L\n              \n            \n            \n              g\n              \n                p\n              \n            \n          \n        \n        \u2265\n        0.9\n      \n    \n    {\\displaystyle {\\frac {g_{L}}{g_{p}}}\\geq 0.9}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0217", "text": "Project the data onto the new basis\n\nThe projected data points are the rows of the matrix \n  \n    \n      \n        \n          T\n        \n        =\n        \n          B\n        \n        \u22c5\n        \n          W\n        \n      \n    \n    {\\displaystyle \\mathbf {T} =\\mathbf {B} \\cdot \\mathbf {W} }", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0218", "text": "That is, the first column of \n  \n    \n      \n        \n          T\n        \n      \n    \n    {\\displaystyle \\mathbf {T} }\n  \n is the projection of the data points onto the first principal component, the second column is the projection onto the second principal component, etc.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0219", "text": "Derivation using the covariance method\nLet X be a d-dimensional random vector expressed as column vector. Without loss of generality, assume X has zero mean.\nWe want to find \n  \n    \n      \n        (\n        \u2217\n        )\n      \n    \n    {\\displaystyle (\\ast )}\n  \n a d \u00d7 d orthonormal transformation matrix P so that PX has a diagonal covariance matrix (that is, PX is a random vector with all its distinct components pairwise uncorrelated).\nA quick computation assuming \n  \n    \n      \n        P", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0220", "text": "P\n      \n    \n    {\\displaystyle P}\n  \n were unitary yields:", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0221", "text": "cov\n                \u2061\n                (\n                P\n                X\n                )\n              \n              \n                \n                =\n                E\n                \u2061\n                [\n                P\n                X\n                 \n                (\n                P\n                X\n                \n                  )\n                  \n                    \u2217", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0222", "text": "\u2217\n                  \n                \n                ]\n              \n            \n            \n              \n              \n                \n                =\n                E\n                \u2061\n                [\n                P\n                X\n                 \n                \n                  X\n                  \n                    \u2217\n                  \n                \n                \n                  P\n                  \n                    \u2217", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0223", "text": "\u2217\n                  \n                \n                ]\n              \n            \n            \n              \n              \n                \n                =\n                P\n                E\n                \u2061\n                [\n                X\n                \n                  X\n                  \n                    \u2217\n                  \n                \n                ]\n                \n                  P", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0224", "text": "P\n                  \n                    \u2217\n                  \n                \n              \n            \n            \n              \n              \n                \n                =\n                P\n                cov\n                \u2061\n                (\n                X\n                )\n                \n                  P\n                  \n                    \u2212\n                    1", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0225", "text": "{\\displaystyle {\\begin{aligned}\\operatorname {cov} (PX)&=\\operatorname {E} [PX~(PX)^{*}]\\\\&=\\operatorname {E} [PX~X^{*}P^{*}]\\\\&=P\\operatorname {E} [XX^{*}]P^{*}\\\\&=P\\operatorname {cov} (X)P^{-1}\\\\\\end{aligned}}}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0226", "text": "Hence \n  \n    \n      \n        (\n        \u2217\n        )\n      \n    \n    {\\displaystyle (\\ast )}\n  \n holds if and only if \n  \n    \n      \n        cov\n        \u2061\n        (\n        X\n        )\n      \n    \n    {\\displaystyle \\operatorname {cov} (X)}\n  \n were diagonalisable by \n  \n    \n      \n        P\n      \n    \n    {\\displaystyle P}\n  \n.\nThis is very constructive, as cov(X) is guaranteed to be a non-negative definite matrix and thus is guaranteed to be diagonalisable by some unitary matrix.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0227", "text": "Covariance-free computation", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0228", "text": "In practical implementations, especially with high dimensional data (large p), the naive covariance method is rarely used because it is not efficient due to high computational and memory costs of explicitly determining the covariance matrix", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0229", "text": ". The covariance-free approach avoids the np2 operations of explicitly calculating and storing the covariance matrix XTX, instead utilizing one of matrix-free methods, for example, based on the function evaluating the product XT(X r) at the cost of 2np operations.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0230", "text": "Iterative computation\nOne way to compute the first principal component efficiently is shown in the following pseudo-code, for a data matrix X with zero mean, without ever computing its covariance matrix.\n\nr = a random vector of length p\nr = r / norm(r)\ndo c times:\n      s = 0 (a vector of length p)\n      for each row x in X\n            s = s + (x \u22c5 r) x\n      \u03bb = rTs // \u03bb is the eigenvalue\n      error = |\u03bb \u22c5 r \u2212 s|\n      r = s / norm(s)\n      exit if error < tolerance\nreturn \u03bb, r", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0231", "text": "This power iteration algorithm simply calculates the vector XT(X r), normalizes, and places the result back in r. The eigenvalue is approximated by rT (XTX) r, which is the Rayleigh quotient on the unit vector r for the covariance matrix XTX . If the largest singular value is well separated from the next largest one, the vector r gets close to the first principal component of X within the number of iterations c, which is small relative to p, at the total cost 2cnp", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0232", "text": ". The power iteration convergence can be accelerated without noticeably sacrificing the small cost per iteration using more advanced matrix-free methods, such as the Lanczos algorithm or the Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) method.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0233", "text": "Subsequent principal components can be computed one-by-one via deflation or simultaneously as a block. In the former approach, imprecisions in already computed approximate principal components additively affect the accuracy of the subsequently computed principal components, thus increasing the error with every new computation. The latter approach in the block power method replaces single-vectors r and s with block-vectors, matrices R and S", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0234", "text": ". Every column of R approximates one of the leading principal components, while all columns are iterated simultaneously. The main calculation is evaluation of the product XT(X R). Implemented, for example, in LOBPCG, efficient blocking eliminates the accumulation of the errors, allows using high-level BLAS matrix-matrix product functions, and typically leads to faster convergence, compared to the single-vector one-by-one technique.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0235", "text": "The NIPALS method", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0236", "text": "Non-linear iterative partial least squares (NIPALS) is a variant the classical power iteration with matrix deflation by subtraction implemented for computing the first few components in a principal component or partial least squares analysis. For very-high-dimensional datasets, such as those generated in the *omics sciences (for example, genomics, metabolomics) it is usually only necessary to compute the first few PCs", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0237", "text": ". The non-linear iterative partial least squares (NIPALS) algorithm updates iterative approximations to the leading scores and loadings t1 and r1T by the power iteration multiplying on every iteration by X on the left and on the right, that is, calculation of the covariance matrix is avoided, just as in the matrix-free implementation of the power iterations to XTX, based on the function evaluating the product XT(X r) = ((X r)TX)T.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0238", "text": "The matrix deflation by subtraction is performed by subtracting the outer product, t1r1T from X leaving the deflated residual matrix used to calculate the subsequent leading PCs.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0239", "text": "For large data matrices, or matrices that have a high degree of column collinearity, NIPALS suffers from loss of orthogonality of PCs due to machine precision round-off errors accumulated in each iteration and matrix deflation by subtraction. A Gram\u2013Schmidt re-orthogonalization algorithm is applied to both the scores and the loadings at each iteration step to eliminate this loss of orthogonality", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0240", "text": ". NIPALS reliance on single-vector multiplications cannot take advantage of high-level BLAS and results in slow convergence for clustered leading singular values\u2014both these deficiencies are resolved in more sophisticated matrix-free block solvers, such as the Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) method.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0241", "text": "Online/sequential estimation\nIn an \"online\" or \"streaming\" situation with data arriving piece by piece rather than being stored in a single batch, it is useful to make an estimate of the PCA projection that can be updated sequentially. This can be done efficiently, but requires different algorithms.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0242", "text": "Qualitative variables\nIn exploratory data analysis, qualitative (categorical) variables not used to construct the model can be projected onto the principal component axes as supplementary elements. The use of supplementary qualitative variables was a defining feature of the French school of data analysis developed in the 1960s and 70s. Based upon the work of Ludovic Lebart, the method is available in the R environment through packages such as FactoMineR.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0243", "text": "Applications\nIntelligence", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0244", "text": "The earliest application of factor analysis was in locating and measuring components of human intelligence. It was believed that intelligence had various uncorrelated components such as spatial intelligence, verbal intelligence, induction, deduction etc and that scores on these could be adduced by factor analysis from results on various tests, to give a single index known as the Intelligence Quotient (IQ)", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0245", "text": ". The pioneering statistical psychologist Spearman actually developed factor analysis in 1904 for his two-factor theory of intelligence, adding a formal technique to the science of psychometrics. In 1924 Thurstone looked for 56 factors of intelligence, developing the notion of Mental Age. Standard IQ tests today are based on this early work.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0246", "text": "Residential differentiation", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0247", "text": "In 1949, Shevky and Williams introduced the theory of factorial ecology, which dominated studies of residential differentiation from the 1950s to the 1970s. Neighbourhoods in a city were recognizable or could be distinguished from one another by various characteristics which could be reduced to three by factor analysis", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0248", "text": ". These were known as 'social rank' (an index of occupational status), 'familism' or family size, and 'ethnicity'; Cluster analysis could then be applied to divide the city into clusters or precincts according to values of the three key factor variables. An extensive literature developed around factorial ecology in urban geography, but the approach went out of fashion after 1980 as being methodologically primitive and having little place in postmodern geographical paradigms.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0249", "text": "One of the problems with factor analysis has always been finding convincing names for the various artificial factors. In 2000, Flood revived the factorial ecology approach to show that principal components analysis actually gave meaningful answers directly, without resorting to factor rotation. The principal components were actually dual variables or shadow prices of 'forces' pushing people together or apart in cities", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0250", "text": ". The first component was 'accessibility', the classic trade-off between demand for travel and demand for space, around which classical urban economics is based. The next two components were 'disadvantage', which keeps people of similar status in separate neighbourhoods (mediated by planning), and ethnicity, where people of similar ethnic backgrounds try to co-locate.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0251", "text": "About the same time, the Australian Bureau of Statistics defined distinct indexes of advantage and disadvantage taking the first principal component of sets of key variables that were thought to be important. These SEIFA indexes are regularly published for various jurisdictions, and are used frequently in spatial analysis.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0252", "text": "Development indexes\nPCA can be used as a formal method for the development of indexes. As an alternative confirmatory composite analysis has been proposed to develop and assess indexes.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0253", "text": "The City Development Index was developed by PCA from about 200 indicators of city outcomes in a 1996 survey of 254 global cities. The first principal component was subject to iterative regression, adding the original variables singly until about 90% of its variation was accounted for. The index ultimately used about 15 indicators but was a good predictor of many more variables. Its comparative value agreed very well with a subjective assessment of the condition of each city", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0254", "text": ". The coefficients on items of infrastructure were roughly proportional to the average costs of providing the underlying services, suggesting the Index was actually a measure of effective physical and social investment in the city.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0255", "text": "The country-level Human Development Index (HDI) from UNDP, which has been published since 1990 and is very extensively used in development studies, has very similar coefficients on similar indicators, strongly suggesting it was originally constructed using PCA.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0256", "text": "Population genetics\nIn 1978 Cavalli-Sforza and others pioneered the use of principal components analysis (PCA) to summarise data on variation in human gene frequencies across regions. The components showed distinctive patterns, including gradients and sinusoidal waves. They interpreted these patterns as resulting from specific ancient migration events.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0257", "text": "Since then, PCA has been ubiquitous in population genetics, with thousands of papers using PCA as a display mechanism. Genetics varies largely according to proximity, so the first two principal components actually show spatial distribution and may be used to map the relative geographical location of different population groups, thereby showing individuals who have wandered from their original locations.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0258", "text": "PCA in genetics has been technically controversial, in that the technique has been performed on discrete non-normal variables and often on binary allele markers. The lack of any measures of standard error in PCA are also an impediment to more consistent usage. In August 2022, the molecular biologist Eran Elhaik published a theoretical paper in Scientific Reports analyzing 12 PCA applications", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0259", "text": ". He concluded that it was easy to manipulate the method, which, in his view, generated results that were 'erroneous, contradictory, and absurd.' Specifically, he argued, the results achieved in population genetics were characterized by cherry-picking and circular reasoning.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0260", "text": "Market research and indexes of attitude\nMarket research has been an extensive user of PCA. It is used to develop customer satisfaction or customer loyalty scores for products, and with clustering, to develop market segments that may be targeted with advertising campaigns, in much the same way as factorial ecology will locate geographical areas with similar characteristics.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0261", "text": "PCA rapidly transforms large amounts of data into smaller, easier-to-digest variables that can be more rapidly and readily analyzed. In any consumer questionnaire, there are series of questions designed to elicit consumer attitudes, and principal components seek out latent variables underlying these attitudes", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0262", "text": ". For example, the Oxford Internet Survey in 2013 asked 2000 people about their attitudes and beliefs, and from these analysts extracted four principal component dimensions, which they identified as 'escape', 'social networking', 'efficiency', and 'problem creating'.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0263", "text": "Another example from Joe Flood in 2008 extracted an attitudinal index toward housing from 28 attitude questions in a national survey of 2697 households in Australia. The first principal component represented a general attitude toward property and home ownership. The index, or the attitude questions it embodied, could be fed into a General Linear Model of tenure choice", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0264", "text": ". The strongest determinant of private renting by far was the attitude index, rather than income, marital status or household type.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0265", "text": "Quantitative finance\nIn quantitative finance, PCA is used\nin financial risk management, and has been applied to other problems such as portfolio optimization.\nPCA is commonly used in problems involving fixed income securities and portfolios, and interest rate derivatives. \nValuations here depend on the entire yield curve, comprising numerous highly correlated instruments, and PCA is used to define a set of components or factors that explain rate movements,\nthereby facilitating the modelling.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0266", "text": "thereby facilitating the modelling.\nOne common risk management application is to calculating value at risk, VaR, applying PCA to the Monte Carlo simulation.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0267", "text": "Here, for each simulation-sample, the components are stressed, and rates, and in turn option values, are then reconstructed; \nwith VaR calculated, finally, over the entire run. \nPCA is also used in hedging exposure to interest rate risk, given partial durations and other sensitivities.\nUnder both, the first three, typically, principal components of the system are of interest (representing \"shift\", \"twist\", and \"curvature\").", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0268", "text": "These principal components are derived from an eigen-decomposition of the covariance matrix of yield at predefined maturities;\nand where the variance of each component is its eigenvalue (and as the components are orthogonal, no correlation need be incorporated in subsequent modelling).\nFor equity, an optimal portfolio is one where the expected return is maximized for a given level of risk, or alternatively, where risk is minimized for a given return; see Markowitz model for discussion.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0269", "text": "Thus, one approach is to reduce portfolio risk, where allocation strategies are applied to the \"principal portfolios\" instead of the underlying stocks.\nA second approach is to enhance portfolio return, using the principal components to select companies' stocks with upside potential.\nPCA has also been used to understand relationships between international equity markets, and within markets between groups of companies in industries or sectors.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0270", "text": "PCA may also be applied to stress testing, essentially an analysis of a bank's ability to endure a hypothetical adverse economic scenario. Its utility is in \"distilling the information contained in [several] macroeconomic variables into a more manageable data set, which can then [be used] for analysis.\" Here, the resulting factors are linked to e.g", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0271", "text": ". interest rates \u2013 based on the largest elements of the factor's eigenvector \u2013 and it is then observed how a \"shock\" to each of the factors affects the implied assets of each of the banks.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0272", "text": "Neuroscience", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0273", "text": "A variant of principal components analysis is used in neuroscience to identify the specific properties of a stimulus that increases a neuron's probability of generating an action potential. This technique is known as spike-triggered covariance analysis", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0274", "text": ". In a typical application an experimenter presents a white noise process as a stimulus (usually either as a sensory input to a test subject, or as a current injected directly into the neuron) and records a train of action potentials, or spikes, produced by the neuron as a result. Presumably, certain features of the stimulus make the neuron more likely to spike", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0275", "text": ". In order to extract these features, the experimenter calculates the covariance matrix of the spike-triggered ensemble, the set of all stimuli (defined and discretized over a finite time window, typically on the order of 100 ms) that immediately preceded a spike", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0276", "text": ". The eigenvectors of the difference between the spike-triggered covariance matrix and the covariance matrix of the prior stimulus ensemble (the set of all stimuli, defined over the same length time window) then indicate the directions in the space of stimuli along which the variance of the spike-triggered ensemble differed the most from that of the prior stimulus ensemble", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0277", "text": ". Specifically, the eigenvectors with the largest positive eigenvalues correspond to the directions along which the variance of the spike-triggered ensemble showed the largest positive change compared to the variance of the prior. Since these were the directions in which varying the stimulus led to a spike, they are often good approximations of the sought after relevant stimulus features.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0278", "text": "In neuroscience, PCA is also used to discern the identity of a neuron from the shape of its action potential. Spike sorting is an important procedure because extracellular recording techniques often pick up signals from more than one neuron. In spike sorting, one first uses PCA to reduce the dimensionality of the space of action potential waveforms, and then performs clustering analysis to associate specific action potentials with individual neurons.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0279", "text": "PCA as a dimension reduction technique is particularly suited to detect coordinated activities of large neuronal ensembles. It has been used in determining collective variables, that is, order parameters, during phase transitions in the brain.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0280", "text": "Relation with other methods\nCorrespondence analysis\nCorrespondence analysis (CA)\nwas developed by Jean-Paul Benz\u00e9cri\nand is conceptually similar to PCA, but scales the data (which should be non-negative) so that rows and columns are treated equivalently. It is traditionally applied to contingency tables.\nCA decomposes the chi-squared statistic associated to this table into orthogonal factors.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0281", "text": "Because CA is a descriptive technique, it can be applied to tables for which the chi-squared statistic is appropriate or not.\nSeveral variants of CA are available including detrended correspondence analysis and canonical correspondence analysis. One special extension is multiple correspondence analysis, which may be seen as the counterpart of principal component analysis for categorical data.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0282", "text": "Factor analysis", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0283", "text": "Principal component analysis creates variables that are linear combinations of the original variables. The new variables have the property that the variables are all orthogonal. The PCA transformation can be helpful as a pre-processing step before clustering. PCA is a variance-focused approach seeking to reproduce the total variable variance, in which components reflect both common and unique variance of the variable", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0284", "text": ". PCA is generally preferred for purposes of data reduction (that is, translating variable space into optimal factor space) but not when the goal is to detect the latent construct or factors.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0285", "text": "Factor analysis is similar to principal component analysis, in that factor analysis also involves linear combinations of variables. Different from PCA, factor analysis is a correlation-focused approach seeking to reproduce the inter-correlations among variables, in which the factors \"represent the common variance of variables, excluding unique variance\"", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0286", "text": ". In terms of the correlation matrix, this corresponds with focusing on explaining the off-diagonal terms (that is, shared co-variance), while PCA focuses on explaining the terms that sit on the diagonal. However, as a side result, when trying to reproduce the on-diagonal terms, PCA also tends to fit relatively well the off-diagonal correlations", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0287", "text": ". Results given by PCA and factor analysis are very similar in most situations, but this is not always the case, and there are some problems where the results are significantly different. Factor analysis is generally used when the research purpose is detecting data structure (that is, latent constructs or factors) or causal modeling. If the factor model is incorrectly formulated or the assumptions are not met, then factor analysis will give erroneous results.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0288", "text": "K-means clustering\nIt has been asserted that the relaxed solution of k-means clustering, specified by the cluster indicators, is given by the principal components, and the PCA subspace spanned by the principal directions is identical to the cluster centroid subspace. However, that PCA is a useful relaxation of k-means clustering was not a new result, and it is straightforward to uncover counterexamples to the statement that the cluster centroid subspace is spanned by the principal directions.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0289", "text": "Non-negative matrix factorization\nNon-negative matrix factorization (NMF) is a dimension reduction method where only non-negative elements in the matrices are used, which is therefore a promising method in astronomy, in the sense that astrophysical signals are non-negative. The PCA components are orthogonal to each other, while the NMF components are all non-negative and therefore constructs a non-orthogonal basis.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0290", "text": "In PCA, the contribution of each component is ranked based on the magnitude of its corresponding eigenvalue, which is equivalent to the fractional residual variance (FRV) in analyzing empirical data. For NMF, its components are ranked based only on the empirical FRV curves. The residual fractional eigenvalue plots, that is, \n  \n    \n      \n        1\n        \u2212\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            k", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0291", "text": "k\n          \n        \n        \n          \u03bb\n          \n            i\n          \n        \n        \n          \n            /\n          \n        \n        \n          \u2211\n          \n            j\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          \u03bb\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle 1-\\sum _{i=1}^{k}\\lambda _{i}{\\Big /}\\sum _{j=1}^{n}\\lambda _{j}}\n  \n as a function of component number", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0292", "text": "as a function of component number \n  \n    \n      \n        k\n      \n    \n    {\\displaystyle k}\n  \n given a total of \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0293", "text": "n\n      \n    \n    {\\displaystyle n}\n  \n components, for PCA have a flat plateau, where no data are captured to remove the quasi-static noise, then the curves drop quickly as an indication of over-fitting (random noise). The FRV curves for NMF is decreasing continuously when the NMF components are constructed sequentially, indicating the continuous capturing of quasi-static noise; then converge to higher levels than PCA, indicating the less over-fitting property of NMF.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0294", "text": "Iconography of correlations", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0295", "text": "It is often difficult to interpret the principal components when the data include many variables of various origins, or when some variables are qualitative. This leads the PCA user to a delicate elimination of several variables. If observations or variables have an excessive impact on the direction of the axes, they should be removed and then projected as supplementary elements", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0296", "text": ". In addition, it is necessary to avoid interpreting the proximities between the points close to the center of the factorial plane.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0297", "text": "The iconography of correlations, on the contrary, which is not a projection on a system of axes, does not have these drawbacks. We can therefore keep all the variables.\nThe principle of the diagram is to underline the \"remarkable\" correlations of the correlation matrix, by a solid line (positive correlation) or dotted line (negative correlation).", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0298", "text": "A strong correlation is not \"remarkable\" if it is not direct, but caused by the effect of a third variable. Conversely, weak correlations can be \"remarkable\". For example, if a variable Y depends on several independent variables, the correlations of Y with each of them are weak and yet \"remarkable\".", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0299", "text": "Generalizations\nSparse PCA\nA particular disadvantage of PCA is that the principal components are usually linear combinations of all input variables. Sparse PCA overcomes this disadvantage by finding linear combinations that contain just a few input variables. It extends the classic method of principal component analysis (PCA) for the reduction of dimensionality of data by adding sparsity constraint on the input variables.\nSeveral approaches have been proposed, including", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0300", "text": "a regression framework,\na convex relaxation/semidefinite programming framework,\na generalized power method framework\nan alternating maximization framework\nforward-backward greedy search and exact methods using branch-and-bound techniques,\nBayesian formulation framework.\nThe methodological and theoretical developments of Sparse PCA as well as its applications in scientific studies were recently reviewed in a survey paper.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0301", "text": "Nonlinear PCA", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0302", "text": "Most of the modern methods for nonlinear dimensionality reduction find their theoretical and algorithmic roots in PCA or K-means. Pearson's original idea was to take a straight line (or plane) which will be \"the best fit\" to a set of data points. Trevor Hastie expanded on this concept by proposing Principal curves as the natural extension for the geometric interpretation of PCA, which explicitly constructs a manifold for data approximation followed by projecting the points onto it", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0303", "text": ". See also the elastic map algorithm and principal geodesic analysis. Another popular generalization is kernel PCA, which corresponds to PCA performed in a reproducing kernel Hilbert space associated with a positive definite kernel.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0304", "text": "In multilinear subspace learning, PCA is generalized to multilinear PCA (MPCA) that extracts features directly from tensor representations. MPCA is solved by performing PCA in each mode of the tensor iteratively. MPCA has been applied to face recognition, gait recognition, etc. MPCA is further extended to uncorrelated MPCA, non-negative MPCA and robust MPCA.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0305", "text": "N-way principal component analysis may be performed with models such as Tucker decomposition, PARAFAC, multiple factor analysis, co-inertia analysis, STATIS, and DISTATIS.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0306", "text": "Robust PCA\nWhile PCA finds the mathematically optimal method (as in minimizing the squared error), it is still sensitive to outliers in the data that produce large errors, something that the method tries to avoid in the first place. It is therefore common practice to remove outliers before computing PCA. However, in some contexts, outliers can be difficult to identify.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0307", "text": "For example, in data mining algorithms like correlation clustering, the assignment of points to clusters and outliers is not known beforehand.\nA recently proposed generalization of PCA based on a weighted PCA increases robustness by assigning different weights to data objects based on their estimated relevancy.\nOutlier-resistant variants of PCA have also been proposed, based on L1-norm formulations (L1-PCA).", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0308", "text": "Robust principal component analysis (RPCA) via decomposition in low-rank and sparse matrices is a modification of PCA that works well with respect to grossly corrupted observations.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0309", "text": "Similar techniques\nIndependent component analysis\nIndependent component analysis (ICA) is directed to similar problems as principal component analysis, but finds additively separable components rather than successive approximations.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0310", "text": "Network component analysis\nGiven a matrix \n  \n    \n      \n        E\n      \n    \n    {\\displaystyle E}\n  \n, it tries to decompose it into two matrices such that \n  \n    \n      \n        E\n        =\n        A\n        P\n      \n    \n    {\\displaystyle E=AP}\n  \n. A key difference from techniques such as PCA and ICA is that some of the entries of \n  \n    \n      \n        A\n      \n    \n    {\\displaystyle A}\n  \n are constrained to be 0. Here \n  \n    \n      \n        P\n      \n    \n    {\\displaystyle P}", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0311", "text": "P\n      \n    \n    {\\displaystyle P}\n  \n is termed the regulatory layer. While in general such a decomposition can have multiple solutions, they prove that if the following conditions are satisfied :", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0312", "text": "A\n      \n    \n    {\\displaystyle A}\n  \n has full column rank\nEach column of \n  \n    \n      \n        A\n      \n    \n    {\\displaystyle A}\n  \n must have at least \n  \n    \n      \n        L\n        \u2212\n        1\n      \n    \n    {\\displaystyle L-1}\n  \n zeroes where \n  \n    \n      \n        L\n      \n    \n    {\\displaystyle L}\n  \n is the number of columns of \n  \n    \n      \n        A\n      \n    \n    {\\displaystyle A}\n  \n (or alternatively the number of rows of", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0313", "text": "P\n      \n    \n    {\\displaystyle P}\n  \n). The justification for this criterion is that if a node is removed from the regulatory layer along with all the output nodes connected to it, the result must still be characterized by a connectivity matrix with full column rank.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0314", "text": "P\n      \n    \n    {\\displaystyle P}\n  \n must have full row rank.\nthen the decomposition is unique up to multiplication by a scalar.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0315", "text": "Discriminant analysis of principal components", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0316", "text": "Discriminant analysis of principal components (DAPC) is a multivariate method used to identify and describe clusters of genetically related individuals. Genetic variation is partitioned into two components: variation between groups and within groups, and it maximizes the former. Linear discriminants are linear combinations of alleles which best separate the clusters. Alleles that most contribute to this discrimination are therefore those that are the most markedly different across groups", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0317", "text": ". The contributions of alleles to the groupings identified by DAPC can allow identifying regions of the genome driving the genetic divergence among groups", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0318", "text": "In DAPC, data are first transformed using a principal components analysis (PCA) and subsequently clusters are identified using discriminant analysis (DA).\nA DAPC can be realized on R using the package Adegenet. (more info: adegenet on the web)", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0319", "text": "Directional component analysis\nDirectional component analysis (DCA) is a method used in the atmospheric sciences for analysing multivariate datasets.\nLike PCA, it allows for dimension reduction, improved visualization and improved interpretability of large data-sets.\nAlso like PCA, it is based on a covariance matrix derived from the input dataset.\nThe difference between PCA and DCA is that DCA additionally requires the input of a vector direction, referred to as the impact.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0320", "text": "Whereas PCA maximises explained variance, DCA maximises probability density given impact.\nThe motivation for DCA is to find components of a multivariate dataset that are both likely (measured using probability density) and important (measured using the impact).\nDCA has been used to find the most likely and most serious heat-wave patterns in weather prediction ensembles\n, and the most likely and most impactful changes in rainfall due to climate change\n.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0321", "text": "Software/source code\nALGLIB \u2013 a C++ and C# library that implements PCA and truncated PCA\nAnalytica \u2013 The built-in EigenDecomp function computes principal components.\nELKI \u2013 includes PCA for projection, including robust variants of PCA, as well as PCA-based clustering algorithms.\nGretl \u2013 principal component analysis can be performed either via the pca command or via the princomp() function.\nJulia \u2013 Supports PCA with the pca function in the MultivariateStats package.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0322", "text": "KNIME \u2013 A java based nodal arranging software for Analysis, in this the nodes called PCA, PCA compute, PCA Apply, PCA inverse make it easily.\nMaple (software) \u2013 The PCA command is used to perform a principal component analysis on a set of data.\nMathematica \u2013 Implements principal component analysis with the PrincipalComponents command using both covariance and correlation methods.\nMathPHP \u2013 PHP mathematics library with support for PCA.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0323", "text": "MATLAB \u2013 The SVD function is part of the basic system. In the Statistics Toolbox, the functions princomp and pca (R2012b) give the principal components, while the function pcares gives the residuals and reconstructed matrix for a low-rank PCA approximation.\nMatplotlib \u2013 a Python library that has a PCA package in the .mlab module.\nmlpack \u2013 Provides an implementation of principal component analysis in C++.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0324", "text": "mrmath \u2013 A high performance math library for Delphi and FreePascal can perform PCA; including robust variants.\nNAG Library \u2013 Principal components analysis is implemented via the g03aa routine (available in both the Fortran versions of the Library).\nNMath \u2013 Proprietary numerical library containing PCA for the .NET Framework.\nGNU Octave \u2013 Free software computational environment mostly compatible with MATLAB, the function princomp gives the principal component.\nOpenCV", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0325", "text": "OpenCV\nOracle Database 12c \u2013 Implemented via DBMS_DATA_MINING.SVDS_SCORING_MODE by specifying setting value SVDS_SCORING_PCA.\nOrange (software) \u2013 Integrates PCA in its visual programming environment. PCA displays a scree plot (degree of explained variance) where user can interactively select the number of principal components.\nOrigin \u2013 Contains PCA in its Pro version.\nQlucore \u2013 Commercial software for analyzing multivariate data with instant response using PCA.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0326", "text": "R \u2013 Free statistical package, the functions princomp and prcomp can be used for principal component analysis; prcomp uses singular value decomposition which generally gives better numerical accuracy. Some packages that implement PCA in R, include, but are not limited to: ade4, vegan, ExPosition, dimRed, and FactoMineR.\nSAS \u2013 Proprietary software; for example, see.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0327", "text": "SAS \u2013 Proprietary software; for example, see.\nscikit-learn \u2013 Python library for machine learning which contains PCA, Probabilistic PCA, Kernel PCA, Sparse PCA and other techniques in the decomposition module.\nScilab \u2013 Free and open-source, cross-platform numerical computational package, the function princomp computes principal component analysis, the function pca computes principal component analysis with standardized variables.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0328", "text": "SPSS \u2013 Proprietary software most commonly used by social scientists for PCA, factor analysis and associated cluster analysis.\nWeka \u2013 Java library for machine learning which contains modules for computing principal components.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0329", "text": "See also\nReferences\nFurther reading\nJackson, J.E. (1991). A User's Guide to Principal Components (Wiley).\nJolliffe, I. T. (1986). Principal Component Analysis. Springer Series in Statistics. Springer-Verlag. pp. 487. CiteSeerX 10.1.1.149.8828. doi:10.1007/b98835. ISBN 978-0-387-95442-4.\nJolliffe, I. T. (2002). Principal Component Analysis. Springer Series in Statistics. New York: Springer-Verlag. doi:10.1007/b98835. ISBN 978-0-387-95442-4.", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0330", "text": "Husson Fran\u00e7ois, L\u00ea S\u00e9bastien & Pag\u00e8s J\u00e9r\u00f4me (2009). Exploratory Multivariate Analysis by Example Using R. Chapman & Hall/CRC The R Series, London. 224p. ISBN 978-2-7535-0938-2\nPag\u00e8s J\u00e9r\u00f4me (2014). Multiple Factor Analysis by Example Using R. Chapman & Hall/CRC The R Series London 272 p", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0331", "text": "External links", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Principal_component_analysis_0332", "text": "University of Copenhagen video by Rasmus Bro on YouTube\nStanford University video by Andrew Ng on YouTube\nA Tutorial on Principal Component Analysis\nA layman's introduction to principal component analysis on YouTube (a video of less than 100 seconds.)\nStatQuest: StatQuest: Principal Component Analysis (PCA), Step-by-Step on YouTube\nLayman's explanation in making sense of principal component analysis, eigenvectors & eigenvalues on Stack Overflow\nSee also the list of Software implementations", "title": "Principal component analysis", "url": "https://en.wikipedia.org/wiki/Principal_component_analysis"}, {"id": "rec_Dimensionality_reduction_0000", "text": "Dimensionality reduction, or dimension reduction, is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its intrinsic dimension. Working in high-dimensional spaces can be undesirable for many reasons; raw data are often sparse as a consequence of the curse of dimensionality, and analyzing the data is usually computationally intractable", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0001", "text": ". Dimensionality reduction is common in fields that deal with large numbers of observations and/or large numbers of variables, such as signal processing, speech recognition, neuroinformatics, and bioinformatics.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0002", "text": "Methods are commonly divided into linear and nonlinear approaches. Linear approaches can be further divided into feature selection and feature extraction. Dimensionality reduction can be used for noise reduction, data visualization, cluster analysis, or as an intermediate step to facilitate other analyses.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0003", "text": "Feature selection\nThe process of feature selection aims to find a suitable subset of the input variables (features, or attributes) for the task at hand. The three strategies are: the filter strategy (e.g., information gain), the wrapper strategy (e.g., accuracy-guided search), and the embedded strategy (features are added or removed while building the model based on prediction errors).", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0004", "text": "Data analysis such as regression or classification can be done in the reduced space more accurately than in the original space.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0005", "text": "Feature projection\nFeature projection (also called feature extraction) transforms the data from the high-dimensional space to a space of fewer dimensions. The data transformation may be linear, as in principal component analysis (PCA), but many nonlinear dimensionality reduction techniques also exist. For multidimensional data, tensor representation can be used in dimensionality reduction through multilinear subspace learning.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0006", "text": "Principal component analysis (PCA)", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0007", "text": "The main linear technique for dimensionality reduction, principal component analysis, performs a linear mapping of the data to a lower-dimensional space in such a way that the variance of the data in the low-dimensional representation is maximized. In practice, the covariance (and sometimes the correlation) matrix of the data is constructed and the eigenvectors on this matrix are computed", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0008", "text": ". The eigenvectors that correspond to the largest eigenvalues (the principal components) can now be used to reconstruct a large fraction of the variance of the original data. Moreover, the first few eigenvectors can often be interpreted in terms of the large-scale physical behavior of the system, because they often contribute the vast majority of the system's energy, especially in low-dimensional systems. Still, this must be proved on a case-by-case basis as not all systems exhibit this behavior", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0009", "text": ". The original space (with dimension of the number of points) has been reduced (with data loss, but hopefully retaining the most important variance) to the space spanned by a few eigenvectors.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0010", "text": "Non-negative matrix factorization (NMF)", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0011", "text": "NMF decomposes a non-negative matrix to the product of two non-negative ones, which has been a promising tool in fields where only non-negative signals exist, such as astronomy", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0012", "text": ". NMF is well known since the multiplicative update rule by Lee & Seung, which has been continuously developed: the inclusion of uncertainties, the consideration of missing data and parallel computation, sequential construction which leads to the stability and linearity of NMF, as well as other updates including handling missing data in digital image processing.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0013", "text": "With a stable component basis during construction, and a linear modeling process, sequential NMF is able to preserve the flux in direct imaging of circumstellar structures in astronomy, as one of the methods of detecting exoplanets, especially for the direct imaging of circumstellar discs. In comparison with PCA, NMF does not remove the mean of the matrices, which leads to physical non-negative fluxes; therefore NMF is able to preserve more information than PCA as demonstrated by Ren et al.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0014", "text": "Kernel PCA\nPrincipal component analysis can be employed in a nonlinear way by means of the kernel trick. The resulting technique is capable of constructing nonlinear mappings that maximize the variance in the data. The resulting technique is called kernel PCA.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0015", "text": "Graph-based kernel PCA\nOther prominent nonlinear techniques include manifold learning techniques such as Isomap, locally linear embedding (LLE), Hessian LLE, Laplacian eigenmaps, and methods based on tangent space analysis. These techniques construct a low-dimensional data representation using a cost function that retains local properties of the data, and can be viewed as defining a graph-based kernel for Kernel PCA.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0016", "text": "More recently, techniques have been proposed that, instead of defining a fixed kernel, try to learn the kernel using semidefinite programming. The most prominent example of such a technique is maximum variance unfolding (MVU). The central idea of MVU is to exactly preserve all pairwise distances between nearest neighbors (in the inner product space) while maximizing the distances between points that are not nearest neighbors.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0017", "text": "An alternative approach to neighborhood preservation is through the minimization of a cost function that measures differences between distances in the input and output spaces", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0018", "text": ". Important examples of such techniques include: classical multidimensional scaling, which is identical to PCA; Isomap, which uses geodesic distances in the data space; diffusion maps, which use diffusion distances in the data space; t-distributed stochastic neighbor embedding (t-SNE), which minimizes the divergence between distributions over pairs of points; and curvilinear component analysis.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0019", "text": "A different approach to nonlinear dimensionality reduction is through the use of autoencoders, a special kind of feedforward neural networks with a bottleneck hidden layer. The training of deep encoders is typically performed using a greedy layer-wise pre-training (e.g., using a stack of restricted Boltzmann machines) that is followed by a finetuning stage based on backpropagation.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0020", "text": "Linear discriminant analysis (LDA)\nLinear discriminant analysis (LDA) is a generalization of Fisher's linear discriminant, a method used in statistics, pattern recognition, and machine learning to find a linear combination of features that characterizes or separates two or more classes of objects or events.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0021", "text": "Generalized discriminant analysis (GDA)\nGDA deals with nonlinear discriminant analysis using kernel function operator. The underlying theory is close to the support-vector machines (SVM) insofar as the GDA method provides a mapping of the input vectors into high-dimensional feature space. Similar to LDA, the objective of GDA is to find a projection for the features into a lower dimensional space by maximizing the ratio of between-class scatter to within-class scatter.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0022", "text": "Autoencoder\nAutoencoders can be used to learn nonlinear dimension reduction functions and codings together with an inverse function from the coding to the original representation.\n\nt-SNE\nT-distributed Stochastic Neighbor Embedding (t-SNE) is a nonlinear dimensionality reduction technique useful for the visualization of high-dimensional datasets. It is not recommended for use in analysis such as clustering or outlier detection since it does not necessarily preserve densities or distances well.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0023", "text": "UMAP\nUniform manifold approximation and projection (UMAP) is a nonlinear dimensionality reduction technique. Visually, it is similar to t-SNE, but it assumes that the data is uniformly distributed on a locally connected Riemannian manifold and that the Riemannian metric is locally constant or approximately locally constant.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0024", "text": "Dimension reduction\nFor high-dimensional datasets, dimension reduction is usually performed prior to applying a k-nearest neighbors (k-NN) algorithm in order to mitigate the curse of dimensionality.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0025", "text": "Feature extraction and dimension reduction can be combined in one step, using principal component analysis (PCA), linear discriminant analysis (LDA), canonical correlation analysis (CCA), or non-negative matrix factorization (NMF) techniques to pre-process the data, followed by clustering via k-NN on feature vectors in a reduced-dimension space. In machine learning, this process is also called low-dimensional embedding.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0026", "text": "For high-dimensional datasets (e.g., when performing similarity search on live video streams, DNA data, or high-dimensional time series), running a fast approximate k-NN search using locality-sensitive hashing, random projection, \"sketches\", or other high-dimensional similarity search techniques from the VLDB conference toolbox may be the only feasible option.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0027", "text": "Applications\nA dimensionality reduction technique that is sometimes used in neuroscience is maximally informative dimensions, which finds a lower-dimensional representation of a dataset such that as much information as possible about the original data is preserved.", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Dimensionality_reduction_0028", "text": "See also\nNotes\nReferences\nExternal links\nJMLR Special Issue on Variable and Feature Selection\nELastic MAPs Archived 2011-07-20 at the Wayback Machine\nLocally Linear Embedding\nVisual Comparison of various dimensionality reduction methods\nA Global Geometric Framework for Nonlinear Dimensionality Reduction", "title": "Dimensionality reduction", "url": "https://en.wikipedia.org/wiki/Dimensionality_reduction"}, {"id": "rec_Reinforcement_learning_0000", "text": "In machine learning and optimal control, reinforcement learning (RL) is concerned with how an intelligent agent should take actions in a dynamic environment in order to maximize a reward signal. Reinforcement learning is one of the three basic machine learning paradigms, alongside supervised learning and unsupervised learning.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0001", "text": "While supervised learning and unsupervised learning algorithms respectively attempt to discover patterns in labeled and unlabeled data, reinforcement learning involves training an agent through interactions with its environment. To learn to maximize rewards from these interactions, the agent makes decisions between trying new actions to learn more about the environment (exploration), or using current knowledge of the environment to take the best action (exploitation)", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0002", "text": ". The search for the optimal balance between these two strategies is known as the exploration\u2013exploitation dilemma.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0003", "text": "The environment is typically stated in the form of a Markov decision process, as many reinforcement learning algorithms use dynamic programming techniques. The main difference between classical dynamic programming methods and reinforcement learning algorithms is that the latter do not assume knowledge of an exact mathematical model of the Markov decision process, and they target large Markov decision processes where exact methods become infeasible.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0004", "text": "Principles", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0005", "text": "Due to its generality, reinforcement learning is studied in many disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimization, multi-agent systems, swarm intelligence, and statistics. In the operations research and control literature, RL is called approximate dynamic programming, or neuro-dynamic programming", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0006", "text": ". The problems of interest in RL have also been studied in the theory of optimal control, which is concerned mostly with the existence and characterization of optimal solutions, and algorithms for their exact computation, and less with learning or approximation (particularly in the absence of a mathematical model of the environment).", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0007", "text": "Basic reinforcement learning is modeled as a Markov decision process:", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0008", "text": "A set of environment and agent states (the state space), \n  \n    \n      \n        \n          \n            S\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {S}}}\n  \n;\nA set of actions (the action space), \n  \n    \n      \n        \n          \n            A\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {A}}}\n  \n, of the agent;", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0009", "text": "P\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n        =\n        Pr\n        (\n        \n          S\n          \n            t\n            +\n            1\n          \n        \n        \n          =\n        \n        \n          s\n          \u2032\n        \n        \u2223\n        \n          S\n          \n            t\n          \n        \n        \n          =\n        \n        s\n        ,", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0010", "text": "=\n        \n        s\n        ,\n        \n          A\n          \n            t\n          \n        \n        \n          =\n        \n        a\n        )\n      \n    \n    {\\displaystyle P_{a}(s,s')=\\Pr(S_{t+1}{=}s'\\mid S_{t}{=}s,A_{t}{=}a)}\n  \n, the transition probability (at time \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n) from state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n to state \n  \n    \n      \n        \n          s\n          \u2032", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0011", "text": "s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n under action \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0012", "text": "R\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle R_{a}(s,s')}\n  \n, the immediate reward after transitioning from \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n to \n  \n    \n      \n        \n          s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n under action \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0013", "text": "a\n      \n    \n    {\\displaystyle a}\n  \n.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0014", "text": "The purpose of reinforcement learning is for the agent to learn an optimal (or near-optimal) policy that maximizes the reward function or other user-provided reinforcement signal that accumulates from immediate rewards. This is similar to processes that appear to occur in animal psychology. For example, biological brains are hardwired to interpret signals such as pain and hunger as negative reinforcements, and interpret pleasure and food intake as positive reinforcements", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0015", "text": ". In some circumstances, animals learn to adopt behaviors that optimize these rewards. This suggests that animals are capable of reinforcement learning.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0016", "text": "A basic reinforcement learning agent interacts with its environment in discrete time steps. At each time step t, the agent receives the current state \n  \n    \n      \n        \n          S\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle S_{t}}\n  \n and reward \n  \n    \n      \n        \n          R\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle R_{t}}\n  \n. It then chooses an action \n  \n    \n      \n        \n          A\n          \n            t", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0017", "text": "A\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle A_{t}}\n  \n from the set of available actions, which is subsequently sent to the environment. The environment moves to a new state \n  \n    \n      \n        \n          S\n          \n            t\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle S_{t+1}}\n  \n and the reward \n  \n    \n      \n        \n          R\n          \n            t\n            +\n            1", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0018", "text": "+\n            1\n          \n        \n      \n    \n    {\\displaystyle R_{t+1}}\n  \n associated with the transition \n  \n    \n      \n        (\n        \n          S\n          \n            t\n          \n        \n        ,\n        \n          A\n          \n            t\n          \n        \n        ,\n        \n          S\n          \n            t\n            +\n            1\n          \n        \n        )\n      \n    \n    {\\displaystyle (S_{t},A_{t},S_{t+1})}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0019", "text": "{\\displaystyle (S_{t},A_{t},S_{t+1})}\n  \n is determined. The goal of a reinforcement learning agent is to learn a policy:", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0020", "text": "\u03c0\n                :\n                \n                  \n                    S\n                  \n                \n                \u00d7\n                \n                  \n                    A\n                  \n                \n                \u2192\n                [\n                0\n                ,\n                1\n                ]", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0021", "text": "\u03c0\n                (\n                s\n                ,\n                a\n                )\n                =\n                Pr\n                (\n                \n                  A\n                  \n                    t\n                  \n                \n                \n                  =\n                \n                a\n                \u2223\n                \n                  S\n                  \n                    t", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0022", "text": "t\n                  \n                \n                \n                  =\n                \n                s\n                )\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}&\\pi :{\\mathcal {S}}\\times {\\mathcal {A}}\\to [0,1]\\\\&\\pi (s,a)=\\Pr(A_{t}{=}a\\mid S_{t}{=}s)\\end{aligned}}}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0023", "text": "that maximizes the expected cumulative reward.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0024", "text": "Formulating the problem as a Markov decision process assumes the agent directly observes the current environmental state; in this case, the problem is said to have full observability. If the agent only has access to a subset of states, or if the observed states are corrupted by noise, the agent is said to have partial observability, and formally the problem must be formulated as a partially observable Markov decision process", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0025", "text": ". In both cases, the set of actions available to the agent can be restricted. For example, the state of an account balance could be restricted to be positive; if the current value of the state is 3 and the state transition attempts to reduce the value by 4, the transition will not be allowed.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0026", "text": "When the agent's performance is compared to that of an agent that acts optimally, the difference in performance yields the notion of regret. In order to act near optimally, the agent must reason about long-term consequences of its actions (i.e., maximize future rewards), although the immediate reward associated with this might be negative.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0027", "text": "Thus, reinforcement learning is particularly well-suited to problems that include a long-term versus short-term reward trade-off. It has been applied successfully to various problems, including energy storage, robot control, photovoltaic generators, backgammon, checkers, Go (AlphaGo), and autonomous driving systems.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0028", "text": "Two elements make reinforcement learning powerful: the use of samples to optimize performance, and the use of function approximation to deal with large environments. Thanks to these two key components, RL can be used in large environments in the following situations:", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0029", "text": "A model of the environment is known, but an analytic solution is not available;\nOnly a simulation model of the environment is given (the subject of simulation-based optimization);\nThe only way to collect information about the environment is to interact with it.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0030", "text": "The first two of these problems could be considered planning problems (since some form of model is available), while the last one could be considered to be a genuine learning problem. However, reinforcement learning converts both planning problems to machine learning problems.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0031", "text": "Exploration\nThe trade-off between exploration and exploitation has been most thoroughly studied through the multi-armed bandit problem and for finite state space Markov decision processes in Burnetas and Katehakis (1997).", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0032", "text": "Reinforcement learning requires clever exploration mechanisms; randomly selecting actions, without reference to an estimated probability distribution, shows poor performance. The case of (small) finite Markov decision processes is relatively well understood. However, due to the lack of algorithms that scale well with the number of states (or scale to problems with infinite state spaces), simple exploration methods are the most practical.\nOne such method is \n  \n    \n      \n        \u03b5", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0033", "text": "\u03b5\n      \n    \n    {\\displaystyle \\varepsilon }\n  \n-greedy, where \n  \n    \n      \n        0\n        <\n        \u03b5\n        <\n        1\n      \n    \n    {\\displaystyle 0<\\varepsilon <1}\n  \n is a parameter controlling the amount of exploration vs. exploitation. With probability \n  \n    \n      \n        1\n        \u2212\n        \u03b5\n      \n    \n    {\\displaystyle 1-\\varepsilon }", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0034", "text": "{\\displaystyle 1-\\varepsilon }\n  \n, exploitation is chosen, and the agent chooses the action that it believes has the best long-term effect (ties between actions are broken uniformly at random). Alternatively, with probability \n  \n    \n      \n        \u03b5\n      \n    \n    {\\displaystyle \\varepsilon }\n  \n, exploration is chosen, and the action is chosen uniformly at random. \n  \n    \n      \n        \u03b5\n      \n    \n    {\\displaystyle \\varepsilon }", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0035", "text": "{\\displaystyle \\varepsilon }\n  \n is usually a fixed parameter but can be adjusted either according to a schedule (making the agent explore progressively less), or adaptively based on heuristics.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0036", "text": "Algorithms for control learning\nEven if the issue of exploration is disregarded and even if the state was observable (assumed hereafter), the problem remains to use past experience to find out which actions lead to higher cumulative rewards.\n\nCriterion of optimality\nPolicy\nThe agent's action selection is modeled as a map called policy:", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0037", "text": "\u03c0\n                :\n                \n                  \n                    A\n                  \n                \n                \u00d7\n                \n                  \n                    S\n                  \n                \n                \u2192\n                [\n                0\n                ,\n                1\n                ]", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0038", "text": "\u03c0\n                (\n                a\n                ,\n                s\n                )\n                =\n                Pr\n                (\n                \n                  A\n                  \n                    t\n                  \n                \n                \n                  =\n                \n                a\n                \u2223\n                \n                  S\n                  \n                    t", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0039", "text": "t\n                  \n                \n                \n                  =\n                \n                s\n                )\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}&\\pi :{\\mathcal {A}}\\times {\\mathcal {S}}\\to [0,1]\\\\&\\pi (a,s)=\\Pr(A_{t}{=}a\\mid S_{t}{=}s)\\end{aligned}}}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0040", "text": "The policy map gives the probability of taking action \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n when in state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n. There are also deterministic policies  \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n for which \n  \n    \n      \n        \u03c0\n        (\n        s\n        )\n      \n    \n    {\\displaystyle \\pi (s)}\n  \n denotes the action that should be played at state \n  \n    \n      \n        s", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0041", "text": "s\n      \n    \n    {\\displaystyle s}\n  \n.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0042", "text": "State-value function\nThe state-value function \n  \n    \n      \n        \n          V\n          \n            \u03c0\n          \n        \n        (\n        s\n        )\n      \n    \n    {\\displaystyle V_{\\pi }(s)}\n  \n is defined as, expected discounted return starting with state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n, i.e. \n  \n    \n      \n        \n          S\n          \n            0\n          \n        \n        =\n        s\n      \n    \n    {\\displaystyle S_{0}=s}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0043", "text": "{\\displaystyle S_{0}=s}\n  \n, and successively following policy \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n. Hence, roughly speaking, the value function estimates \"how good\" it is to be in a given state.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0044", "text": "V\n          \n            \u03c0\n          \n        \n        (\n        s\n        )\n        =\n        \n          \n            E\n          \n        \n        \u2061\n        [\n        G\n        \u2223\n        \n          S\n          \n            0\n          \n        \n        \n          =\n        \n        s\n        ]\n        =\n        \n          \n            E\n          \n        \n        \u2061\n        \n          [\n          \n            \n              \u2211\n              \n                t", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0045", "text": "\u2211\n              \n                t\n                =\n                0\n              \n              \n                \u221e\n              \n            \n            \n              \u03b3\n              \n                t\n              \n            \n            \n              R\n              \n                t\n                +\n                1\n              \n            \n            \u2223\n            \n              S\n              \n                0", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0046", "text": "=\n            \n            s\n          \n          ]\n        \n        ,\n      \n    \n    {\\displaystyle V_{\\pi }(s)=\\operatorname {\\mathbb {E} } [G\\mid S_{0}{=}s]=\\operatorname {\\mathbb {E} } \\left[\\sum _{t=0}^{\\infty }\\gamma ^{t}R_{t+1}\\mid S_{0}{=}s\\right],}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0047", "text": "where the random variable \n  \n    \n      \n        G\n      \n    \n    {\\displaystyle G}\n  \n denotes the discounted return, and is defined as the sum of future discounted rewards:", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0048", "text": "G\n        =\n        \n          \u2211\n          \n            t\n            =\n            0\n          \n          \n            \u221e\n          \n        \n        \n          \u03b3\n          \n            t\n          \n        \n        \n          R\n          \n            t\n            +\n            1\n          \n        \n        =\n        \n          R\n          \n            1\n          \n        \n        +\n        \u03b3\n        \n          R\n          \n            2\n          \n        \n        +", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0049", "text": "2\n          \n        \n        +\n        \n          \u03b3\n          \n            2\n          \n        \n        \n          R\n          \n            3\n          \n        \n        +\n        \u22ef\n        ,\n      \n    \n    {\\displaystyle G=\\sum _{t=0}^{\\infty }\\gamma ^{t}R_{t+1}=R_{1}+\\gamma R_{2}+\\gamma ^{2}R_{3}+\\cdots ,}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0050", "text": "where \n  \n    \n      \n        \n          R\n          \n            t\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle R_{t+1}}\n  \n is the reward for transitioning from state \n  \n    \n      \n        \n          S\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle S_{t}}\n  \n to \n  \n    \n      \n        \n          S\n          \n            t\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle S_{t+1}}\n  \n,", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0051", "text": "{\\displaystyle S_{t+1}}\n  \n, \n  \n    \n      \n        0\n        \u2264\n        \u03b3\n        <\n        1\n      \n    \n    {\\displaystyle 0\\leq \\gamma <1}\n  \n is the discount rate. \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n is less than 1, so rewards in the distant future are weighted less than rewards in the immediate future.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0052", "text": "The algorithm must find a policy with maximum expected discounted return. From the theory of Markov decision processes it is known that, without loss of generality, the search can be restricted to the set of so-called stationary policies. A policy is stationary if the action-distribution returned by it depends only on the last state visited (from the observation agent's history). The search can be further restricted to deterministic stationary policies", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0053", "text": ". A deterministic stationary policy deterministically selects actions based on the current state. Since any such policy can be identified with a mapping from the set of states to the set of actions, these policies can be identified with such mappings with no loss of generality.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0054", "text": "Brute force\nThe brute force approach entails two steps:", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0055", "text": "For each possible policy, sample returns while following it\nChoose the policy with the largest expected discounted return\nOne problem with this is that the number of policies can be large, or even infinite. Another is that the variance of the returns may be large, which requires many samples to accurately estimate the discounted return of each policy.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0056", "text": "These problems can be ameliorated if we assume some structure and allow samples generated from one policy to influence the estimates made for others. The two main approaches for achieving this are value function estimation and direct policy search.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0057", "text": "Value function\nValue function approaches attempt to find a policy that maximizes the discounted return by maintaining a set of estimates of expected discounted returns \n  \n    \n      \n        \n          \n            E\n          \n        \n        \u2061\n        [\n        G\n        ]\n      \n    \n    {\\displaystyle \\operatorname {\\mathbb {E} } [G]}\n  \n for some policy (usually either the \"current\" [on-policy] or the optimal [off-policy] one).", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0058", "text": "These methods rely on the theory of Markov decision processes, where optimality is defined in a sense stronger than the one above: A policy is optimal if it achieves the best-expected discounted return from any initial state (i.e., initial distributions play no role in this definition). Again, an optimal policy can always be found among stationary policies.\nTo define optimality in a formal manner, define the state-value of a policy \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0059", "text": "\u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n by", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0060", "text": "V\n          \n            \u03c0\n          \n        \n        (\n        s\n        )\n        =\n        \n          \n            E\n          \n        \n        \u2061\n        [\n        G\n        \u2223\n        s\n        ,\n        \u03c0\n        ]\n        ,\n      \n    \n    {\\displaystyle V^{\\pi }(s)=\\operatorname {\\mathbb {E} } [G\\mid s,\\pi ],}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0061", "text": "where \n  \n    \n      \n        G\n      \n    \n    {\\displaystyle G}\n  \n stands for the discounted return associated with following \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n from the initial state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n. Defining \n  \n    \n      \n        \n          V\n          \n            \u2217\n          \n        \n        (\n        s\n        )\n      \n    \n    {\\displaystyle V^{*}(s)}\n  \n as the maximum possible state-value of", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0062", "text": "as the maximum possible state-value of \n  \n    \n      \n        \n          V\n          \n            \u03c0\n          \n        \n        (\n        s\n        )\n      \n    \n    {\\displaystyle V^{\\pi }(s)}\n  \n, where \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n is allowed to change,", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0063", "text": "V\n          \n            \u2217\n          \n        \n        (\n        s\n        )\n        =\n        \n          max\n          \n            \u03c0\n          \n        \n        \n          V\n          \n            \u03c0\n          \n        \n        (\n        s\n        )\n        .\n      \n    \n    {\\displaystyle V^{*}(s)=\\max _{\\pi }V^{\\pi }(s).}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0064", "text": "A policy that achieves these optimal state-values in each state is called optimal. Clearly, a policy that is optimal in this sense is also optimal in the sense that it maximizes the expected discounted return, since \n  \n    \n      \n        \n          V\n          \n            \u2217\n          \n        \n        (\n        s\n        )\n        =\n        \n          max\n          \n            \u03c0\n          \n        \n        \n          E\n        \n        [\n        G\n        \u2223\n        s\n        ,\n        \u03c0", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0065", "text": "G\n        \u2223\n        s\n        ,\n        \u03c0\n        ]\n      \n    \n    {\\displaystyle V^{*}(s)=\\max _{\\pi }\\mathbb {E} [G\\mid s,\\pi ]}\n  \n, where \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n is a state randomly sampled from the distribution \n  \n    \n      \n        \u03bc\n      \n    \n    {\\displaystyle \\mu }\n  \n of initial states (so \n  \n    \n      \n        \u03bc\n        (\n        s\n        )\n        =\n        Pr\n        (\n        \n          S\n          \n            0", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0066", "text": "S\n          \n            0\n          \n        \n        =\n        s\n        )\n      \n    \n    {\\displaystyle \\mu (s)=\\Pr(S_{0}=s)}\n  \n).\nAlthough state-values suffice to define optimality, it is useful to define action-values. Given a state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n, an action \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n and a policy \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n, the action-value of the pair", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0067", "text": ", the action-value of the pair \n  \n    \n      \n        (\n        s\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle (s,a)}\n  \n under \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n is defined by", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0068", "text": "Q\n          \n            \u03c0\n          \n        \n        (\n        s\n        ,\n        a\n        )\n        =\n        \n          \n            E\n          \n        \n        \u2061\n        [\n        G\n        \u2223\n        s\n        ,\n        a\n        ,\n        \u03c0\n        ]\n        ,\n      \n    \n    {\\displaystyle Q^{\\pi }(s,a)=\\operatorname {\\mathbb {E} } [G\\mid s,a,\\pi ],}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0069", "text": "where \n  \n    \n      \n        G\n      \n    \n    {\\displaystyle G}\n  \n now stands for the random discounted return associated with first taking action \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n in state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n and following \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n, thereafter.\nThe theory of Markov decision processes states that if \n  \n    \n      \n        \n          \u03c0\n          \n            \u2217", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0070", "text": "\u03c0\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle \\pi ^{*}}\n  \n is an optimal policy, we act optimally (take the optimal action) by choosing the action from \n  \n    \n      \n        \n          Q\n          \n            \n              \u03c0\n              \n                \u2217\n              \n            \n          \n        \n        (\n        s\n        ,\n        \u22c5\n        )\n      \n    \n    {\\displaystyle Q^{\\pi ^{*}}(s,\\cdot )}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0071", "text": "{\\displaystyle Q^{\\pi ^{*}}(s,\\cdot )}\n  \n with the highest action-value at each state, \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n. The action-value function of such an optimal policy (\n  \n    \n      \n        \n          Q\n          \n            \n              \u03c0\n              \n                \u2217\n              \n            \n          \n        \n      \n    \n    {\\displaystyle Q^{\\pi ^{*}}}\n  \n) is called the optimal action-value function and is commonly denoted by", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0072", "text": "Q\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle Q^{*}}\n  \n. In summary, the knowledge of the optimal action-value function alone suffices to know how to act optimally.\nAssuming full knowledge of the Markov decision process, the two basic approaches to compute the optimal action-value function are value iteration and policy iteration. Both algorithms compute a sequence of functions \n  \n    \n      \n        \n          Q", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0073", "text": "Q\n          \n            k\n          \n        \n      \n    \n    {\\displaystyle Q_{k}}\n  \n (\n  \n    \n      \n        k\n        =\n        0\n        ,\n        1\n        ,\n        2\n        ,\n        \u2026\n      \n    \n    {\\displaystyle k=0,1,2,\\ldots }\n  \n) that converge to \n  \n    \n      \n        \n          Q\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle Q^{*}}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0074", "text": "{\\displaystyle Q^{*}}\n  \n. Computing these functions involves computing expectations over the whole state-space, which is impractical for all but the smallest (finite) Markov decision processes. In reinforcement learning methods, expectations are approximated by averaging over samples and using function approximation techniques to cope with the need to represent value functions over large state-action spaces.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0075", "text": "Monte Carlo methods", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0076", "text": "Monte Carlo methods are used to solve reinforcement learning problems by averaging sample returns. Unlike methods that require full knowledge of the environment's dynamics, Monte Carlo methods rely solely on actual or simulated experience\u2014sequences of states, actions, and rewards obtained from interaction with an environment. This makes them applicable in situations where the complete dynamics are unknown", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0077", "text": ". Learning from actual experience does not require prior knowledge of the environment and can still lead to optimal behavior. When using simulated experience, only a model capable of generating sample transitions is required, rather than a full specification of transition probabilities, which is necessary for dynamic programming methods.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0078", "text": "Monte Carlo methods apply to episodic tasks, where experience is divided into episodes that eventually terminate. Policy and value function updates occur only after the completion of an episode, making these methods incremental on an episode-by-episode basis, though not on a step-by-step (online) basis", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0079", "text": ". The term \"Monte Carlo\" generally refers to any method involving random sampling; however, in this context, it specifically refers to methods that compute averages from complete returns, rather than partial returns.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0080", "text": "These methods function similarly to the bandit algorithms, in which returns are averaged for each state-action pair. The key difference is that actions taken in one state affect the returns of subsequent states within the same episode, making the problem non-stationary. To address this non-stationarity, Monte Carlo methods use the framework of general policy iteration (GPI)", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0081", "text": ". While dynamic programming computes value functions using full knowledge of the Markov decision process, Monte Carlo methods learn these functions through sample returns. The value functions and policies interact similarly to dynamic programming to achieve optimality, first addressing the prediction problem and then extending to policy improvement and control, all based on sampled experience.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0082", "text": "Temporal difference methods\nThe first problem is corrected by allowing the procedure to change the policy (at some or all states) before the values settle. This too may be problematic as it might prevent convergence. Most current algorithms do this, giving rise to the class of generalized policy iteration algorithms. Many actor-critic methods belong to this category.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0083", "text": "The second issue can be corrected by allowing trajectories to contribute to any state-action pair in them. This may also help to some extent with the third problem, although a better solution when returns have high variance is Sutton's temporal difference (TD) methods that are based on the recursive Bellman equation", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0084", "text": ". The computation in TD methods can be incremental (when after each transition the memory is changed and the transition is thrown away), or batch (when the transitions are batched and the estimates are computed once based on the batch). Batch methods, such as the least-squares temporal difference method, may use the information in the samples better, while incremental methods are the only choice when batch methods are infeasible due to their high computational or memory complexity", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0085", "text": ". Some methods try to combine the two approaches. Methods based on temporal differences also overcome the fourth issue.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0086", "text": "Another problem specific to TD comes from their reliance on the recursive Bellman equation. Most TD methods have a so-called \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n parameter \n  \n    \n      \n        (\n        0\n        \u2264\n        \u03bb\n        \u2264\n        1\n        )\n      \n    \n    {\\displaystyle (0\\leq \\lambda \\leq 1)}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0087", "text": "{\\displaystyle (0\\leq \\lambda \\leq 1)}\n  \n that can continuously interpolate between Monte Carlo methods that do not rely on the Bellman equations and the basic TD methods that rely entirely on the Bellman equations. This can be effective in palliating this issue.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0088", "text": "Function approximation methods\nIn order to address the fifth issue, function approximation methods are used. Linear function approximation starts with a mapping \n  \n    \n      \n        \u03d5\n      \n    \n    {\\displaystyle \\phi }\n  \n that assigns a finite-dimensional vector to each state-action pair. Then, the action values of a state-action pair \n  \n    \n      \n        (\n        s\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle (s,a)}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0089", "text": "{\\displaystyle (s,a)}\n  \n are obtained by linearly combining the components of \n  \n    \n      \n        \u03d5\n        (\n        s\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle \\phi (s,a)}\n  \n with some weights \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n:", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0090", "text": "Q\n        (\n        s\n        ,\n        a\n        )\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            d\n          \n        \n        \n          \u03b8\n          \n            i\n          \n        \n        \n          \u03d5\n          \n            i\n          \n        \n        (\n        s\n        ,\n        a\n        )\n        .\n      \n    \n    {\\displaystyle Q(s,a)=\\sum _{i=1}^{d}\\theta _{i}\\phi _{i}(s,a).}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0091", "text": "The algorithms then adjust the weights, instead of adjusting the values associated with the individual state-action pairs. Methods based on ideas from nonparametric statistics (which can be seen to construct their own features) have been explored.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0092", "text": "Value iteration can also be used as a starting point, giving rise to the Q-learning algorithm and its many variants. Including Deep Q-learning methods when a neural network is used to represent Q, with various applications in stochastic search problems.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0093", "text": "The problem with using action-values is that they may need highly precise estimates of the competing action values that can be hard to obtain when the returns are noisy, though this problem is mitigated to some extent by temporal difference methods. Using the so-called compatible function approximation method compromises generality and efficiency.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0094", "text": "Direct policy search\nAn alternative method is to search directly in (some subset of) the policy space, in which case the problem becomes a case of stochastic optimization. The two approaches available are gradient-based and gradient-free methods.\nGradient-based methods (policy gradient methods) start with a mapping from a finite-dimensional (parameter) space to the space of policies: given the parameter vector \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n, let", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0095", "text": "{\\displaystyle \\theta }\n  \n, let \n  \n    \n      \n        \n          \u03c0\n          \n            \u03b8\n          \n        \n      \n    \n    {\\displaystyle \\pi _{\\theta }}\n  \n denote the policy associated to \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n. Defining the performance function by \n  \n    \n      \n        \u03c1\n        (\n        \u03b8\n        )\n        =\n        \n          \u03c1\n          \n            \n              \u03c0\n              \n                \u03b8", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0096", "text": "\u03b8\n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\rho (\\theta )=\\rho ^{\\pi _{\\theta }}}\n  \n under mild conditions this function will be differentiable as a function of the parameter vector \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n. If the gradient of \n  \n    \n      \n        \u03c1\n      \n    \n    {\\displaystyle \\rho }", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0097", "text": "{\\displaystyle \\rho }\n  \n was known, one could use gradient ascent. Since an analytic expression for the gradient is not available, only a noisy estimate is available. Such an estimate can be constructed in many ways, giving rise to algorithms such as Williams's REINFORCE method (which is known as the likelihood ratio method in the simulation-based optimization literature).", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0098", "text": "A large class of methods avoids relying on gradient information. These include simulated annealing, cross-entropy search or methods of evolutionary computation. Many gradient-free methods can achieve (in theory and in the limit) a global optimum.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0099", "text": "Policy search methods may converge slowly given noisy data. For example, this happens in episodic problems when the trajectories are long and the variance of the returns is large. Value-function based methods that rely on temporal differences might help in this case. In recent years, actor\u2013critic methods have been proposed and performed well on various problems.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0100", "text": "Policy search methods have been used in the robotics context. Many policy search methods may get stuck in local optima (as they are based on local search).", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0101", "text": "Model-based algorithms", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0102", "text": "Finally, all of the above methods can be combined with algorithms that first learn a model of the Markov decision process, the probability of each next state given an action taken from an existing state. For instance, the Dyna algorithm learns a model from experience, and uses that to provide more modelled transitions for a value function, in addition to the real transitions", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0103", "text": ". Such methods can sometimes be extended to use of non-parametric models, such as when the transitions are simply stored and \"replayed\" to the learning algorithm.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0104", "text": "Model-based methods can be more computationally intensive than model-free approaches, and their utility can be limited by the extent to which the Markov decision process can be learnt.\nThere are other ways to use models than to update a value function. For instance, in model predictive control the model is used to update the behavior directly.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0105", "text": "Theory\nBoth the asymptotic and finite-sample behaviors of most algorithms are well understood. Algorithms with provably good online performance (addressing the exploration issue) are known.\nEfficient exploration of Markov decision processes is given in Burnetas and Katehakis (1997). Finite-time performance bounds have also appeared for many algorithms, but these bounds are expected to be rather loose and thus more work is needed to better understand the relative advantages and limitations.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0106", "text": "For incremental algorithms, asymptotic convergence issues have been settled. Temporal-difference-based algorithms converge under a wider set of conditions than was previously possible (for example, when used with arbitrary, smooth function approximation).", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0107", "text": "Research\nResearch topics include:", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0108", "text": "actor-critic architecture\nactor-critic-scenery architecture\nadaptive methods that work with fewer (or no) parameters under a large number of conditions\nbug detection in software projects\ncontinuous learning\ncombinations with logic-based frameworks (e.g., temporal-logic specifications, reward machines, and probabilistic argumentation).\nexploration in large Markov decision processes\nentity-based reinforcement learning\nhuman feedback", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0109", "text": "human feedback\ninteraction between implicit and explicit learning in skill acquisition\nintrinsic motivation which differentiates information-seeking, curiosity-type behaviours from task-dependent goal-directed behaviours large-scale empirical evaluations\nlarge (or continuous) action spaces\nmodular and hierarchical reinforcement learning\nmultiagent/distributed reinforcement learning is a topic of interest. Applications are expanding.\noccupant-centric control\noptimization of computing resources", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0110", "text": "optimization of computing resources\npartial information (e.g., using predictive state representation)\nreward function based on maximising novel information\nsample-based planning (e.g., based on Monte Carlo tree search).\nsecurities trading\ntransfer learning\nTD learning modeling dopamine-based learning in the brain. Dopaminergic projections from the substantia nigra to the basal ganglia function are the prediction error.\nvalue-function and policy search methods", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0111", "text": "Comparison of key algorithms\nThe following table lists the key algorithms for learning a policy depending on several criteria:", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0112", "text": "The algorithm can be on-policy (it performs policy updates using trajectories sampled via the current policy) or off-policy.\nThe action space may be discrete (e.g. the action space could be \"going up\", \"going left\", \"going right\", \"going down\", \"stay\") or continuous (e.g. moving the arm with a given angle).\nThe state space may be discrete (e.g. the agent could be in a cell in a grid) or continuous (e.g. the agent could be located at a given position in the plane).", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0113", "text": "Associative reinforcement learning\nAssociative reinforcement learning tasks combine facets of stochastic learning automata tasks and supervised learning pattern classification tasks. In associative reinforcement learning tasks, the learning system interacts in a closed loop with its environment.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0114", "text": "Deep reinforcement learning\nThis approach extends reinforcement learning by using a deep neural network and without explicitly designing the state space. The work on learning ATARI games by Google DeepMind increased attention to deep reinforcement learning or end-to-end reinforcement learning.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0115", "text": "Adversarial deep reinforcement learning", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0116", "text": "Adversarial deep reinforcement learning is an active area of research in reinforcement learning focusing on vulnerabilities of learned policies. In this research area some studies initially showed that reinforcement learning policies are susceptible to imperceptible adversarial manipulations", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0117", "text": ". While some methods have been proposed to overcome these susceptibilities, in the most recent studies it has been shown that these proposed solutions are far from providing an accurate representation of current vulnerabilities of deep reinforcement learning policies.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0118", "text": "Fuzzy reinforcement learning\nBy introducing fuzzy inference in reinforcement learning, approximating the state-action value function with fuzzy rules in continuous space becomes possible. The IF - THEN form of fuzzy rules make this approach suitable for expressing the results in a form close to natural language. Extending FRL with Fuzzy Rule Interpolation allows the use of reduced size sparse fuzzy rule-bases to emphasize cardinal rules (most important state-action values).", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0119", "text": "Inverse reinforcement learning", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0120", "text": "In inverse reinforcement learning (IRL), no reward function is given. Instead, the reward function is inferred given an observed behavior from an expert. The idea is to mimic observed behavior, which is often optimal or close to optimal. One popular IRL paradigm is named maximum entropy inverse reinforcement learning (MaxEnt IRL)", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0121", "text": ". MaxEnt IRL estimates the parameters of a linear model of the reward function by maximizing the entropy of the probability distribution of observed trajectories subject to constraints related to matching expected feature counts. Recently it has been shown that MaxEnt IRL is a particular case of a more general framework named random utility inverse reinforcement learning (RU-IRL). RU-IRL is based on random utility theory and Markov decision processes", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0122", "text": ". While prior IRL approaches assume that the apparent random behavior of an observed agent is due to it following a random policy, RU-IRL assumes that the observed agent follows a deterministic policy but randomness in observed behavior is due to the fact that an observer only has partial access to the features the observed agent uses in decision making", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0123", "text": ". The utility function is modeled as a random variable to account for the ignorance of the observer regarding the features the observed agent actually considers in its utility function.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0124", "text": "Multi-objective reinforcement learning\nMulti-objective reinforcement learning (MORL) is a form of reinforcement learning concerned with conflicting alternatives. It is distinct from multi-objective optimization in that it is concerned with agents acting in environments.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0125", "text": "Safe reinforcement learning", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0126", "text": "Safe reinforcement learning (SRL) can be defined as the process of learning policies that maximize the expectation of the return in problems in which it is important to ensure reasonable system performance and/or respect safety constraints during the learning and/or deployment processes. An alternative approach is risk-averse reinforcement learning, where instead of the expected return, a risk-measure of the return is optimized, such as the conditional value at risk (CVaR)", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0127", "text": ". In addition to mitigating risk, the CVaR objective increases robustness to model uncertainties. However, CVaR optimization in risk-averse RL requires special care, to prevent gradient bias and blindness to success.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0128", "text": "Self-reinforcement learning\nSelf-reinforcement learning (or self-learning), is a learning paradigm which does not use the concept of immediate reward \n  \n    \n      \n        \n          R\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle R_{a}(s,s')}\n  \n after transition from \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n to \n  \n    \n      \n        \n          s", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0129", "text": "to \n  \n    \n      \n        \n          s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n with action \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0130", "text": "a\n      \n    \n    {\\displaystyle a}\n  \n. It does not use an external reinforcement, it only uses the agent internal self-reinforcement. The internal self-reinforcement is provided by mechanism of feelings and emotions. In the learning process emotions are backpropagated by a mechanism of secondary reinforcement. The learning equation does not include the immediate reward, it only includes the state evaluation.\nThe self-reinforcement algorithm updates a memory matrix", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0131", "text": "W\n        =\n        \u2016\n        w\n        (\n        a\n        ,\n        s\n        )\n        \u2016\n      \n    \n    {\\displaystyle W=\\|w(a,s)\\|}\n  \n such that in each iteration executes the following machine learning routine:", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0132", "text": "In situation \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n perform action \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n.\nReceive a consequence situation \n  \n    \n      \n        \n          s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n.\nCompute state evaluation \n  \n    \n      \n        v\n        (\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle v(s')}\n  \n of how good is to be in the consequence situation", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0133", "text": "s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n.\nUpdate crossbar memory \n  \n    \n      \n        \n          w\n          \u2032\n        \n        (\n        a\n        ,\n        s\n        )\n        =\n        w\n        (\n        a\n        ,\n        s\n        )\n        +\n        v\n        (\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle w'(a,s)=w(a,s)+v(s')}\n  \n.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0134", "text": "{\\displaystyle w'(a,s)=w(a,s)+v(s')}\n  \n.\nInitial conditions of the memory are received as input from the genetic environment. It is a system with only one input (situation), and only one output (action, or behavior).", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0135", "text": "Self-reinforcement (self-learning) was introduced in 1982 along with a neural network capable of self-reinforcement learning, named Crossbar Adaptive Array (CAA). The CAA computes, in a crossbar fashion, both decisions about actions and emotions (feelings) about consequence states. The system is driven by the interaction between cognition and emotion.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0136", "text": "Reinforcement Learning in Natural Language Processing", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0137", "text": "In recent years, reinforcement learning has become a significant concept in natural language processing (NLP), where tasks are often sequential decision-making rather than static classification. Reinforcement learning is where an agent take actions in an environment to maximize the accumulation of rewards", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0138", "text": ". This framework is best fit for many NLP tasks, including dialogue generation, text summarization, and machine translation, where the quality of the output depends on optimizing long-term or human-centered goals rather than the prediction of single correct label.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0139", "text": "Early application of RL in NLP emerged in dialogue systems, where conversation was determined as a series of actions optimized for fluency and coherence. These early attempts, including policy gradient and sequence-level training techniques, laid a foundation for the broader application of reinforcement learning to other areas of NLP.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0140", "text": "A major breakthrough happened with the introduction of reinforcement learning from human feedback (RLHF), a method in which human feedback ratings are used to train a reward model that guides the RL agent. Unlike traditional rule-based or supervised systems, RLHF allows models to align their behavior with human judgments on complex and subjective tasks", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0141", "text": ". This technique was initially used in the development of InstructGPT, an effective language model trained to follow human instructions and later in ChatGPT which incorporates RLHF for improving output responses and ensuring safety.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0142", "text": "More recently, researchers have explored the use of offline RL in NLP to improve dialogue systems without the need of live human interaction. These methods optimize for user engagement, coherence, and diversity based on past conversation logs and pre-trained reward models.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0143", "text": "One example is DeepSeek-R1, which incorporates multi-stage training and cold-start data before RL. DeepSeek-R1 achieves performance comparable to OpenAI-o1-1217 on reasoning tasks. This model was trained via large-scale reinforcement learning (RL) without supervised fine-tuning (SFT) as a preliminary step.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0144", "text": "Statistical comparison of reinforcement learning algorithms", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0145", "text": "Efficient comparison of RL algorithms is essential for research, deployment and monitoring of RL systems. To compare different algorithms on a given environment, an agent can be trained for each algorithm. Since the performance is sensitive to implementation details, all algorithms should be implemented as closely as possible to each other. After the training is finished, the agents can be run on a sample of test episodes, and their scores (returns) can be compared", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0146", "text": ". Since episodes are typically assumed to be i.i.d, standard statistical tools can be used for hypothesis testing, such as T-test and permutation test. This requires to accumulate all the rewards within an episode into a single number\u2014the episodic return. However, this causes a loss of information, as different time-steps are averaged together, possibly with different levels of noise", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0147", "text": ". Whenever the noise level varies across the episode, the statistical power can be improved significantly, by weighting the rewards according to their estimated noise.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0148", "text": "Challenges and limitations\nDespite significant advancements, reinforcement learning (RL) continues to face several challenges and limitations that hinder its widespread application in real-world scenarios.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0149", "text": "Sample inefficiency", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0150", "text": "RL algorithms often require a large number of interactions with the environment to learn effective policies, leading to high computational costs and time-intensive to train the agent. For instance, OpenAI's Dota-playing bot utilized thousands of years of simulated gameplay to achieve human-level performance", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0151", "text": ". Techniques like experience replay and curriculum learning have been proposed to deprive sample inefficiency, but these techniques add more complexity and are not always sufficient for real-world applications.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0152", "text": "Stability and convergence issues\nTraining RL models, particularly for deep neural network-based models, can be unstable and prone to divergence. A small change in the policy or environment can lead to extreme fluctuations in performance, making it difficult to achieve consistent results. This instability is further enhanced in the case of the continuous or high-dimensional action space, where the learning step becomes more complex and less predictable.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0153", "text": "Generalization and transferability\nThe RL agents trained in specific environments often struggle to generalize their learned policies to new, unseen scenarios. This is the major setback preventing the application of RL to dynamic real-world environments where adaptability is crucial. The challenge is to develop such algorithms that can transfer knowledge across tasks and environments without extensive retraining.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0154", "text": "Bias and reward function issues\nDesigning appropriate reward functions is critical in RL because poorly designed reward functions can lead to unintended behaviors. In addition, RL systems trained on biased data may perpetuate existing biases and lead to discriminatory or unfair outcomes. Both of these issues requires careful consideration of reward structures and data sources to ensure fairness and desired behaviors.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0155", "text": "See also\nReferences\nFurther reading\nAnnaswamy, Anuradha M. (3 May 2023). \"Adaptive Control and Intersections with Reinforcement Learning\". Annual Review of Control, Robotics, and Autonomous Systems. 6 (1): 65\u201393. doi:10.1146/annurev-control-062922-090153. ISSN 2573-5144. S2CID 255702873.\nAuer, Peter; Jaksch, Thomas; Ortner, Ronald (2010). \"Near-optimal regret bounds for reinforcement learning\". Journal of Machine Learning Research. 11: 1563\u20131600.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0156", "text": "Bertsekas, Dimitri P. (2023) [2019]. Reinforcement Learning and Optimal Control (1st ed.). Athena Scientific. ISBN 978-1-886-52939-7.\nBusoniu, Lucian; Babuska, Robert; De Schutter, Bart; Ernst, Damien (2010). Reinforcement Learning and Dynamic Programming using Function Approximators. Taylor & Francis CRC Press. ISBN 978-1-4398-2108-4.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0157", "text": "Fran\u00e7ois-Lavet, Vincent; Henderson, Peter; Islam, Riashat; Bellemare, Marc G.; Pineau, Joelle (2018). \"An Introduction to Deep Reinforcement Learning\". Foundations and Trends in Machine Learning. 11 (3\u20134): 219\u2013354. arXiv:1811.12560. Bibcode:2018arXiv181112560F. doi:10.1561/2200000071. S2CID 54434537.\nLi, Shengbo Eben (2023). Reinforcement Learning for Sequential Decision and Optimal Control (1st ed.). Springer Verlag, Singapore. doi:10.1007/978-981-19-7784-8. ISBN 978-9-811-97783-1.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0158", "text": "Powell, Warren (2011). Approximate dynamic programming: solving the curses of dimensionality. Wiley-Interscience. Archived from the original on 2016-07-31. Retrieved 2010-09-08.\nSutton, Richard S. (1988). \"Learning to predict by the method of temporal differences\". Machine Learning. 3 (1): 9\u201344. Bibcode:1988MLear...3....9S. doi:10.1007/BF00115009.\nSutton, Richard S.; Barto, Andrew G. (2018) [1998]. Reinforcement Learning: An Introduction (2nd ed.). MIT Press. ISBN 978-0-262-03924-6.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0159", "text": "Szita, Istvan; Szepesvari, Csaba (2010). \"Model-based Reinforcement Learning with Nearly Tight Exploration Complexity Bounds\" (PDF). ICML 2010. Omnipress. pp. 1031\u20131038. Archived from the original (PDF) on 2010-07-14.", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Reinforcement_learning_0160", "text": "External links\nDissecting Reinforcement Learning Series of blog post on reinforcement learning with Python code\nA (Long) Peek into Reinforcement Learning\nQSMM \u2013 reinforcement learning through adaptive probabilistic assembler programs", "title": "Reinforcement learning", "url": "https://en.wikipedia.org/wiki/Reinforcement_learning"}, {"id": "rec_Qlearning_0000", "text": "Q-learning is a reinforcement learning algorithm that trains an agent to assign values to its possible actions based on its current state, without requiring a model of the environment (model-free). It can handle problems with stochastic transitions and rewards without requiring adaptations.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0001", "text": "For example, in a grid maze, an agent learns to reach an exit worth 10 points. At a junction, Q-learning might assign a higher value to moving right than left if right gets to the exit faster, improving this choice by trying both directions over time.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0002", "text": "For any finite Markov decision process, Q-learning finds an optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps, starting from the current state. Q-learning can identify an optimal action-selection policy for any given finite Markov decision process, given infinite exploration time and a partly random policy.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0003", "text": "\"Q\" refers to the function that the algorithm computes: the expected reward\u2014that is, the quality\u2014of an action taken in a given state.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0004", "text": "Reinforcement learning\nReinforcement learning involves an agent, a set of states \n  \n    \n      \n        \n          \n            S\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {S}}}\n  \n, and a set \n  \n    \n      \n        \n          \n            A\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {A}}}\n  \n of actions per state. By performing an action \n  \n    \n      \n        a\n        \u2208\n        \n          \n            A", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0005", "text": "A\n          \n        \n      \n    \n    {\\displaystyle a\\in {\\mathcal {A}}}\n  \n, the agent transitions from state to state. Executing an action in a specific state provides the agent with a reward (a numerical score).", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0006", "text": "The goal of the agent is to maximize its total reward. It does this by adding the maximum reward attainable from future states to the reward for achieving its current state, effectively influencing the current action by the potential future reward. This potential reward is a weighted sum of expected values of the rewards of all future steps starting from the current state.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0007", "text": "As an example, consider the process of boarding a train, in which the reward is measured by the negative of the total time spent boarding (alternatively, the cost of boarding the train is equal to the boarding time). One strategy is to enter the train door as soon as they open, minimizing the initial wait time for yourself", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0008", "text": ". If the train is crowded, however, then you will have a slow entry after the initial action of entering the door as people are fighting you to depart the train as you attempt to board. The total boarding time, or cost, is then:", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0009", "text": "0 seconds wait time + 15 seconds fight time\nOn the next day, by random chance (exploration), you decide to wait and let other people depart first. This initially results in a longer wait time. However, less time is spent fighting the departing passengers. Overall, this path has a higher reward than that of the previous day, since the total boarding time is now:", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0010", "text": "5 second wait time + 0 second fight time\nThrough exploration, despite the initial (patient) action resulting in a larger cost (or negative reward) than in the forceful strategy, the overall cost is lower, thus revealing a more rewarding strategy.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0011", "text": "Algorithm\nAfter \n  \n    \n      \n        \u0394\n        t\n      \n    \n    {\\displaystyle \\Delta t}\n  \n steps into the future the agent will decide some next step. The weight for this step is calculated as \n  \n    \n      \n        \n          \u03b3\n          \n            \u0394\n            t\n          \n        \n      \n    \n    {\\displaystyle \\gamma ^{\\Delta t}}\n  \n, where \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n (the discount factor) is a number between 0 and 1 (", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0012", "text": "0\n        \u2264\n        \u03b3\n        \u2264\n        1\n      \n    \n    {\\displaystyle 0\\leq \\gamma \\leq 1}\n  \n). Assuming \n  \n    \n      \n        \u03b3\n        <\n        1\n      \n    \n    {\\displaystyle \\gamma <1}\n  \n, it has the effect of valuing rewards received earlier higher than those received later (reflecting the value of a \"good start\"). \n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0013", "text": "{\\displaystyle \\gamma }\n  \n may also be interpreted as the probability to succeed (or survive) at every step \n  \n    \n      \n        \u0394\n        t\n      \n    \n    {\\displaystyle \\Delta t}\n  \n.\nThe algorithm, therefore, has a function that calculates the quality of a state\u2013action combination:", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0014", "text": "Q\n        :\n        \n          \n            S\n          \n        \n        \u00d7\n        \n          \n            A\n          \n        \n        \u2192\n        \n          R\n        \n      \n    \n    {\\displaystyle Q:{\\mathcal {S}}\\times {\\mathcal {A}}\\to \\mathbb {R} }\n  \n.\nBefore learning begins, \u2060\n  \n    \n      \n        Q\n      \n    \n    {\\displaystyle Q}\n  \n\u2060 is initialized to a possibly arbitrary fixed value (chosen by the programmer). Then, at each time \n  \n    \n      \n        t", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0015", "text": "t\n      \n    \n    {\\displaystyle t}\n  \n the agent selects an action \n  \n    \n      \n        \n          A\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle A_{t}}\n  \n, observes a reward \n  \n    \n      \n        \n          R\n          \n            t\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle R_{t+1}}\n  \n, enters a new state \n  \n    \n      \n        \n          S\n          \n            t\n            +\n            1", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0016", "text": "t\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle S_{t+1}}\n  \n (that may depend on both the previous state \n  \n    \n      \n        \n          S\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle S_{t}}\n  \n and the selected action), and \n  \n    \n      \n        Q\n      \n    \n    {\\displaystyle Q}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0017", "text": "Q\n      \n    \n    {\\displaystyle Q}\n  \n is updated. The core of the algorithm is a Bellman equation as a simple value iteration update, using the weighted average of the current value and the new information:", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0018", "text": "Q\n          \n            n\n            e\n            w\n          \n        \n        (\n        \n          S\n          \n            t\n          \n        \n        ,\n        \n          A\n          \n            t\n          \n        \n        )\n        \u2190\n        (\n        1\n        \u2212\n        \n          \n            \n              \u03b1\n              \u23df\n            \n          \n          \n            learning rate\n          \n        \n        )\n        \u22c5", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0019", "text": ")\n        \u22c5\n        \n          \n            \n              \n                Q\n                (\n                \n                  S\n                  \n                    t\n                  \n                \n                ,\n                \n                  A\n                  \n                    t\n                  \n                \n                )\n              \n              \u23df\n            \n          \n          \n            current value\n          \n        \n        +", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0020", "text": "+\n        \n          \n            \n              \u03b1\n              \u23df\n            \n          \n          \n            learning rate\n          \n        \n        \u22c5\n        \n          \n            (\n          \n        \n        \n          \n            \n              \n                \n                  \n                    \n                      \n                        R\n                        \n                          t\n                          +", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0021", "text": "+\n                          1\n                        \n                      \n                      \u23df\n                    \n                  \n                  \n                    reward\n                  \n                \n                +\n                \n                  \n                    \n                      \u03b3\n                      \u23df\n                    \n                  \n                  \n                    discount factor", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0022", "text": "\u22c5\n                \n                  \n                    \n                      \n                        \n                          max\n                          \n                            a\n                          \n                        \n                        Q\n                        (\n                        \n                          S\n                          \n                            t\n                            +", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0023", "text": "+\n                            1\n                          \n                        \n                        ,\n                        a\n                        )\n                      \n                      \u23df\n                    \n                  \n                  \n                    estimate of optimal future value\n                  \n                \n              \n              \u23df", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0024", "text": "new value (temporal difference target)\n          \n        \n        \n          \n            )", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0025", "text": "{\\displaystyle Q^{new}(S_{t},A_{t})\\leftarrow (1-\\underbrace {\\alpha } _{\\text{learning rate}})\\cdot \\underbrace {Q(S_{t},A_{t})} _{\\text{current value}}+\\underbrace {\\alpha } _{\\text{learning rate}}\\cdot {\\bigg (}\\underbrace {\\underbrace {R_{t+1}} _{\\text{reward}}+\\underbrace {\\gamma } _{\\text{discount factor}}\\cdot \\underbrace {\\max _{a}Q(S_{t+1},a)} _{\\text{estimate of optimal future value}}} _{\\text{new value (temporal difference target)}}{\\bigg )}}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0026", "text": "where \n  \n    \n      \n        \n          R\n          \n            t\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle R_{t+1}}\n  \n is the reward received when moving from the state \n  \n    \n      \n        \n          S\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle S_{t}}\n  \n to the state \n  \n    \n      \n        \n          S\n          \n            t\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle S_{t+1}}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0027", "text": "{\\displaystyle S_{t+1}}\n  \n, and \n  \n    \n      \n        \u03b1\n      \n    \n    {\\displaystyle \\alpha }\n  \n is the learning rate \n  \n    \n      \n        (\n        0\n        <\n        \u03b1\n        \u2264\n        1\n        )\n      \n    \n    {\\displaystyle (0<\\alpha \\leq 1)}\n  \n.\nNote that \n  \n    \n      \n        \n          Q\n          \n            n\n            e\n            w\n          \n        \n        (\n        \n          S\n          \n            t\n          \n        \n        ,", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0028", "text": "t\n          \n        \n        ,\n        \n          A\n          \n            t\n          \n        \n        )\n      \n    \n    {\\displaystyle Q^{new}(S_{t},A_{t})}\n  \n is the sum of three terms:", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0029", "text": "(\n        1\n        \u2212\n        \u03b1\n        )\n        Q\n        (\n        \n          S\n          \n            t\n          \n        \n        ,\n        \n          A\n          \n            t\n          \n        \n        )\n      \n    \n    {\\displaystyle (1-\\alpha )Q(S_{t},A_{t})}\n  \n: the current value (weighted by one minus the learning rate)", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0030", "text": "\u03b1\n        \n        \n          R\n          \n            t\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle \\alpha \\,R_{t+1}}\n  \n: the reward \n  \n    \n      \n        \n          R\n          \n            t\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle R_{t+1}}\n  \n to obtain if action \n  \n    \n      \n        \n          A\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle A_{t}}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0031", "text": "{\\displaystyle A_{t}}\n  \n is taken when in state \n  \n    \n      \n        \n          S\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle S_{t}}\n  \n (weighted by learning rate)", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0032", "text": "\u03b1\n        \u03b3\n        \n          max\n          \n            a\n          \n        \n        Q\n        (\n        \n          S\n          \n            t\n            +\n            1\n          \n        \n        ,\n        a\n        )\n      \n    \n    {\\displaystyle \\alpha \\gamma \\max _{a}Q(S_{t+1},a)}\n  \n: the maximum reward that can be obtained from state \n  \n    \n      \n        \n          S\n          \n            t\n            +\n            1", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0033", "text": "1\n          \n        \n      \n    \n    {\\displaystyle S_{t+1}}\n  \n(weighted by learning rate and discount factor)\nAn episode of the algorithm ends when state \n  \n    \n      \n        \n          S\n          \n            t\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle S_{t+1}}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0034", "text": "{\\displaystyle S_{t+1}}\n  \n is a final or terminal state. However, Q-learning can also learn in non-episodic tasks (as a result of the property of convergent infinite series). If the discount factor is lower than 1, the action values are finite even if the problem can contain infinite loops or paths.\nFor all final states \n  \n    \n      \n        \n          s\n          \n            f\n          \n        \n      \n    \n    {\\displaystyle s_{f}}\n  \n, \n  \n    \n      \n        Q\n        (", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0035", "text": ", \n  \n    \n      \n        Q\n        (\n        \n          s\n          \n            f\n          \n        \n        ,\n        a\n        )\n      \n    \n    {\\displaystyle Q(s_{f},a)}\n  \n is never updated, but is set to the reward value \n  \n    \n      \n        r\n      \n    \n    {\\displaystyle r}\n  \n observed for state \n  \n    \n      \n        \n          s\n          \n            f\n          \n        \n      \n    \n    {\\displaystyle s_{f}}\n  \n. In most cases, \n  \n    \n      \n        Q\n        (", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0036", "text": "Q\n        (\n        \n          s\n          \n            f\n          \n        \n        ,\n        a\n        )\n      \n    \n    {\\displaystyle Q(s_{f},a)}\n  \n can be taken to equal zero.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0037", "text": "Influence of variables\nLearning rate\nThe learning rate or step size determines to what extent newly acquired information overrides old information. A factor of 0 makes the agent learn nothing (exclusively exploiting prior knowledge), while a factor of 1 makes the agent consider only the most recent information (ignoring prior knowledge to explore possibilities). In fully deterministic environments, a learning rate of \n  \n    \n      \n        \n          \u03b1\n          \n            t", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0038", "text": "\u03b1\n          \n            t\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle \\alpha _{t}=1}\n  \n is optimal. When the problem is stochastic, the algorithm converges under some technical conditions on the learning rate that require it to decrease to zero. In practice, often a constant learning rate is used, such as \n  \n    \n      \n        \n          \u03b1\n          \n            t\n          \n        \n        =\n        0.1\n      \n    \n    {\\displaystyle \\alpha _{t}=0.1}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0039", "text": "{\\displaystyle \\alpha _{t}=0.1}\n  \n for all \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0040", "text": "Discount factor\nThe discount factor \u2060\n  \n    \n      \n        \u03b3\n      \n    \n    {\\displaystyle \\gamma }\n  \n\u2060 determines the importance of future rewards. A factor of 0 will make the agent \"myopic\" (or short-sighted) by only considering current rewards, i.e. \n  \n    \n      \n        \n          r\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle r_{t}}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0041", "text": "{\\displaystyle r_{t}}\n  \n (in the update rule above), while a factor approaching 1 will make it strive for a long-term high reward. If the discount factor meets or exceeds 1, the action values may diverge. For \u2060\n  \n    \n      \n        \u03b3\n        =\n        1\n      \n    \n    {\\displaystyle \\gamma =1}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0042", "text": "\u2060, without a terminal state, or if the agent never reaches one, all environment histories become infinitely long, and utilities with additive, undiscounted rewards generally become infinite. Even with a discount factor only slightly lower than 1, Q-function learning leads to propagation of errors and instabilities when the value function is approximated with an artificial neural network", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0043", "text": ". In that case, starting with a lower discount factor and increasing it towards its final value accelerates learning.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0044", "text": "Initial conditions (Q0)\nSince Q-learning is an iterative algorithm, it implicitly assumes an initial condition before the first update occurs. High initial values, also known as \"optimistic initial conditions\", can encourage exploration: no matter what action is selected, the update rule will cause it to have lower values than the other alternative, thus increasing their choice probability. The first reward \n  \n    \n      \n        r\n      \n    \n    {\\displaystyle r}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0045", "text": "r\n      \n    \n    {\\displaystyle r}\n  \n can be used to reset the initial conditions. According to this idea, the first time an action is taken the reward is used to set the value of \n  \n    \n      \n        Q\n      \n    \n    {\\displaystyle Q}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0046", "text": "Q\n      \n    \n    {\\displaystyle Q}\n  \n. This allows immediate learning in case of fixed deterministic rewards. A model that incorporates reset of initial conditions (RIC) is expected to predict participants' behavior better than a model that assumes any arbitrary initial condition (AIC). RIC seems to be consistent with human behaviour in repeated binary choice experiments.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0047", "text": "Implementation\nQ-learning at its simplest stores data in tables. This approach falters with increasing numbers of states/actions since the likelihood of the agent visiting a particular state and performing a particular action is increasingly small.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0048", "text": "Function approximation\nQ-learning can be combined with function approximation. This makes it possible to apply the algorithm to larger problems, even when the state space is continuous.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0049", "text": "One solution is to use an (adapted) artificial neural network as a function approximator. Another possibility is to integrate Fuzzy Rule Interpolation (FRI) and use sparse fuzzy rule-bases instead of discrete Q-tables or ANNs, which has the advantage of being a human-readable knowledge representation form. Function approximation may speed up learning in finite problems, due to the fact that the algorithm can generalize earlier experiences to previously unseen states.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0050", "text": "Quantization", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0051", "text": "Another technique to decrease the state/action space quantizes possible values. Consider the example of learning to balance a stick on a finger. To describe a state at a certain point in time involves the position of the finger in space, its velocity, the angle of the stick and the angular velocity of the stick. This yields a four-element vector that describes one state, i.e. a snapshot of one state encoded into four values. The problem is that infinitely many possible states are present", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0052", "text": ". To shrink the possible space of valid actions multiple values can be assigned to a bucket. The exact distance of the finger from its starting position (-Infinity to Infinity) is not known, but rather whether it is far away or not (Near, Far).", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0053", "text": "History\nQ-learning was introduced by Chris Watkins in 1989. A convergence proof was presented by Watkins and Peter Dayan in 1992, building on Watkins' doctoral dissertation, Learning from Delayed Rewards. Eight years earlier in 1981, the same problem, under the name of \"Delayed reinforcement learning,\" was solved by Bozinovski's Crossbar Adaptive Array (CAA). The memory matrix \n  \n    \n      \n        W\n        =\n        \u2016\n        w\n        (\n        a\n        ,\n        s\n        )\n        \u2016", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0054", "text": "a\n        ,\n        s\n        )\n        \u2016\n      \n    \n    {\\displaystyle W=\\|w(a,s)\\|}\n  \n was the same as the eight years later Q-table of Q-learning. The architecture introduced the term \"state evaluation\" in reinforcement learning. The crossbar learning algorithm, written in mathematical pseudocode in the paper, in each iteration performs the following computation:", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0055", "text": "In state s perform action a;\nReceive consequence state s';\nCompute state evaluation \u2060\n  \n    \n      \n        v\n        (\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle v(s')}\n  \n\u2060;\nUpdate crossbar value \n  \n    \n      \n        \n          w\n          \u2032\n        \n        (\n        a\n        ,\n        s\n        )\n        =\n        w\n        (\n        a\n        ,\n        s\n        )\n        +\n        v\n        (\n        \n          s\n          \u2032\n        \n        )", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0056", "text": "s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle w'(a,s)=w(a,s)+v(s')}\n  \n.\nThe term \"secondary reinforcement\" is borrowed from animal learning theory, to model state values via backpropagation: the state value \u2060\n  \n    \n      \n        v\n        (\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle v(s')}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0057", "text": "{\\displaystyle v(s')}\n  \n\u2060 of the consequence situation is backpropagated to the previously encountered situations. CAA computes state values vertically and actions horizontally (the \"crossbar\"). Demonstration graphs showing delayed reinforcement learning contained states (desirable, undesirable, and neutral states), which were computed by the state evaluation function. This learning system was a forerunner of the Q-learning algorithm.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0058", "text": "In 2014, Google DeepMind patented an application of Q-learning to deep learning, entitled \"deep reinforcement learning\" or \"deep Q-learning,\" that can play Atari 2600 games at expert human levels.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0059", "text": "Variants\nDeep Q-learning", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0060", "text": "The DeepMind system used a deep convolutional neural network, with layers of tiled convolutional filters to mimic the effects of receptive fields. Reinforcement learning is unstable or divergent when a nonlinear function approximator such as a neural network is used to represent Q", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0061", "text": ". This instability comes from the correlations present in the sequence of observations, the fact that small updates to Q may significantly change the policy of the agent and the data distribution, and the correlations between Q and the target values. The method can be used for stochastic search in various domains and applications.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0062", "text": "The technique used experience replay, a biologically inspired mechanism that uses a random sample of prior actions instead of the most recent action to proceed. This removes correlations in the observation sequence and smooths changes in the data distribution. Iterative updates adjust Q towards target values that are only periodically updated, further reducing correlations with the target.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0063", "text": "Double Q-learning\nBecause the future maximum approximated action value in Q-learning is evaluated using the same Q function as in current action selection policy, in noisy environments Q-learning can sometimes overestimate the action values, slowing the learning. A variant called Double Q-learning was proposed to correct this. Double Q-learning is an off-policy reinforcement learning algorithm, where a different policy is used for value evaluation than what is used to select the next action.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0064", "text": "In practice, two separate value functions  \n  \n    \n      \n        \n          Q\n          \n            A\n          \n        \n      \n    \n    {\\displaystyle Q^{A}}\n  \n and \n  \n    \n      \n        \n          Q\n          \n            B\n          \n        \n      \n    \n    {\\displaystyle Q^{B}}\n  \n are trained in a mutually symmetric fashion using separate experiences. The double Q-learning update step is then as follows:", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0065", "text": "Q\n          \n            t\n            +\n            1\n          \n          \n            A\n          \n        \n        (\n        \n          s\n          \n            t\n          \n        \n        ,\n        \n          a\n          \n            t\n          \n        \n        )\n        =\n        \n          Q\n          \n            t\n          \n          \n            A\n          \n        \n        (\n        \n          s\n          \n            t", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0066", "text": "t\n          \n        \n        ,\n        \n          a\n          \n            t\n          \n        \n        )\n        +\n        \n          \u03b1\n          \n            t\n          \n        \n        (\n        \n          s\n          \n            t\n          \n        \n        ,\n        \n          a\n          \n            t\n          \n        \n        )\n        \n          (\n          \n            \n              r\n              \n                t", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0067", "text": "t\n              \n            \n            +\n            \u03b3\n            \n              Q\n              \n                t\n              \n              \n                B\n              \n            \n            \n              (\n              \n                \n                  s\n                  \n                    t\n                    +\n                    1\n                  \n                \n                ,", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0068", "text": "a\n                      r\n                      g\n                       \n                      m\n                      a\n                      x\n                    \n                  \n                  \n                    a\n                  \n                \n                \u2061\n                \n                  Q\n                  \n                    t\n                  \n                  \n                    A", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0069", "text": "A\n                  \n                \n                (\n                \n                  s\n                  \n                    t\n                    +\n                    1\n                  \n                \n                ,\n                a\n                )\n              \n              )\n            \n            \u2212\n            \n              Q\n              \n                t\n              \n              \n                A\n              \n            \n            (", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0070", "text": "(\n            \n              s\n              \n                t\n              \n            \n            ,\n            \n              a\n              \n                t\n              \n            \n            )\n          \n          )", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0071", "text": ")\n        \n      \n    \n    {\\displaystyle Q_{t+1}^{A}(s_{t},a_{t})=Q_{t}^{A}(s_{t},a_{t})+\\alpha _{t}(s_{t},a_{t})\\left(r_{t}+\\gamma Q_{t}^{B}\\left(s_{t+1},\\mathop {\\operatorname {arg~max} } _{a}Q_{t}^{A}(s_{t+1},a)\\right)-Q_{t}^{A}(s_{t},a_{t})\\right)}\n  \n, and", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0072", "text": "Q\n          \n            t\n            +\n            1\n          \n          \n            B\n          \n        \n        (\n        \n          s\n          \n            t\n          \n        \n        ,\n        \n          a\n          \n            t\n          \n        \n        )\n        =\n        \n          Q\n          \n            t\n          \n          \n            B\n          \n        \n        (\n        \n          s\n          \n            t", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0073", "text": "t\n          \n        \n        ,\n        \n          a\n          \n            t\n          \n        \n        )\n        +\n        \n          \u03b1\n          \n            t\n          \n        \n        (\n        \n          s\n          \n            t\n          \n        \n        ,\n        \n          a\n          \n            t\n          \n        \n        )\n        \n          (\n          \n            \n              r\n              \n                t", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0074", "text": "t\n              \n            \n            +\n            \u03b3\n            \n              Q\n              \n                t\n              \n              \n                A\n              \n            \n            \n              (\n              \n                \n                  s\n                  \n                    t\n                    +\n                    1\n                  \n                \n                ,", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0075", "text": "a\n                      r\n                      g\n                       \n                      m\n                      a\n                      x\n                    \n                  \n                  \n                    a\n                  \n                \n                \u2061\n                \n                  Q\n                  \n                    t\n                  \n                  \n                    B", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0076", "text": "B\n                  \n                \n                (\n                \n                  s\n                  \n                    t\n                    +\n                    1\n                  \n                \n                ,\n                a\n                )\n              \n              )\n            \n            \u2212\n            \n              Q\n              \n                t\n              \n              \n                B\n              \n            \n            (", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0077", "text": "(\n            \n              s\n              \n                t\n              \n            \n            ,\n            \n              a\n              \n                t\n              \n            \n            )\n          \n          )\n        \n        .", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0078", "text": ")\n        \n        .\n      \n    \n    {\\displaystyle Q_{t+1}^{B}(s_{t},a_{t})=Q_{t}^{B}(s_{t},a_{t})+\\alpha _{t}(s_{t},a_{t})\\left(r_{t}+\\gamma Q_{t}^{A}\\left(s_{t+1},\\mathop {\\operatorname {arg~max} } _{a}Q_{t}^{B}(s_{t+1},a)\\right)-Q_{t}^{B}(s_{t},a_{t})\\right).}", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0079", "text": "Now the estimated value of the discounted future is evaluated using a different policy, which solves the overestimation issue.\nThis algorithm was later modified in 2015 and combined with deep learning, as in the DQN algorithm, resulting in Double DQN, which outperforms the original DQN algorithm.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0080", "text": "Others\nDelayed Q-learning is an alternative implementation of the online Q-learning algorithm, with probably approximately correct (PAC) learning.\nGreedy GQ is a variant of Q-learning to use in combination with (linear) function approximation. The advantage of Greedy GQ is that convergence is guaranteed even when function approximation is used to estimate the action values.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0081", "text": "Distributional Q-learning is a variant of Q-learning which seeks to model the distribution of returns rather than the expected return of each action. It has been observed to facilitate estimate by deep neural networks and can enable alternative control methods, such as risk-sensitive control.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0082", "text": "Multi-agent learning\nQ-learning has been proposed in the multi-agent setting (see Section 4.1.2 in ). One approach consists in pretending the environment is passive. Littman proposes the minimax Q learning algorithm.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0083", "text": "Limitations\nThe standard Q-learning algorithm (using a \n  \n    \n      \n        Q\n      \n    \n    {\\displaystyle Q}\n  \n table) applies only to discrete action and state spaces. Discretization of these values leads to inefficient learning, largely due to the curse of dimensionality. However, there are adaptations of Q-learning that attempt to solve this problem such as Wire-fitted Neural Network Q-Learning.", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0084", "text": "See also\nReinforcement learning\nTemporal difference learning\nSARSA\nIterated prisoner's dilemma\nGame theory", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0085", "text": "References\nExternal links\nWatkins, C.J.C.H. (1989). Learning from Delayed Rewards. PhD thesis, Cambridge University, Cambridge, England.\nStrehl, Li, Wiewiora, Langford, Littman (2006). PAC model-free reinforcement learning\nReinforcement Learning: An Introduction by Richard Sutton and Andrew S. Barto, an online textbook. See \"6.5 Q-Learning: Off-Policy TD Control\".\nPiqle: a Generic Java Platform for Reinforcement Learning", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Qlearning_0086", "text": "Reinforcement Learning Maze, a demonstration of guiding an ant through a maze using Q-learning\nQ-learning work by Gerald Tesauro", "title": "Q-learning", "url": "https://en.wikipedia.org/wiki/Q-learning"}, {"id": "rec_Markov_decision_process_0000", "text": "A Markov decision process (MDP) is a mathematical model for sequential decision making when outcomes are uncertain. It is a type of stochastic decision process, and is often solved using the methods of stochastic dynamic programming.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0001", "text": "Originating from operations research in the 1950s, MDPs have since gained recognition in a variety of fields, including ecology, economics, healthcare, telecommunications and reinforcement learning. Reinforcement learning utilizes the MDP framework to model the interaction between a learning agent and its environment. In this framework, the interaction is characterized by states, actions, and rewards", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0002", "text": ". The MDP framework is designed to provide a simplified representation of key elements of artificial intelligence challenges. This modeling framework incorporates the understanding of cause and effect, the management of uncertainty and nondeterminism, and the pursuit of explicit goals.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0003", "text": "The name comes from its connection to Markov chains, a concept developed by the Russian mathematician Andrey Markov. The \"Markov\" in \"Markov decision process\" refers to the underlying structure of state transitions that still follow the Markov property. The process is called a \"decision process\" because it involves making decisions that influence these state transitions, extending the concept of a Markov chain into the realm of decision-making under uncertainty.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0004", "text": "Definition\nA Markov decision process is a 4-tuple \n  \n    \n      \n        (\n        S\n        ,\n        A\n        ,\n        \n          P\n          \n            a\n          \n        \n        ,\n        \n          R\n          \n            a\n          \n        \n        )\n      \n    \n    {\\displaystyle (S,A,P_{a},R_{a})}\n  \n, where:", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0005", "text": "S\n      \n    \n    {\\displaystyle S}\n  \n is a set of states called the state space. The state space may be discrete or continuous, like the set of real numbers.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0006", "text": "A\n      \n    \n    {\\displaystyle A}\n  \n is a set of actions called the action space (alternatively, \n  \n    \n      \n        \n          A\n          \n            s\n          \n        \n      \n    \n    {\\displaystyle A_{s}}\n  \n is the set of actions available from state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n). As for state, this set may be discrete or continuous.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0007", "text": "P\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle P_{a}(s,s')}\n  \n is, on an intuitive level, the probability that action \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n in state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n at time \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n will lead to state", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0008", "text": "{\\displaystyle t}\n  \n will lead to state \n  \n    \n      \n        \n          s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n at time \n  \n    \n      \n        t\n        +\n        1\n      \n    \n    {\\displaystyle t+1}\n  \n. In general, this probability transition is defined to satisfy \n  \n    \n      \n        Pr\n        (\n        \n          s\n          \n            t\n            +\n            1\n          \n        \n        \u2208\n        \n          S\n          \u2032\n        \n        \u2223", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0009", "text": "S\n          \u2032\n        \n        \u2223\n        \n          s\n          \n            t\n          \n        \n        =\n        s\n        ,\n        \n          a\n          \n            t\n          \n        \n        =\n        a\n        )\n        =\n        \n          \u222b\n          \n            \n              S\n              \u2032\n            \n          \n        \n        \n          P\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0010", "text": "s\n          \u2032\n        \n        )\n        d\n        \n          s\n          \u2032\n        \n        ,\n      \n    \n    {\\displaystyle \\Pr(s_{t+1}\\in S'\\mid s_{t}=s,a_{t}=a)=\\int _{S'}P_{a}(s,s')ds',}\n  \n for every \n  \n    \n      \n        \n          S\n          \u2032\n        \n        \u2286\n        S\n      \n    \n    {\\displaystyle S'\\subseteq S}\n  \n measurable. In case the state space is discrete, the integral is intended with respect to the counting measure, so that the latter simplifies as", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0011", "text": "P\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n        =\n        Pr\n        (\n        \n          s\n          \n            t\n            +\n            1\n          \n        \n        =\n        \n          s\n          \u2032\n        \n        \u2223\n        \n          s\n          \n            t\n          \n        \n        =\n        s\n        ,\n        \n          a\n          \n            t", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0012", "text": "a\n          \n            t\n          \n        \n        =\n        a\n        )\n      \n    \n    {\\displaystyle P_{a}(s,s')=\\Pr(s_{t+1}=s'\\mid s_{t}=s,a_{t}=a)}\n  \n; in case \n  \n    \n      \n        S\n        \u2286\n        \n          \n            R\n          \n          \n            d\n          \n        \n      \n    \n    {\\displaystyle S\\subseteq \\mathbb {R} ^{d}}\n  \n, the integral is usually intended with respect to the Lebesgue measure.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0013", "text": "R\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle R_{a}(s,s')}\n  \n is the immediate reward (or expected immediate reward) received after action \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n is taken to transition from state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n to state \n  \n    \n      \n        \n          s", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0014", "text": "to state \n  \n    \n      \n        \n          s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n. The reward is in general a random variable.\nA policy function \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n is a (potentially probabilistic) mapping from state space (\n  \n    \n      \n        S\n      \n    \n    {\\displaystyle S}\n  \n) to action space (\n  \n    \n      \n        A\n      \n    \n    {\\displaystyle A}\n  \n).", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0015", "text": "Optimization objective\nThe goal in a Markov decision process is to find a good \"policy\" for the decision maker: a function \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n that specifies the action \n  \n    \n      \n        \u03c0\n        (\n        s\n        )\n      \n    \n    {\\displaystyle \\pi (s)}\n  \n that the decision maker will choose when in state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0016", "text": "s\n      \n    \n    {\\displaystyle s}\n  \n. Once a Markov decision process is combined with a policy in this way, this fixes the action for each state and the resulting combination behaves like a Markov chain (since the action chosen in state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n is completely determined by \n  \n    \n      \n        \u03c0\n        (\n        s\n        )\n      \n    \n    {\\displaystyle \\pi (s)}\n  \n).\nThe objective is to choose a policy", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0017", "text": "\u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n that will maximize some cumulative function of the random rewards, typically the expected discounted sum over a potentially infinite horizon:", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0018", "text": "E\n        \n          [\n          \n            \n              \u2211\n              \n                t\n                =\n                0\n              \n              \n                \u221e\n              \n            \n            \n              \n                \u03b3\n                \n                  t\n                \n              \n              \n                R\n                \n                  \n                    a\n                    \n                      t", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0019", "text": "t\n                    \n                  \n                \n              \n              (\n              \n                s\n                \n                  t\n                \n              \n              ,\n              \n                s\n                \n                  t\n                  +\n                  1\n                \n              \n              )\n            \n          \n          ]", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0020", "text": "]\n        \n      \n    \n    {\\displaystyle E\\left[\\sum _{t=0}^{\\infty }{\\gamma ^{t}R_{a_{t}}(s_{t},s_{t+1})}\\right]}\n  \n (where we choose \n  \n    \n      \n        \n          a\n          \n            t\n          \n        \n        =\n        \u03c0\n        (\n        \n          s\n          \n            t\n          \n        \n        )\n      \n    \n    {\\displaystyle a_{t}=\\pi (s_{t})}\n  \n, i.e. actions given by the policy). And the expectation is taken over", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0021", "text": "s\n          \n            t\n            +\n            1\n          \n        \n        \u223c\n        \n          P\n          \n            \n              a\n              \n                t\n              \n            \n          \n        \n        (\n        \n          s\n          \n            t\n          \n        \n        ,\n        \n          s\n          \n            t\n            +\n            1\n          \n        \n        )", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0022", "text": ")\n      \n    \n    {\\displaystyle s_{t+1}\\sim P_{a_{t}}(s_{t},s_{t+1})}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0023", "text": "where \n  \n    \n      \n         \n        \u03b3\n         \n      \n    \n    {\\displaystyle \\ \\gamma \\ }\n  \n is the discount factor satisfying \n  \n    \n      \n        0\n        \u2264\n         \n        \u03b3\n         \n        \u2264\n         \n        1\n      \n    \n    {\\displaystyle 0\\leq \\ \\gamma \\ \\leq \\ 1}\n  \n, which is usually close to \n  \n    \n      \n        1\n      \n    \n    {\\displaystyle 1}\n  \n (for example, \n  \n    \n      \n        \u03b3\n        =\n        1\n        \n          /\n        \n        (\n        1", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0024", "text": "/\n        \n        (\n        1\n        +\n        r\n        )\n      \n    \n    {\\displaystyle \\gamma =1/(1+r)}\n  \n for some discount rate \n  \n    \n      \n        r\n      \n    \n    {\\displaystyle r}\n  \n). A lower discount factor makes the decision maker more short-sighted, in that it comparatively disregards the effect that following its current policy has at times lying further in the future.\nAnother possible, but strictly related, objective that is commonly used is the", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0025", "text": "H\n        \u2212\n      \n    \n    {\\displaystyle H-}\n  \nstep return. This time, instead of using a discount factor \n  \n    \n      \n         \n        \u03b3\n         \n      \n    \n    {\\displaystyle \\ \\gamma \\ }\n  \n, the agent is interested only in the first \n  \n    \n      \n        H\n      \n    \n    {\\displaystyle H}\n  \n steps of the process, with each reward having the same weight.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0026", "text": "E\n        \n          [\n          \n            \n              \u2211\n              \n                t\n                =\n                0\n              \n              \n                H\n                \u2212\n                1\n              \n            \n            \n              \n                R\n                \n                  \n                    a\n                    \n                      t", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0027", "text": "(\n              \n                s\n                \n                  t\n                \n              \n              ,\n              \n                s\n                \n                  t\n                  +\n                  1\n                \n              \n              )\n            \n          \n          ]\n        \n      \n    \n    {\\displaystyle E\\left[\\sum _{t=0}^{H-1}{R_{a_{t}}(s_{t},s_{t+1})}\\right]}\n  \n (where we choose", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0028", "text": "(where we choose \n  \n    \n      \n        \n          a\n          \n            t\n          \n        \n        =\n        \u03c0\n        (\n        \n          s\n          \n            t\n          \n        \n        )\n      \n    \n    {\\displaystyle a_{t}=\\pi (s_{t})}\n  \n, i.e. actions given by the policy). And the expectation is taken over \n  \n    \n      \n        \n          s\n          \n            t\n            +\n            1\n          \n        \n        \u223c\n        \n          P", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0029", "text": "P\n          \n            \n              a\n              \n                t\n              \n            \n          \n        \n        (\n        \n          s\n          \n            t\n          \n        \n        ,\n        \n          s\n          \n            t\n            +\n            1\n          \n        \n        )\n      \n    \n    {\\displaystyle s_{t+1}\\sim P_{a_{t}}(s_{t},s_{t+1})}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0030", "text": "where \n  \n    \n      \n         \n        H\n         \n      \n    \n    {\\displaystyle \\ H\\ }\n  \n is the time horizon. Compared to the previous objective, the latter one is more used in Learning Theory.\nA policy that maximizes the function above is called an optimal policy and is usually denoted \n  \n    \n      \n        \n          \u03c0\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle \\pi ^{*}}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0031", "text": "{\\displaystyle \\pi ^{*}}\n  \n. A particular MDP may have multiple distinct optimal policies. Because of the Markov property, it can be shown that the optimal policy is a function of the current state, as assumed above. When \n  \n    \n      \n        \n          R\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle R_{a}(s,s')}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0032", "text": "{\\displaystyle R_{a}(s,s')}\n  \n is deterministic, there will always exist an optimal policy \n  \n    \n      \n        \n          \u03c0\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle \\pi ^{*}}\n  \n which is deterministic as well.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0033", "text": "Simulator models\nIn many cases, it is difficult to represent the transition probability distributions, \n  \n    \n      \n        \n          P\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle P_{a}(s,s')}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0034", "text": "{\\displaystyle P_{a}(s,s')}\n  \n, explicitly. In such cases, a simulator can be used to model the MDP implicitly by providing samples from the transition distributions. One common form of implicit MDP model is an episodic environment simulator that can be started from an initial state and yields a subsequent state and reward every time it receives an action input. In this manner, trajectories of states, actions, and rewards, often called episodes may be produced.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0035", "text": "Another form of simulator is a generative model, a single step simulator that can generate samples of the next state and reward given any state and action. (Note that this is a different meaning from the term generative model in the context of statistical classification.) In algorithms that are expressed using pseudocode, \n  \n    \n      \n        G\n      \n    \n    {\\displaystyle G}\n  \n is often used to represent a generative model. For example, the expression \n  \n    \n      \n        \n          s", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0036", "text": "s\n          \u2032\n        \n        ,\n        r\n        \u2190\n        G\n        (\n        s\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle s',r\\gets G(s,a)}\n  \n might denote the action of sampling from the generative model where \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n and \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n are the current state and action, and \n  \n    \n      \n        \n          s\n          \u2032", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0037", "text": "s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n and \n  \n    \n      \n        r\n      \n    \n    {\\displaystyle r}\n  \n are the new state and reward. Compared to an episodic simulator, a generative model has the advantage that it can yield data from any state, not only those encountered in a trajectory.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0038", "text": "These model classes form a hierarchy of information content: an explicit model trivially yields a generative model through sampling from the distributions, and repeated application of a generative model yields an episodic simulator. In the opposite direction, it is only possible to learn approximate models through regression. The type of model available for a particular MDP plays a significant role in determining which solution algorithms are appropriate", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0039", "text": ". For example, the dynamic programming algorithms described in the next section require an explicit model, and Monte Carlo tree search requires a generative model (or an episodic simulator that can be copied at any state), whereas most reinforcement learning algorithms require only an episodic simulator.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0040", "text": "Example\nAn example of MDP is the Pole-Balancing model, which comes from classic control theory.\nIn this example, we have", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0041", "text": "S\n      \n    \n    {\\displaystyle S}\n  \n is the set of ordered tuples \n  \n    \n      \n        (\n        \u03b8\n        ,\n        \n          \n            \n              \u03b8\n              \u02d9\n            \n          \n        \n        ,\n        x\n        ,\n        \n          \n            \n              x\n              \u02d9\n            \n          \n        \n        )\n      \n    \n    {\\displaystyle (\\theta ,{\\dot {\\theta }},x,{\\dot {x}})}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0042", "text": "given by pole angle, angular velocity, position of the cart and its speed.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0043", "text": "A\n      \n    \n    {\\displaystyle A}\n  \n is \n  \n    \n      \n        {\n        \u2212\n        1\n        ,\n        1\n        }\n      \n    \n    {\\displaystyle \\{-1,1\\}}\n  \n, corresponding to applying a force on the left (right) on the cart.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0044", "text": "P\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle P_{a}(s,s')}\n  \n is the transition of the system, which in this case is going to be deterministic and driven by the laws of mechanics.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0045", "text": "R\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle R_{a}(s,s')}\n  \n is \n  \n    \n      \n        1\n      \n    \n    {\\displaystyle 1}\n  \n if the pole is up after the transition, zero otherwise. Therefore, this function only depend on \n  \n    \n      \n        \n          s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n in this specific case.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0046", "text": "Algorithms", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0047", "text": "Solutions for MDPs with finite state and action spaces may be found through a variety of methods such as dynamic programming. The algorithms in this section apply to MDPs with finite state and action spaces and explicitly given transition probabilities and reward functions, but the basic concepts may be extended to handle other problem classes, for example using function approximation", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0048", "text": ". Also, some processes with countably infinite state and action spaces can be exactly reduced to ones with finite state and action spaces.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0049", "text": "The standard family of algorithms to calculate optimal policies for finite state and action MDPs requires storage for two arrays indexed by state: value \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n, which contains real values, and policy \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n, which contains actions. At the end of the algorithm, \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n will contain the solution and", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0050", "text": "will contain the solution and \n  \n    \n      \n        V\n        (\n        s\n        )\n      \n    \n    {\\displaystyle V(s)}\n  \n will contain the discounted sum of the rewards to be earned (on average) by following that solution from state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0051", "text": "s\n      \n    \n    {\\displaystyle s}\n  \n.\nThe algorithm has two steps, (1) a value update and (2) a policy update, which are repeated in some order for all the states until no further changes take place.  Both recursively update a new estimation of the optimal policy and state value using an older estimation of those values.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0052", "text": "V\n        (\n        s\n        )\n        :=\n        \n          \u2211\n          \n            \n              s\n              \u2032\n            \n          \n        \n        \n          P\n          \n            \u03c0\n            (\n            s\n            )\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n        \n          (\n          \n            \n              R\n              \n                \u03c0\n                (\n                s", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0053", "text": "(\n                s\n                )\n              \n            \n            (\n            s\n            ,\n            \n              s\n              \u2032\n            \n            )\n            +\n            \u03b3\n            V\n            (\n            \n              s\n              \u2032\n            \n            )\n          \n          )\n        \n      \n    \n    {\\displaystyle V(s):=\\sum _{s'}P_{\\pi (s)}(s,s')\\left(R_{\\pi (s)}(s,s')+\\gamma V(s')\\right)}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0054", "text": "\u03c0\n        (\n        s\n        )\n        :=\n        \n          argmax\n          \n            a\n          \n        \n        \u2061\n        \n          {\n          \n            \n              \u2211\n              \n                \n                  s\n                  \u2032\n                \n              \n            \n            \n              P\n              \n                a\n              \n            \n            (\n            s\n            ,\n            \n              s", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0055", "text": ",\n            \n              s\n              \u2032\n            \n            )\n            \n              (\n              \n                \n                  R\n                  \n                    a\n                  \n                \n                (\n                s\n                ,\n                \n                  s\n                  \u2032\n                \n                )\n                +\n                \u03b3\n                V\n                (\n                \n                  s", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0056", "text": "s\n                  \u2032\n                \n                )\n              \n              )\n            \n          \n          }\n        \n      \n    \n    {\\displaystyle \\pi (s):=\\operatorname {argmax} _{a}\\left\\{\\sum _{s'}P_{a}(s,s')\\left(R_{a}(s,s')+\\gamma V(s')\\right)\\right\\}}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0057", "text": "Their order depends on the variant of the algorithm; one can also do them for all states at once or state by state, and more often to some states than others. As long as no state is permanently excluded from either of the steps, the algorithm will eventually arrive at the correct solution.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0058", "text": "Notable variants\nValue iteration\nIn value iteration (Bellman 1957), which is also called backward induction,\nthe \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n function is not used; instead, the value of \n  \n    \n      \n        \u03c0\n        (\n        s\n        )\n      \n    \n    {\\displaystyle \\pi (s)}\n  \n is calculated within \n  \n    \n      \n        V\n        (\n        s\n        )\n      \n    \n    {\\displaystyle V(s)}\n  \n whenever it is needed. Substituting the calculation of", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0059", "text": "\u03c0\n        (\n        s\n        )\n      \n    \n    {\\displaystyle \\pi (s)}\n  \n into the calculation of \n  \n    \n      \n        V\n        (\n        s\n        )\n      \n    \n    {\\displaystyle V(s)}\n  \n gives the combined step;", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0060", "text": "V\n          \n            i\n            +\n            1\n          \n        \n        (\n        s\n        )\n        :=\n        \n          max\n          \n            a\n          \n        \n        \n          {\n          \n            \n              \u2211\n              \n                \n                  s\n                  \u2032\n                \n              \n            \n            \n              P\n              \n                a", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0061", "text": "a\n              \n            \n            (\n            s\n            ,\n            \n              s\n              \u2032\n            \n            )\n            \n              (\n              \n                \n                  R\n                  \n                    a\n                  \n                \n                (\n                s\n                ,\n                \n                  s\n                  \u2032\n                \n                )\n                +\n                \u03b3", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0062", "text": "+\n                \u03b3\n                \n                  V\n                  \n                    i\n                  \n                \n                (\n                \n                  s\n                  \u2032\n                \n                )\n              \n              )\n            \n          \n          }\n        \n        ,\n      \n    \n    {\\displaystyle V_{i+1}(s):=\\max _{a}\\left\\{\\sum _{s'}P_{a}(s,s')\\left(R_{a}(s,s')+\\gamma V_{i}(s')\\right)\\right\\},}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0063", "text": "where \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n is the iteration number. Value iteration starts at \n  \n    \n      \n        i\n        =\n        0\n      \n    \n    {\\displaystyle i=0}\n  \n and \n  \n    \n      \n        \n          V\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle V_{0}}\n  \n as a guess of the value function. It then iterates, repeatedly computing \n  \n    \n      \n        \n          V\n          \n            i\n            +\n            1", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0064", "text": "i\n            +\n            1\n          \n        \n      \n    \n    {\\displaystyle V_{i+1}}\n  \n for all states \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n, until \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0065", "text": "V\n      \n    \n    {\\displaystyle V}\n  \n converges with the left-hand side equal to the right-hand side (which is the \"Bellman equation\" for this problem). Lloyd Shapley's 1953 paper on stochastic games included as a special case the value iteration method for MDPs, but this was recognized only later on.\nValue iteration is guaranteed to converge for \n  \n    \n      \n        \u03b3\n        <\n        1\n      \n    \n    {\\displaystyle \\gamma <1}\n  \n by the Banach fixed-point theorem.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0066", "text": "Policy iteration\nIn policy iteration, one first performs Value Determination by solving for \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n from the linear system described in step one, then performs Policy Improvement by computing \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0067", "text": "\u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n as in step two, then repeats both steps until the policy converges. (Policy iteration was invented by Howard to optimize Sears catalogue mailing, which he had been optimizing using value iteration.)\nSince policy iteration effectively interleaves a linear inverse problem with a nonlinear operation, it may interpreted as a type of relaxation method.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0068", "text": "This variant has the advantage that there is a definite stopping condition. Since there is a unique solution \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n for each policy \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n, the algorithm is completed once the Policy Improvement produces the same policy twice consecutively.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0069", "text": "While there are situations where policy iteration may be faster than value iteration (e.g. when the action space is significantly larger than the state space), policy iteration is usually slower than value iteration for a large number of possible states.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0070", "text": "Modified policy iteration\nIn modified policy iteration (van Nunen 1976; Puterman & Shin 1978), step one is repeated several times, and then step two is performed once. Then step one is again repeated several times and so on.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0071", "text": "Prioritized sweeping\nIn this variant, the steps are preferentially applied to states which are in some way important \u2013 whether based on the algorithm (there were large changes in \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n or \n  \n    \n      \n        \u03c0\n      \n    \n    {\\displaystyle \\pi }\n  \n around those states recently) or based on use (those states are near the starting state, or otherwise of interest to the person or program using the algorithm).", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0072", "text": "Computational complexity", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0073", "text": "Algorithms for finding optimal policies with time complexity polynomial in the size of the problem representation exist for finite MDPs. Thus, decision problems based on MDPs are in computational complexity class P. However, due to the curse of dimensionality, the size of the problem representation is often exponential in the number of state and action variables, limiting exact solution techniques to problems that have a compact representation", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0074", "text": ". In practice, online planning techniques such as Monte Carlo tree search can find useful solutions in larger problems, and, in theory, it is possible to construct online planning algorithms that can find an arbitrarily near-optimal policy with no computational complexity dependence on the size of the state space.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0075", "text": "Extensions and generalizations\nA Markov decision process is a stochastic game with only one player.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0076", "text": "Partial observability\nThe solution above assumes that the state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n is known when action is to be taken; otherwise \n  \n    \n      \n        \u03c0\n        (\n        s\n        )\n      \n    \n    {\\displaystyle \\pi (s)}\n  \n cannot be calculated. When this assumption is not true, the problem is called a partially observable Markov decision process or POMDP.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0077", "text": "Constrained Markov decision processes\nConstrained Markov decision processes (CMDPS) are extensions to Markov decision process (MDPs). There are three fundamental differences between MDPs and CMDPs.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0078", "text": "There are multiple costs incurred after applying an action instead of one.\nCMDPs are solved with linear programs only, and dynamic programming does not work.\nThe final policy depends on the starting state.\nThe method of Lagrange multipliers applies to CMDPs.\nMany Lagrangian-based algorithms have been developed.\n\nNatural policy gradient primal-dual method.\nThere are a number of applications for CMDPs. It has recently been used in motion planning scenarios in robotics.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0079", "text": "Continuous-time Markov decision process", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0080", "text": "In discrete-time Markov Decision Processes, decisions are made at discrete time intervals. However, for continuous-time Markov decision processes, decisions can be made at any time the decision maker chooses. In comparison to discrete-time Markov decision processes, continuous-time Markov decision processes can better model the decision-making process for a system that has continuous dynamics, i.e., the system dynamics are defined by ordinary differential equations (ODEs)", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0081", "text": ". This modelling framework can be applied to areas such as queueing systems, epidemic processes, and population processes.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0082", "text": "Like the discrete-time Markov decision processes, in continuous-time Markov decision processes the agent aims to find the optimal policy that would maximize the expected cumulative reward. The key difference with the standard case is that, due to the continuous nature of the time variable, summation is replaced by an integral:", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0083", "text": "max\n        \n          E\n          \n            \u03c0\n          \n        \n        \u2061\n        \n          [\n          \n            \n              \n              \n                \n                  \u222b\n                  \n                    0\n                  \n                  \n                    \u221e\n                  \n                \n                \n                  \u03b3\n                  \n                    t\n                  \n                \n                r", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0084", "text": "r\n                (\n                s\n                (\n                t\n                )\n                ,\n                \u03c0\n                (\n                s\n                (\n                t\n                )\n                )\n                )\n                \n                d\n                t\n                \n              \n              |\n            \n            \n              s\n              \n                0", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0085", "text": "0\n              \n            \n          \n          ]\n        \n      \n    \n    {\\displaystyle \\max \\operatorname {E} _{\\pi }\\left[\\left.\\int _{0}^{\\infty }\\gamma ^{t}r(s(t),\\pi (s(t)))\\,dt\\;\\right|s_{0}\\right]}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0086", "text": "where \n  \n    \n      \n        0\n        \u2264\n        \u03b3\n        <\n        1.\n      \n    \n    {\\displaystyle 0\\leq \\gamma <1.}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0087", "text": "Discrete space: Linear programming formulation", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0088", "text": "If the state space and action space are finite, we could use linear programming to find the optimal policy, which was one of the earliest approaches applied. Here we only consider the ergodic model, which means our continuous-time MDP becomes an ergodic continuous-time Markov chain under a stationary policy. Under this assumption, although the decision maker can make a decision at any time in the current state, there is no benefit in taking multiple actions", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0089", "text": ". It is better to take an action only at the time when system is transitioning from the current state to another state. Under some conditions, if our optimal value function", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0090", "text": "V\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle V^{*}}\n  \n is independent of state \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n, we will have the following inequality:", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0091", "text": "g\n        \u2265\n        R\n        (\n        i\n        ,\n        a\n        )\n        +\n        \n          \u2211\n          \n            j\n            \u2208\n            S\n          \n        \n        q\n        (\n        j\n        \u2223\n        i\n        ,\n        a\n        )\n        h\n        (\n        j\n        )\n        \n        \u2200\n        i\n        \u2208\n        S\n        \n           and \n        \n        a\n        \u2208\n        A\n        (\n        i\n        )", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0092", "text": "(\n        i\n        )\n      \n    \n    {\\displaystyle g\\geq R(i,a)+\\sum _{j\\in S}q(j\\mid i,a)h(j)\\quad \\forall i\\in S{\\text{ and }}a\\in A(i)}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0093", "text": "If there exists a function \n  \n    \n      \n        h\n      \n    \n    {\\displaystyle h}\n  \n, then \n  \n    \n      \n        \n          \n            \n              \n                V\n                \u00af\n              \n            \n          \n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle {\\bar {V}}^{*}}\n  \n will be the smallest \n  \n    \n      \n        g\n      \n    \n    {\\displaystyle g}\n  \n satisfying the above equation. In order to find", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0094", "text": "V\n                \u00af\n              \n            \n          \n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle {\\bar {V}}^{*}}\n  \n, we could use the following linear programming model:", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0095", "text": "Primal linear program(P-LP)", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0096", "text": "Minimize\n                \n                \n              \n              \n                g\n              \n            \n            \n              \n                \n                  s.t\n                \n                \n              \n              \n                g\n                \u2212\n                \n                  \u2211\n                  \n                    j\n                    \u2208", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0097", "text": "j\n                    \u2208\n                    S\n                  \n                \n                q\n                (\n                j\n                \u2223\n                i\n                ,\n                a\n                )\n                h\n                (\n                j\n                )\n                \u2265\n                R\n                (\n                i\n                ,\n                a\n                )\n                \n                \n                \u2200", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0098", "text": "\u2200\n                i\n                \u2208\n                S\n                ,\n                \n                a\n                \u2208\n                A\n                (\n                i\n                )\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}{\\text{Minimize}}\\quad &g\\\\{\\text{s.t}}\\quad &g-\\sum _{j\\in S}q(j\\mid i,a)h(j)\\geq R(i,a)\\,\\,\\forall i\\in S,\\,a\\in A(i)\\end{aligned}}}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0099", "text": "Dual linear program(D-LP)", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0100", "text": "Maximize\n                \n              \n              \n                \n                \n                  \u2211\n                  \n                    i\n                    \u2208\n                    S\n                  \n                \n                \n                  \u2211\n                  \n                    a\n                    \u2208\n                    A\n                    (\n                    i", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0101", "text": "(\n                    i\n                    )\n                  \n                \n                R\n                (\n                i\n                ,\n                a\n                )\n                y\n                (\n                i\n                ,\n                a\n                )\n              \n            \n            \n              \n                \n                  s.t.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0102", "text": "\u2211\n                  \n                    i\n                    \u2208\n                    S\n                  \n                \n                \n                  \u2211\n                  \n                    a\n                    \u2208\n                    A\n                    (\n                    i\n                    )\n                  \n                \n                q\n                (\n                j\n                \u2223", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0103", "text": "j\n                \u2223\n                i\n                ,\n                a\n                )\n                y\n                (\n                i\n                ,\n                a\n                )\n                =\n                0\n                \n                \u2200\n                j\n                \u2208\n                S\n                ,\n              \n            \n            \n              \n              \n                \n                \n                  \u2211", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0104", "text": "\u2211\n                  \n                    i\n                    \u2208\n                    S\n                  \n                \n                \n                  \u2211\n                  \n                    a\n                    \u2208\n                    A\n                    (\n                    i\n                    )\n                  \n                \n                y\n                (\n                i\n                ,\n                a\n                )", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0105", "text": "a\n                )\n                =\n                1\n                ,\n              \n            \n            \n              \n              \n                y\n                (\n                i\n                ,\n                a\n                )\n                \u2265\n                0\n                \n                \u2200\n                a\n                \u2208\n                A\n                (\n                i\n                )\n                \n                   and", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0106", "text": "and \n                \n                \u2200\n                i\n                \u2208\n                S\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}{\\text{Maximize}}&\\sum _{i\\in S}\\sum _{a\\in A(i)}R(i,a)y(i,a)\\\\{\\text{s.t.}}&\\sum _{i\\in S}\\sum _{a\\in A(i)}q(j\\mid i,a)y(i,a)=0\\quad \\forall j\\in S,\\\\&\\sum _{i\\in S}\\sum _{a\\in A(i)}y(i,a)=1,\\\\&y(i,a)\\geq 0\\qquad \\forall a\\in A(i){\\text{ and }}\\forall i\\in S\\end{aligned}}}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0107", "text": "y\n        (\n        i\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle y(i,a)}\n  \n is a feasible solution to the D-LP if \n  \n    \n      \n        y\n        (\n        i\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle y(i,a)}\n  \n is nonnative and satisfied the constraints in the D-LP problem. A feasible solution \n  \n    \n      \n        \n          y\n          \n            \u2217\n          \n        \n        (\n        i\n        ,\n        a\n        )", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0108", "text": "i\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle y^{*}(i,a)}\n  \n to the D-LP is said to be an optimal solution if", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0109", "text": "\u2211\n                  \n                    i\n                    \u2208\n                    S\n                  \n                \n                \n                  \u2211\n                  \n                    a\n                    \u2208\n                    A\n                    (\n                    i\n                    )\n                  \n                \n                R\n                (\n                i", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0110", "text": "(\n                i\n                ,\n                a\n                )\n                \n                  y\n                  \n                    \u2217\n                  \n                \n                (\n                i\n                ,\n                a\n                )\n                \u2265\n                \n                  \u2211\n                  \n                    i\n                    \u2208\n                    S", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0111", "text": "\u2211\n                  \n                    a\n                    \u2208\n                    A\n                    (\n                    i\n                    )\n                  \n                \n                R\n                (\n                i\n                ,\n                a\n                )\n                y\n                (\n                i\n                ,\n                a\n                )", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0112", "text": "{\\displaystyle {\\begin{aligned}\\sum _{i\\in S}\\sum _{a\\in A(i)}R(i,a)y^{*}(i,a)\\geq \\sum _{i\\in S}\\sum _{a\\in A(i)}R(i,a)y(i,a)\\end{aligned}}}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0113", "text": "for all feasible solution \n  \n    \n      \n        y\n        (\n        i\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle y(i,a)}\n  \n to the D-LP. Once we have found the optimal solution \n  \n    \n      \n        \n          y\n          \n            \u2217\n          \n        \n        (\n        i\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle y^{*}(i,a)}\n  \n, we can use it to establish the optimal policies.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0114", "text": "Continuous space: Hamilton\u2013Jacobi\u2013Bellman equation\nIn continuous-time MDP, if the state space and action space are continuous, the optimal criterion could be found by solving the Hamilton\u2013Jacobi\u2013Bellman (HJB) partial differential equation. In order to discuss the HJB equation, we need to reformulate\nour problem", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0115", "text": "V\n                (\n                s\n                (\n                0\n                )\n                ,\n                0\n                )\n                =", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0116", "text": "max\n                  \n                    a\n                    (\n                    t\n                    )\n                    =\n                    \u03c0\n                    (\n                    s\n                    (\n                    t\n                    )\n                    )\n                  \n                \n                \n                  \u222b", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0117", "text": "\u222b\n                  \n                    0\n                  \n                  \n                    T\n                  \n                \n                r\n                (\n                s\n                (\n                t\n                )\n                ,\n                a\n                (\n                t\n                )\n                )\n                \n                d\n                t\n                +\n                D\n                [\n                s", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0118", "text": "[\n                s\n                (\n                T\n                )\n                ]\n              \n            \n            \n              \n                \n                  s.t.\n                \n                \n              \n              \n                \n                  \n                    \n                      d\n                      s\n                      (\n                      t\n                      )", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0119", "text": "d\n                      t\n                    \n                  \n                \n                =\n                f\n                [\n                t\n                ,\n                s\n                (\n                t\n                )\n                ,\n                a\n                (\n                t\n                )\n                ]", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0120", "text": "{\\displaystyle {\\begin{aligned}V(s(0),0)={}&\\max _{a(t)=\\pi (s(t))}\\int _{0}^{T}r(s(t),a(t))\\,dt+D[s(T)]\\\\{\\text{s.t.}}\\quad &{\\frac {ds(t)}{dt}}=f[t,s(t),a(t)]\\end{aligned}}}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0121", "text": "D\n        (\n        \u22c5\n        )\n      \n    \n    {\\displaystyle D(\\cdot )}\n  \n is the terminal reward function, \n  \n    \n      \n        s\n        (\n        t\n        )\n      \n    \n    {\\displaystyle s(t)}\n  \n is the system state vector, \n  \n    \n      \n        a\n        (\n        t\n        )\n      \n    \n    {\\displaystyle a(t)}\n  \n is the system control vector we try to find. \n  \n    \n      \n        f\n        (\n        \u22c5\n        )\n      \n    \n    {\\displaystyle f(\\cdot )}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0122", "text": "{\\displaystyle f(\\cdot )}\n  \n shows how the state vector changes over time. The Hamilton\u2013Jacobi\u2013Bellman equation is as follows:", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0123", "text": "0\n        =\n        \n          max\n          \n            a\n          \n        \n        (\n        r\n        (\n        t\n        ,\n        s\n        ,\n        a\n        )\n        +\n        \n          \n            \n              \u2202\n              V\n              (\n              t\n              ,\n              s\n              )\n            \n            \n              \u2202\n              s\n            \n          \n        \n        f\n        (\n        t\n        ,\n        s\n        ,", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0124", "text": "(\n        t\n        ,\n        s\n        ,\n        a\n        )\n        )\n      \n    \n    {\\displaystyle 0=\\max _{a}(r(t,s,a)+{\\frac {\\partial V(t,s)}{\\partial s}}f(t,s,a))}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0125", "text": "We could solve the equation to find the optimal value function \n  \n    \n      \n        \n          V\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle V^{*}}\n  \n, which in turns yield the optimal control at any time \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n, \n  \n    \n      \n        a\n        (\n        t\n        )\n      \n    \n    {\\displaystyle a(t)}\n  \n through", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0126", "text": "a\n        (\n        t\n        )\n        =\n        \n          \n            argmax\n            a\n          \n        \n        (\n        r\n        (\n        t\n        ,\n        s\n        ,\n        a\n        )\n        +\n        \n          \n            \n              \u2202\n              \n                V\n                \n                  \u2217\n                \n              \n              (\n              t\n              ,\n              s\n              )", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0127", "text": ")\n            \n            \n              \u2202\n              s\n            \n          \n        \n        f\n        (\n        t\n        ,\n        s\n        ,\n        a\n        )\n        )\n        .\n      \n    \n    {\\displaystyle a(t)={\\underset {a}{\\text{argmax}}}(r(t,s,a)+{\\frac {\\partial V^{*}(t,s)}{\\partial s}}f(t,s,a)).}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0128", "text": "Reinforcement learning\nReinforcement learning is an interdisciplinary area of machine learning and optimal control that has, as main objective, finding an approximately optimal policy for MDPs where transition probabilities and rewards are unknown.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0129", "text": "Reinforcement learning can solve Markov-Decision processes without explicit specification of the transition probabilities which are instead needed to perform policy iteration. In this setting, transition probabilities and rewards must be learned from experience, i.e. by letting an agent interact with the MDP for a given number of steps. Both on a theoretical and on a practical level, effort is put in maximizing the sample efficiency, i.e", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0130", "text": ". minimimizing the number of samples needed to learn a policy whose performance is", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0131", "text": "\u03b5\n        \u2212\n      \n    \n    {\\displaystyle \\varepsilon -}\n  \nclose to the optimal one (due to the stochastic nature of the process, learning the optimal policy with a finite number of samples is, in general, impossible).", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0132", "text": "Reinforcement Learning for discrete MDPs\nFor the purpose of this section, it is useful to define a further function, which corresponds to taking the action \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n and then continuing optimally (or according to whatever policy one currently has):", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0133", "text": "Q\n        (\n        s\n        ,\n        a\n        )\n        =\n        \n          \u2211\n          \n            \n              s\n              \u2032\n            \n          \n        \n        \n          P\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032\n        \n        )\n        (\n        \n          R\n          \n            a\n          \n        \n        (\n        s\n        ,\n        \n          s\n          \u2032", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0134", "text": ",\n        \n          s\n          \u2032\n        \n        )\n        +\n        \u03b3\n        V\n        (\n        \n          s\n          \u2032\n        \n        )\n        )\n        .\n         \n      \n    \n    {\\displaystyle \\ Q(s,a)=\\sum _{s'}P_{a}(s,s')(R_{a}(s,s')+\\gamma V(s')).\\ }", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0135", "text": "While this function is also unknown, experience during learning is based on \n  \n    \n      \n        (\n        s\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle (s,a)}\n  \n pairs (together with the outcome \n  \n    \n      \n        \n          s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n; that is, \"I was in state \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n and I tried doing \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n and", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0136", "text": "{\\displaystyle a}\n  \n and \n  \n    \n      \n        \n          s\n          \u2032\n        \n      \n    \n    {\\displaystyle s'}\n  \n happened\"). Thus, one has an array \n  \n    \n      \n        Q\n      \n    \n    {\\displaystyle Q}\n  \n and uses experience to update it directly. This is known as Q-learning.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0137", "text": "Other scopes\nLearning automata", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0138", "text": "Another application of MDP process in machine learning theory is called learning automata. This is also one type of reinforcement learning if the environment is stochastic. The first detail learning automata paper is surveyed by Narendra and Thathachar (1974), which were originally described explicitly as finite-state automata. Similar to reinforcement learning, a learning automata algorithm also has the advantage of solving the problem when probability or rewards are unknown", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0139", "text": ". The difference between learning automata and Q-learning is that the former technique omits the memory of Q-values, but updates the action probability directly to find the learning result. Learning automata is a learning scheme with a rigorous proof of convergence.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0140", "text": "In learning automata theory, a stochastic automaton consists of:", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0141", "text": "a set x of possible inputs,\na set \u03a6 = { \u03a61, ..., \u03a6s } of possible internal states,\na set \u03b1 = { \u03b11, ..., \u03b1r } of possible outputs, or actions, with r \u2264 s,\nan initial state probability vector p(0) = \u226a p1(0), ..., ps(0) \u226b,\na computable function A which after each time step t generates p(t + 1) from p(t), the current input, and the current state, and\na function G: \u03a6 \u2192 \u03b1 which generates the output at each time step.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0142", "text": "The states of such an automaton correspond to the states of a \"discrete-state discrete-parameter Markov process\". At each time step t = 0,1,2,3,..., the automaton reads an input from its environment, updates P(t) to P(t + 1) by A, randomly chooses a successor state according to the probabilities P(t + 1) and outputs the corresponding action. The automaton's environment, in turn, reads the action and sends the next input to the automaton.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0143", "text": "Category theoretic interpretation\nOther than the rewards, a Markov decision process \n  \n    \n      \n        (\n        S\n        ,\n        A\n        ,\n        P\n        )\n      \n    \n    {\\displaystyle (S,A,P)}\n  \n can be understood in terms of Category theory. Namely, let \n  \n    \n      \n        \n          \n            A\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {A}}}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0144", "text": "{\\displaystyle {\\mathcal {A}}}\n  \n denote the free monoid with generating set A. Let Dist denote the Kleisli category of the Giry monad. Then a functor \n  \n    \n      \n        \n          \n            A\n          \n        \n        \u2192\n        \n          D\n          i\n          s\n          t\n        \n      \n    \n    {\\displaystyle {\\mathcal {A}}\\to \\mathbf {Dist} }\n  \n encodes both the set S of states and the probability function P.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0145", "text": "In this way, Markov decision processes could be generalized from monoids (categories with one object) to arbitrary categories. One can call the result \n  \n    \n      \n        (\n        \n          \n            C\n          \n        \n        ,\n        F\n        :\n        \n          \n            C\n          \n        \n        \u2192\n        \n          D\n          i\n          s\n          t\n        \n        )\n      \n    \n    {\\displaystyle ({\\mathcal {C}},F:{\\mathcal {C}}\\to \\mathbf {Dist} )}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0146", "text": "a context-dependent Markov decision process, because moving from one object to another in \n  \n    \n      \n        \n          \n            C\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {C}}}\n  \n changes the set of available actions and the set of possible states.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0147", "text": "Alternative notations\nThe terminology and notation for MDPs are not entirely settled. There are two main streams \u2014 one focuses on maximization problems from contexts like economics, using the terms action, reward, value, and calling the discount factor \u03b2 or \u03b3, while the other focuses on minimization problems from engineering and navigation, using the terms control, cost, cost-to-go, and calling the discount factor \u03b1. In addition, the notation for the transition probability varies.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0148", "text": "In addition, transition probability is sometimes written \n  \n    \n      \n        Pr\n        (\n        s\n        ,\n        a\n        ,\n        \n          s\n          \u2032\n        \n        )\n      \n    \n    {\\displaystyle \\Pr(s,a,s')}\n  \n, \n  \n    \n      \n        Pr\n        (\n        \n          s\n          \u2032\n        \n        \u2223\n        s\n        ,\n        a\n        )\n      \n    \n    {\\displaystyle \\Pr(s'\\mid s,a)}\n  \n or, rarely, \n  \n    \n      \n        \n          p", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0149", "text": "p\n          \n            \n              s\n              \u2032\n            \n            s\n          \n        \n        (\n        a\n        )\n        .\n      \n    \n    {\\displaystyle p_{s's}(a).}", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0150", "text": "See also\nReferences\nSources\nBellman, R. (1957), Dynamic Programming, Princeton University Press, ISBN 978-0-486-42809-3 {{citation}}: ISBN / Date incompatibility (help). Dover paperback edition (2003)", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0151", "text": "Further reading\nBellman., R. E. (2003) [1957]. Dynamic Programming (Dover paperback ed.). Princeton, NJ: Princeton University Press. ISBN 978-0-486-42809-3.\nBertsekas, D. (1995). Dynamic Programming and Optimal Control. Vol. 2. MA: Athena.\nDerman, C. (1970). Finite state Markovian decision processes. Academic Press.\nFeinberg, E.A.; Shwartz, A., eds. (2002). Handbook of Markov Decision Processes. Boston, MA: Kluwer. ISBN 9781461508052.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0152", "text": "Guo, X.; Hern\u00e1ndez-Lerma, O. (2009). Continuous-Time Markov Decision Processes. Stochastic Modelling and Applied Probability. Springer. ISBN 9783642025464.\nMeyn, S. P. (2007). Control Techniques for Complex Networks. Cambridge University Press. ISBN 978-0-521-88441-9. Archived from the original on 19 June 2010. Appendix contains abridged \"Meyn & Tweedie\". Archived from the original on 18 December 2012.\nPuterman., M. L. (1994). Markov Decision Processes. Wiley.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Markov_decision_process_0153", "text": "Ross, S. M. (1983). Introduction to stochastic dynamic programming (PDF). Academic press. Archived from the original (PDF) on 2022-03-04. Retrieved 2019-01-19.\nSutton, R. S.; Barto, A. G. (2017). Reinforcement Learning: An Introduction. Cambridge, MA: The MIT Press.\nTijms., H.C. (2003). A First Course in Stochastic Models. Wiley. ISBN 9780470864289.", "title": "Markov decision process", "url": "https://en.wikipedia.org/wiki/Markov_decision_process"}, {"id": "rec_Computer_vision_0000", "text": "Computer vision tasks include methods for acquiring, processing, analyzing, and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g. in the form of decisions. \"Understanding\" in this context signifies the transformation of visual images into descriptions of the world that make sense to thought processes and can elicit appropriate action", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0001", "text": ". This image understanding can be seen as the disentangling of symbolic information from image data using models constructed with the aid of geometry, physics, statistics, and learning theory.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0002", "text": "The scientific discipline of computer vision is concerned with the theory behind artificial systems that extract information from images. Image data can take many forms, such as video sequences, views from multiple cameras, multi-dimensional data from a 3D scanner, 3D point clouds from LiDaR sensors, or medical scanning devices. The technological discipline of computer vision seeks to apply its theories and models to the construction of computer vision systems.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0003", "text": "Subdisciplines of computer vision include scene reconstruction, object detection, event detection, activity recognition, video tracking, object recognition, 3D pose estimation, learning, indexing, motion estimation, visual servoing, 3D scene modeling, and image restoration.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0004", "text": "Definition", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0005", "text": "Computer vision is an interdisciplinary field that deals with how computers can be made to gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to automate tasks that the human visual system can do. \"Computer vision is concerned with the automatic extraction, analysis, and understanding of useful information from a single image or a sequence of images", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0006", "text": ". It involves the development of a theoretical and algorithmic basis to achieve automatic visual understanding.\" As a scientific discipline, computer vision is concerned with the theory behind artificial systems that extract information from images. The image data can take many forms, such as video sequences, views from multiple cameras, or multi-dimensional data from a medical scanner", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0007", "text": ". As a technological discipline, computer vision seeks to apply its theories and models for the construction of computer vision systems. Machine vision refers to a systems engineering discipline, especially in the context of factory automation. In more recent times, the terms computer vision and machine vision have converged to a greater degree.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0008", "text": "History\nIn the late 1960s, computer vision began at universities that were pioneering artificial intelligence. It was meant to mimic the human visual system as a stepping stone to endowing robots with intelligent behavior. In 1966, it was believed that this could be achieved through an undergraduate summer project, by attaching a camera to a computer and having it \"describe what it saw\".", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0009", "text": "What distinguished computer vision from the prevalent field of digital image processing at that time was a desire to extract three-dimensional structure from images with the goal of achieving full scene understanding", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0010", "text": ". Studies in the 1970s formed the early foundations for many of the computer vision algorithms that exist today, including extraction of edges from images, labeling of lines, non-polyhedral and polyhedral modeling, representation of objects as interconnections of smaller structures, optical flow, and motion estimation.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0011", "text": "The next decade saw studies based on more rigorous mathematical analysis and quantitative aspects of computer vision. These include the concept of scale-space, the inference of shape from various cues such as shading, texture and focus, and contour models known as snakes. Researchers also realized that many of these mathematical concepts could be treated within the same optimization framework as regularization and Markov random fields.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0012", "text": "By the 1990s, some of the previous research topics became more active than others. Research in projective 3-D reconstructions led to better understanding of camera calibration. With the advent of optimization methods for camera calibration, it was realized that a lot of the ideas were already explored in bundle adjustment theory from the field of photogrammetry. This led to methods for sparse 3-D reconstructions of scenes from multiple images", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0013", "text": ". Progress was made on the dense stereo correspondence problem and further multi-view stereo techniques. At the same time, variations of graph cut were used to solve image segmentation. This decade also marked the first time statistical learning techniques were used in practice to recognize faces in images (see Eigenface). Toward the end of the 1990s, a significant change came about with the increased interaction between the fields of computer graphics and computer vision", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0014", "text": ". This included image-based rendering, image morphing, view interpolation, panoramic image stitching and early light-field rendering.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0015", "text": "Recent work has seen the resurgence of feature-based methods used in conjunction with machine learning techniques and complex optimization frameworks. \nThe advancement of Deep Learning techniques has brought further life to the field of computer vision. The accuracy of deep learning algorithms on several benchmark computer vision data sets for tasks ranging from classification, segmentation and optical flow has surpassed prior methods.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0016", "text": "Related fields\nSolid-state physics", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0017", "text": "Solid-state physics is another field that is closely related to computer vision. Most computer vision systems rely on image sensors, which detect electromagnetic radiation, which is typically in the form of either visible, infrared or ultraviolet light. The sensors are designed using quantum physics. The process by which light interacts with surfaces is explained using physics. Physics explains the behavior of optics which are a core part of most imaging systems", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0018", "text": ". Sophisticated image sensors even require quantum mechanics to provide a complete understanding of the image formation process. Also, various measurement problems in physics can be addressed using computer vision, for example, motion in fluids.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0019", "text": "Neurobiology", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0020", "text": "Neurobiology has greatly influenced the development of computer vision algorithms. Over the last century, there has been an extensive study of eyes, neurons, and brain structures devoted to the processing of visual stimuli in both humans and various animals. This has led to a coarse yet convoluted description of how natural vision systems operate in order to solve certain vision-related tasks", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0021", "text": ". These results have led to a sub-field within computer vision where artificial systems are designed to mimic the processing and behavior of biological systems at different levels of complexity. Also, some of the learning-based methods developed within computer vision (e.g. neural net and deep learning based image and feature analysis and classification) have their background in neurobiology", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0022", "text": ".  The Neocognitron, a neural network developed in the 1970s by Kunihiko Fukushima, is an early example of computer vision taking direct inspiration from neurobiology, specifically the primary visual cortex.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0023", "text": "Some strands of computer vision research are closely related to the study of biological vision\u2014indeed, just as many strands of AI research are closely tied with research into human intelligence and the use of stored knowledge to interpret, integrate, and utilize visual information. The field of biological vision studies and models the physiological processes behind visual perception in humans and other animals", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0024", "text": ". Computer vision, on the other hand, develops and describes the algorithms implemented in software and hardware behind artificial vision systems. An interdisciplinary exchange between biological and computer vision has proven fruitful for both fields.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0025", "text": "Signal processing", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0026", "text": "Yet another field related to computer vision is signal processing. Many methods for processing one-variable signals, typically temporal signals, can be extended in a natural way to the processing of two-variable signals or multi-variable signals in computer vision. However, because of the specific nature of images, there are many methods developed within computer vision that have no counterpart in the processing of one-variable signals", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0027", "text": ". Together with the multi-dimensionality of the signal, this defines a subfield in signal processing as a part of computer vision.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0028", "text": "Robotic navigation\nRobot navigation sometimes deals with autonomous path planning or deliberation for robotic systems to navigate through an environment. A detailed understanding of these environments is required to navigate through them. Information about the environment could be provided by a computer vision system, acting as a vision sensor and providing high-level information about the environment and the robot", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0029", "text": "Visual computing\nOther fields", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0030", "text": "Besides the above-mentioned views on computer vision, many of the related research topics can also be studied from a purely mathematical point of view. For example, many methods in computer vision are based on statistics, optimization or geometry", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0031", "text": ". Finally, a significant part of the field is devoted to the implementation aspect of computer vision; how existing methods can be realized in various combinations of software and hardware, or how these methods can be modified in order to gain processing speed without losing too much performance. Computer vision is also used in fashion eCommerce, inventory management, patent search, furniture, and the beauty industry.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0032", "text": "Distinctions", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0033", "text": "The fields most closely related to computer vision are image processing, image analysis and machine vision. There is a significant overlap in the range of techniques and applications that these cover. This implies that the basic techniques that are used and developed in these fields are similar, something which can be interpreted as there is only one field with different names", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0034", "text": ". On the other hand, it appears to be necessary for research groups, scientific journals, conferences, and companies to present or market themselves as belonging specifically to one of these fields and, hence, various characterizations which distinguish each of the fields from the others have been presented", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0035", "text": ". In image processing, the input and output are both images, whereas in computer vision, the input is an image or video, and the output could be an enhanced image, an analysis of the image's content, or even a system's behavior based on that analysis.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0036", "text": "Computer graphics produces image data from 3D models, and computer vision often produces 3D models from image data. There is also a trend towards a combination of the two disciplines, e.g., as explored in augmented reality.\nThe following characterizations appear relevant but should not be taken as universally accepted:", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0037", "text": "Image processing and image analysis tend to focus on 2D images, how to transform one image to another, e.g., by pixel-wise operations such as contrast enhancement, local operations such as edge extraction or noise removal, or geometrical transformations such as rotating the image. This characterization implies that image processing/analysis neither requires assumptions nor produces interpretations about the image content.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0038", "text": "Computer vision includes 3D analysis from 2D images. This analyzes the 3D scene projected onto one or several images, e.g., how to reconstruct structure or other information about the 3D scene from one or several images. Computer vision often relies on more or less complex assumptions about the scene depicted in an image.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0039", "text": "Machine vision is the process of applying a range of technologies and methods to provide imaging-based automatic inspection, process control, and robot guidance in industrial applications. Machine vision tends to focus on applications, mainly in manufacturing, e.g., vision-based robots and systems for vision-based inspection, measurement, or picking (such as bin picking)", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0040", "text": ". This implies that image sensor technologies and control theory often are integrated with the processing of image data to control a robot and that real-time processing is emphasized by means of efficient implementations in hardware and software. It also implies that external conditions such as lighting can be and are often more controlled in machine vision than they are in general computer vision, which can enable the use of different algorithms.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0041", "text": "There is also a field called imaging which primarily focuses on the process of producing images, but sometimes also deals with the processing and analysis of images. For example, medical imaging includes substantial work on the analysis of image data in medical applications. Progress in convolutional neural networks (CNNs) has improved the accurate detection of disease in medical images, particularly in cardiology, pathology, dermatology, and radiology.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0042", "text": "Finally, pattern recognition is a field that uses various methods to extract information from signals in general, mainly based on statistical approaches and artificial neural networks. A significant part of this field is devoted to applying these methods to image data.\nPhotogrammetry also overlaps with computer vision, e.g., stereophotogrammetry vs. computer stereo vision.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0043", "text": "Applications", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0044", "text": "Applications range from tasks such as industrial machine vision systems which, say, inspect bottles speeding by on a production line, to research into artificial intelligence and computers or robots that can comprehend the world around them. The computer vision and machine vision fields have significant overlap. Computer vision covers the core technology of automated image analysis which is used in many fields", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0045", "text": ". Machine vision usually refers to a process of combining automated image analysis with other methods and technologies to provide automated inspection and robot guidance in industrial applications. In many computer-vision applications, computers are pre-programmed to solve a particular task, but methods based on learning are now becoming increasingly common. Examples of applications of computer vision include systems for:", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0046", "text": "Automatic inspection, e.g., in manufacturing applications;\nAssisting humans in identification tasks, e.g., a species identification system;\nControlling processes, e.g., an industrial robot;\nDetecting events, e.g., for visual surveillance or people counting, e.g., in the restaurant industry;\nInteraction, e.g., as the input to a device for computer-human interaction;", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0047", "text": "MediaPipe, an open-source framework from Google for AI edge device computing, e.g., face detection, image classification, object detection;\nmonitoring agricultural crops, e.g. an open-source vision transformers model  has been developed to help farmers automatically detect strawberry diseases with 98.4% accuracy.\nModeling objects or environments, e.g., medical image analysis or topographical modeling;\nNavigation, e.g., by an autonomous vehicle or mobile robot;", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0048", "text": "Organizing information, e.g., for indexing databases of images and image sequences.\nTracking surfaces or planes in 3D coordinates for allowing Augmented Reality experiences.\nAnalyzing the condition of facilities in industry or construction.\nAutomatic real-time lip-reading for devices and apps to assist people with disabilities.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0049", "text": "For 2024, the leading areas of computer vision were industry (market size US$5.22 billion), medicine (market size US$2.6 billion), military (market size US$996.2 million).", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0050", "text": "Medicine", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0051", "text": "One of the most prominent application fields is medical computer vision, or medical image processing, characterized by the extraction of information from image data to diagnose a patient. An example of this is the detection of tumours, arteriosclerosis or other malign changes, and a variety of dental pathologies; measurements of organ dimensions, blood flow, etc. are another example", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0052", "text": ". are another example. It also supports medical research by providing new information: e.g., about the structure of the brain or the quality of medical treatments. Applications of computer vision in the medical area also include enhancement of images interpreted by humans\u2014ultrasonic images or X-ray images, for example\u2014to reduce the influence of noise.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0053", "text": "Machine vision", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0054", "text": "A second application area in computer vision is in industry, sometimes called machine vision, where information is extracted for the purpose of supporting a production process. One example is quality control where details or final products are being automatically inspected in order to find defects", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0055", "text": ". One of the most prevalent fields for such inspection is the Wafer industry in which every single Wafer is being measured and inspected for inaccuracies or defects to prevent a computer chip from coming to market in an unusable manner. Another example is a measurement of the position and orientation of details to be picked up by a robot arm. Machine vision is also heavily used in the agricultural processes to remove undesirable foodstuff from bulk material, a process called optical sorting.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0056", "text": "Military", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0057", "text": "The obvious examples are the detection of enemy soldiers or vehicles and missile guidance. More advanced systems for missile guidance send the missile to an area rather than a specific target, and target selection is made when the missile reaches the area based on locally acquired image data", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0058", "text": ". Modern military concepts, such as \"battlefield awareness\", imply that various sensors, including image sensors, provide a rich set of information about a combat scene that can be used to support strategic decisions. In this case, automatic processing of the data is used to reduce complexity and to fuse information from multiple sensors to increase reliability.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0059", "text": "Autonomous vehicles", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0060", "text": "One of the newer application areas is autonomous vehicles, which include submersibles, land-based vehicles (small robots with wheels, cars, or trucks), aerial vehicles, and unmanned aerial vehicles (UAV). The level of autonomy ranges from fully autonomous (unmanned) vehicles to vehicles where computer-vision-based systems support a driver or a pilot in various situations", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0061", "text": ". Fully autonomous vehicles typically use computer vision for navigation, e.g., for knowing where they are or mapping their environment (SLAM), for detecting obstacles. It can also be used for detecting certain task-specific events, e.g., a UAV looking for forest fires. Examples of supporting systems are obstacle warning systems in cars, cameras and LiDAR sensors in vehicles, and systems for autonomous landing of aircraft", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0062", "text": ". Several car manufacturers have demonstrated systems for autonomous driving of cars. There are ample examples of military autonomous vehicles ranging from advanced missiles to UAVs for recon missions or missile guidance. Space exploration is already being made with autonomous vehicles using computer vision, e.g., NASA's Curiosity and CNSA's Yutu-2 rover.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0063", "text": "Tactile feedback", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0064", "text": "Materials such as rubber and silicon are being used to create sensors that allow for applications such as detecting microundulations and calibrating robotic hands. Rubber can be used in order to create a mold that can be placed over a finger, inside of this mold would be multiple strain gauges. The finger mold and sensors could then be placed on top of a small sheet of rubber containing an array of rubber pins. A user can then wear the finger mold and trace a surface", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0065", "text": ". A computer can then read the data from the strain gauges and measure if one or more of the pins are being pushed upward. If a pin is being pushed upward then the computer can recognize this as an imperfection in the surface. This sort of technology is useful in order to receive accurate data on imperfections on a very large surface. Another variation of this finger mold sensor are sensors that contain a camera suspended in silicon", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0066", "text": ". The silicon forms a dome around the outside of the camera and embedded in the silicon are point markers that are equally spaced. These cameras can then be placed on devices such as robotic hands in order to allow the computer to receive highly accurate tactile data.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0067", "text": "Other application areas include:", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0068", "text": "Support of visual effects creation for cinema and broadcast, e.g., camera tracking (match moving).\nSurveillance.\nDriver drowsiness detection\nTracking and counting organisms in the biological sciences", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0069", "text": "Typical tasks\nEach of the application areas described above employ a range of computer vision tasks; more or less well-defined measurement problems or processing problems, which can be solved using a variety of methods. Some examples of typical computer vision tasks are presented below.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0070", "text": "Computer vision tasks include methods for acquiring, processing, analyzing and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g., in the forms of decisions. Understanding in this context means the transformation of visual images into descriptions of the world that can interface with other thought processes and elicit appropriate action", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0071", "text": ". This image understanding can be seen as the disentangling of symbolic information from image data using models constructed with the aid of geometry, physics, statistics, and learning theory.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0072", "text": "Recognition\nThe classical problem in computer vision, image processing, and machine vision is that of determining whether or not the image data contains some specific object, feature, or activity. Different varieties of recognition problem are described in the literature.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0073", "text": "Object recognition (also called object classification) \u2013 one or several pre-specified or learned objects or object classes can be recognized, usually together with their 2D positions in the image or 3D poses in the scene. Blippar, Google Goggles, and LikeThat provide stand-alone programs that illustrate this functionality.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0074", "text": "Identification \u2013 an individual instance of an object is recognized. Examples include identification of a specific person's face or fingerprint, identification of handwritten digits, or the identification of a specific vehicle.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0075", "text": "Detection \u2013 the image data are scanned for specific objects along with their locations. Examples include the detection of an obstacle in the car's field of view and possible abnormal cells or tissues in medical images or the detection of a vehicle in an automatic road toll system", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0076", "text": ". Detection based on relatively simple and fast computations is sometimes used for finding smaller regions of interesting image data which can be further analyzed by more computationally demanding techniques to produce a correct interpretation.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0077", "text": "Currently, the best algorithms for such tasks are based on convolutional neural networks. An illustration of their capabilities is given by the ImageNet Large Scale Visual Recognition Challenge; this is a benchmark in object classification and detection, with millions of images and 1000 object classes used in the competition. Performance of convolutional neural networks on the ImageNet tests is now close to that of humans", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0078", "text": ". The best algorithms still struggle with objects that are small or thin, such as a small ant on the stem of a flower or a person holding a quill in their hand. They also have trouble with images that have been distorted with filters (an increasingly common phenomenon with modern digital cameras). By contrast, those kinds of images rarely trouble humans. Humans, however, tend to have trouble with other issues", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0079", "text": ". For example, they are not good at classifying objects into fine-grained classes, such as the particular breed of dog or species of bird, whereas convolutional neural networks handle this with ease.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0080", "text": "Several specialized tasks based on recognition exist, such as:", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0081", "text": "Content-based image retrieval \u2013 finding all images in a larger set of images which have a specific content. The content can be specified in different ways, for example in terms of similarity relative to a target image (give me all images similar to image X) by utilizing reverse image search techniques, or in terms of high-level search criteria given as text input (give me all images which contain many houses, are taken during winter and have no cars in them).", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0082", "text": "Pose estimation \u2013 estimating the position or orientation of a specific object relative to the camera. An example application for this technique would be assisting a robot arm in retrieving objects from a conveyor belt in an assembly line situation or picking parts from a bin.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0083", "text": "Optical character recognition (OCR) \u2013 identifying characters in images of printed or handwritten text, usually with a view to encoding the text in a format more amenable to editing or indexing (e.g. ASCII). A related task is reading of 2D codes such as data matrix and QR codes.\nFacial recognition \u2013  a technology that enables the matching of faces in digital images or video frames to a face database, which is now widely used for mobile phone facelock, smart door locking, etc.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0084", "text": "Emotion recognition \u2013 a subset of facial recognition, emotion recognition refers to the process of classifying human emotions. Psychologists caution, however, that internal emotions cannot be reliably detected from faces.\nShape Recognition Technology (SRT) in people counter systems differentiating human beings (head and shoulder patterns) from objects.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0085", "text": "Human activity recognition - deals with recognizing the activity from a series of video frames, such as, if the person is picking up an object or walking.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0086", "text": "Motion analysis\nSeveral tasks relate to motion estimation, where an image sequence is processed to produce an estimate of the velocity either at each points in the image or in the 3D scene or even of the camera that produces the images. Examples of such tasks are:", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0087", "text": "Egomotion \u2013 determining the 3D rigid motion (rotation and translation) of the camera from an image sequence produced by the camera.\nTracking \u2013 following the movements of a (usually) smaller set of interest points or objects (e.g., vehicles, objects, humans or other organisms) in the image sequence. This has vast industry applications as most high-running machinery can be monitored in this way.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0088", "text": "Optical flow \u2013 to determine, for each point in the image, how that point is moving relative to the image plane, i.e., its apparent motion. This motion is a result of both how the corresponding 3D point is moving in the scene and how the camera is moving relative to the scene.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0089", "text": "Scene reconstruction", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0090", "text": "Given one or (typically) more images of a scene, or a video, scene reconstruction aims at computing a 3D model of the scene. In the simplest case, the model can be a set of 3D points. More sophisticated methods produce a complete 3D surface model. The advent of 3D imaging not requiring motion or scanning, and related processing algorithms is enabling rapid advances in this field. Grid-based 3D sensing can be used to acquire 3D images from multiple angles", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0091", "text": ". Algorithms are now available to stitch multiple 3D images together into point clouds and 3D models.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0092", "text": "Image restoration", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0093", "text": "Image restoration comes into the picture when the original image is degraded or damaged due to some external factors like lens wrong positioning, transmission interference, low lighting or motion blurs, etc., which is referred to as noise. When the images are degraded or damaged, the information to be extracted from them also gets damaged. Therefore, we need to recover or restore the image as it was intended to be", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0094", "text": ". The aim of image restoration is the removal of noise (sensor noise, motion blur, etc.) from images. The simplest possible approach for noise removal is various types of filters, such as low-pass filters or median filters. More sophisticated methods assume a model of how the local image structures look to distinguish them from noise", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0095", "text": ". By first analyzing the image data in terms of the local image structures, such as lines or edges, and then controlling the filtering based on local information from the analysis step, a better level of noise removal is usually obtained compared to the simpler approaches.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0096", "text": "An example in this field is inpainting.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0097", "text": "System methods", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0098", "text": "The organization of a computer vision system is highly application-dependent. Some systems are stand-alone applications that solve a specific measurement or detection problem, while others constitute a sub-system of a larger design which, for example, also contains sub-systems for control of mechanical actuators, planning, information databases, man-machine interfaces, etc", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0099", "text": ". The specific implementation of a computer vision system also depends on whether its functionality is pre-specified or if some part of it can be learned or modified during operation. Many functions are unique to the application. There are, however, typical functions that are found in many computer vision systems.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0100", "text": "Image acquisition \u2013 A digital image is produced by one or several image sensors, which, besides various types of light-sensitive cameras, include range sensors, tomography devices, radar, ultra-sonic cameras, etc. Depending on the type of sensor, the resulting image data is an ordinary 2D image, a 3D volume, or an image sequence", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0101", "text": ". The pixel values typically correspond to light intensity in one or several spectral bands (gray images or colour images) but can also be related to various physical measures, such as depth, absorption or reflectance of sonic or electromagnetic waves, or magnetic resonance imaging.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0102", "text": "Pre-processing \u2013 Before a computer vision method can be applied to image data in order to extract some specific piece of information, it is usually necessary to process the data in order to ensure that it satisfies certain assumptions implied by the method. Examples are:\nRe-sampling to ensure that the image coordinate system is correct.\nNoise reduction to ensure that sensor noise does not introduce false information.\nContrast enhancement to ensure that relevant information can be detected.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0103", "text": "Scale space representation to enhance image structures at locally appropriate scales.\nFeature extraction \u2013 Image features at various levels of complexity are extracted from the image data. Typical examples of such features are:\nLines, edges and ridges.\nLocalized interest points such as corners, blobs or points.\nMore complex features may be related to texture, shape, or motion.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0104", "text": "Detection/segmentation \u2013 At some point in the processing, a decision is made about which image points or regions of the image are relevant for further processing. Examples are:\nSelection of a specific set of interest points.\nSegmentation of one or multiple image regions that contain a specific object of interest.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0105", "text": "Segmentation of image into nested scene architecture comprising foreground, object groups, single objects or salient object parts (also referred to as spatial-taxon scene hierarchy), while the visual salience is often implemented as spatial and temporal attention.\nSegmentation or co-segmentation of one or multiple videos into a series of per-frame foreground masks while maintaining its temporal semantic continuity.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0106", "text": "High-level processing \u2013 At this step, the input is typically a small set of data, for example, a set of points or an image region, which is assumed to contain a specific object. The remaining processing deals with, for example:\nVerification that the data satisfies model-based and application-specific assumptions.\nEstimation of application-specific parameters, such as object pose or object size.\nImage recognition \u2013 classifying a detected object into different categories.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0107", "text": "Image registration \u2013 comparing and combining two different views of the same object.\nDecision making Making the final decision required for the application, for example:\nPass/fail on automatic inspection applications.\nMatch/no-match in recognition applications.\nFlag for further human review in medical, military, security and recognition applications.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0108", "text": "Image-understanding systems\nImage-understanding systems (IUS) include three levels of abstraction as follows: low level includes image primitives such as edges, texture elements, or regions; intermediate level includes boundaries, surfaces and volumes; and high level includes objects, scenes, or events. Many of these requirements are entirely topics for further research.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0109", "text": "The representational requirements in the designing of IUS for these levels are: representation of prototypical concepts, concept organization, spatial knowledge, temporal knowledge, scaling, and description by comparison and differentiation.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0110", "text": "While inference refers to the process of deriving new, not explicitly represented facts from currently known facts, control refers to the process that selects which of the many inference, search, and matching techniques should be applied at a particular stage of processing", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0111", "text": ". Inference and control requirements for IUS are: search and hypothesis activation, matching and hypothesis testing, generation and use of expectations, change and focus of attention, certainty and strength of belief, inference and goal satisfaction.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0112", "text": "Hardware", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0113", "text": "There are many kinds of computer vision systems; however, all of them contain these basic elements: a power source, at least one image acquisition device (camera, ccd, etc.), a processor, and control and communication cables or some kind of wireless interconnection mechanism. In addition, a practical vision system contains software, as well as a display in order to monitor the system", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0114", "text": ". Vision systems for inner spaces, as most industrial ones, contain an illumination system and may be placed in a controlled environment. Furthermore, a completed system includes many accessories, such as camera supports, cables, and connectors.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0115", "text": "Most computer vision systems use visible-light cameras passively viewing a scene at frame rates of at most 60 frames per second (usually far slower).", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0116", "text": "A few computer vision systems use image-acquisition hardware with active illumination or something other than visible light or both, such as structured-light 3D scanners, thermographic cameras, hyperspectral imagers, radar imaging, lidar scanners, magnetic resonance images, side-scan sonar, synthetic aperture sonar, etc. Such hardware captures \"images\" that are then processed often using the same computer vision algorithms used to process visible-light images.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0117", "text": "While traditional broadcast and consumer video systems operate at a rate of 30 frames per second, advances in digital signal processing and consumer graphics hardware has made high-speed image acquisition, processing, and display possible for real-time systems on the order of hundreds to thousands of frames per second. For applications in robotics, fast, real-time video systems are critically important and often can simplify the processing needed for certain algorithms", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0118", "text": ". When combined with a high-speed projector, fast image acquisition allows 3D measurement and feature tracking to be realized.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0119", "text": "Egocentric vision systems are composed of a wearable camera that automatically take pictures from a first-person perspective.\nAs of 2016, vision processing units are emerging as a new class of processors to complement CPUs and graphics processing units (GPUs) in this role.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0120", "text": "See also\nLists\nOutline of computer vision\nList of emerging technologies\nOutline of artificial intelligence", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0121", "text": "References\nFurther reading\nJames E. Dobson (2023). The Birth of Computer Vision. University of Minnesota Press. ISBN 978-1-5179-1421-9.\nDavid Marr (1982). Vision. W. H. Freeman and Company. ISBN 978-0-7167-1284-8.\nAzriel Rosenfeld; Avinash Kak (1982). Digital Picture Processing. Academic Press. ISBN 978-0-12-597301-4.\nBarghout, Lauren; Lawrence W. Lee (2003). Perceptual information processing system. U.S. Patent Application 10/618,543. ISBN 978-0-262-08159-7.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0122", "text": "Berthold K.P. Horn (1986). Robot Vision. MIT Press. ISBN 978-0-262-08159-7.\nMichael C. Fairhurst (1988). Computer Vision for robotic systems. Prentice Hall. ISBN 978-0-13-166919-2.\nOlivier Faugeras (1993). Three-Dimensional Computer Vision, A Geometric Viewpoint. MIT Press. ISBN 978-0-262-06158-2.\nTony Lindeberg (1994). Scale-Space Theory in Computer Vision. Springer. ISBN 978-0-7923-9418-1.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0123", "text": "James L. Crowley; Henrik I. Christensen, eds. (1995). Vision as Process. Springer-Verlag. ISBN 978-3-540-58143-7.\nG\u00f6sta H. Granlund; Hans Knutsson (1995). Signal Processing for Computer Vision. Kluwer Academic Publisher. ISBN 978-0-7923-9530-0.\nReinhard Klette; Karsten Schluens; Andreas Koschan (1998). Computer Vision \u2013 Three-Dimensional Data from Images. Springer, Singapore. ISBN 978-981-3083-71-4.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0124", "text": "Emanuele Trucco; Alessandro Verri (1998). Introductory Techniques for 3-D Computer Vision. Prentice Hall. ISBN 978-0-13-261108-4.\nBernd J\u00e4hne (2002). Digital Image Processing. Springer. ISBN 978-3-540-67754-3.\nRichard Hartley and Andrew Zisserman (2003). Multiple View Geometry in Computer Vision. Cambridge University Press. ISBN 978-0-521-54051-3.\nG\u00e9rard Medioni; Sing Bing Kang (2004). Emerging Topics in Computer Vision. Prentice Hall. ISBN 978-0-13-101366-7.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0125", "text": "R. Fisher; K Dawson-Howe; A. Fitzgibbon; C. Robertson; E. Trucco (2005). Dictionary of Computer Vision and Image Processing. John Wiley. ISBN 978-0-470-01526-1.\nNikos Paragios and Yunmei Chen and Olivier Faugeras (2005). Handbook of Mathematical Models in Computer Vision. Springer. ISBN 978-0-387-26371-7.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0126", "text": "Wilhelm Burger; Mark J. Burge (2007). Digital Image Processing: An Algorithmic Approach Using Java. Springer. ISBN 978-1-84628-379-6. Archived from the original on 2014-05-17. Retrieved 2007-06-13.\nPedram Azad; Tilo Gockel; R\u00fcdiger Dillmann (2008). Computer Vision \u2013 Principles and Practice. Elektor International Media BV. ISBN 978-0-905705-71-2.\nRichard Szeliski (2010). Computer Vision: Algorithms and Applications. Springer-Verlag. ISBN 978-1-84882-934-3.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0127", "text": "J. R. Parker (2011). Algorithms for Image Processing and Computer Vision (2nd ed.). Wiley. ISBN 978-0-470-64385-3.\nRichard J. Radke (2013). Computer Vision for Visual Effects. Cambridge University Press. ISBN 978-0-521-76687-6.\nNixon, Mark; Aguado, Alberto (2019). Feature Extraction and Image Processing for Computer Vision (4th ed.). Academic Press. ISBN 978-0-12-814976-8.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0128", "text": "External links\nUSC Iris computer vision conference list\nComputer vision papers on the web \u2013 a complete list of papers of the most relevant computer vision conferences.\nComputer Vision Online Archived 2011-11-30 at the Wayback Machine \u2013 news, source code, datasets and job offers related to computer vision\nCVonline \u2013 Bob Fisher's Compendium of Computer Vision.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Computer_vision_0129", "text": "British Machine Vision Association \u2013 supporting computer vision research within the UK via the BMVC and MIUA conferences, Annals of the BMVA (open-source journal), BMVA Summer School and one-day meetings\nComputer Vision Container, Joe Hoeller GitHub: Widely adopted open-source container for GPU accelerated computer vision applications. Used by researchers, universities, private companies, as well as the U.S. Gov't.", "title": "Computer vision", "url": "https://en.wikipedia.org/wiki/Computer_vision"}, {"id": "rec_Object_detection_0000", "text": "Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos. Well-researched domains of object detection include face detection and pedestrian detection. Object detection has applications in many areas of computer vision, including image retrieval and video surveillance.", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0001", "text": "Uses\nIt is widely used in computer vision tasks such as image annotation, vehicle counting, activity recognition, face detection, face recognition, video object co-segmentation. It is also used in tracking objects, for example tracking a ball during a football match, tracking movement of a cricket bat, or tracking a person in a video.", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0002", "text": "Often, the test images are sampled from a different data distribution, making the object detection task significantly more difficult. To address the challenges caused by the domain gap between training and test data, many unsupervised domain adaptation approaches have been proposed. A simple and straightforward solution for reducing the domain gap is to apply an image-to-image translation approach, such as cycle-GAN", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0003", "text": ". Among other uses, cross-domain object detection is applied in autonomous driving, where models can be trained on a vast amount of video game scenes, since the labels can be generated without manual labor.", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0004", "text": "Concept\nEvery object class has its own special features that help in classifying the class \u2013 for example all circles are round.", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0005", "text": "Object class detection uses these special features. For example, when looking for circles, objects that are at a particular distance from a point (i.e. the center) are sought. Similarly, when looking for squares, objects that are perpendicular at corners and have equal side lengths are needed.  A similar approach is used for face identification where eyes, nose, and lips can be found and features like skin color and distance between eyes can be found.", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0006", "text": "Benchmarks\nFor object localization, true positive is often measured by the thresholded intersection over union. For example, if there is a traffic sign in the image, with a bounding box drawn by a human (\"ground truth label\"), then a neural network has detected the traffic sign (a true positive) at 0.5 threshold iff it has drawn a bounding box whose IoU with the ground truth is above 0.5. Otherwise, the bounding box is a false positive.", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0007", "text": "If there is only a single ground truth bounding box, but multiple predictions, then the IoU of each prediction is calculated. The prediction with the highest IoU is a true positive if it is above threshold, else it is a false positive. All other predicted bounding boxes are false positives. If there is no prediction with an IoU above the threshold, then the ground truth label has a false negative.", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0008", "text": "For simultaneous object localization and classification, a true positive is one where the class label is correct, and the bounding box has an IoU exceeding the threshold.\nSimultaneous object localization and classification is benchmarked by the mean average precision (mAP). The average precision (AP) of the network for a class of objects is the area under the precision-recall curve as the confidence threshold is varied. The mAP is the average of AP over all classes.", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0009", "text": "Methods\nMethods for object detection generally fall into either neural network-based or non-neural approaches. For non-neural approaches, it becomes necessary to first define features using one of the methods below, then using a technique such as support vector machine (SVM) to do the classification. On the other hand, neural techniques are able to do end-to-end object detection without specifically defining features, and are typically based on convolutional neural networks (CNN).", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0010", "text": "Non-neural approaches:\nViola\u2013Jones object detection framework based on Haar features\nScale-invariant feature transform (SIFT)\nHistogram of oriented gradients (HOG) features\nNeural network approaches:\nOverFeat.\nRegion Proposals (R-CNN, Fast R-CNN, Faster R-CNN, cascade R-CNN.)\nYou Only Look Once (YOLO).\nSingle Shot MultiBox Detector (SSD) \nSingle-Shot Refinement Neural Network for Object Detection (RefineDet) \nRetina-Net \nDeformable convolutional networks", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0011", "text": "See also\nFeature detection (computer vision)\nMoving object detection\nSmall object detection\nOutline of object recognition\nTeknomo\u2013Fernandez algorithm\n\nReferences\nFurther reading\nZou, Zhengxia; Chen, Keyan; Shi, Zhenwei; Guo, Yuhong; Ye, Jieping (March 2023). \"Object Detection in 20 Years: A Survey\". Proceedings of the IEEE. 111 (3): 257\u2013276. doi:10.1109/JPROC.2023.3238524. ISSN 0018-9219.", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0012", "text": "External links\nJoshi, Snehal (2023-10-24). \"Top Object Detection Models\". hitechbpo.com.\nWeng, Lilian (2017-10-29). \"Object Detection for Dummies Part 1: Gradient Vector, HOG, and SS\". lilianweng.github.io. Retrieved 2024-09-11.\nWeng, Lilian (2017-12-15). \"Object Detection for Dummies Part 2: CNN, DPM and Overfeat\". lilianweng.github.io. Retrieved 2024-09-11.\nWeng, Lilian (2017-12-31). \"Object Detection for Dummies Part 3: R-CNN Family\". lilianweng.github.io. Retrieved 2024-09-11.", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Object_detection_0013", "text": "Weng, Lilian (2018-12-27). \"Object Detection Part 4: Fast Detection Models\". lilianweng.github.io. Retrieved 2024-09-11.\nMultiple object class detection\nSpatio-temporal action localization\nOnline Object Detection Demo\nVideo object detection and co-segmentation", "title": "Object detection", "url": "https://en.wikipedia.org/wiki/Object_detection"}, {"id": "rec_Image_segmentation_0000", "text": "In digital image processing and computer vision, image segmentation is the process of partitioning a digital image into multiple image segments, also known as image regions or image objects (sets of pixels). The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze. Image segmentation is typically used to locate objects and boundaries (lines, curves, etc.) in images", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0001", "text": ". More precisely, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0002", "text": "The result of image segmentation is a set of segments that collectively cover the entire image, or a set of contours extracted from the image (see edge detection). Each of the pixels in a region are similar with respect to some characteristic or computed property, such as color, intensity, or texture. Adjacent regions are significantly different with respect to the same characteristic(s)", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0003", "text": ". When applied to a stack of images, typical in medical imaging, the resulting contours after image segmentation can be used to create 3D reconstructions with the help of geometry reconstruction algorithms like marching cubes.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0004", "text": "Applications\nSome of the practical applications of image segmentation are:", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0005", "text": "Content-based image retrieval\nMachine vision\nMedical imaging, and imaging studies in biomedical research, including volume rendered images from computed tomography, magnetic resonance imaging, as well as volume electron microscopy techniques such as FIB-SEM.\nLocate tumors and other pathologies\nMeasure tissue volumes\nDiagnosis, study of anatomical structure\nSurgery planning\nVirtual surgery simulation\nIntra-surgery navigation\nRadiotherapy", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0006", "text": "Digital Pathology and Histopathology. Nuclei instance segmentation in variously stained whole slide images (WSIs) refers to the automatic delineation of individual cells' nuclei borders. This task is a specialized subproblem of instance segmentation, with high practical importance in biomedical research and diagnostics (e.g. cell counting, morphometric feature extraction, tumor grading, prognostic biomarker extraction)", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0007", "text": ". The modern approach to the problem is the use of deep learning architectures developed to encompass the main challenges of separating overlapping or fused nuclei. The availability of few humanly-annotated datasets poses another challenge to the problem.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0008", "text": "Object detection\nPedestrian detection\nFace detection\nBrake light detection\nLocate objects in satellite images (roads, forests, crops, etc.)\nRecognition Tasks\nFace recognition\nFingerprint recognition\nIris recognition\nProhibited Item at Airport security checkpoints\nTraffic control systems\nVideo surveillance\nVideo object co-segmentation and action localization", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0009", "text": "Several general-purpose algorithms and techniques have been developed for image segmentation. To be useful, these techniques must typically be combined with a domain's specific knowledge in order to effectively solve the domain's segmentation problems.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0010", "text": "Classes of segmentation techniques\nThere are two classes of segmentation techniques.\n\nClassical computer vision approaches\nAI based techniques", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0011", "text": "Groups of image segmentation\nSemantic segmentation is an approach detecting, for every pixel, the belonging class. For example, in a figure with many people, all the pixels belonging to persons will have the same class id and the pixels in the background will be classified as background.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0012", "text": "Instance segmentation is an approach that identifies, for every pixel, the specific belonging instance of the object. It detects each distinct object of interest in the image. For example, when each person in a figure is segmented as an individual object.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0013", "text": "Panoptic segmentation combines both semantic and instance segmentation. Like semantic segmentation, panoptic segmentation is an approach that identifies, for every pixel, the belonging class. Moreover, like in instance segmentation, panoptic segmentation distinguishes different instances of the same class.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0014", "text": "Thresholding\nThe simplest method of image segmentation is called the thresholding method. This method is based on a clip-level (or a threshold value) to turn a gray-scale image into a binary image.\nThe key of this method is to select the threshold value (or values when multiple-levels are selected). Several popular methods are used in industry including the maximum entropy method, balanced histogram thresholding, Otsu's method (maximum variance), and k-means clustering.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0015", "text": "Recently, methods have been developed for thresholding computed tomography (CT) images. The key idea is that, unlike Otsu's method, the thresholds are derived from the radiographs instead of the (reconstructed) image.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0016", "text": "New methods suggest the use of multi-dimensional, fuzzy rule-based, non-linear thresholds. In these approaches, the decision regarding each pixel's membership in a segment is based on multi-dimensional rules derived from fuzzy logic and evolutionary algorithms, considering factors such as image lighting, environment, and application.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0017", "text": "Clustering methods\nThe K-means algorithm is an iterative technique that is used to partition an image into K clusters. The basic algorithm is", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0018", "text": "Pick K cluster centers, either randomly or based on some heuristic method, for example K-means++\nAssign each pixel in the image to the cluster that minimizes the distance between the pixel and the cluster center\nRe-compute the cluster centers by averaging all of the pixels in the cluster\nRepeat steps 2 and 3 until convergence is attained (i.e. no pixels change clusters)", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0019", "text": "In this case, distance is the squared or absolute difference between a pixel and a cluster center. The difference is typically based on pixel color, intensity, texture, and location, or a weighted combination of these factors. K can be selected manually, randomly, or by a heuristic. This algorithm is guaranteed to converge, but it may not return the optimal solution. The quality of the solution depends on the initial set of clusters and the value of K.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0020", "text": "The Mean Shift algorithm is a technique that is used to partition an image into an unknown apriori number of clusters. This has the advantage of not having to start with an initial guess of such parameter which makes it a better general solution for more diverse cases.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0021", "text": "Motion and interactive segmentation\nMotion based segmentation is a technique that relies on motion in the image to perform segmentation.\nThe idea is simple: look at the differences between a pair of images. Assuming the object of interest is moving, the difference will be exactly that object.\nImproving on this idea, Kenney et al. proposed interactive segmentation [2]. They use a robot to poke objects in order to generate the motion signal necessary for motion-based segmentation.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0022", "text": "Interactive segmentation follows the interactive perception framework proposed by Dov Katz [3] and Oliver Brock [4].\nAnother technique that is based on motion is rigid motion segmentation.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0023", "text": "Compression-based methods", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0024", "text": "Compression based methods postulate that the optimal segmentation is the one that minimizes, over all possible segmentations, the coding length of the data. The connection between these two concepts is that segmentation tries to find patterns in an image and any regularity in the image can be used to compress it. The method describes each segment by its texture and boundary shape", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0025", "text": ". Each of these components is modeled by a probability distribution function and its coding length is computed as follows:", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0026", "text": "The boundary encoding leverages the fact that regions in natural images tend to have a smooth contour. This prior is used by Huffman coding to encode the difference chain code of the contours in an image. Thus, the smoother a boundary is, the shorter coding length it attains.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0027", "text": "Texture is encoded by lossy compression in a way similar to minimum description length (MDL) principle, but here the length of the data given the model is approximated by the number of samples times the entropy of the model. The texture in each region is modeled by a multivariate normal distribution whose entropy has a closed form expression. An interesting property of this model is that the estimated entropy bounds the true entropy of the data from above", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0028", "text": ". This is because among all distributions with a given mean and covariance, normal distribution has the largest entropy. Thus, the true coding length cannot be more than what the algorithm tries to minimize.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0029", "text": "For any given segmentation of an image, this scheme yields the number of bits required to encode that image based on the given segmentation. Thus, among all possible segmentations of an image, the goal is to find the segmentation which produces the shortest coding length. This can be achieved by a simple agglomerative clustering method. The distortion in the lossy compression determines the coarseness of the segmentation and its optimal value may differ for each image", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0030", "text": ". This parameter can be estimated heuristically from the contrast of textures in an image. For example, when the textures in an image are similar, such as in camouflage images, stronger sensitivity and thus lower quantization is required.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0031", "text": "Histogram-based methods\nHistogram-based methods are very efficient compared to other image segmentation methods because they typically require only one pass through the pixels. In this technique, a histogram is computed from all of the pixels in the image, and the peaks and valleys in the histogram are used to locate the clusters in the image. Color or intensity can be used as the measure.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0032", "text": "A refinement of this technique is to recursively apply the histogram-seeking method to clusters in the image in order to divide them into smaller clusters. This operation is repeated with smaller and smaller clusters until no more clusters are formed.\nOne disadvantage of the histogram-seeking method is that it may be difficult to identify significant peaks and valleys in the image.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0033", "text": "Histogram-based approaches can also be quickly adapted to apply to multiple frames, while maintaining their single pass efficiency. The histogram can be done in multiple fashions when multiple frames are considered. The same approach that is taken with one frame can be applied to multiple, and after the results are merged, peaks and valleys that were previously difficult to identify are more likely to be distinguishable", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0034", "text": ". The histogram can also be applied on a per-pixel basis where the resulting information is used to determine the most frequent color for the pixel location. This approach segments based on active objects and a static environment, resulting in a different type of segmentation useful in video tracking.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0035", "text": "Edge detection\nEdge detection is a well-developed field on its own within image processing. Region boundaries and edges are closely related,\nsince there is often a sharp adjustment in intensity at the region boundaries. Edge detection techniques have therefore been used as the base of another segmentation technique.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0036", "text": "The edges identified by edge detection are often disconnected. To segment an object from an image however, one needs closed region boundaries. The desired edges are the boundaries between such objects or spatial-taxons.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0037", "text": "Spatial-taxons are information granules, consisting of a crisp pixel region, stationed at abstraction levels within a hierarchical nested scene architecture. They are similar to the Gestalt psychological designation of figure-ground, but are extended to include foreground, object groups, objects and salient object parts. Edge detection methods can be applied to the spatial-taxon region, in the same manner they would be applied to a silhouette", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0038", "text": ". This method is particularly useful when the disconnected edge is part of an illusory contour", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0039", "text": "Segmentation methods can also be applied to edges obtained from edge detectors. Lindeberg and Li developed an integrated method that segments edges into straight and curved edge segments for parts-based object recognition, based on a minimum description length (MDL) criterion that was optimized by a split-and-merge-like method with candidate breakpoints obtained from complementary junction cues to obtain more likely points at which to consider partitions into different segments.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0040", "text": "Isolated Point Detection\nThe detection of isolated points in an image is a fundamental part of image segmentation. This process primarily depends on the second derivative, indicating the use of the Laplacian operator. The Laplacian of a function \n  \n    \n      \n        f\n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle f(x,y)}\n  \n is given by:", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0041", "text": "\u2207\n          \n            2\n          \n        \n        f\n        (\n        x\n        ,\n        y\n        )\n        =\n        \n          \n            \n              \n                \u2202\n                \n                  2\n                \n              \n              f\n            \n            \n              \u2202\n              \n                x\n                \n                  2\n                \n              \n            \n          \n        \n        +", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0042", "text": "+\n        \n          \n            \n              \n                \u2202\n                \n                  2\n                \n              \n              f\n            \n            \n              \u2202\n              \n                y\n                \n                  2\n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\nabla ^{2}f(x,y)={\\frac {\\partial ^{2}f}{\\partial x^{2}}}+{\\frac {\\partial ^{2}f}{\\partial y^{2}}}}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0043", "text": "The Laplacian operator is employed such that the partial derivatives are derived from a specific equation. The second partial derivative of \n  \n    \n      \n        f\n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle f(x,y)}\n  \n with respect to \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n and \n  \n    \n      \n        y\n      \n    \n    {\\displaystyle y}\n  \n are given by:", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0044", "text": "\u2202\n                \n                  2\n                \n              \n              f\n              (\n              x\n              ,\n              y\n              )\n            \n            \n              \u2202\n              \n                x\n                \n                  2\n                \n              \n            \n          \n        \n        =\n        f\n        (\n        x\n        +\n        1\n        ,", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0045", "text": "(\n        x\n        +\n        1\n        ,\n        y\n        )\n        +\n        f\n        (\n        x\n        \u2212\n        1\n        ,\n        y\n        )\n        \u2212\n        2\n        f\n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle {\\frac {\\partial ^{2}f(x,y)}{\\partial x^{2}}}=f(x+1,y)+f(x-1,y)-2f(x,y)}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0046", "text": "\u2202\n                \n                  2\n                \n              \n              f\n              (\n              x\n              ,\n              y\n              )\n            \n            \n              \u2202\n              \n                y\n                \n                  2\n                \n              \n            \n          \n        \n        =\n        f\n        (\n        x\n        ,\n        y\n        +", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0047", "text": "(\n        x\n        ,\n        y\n        +\n        1\n        )\n        +\n        f\n        (\n        x\n        ,\n        y\n        \u2212\n        1\n        )\n        \u2212\n        2\n        f\n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle {\\frac {\\partial ^{2}f(x,y)}{\\partial y^{2}}}=f(x,y+1)+f(x,y-1)-2f(x,y)}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0048", "text": "These partial derivatives are then used to compute the Laplacian as:", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0049", "text": "\u2207\n          \n            2\n          \n        \n        f\n        (\n        x\n        ,\n        y\n        )\n        =\n        f\n        (\n        x\n        +\n        1\n        ,\n        y\n        )\n        +\n        f\n        (\n        x\n        \u2212\n        1\n        ,\n        y\n        )\n        +\n        f\n        (\n        x\n        ,\n        y\n        +\n        1\n        )\n        +\n        f\n        (\n        x\n        ,\n        y\n        \u2212\n        1", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0050", "text": "x\n        ,\n        y\n        \u2212\n        1\n        )\n        \u2212\n        4\n        f\n        (\n        x\n        ,\n        y\n        )\n      \n    \n    {\\displaystyle \\nabla ^{2}f(x,y)=f(x+1,y)+f(x-1,y)+f(x,y+1)+f(x,y-1)-4f(x,y)}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0051", "text": "This mathematical expression can be implemented by convolving with an appropriate mask. If we extend this equation to three dimensions (x,y,z), the intensity at each pixel location around a central pixel at (x, y, z) is replaced by their corresponding values. This equation becomes particularly useful when we assume that all pixels have unit spacing along each axis.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0052", "text": "A sphere mask has been developed for use with three-dimensional datasets. The sphere mask is designed to use only integer arithmetic during calculations, thereby eliminating the need for floating-point hardware or software.\nWhen applying these concepts to actual images represented as arrays of numbers, we need to consider what happens when we reach an edge or border region. The function \n  \n    \n      \n        g\n        (\n        x\n        ,\n        y\n        )", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0053", "text": ",\n        y\n        )\n      \n    \n    {\\displaystyle g(x,y)}\n  \n is defined as:", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0054", "text": "g\n        (\n        x\n        ,\n        y\n        )\n        =\n        \n          \n            {\n            \n              \n                \n                  1\n                \n                \n                  \n                    if \n                  \n                  \n                    |\n                  \n                  R\n                  (\n                  x\n                  ,\n                  y\n                  )", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0055", "text": ")\n                  \n                    |\n                  \n                  \u2265\n                  T\n                \n              \n              \n                \n                  0\n                \n                \n                  \n                    otherwise\n                  \n                \n              \n            \n            \n          \n        \n      \n    \n    {\\displaystyle g(x,y)={\\begin{cases}1&{\\text{if }}|R(x,y)|\\geq T\\\\0&{\\text{otherwise}}\\end{cases}}}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0056", "text": "This above equation is used to determine whether a point in the image is an isolated point based on the response magnitude \n  \n    \n      \n        \n          |\n        \n        R\n        (\n        x\n        ,\n        y\n        )\n        \n          |\n        \n      \n    \n    {\\displaystyle |R(x,y)|}\n  \n and a threshold value \n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0057", "text": "T\n      \n    \n    {\\displaystyle T}\n  \n. If the response magnitude is greater than or equal to the threshold, the function returns 1, indicating the presence of an isolated point; otherwise, it returns 0. This helps in the effective detection and segmentation of isolated points in the image.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0058", "text": "Application of Isolated Point Detection in X-ray Image Processing", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0059", "text": "The detection of isolated points has significant applications in various fields, including X-ray image processing. For instance, an original X-ray image of a turbine blade can be examined pixel-by-pixel to detect porosity in the upper-right quadrant of the blade. The result of applying an edge detector's response to this X-ray image can be approximated. This demonstrates the segmentation of isolated points in an image with the aid of single-pixel probes.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0060", "text": "Dual clustering method", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0061", "text": "This method is a combination of three characteristics of the image: partition of the image based on histogram analysis is checked by high compactness of the clusters (objects), and high gradients of their borders. For that purpose two spaces have to be introduced: one space is the one-dimensional histogram of brightness H = H(B); the second space is the dual 3-dimensional space of the original image itself B = B(x, y)", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0062", "text": ". The first space allows to measure how compactly the brightness of the image is distributed by calculating a minimal clustering kmin. Threshold brightness T corresponding to kmin defines the binary (black-and-white) image \u2013 bitmap b = \u03c6(x, y), where \u03c6(x, y) = 0, if B(x, y) < T, and \u03c6(x, y) = 1, if B(x, y) \u2265 T. The bitmap b is an object in dual space. On that bitmap a measure has to be defined reflecting how compact distributed black (or white) pixels are", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0063", "text": ". So, the goal is to find objects with good borders. For all T the measure MDC = G/(k \u00d7 L) has to be calculated (where k is difference in brightness between the object and the background, L is length of all borders, and G is mean gradient on the borders). Maximum of MDC defines the segmentation.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0064", "text": "Region-growing methods\nRegion-growing methods rely mainly on the assumption that the neighboring pixels within one region have similar values. The common procedure is to compare one pixel with its neighbors. If a similarity criterion is satisfied, the pixel can be set to belong to the same cluster as one or more of its neighbors. The selection of the similarity criterion is significant and the results are influenced by noise in all instances.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0065", "text": "The method of Statistical Region Merging (SRM) starts by building the graph of pixels using 4-connectedness with edges weighted by the absolute value of the intensity difference. Initially each pixel forms a single pixel region. SRM then sorts those edges in a priority queue and decides whether or not to merge the current regions belonging to the edge pixels using a statistical predicate.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0066", "text": "One region-growing method is the seeded region growing method. This method takes a set of seeds as input along with the image. The seeds mark each of the objects to be segmented. The regions are iteratively grown by comparison of all unallocated neighboring pixels to the regions. The difference between a pixel's intensity value and the region's mean, \n  \n    \n      \n        \u03b4\n      \n    \n    {\\displaystyle \\delta }", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0067", "text": "{\\displaystyle \\delta }\n  \n, is used as a measure of similarity. The pixel with the smallest difference measured in this way is assigned to the respective region. This process continues until all pixels are assigned to a region. Because seeded region growing requires seeds as additional input, the segmentation results are dependent on the choice of seeds, and noise in the image can cause the seeds to be poorly placed.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0068", "text": "Another region-growing method is the unseeded region growing method. It is a modified algorithm that does not require explicit seeds. It starts with a single region \n  \n    \n      \n        \n          A\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle A_{1}}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0069", "text": "{\\displaystyle A_{1}}\n  \n\u2014the pixel chosen here does not markedly influence the final segmentation. At each iteration it considers the neighboring pixels in the same way as seeded region growing. It differs from seeded region growing in that if the minimum \n  \n    \n      \n        \u03b4\n      \n    \n    {\\displaystyle \\delta }\n  \n is less than a predefined threshold \n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}\n  \n then it is added to the respective region", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0070", "text": "then it is added to the respective region \n  \n    \n      \n        \n          A\n          \n            j\n          \n        \n      \n    \n    {\\displaystyle A_{j}}\n  \n. If not, then the pixel is considered different from all current regions \n  \n    \n      \n        \n          A\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle A_{i}}\n  \n and a new region \n  \n    \n      \n        \n          A\n          \n            n\n            +\n            1", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0071", "text": "+\n            1\n          \n        \n      \n    \n    {\\displaystyle A_{n+1}}\n  \n is created with this pixel.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0072", "text": "is created with this pixel.\nOne variant of this technique, proposed by Haralick and Shapiro (1985), is based on pixel intensities. The mean and scatter of the region and the intensity of the candidate pixel are used to compute a test statistic. If the test statistic is sufficiently small, the pixel is added to the region, and the region's mean and scatter are recomputed. Otherwise, the pixel is rejected, and is used to form a new region.\nA special region-growing method is called", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0073", "text": "\u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n-connected segmentation (see also lambda-connectedness). It is based on pixel intensities and neighborhood-linking paths. A degree of connectivity (connectedness) is calculated based on a path that is formed by pixels. For a certain value of \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n, two pixels are called \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0074", "text": "{\\displaystyle \\lambda }\n  \n-connected if there is a path linking those two pixels and the connectedness of this path is at least \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n. \n  \n    \n      \n        \u03bb\n      \n    \n    {\\displaystyle \\lambda }\n  \n-connectedness is an equivalence relation.\nSplit-and-merge segmentation is based on a quadtree partition of an image. It is sometimes called quadtree segmentation.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0075", "text": "This method starts at the root of the tree that represents the whole image. If it is found non-uniform (not homogeneous), then it is split into four child squares (the splitting process), and so on. If, in contrast, four child squares are homogeneous, they are merged as several connected components (the merging process). The node in the tree is a segmented node. This process continues recursively until no further splits or merges are possible", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0076", "text": ". When a special data structure is involved in the implementation of the algorithm of the method, its time complexity can reach", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0077", "text": "O\n        (\n        n\n        log\n        \u2061\n        n\n        )\n      \n    \n    {\\displaystyle O(n\\log n)}\n  \n, an optimal algorithm of the method.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0078", "text": "Partial differential equation-based methods", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0079", "text": "Using a partial differential equation (PDE)-based method and solving the PDE equation by a numerical scheme, one can segment the image. Curve propagation is a popular technique in this category, with numerous applications to object extraction, object tracking, stereo reconstruction, etc. The central idea is to evolve an initial curve towards the lowest potential of a cost function, where its definition reflects the task to be addressed", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0080", "text": ". As for most inverse problems, the minimization of the cost functional is non-trivial and imposes certain smoothness constraints on the solution, which in the present case can be expressed as geometrical constraints on the evolving curve.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0081", "text": "Parametric methods", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0082", "text": "Lagrangian techniques are based on parameterizing the contour according to some sampling strategy and then evolving each element according to image and internal terms", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0083", "text": ". Such techniques are fast and efficient, however the original \"purely parametric\" formulation (due to Kass, Witkin and Terzopoulos in 1987 and known as \"snakes\"), is generally criticized for its limitations regarding the choice of sampling strategy, the internal geometric properties of the curve, topology changes (curve splitting and merging), addressing problems in higher dimensions, etc.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0084", "text": ". Nowadays, efficient \"discretized\" formulations have been developed to address these limitations while maintaining high efficiency. In both cases, energy minimization is generally conducted using a steepest-gradient descent, whereby derivatives are computed using, e.g., finite differences.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0085", "text": "Level-set methods", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0086", "text": "The level-set method was initially proposed to track moving interfaces by Dervieux and Thomasset in 1979 and 1981 and was later reinvented by Osher and Sethian in 1988. This has spread across various imaging domains in the late 1990s. It can be used to efficiently address the problem of curve/surface/etc. propagation in an implicit manner. The central idea is to represent the evolving contour using a signed function whose zero corresponds to the actual contour", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0087", "text": ". Then, according to the motion equation of the contour, one can easily derive a similar flow for the implicit surface that when applied to the zero level will reflect the propagation of the contour. The level-set method affords numerous advantages: it is implicit, is parameter-free, provides a direct way to estimate the geometric properties of the evolving structure, allows for change of topology, and is intrinsic", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0088", "text": ". It can be used to define an optimization framework, as proposed by Zhao, Merriman and Osher in 1996. One can conclude that it is a very convenient framework for addressing numerous applications of computer vision and medical image analysis. Research into various level-set data structures has led to very efficient implementations of this method.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0089", "text": "Fast marching methods\nThe fast marching method has been used in image segmentation, and this model has been improved (permitting both positive and negative propagation speeds) in an approach called the generalized fast marching method.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0090", "text": "Variational methods\nThe goal of variational methods is to find a segmentation\nwhich is optimal with respect to a specific energy functional. The functionals consist of a data fitting term and a regularizing terms. A classical representative is the Potts model defined for an image \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n by", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0091", "text": "argmin\n          \n            u\n          \n        \n        \u2061\n        \u03b3\n        \u2016\n        \u2207\n        u\n        \n          \u2016\n          \n            0\n          \n        \n        +\n        \u222b\n        (\n        u\n        \u2212\n        f\n        \n          )\n          \n            2\n          \n        \n        \n        d\n        x\n        .\n      \n    \n    {\\displaystyle \\operatorname {argmin} _{u}\\gamma \\|\\nabla u\\|_{0}+\\int (u-f)^{2}\\,dx.}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0092", "text": "A minimizer \n  \n    \n      \n        \n          u\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle u^{*}}\n  \n is a piecewise constant image which has an optimal tradeoff between the squared L2 distance to the given image \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n and the total length of its jump set. The jump set of \n  \n    \n      \n        \n          u\n          \n            \u2217\n          \n        \n      \n    \n    {\\displaystyle u^{*}}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0093", "text": "{\\displaystyle u^{*}}\n  \n defines a segmentation. The relative weight of the energies is tuned by the parameter \n  \n    \n      \n        \u03b3\n        >\n        0\n      \n    \n    {\\displaystyle \\gamma >0}\n  \n. The binary variant of the Potts model, i.e., if the range of \n  \n    \n      \n        u\n      \n    \n    {\\displaystyle u}\n  \n is restricted to two values, is often called Chan-Vese model. An important generalization is the Mumford-Shah model given by", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0094", "text": "argmin\n          \n            u\n            ,\n            K\n          \n        \n        \u2061\n        \u03b3\n        \n          |\n        \n        K\n        \n          |\n        \n        +\n        \u03bc\n        \n          \u222b\n          \n            \n              K\n              \n                C\n              \n            \n          \n        \n        \n          |\n        \n        \u2207\n        u\n        \n          \n            |\n          \n          \n            2", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0095", "text": "2\n          \n        \n        \n        d\n        x\n        +\n        \u222b\n        (\n        u\n        \u2212\n        f\n        \n          )\n          \n            2\n          \n        \n        \n        d\n        x\n        .\n      \n    \n    {\\displaystyle \\operatorname {argmin} _{u,K}\\gamma |K|+\\mu \\int _{K^{C}}|\\nabla u|^{2}\\,dx+\\int (u-f)^{2}\\,dx.}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0096", "text": "The functional value is the sum of the total length of the segmentation curve \n  \n    \n      \n        K\n      \n    \n    {\\displaystyle K}\n  \n, the smoothness of the approximation \n  \n    \n      \n        u\n      \n    \n    {\\displaystyle u}\n  \n, and its distance to the original image \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n. The weight of the smoothness penalty is adjusted by \n  \n    \n      \n        \u03bc\n        >\n        0\n      \n    \n    {\\displaystyle \\mu >0}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0097", "text": "{\\displaystyle \\mu >0}\n  \n. The Potts model is often called piecewise constant Mumford-Shah model as it can be seen as the degenerate case \n  \n    \n      \n        \u03bc\n        \u2192\n        \u221e\n      \n    \n    {\\displaystyle \\mu \\to \\infty }\n  \n. The optimization problems are known to be NP-hard in general but near-minimizing strategies work well in practice. Classical algorithms are graduated non-convexity and Ambrosio-Tortorelli approximation.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0098", "text": "Graph partitioning methods", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0099", "text": "Graph partitioning methods are an effective tools for image segmentation since they model the impact of pixel neighborhoods on a given cluster of pixels or pixel, under the assumption of homogeneity in images. In these methods, the image is modeled as a weighted, undirected graph. Usually a pixel or a group of pixels are associated with nodes and edge weights define the (dis)similarity between the neighborhood pixels", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0100", "text": ". The graph (image) is then partitioned according to a criterion designed to model \"good\" clusters. Each partition of the nodes (pixels) output from these algorithms are considered an object segment in the image; see Segmentation-based object categorization. Some popular algorithms of this category are normalized cuts, random walker, minimum cut, isoperimetric partitioning, minimum spanning tree-based segmentation, and segmentation-based object categorization.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0101", "text": "Markov random fields", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0102", "text": "The application of Markov random fields (MRF) for images was suggested in early 1984 by Geman and Geman. Their strong mathematical foundation and ability to provide a global optimum even when defined on local features proved to be the foundation for novel research in the domain of image analysis, de-noising and segmentation", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0103", "text": ". MRFs are completely characterized by their prior probability distributions, marginal probability distributions, cliques, smoothing constraint as well as criterion for updating values. The criterion for image segmentation using MRFs is restated as finding the labelling scheme which has maximum probability for a given set of features. The broad categories of image segmentation using MRFs are supervised and unsupervised segmentation.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0104", "text": "Supervised image segmentation using MRF and MAP\nIn terms of image segmentation, the function that MRFs seek to maximize is the probability of identifying a labelling scheme given a particular set of features are detected in the image. This is a restatement of the maximum a posteriori estimation method.\n\nThe generic algorithm for image segmentation using MAP is given below:", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0105", "text": "Optimization algorithms\nEach optimization algorithm is an adaptation of models from a variety of fields and they are set apart by their unique cost functions. The common trait of cost functions is to penalize change in pixel value as well as difference in pixel label when compared to labels of neighboring pixels.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0106", "text": "Iterated conditional modes/gradient descent\nThe iterated conditional modes (ICM) algorithm tries to reconstruct the ideal labeling scheme by changing the values of each pixel over each iteration and evaluating the energy of the new labeling scheme using the cost function given below,", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0107", "text": "\u03b1\n        (\n        1\n        \u2212\n        \u03b4\n        (\n        \n          \u2113\n          \n            i\n          \n        \n        \u2212\n        \n          \u2113\n          \n            \n              initial \n            \n            i\n          \n        \n        )\n        +\n        \u03b2\n        \n          \u03a3\n          \n            q\n            \u2208\n            N\n            (\n            i\n            )\n          \n        \n        (\n        1\n        \u2212\n        \u03b4\n        (", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0108", "text": "1\n        \u2212\n        \u03b4\n        (\n        \n          \u2113\n          \n            i\n          \n        \n        ,\n        \n          \u2113\n          \n            q\n            (\n            i\n            )\n          \n        \n        )\n        )\n        .\n      \n    \n    {\\displaystyle \\alpha (1-\\delta (\\ell _{i}-\\ell _{{\\text{initial }}i})+\\beta \\Sigma _{q\\in N(i)}(1-\\delta (\\ell _{i},\\ell _{q(i)})).}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0109", "text": "where \u03b1 is the penalty for change in pixel label and \u03b2 is the penalty for difference in label between\nneighboring pixels and chosen pixel. Here \n  \n    \n      \n        N\n        (\n        i\n        )\n      \n    \n    {\\displaystyle N(i)}\n  \n is neighborhood of pixel i and \u03b4 is the Kronecker delta function. A major issue with ICM is that, similar to gradient descent, it has a tendency to rest over local maxima and thus not obtain a globally optimal labeling scheme.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0110", "text": "Simulated annealing (SA)\nDerived as an analogue of annealing in metallurgy, simulated annealing (SA) uses change in pixel label over iterations and estimates the difference in energy of each newly formed graph to the initial data. If the newly formed graph is more profitable, in terms of low energy cost, given by:", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0111", "text": "\u0394\n        U\n        =\n        \n          U\n          \n            new\n          \n        \n        \u2212\n        \n          U\n          \n            old\n          \n        \n      \n    \n    {\\displaystyle \\Delta U=U^{\\text{new}}-U^{\\text{old}}}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0112", "text": "\u2113\n          \n            i\n          \n        \n        =\n        \n          \n            {\n            \n              \n                \n                  \n                    \u2113\n                    \n                      i\n                    \n                    \n                      new\n                    \n                  \n                  ,\n                \n                \n                  \n                    if", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0113", "text": "if \n                  \n                  \u0394\n                  U\n                  \u2264\n                  0\n                  ,\n                \n              \n              \n                \n                  \n                    \u2113\n                    \n                      i\n                    \n                    \n                      new\n                    \n                  \n                  ,\n                \n                \n                  \n                    if", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0114", "text": "if \n                  \n                  \u0394\n                  U\n                  >\n                  0\n                  \n                     and \n                  \n                  \u03b4\n                  <\n                  \n                    e\n                    \n                      \u2212\n                      \u0394\n                      U\n                      \n                        /\n                      \n                      T", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0115", "text": "T\n                    \n                  \n                  ,\n                  \n                    \u2113\n                    \n                      i\n                    \n                    \n                      old", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0116", "text": "{\\displaystyle \\ell _{i}={\\begin{cases}\\ell _{i}^{\\text{new}},&{\\text{if }}\\Delta U\\leq 0,\\\\\\ell _{i}^{\\text{new}},&{\\text{if }}\\Delta U>0{\\text{ and }}\\delta <e^{-\\Delta U/T},\\ell _{i}^{\\text{old}}\\end{cases}}}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0117", "text": "the algorithm selects the newly formed graph. Simulated annealing requires the input of temperature schedules which directly affects the speed of convergence of the system, as well as energy threshold for minimization to occur.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0118", "text": "Alternative algorithms\nA range of other methods exist for solving simple as well as higher order MRFs. They include Maximization of Posterior Marginal, Multi-scale MAP estimation, Multiple Resolution segmentation and more. Apart from likelihood estimates, graph-cut using maximum flow and other highly constrained graph based methods exist for solving MRFs.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0119", "text": "Image segmentation using MAP and expectation\u2013maximization\nThe expectation\u2013maximization algorithm is utilized to iteratively estimate the a posterior probabilities and distributions of labeling when no training data is available and no estimate of segmentation model can be formed. A general approach is to use histograms to represent the features of an image and proceed as outlined briefly in this three-step algorithm:\n1. A random estimate of the model parameters is utilized.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0120", "text": "2. E step: Estimate class statistics based on the random segmentation model defined. Using these, compute the conditional probability of belonging to a label given the feature set is calculated using naive Bayes' theorem.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0121", "text": "P\n        (\n        \u03bb\n        \u2223\n        \n          f\n          \n            i\n          \n        \n        )\n        =\n        \n          \n            \n              P\n              (\n              \n                f\n                \n                  i\n                \n              \n              \u2223\n              \u03bb\n              )\n              P\n              (\n              \u03bb\n              )\n            \n            \n              \n                \u03a3", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0122", "text": "\u03a3\n                \n                  \u03bb\n                  \u2208\n                  \u039b\n                \n              \n              P\n              (\n              \n                f\n                \n                  i\n                \n              \n              \u2223\n              \u03bb\n              )\n              P\n              (\n              \u03bb\n              )", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0123", "text": "{\\displaystyle P(\\lambda \\mid f_{i})={\\frac {P(f_{i}\\mid \\lambda )P(\\lambda )}{\\Sigma _{\\lambda \\in \\Lambda }P(f_{i}\\mid \\lambda )P(\\lambda )}}}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0124", "text": "Here \n  \n    \n      \n        \u03bb\n        \u2208\n        \u039b\n      \n    \n    {\\displaystyle \\lambda \\in \\Lambda }\n  \n, the set of all possible labels.\n3. M step: The established relevance of a given feature set to a labeling scheme is now used to compute the a priori estimate of a given label in the second part of the algorithm. Since the actual number of total labels is unknown (from a training data set), a hidden estimate of the number of labels given by the user is utilized in computations.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0125", "text": "P\n        (\n        \u03bb\n        )\n        =\n        \n          \n            \n              \n                \u03a3\n                \n                  \u03bb\n                  \u2208\n                  \u039b\n                \n              \n              P\n              (\n              \u03bb\n              \u2223\n              \n                f\n                \n                  i\n                \n              \n              )\n            \n            \n              \n                |", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0126", "text": "|\n              \n              \u03a9\n              \n                |\n              \n            \n          \n        \n      \n    \n    {\\displaystyle P(\\lambda )={\\frac {\\Sigma _{\\lambda \\in \\Lambda }P(\\lambda \\mid f_{i})}{|\\Omega |}}}", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0127", "text": "where \n  \n    \n      \n        \u03a9\n      \n    \n    {\\displaystyle \\Omega }\n  \n is the set of all possible features.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0128", "text": "Disadvantages of MAP and EM based image segmentation\nExact MAP estimates cannot be easily computed.\nApproximate MAP estimates are computationally expensive to calculate.\nExtension to multi-class labeling degrades performance and increases storage required.\nReliable estimation of parameters for EM is required for global optima to be achieved.\nBased on method of optimization, segmentation may cluster to local minima.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0129", "text": "Watershed transformation\nThe watershed transformation considers the gradient magnitude of an image as a topographic surface. Pixels having the highest gradient magnitude intensities (GMIs) correspond to watershed lines, which represent the region boundaries. Water placed on any pixel enclosed by a common watershed line flows downhill to a common local intensity minimum (LIM). Pixels draining to a common minimum form a catch basin, which represents a segment.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0130", "text": "Model-based segmentation", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0131", "text": "The central assumption of model-based approaches is that the structures of interest have a tendency towards a particular shape. Therefore, one can seek a probabilistic model that characterizes the shape and its variation. When segmenting an image, constraints can be imposed using this model as a prior", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0132", "text": ". Such a task may involve (i) registration of the training examples to a common pose, (ii) probabilistic representation of the variation of the registered samples, and (iii) statistical inference between the model and the image. Other important methods in the literature for model-based segmentation include active shape models and active appearance models.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0133", "text": "Multi-scale segmentation\nImage segmentations are computed at multiple scales in scale space and sometimes propagated from coarse to fine scales; see scale-space segmentation.\nSegmentation criteria can be arbitrarily complex and may take into account global as well as local criteria. A common requirement is that each region must be connected in some sense.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0134", "text": "One-dimensional hierarchical signal segmentation\nWitkin's seminal work in scale space included the notion that a one-dimensional signal could be unambiguously segmented into regions, with one scale parameter controlling the scale of segmentation.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0135", "text": "A key observation is that the zero-crossings of the second derivatives (minima and maxima of the first derivative or slope) of multi-scale-smoothed versions of a signal form a nesting tree, which defines hierarchical relations between segments at different scales. Specifically, slope extrema at coarse scales can be traced back to corresponding features at fine scales", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0136", "text": ". When a slope maximum and slope minimum annihilate each other at a larger scale, the three segments that they separated merge into one segment, thus defining the hierarchy of segments.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0137", "text": "Image segmentation and primal sketch\nThere have been numerous research works in this area, out of which a few have now reached a state where they can be applied either with interactive manual intervention (usually with application to medical imaging) or fully automatically. The following is a brief overview of some of the main research ideas that current approaches are based upon.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0138", "text": "The nesting structure that Witkin described is, however, specific for one-dimensional signals and does not trivially transfer to higher-dimensional images. Nevertheless, this general idea has inspired several other authors to investigate coarse-to-fine schemes for image segmentation. Koenderink proposed to study how iso-intensity contours evolve over scales and this approach was investigated in more detail by Lifshitz and Pizer", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0139", "text": ". Unfortunately, however, the intensity of image features changes over scales, which implies that it is hard to trace coarse-scale image features to finer scales using iso-intensity information.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0140", "text": "Lindeberg studied the problem of linking local extrema and saddle points over scales, and proposed an image representation called the scale-space primal sketch which makes explicit the relations between structures at different scales, and also makes explicit which image features are stable over large ranges of scale including locally appropriate scales for those", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0141", "text": ". Bergholm proposed to detect edges at coarse scales in scale-space and then trace them back to finer scales with manual choice of both the coarse detection scale and the fine localization scale.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0142", "text": "Gauch and Pizer studied the complementary problem of ridges and valleys at multiple scales and developed a tool for interactive image segmentation based on multi-scale watersheds. The use of multi-scale watershed with application to the gradient map has also been investigated by Olsen and Nielsen and been carried over to clinical use by Dam. Vincken et al. proposed a hyperstack for defining probabilistic relations between image structures at different scales", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0143", "text": ". The use of stable image structures over scales has been furthered by Ahuja and his co-workers into a fully automated system. A fully automatic brain segmentation algorithm based on closely related ideas of multi-scale watersheds has been presented by Undeman and Lindeberg and been extensively tested in brain databases.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0144", "text": "These ideas for multi-scale image segmentation by linking image structures over scales have also been picked up by Florack and Kuijper. Bijaoui and Ru\u00e9 associate structures detected in scale-space above a minimum noise threshold into an object tree which spans multiple scales and corresponds to a kind of feature in the original signal. Extracted features are accurately reconstructed using an iterative conjugate gradient matrix method.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0145", "text": "Semi-automatic segmentation\nIn one kind of segmentation, the user outlines the region of interest with the mouse clicks and algorithms are applied so that the path that best fits the edge of the image is shown.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0146", "text": "Techniques like SIOX, Livewire, Intelligent Scissors or IT-SNAPS are used in this kind of segmentation. In an alternative kind of semi-automatic segmentation, the algorithms return a spatial-taxon (i.e. foreground, object-group, object or object-part) selected by the user or designated via prior probabilities.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0147", "text": "Trainable segmentation", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0148", "text": "Most of the aforementioned segmentation methods are based only on color information of pixels in the image. Humans use much more knowledge when performing image segmentation, but implementing this knowledge would cost considerable human engineering and computational time, and would require a huge domain knowledge database which does not currently exist", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0149", "text": ". Trainable segmentation methods, such as neural network segmentation, overcome these issues by modeling the domain knowledge from a dataset of labeled pixels.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0150", "text": "An image segmentation neural network can process small areas of an image to extract simple features such as edges. Another neural network, or any decision-making mechanism, can then combine these features to label the areas of an image accordingly. A type of network designed this way is the Kohonen map.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0151", "text": "Pulse-coupled neural networks (PCNNs) are neural models proposed by modeling a cat's visual cortex and developed for high-performance biomimetic image processing. In 1989, Reinhard Eckhorn introduced a neural model to emulate the mechanism of a cat's visual cortex. The Eckhorn model provided a simple and effective tool for studying the visual cortex of small mammals, and was soon recognized as having significant application potential in image processing", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0152", "text": ". In 1994, the Eckhorn model was adapted to be an image processing algorithm by John L. Johnson, who termed this algorithm Pulse-Coupled Neural Network. Over the past decade, PCNNs have been utilized for a variety of image processing applications, including: image segmentation, feature generation, face extraction, motion detection, region growing, noise reduction, and so on. A PCNN is a two-dimensional neural network", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0153", "text": ". A PCNN is a two-dimensional neural network. Each neuron in the network corresponds to one pixel in an input image, receiving its corresponding pixel's color information (e.g. intensity) as an external stimulus. Each neuron also connects with its neighboring neurons, receiving local stimuli from them. The external and local stimuli are combined in an internal activation system, which accumulates the stimuli until it exceeds a dynamic threshold, resulting in a pulse output", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0154", "text": ". Through iterative computation, PCNN neurons produce temporal series of pulse outputs. The temporal series of pulse outputs contain information of input images and can be utilized for various image processing applications, such as image segmentation and feature generation", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0155", "text": ". Compared with conventional image processing means, PCNNs have several significant merits, including robustness against noise, independence of geometric variations in input patterns, capability of bridging minor intensity variations in input patterns, etc.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0156", "text": "In 2015, convolutional neural networks reached state of the art in semantic segmentation. U-Net is an architecture which takes as input an image and outputs a label for each pixel. U-Net initially was developed to detect cell boundaries in biomedical images. U-Net follows classical autoencoder architecture, as such it contains two sub-structures", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0157", "text": ". The encoder structure follows the traditional stack of convolutional and max pooling layers to increase the receptive field as it goes through the layers. It is used to capture the context in the image. The decoder structure utilizes transposed convolution layers for upsampling so that the end dimensions are close to that of the input image", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0158", "text": ". Skip connections are placed between convolution and transposed convolution layers of the same shape in order to preserve details that would have been lost otherwise.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0159", "text": "In addition to pixel-level semantic segmentation tasks which assign a given category to each pixel, modern segmentation applications include instance-level semantic segmentation tasks in which each individual in a given category must be uniquely identified, as well as panoptic segmentation tasks which combines these two tasks to provide a more complete scene segmentation.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0160", "text": "Segmentation of related images and videos", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0161", "text": "Related images such as a photo album or a sequence of video frames often contain semantically similar objects and scenes, therefore it is often beneficial to exploit such correlations. The task of simultaneously segmenting scenes from related images or video frames is termed co-segmentation, which is typically used in human action localization", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0162", "text": ". Unlike conventional bounding box-based object detection, human action localization methods provide finer-grained results, typically per-image segmentation masks delineating the human object of interest and its action category (e.g., Segment-Tube). Techniques such as dynamic Markov Networks, CNN and LSTM are often employed to exploit the inter-frame correlations.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0163", "text": "Other methods\nThere are many other methods of segmentation like multispectral segmentation or connectivity-based segmentation based on DTI images.", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0164", "text": "See also\nObject co-segmentation \u2013 Image segmentation in computer vision\nComputer vision \u2013 Computerized information extraction from images\nImage-based meshing\nRange image segmentation\nVector quantization \u2013 Classical quantization technique from signal processing\nImage quantization \u2013 Lossy compression techniquePages displaying short descriptions of redirect targets\nColor quantization \u2013 Image processing technique", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0165", "text": "Color quantization \u2013 Image processing technique\nObject-based image analysis \u2013 Extraction of information from images via digital image processing techniquesPages displaying short descriptions of redirect targets\nList of manual image annotation tools\nRigid motion segmentation\nText segmentation \u2013 Human writing practice", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0166", "text": "Notes\nReferences\nExternal links\nSome sample code that performs basic segmentation, by Syed Zainudeen. University Technology of Malaysia.\nGeneralized Fast Marching method by Forcadel et al. [2008] for applications in image segmentation.\nImage Processing Research Group An Online Open Image Processing Research Community.\nSegmentation methods in image processing and analysis and Minimizing energy to segment images by Mathworks", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Image_segmentation_0167", "text": "More image segmentation methods with detailed algorithms Archived 1 November 2019 at the Wayback Machine by Yu-Hsiang Wang (\u738b\u6631\u7fd4), National Taiwan University, Taipei, Taiwan, ROC\nOnline demonstration of piecewise linear image segmentation by IPOL Journal", "title": "Image segmentation", "url": "https://en.wikipedia.org/wiki/Image_segmentation"}, {"id": "rec_Information_retrieval_0000", "text": "Information retrieval (IR) in computing and information science is the task of identifying and retrieving information system resources that are relevant to an information need. The information need can be specified in the form of a search query. In the case of document retrieval, queries can be based on full-text or other content-based indexing", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0001", "text": ". Information retrieval is the science of searching for information in a document, searching for documents themselves, and also searching for the metadata that describes data, and for databases of texts, images, or sounds. Cross-modal retrieval implies retrieval across modalities.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0002", "text": "Automated information retrieval systems are used to reduce what has been called information overload. An IR system is a software system that provides access to books, journals, and other documents, as well as storing and managing those documents. Web search engines are the most visible IR applications.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0003", "text": "Overview\nAn information retrieval process begins when a user enters a query into the system. Queries are formal statements of information needs, for example search strings in web search engines. In information retrieval, a query does not uniquely identify a single object in the collection. Instead, several objects may match the query, perhaps with different degrees of relevance.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0004", "text": "An object is an entity that is represented by information in a content collection or database. User queries are matched against the database information. However, as opposed to classical SQL queries of a database, in information retrieval the results returned may or may not match the query, so results are typically ranked. This ranking of results is a key difference of information retrieval searching compared to database searching.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0005", "text": "Depending on the application the data objects may be, for example, text documents, images, audio, mind maps or videos. Often the documents themselves are not kept or stored directly in the IR system, but are instead represented in the system by document surrogates or metadata.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0006", "text": "Most IR systems compute a numeric score on how well each object in the database matches the query, and rank the objects according to this value. The top ranking objects are then shown to the user. The process may then be iterated if the user wishes to refine the query.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0007", "text": "History", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0008", "text": "The idea of using computers to search for relevant pieces of information was popularized in the article As We May Think by Vannevar Bush in 1945. It would appear that Bush was inspired by patents for a 'statistical machine' \u2013 filed by Emanuel Goldberg in the 1920s and 1930s \u2013 that searched for documents stored on film. The first description of a computer searching for information was described by Holmstrom in 1948, detailing an early mention of the Univac computer", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0009", "text": ". Automated information retrieval systems were introduced in the 1950s: one even featured in the 1957 romantic comedy Desk Set. In the 1960s, the first large information retrieval research group was formed by Gerard Salton at Cornell. By the 1970s several different retrieval techniques had been shown to perform well on small text corpora such as the Cranfield collection (several thousand documents)", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0010", "text": ". Large-scale retrieval systems, such as the Lockheed Dialog system, came into use early in the 1970s.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0011", "text": "In 1992, the US Department of Defense along with the National Institute of Standards and Technology (NIST), cosponsored the Text Retrieval Conference (TREC) as part of the TIPSTER text program. The aim of this was to look into the information retrieval community by supplying the infrastructure that was needed for evaluation of text retrieval methodologies on a very large text collection. This catalyzed research on methods that scale to huge corpora", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0012", "text": ". The introduction of web search engines has boosted the need for very large scale retrieval systems even further.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0013", "text": "By the late 1990s, the rise of the World Wide Web fundamentally transformed information retrieval. While early search engines such as AltaVista (1995) and Yahoo! (1994) offered keyword-based retrieval, they were limited in scale and ranking refinement. The breakthrough came in 1998 with the founding of Google, which introduced the PageRank algorithm, using the web's hyperlink structure to assess page importance and improve relevance ranking.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0014", "text": "During the 2000s, web search systems evolved rapidly with the integration of machine learning techniques. These systems began to incorporate user behavior data (e.g., click-through logs), query reformulation, and content-based signals to improve search accuracy and personalization. In 2009, Microsoft launched Bing, introducing features that would later incorporate semantic web technologies through the development of its Satori knowledge base", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0015", "text": ". Academic analysis have highlighted Bing's semantic capabilities, including structured data use and entity recognition, as part of a broader industry shift toward improving search relevance and understanding user intent through natural language processing.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0016", "text": "A major leap occurred in 2018, when Google deployed BERT (Bidirectional Encoder Representations from Transformers) to better understand the contextual meaning of queries and documents. This marked one of the first times deep neural language models were used at scale in real-world retrieval systems. BERT's bidirectional training enabled a more refined comprehension of word relationships in context, improving the handling of natural language queries", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0017", "text": ". Because of its success, transformer-based models gained traction in academic research and commercial search applications.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0018", "text": "Simultaneously, the research community began exploring neural ranking models that outperformed traditional lexical-based methods. Long-standing benchmarks such as the Text REtrieval Conference (TREC), initiated in 1992, and more recent evaluation frameworks Microsoft MARCO(MAchine Reading COmprehension) (2019) became central to training and evaluating retrieval systems across multiple tasks and domains", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0019", "text": ". MS MARCO has also been adopted in the TREC Deep Learning Tracks, where it serves as a core dataset for evaluating advances in neural ranking models within a standardized benchmarking environment.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0020", "text": "As deep learning became integral to information retrieval systems, researchers began to categorize neural approaches into three broad classes: sparse, dense, and hybrid models. Sparse models, including traditional term-based methods and learned variants like SPLADE, rely on interpretable representations and inverted indexes to enable efficient exact term matching with added semantic signals", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0021", "text": ". Dense models, such as dual-encoder architectures like ColBERT, use continuous vector embeddings to support semantic similarity beyond keyword overlap. Hybrid models aim to combine the advantages of both, balancing the lexical (token) precision of sparse methods with the semantic depth of dense models. This way of categorizing models balances scalability, relevance, and efficiency in retrieval systems.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0022", "text": "As IR systems increasingly rely on deep learning, concerns around bias, fairness, and explainability have also come to the picture. Research is now focused not just on relevance and efficiency, but on transparency, accountability, and user trust in retrieval algorithms.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0023", "text": "Applications\nAreas where information retrieval techniques are employed include (the entries are in alphabetical order within each category):\n\nGeneral applications\nDigital libraries\nInformation filtering\nRecommender systems\nMedia search\nBlog search\nImage retrieval\n3D retrieval\nMusic retrieval\nNews search\nSpeech retrieval\nVideo retrieval\nSearch engines\nSite search\nDesktop search\nEnterprise search\nFederated search\nMobile search\nSocial search\nWeb search", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0024", "text": "Domain-specific applications\nExpert search finding\nGenomic information retrieval\nGeographic information retrieval\nInformation retrieval for chemical structures\nInformation retrieval in software engineering\nLegal information retrieval\nVertical search\n\nOther retrieval methods\nMethods/Techniques in which information retrieval techniques are employed include:", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0025", "text": "Cross-modal retrieval\nAdversarial information retrieval\nAutomatic summarization\nMulti-document summarization\nCompound term processing\nCross-lingual retrieval\nDocument classification\nSpam filtering\nQuestion answering", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0026", "text": "Model types\nIn order to effectively retrieve relevant documents by IR strategies, the documents are typically transformed into a suitable representation. Each retrieval strategy incorporates a specific model for its document representation purposes. The picture on the right illustrates the relationship of some common models. In the picture, the models are categorized according to two dimensions: the mathematical basis and the properties of the model.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0027", "text": "First dimension: mathematical basis\nSet-theoretic models represent documents as sets of words or phrases. Similarities are usually derived from set-theoretic operations on those sets. Common models are:\nStandard Boolean model\nExtended Boolean model\nFuzzy retrieval\nAlgebraic models represent documents and queries usually as vectors, matrices, or tuples. The similarity of the query vector and document vector is represented as a scalar value.\nVector space model\nGeneralized vector space model", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0028", "text": "Vector space model\nGeneralized vector space model\n(Enhanced) Topic-based Vector Space Model\nExtended Boolean model\nLatent semantic indexing a.k.a. latent semantic analysis\nProbabilistic models treat the process of document retrieval as a probabilistic inference. Similarities are computed as probabilities that a document is relevant for a given query. Probabilistic theorems like Bayes' theorem are often used in these models.\nBinary Independence Model", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0029", "text": "Binary Independence Model\nProbabilistic relevance model on which is based the okapi (BM25) relevance function\nUncertain inference\nLanguage models\nDivergence-from-randomness model\nLatent Dirichlet allocation", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0030", "text": "Latent Dirichlet allocation\nFeature-based retrieval models view documents as vectors of values of feature functions (or just features) and seek the best way to combine these features into a single relevance score, typically by learning to rank methods. Feature functions are arbitrary functions of document and query, and as such can easily incorporate almost any other retrieval model as just another feature.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0031", "text": "Data fusion models: Data fusion in information retrieval combines results from multiple search systems or retrieval models to improve performance. By merging ranked lists, it leverages the strengths of diverse approaches, often enhancing recall and precision. Common methods include score normalization and voting techniques like CombSUM or Borda count", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0032", "text": ". This meta-search strategy is particularly effective when individual systems have complementary coverage or when query difficulty varies, producing a more robust and reliable final ranking sparsity.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0033", "text": "Second dimension: properties of the model\nModels without term-interdependencies treat different terms/words as independent. This fact is usually represented in vector space models by the orthogonality assumption of term vectors or in probabilistic models by an independency assumption for term variables.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0034", "text": "Models with immanent term interdependencies allow a representation of interdependencies between terms. However the degree of the interdependency between two terms is defined by the model itself. It is usually directly or indirectly derived (e.g. by dimensional reduction) from the co-occurrence of those terms in the whole set of documents.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0035", "text": "Models with transcendent term interdependencies allow a representation of interdependencies between terms, but they do not allege how the interdependency between two terms is defined. They rely on an external source for the degree of interdependency between two terms. (For example, a human or sophisticated algorithms.)", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0036", "text": "Third Dimension: representational approach-based classification\nIn addition to the theoretical distinctions, modern information retrieval models are also categorized on how queries and documents are represented and compared, using a practical classification distinguishing between sparse, dense and hybrid models.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0037", "text": "Sparse models utilize interpretable, term-based representations and typically rely on inverted index structures. Classical methods such as TF-IDF and BM25 fall under this category, along with more recent learned sparse models that integrate neural architectures while retaining sparsity.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0038", "text": "Dense models represent queries and documents as continuous vectors using deep learning models, typically transformer-based encoders. These models enable semantic similarity matching beyond exact term overlap and are used in tasks involving semantic search and question answering.\nHybrid models aim to combine the strengths of both approaches, integrating lexical (tokens) and semantic signals through score fusion, late interaction, or multi-stage ranking pipelines.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0039", "text": "This classification has become increasingly common in both academic and the real world applications and is getting widely adopted and used in evaluation benchmarks for Information Retrieval models.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0040", "text": "Performance and correctness measures", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0041", "text": "The evaluation of an information retrieval system' is the process of assessing how well a system meets the information needs of its users. In general, measurement considers a collection of documents to be searched and a search query. Traditional evaluation metrics, designed for Boolean retrieval or top-k retrieval, include precision and recall. All measures assume a ground truth notion of relevance: every document is known to be either relevant or non-relevant to a particular query", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0042", "text": ". In practice, queries may be ill-posed and there may be different shades of relevance.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0043", "text": "Libraries for searching and indexing\nLemur\nLucene\nSolr\nElasticsearch\nManatee\nManticore search\nSphinx\nTerrier Search Engine\nXapian", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0044", "text": "Timeline\nBefore the 1900s\n1801: Joseph Marie Jacquard invents the Jacquard loom, the first machine to use punched cards to control a sequence of operations.\n1880s: Herman Hollerith invents an electro-mechanical data tabulator using punch cards as a machine readable medium.\n1890 Hollerith cards, keypunches and tabulators used to process the 1890 US census data.\n1920s\u20131930s", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0045", "text": "1920s\u20131930s\nEmanuel Goldberg submits patents for his \"Statistical Machine\", a document search engine that used photoelectric cells and pattern recognition to search the metadata on rolls of microfilmed documents.\n1940s\u20131950s\nlate 1940s: The US military confronted problems of indexing and retrieval of wartime scientific research documents captured from Germans.\n1945: Vannevar Bush's As We May Think appeared in Atlantic Monthly.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0046", "text": "1947: Hans Peter Luhn (research engineer at IBM since 1941) began work on a mechanized punch card-based system for searching chemical compounds.\n1950s: Growing concern in the US for a \"science gap\" with the USSR motivated, encouraged funding and provided a backdrop for mechanized literature searching systems (Allen Kent et al.) and the invention of the citation index by Eugene Garfield.\n1950: The term \"information retrieval\" was coined by Calvin Mooers.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0047", "text": "1951: Philip Bagley conducted the earliest experiment in computerized document retrieval in a master thesis at MIT.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0048", "text": "1955: Allen Kent joined Case Western Reserve University, and eventually became associate director of the Center for Documentation and Communications Research. That same year, Kent and colleagues published a paper in American Documentation describing the precision and recall measures as well as detailing a proposed \"framework\" for evaluating an IR system which included statistical sampling methods for determining the number of relevant documents not retrieved.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0049", "text": "1958: International Conference on Scientific Information Washington DC included consideration of IR systems as a solution to problems identified. See: Proceedings of the International Conference on Scientific Information, 1958 (National Academy of Sciences, Washington, DC, 1959)\n1959: Hans Peter Luhn published \"Auto-encoding of documents for information retrieval\".\n1960s:\nearly 1960s: Gerard Salton began work on IR at Harvard, later moved to Cornell.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0050", "text": "1960: Melvin Earl Maron and John Lary Kuhns published \"On relevance, probabilistic indexing, and information retrieval\" in the Journal of the ACM 7(3):216\u2013244, July 1960.\n1962:\n* Cyril W. Cleverdon published early findings of the Cranfield studies, developing a model for IR system evaluation. See: Cyril W. Cleverdon, \"Report on the Testing and Analysis of an Investigation into the Comparative Efficiency of Indexing Systems\". Cranfield Collection of Aeronautics, Cranfield, England, 1962.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0051", "text": "* Kent published Information Analysis and Retrieval.\n1963:\n* Weinberg report \"Science, Government and Information\" gave a full articulation of the idea of a \"crisis of scientific information\". The report was named after Dr. Alvin Weinberg.\n* Joseph Becker and Robert M. Hayes published text on information retrieval. Becker, Joseph; Hayes, Robert Mayo. Information storage and retrieval: tools, elements, theories. New York, Wiley (1963).\n1964:", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0052", "text": "1964:\n* Karen Sp\u00e4rck Jones finished her thesis at Cambridge, Synonymy and Semantic Classification, and continued work on computational linguistics as it applies to IR.\n* The National Bureau of Standards sponsored a symposium titled \"Statistical Association Methods for Mechanized Documentation\". Several highly significant papers, including G. Salton's first published reference (we believe) to the SMART system.\nmid-1960s:", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0053", "text": "mid-1960s:\n* National Library of Medicine (NLM) developed MEDLARS Medical Literature Analysis and Retrieval System, the first major machine-readable database and batch-retrieval system.\n* Project Intrex at MIT.\n1965: J. C. R. Licklider published Libraries of the Future.\n1966: Don Swanson was involved in studies at University of Chicago on Requirements for Future Catalogs.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0054", "text": "late 1960s: F. Wilfrid Lancaster completed evaluation studies of the MEDLARS system and published the first edition of his text on information retrieval.\n1968:\n* Gerard Salton published Automatic Information Organization and Retrieval.\n* John W. Sammon, Jr.'s RADC Tech report \"Some Mathematics of Information Storage and Retrieval...\" outlined the vector model.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0055", "text": "1969: Sammon's \"A nonlinear mapping for data structure analysis Archived 2017-08-08 at the Wayback Machine\" (IEEE Transactions on Computers) was the first proposal for visualization interface to an IR system.\n1970s\nearly 1970s:\n* First online systems\u2014NLM's AIM-TWX, MEDLINE; Lockheed's Dialog; SDC's ORBIT.\n* Theodor Nelson promoting concept of hypertext, published Computer Lib/Dream Machines.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0056", "text": "1971: Nicholas Jardine and Cornelis J. van Rijsbergen published \"The use of hierarchic clustering in information retrieval\", which articulated the \"cluster hypothesis\".\n1975: Three highly influential publications by Salton fully articulated his vector processing framework and term discrimination model:\n* A Theory of Indexing (Society for Industrial and Applied Mathematics)\n* A Theory of Term Importance in Automatic Text Analysis (JASIS v. 26)", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0057", "text": "* A Vector Space Model for Automatic Indexing (CACM 18:11)\n1978: The First ACM SIGIR conference.\n1979: C. J. van Rijsbergen published Information Retrieval (Butterworths). Heavy emphasis on probabilistic models.\n1979: Tamas Doszkocs implemented the CITE natural language user interface for MEDLINE at the National Library of Medicine. The CITE system supported free form query input, ranked output and relevance feedback.\n1980s", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0058", "text": "1980s\n1980: First international ACM SIGIR conference, joint with British Computer Society IR group in Cambridge.\n1982: Nicholas J. Belkin, Robert N. Oddy, and Helen M. Brooks proposed the ASK (Anomalous State of Knowledge) viewpoint for information retrieval. This was an important concept, though their automated analysis tool proved ultimately disappointing.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0059", "text": "1983: Salton (and Michael J. McGill) published Introduction to Modern Information Retrieval (McGraw-Hill), with heavy emphasis on vector models.\n1985: David Blair and Bill Maron publish: An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0060", "text": "1986: Donald A.B. Lindberg M.D., NLM Director, implemented a direct health professional interface to MEDLINE and other MEDLARS databases. Grateful Med, a pun on the Grateful Dead, was adapted from Microsearch, an ELHILL user interface that assembled query language prior to connecting to the NLM mainframe. After retrieving the query results, Grateful Med disconnected from the mainframe to keep search costs low.\nmid-1980s: Efforts to develop end-user versions of commercial IR systems.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0061", "text": "1985\u20131993: Key papers on and experimental systems for visualization interfaces.\nWork by Donald B. Crouch, Robert R. Korfhage, Matthew Chalmers, Anselm Spoerri and others.\n1989: First World Wide Web proposals by Tim Berners-Lee at CERN.\n1990s\n1992: First TREC conference.\n1997: Publication of Korfhage's Information Storage and Retrieval with emphasis on visualization and multi-reference point systems.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0062", "text": "1998: Google is founded by Larry Page and Sergey Brin. It introduces the PageRank algorithm, which evaluates the importance of web pages based on hyperlink structure.\n1999: Publication of Ricardo Baeza-Yates and Berthier Ribeiro-Neto's Modern Information Retrieval by Addison Wesley, the first book that attempts to cover all IR.\n2000s", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0063", "text": "2000s\n2001: Wikipedia launches as a free, collaborative online encyclopedia. It quickly becomes a major resource for information retrieval, particularly for natural language processing and semantic search benchmarks.\n2009: Microsoft launches Bing, introducing features such as related searches, semantic suggestions, and later incorporating deep learning techniques into its ranking algorithms.\n2010s", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0064", "text": "2010s\n2013: Google's Hummingbird algorithm goes live, marking a shift from keyword matching toward understanding query intent and semantic context in search queries.\n2018: Google AI researchers release BERT (Bidirectional Encoder Representations from Transformers), enabling deep bidirectional understanding of language and improving document ranking and query understanding in IR.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0065", "text": "2019: Microsoft introduces MS MARCO (Microsoft MAchine Reading COmprehension), a large-scale dataset designed for training and evaluating machine reading and passage ranking models.\n2020s\n2020: The ColBERT (Contextualized Late Interaction over BERT) model, designed for efficient passage retrieval using contextualized embeddings, was introduced at SIGIR 2020.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0066", "text": "2021: SPLADE is introduced at SIGIR 2021. It's a sparse neural retrieval model that balances lexical and semantic features using masked language modeling and sparsity regularization.\n2022: The BEIR benchmark is released to evaluate zero-shot IR across 18 datasets covering diverse tasks. It standardizes comparisons between dense, sparse, and hybrid IR models.", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0067", "text": "Major conferences\nSIGIR: Special Interest Group on Information Retrieval\nECIR: European Conference on Information Retrieval\nCIKM: Conference on Information and Knowledge Management\nWWW: International World Wide Web Conference\n\nAwards in the field\nTony Kent Strix award\nGerard Salton Award\nKaren Sp\u00e4rck Jones Award\n\nSee also\nReferences\nFurther reading\nExternal links", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0068", "text": "ACM SIGIR: Information Retrieval Special Interest Group\nBCS IRSG: British Computer Society \u2013 Information Retrieval Specialist Group\nText Retrieval Conference (TREC)\nForum for Information Retrieval Evaluation (FIRE)\nInformation Retrieval (online book) by C. J. van Rijsbergen\nInformation Retrieval Wiki Archived 2015-11-24 at the Wayback Machine\nInformation Retrieval Facility Archived 2008-05-22 at the Wayback Machine\nTREC report on information retrieval evaluation techniques", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Information_retrieval_0069", "text": "How eBay measures search relevance\nInformation retrieval performance evaluation tool @ Athena Research Centre", "title": "Information retrieval", "url": "https://en.wikipedia.org/wiki/Information_retrieval"}, {"id": "rec_Tfidf_0000", "text": "In information retrieval, tf\u2013idf (term frequency\u2013inverse document frequency, TF*IDF, TFIDF, TF\u2013IDF, or Tf\u2013idf) is a measure of importance of a word to a document in a collection or corpus, adjusted for the fact that some words appear more frequently in general. Like the bag-of-words model, it models a document as a multiset of words, without word order. It is a refinement over the simple bag-of-words model, by allowing the weight of words to depend on the rest of the corpus.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0001", "text": "It was often used as a weighting factor in searches of information retrieval, text mining, and user modeling. A survey conducted in 2015 showed that 83% of text-based recommender systems in digital libraries used tf\u2013idf. Variations of the tf\u2013idf weighting scheme were often used by search engines as a central tool in scoring and ranking a document's relevance given a user query", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0002", "text": ". In applied search engine optimization practice, tf\u2013idf is also described as a method for analysing term importance within web pages and supporting semantic SEO techniques. One of the simplest ranking functions is computed by summing the tf\u2013idf for each query term; many more sophisticated ranking functions are variants of this simple model.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0003", "text": "Motivations\nKaren Sp\u00e4rck Jones (1972) conceived a statistical interpretation of term-specificity called Inverse Document Frequency (idf), which became a cornerstone of term weighting:\n\nThe specificity of a term can be quantified as an inverse function of the number of documents in which it occurs.For example, the df (document frequency) and idf for some words in Shakespeare's 37 plays might be represented as follows:", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0004", "text": "We see that \"Romeo\", \"Falstaff\", and \"salad\" appears in very few plays, so seeing these words, one could get a good idea as to which play it might be. In contrast, \"good\" and \"sweet\" appears in every play and are completely uninformative as to which play it is.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0005", "text": "Definition\nThe tf\u2013idf is the product of two statistics, term frequency and inverse document frequency. There are various ways for determining the exact values of both statistics.\nA formula that aims to define the importance of a keyword or phrase within a document or a web page.\n\nTerm frequency\nTerm frequency, tf(t,d), is the relative frequency of term t within document d,", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0006", "text": "t\n          f\n        \n        (\n        t\n        ,\n        d\n        )\n        =\n        \n          \n            \n              f\n              \n                t\n                ,\n                d\n              \n            \n            \n              \n                \u2211\n                \n                  \n                    t\n                    \u2032\n                  \n                  \u2208\n                  d", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0007", "text": "f\n                  \n                    \n                      t\n                      \u2032\n                    \n                    ,\n                    d\n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\mathrm {tf} (t,d)={\\frac {f_{t,d}}{\\sum _{t'\\in d}{f_{t',d}}}}}\n  \n,", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0008", "text": ",\nwhere ft,d is the raw count of a term in a document, i.e., the number of times that term t occurs in document d. Note the denominator is simply the total number of terms in document d (counting each occurrence of the same term separately). There are various other ways to define term frequency:", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0009", "text": "the raw count itself: tf(t,d) = ft,d\nBoolean \"frequencies\": tf(t,d) = 1 if t occurs in d and 0 otherwise;\nlogarithmically scaled frequency: tf(t,d) = log (1 + ft,d);\naugmented frequency, to prevent a bias towards longer documents, e.g. raw frequency divided by the raw frequency of the most frequently occurring term in the document:", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0010", "text": "t\n          f\n        \n        (\n        t\n        ,\n        d\n        )\n        =\n        0.5\n        +\n        0.5\n        \u22c5\n        \n          \n            \n              f\n              \n                t\n                ,\n                d\n              \n            \n            \n              max\n              {\n              \n                f\n                \n                  \n                    t\n                    \u2032", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0011", "text": "\u2032\n                  \n                  ,\n                  d\n                \n              \n              :\n              \n                t\n                \u2032\n              \n              \u2208\n              d\n              }\n            \n          \n        \n      \n    \n    {\\displaystyle \\mathrm {tf} (t,d)=0.5+0.5\\cdot {\\frac {f_{t,d}}{\\max\\{f_{t',d}:t'\\in d\\}}}}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0012", "text": "Inverse document frequency\nThe inverse document frequency is a measure of how much information the word provides, i.e., how common or rare it is across all documents. It is the logarithmically scaled inverse fraction of the documents that contain the word (obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient):", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0013", "text": "i\n          d\n          f\n        \n        (\n        t\n        ,\n        D\n        )\n        =\n        log\n        \u2061\n        \n          \n            N\n            \n              n\n              \n                t\n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\mathrm {idf} (t,D)=\\log {\\frac {N}{n_{t}}}}\n  \n\nwith\n\n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n: is the set of all documents in the corpus", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0014", "text": "N\n        =\n        \n          \n            |\n          \n          D\n          \n            |\n          \n        \n      \n    \n    {\\displaystyle N={|D|}}\n  \n: total number of documents in the corpus", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0015", "text": "n\n          \n            t\n          \n        \n        =\n        \n          |\n        \n        {\n        d\n        \u2208\n        D\n        :\n        t\n        \u2208\n        d\n        }\n        \n          |\n        \n      \n    \n    {\\displaystyle n_{t}=|\\{d\\in D:t\\in d\\}|}\n  \n : number of documents where the term \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n appears (i.e., \n  \n    \n      \n        \n          t\n          f\n        \n        (\n        t", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0016", "text": "f\n        \n        (\n        t\n        ,\n        d\n        )\n        \u2260\n        0\n      \n    \n    {\\displaystyle \\mathrm {tf} (t,d)\\neq 0}\n  \n). If the term is not in the corpus, this will lead to a division-by-zero. It is therefore common to adjust the numerator to \n  \n    \n      \n        1\n        +\n        N\n      \n    \n    {\\displaystyle 1+N}\n  \n and the denominator to \n  \n    \n      \n        1\n        +\n        \n          |\n        \n        {\n        d\n        \u2208\n        D", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0017", "text": "{\n        d\n        \u2208\n        D\n        :\n        t\n        \u2208\n        d\n        }\n        \n          |\n        \n      \n    \n    {\\displaystyle 1+|\\{d\\in D:t\\in d\\}|}\n  \n.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0018", "text": "Term frequency\u2013inverse document frequency\nThen tf\u2013idf is calculated as", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0019", "text": "t\n          f\n          i\n          d\n          f\n        \n        (\n        t\n        ,\n        d\n        ,\n        D\n        )\n        =\n        \n          t\n          f\n        \n        (\n        t\n        ,\n        d\n        )\n        \u22c5\n        \n          i\n          d\n          f\n        \n        (\n        t\n        ,\n        D\n        )\n      \n    \n    {\\displaystyle \\mathrm {tfidf} (t,d,D)=\\mathrm {tf} (t,d)\\cdot \\mathrm {idf} (t,D)}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0020", "text": "A high weight in tf\u2013idf is reached by a high term frequency (in the given document) and a low document frequency of the term in the whole collection of documents; the weights hence tend to filter out common terms. Since the ratio inside the idf's log function is always greater than or equal to 1, the value of idf (and tf\u2013idf) is greater than or equal to 0. As a term appears in more documents, the ratio inside the logarithm approaches 1, bringing the idf and tf\u2013idf closer to 0.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0021", "text": "Justification of idf\nIdf was introduced as \"term specificity\" by Karen Sp\u00e4rck Jones in a 1972 paper. Although it has worked well as a heuristic, its theoretical foundations have been troublesome for at least three decades afterward, with many researchers trying to find information theoretic justifications for it.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0022", "text": "Sp\u00e4rck Jones's own explanation did not propose much theory, aside from a connection to Zipf's law. Attempts have been made to put idf on a probabilistic footing, by estimating the probability that a given document d contains a term t as the relative document frequency,", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0023", "text": "P\n        (\n        t\n        \n          |\n        \n        D\n        )\n        =\n        \n          \n            \n              \n                |\n              \n              {\n              d\n              \u2208\n              D\n              :\n              t\n              \u2208\n              d\n              }\n              \n                |\n              \n            \n            N\n          \n        \n        ,", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0024", "text": ",\n      \n    \n    {\\displaystyle P(t|D)={\\frac {|\\{d\\in D:t\\in d\\}|}{N}},}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0025", "text": "so that we can define idf as", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0026", "text": "i\n                  d\n                  f\n                \n              \n              \n                \n                =\n                \u2212\n                log\n                \u2061\n                P\n                (\n                t\n                \n                  |\n                \n                D\n                )", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0027", "text": "=\n                log\n                \u2061\n                \n                  \n                    1\n                    \n                      P\n                      (\n                      t\n                      \n                        |\n                      \n                      D\n                      )", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0028", "text": "=\n                log\n                \u2061\n                \n                  \n                    N\n                    \n                      \n                        |\n                      \n                      {\n                      d\n                      \u2208\n                      D\n                      :\n                      t\n                      \u2208\n                      d\n                      }", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0029", "text": "d\n                      }\n                      \n                        |\n                      \n                    \n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}\\mathrm {idf} &=-\\log P(t|D)\\\\&=\\log {\\frac {1}{P(t|D)}}\\\\&=\\log {\\frac {N}{|\\{d\\in D:t\\in d\\}|}}\\end{aligned}}}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0030", "text": "Namely, the inverse document frequency is the logarithm of \"inverse\" relative document frequency.\nThis probabilistic interpretation in turn takes the same form as that of self-information. However, applying such information-theoretic notions to problems in information retrieval leads to problems when trying to define the appropriate event spaces for the required probability distributions: not only documents need to be taken into account, but also queries and terms.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0031", "text": "Link with information theory\nBoth term frequency and inverse document frequency can be formulated in terms of information theory; it helps to understand why their product has a meaning in terms of joint informational content of a document. A characteristic assumption about the distribution \n  \n    \n      \n        p\n        (\n        d\n        ,\n        t\n        )\n      \n    \n    {\\displaystyle p(d,t)}\n  \n is that:", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0032", "text": "p\n        (\n        d\n        \n          |\n        \n        t\n        )\n        =\n        \n          \n            1\n            \n              \n                |\n              \n              {\n              d\n              \u2208\n              D\n              :\n              t\n              \u2208\n              d\n              }\n              \n                |\n              \n            \n          \n        \n      \n    \n    {\\displaystyle p(d|t)={\\frac {1}{|\\{d\\in D:t\\in d\\}|}}}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0033", "text": "This assumption and its implications, according to Aizawa: \"represent the heuristic that tf\u2013idf employs.\"\nThe conditional entropy of a \"randomly chosen\" document in the corpus \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n, conditional to the fact it contains a specific term \n  \n    \n      \n        t\n      \n    \n    {\\displaystyle t}\n  \n (and assuming that all documents have equal probability to be chosen) is:", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0034", "text": "H\n        (\n        \n          \n            D\n          \n        \n        \n          |\n        \n        \n          \n            T\n          \n        \n        =\n        t\n        )\n        =\n        \u2212\n        \n          \u2211\n          \n            d\n          \n        \n        \n          p\n          \n            d\n            \n              |\n            \n            t\n          \n        \n        log\n        \u2061\n        \n          p\n          \n            d", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0035", "text": "p\n          \n            d\n            \n              |\n            \n            t\n          \n        \n        =\n        \u2212\n        log\n        \u2061\n        \n          \n            1\n            \n              \n                |\n              \n              {\n              d\n              \u2208\n              D\n              :\n              t\n              \u2208\n              d\n              }\n              \n                |\n              \n            \n          \n        \n        =\n        log", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0036", "text": "=\n        log\n        \u2061\n        \n          \n            \n              \n                |\n              \n              {\n              d\n              \u2208\n              D\n              :\n              t\n              \u2208\n              d\n              }\n              \n                |\n              \n            \n            \n              \n                |\n              \n              D\n              \n                |", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0037", "text": "+\n        log\n        \u2061\n        \n          |\n        \n        D\n        \n          |\n        \n        =\n        \u2212\n        \n          i\n          d\n          f\n        \n        (\n        t\n        )\n        +\n        log\n        \u2061\n        \n          |\n        \n        D\n        \n          |", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0038", "text": "|\n        \n      \n    \n    {\\displaystyle H({\\cal {D}}|{\\cal {T}}=t)=-\\sum _{d}p_{d|t}\\log p_{d|t}=-\\log {\\frac {1}{|\\{d\\in D:t\\in d\\}|}}=\\log {\\frac {|\\{d\\in D:t\\in d\\}|}{|D|}}+\\log |D|=-\\mathrm {idf} (t)+\\log |D|}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0039", "text": "In terms of notation, \n  \n    \n      \n        \n          \n            D\n          \n        \n      \n    \n    {\\displaystyle {\\cal {D}}}\n  \n and \n  \n    \n      \n        \n          \n            T\n          \n        \n      \n    \n    {\\displaystyle {\\cal {T}}}\n  \n are \"random variables\" corresponding to respectively draw a document or a term. The mutual information can be expressed as", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0040", "text": "M\n        (\n        \n          \n            T\n          \n        \n        ;\n        \n          \n            D\n          \n        \n        )\n        =\n        H\n        (\n        \n          \n            D\n          \n        \n        )\n        \u2212\n        H\n        (\n        \n          \n            D\n          \n        \n        \n          |\n        \n        \n          \n            T\n          \n        \n        )\n        =\n        \n          \u2211\n          \n            t", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0041", "text": "\u2211\n          \n            t\n          \n        \n        \n          p\n          \n            t\n          \n        \n        \u22c5\n        (\n        H\n        (\n        \n          \n            D\n          \n        \n        )\n        \u2212\n        H\n        (\n        \n          \n            D\n          \n        \n        \n          |\n        \n        W\n        =\n        t\n        )\n        )\n        =\n        \n          \u2211\n          \n            t\n          \n        \n        \n          p", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0042", "text": "p\n          \n            t\n          \n        \n        \u22c5\n        \n          i\n          d\n          f\n        \n        (\n        t\n        )\n      \n    \n    {\\displaystyle M({\\cal {T}};{\\cal {D}})=H({\\cal {D}})-H({\\cal {D}}|{\\cal {T}})=\\sum _{t}p_{t}\\cdot (H({\\cal {D}})-H({\\cal {D}}|W=t))=\\sum _{t}p_{t}\\cdot \\mathrm {idf} (t)}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0043", "text": "The last step is to expand \n  \n    \n      \n        \n          p\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle p_{t}}\n  \n, the unconditional probability to draw a term, with respect to the (random) choice of a document, to obtain:", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0044", "text": "M\n        (\n        \n          \n            T\n          \n        \n        ;\n        \n          \n            D\n          \n        \n        )\n        =\n        \n          \u2211\n          \n            t\n            ,\n            d\n          \n        \n        \n          p\n          \n            t\n            \n              |\n            \n            d\n          \n        \n        \u22c5\n        \n          p\n          \n            d\n          \n        \n        \u22c5\n        \n          i", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0045", "text": "\u22c5\n        \n          i\n          d\n          f\n        \n        (\n        t\n        )\n        =\n        \n          \u2211\n          \n            t\n            ,\n            d\n          \n        \n        \n          t\n          f\n        \n        (\n        t\n        ,\n        d\n        )\n        \u22c5\n        \n          \n            1\n            \n              \n                |\n              \n              D\n              \n                |", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0046", "text": "\u22c5\n        \n          i\n          d\n          f\n        \n        (\n        t\n        )\n        =\n        \n          \n            1\n            \n              \n                |\n              \n              D\n              \n                |\n              \n            \n          \n        \n        \n          \u2211\n          \n            t\n            ,\n            d\n          \n        \n        \n          t\n          f\n        \n        (\n        t", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0047", "text": "f\n        \n        (\n        t\n        ,\n        d\n        )\n        \u22c5\n        \n          i\n          d\n          f\n        \n        (\n        t\n        )\n        .\n      \n    \n    {\\displaystyle M({\\cal {T}};{\\cal {D}})=\\sum _{t,d}p_{t|d}\\cdot p_{d}\\cdot \\mathrm {idf} (t)=\\sum _{t,d}\\mathrm {tf} (t,d)\\cdot {\\frac {1}{|D|}}\\cdot \\mathrm {idf} (t)={\\frac {1}{|D|}}\\sum _{t,d}\\mathrm {tf} (t,d)\\cdot \\mathrm {idf} (t).}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0048", "text": "This expression shows that summing the Tf\u2013idf of all possible terms and documents recovers the mutual information between documents and term taking into account all the specificities of their joint distribution. Each Tf\u2013idf hence carries the \"bit of information\" attached to a term x document pair.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0049", "text": "Link with statistical theory\nTf\u2013idf is closely related to the negative logarithmically transformed p-value from a one-tailed formulation of Fisher's exact test when the underlying corpus documents satisfy certain idealized assumptions.\n\nExample of tf\u2013idf\nSuppose that we have term count tables of a corpus consisting of only two documents:\n\n\"This is a sample A.\"\n\"This is another example, another example, example.\"", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0050", "text": "The calculation of tf\u2013idf for the term \"this\" is performed as follows:\nIn its raw frequency form, tf is just the frequency of the \"this\" for each document. In each document, the word \"this\" appears once; but as the document 2 has more words, its relative frequency is smaller.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0051", "text": "t\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            t\n            h\n            i\n            \n              s\n              \u2033\n            \n          \n        \n        ,\n        \n          d\n          \n            1\n          \n        \n        )\n        =\n        \n          \n            1\n            5\n          \n        \n        =\n        0.2", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0052", "text": "=\n        0.2\n      \n    \n    {\\displaystyle \\mathrm {tf} ({\\mathsf {''this''}},d_{1})={\\frac {1}{5}}=0.2}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0053", "text": "t\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            t\n            h\n            i\n            \n              s\n              \u2033\n            \n          \n        \n        ,\n        \n          d\n          \n            2\n          \n        \n        )\n        =\n        \n          \n            1\n            7\n          \n        \n        \u2248\n        0.14", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0054", "text": "\u2248\n        0.14\n      \n    \n    {\\displaystyle \\mathrm {tf} ({\\mathsf {''this''}},d_{2})={\\frac {1}{7}}\\approx 0.14}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0055", "text": "An idf is constant per corpus, and accounts for the ratio of documents that include the word \"this\". In this case, we have a corpus of two documents and all of them include the word \"this\".", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0056", "text": "i\n          d\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            t\n            h\n            i\n            \n              s\n              \u2033\n            \n          \n        \n        ,\n        D\n        )\n        =\n        log\n        \u2061\n        \n          (\n          \n            \n              2\n              2\n            \n          \n          )\n        \n        =\n        0", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0057", "text": "=\n        0\n      \n    \n    {\\displaystyle \\mathrm {idf} ({\\mathsf {''this''}},D)=\\log \\left({\\frac {2}{2}}\\right)=0}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0058", "text": "So tf\u2013idf is zero for the word \"this\", which implies that the word is not very informative as it appears in all documents.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0059", "text": "t\n          f\n          i\n          d\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            t\n            h\n            i\n            \n              s\n              \u2033\n            \n          \n        \n        ,\n        \n          d\n          \n            1\n          \n        \n        ,\n        D\n        )\n        =\n        0.2\n        \u00d7\n        0\n        =\n        0", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0060", "text": "0\n        =\n        0\n      \n    \n    {\\displaystyle \\mathrm {tfidf} ({\\mathsf {''this''}},d_{1},D)=0.2\\times 0=0}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0061", "text": "t\n          f\n          i\n          d\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            t\n            h\n            i\n            \n              s\n              \u2033\n            \n          \n        \n        ,\n        \n          d\n          \n            2\n          \n        \n        ,\n        D\n        )\n        =\n        0.14\n        \u00d7\n        0\n        =\n        0", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0062", "text": "0\n        =\n        0\n      \n    \n    {\\displaystyle \\mathrm {tfidf} ({\\mathsf {''this''}},d_{2},D)=0.14\\times 0=0}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0063", "text": "The word \"example\" is more interesting - it occurs three times, but only in the second document:", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0064", "text": "t\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            e\n            x\n            a\n            m\n            p\n            l\n            \n              e\n              \u2033\n            \n          \n        \n        ,\n        \n          d\n          \n            1\n          \n        \n        )\n        =\n        \n          \n            0\n            5\n          \n        \n        =\n        0", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0065", "text": "=\n        0\n      \n    \n    {\\displaystyle \\mathrm {tf} ({\\mathsf {''example''}},d_{1})={\\frac {0}{5}}=0}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0066", "text": "t\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            e\n            x\n            a\n            m\n            p\n            l\n            \n              e\n              \u2033\n            \n          \n        \n        ,\n        \n          d\n          \n            2\n          \n        \n        )\n        =\n        \n          \n            3\n            7\n          \n        \n        \u2248\n        0.429", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0067", "text": "\u2248\n        0.429\n      \n    \n    {\\displaystyle \\mathrm {tf} ({\\mathsf {''example''}},d_{2})={\\frac {3}{7}}\\approx 0.429}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0068", "text": "i\n          d\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            e\n            x\n            a\n            m\n            p\n            l\n            \n              e\n              \u2033\n            \n          \n        \n        ,\n        D\n        )\n        =\n        log\n        \u2061\n        \n          (\n          \n            \n              2\n              1\n            \n          \n          )", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0069", "text": ")\n        \n        =\n        0.301\n      \n    \n    {\\displaystyle \\mathrm {idf} ({\\mathsf {''example''}},D)=\\log \\left({\\frac {2}{1}}\\right)=0.301}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0070", "text": "Finally,", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0071", "text": "t\n          f\n          i\n          d\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            e\n            x\n            a\n            m\n            p\n            l\n            \n              e\n              \u2033\n            \n          \n        \n        ,\n        \n          d\n          \n            1\n          \n        \n        ,\n        D\n        )\n        =\n        \n          t\n          f", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0072", "text": "=\n        \n          t\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            e\n            x\n            a\n            m\n            p\n            l\n            \n              e\n              \u2033\n            \n          \n        \n        ,\n        \n          d\n          \n            1\n          \n        \n        )\n        \u00d7\n        \n          i\n          d\n          f\n        \n        (", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0073", "text": "(\n        \n          \n            \n              \n              \u2033\n            \n            e\n            x\n            a\n            m\n            p\n            l\n            \n              e\n              \u2033\n            \n          \n        \n        ,\n        D\n        )\n        =\n        0\n        \u00d7\n        0.301\n        =\n        0", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0074", "text": "0.301\n        =\n        0\n      \n    \n    {\\displaystyle \\mathrm {tfidf} ({\\mathsf {''example''}},d_{1},D)=\\mathrm {tf} ({\\mathsf {''example''}},d_{1})\\times \\mathrm {idf} ({\\mathsf {''example''}},D)=0\\times 0.301=0}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0075", "text": "t\n          f\n          i\n          d\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            e\n            x\n            a\n            m\n            p\n            l\n            \n              e\n              \u2033\n            \n          \n        \n        ,\n        \n          d\n          \n            2\n          \n        \n        ,\n        D\n        )\n        =\n        \n          t\n          f", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0076", "text": "=\n        \n          t\n          f\n        \n        (\n        \n          \n            \n              \n              \u2033\n            \n            e\n            x\n            a\n            m\n            p\n            l\n            \n              e\n              \u2033\n            \n          \n        \n        ,\n        \n          d\n          \n            2\n          \n        \n        )\n        \u00d7\n        \n          i\n          d\n          f\n        \n        (", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0077", "text": "(\n        \n          \n            \n              \n              \u2033\n            \n            e\n            x\n            a\n            m\n            p\n            l\n            \n              e\n              \u2033\n            \n          \n        \n        ,\n        D\n        )\n        =\n        0.429\n        \u00d7\n        0.301\n        \u2248\n        0.129", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0078", "text": "0.301\n        \u2248\n        0.129\n      \n    \n    {\\displaystyle \\mathrm {tfidf} ({\\mathsf {''example''}},d_{2},D)=\\mathrm {tf} ({\\mathsf {''example''}},d_{2})\\times \\mathrm {idf} ({\\mathsf {''example''}},D)=0.429\\times 0.301\\approx 0.129}", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0079", "text": "(using the base 10 logarithm).", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0080", "text": "Beyond terms", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0081", "text": "The idea behind tf\u2013idf also applies to entities other than terms. In 1998, the concept of idf was applied to citations. The authors argued that \"if a very uncommon citation is shared by two documents, this should be weighted more highly than a citation made by a large number of documents\". In addition, tf\u2013idf was applied to \"visual words\" with the purpose of conducting object matching in videos, and entire sentences", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0082", "text": ". However, the concept of tf\u2013idf did not prove to be more effective in all cases than a plain tf scheme (without idf). When tf\u2013idf was applied to citations, researchers could find no improvement over a simple citation-count weight that had no idf component.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0083", "text": "Derivatives", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0084", "text": "A number of term-weighting schemes have derived from tf\u2013idf. One of them is TF\u2013PDF (term frequency * proportional document frequency). TF\u2013PDF was introduced in 2001 in the context of identifying emerging topics in the media. The PDF component measures the difference of how often a term occurs in different domains. Another derivate is TF\u2013IDuF. In TF\u2013IDuF, idf is not calculated based on the document corpus that is to be searched or recommended", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0085", "text": ". Instead, idf is calculated on users' personal document collections. The authors report that TF\u2013IDuF was equally effective as tf\u2013idf but could also be applied in situations when, e.g., a user modeling system has no access to a global document corpus. The DELTA TF-IDF  derivative uses the difference in importance of a term across two specific classes, like positive and negative sentiment", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0086", "text": ". For example, it can assign a high score to a word like \"excellent\" in positive reviews and a low score to the same word in negative reviews. This helps identify words that strongly indicate the sentiment of a document, potentially leading to improved accuracy in text classification tasks.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0087", "text": "See also\nReferences\nSalton, G; McGill, M. J. (1986). Introduction to modern information retrieval. McGraw-Hill. ISBN 978-0-07-054484-0.\nSalton, G.; Fox, E. A.; Wu, H. (1983). \"Extended Boolean information retrieval\". Communications of the ACM. 26 (11): 1022\u20131036. doi:10.1145/182.358466. hdl:1813/6351. S2CID 207180535.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0088", "text": "Salton, G.; Buckley, C. (1988). \"Term-weighting approaches in automatic text retrieval\" (PDF). Information Processing & Management. 24 (5): 513\u2013523. doi:10.1016/0306-4573(88)90021-0. hdl:1813/6721. S2CID 7725217.\nWu, H. C.; Luk, R.W.P.; Wong, K.F.; Kwok, K.L. (2008). \"Interpreting TF-IDF term weights as making relevance decisions\". ACM Transactions on Information Systems. 26 (3): 1. doi:10.1145/1361684.1361686. hdl:10397/10130. S2CID 18303048.", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0089", "text": "External links and suggested reading\nGensim is a Python library for vector space modeling and includes tf\u2013idf weighting.\nAnatomy of a search engine Archived 2009-03-10 at the Wayback Machine\ntf\u2013idf and related definitions as used in Lucene\nTfidfTransformer in scikit-learn", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Tfidf_0090", "text": "TfidfTransformer in scikit-learn\nText to Matrix Generator (TMG)  MATLAB toolbox that can be used for various tasks in text mining (TM) specifically  i) indexing, ii) retrieval, iii) dimensionality reduction, iv) clustering, v) classification. The indexing step offers the user the ability to apply local and global weighting methods, including tf\u2013idf.\nTerm-frequency explained Explanation of term-frequency", "title": "Tf\u2013idf", "url": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"}, {"id": "rec_Okapi_BM25_0000", "text": "In information retrieval, Okapi BM25 (BM is an abbreviation of best matching) is a ranking function used by search engines to estimate the relevance of documents to a given search query. It is based on the probabilistic retrieval framework developed in the 1970s and 1980s by Stephen E. Robertson, Karen Sp\u00e4rck Jones, and others.", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0001", "text": "The name of the actual ranking function is BM25. The fuller name, Okapi BM25, includes the name of the first system to use it, which was the Okapi information retrieval system, implemented at London's City University in the 1980s and 1990s. BM25 and its newer variants, e.g. BM25F (a version of BM25 that can take document structure and anchor text into account), represent TF-IDF-like retrieval functions used in document retrieval.", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0002", "text": "The ranking function\nBM25 is a bag-of-words retrieval function that ranks a set of documents based on the query terms appearing in each document, regardless of their proximity within the document. It is a family of scoring functions with slightly different components and parameters. One of the most prominent instantiations of the function is as follows.\nGiven a query Q, containing keywords \n  \n    \n      \n        \n          q\n          \n            1\n          \n        \n        ,\n        .", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0003", "text": ",\n        .\n        .\n        .\n        ,\n        \n          q\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle q_{1},...,q_{n}}\n  \n, the BM25 score of a document D is:", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0004", "text": "score\n        \n        (\n        D\n        ,\n        Q\n        )\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          IDF\n        \n        (\n        \n          q\n          \n            i\n          \n        \n        )\n        \u22c5\n        \n          \n            \n              f\n              (\n              \n                q", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0005", "text": "q\n                \n                  i\n                \n              \n              ,\n              D\n              )\n              \u22c5\n              (\n              \n                k\n                \n                  1\n                \n              \n              +\n              1\n              )\n            \n            \n              f\n              (\n              \n                q\n                \n                  i", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0006", "text": ",\n              D\n              )\n              +\n              \n                k\n                \n                  1\n                \n              \n              \u22c5\n              \n                (\n                \n                  1\n                  \u2212\n                  b\n                  +\n                  b\n                  \u22c5\n                  \n                    \n                      \n                        \n                          |", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0007", "text": "|\n                        \n                        D\n                        \n                          |\n                        \n                      \n                      avgdl\n                    \n                  \n                \n                )", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0008", "text": "{\\displaystyle {\\text{score}}(D,Q)=\\sum _{i=1}^{n}{\\text{IDF}}(q_{i})\\cdot {\\frac {f(q_{i},D)\\cdot (k_{1}+1)}{f(q_{i},D)+k_{1}\\cdot \\left(1-b+b\\cdot {\\frac {|D|}{\\text{avgdl}}}\\right)}}}", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0009", "text": "where \n  \n    \n      \n        f\n        (\n        \n          q\n          \n            i\n          \n        \n        ,\n        D\n        )\n      \n    \n    {\\displaystyle f(q_{i},D)}\n  \n is the number of times that the keyword \n  \n    \n      \n        \n          q\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle q_{i}}\n  \n occurs in the document D, \n  \n    \n      \n        \n          |\n        \n        D\n        \n          |\n        \n      \n    \n    {\\displaystyle |D|}", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0010", "text": "{\\displaystyle |D|}\n  \n is the length of the document D in words, and avgdl is the average document length in the text collection from which documents are drawn. \n  \n    \n      \n        \n          k\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle k_{1}}\n  \n and b are free parameters, usually chosen, in absence of an advanced optimization, as \n  \n    \n      \n        \n          k\n          \n            1\n          \n        \n        \u2208\n        [", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0011", "text": "\u2208\n        [\n        1.2\n        ,\n        2.0\n        ]\n      \n    \n    {\\displaystyle k_{1}\\in [1.2,2.0]}\n  \n and \n  \n    \n      \n        b\n        =\n        0.75\n      \n    \n    {\\displaystyle b=0.75}\n  \n. \n  \n    \n      \n        \n          IDF\n        \n        (\n        \n          q\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle {\\text{IDF}}(q_{i})}\n  \n is the IDF (inverse document frequency) weight of the query term", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0012", "text": "q\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle q_{i}}\n  \n. It is usually computed as:", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0013", "text": "IDF\n        \n        (\n        \n          q\n          \n            i\n          \n        \n        )\n        =\n        ln\n        \u2061\n        \n          (\n          \n            \n              \n                \n                  N\n                  \u2212\n                  n\n                  (\n                  \n                    q\n                    \n                      i\n                    \n                  \n                  )\n                  +", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0014", "text": ")\n                  +\n                  0.5\n                \n                \n                  n\n                  (\n                  \n                    q\n                    \n                      i\n                    \n                  \n                  )\n                  +\n                  0.5\n                \n              \n            \n            +\n            1\n          \n          )", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0015", "text": ")\n        \n      \n    \n    {\\displaystyle {\\text{IDF}}(q_{i})=\\ln \\left({\\frac {N-n(q_{i})+0.5}{n(q_{i})+0.5}}+1\\right)}", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0016", "text": "where  N is the total number of documents in the collection, and \n  \n    \n      \n        n\n        (\n        \n          q\n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle n(q_{i})}\n  \n is the number of documents containing \n  \n    \n      \n        \n          q\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle q_{i}}\n  \n.", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0017", "text": "{\\displaystyle q_{i}}\n  \n.\nThere are several interpretations for IDF and slight variations on its formula. In the original BM25 derivation, the IDF component is derived from the Binary Independence Model.", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0018", "text": "IDF information theoretic interpretation\nHere is an interpretation from information theory. Suppose a query term \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n appears in \n  \n    \n      \n        n\n        (\n        q\n        )\n      \n    \n    {\\displaystyle n(q)}\n  \n documents. Then a randomly picked document \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n will contain the term with probability \n  \n    \n      \n        \n          \n            \n              n", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0019", "text": "n\n              (\n              q\n              )\n            \n            N\n          \n        \n      \n    \n    {\\displaystyle {\\frac {n(q)}{N}}}\n  \n (where \n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}\n  \n is again the cardinality of the set of documents in the collection). Therefore, the information content of the message \"\n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n contains \n  \n    \n      \n        q", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0020", "text": "contains \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n\" is:", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0021", "text": "\u2212\n        log\n        \u2061\n        \n          \n            \n              n\n              (\n              q\n              )\n            \n            N\n          \n        \n        =\n        log\n        \u2061\n        \n          \n            N\n            \n              n\n              (\n              q\n              )\n            \n          \n        \n        .\n      \n    \n    {\\displaystyle -\\log {\\frac {n(q)}{N}}=\\log {\\frac {N}{n(q)}}.}", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0022", "text": "Now suppose we have two query terms \n  \n    \n      \n        \n          q\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle q_{1}}\n  \n and \n  \n    \n      \n        \n          q\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle q_{2}}\n  \n. If the two terms occur in documents entirely independently of each other, then the probability of seeing both \n  \n    \n      \n        \n          q\n          \n            1", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0023", "text": "1\n          \n        \n      \n    \n    {\\displaystyle q_{1}}\n  \n and \n  \n    \n      \n        \n          q\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle q_{2}}\n  \n in a randomly picked document \n  \n    \n      \n        D\n      \n    \n    {\\displaystyle D}\n  \n is:", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0024", "text": "n\n              (\n              \n                q\n                \n                  1\n                \n              \n              )\n            \n            N\n          \n        \n        \u22c5\n        \n          \n            \n              n\n              (\n              \n                q\n                \n                  2\n                \n              \n              )\n            \n            N\n          \n        \n        ,", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0025", "text": "N\n          \n        \n        ,\n      \n    \n    {\\displaystyle {\\frac {n(q_{1})}{N}}\\cdot {\\frac {n(q_{2})}{N}},}", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0026", "text": "and the information content of such an event is:", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0027", "text": "\u2211\n          \n            i\n            =\n            1\n          \n          \n            2\n          \n        \n        log\n        \u2061\n        \n          \n            N\n            \n              n\n              (\n              \n                q\n                \n                  i\n                \n              \n              )\n            \n          \n        \n        .\n      \n    \n    {\\displaystyle \\sum _{i=1}^{2}\\log {\\frac {N}{n(q_{i})}}.}", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0028", "text": "With a small variation, this is exactly what is expressed by the IDF component of BM25.", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0029", "text": "Modifications\nAt the extreme values of the coefficient b BM25 turns into ranking functions known as BM11 (for \n  \n    \n      \n        b\n        =\n        1\n      \n    \n    {\\displaystyle b=1}\n  \n) and BM15 (for \n  \n    \n      \n        b\n        =\n        0\n      \n    \n    {\\displaystyle b=0}\n  \n).", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0030", "text": "{\\displaystyle b=0}\n  \n).\nBM25F (or the BM25 model with Extension to Multiple Weighted Fields) is a modification of BM25 in which the document is considered to be composed from several fields (such as headlines, main text, anchor text) with possibly different degrees of importance, term relevance saturation and length normalization. BM25F defines each type of field as a stream, applying a per-stream weighting to scale each stream against the calculated score.", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0031", "text": "BM25+ is an extension of BM25. BM25+ was developed to address one deficiency of the standard BM25 in which the component of term frequency normalization by document length is not properly lower-bounded; as a result of this deficiency, long documents which do match the query term can often be scored unfairly by BM25 as having a similar relevancy to shorter documents that do not contain the query term at all. The scoring formula of BM25+ only has one additional free parameter", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0032", "text": "\u03b4\n      \n    \n    {\\displaystyle \\delta }\n  \n (a default value is 1.0 in absence of a training data) as compared with BM25:", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0033", "text": "score\n        \n        (\n        D\n        ,\n        Q\n        )\n        =\n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            n\n          \n        \n        \n          IDF\n        \n        (\n        \n          q\n          \n            i\n          \n        \n        )\n        \u22c5\n        \n          [\n          \n            \n              \n                \n                  f\n                  (", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0034", "text": "f\n                  (\n                  \n                    q\n                    \n                      i\n                    \n                  \n                  ,\n                  D\n                  )\n                  \u22c5\n                  (\n                  \n                    k\n                    \n                      1\n                    \n                  \n                  +\n                  1\n                  )", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0035", "text": "f\n                  (\n                  \n                    q\n                    \n                      i\n                    \n                  \n                  ,\n                  D\n                  )\n                  +\n                  \n                    k\n                    \n                      1\n                    \n                  \n                  \u22c5\n                  \n                    (", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0036", "text": "(\n                    \n                      1\n                      \u2212\n                      b\n                      +\n                      b\n                      \u22c5\n                      \n                        \n                          \n                            \n                              |\n                            \n                            D\n                            \n                              |", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0037", "text": "avgdl\n                        \n                      \n                    \n                    )\n                  \n                \n              \n            \n            +\n            \u03b4\n          \n          ]\n        \n      \n    \n    {\\displaystyle {\\text{score}}(D,Q)=\\sum _{i=1}^{n}{\\text{IDF}}(q_{i})\\cdot \\left[{\\frac {f(q_{i},D)\\cdot (k_{1}+1)}{f(q_{i},D)+k_{1}\\cdot \\left(1-b+b\\cdot {\\frac {|D|}{\\text{avgdl}}}\\right)}}+\\delta \\right]}", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0038", "text": "References\nGeneral references\nStephen E. Robertson; Steve Walker; Susan Jones; Micheline Hancock-Beaulieu & Mike Gatford (November 1994). Okapi at TREC-3. Proceedings of the Third Text REtrieval Conference (TREC 1994). Gaithersburg, USA.\nStephen E. Robertson; Steve Walker & Micheline Hancock-Beaulieu (November 1998). Okapi at TREC-7. Proceedings of the Seventh Text REtrieval Conference. Gaithersburg, USA.", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0039", "text": "Sp\u00e4rck Jones, K.; Walker, S.; Robertson, S. E. (2000). \"A probabilistic model of information retrieval: Development and comparative experiments: Part 1\". Information Processing & Management. 36 (6): 779\u2013808. CiteSeerX 10.1.1.134.6108. doi:10.1016/S0306-4573(00)00015-7.", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0040", "text": "Sp\u00e4rck Jones, K.; Walker, S.; Robertson, S. E. (2000). \"A probabilistic model of information retrieval: Development and comparative experiments: Part 2\". Information Processing & Management. 36 (6): 809\u2013840. doi:10.1016/S0306-4573(00)00016-9.\nStephen Robertson & Hugo Zaragoza (2009). \"The Probabilistic Relevance Framework: BM25 and Beyond\". Foundations and Trends in Information Retrieval. 3 (4): 333\u2013389. CiteSeerX 10.1.1.156.5282. doi:10.1561/1500000019. S2CID 207178704.", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Okapi_BM25_0041", "text": "External links\nRobertson, Stephen; Zaragoza, Hugo (2009). The Probabilistic Relevance Framework: BM25 and Beyond (PDF). NOW Publishers, Inc. ISBN 978-1-60198-308-4.", "title": "Okapi BM25", "url": "https://en.wikipedia.org/wiki/Okapi_BM25"}, {"id": "rec_Recommender_system_0000", "text": "A recommender system, also called a recommendation algorithm, recommendation engine, or recommendation platform, is a type of information filtering system that suggests items most relevant to a particular user. The value of these systems becomes particularly evident in scenarios where users must select from a large number of options, such as products, media, or content", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0001", "text": ". Major social media platforms and streaming services rely on recommender systems that employ machine learning to analyze user behavior and preferences, thereby enabling personalized content feeds.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0002", "text": "Typically, the suggestions refer to a variety decision-making processes, including the selection of a product, musical selection, or online news source to read. The implementation of recommender systems is pervasive, with commonly recognised examples including the generation of playlist for video and music services, the provision of product recommendations for e-commerce platforms, and the recommendation of content on social media platforms and the open web", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0003", "text": ". These systems can operate using a single type of input, such as music, or multiple inputs from diverse platforms, including news, books and search queries. Additionally, popular recommender systems have been developed for specific topics, such as restaurants and online dating services. Recommender systems have also been developed to explore research articles and experts, collaborators, and financial services.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0004", "text": "A content discovery platform is a software recommendation platform that employs recommender system tools. It utilizes user metadata in order to identify and suggest relevant content, whilst reducing ongoing maintenance and development costs. A content discovery platform delivers personalized content to websites, mobile devices, and set-top boxes", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0005", "text": ". A large range of content discovery platforms currently exist for various forms of content ranging from news articles and academic journal articles to television. As operators compete to serve as the gateway to home entertainment, personalized television emerges as a key service differentiator", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0006", "text": ". Academic content discovery has recently become another area of interest, the emergence of numerous companies dedicated to assisting academic researchers in keeping up to date with relevant academic content and facilitating serendipitous discovery of new content.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0007", "text": "Overview", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0008", "text": "Recommender systems usually make use of either or both collaborative filtering and content-based filtering, as well as other systems such as knowledge-based systems. Collaborative filtering approaches build a model from a user's past behavior (e.g., items previously purchased or selected and/or numerical ratings given to those items) as well as similar decisions made by other users. This model is then used to predict items (or ratings for items) that the user may have an interest in", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0009", "text": ". Content-based filtering approaches utilize a series of discrete, pre-tagged characteristics of an item in order to recommend additional items with similar properties.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0010", "text": "Example\nThe differences between collaborative and content-based filtering can be demonstrated by comparing two early music recommender systems, Last.fm and Pandora Radio. We can also look at how these methods are applied in e-commerce, for example, on platforms like Amazon.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0011", "text": "Last.fm creates a \"station\" of recommended songs by observing what bands and individual tracks the user has listened to on a regular basis and comparing those against the listening behavior of other users. Last.fm will play tracks that do not appear in the user's library, but are often played by other users with similar interests. As this approach leverages the behavior of users, it is an example of a collaborative filtering technique.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0012", "text": "Pandora uses the properties of a song or artist (a subset of the 450 attributes provided by the Music Genome Project) to seed a \"station\" that plays music with similar properties. User feedback is used to refine the station's results, deemphasizing certain attributes when a user \"dislikes\" a particular song and emphasizing other attributes when a user \"likes\" a song. This is an example of a content-based approach.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0013", "text": "In e-commerce, Amazon's well-known \"customers who bought X also bought Y\" feature is a prime example of collaborative filtering. It also uses content-based filtering when it recommends a book by the same author you've previously read or a pair of shoes in a similar style to ones you've viewed.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0014", "text": "Each type of system has its strengths and weaknesses. In the above example, Last.fm requires a large amount of information about a user to make accurate recommendations. This is an example of the cold start problem, and is common in collaborative filtering systems. Whereas Pandora needs very little information to start, it is far more limited in scope (for example, it can only make recommendations that are similar to the original seed).", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0015", "text": "Alternative implementations\nRecommender systems are a useful alternative to search algorithms since they help users discover items they might not have found otherwise. Of note, recommender systems are often implemented using search engines indexing non-traditional data. In some cases, like in the Gonzalez v. Google Supreme Court case, may argue that search and recommendation algorithms are different technologies.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0016", "text": "Recommender systems have been the focus of several granted patents, and there are more than 50 software libraries that support the development of recommender systems including LensKit, RecBole, ReChorus and RecPack.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0017", "text": "History\nElaine Rich created the first recommender system in 1979, called Grundy. She looked for a way to recommend users books they might like. Her idea was to create a system that asks users specific questions and classifies them into classes of preferences, or \"stereotypes\", depending on their answers. Depending on users' stereotype membership, they would then get recommendations for books they might like.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0018", "text": "Another early recommender system, called a \"digital bookshelf\", was described in a 1990 technical report by Jussi Karlgren at Columbia University,", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0019", "text": "and implemented at scale and worked through in technical reports and publications from 1994 onwards by Jussi Karlgren, then at SICS,\nand research groups led by Pattie Maes at MIT, Will Hill at Bellcore, and Paul Resnick, also at MIT, whose work with GroupLens was awarded the 2010 ACM Software Systems Award.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0020", "text": "Montaner provided the first overview of recommender systems from an intelligent agent perspective. Adomavicius provided a new, alternate overview of recommender systems.  Herlocker provides an additional overview of evaluation techniques for recommender systems, and Beel et al. discussed the problems of offline evaluations. Beel et al. have also provided literature surveys on available research paper recommender systems and existing challenges.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0021", "text": "Approaches\nCollaborative filtering", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0022", "text": "One approach to the design of recommender systems that has wide use is collaborative filtering. Collaborative filtering is based on the assumption that people who agreed in the past will agree in the future, and that they will like similar kinds of items as they liked in the past. The system generates recommendations using only information about rating profiles for different users or items", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0023", "text": ". By locating peer users/items with a rating history similar to the current user or item, they generate recommendations using this neighborhood. This approach is a cornerstone for e-commerce sites that analyze the purchasing patterns of thousands of users to suggest what you might like. Collaborative filtering methods are classified as memory-based and model-based", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0024", "text": ". A well-known example of memory-based approaches is the user-based algorithm, while that of model-based approaches is matrix factorization (recommender systems).", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0025", "text": "A key advantage of the collaborative filtering approach is that it does not rely on machine analyzable content and therefore it is capable of accurately recommending complex items such as movies without requiring an \"understanding\" of the item itself. Many algorithms have been used in measuring user similarity or item similarity in recommender systems. For example, the k-nearest neighbor (k-NN) approach and the Pearson Correlation as first implemented by Allen.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0026", "text": "When building a model from a user's behavior, a distinction is often made between explicit and implicit forms of data collection.\nExamples of explicit data collection include the following:", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0027", "text": "Asking a user to rate an item on a sliding scale.\nAsking a user to search.\nAsking a user to rank a collection of items from favorite to least favorite.\nPresenting two items to a user and asking him/her to choose the better one of them.\nAsking a user to create a list of items that he/she likes (see Rocchio classification or other similar techniques).\nExamples of implicit data collection include the following:", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0028", "text": "Observing the items that a user views in an online store.\nAnalyzing item/user viewing times.\nKeeping a record of the items that a user purchases online.\nObtaining a list of items that a user has listened to or watched on his/her computer.\nAnalyzing the user's social network and discovering similar likes and dislikes.\nCollaborative filtering approaches often suffer from three problems: cold start, scalability, and sparsity.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0029", "text": "Cold start: For a new user or item, there is not enough data to make accurate recommendations. Note: one commonly implemented solution to this problem is the multi-armed bandit algorithm.\nScalability: There are millions of users and products in many of the environments in which these systems make recommendations. Thus, a large amount of computation power is often necessary to calculate recommendations.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0030", "text": "Sparsity: The number of items sold on major e-commerce sites is extremely large. The most active users will only have rated a small subset of the overall database. Thus, even the most popular items have very few ratings.\nOne of the most famous examples of collaborative filtering is item-to-item collaborative filtering (people who buy x also buy y), an algorithm popularized by Amazon.com's recommender system.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0031", "text": "Many social networks originally used collaborative filtering to recommend new friends, groups, and other social connections by examining the network of connections between a user and their friends. Collaborative filtering is still used as part of hybrid systems. This technique can employ embeddings, a machine learning technique.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0032", "text": "Content-based filtering", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0033", "text": "Another common approach when designing recommender systems is content-based filtering. Content-based filtering methods are based on a description of the item and a profile of the user's preferences. These methods are best suited to situations where there is known data on an item (name, location, description, etc.), but not on the user", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0034", "text": ". Content-based recommenders treat recommendation as a user-specific classification problem and learn a classifier for the user's likes and dislikes based on an item's features.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0035", "text": "In this system, keywords are used to describe the items, and a user profile is built to indicate the type of item this user likes. In other words, these algorithms try to recommend items similar to those that a user liked in the past or is examining in the present. It does not rely on a user sign-in mechanism to generate this often temporary profile. In particular, various candidate items are compared with items previously rated by the user, and the best-matching items are recommended", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0036", "text": ". This approach has its roots in information retrieval and information filtering research.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0037", "text": "To create a user profile, the system mostly focuses on two types of information:", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0038", "text": "A model of the user's preference.\nA history of the user's interaction with the recommender system.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0039", "text": "Basically, these methods use an item profile (i.e., a set of discrete attributes and features) characterizing the item within the system. To abstract the features of the items in the system, an item presentation algorithm is applied. A widely used algorithm is the tf\u2013idf representation (also called vector space representation). The system creates a content-based profile of users based on a weighted vector of item features", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0040", "text": ". The weights denote the importance of each feature to the user and can be computed from individually rated content vectors using a variety of techniques. Simple approaches use the average values of the rated item vector while other sophisticated methods use machine learning techniques such as Bayesian Classifiers, cluster analysis, decision trees, and artificial neural networks in order to estimate the probability that the user is going to like the item.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0041", "text": "A key issue with content-based filtering is whether the system can learn user preferences from users' actions regarding one content source and use them across other content types. When the system is limited to recommending content of the same type as the user is already using, the value from the recommendation system is significantly less than when other content types from other services can be recommended. For example, recommending news articles based on news browsing is useful", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0042", "text": ". Still, it would be much more useful when music, videos, products, discussions, etc., from different services, can be recommended based on news browsing. To overcome this, most content-based recommender systems now use some form of the hybrid system.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0043", "text": "Content-based recommender systems can also include opinion-based recommender systems. In some cases, users are allowed to leave text reviews or feedback on the items. These user-generated texts are implicit data for the recommender system because they are potentially rich resources of both feature/aspects of the item and users' evaluation/sentiment to the item", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0044", "text": ". Features extracted from the user-generated reviews are improved metadata of items, because as they also reflect aspects of the item like metadata, extracted features are widely concerned by the users. Sentiments extracted from the reviews can be seen as users' rating scores on the corresponding features", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0045", "text": ". Popular approaches of opinion-based recommender system utilize various techniques including text mining, information retrieval, sentiment analysis (see also Multimodal sentiment analysis) and deep learning.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0046", "text": "Hybrid recommendations approaches", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0047", "text": "Most recommender systems now use a hybrid approach, combining collaborative filtering, content-based filtering, and other approaches. E-commerce platforms frequently use hybrid approaches to overcome problems like the cold start problem, where a new user has no history for collaborative filtering to analyze. There is no reason why several different techniques of the same type could not be hybridized", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0048", "text": ". Hybrid approaches can be implemented in several ways: by making content-based and collaborative-based predictions separately and then combining them; by adding content-based capabilities to a collaborative-based approach (and vice versa); or by unifying the approaches into one model", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0049", "text": ". Several studies that empirically compared the performance of the hybrid with the pure collaborative and content-based methods and demonstrated that the hybrid methods can provide more accurate recommendations than pure approaches. These methods can also be used to overcome some of the common problems in recommender systems such as cold start and the sparsity problem, as well as the knowledge engineering bottleneck in knowledge-based approaches.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0050", "text": "Netflix is a good example of the use of hybrid recommender systems. The website makes recommendations by comparing the watching and searching habits of similar users (i.e., collaborative filtering) as well as by offering movies that share characteristics with films that a user has rated highly (content-based filtering).\nSome hybridization techniques include:", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0051", "text": "Weighted: Combining the score of different recommendation components numerically.\nSwitching: Choosing among recommendation components and applying the selected one.\nMixed: Recommendations from different recommenders are presented together to give the recommendation.\nCascade: Recommenders are given strict priority, with the lower priority ones breaking ties in the scoring of the higher ones.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0052", "text": "Meta-level: One recommendation technique is applied and produces some sort of model, which is then the input used by the next technique.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0053", "text": "Technologies\nSession-based recommender systems", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0054", "text": "These recommender systems use the interactions of a user within a session to generate recommendations. Session-based recommender systems are used at YouTube and Amazon. These are particularly useful when history (such as past clicks, purchases) of a user is not available or not relevant in the current user session. Domains where session-based recommendations are particularly relevant include video, e-commerce, travel, music and more", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0055", "text": ". Most instances of session-based recommender systems rely on the sequence of recent interactions within a session without requiring any additional details (historical, demographic) of the user. Techniques for session-based recommendations are mainly based on generative sequential models such as recurrent neural networks, transformers, and other deep-learning-based approaches.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0056", "text": "Reinforcement learning for recommender systems", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0057", "text": "The recommendation problem can be seen as a special instance of a reinforcement learning problem whereby the user is the environment upon which the agent, the recommendation system acts upon in order to receive a reward, for instance, a click or engagement by the user. One aspect of reinforcement learning that is of particular use in the area of recommender systems is the fact that the models or policies can be learned by providing a reward to the recommendation agent", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0058", "text": ". This is in contrast to traditional learning techniques which rely on supervised learning approaches that are less flexible, reinforcement learning recommendation techniques allow to potentially train models that can be optimized directly on metrics of engagement, and user interest.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0059", "text": "Multi-criteria recommender systems", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0060", "text": "Multi-criteria recommender systems (MCRS) can be defined as recommender systems that incorporate preference information upon multiple criteria. Instead of developing recommendation techniques based on a single criterion value, the overall preference of user u for the item i, these systems try to predict a rating for unexplored items of u by exploiting preference information on multiple criteria that affect this overall preference value", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0061", "text": ". Several researchers approach MCRS as a multi-criteria decision making (MCDM) problem, and apply MCDM methods and techniques to implement MCRS systems. See this chapter for an extended introduction.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0062", "text": "Risk-aware recommender systems", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0063", "text": "The majority of existing approaches to recommender systems focus on recommending the most relevant content to users using contextual information, yet do not take into account the risk of disturbing the user with unwanted notifications. It is important to consider the risk of upsetting the user by pushing recommendations in certain circumstances, for instance, during a professional meeting, early morning, or late at night", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0064", "text": ". Therefore, the performance of the recommender system depends in part on the degree to which it has incorporated the risk into the recommendation process. One option to manage this issue is DRARS, a system which models the context-aware recommendation as a bandit problem. This system combines a content-based technique and a contextual bandit algorithm.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0065", "text": "Mobile recommender systems\nMobile recommender systems make use of internet-accessing smartphones to offer personalized, context-sensitive recommendations. This is a particularly difficult area of research as mobile data is more complex than data that recommender systems often have to deal with. It is heterogeneous, noisy, requires spatial and temporal auto-correlation, and has validation and generality problems.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0066", "text": "There are three factors that could affect the mobile recommender systems and the accuracy of prediction results: the context, the recommendation method and privacy. Additionally, mobile recommender systems suffer from a transplantation problem \u2013 recommendations may not apply in all regions (for instance, it would be unwise to recommend a recipe in an area where all of the ingredients may not be available).", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0067", "text": "One example of a mobile recommender system are the approaches taken by companies such as Uber and Lyft to generate driving routes for taxi drivers in a city. This system uses GPS data of the routes that taxi drivers take while working, which includes location (latitude and longitude), time stamps, and operational status (with or without passengers). It uses this data to recommend a list of pickup points along a route, with the goal of optimizing occupancy times and profits.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0068", "text": "Generative recommenders", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0069", "text": "Generative recommenders (GR) represent an approach that transforms recommendation tasks into sequential transduction problems, where user actions are treated like tokens in a generative modeling framework", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0070", "text": ". In one method, known as HSTU (Hierarchical Sequential Transduction Units), high-cardinality, non-stationary, and streaming datasets are efficiently processed as sequences, enabling the model to learn from trillions of parameters and to handle user action histories orders of magnitude longer than before", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0071", "text": ". By turning all of the system\u2019s varied data into a single stream of tokens and using a custom self-attention approach instead of traditional neural network layers, generative recommenders make the model much simpler and less memory-hungry. As a result, it can improve recommendation quality in test simulations and in real-world tests, while being faster than previous Transformer-based systems when handling long lists of user actions", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0072", "text": ". Ultimately, this approach allows the model\u2019s performance to grow steadily as more computing power is used, laying a foundation for efficient and scalable \u201cfoundation models\u201d for recommendations.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0073", "text": "The Netflix Prize", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0074", "text": "One of the events that energized research in recommender systems was the Netflix Prize. From 2006 to 2009, Netflix sponsored a competition, offering a grand prize of $1,000,000 to the team that could take an offered dataset of over 100 million movie ratings and return recommendations that were 10% more accurate than those offered by the company's existing recommender system. This competition energized the search for new and more accurate algorithms", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0075", "text": ". On 21 September 2009, the grand prize of US$1,000,000 was given to the BellKor's Pragmatic Chaos team using tiebreaking rules.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0076", "text": "The most accurate algorithm in 2007 used an ensemble method of 107 different algorithmic approaches, blended into a single prediction. As stated by the winners, Bell et al.:", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0077", "text": "Predictive accuracy is substantially improved when blending multiple predictors. Our experience is that most efforts should be concentrated in deriving substantially different approaches, rather than refining a single technique.  Consequently, our solution is an ensemble of many methods.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0078", "text": "Many benefits accrued to the web due to the Netflix project. Some teams have taken their technology and applied it to other markets. Some members from the team that finished second place founded Gravity R&D, a recommendation engine that's active in the RecSys community. 4-Tell, Inc. created a Netflix project\u2013derived solution for ecommerce websites.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0079", "text": "A number of privacy issues arose around the dataset offered by Netflix for the Netflix Prize competition. Although the data sets were anonymized in order to preserve customer privacy, in 2007 two researchers from the University of Texas were able to identify individual users by matching the data sets with film ratings on the Internet Movie Database (IMDb). As a result, in December 2009, an anonymous Netflix user sued Netflix in Doe v", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0080", "text": ". Netflix, alleging that Netflix had violated United States fair trade laws and the Video Privacy Protection Act by releasing the datasets. This, as well as concerns from the Federal Trade Commission, led to the cancellation of a second Netflix Prize competition in 2010.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0081", "text": "Evaluation\nPerformance measures\nEvaluation is important in assessing the effectiveness of recommendation algorithms. To measure the effectiveness of recommender systems, and compare different approaches, three types of evaluations are available: user studies, online evaluations (A/B tests), and offline evaluations.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0082", "text": "The commonly used metrics are the mean squared error and root mean squared error, the latter having been used in the Netflix Prize. The information retrieval metrics such as precision and recall or discounted cumulative gain (DCG) are useful to assess the quality of a recommendation method. Diversity, novelty, and coverage are also considered as important aspects in evaluation. However, many of the classic evaluation measures are highly criticized.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0083", "text": "Evaluating the performance of a recommendation algorithm on a fixed test dataset will always be extremely challenging as it is impossible to accurately predict the reactions of real users to the recommendations. Hence any metric that computes the effectiveness of an algorithm in offline data will be imprecise.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0084", "text": "User studies are rather a small scale. A few dozens or hundreds of users are presented recommendations created by different recommendation approaches, and then the users judge which recommendations are best.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0085", "text": "In A/B tests, recommendations are shown to typically thousands of users of a real product, and the recommender system randomly picks at least two different recommendation approaches to generate recommendations. The effectiveness is measured with implicit measures of effectiveness such as conversion rate or click-through rate.\nOffline evaluations are based on historic data, e.g. a dataset that contains information about how users previously rated movies.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0086", "text": "The effectiveness of recommendation approaches is then measured based on how well a recommendation approach can predict the users' ratings in the dataset. While a rating is an explicit expression of whether a user liked a movie, such information is not available in all domains. For instance, in the domain of citation recommender systems, users typically do not rate a citation or recommended article. In such cases, offline evaluations may use implicit measures of effectiveness", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0087", "text": ". For instance, it may be assumed that a recommender system is effective that is able to recommend as many articles as possible that are contained in a research article's reference list. However, this kind of offline evaluations is seen critical by many researchers. For instance, it has been shown that results of offline evaluations have low correlation with results from user studies or A/B tests", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0088", "text": ". A dataset popular for offline evaluation has been shown to contain duplicate data and thus to lead to wrong conclusions in the evaluation of algorithms. Often, results of so-called offline evaluations do not correlate with actually assessed user-satisfaction. This is probably because offline training is highly biased toward the highly reachable items, and offline testing data is highly influenced by the outputs of the online recommendation module", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0089", "text": ". Researchers have concluded that the results of offline evaluations should be viewed critically.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0090", "text": "Researchers have identified limits to the practical usefulness of accuracy-based evaluation metrics in recommender systems, sometimes described as an accuracy barrier or magic barrier.  In this view, continued improvements in offline accuracy do not necessarily translate into better user experience, satisfaction, or real-world system impact. As a result, accuracy gains beyond a certain point may provide diminishing returns when evaluated solely through offline experiments.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0091", "text": "Beyond accuracy\nTypically, research on recommender systems is concerned with finding the most accurate recommendation algorithms. However, there are a number of factors that are also important.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0092", "text": "Diversity \u2013 Users tend to be more satisfied with recommendations when there is a higher intra-list diversity, e.g. items from different artists.\nRecommender persistence \u2013 In some situations, it is more effective to re-show recommendations, or let users re-rate items, than showing new items. There are several reasons for this. Users may ignore items when they are shown for the first time, for instance, because they had no time to inspect the recommendations carefully.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0093", "text": "Privacy \u2013 Recommender systems usually have to deal with privacy concerns because users have to reveal sensitive information. Building user profiles using collaborative filtering can be problematic from a privacy point of view. Many European countries have a strong culture of data privacy, and every attempt to introduce any level of user profiling can result in a negative customer response. Much research has been conducted on ongoing privacy issues in this space", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0094", "text": ". The Netflix Prize is particularly notable for the detailed personal information released in its dataset. Ramakrishnan et al. have conducted an extensive overview of the trade-offs between personalization and privacy and found that the combination of weak ties (an unexpected connection that provides serendipitous recommendations) and other data sources can be used to uncover identities of users in an anonymized dataset.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0095", "text": "User demographics \u2013 Beel et al. found that user demographics may influence how satisfied users are with recommendations. In their paper they show that elderly users tend to be more interested in recommendations than younger users.\nRobustness \u2013 When users can participate in the recommender system, the issue of fraud must be addressed.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0096", "text": "Serendipity \u2013 Serendipity is a measure of \"how surprising the recommendations are\". For instance, a recommender system that recommends milk to a customer in a grocery store might be perfectly accurate, but it is not a good recommendation because it is an obvious item for the customer to buy. \"[Serendipity] serves two purposes: First, the chance that users lose interest because the choice set is too uniform decreases", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0097", "text": ". Second, these items are needed for algorithms to learn and improve themselves\".", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0098", "text": "Trust \u2013 A recommender system is of little value for a user if the user does not trust the system. Trust can be built by a recommender system by explaining how it generates recommendations, and why it recommends an item.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0099", "text": "Labelling \u2013 User satisfaction with recommendations may be influenced by the labeling of the recommendations. For instance, in the cited study click-through rate (CTR) for recommendations labeled as \"Sponsored\" were lower (CTR=5.93%) than CTR for identical recommendations labeled as \"Organic\" (CTR=8.86%). Recommendations with no label performed best (CTR=9.87%) in that study.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0100", "text": "Reproducibility", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0101", "text": "Recommender systems are notoriously difficult to evaluate offline, with some researchers claiming that this has led to a reproducibility crisis in recommender systems publications. The topic of reproducibility seems to be a recurrent issue in some Machine Learning publication venues, but does not have a considerable effect beyond the world of scientific publication", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0102", "text": ". In the context of recommender systems a 2019 paper surveyed a small number of hand-picked publications applying deep learning or neural methods to the top-k recommendation problem, published in top conferences (SIGIR, KDD, WWW, RecSys, IJCAI), has shown that on average less than 40% of articles could be reproduced by the authors of the survey, with as little as 14% in some conferences", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0103", "text": ". The articles considers a number of potential problems in today's research scholarship and suggests improved scientific practices in that area.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0104", "text": "More recent work on benchmarking a set of the same methods came to qualitatively very different results whereby neural methods were found to be among the best performing methods. Deep learning and neural methods for recommender systems have been used in the winning solutions in several recent recommender system challenges, WSDM, RecSys Challenge.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0105", "text": "Moreover, neural and deep learning methods are widely used in industry where they are extensively tested. The topic of reproducibility is not new in recommender systems. By 2011, Ekstrand, Konstan, et al. criticized that \"it is currently difficult to reproduce and extend recommender systems research results,\" and that evaluations are \"not handled consistently\"", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0106", "text": ". Konstan and Adomavicius conclude that \"the Recommender Systems research community is facing a crisis where a significant number of papers present results that contribute little to collective knowledge [...] often because the research lacks the [...] evaluation to be properly judged and, hence, to provide meaningful contributions.\" As a consequence, much research about recommender systems can be considered as not reproducible", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0107", "text": ". Hence, operators of recommender systems find little guidance in the current research for answering the question, which recommendation approaches to use in a recommender systems. Said and Bellog\u00edn conducted a study of papers published in the field, as well as benchmarked some of the most popular frameworks for recommendation and found large inconsistencies in results, even when the same algorithms and data sets were used", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0108", "text": ". Some researchers demonstrated that minor variations in the recommendation algorithms or scenarios led to strong changes in the effectiveness of a recommender system", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0109", "text": ". They conclude that seven actions are necessary to improve the current situation: \"(1) survey other research fields and learn from them, (2) find a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research.\"", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0110", "text": "Artificial intelligence applications in recommendation", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0111", "text": "Artificial intelligence (AI) applications in recommendation systems are the advanced methodologies that leverage AI technologies, to enhance the performance recommendation engines. The AI-based recommender can analyze complex data sets, learning from user behavior, preferences, and interactions to generate highly accurate and personalized content or product suggestions. The integration of AI in recommendation systems has marked a significant evolution from traditional recommendation methods", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0112", "text": ". Traditional methods often relied on inflexible algorithms that could suggest items based on general user trends or apparent similarities in content. In comparison, AI-powered systems have the capability to detect patterns and subtle distinctions that may be overlooked by traditional methods. These systems can adapt to specific individual preferences, thereby offering recommendations that are more aligned with individual user needs", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0113", "text": ". This approach marks a shift towards more personalized, user-centric suggestions.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0114", "text": "Recommendation systems widely adopt AI techniques such as machine learning, deep learning, and natural language processing. These advanced methods enhance system capabilities to predict user preferences and deliver personalized content more accurately. Each technique contributes uniquely. The following sections will introduce specific AI models utilized by a recommendation system by illustrating their theories and functionalities.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0115", "text": "KNN-based collaborative filters", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0116", "text": "Collaborative filtering (CF) is one of the most commonly used recommendation system algorithms. It generates personalized suggestions for users based on explicit or implicit behavioral patterns to form predictions. Specifically, it relies on external feedback such as star ratings, purchasing history and so on to make judgments. CF make predictions about users' preference based on similarity measurements", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0117", "text": ". Essentially, the underlying theory is: \"if user A is similar to user B, and if A likes item C, then it is likely that B also likes item C.\"", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0118", "text": "There are many models available for collaborative filtering. For AI-applied collaborative filtering, a common model is called K-nearest neighbors. The ideas are as follows:", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0119", "text": "Data Representation: Create a n-dimensional space where each axis represents a user's trait (ratings, purchases, etc.). Represent the user as a point in that space.\nStatistical Distance: 'Distance' measures how far apart users are in this space. See statistical distance for computational details\nIdentifying Neighbors: Based on the computed distances, find k nearest neighbors of the user to which we want to make recommendations", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0120", "text": "Forming Predictive Recommendations: The system will analyze the similar preference of the k neighbors. The system will make recommendations based on that similarity", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0121", "text": "Neural networks", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0122", "text": "An artificial neural network (ANN), is a deep learning model structure which aims to mimic a human brain. They comprise a series of neurons, each responsible for receiving and processing information transmitted from other interconnected neurons. Similar to a human brain, these neurons will change activation state based on incoming signals (training input and backpropagated output), allowing the system to adjust activation weights during the network learning phase", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0123", "text": ". ANN is usually designed to be a black-box model. Unlike regular machine learning where the underlying theoretical components are formal and rigid, the collaborative effects of neurons are not entirely clear, but modern experiments has shown the predictive power of ANN.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0124", "text": "ANN is widely used in recommendation systems for its power to utilize various data. Other than feedback data, ANN can incorporate non-feedback data which are too intricate for collaborative filtering to learn, and the unique structure allows ANN to identify extra signal from non-feedback data to boost user experience. Following are some examples:", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0125", "text": "Time and Seasonality: what specify time and date or a season that a user interacts with the platform\nUser Navigation Patterns: sequence of pages visited, time spent on different parts of a website, mouse movement, etc.\nExternal Social Trends: information from outer social media\n\nTwo-Tower Model\nThe Two-Tower model is a neural architecture commonly employed in large-scale recommendation systems, particularly for candidate retrieval tasks. It consists of two neural networks:", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0126", "text": "User Tower: Encodes user-specific features, such as interaction history or demographic data.\nItem Tower: Encodes item-specific features, such as metadata or content embeddings.\nThe outputs of the two towers are fixed-length embeddings that represent users and items in a shared vector space. A similarity metric, such as dot product or cosine similarity, is used to measure relevance between a user and an item.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0127", "text": "This model is highly efficient for large datasets as embeddings can be pre-computed for items, allowing rapid retrieval during inference. It is often used in conjunction with ranking models for end-to-end recommendation pipelines.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0128", "text": "Natural language processing", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0129", "text": "Natural language processing is a series of AI algorithms to make natural human language accessible and analyzable to a machine. It is a fairly modern technique inspired by the growing amount of textual information. For application in recommendation system, a common case is the Amazon customer review. Amazon will analyze the feedbacks comments from each customer and report relevant data to other customers for reference", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0130", "text": ". The recent years have witnessed the development of various text analysis models, including latent semantic analysis (LSA), singular value decomposition (SVD), latent Dirichlet allocation (LDA), etc. Their uses have consistently aimed to provide customers with more precise and tailored recommendations.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0131", "text": "Specific applications\nE-commerce", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0132", "text": "E-commerce\nRecommender systems are essential for modern e-commerce platforms, playing a key role in improving the customer experience and increasing sales. These systems analyze customer data to provide personalized product suggestions, helping users discover items they might not have found on their own. A study by J. Leskovec et al. highlighted that such systems are crucial when an individual needs to choose from a potentially overwhelming number of items that a service may offer.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0133", "text": "E-commerce recommenders typically use a combination of filtering techniques to generate these suggestions. Collaborative filtering is a core method, recommending products based on the purchasing and Browse habits of similar users. Another widely used approach is content-based filtering, which recommends items with similar attributes to those a user has previously shown interest in", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0134", "text": ". Many e-commerce platforms use a hybrid approach, combining these techniques to create more accurate and diverse recommendations, which helps to address issues like the \"cold start\" problem for new users or products.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0135", "text": "These systems are implemented in several ways across e-commerce sites to maximize their effectiveness at different stages of the shopping process:", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0136", "text": "On the homepage: Displaying personalized product lists, such as \u201cRecommended for you,\u201d based on a user's overall history.\nOn the product detail page: When a customer views a specific product, the system can show sections like \u201cS\u00favisiace produkty\u201d (Related products) or \u201cK tomuto produktu sa hod\u00ed\u201d (Goes well with this product). These recommendations help with upselling or cross-selling.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0137", "text": "In the shopping cart: When a customer has an item in their cart, the system can suggest complementary accessories or related items. For instance, if someone buys a camera, the system might recommend a memory card or a carrying case.\nThrough pop-up windows and notifications: Recommendations can also appear in timely pop-up windows, such as when a user attempts to leave the site or after a purchase is completed, with the goal of prompting them to \u201ctake another look\u201d or \u201cdiscover more.\u201d", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0138", "text": "In personalized email marketing: The system automatically sends emails with product recommendations based on a customer's past purchases or Browse history, increasing conversion rates even after the customer has left the site.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0139", "text": "The effective use of recommender systems can lead to a significant increase in key performance indicators for e-commerce, including higher conversion rates, larger average order values from cross-sells and upsells, and improved customer satisfaction and retention. These systems are powered by a range of technologies, from traditional machine learning models to advanced deep learning architectures that can process complex user behavior and product data.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0140", "text": "Academic content discovery", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0141", "text": "An emerging market for content discovery platforms is academic content. Approximately 6000 academic journal articles are published daily, making it increasingly difficult for researchers to balance time management with staying up to date with relevant research", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0142", "text": ". Though traditional tools academic search tools such as Google Scholar or PubMed provide a readily accessible database of journal articles, content recommendation in these cases are performed in a 'linear' fashion, with users setting 'alarms' for new publications based on keywords, journals or particular authors.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0143", "text": "Google Scholar provides an 'Updates' tool that suggests articles by using a statistical model that takes a researchers' authorized paper and citations as input. Whilst these recommendations have been noted to be extremely good, this poses a problem with early career researchers which may be lacking a sufficient body of work to produce accurate recommendations.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0144", "text": "Decision-making", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0145", "text": "In contrast to an engagement-based ranking system employed by social media and other digital platforms, a bridging-based ranking optimizes for content that is unifying instead of polarizing. Examples include Polis and Remesh which have been used around the world to help find more consensus around specific political issues. Twitter has also used this approach for managing its community notes, which YouTube planned to pilot in 2024", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0146", "text": ". Aviv Ovadya also argues for implementing bridging-based algorithms in major platforms by empowering deliberative groups that are representative of the platform's users to control the design and implementation of the algorithm.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0147", "text": "Television", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0148", "text": "As the connected television landscape continues to evolve, search and recommendation are seen as having an even more pivotal role in the discovery of content. With broadband-connected devices, consumers are projected to have access to content from linear broadcast sources as well as internet television", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0149", "text": ". Therefore, there is a risk that the market could become fragmented, leaving it to the viewer to visit various locations and find what they want to watch in a way that is time-consuming and complicated for them. By using a search and recommendation engine, viewers are provided with a central 'portal' from which to discover content from several sources in just one location.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0150", "text": "See also\nReferences\nFurther reading\nBooks\nKim Falk (d 2019), Practical Recommender Systems, Manning Publications, ISBN 9781617292705\nBharat Bhasker; K. Srikumar (2010). Recommender Systems in E-Commerce. CUP. ISBN 978-0-07-068067-8. Archived from the original on September 1, 2010.\nJannach, Dietmar; Markus Zanker; Alexander Felfernig; Gerhard Friedrich (2010). Recommender Systems: An Introduction. CUP. ISBN 978-0-521-49336-9. Archived from the original on August 31, 2015.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0151", "text": "Seaver, Nick (2022). Computing Taste: Algorithms and the Makers of Music Recommendation. University of Chicago Press.\nScientific articles\nRobert M. Bell; Jim Bennett; Yehuda Koren & Chris Volinsky (May 2009). \"The Million Dollar Programming Prize\". IEEE Spectrum. Archived from the original on May 11, 2009. Retrieved December 10, 2018.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Recommender_system_0152", "text": "Prem Melville, Raymond J. Mooney, and Ramadass Nagarajan. (2002) Content-Boosted Collaborative Filtering for Improved Recommendations.  Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI-2002), pp. 187\u2013192, Edmonton, Canada, July 2002.", "title": "Recommender system", "url": "https://en.wikipedia.org/wiki/Recommender_system"}, {"id": "rec_Collaborative_filtering_0000", "text": "Collaborative filtering (CF) is, besides content-based filtering, one of two major techniques used by recommender systems. Collaborative filtering has two senses, a narrow one and a more general one.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0001", "text": "In the newer, narrower sense, collaborative filtering is a method of making automatic predictions (filtering) about a user's interests by utilizing preferences or taste information collected from many users (collaborating). This approach assumes that if persons A and B share similar opinions on one issue, they are more likely to agree on other issues compared to a random pairing of A with another person", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0002", "text": ". For instance, a collaborative filtering system for television programming could predict which shows a user might enjoy based on a limited list of the user's tastes (likes or dislikes). These predictions are specific to the user, but use information gleaned from many users. This differs from the simpler approach of giving an average (non-specific) score for each item of interest, for example based on its number of votes.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0003", "text": "In the more general sense, collaborative filtering is the process of filtering information or patterns using techniques involving collaboration among multiple agents, viewpoints, data sources, etc. Applications of collaborative filtering typically involve very large data sets", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0004", "text": ". Collaborative filtering methods have been applied to many kinds of data including: sensing and monitoring data, such as in mineral exploration, environmental sensing over large areas or multiple sensors; financial data, such as financial service institutions that integrate many financial sources; and user data from electronic commerce and web applications.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0005", "text": "This article focuses on collaborative filtering for user data, but some of the methods also apply to other major applications.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0006", "text": "Overview\nThe growth of the Internet has made it much more difficult to effectively extract useful information from all the available online information. The overwhelming amount of data necessitates mechanisms for efficient information filtering. Collaborative filtering is one of the techniques used for dealing with this problem.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0007", "text": "The motivation for collaborative filtering comes from the idea that people often get the best recommendations from someone with tastes similar to themselves. Collaborative filtering encompasses techniques for matching people with similar interests and making recommendations on this basis.\nCollaborative filtering algorithms often require (1) users' active participation, (2) an easy way to represent users' interests, and (3) algorithms that are able to match people with similar interests.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0008", "text": "Typically, the workflow of a collaborative filtering system is:", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0009", "text": "A user expresses his or her preferences by rating items (e.g. books, movies, or music recordings) of the system. These ratings can be viewed as an approximate representation of the user's interest in the corresponding domain.\nThe system matches this user's ratings against other users' and finds the people with most \"similar\" tastes.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0010", "text": "With similar users, the system recommends items that the similar users have rated highly but not yet being rated by this user (presumably the absence of rating is often considered as the unfamiliarity of an item)\nA key problem of collaborative filtering is how to combine and weight the preferences of user neighbors. Sometimes, users can immediately rate the recommended items. As a result, the system gains an increasingly accurate representation of user preferences over time.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0011", "text": "Methodology\nCollaborative filtering systems have many forms, but many common systems can be reduced to two steps:", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0012", "text": "Look for users who share the same rating patterns with the active user (the user whom the prediction is for).\nUse the ratings from those like-minded users found in step 1 to calculate a prediction for the active user\nThis falls under the category of user-based collaborative filtering. A specific application of this is the user-based Nearest Neighbor algorithm.\nAlternatively, item-based collaborative filtering (users who bought x also bought y), proceeds in an item-centric manner:", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0013", "text": "Build an item-item matrix determining relationships between pairs of items\nInfer the tastes of the current user by examining the matrix and matching that user's data\nSee, for example, the Slope One item-based collaborative filtering family.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0014", "text": "Another form of collaborative filtering can be based on implicit observations of normal user behavior (as opposed to the artificial behavior imposed by a rating task). These systems observe what a user has done together with what all users have done (what music they have listened to, what items they have bought) and use that data to predict the user's behavior in the future, or to predict how a user might like to behave given the chance", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0015", "text": ". These predictions then have to be filtered through business logic to determine how they might affect the actions of a business system. For example, it is not useful to offer to sell somebody a particular album of music if they already have demonstrated that they own that music.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0016", "text": "Relying on a scoring or rating system which is averaged across all users ignores specific demands of a user, and is particularly poor in tasks where there is large variation in interest (as in the recommendation of music). However, there are other methods to combat information explosion, such as web search and data clustering.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0017", "text": "Types\nMemory-based\nThe memory-based approach uses user rating data to compute the similarity between users or items. Typical examples of this approach are neighbourhood-based CF and item-based/user-based top-N recommendations. For example, in user based approaches, the value of ratings user u gives to item i is calculated as an aggregation of some similar users' rating of the item:", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0018", "text": "r\n          \n            u\n            ,\n            i\n          \n        \n        =\n        \n          aggr\n          \n            \n              u\n              \n                \u2032\n              \n            \n            \u2208\n            U\n          \n        \n        \u2061\n        \n          r\n          \n            \n              u\n              \n                \u2032\n              \n            \n            ,\n            i", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0019", "text": "i\n          \n        \n      \n    \n    {\\displaystyle r_{u,i}=\\operatorname {aggr} _{u^{\\prime }\\in U}r_{u^{\\prime },i}}", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0020", "text": "where U denotes the set of top N users that are most similar to user u who rated item i. Some examples of the aggregation function include:", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0021", "text": "r\n          \n            u\n            ,\n            i\n          \n        \n        =\n        \n          \n            1\n            N\n          \n        \n        \n          \u2211\n          \n            \n              u\n              \n                \u2032\n              \n            \n            \u2208\n            U\n          \n        \n        \n          r\n          \n            \n              u\n              \n                \u2032\n              \n            \n            ,", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0022", "text": ",\n            i\n          \n        \n      \n    \n    {\\displaystyle r_{u,i}={\\frac {1}{N}}\\sum \\limits _{u^{\\prime }\\in U}r_{u^{\\prime },i}}", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0023", "text": "r\n          \n            u\n            ,\n            i\n          \n        \n        =\n        k\n        \n          \u2211\n          \n            \n              u\n              \n                \u2032\n              \n            \n            \u2208\n            U\n          \n        \n        simil\n        \u2061\n        (\n        u\n        ,\n        \n          u\n          \n            \u2032\n          \n        \n        )\n        \n          r\n          \n            \n              u", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0024", "text": "u\n              \n                \u2032\n              \n            \n            ,\n            i\n          \n        \n      \n    \n    {\\displaystyle r_{u,i}=k\\sum \\limits _{u^{\\prime }\\in U}\\operatorname {simil} (u,u^{\\prime })r_{u^{\\prime },i}}", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0025", "text": "where k is a normalizing factor defined as \n  \n    \n      \n        k\n        =\n        1\n        \n          /\n        \n        \n          \u2211\n          \n            \n              u\n              \n                \u2032\n              \n            \n            \u2208\n            U\n          \n        \n        \n          |\n        \n        simil\n        \u2061\n        (\n        u\n        ,\n        \n          u\n          \n            \u2032\n          \n        \n        )\n        \n          |", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0026", "text": "|\n        \n      \n    \n    {\\displaystyle k=1/\\sum _{u^{\\prime }\\in U}|\\operatorname {simil} (u,u^{\\prime })|}\n  \n, and", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0027", "text": "r\n          \n            u\n            ,\n            i\n          \n        \n        =\n        \n          \n            \n              \n                r\n                \n                  u\n                \n              \n              \u00af\n            \n          \n        \n        +\n        k\n        \n          \u2211\n          \n            \n              u\n              \n                \u2032\n              \n            \n            \u2208\n            U", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0028", "text": "\u2208\n            U\n          \n        \n        simil\n        \u2061\n        (\n        u\n        ,\n        \n          u\n          \n            \u2032\n          \n        \n        )\n        (\n        \n          r\n          \n            \n              u\n              \n                \u2032\n              \n            \n            ,\n            i\n          \n        \n        \u2212\n        \n          \n            \n              \n                r\n                \n                  \n                    u", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0029", "text": "u\n                    \n                      \u2032\n                    \n                  \n                \n              \n              \u00af\n            \n          \n        \n        )\n      \n    \n    {\\displaystyle r_{u,i}={\\bar {r_{u}}}+k\\sum \\limits _{u^{\\prime }\\in U}\\operatorname {simil} (u,u^{\\prime })(r_{u^{\\prime },i}-{\\bar {r_{u^{\\prime }}}})}", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0030", "text": "where \n  \n    \n      \n        \n          \n            \n              \n                r\n                \n                  u\n                \n              \n              \u00af\n            \n          \n        \n      \n    \n    {\\displaystyle {\\bar {r_{u}}}}\n  \n is the average rating of user u for all the items rated by u.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0031", "text": "The neighborhood-based algorithm calculates the similarity between two users or items, and produces a prediction for the user by taking the weighted average of all the ratings. Similarity computation between items or users is an important part of this approach. Multiple measures, such as Pearson correlation and vector cosine based similarity are used for this.\nThe Pearson correlation similarity of two users x, y is defined as", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0032", "text": "simil\n        \u2061\n        (\n        x\n        ,\n        y\n        )\n        =\n        \n          \n            \n              \n                \u2211\n                \n                  i\n                  \u2208\n                  \n                    I\n                    \n                      x\n                      y\n                    \n                  \n                \n              \n              (\n              \n                r\n                \n                  x", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0033", "text": "x\n                  ,\n                  i\n                \n              \n              \u2212\n              \n                \n                  \n                    \n                      r\n                      \n                        x\n                      \n                    \n                    \u00af\n                  \n                \n              \n              )\n              (\n              \n                r\n                \n                  y", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0034", "text": "y\n                  ,\n                  i\n                \n              \n              \u2212\n              \n                \n                  \n                    \n                      r\n                      \n                        y\n                      \n                    \n                    \u00af\n                  \n                \n              \n              )", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0035", "text": "\u2211\n                    \n                      i\n                      \u2208\n                      \n                        I\n                        \n                          x\n                          y\n                        \n                      \n                    \n                  \n                  (\n                  \n                    r\n                    \n                      x\n                      ,\n                      i", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0036", "text": ",\n                      i\n                    \n                  \n                  \u2212\n                  \n                    \n                      \n                        \n                          r\n                          \n                            x\n                          \n                        \n                        \u00af\n                      \n                    \n                  \n                  \n                    )", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0037", "text": ")\n                    \n                      2\n                    \n                  \n                \n              \n              \n                \n                  \n                    \u2211\n                    \n                      i\n                      \u2208\n                      \n                        I\n                        \n                          x\n                          y", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0038", "text": "(\n                  \n                    r\n                    \n                      y\n                      ,\n                      i\n                    \n                  \n                  \u2212\n                  \n                    \n                      \n                        \n                          r\n                          \n                            y", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0039", "text": "\u00af\n                      \n                    \n                  \n                  \n                    )\n                    \n                      2", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0040", "text": "{\\displaystyle \\operatorname {simil} (x,y)={\\frac {\\sum \\limits _{i\\in I_{xy}}(r_{x,i}-{\\bar {r_{x}}})(r_{y,i}-{\\bar {r_{y}}})}{{\\sqrt {\\sum \\limits _{i\\in I_{xy}}(r_{x,i}-{\\bar {r_{x}}})^{2}}}{\\sqrt {\\sum \\limits _{i\\in I_{xy}}(r_{y,i}-{\\bar {r_{y}}})^{2}}}}}}", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0041", "text": "where Ixy is the set of items rated by both user x and user y.\nThe cosine-based approach defines the cosine-similarity between two users x and y as:", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0042", "text": "simil\n        \u2061\n        (\n        x\n        ,\n        y\n        )\n        =\n        cos\n        \u2061\n        (\n        \n          \n            \n              x\n              \u2192\n            \n          \n        \n        ,\n        \n          \n            \n              y\n              \u2192\n            \n          \n        \n        )\n        =\n        \n          \n            \n              \n                \n                  \n                    x\n                    \u2192", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0043", "text": "x\n                    \u2192\n                  \n                \n              \n              \u22c5\n              \n                \n                  \n                    y\n                    \u2192\n                  \n                \n              \n            \n            \n              \n                |\n              \n              \n                |\n              \n              \n                \n                  \n                    x\n                    \u2192", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0044", "text": "\u2192\n                  \n                \n              \n              \n                |\n              \n              \n                |\n              \n              \u00d7\n              \n                |\n              \n              \n                |\n              \n              \n                \n                  \n                    y\n                    \u2192\n                  \n                \n              \n              \n                |", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0045", "text": "|\n              \n              \n                |\n              \n            \n          \n        \n        =\n        \n          \n            \n              \n                \u2211\n                \n                  i\n                  \u2208\n                  \n                    I\n                    \n                      x\n                      y\n                    \n                  \n                \n              \n              \n                r\n                \n                  x", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0046", "text": "x\n                  ,\n                  i\n                \n              \n              \n                r\n                \n                  y\n                  ,\n                  i\n                \n              \n            \n            \n              \n                \n                  \n                    \u2211\n                    \n                      i\n                      \u2208\n                      \n                        I", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0047", "text": "x\n                        \n                      \n                    \n                  \n                  \n                    r\n                    \n                      x\n                      ,\n                      i\n                    \n                    \n                      2\n                    \n                  \n                \n              \n              \n                \n                  \n                    \u2211", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0048", "text": "\u2211\n                    \n                      i\n                      \u2208\n                      \n                        I\n                        \n                          y\n                        \n                      \n                    \n                  \n                  \n                    r\n                    \n                      y\n                      ,\n                      i\n                    \n                    \n                      2", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0049", "text": "2\n                    \n                  \n                \n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\operatorname {simil} (x,y)=\\cos({\\vec {x}},{\\vec {y}})={\\frac {{\\vec {x}}\\cdot {\\vec {y}}}{||{\\vec {x}}||\\times ||{\\vec {y}}||}}={\\frac {\\sum \\limits _{i\\in I_{xy}}r_{x,i}r_{y,i}}{{\\sqrt {\\sum \\limits _{i\\in I_{x}}r_{x,i}^{2}}}{\\sqrt {\\sum \\limits _{i\\in I_{y}}r_{y,i}^{2}}}}}}", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0050", "text": "The user based top-N recommendation algorithm uses a similarity-based vector model to identify the k most similar users to an active user. After the k most similar users are found, their corresponding user-item matrices are aggregated to identify the set of items to be recommended. A popular method to find the similar users is the Locality-sensitive hashing, which implements the nearest neighbor mechanism in linear time.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0051", "text": "The advantages with this approach include: the explainability of the results, which is an important aspect of recommendation systems; easy creation and use; easy facilitation of new data; content-independence of the items being recommended; good scaling with co-rated items.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0052", "text": "There are also several disadvantages of this approach. Its performance decreases when data is sparse, which is common for web-related items. This hinders the scalability of this approach and creates problems with large datasets. Although it can efficiently handle new users because it relies on a data structure, adding new items becomes more complicated because that representation usually relies on a specific vector space", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0053", "text": ". Adding new items requires inclusion of the new item and the re-insertion of all the elements in the structure.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0054", "text": "Model-based\nAn alternative to memory-based methods is to learn models to predict users' rating of unrated items. Model-based CF algorithms include Bayesian networks, clustering models, latent semantic models such as singular value decomposition, probabilistic latent semantic analysis, multiple multiplicative factor, latent Dirichlet allocation and Markov decision process-based models.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0055", "text": "Through this approach, dimensionality reduction methods are mostly used for improving robustness and accuracy of memory-based methods. Specifically, methods like singular value decomposition, principal component analysis, known as latent factor models, compress a user-item matrix into a low-dimensional representation in terms of latent factors. This transforms the large matrix that contains many missing values, into a much smaller matrix", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0056", "text": ". A compressed matrix can be used to find neighbors of a user or item as per the previous section. Compression has two advantages in large, sparse data: it is more accurate and scales better.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0057", "text": "Hybrid\nA number of applications combine the memory-based and the model-based CF algorithms. These overcome the limitations of native CF approaches and improve prediction performance. Importantly, they overcome the CF problems such as sparsity and loss of information. However, they have increased complexity and are expensive to implement. Usually most commercial recommender systems are hybrid, for example, the Google news recommender system.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0058", "text": "Deep-learning\nIn recent years, many neural and deep-learning techniques have been proposed for collaborative filtering. Some generalize traditional matrix factorization algorithms via a non-linear neural architecture, or leverage new model types like Variational Autoencoders.  Deep learning has been applied to many scenarios (context-aware, sequence-aware, social tagging etc.).", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0059", "text": "However, deep learning effectiveness for collaborative recommendation has been questioned. A systematic analysis of publications using deep learning or neural methods to the top-k recommendation problem, published in top conferences (SIGIR, KDD, WWW, RecSys), found that, on average, less than 40% of articles are reproducible, and only 14% in some conferences", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0060", "text": ". Overall, the study identifies 18 articles, only 7 of them could be reproduced and 6 could be outperformed by older and simpler properly tuned baselines. The article highlights potential problems in today's research scholarship and calls for improved scientific practices. Similar issues have been spotted by others and also in sequence-aware recommender systems.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0061", "text": "Context-aware collaborative filtering", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0062", "text": "Many recommender systems simply ignore other contextual information existing alongside user's rating in providing item recommendation. However, by pervasive availability of contextual information such as time, location, social information, and type of the device that user is using, it is becoming more important than ever for a successful recommender system to provide a context-sensitive recommendation", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0063", "text": ". According to Charu Aggrawal, \"Context-sensitive recommender systems tailor their recommendations to additional information that defines the specific situation under which recommendations are made. This additional information is referred to as the context.\"", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0064", "text": "Taking contextual information into consideration, we will have additional dimension to the existing user-item rating matrix. As an instance, assume a music recommender system which provides different recommendations in corresponding to time of the day. In this case, it is possible a user have different preferences for a music in different time of a day", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0065", "text": ". Thus, instead of using user-item matrix, we may use tensor of order 3 (or higher for considering other contexts) to represent context-sensitive users' preferences.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0066", "text": "In order to take advantage of collaborative filtering and particularly neighborhood-based methods, approaches can be extended from a two-dimensional rating matrix into a tensor of higher order. For this purpose, the approach is to find the most similar/like-minded users to a target user; one can extract and compute similarity of slices (e.g. item-time matrix) corresponding to each user", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0067", "text": ". item-time matrix) corresponding to each user. Unlike the context-insensitive case for which similarity of two rating vectors are calculated, in the context-aware approaches, the similarity of rating matrices corresponding to each user is calculated by using Pearson coefficients. After the most like-minded users are found, their corresponding ratings are aggregated to identify the set of items to be recommended to the target user.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0068", "text": "The most important disadvantage of taking context into recommendation model is to be able to deal with larger dataset that contains much more missing values in comparison to user-item rating matrix. Therefore, similar to matrix factorization methods, tensor factorization techniques can be used to reduce dimensionality of original data before using any neighborhood-based methods.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0069", "text": "Application on social web\nUnlike the traditional model of mainstream media, in which there are few editors who set guidelines, collaboratively filtered social media can have a very large number of editors, and content improves as the number of participants increases. Services like Reddit, YouTube, and Last.fm are typical examples of collaborative filtering based media.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0070", "text": "One scenario of collaborative filtering application is to recommend interesting or popular information as judged by the community. As a typical example, stories appear in the front page of Reddit as they are \"voted up\" (rated positively) by the community. As the community becomes larger and more diverse, the promoted stories can better reflect the average interest of the community members.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0071", "text": "Wikipedia is another application of collaborative filtering. Volunteers contribute to the encyclopedia by filtering out facts from falsehoods.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0072", "text": "Another aspect of collaborative filtering systems is the ability to generate more personalized recommendations by analyzing information from the past activity of a specific user, or the history of other users deemed to be of similar taste to a given user. These resources are used as user profiling and helps the site recommend content on a user-by-user basis", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0073", "text": ". The more a given user makes use of the system, the better the recommendations become, as the system gains data to improve its model of that user.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0074", "text": "Problems", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0075", "text": "A collaborative filtering system does not necessarily succeed in automatically matching content to one's preferences. Unless the platform achieves unusually good diversity and independence of opinions, one point of view will always dominate another in a particular community", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0076", "text": ". As in the personalized recommendation scenario, the introduction of new users or new items can cause the cold start problem, as there will be insufficient data on these new entries for the collaborative filtering to work accurately. In order to make appropriate recommendations for a new user, the system must first learn the user's preferences by analysing past voting or rating activities", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0077", "text": ". The collaborative filtering system requires a substantial number of users to rate a new item before that item can be recommended.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0078", "text": "Challenges\nData sparsity\nIn practice, many commercial recommender systems are based on large datasets. As a result, the user-item matrix used for collaborative filtering could be extremely large and sparse, which brings about challenges in the performance of the recommendation.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0079", "text": "One typical problem caused by the data sparsity is the cold start problem. As collaborative filtering methods recommend items based on users' past preferences, new users will need to rate a sufficient number of items to enable the system to capture their preferences accurately and thus provides reliable recommendations.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0080", "text": "Similarly, new items also have the same problem. When new items are added to the system, they need to be rated by a substantial number of users before they could be recommended to users who have similar tastes to the ones who rated them. The new item problem does not affect content-based recommendations, because the recommendation of an item is based on its discrete set of descriptive qualities rather than its ratings.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0081", "text": "Scalability\nAs the numbers of users and items grow, traditional CF algorithms will suffer serious scalability problems. For example, with tens of millions of customers \n  \n    \n      \n        O\n        (\n        M\n        )\n      \n    \n    {\\displaystyle O(M)}\n  \n and millions of items \n  \n    \n      \n        O\n        (\n        N\n        )\n      \n    \n    {\\displaystyle O(N)}\n  \n, a CF algorithm with the complexity of \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0082", "text": "n\n      \n    \n    {\\displaystyle n}\n  \n is already too large. As well, many systems need to react immediately to online requirements and make recommendations for all users regardless of their millions of users, with most computations happening in very large memory machines.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0083", "text": "Synonyms\nSynonyms refers to the tendency of a number of the same or very similar items to have different names or entries. Most recommender systems are unable to discover this latent association and thus treat these products differently.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0084", "text": "For example, the seemingly different items \"children's movie\" and \"children's film\" are actually referring to the same item. Indeed, the degree of variability in descriptive term usage is greater than commonly suspected. The prevalence of synonyms decreases the recommendation performance of CF systems. Topic Modeling (like the Latent Dirichlet Allocation technique) could solve this by grouping different words belonging to the same topic.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0085", "text": "Gray sheep\nGray sheep refers to the users whose opinions do not consistently agree or disagree with any group of people and thus do not benefit from collaborative filtering. Black sheep are a group whose idiosyncratic tastes make recommendations nearly impossible. Although this is a failure of the recommender system, non-electronic recommenders also have great problems in these cases, so having black sheep is an acceptable failure.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0086", "text": "Shilling attacks\nIn a recommendation system where everyone can give the ratings, people may give many positive ratings for their own items and negative ratings for their competitors'. It is often necessary for the collaborative filtering systems to introduce precautions to discourage such manipulations.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0087", "text": "Diversity and the long tail", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0088", "text": "Collaborative filters are expected to increase diversity because they help us discover new products. Some algorithms, however, may unintentionally do the opposite. Because collaborative filters recommend products based on past sales or ratings, they cannot usually recommend products with limited historical data. This can create a rich-get-richer effect for popular products, akin to positive feedback. This bias toward popularity can prevent what are otherwise better consumer-product matches", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0089", "text": ". A Wharton study details this phenomenon along with several ideas that may promote diversity and the \"long tail.\" Several collaborative filtering algorithms have been developed to promote diversity and the \"long tail\" by recommending novel, unexpected, and serendipitous items.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0090", "text": "Innovations\nNew algorithms have been developed for CF as a result of the Netflix prize.\nCross-System Collaborative Filtering where user profiles across multiple recommender systems are combined in a multitask manner; this way, preference pattern sharing is achieved across models.\nRobust collaborative filtering, where recommendation is stable towards efforts of manipulation. This research area is still active and not completely solved.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0091", "text": "Auxiliary information", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0092", "text": "User-item matrix is a basic foundation of traditional collaborative filtering techniques, and it suffers from data sparsity problem (i.e. cold start). As a consequence, except for user-item matrix, researchers are trying to gather more auxiliary information to help boost recommendation performance and develop personalized recommender systems. Generally, there are two popular auxiliary information: attribute information and interaction information", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0093", "text": ". Attribute information describes a user's or an item's properties. For example, user attribute might include general profile (e.g. gender and age) and social contacts (e.g. followers or friends in social networks); Item attribute means properties like category, brand or content. In addition, interaction information refers to the implicit data showing how users interplay with the item. Widely used interaction information contains tags, comments or reviews and browsing history etc", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0094", "text": ". Auxiliary information plays a significant role in a variety of aspects. Explicit social links, as a reliable representative of trust or friendship, is always employed in similarity calculation to find similar persons who share interest with the target user. The interaction-associated information \u2013 tags \u2013 is taken as a third dimension (in addition to user and item) in advanced collaborative filtering to construct a 3-dimensional tensor structure for exploration of recommendation.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0095", "text": "See also\nReferences\nExternal links\nBeyond Recommender Systems: Helping People Help Each Other, page 12, 2001\nRecommender Systems. Prem Melville and Vikas Sindhwani. In Encyclopedia of Machine Learning, Claude Sammut and Geoffrey Webb (Eds), Springer, 2010.\nRecommender Systems in industrial contexts \u2013 PHD thesis (2012) including a comprehensive overview of many collaborative recommender systems", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0096", "text": "Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. Adomavicius, G. and Tuzhilin, A. IEEE Transactions on Knowledge and Data Engineering 06.2005\nEvaluating collaborative filtering recommender systems (DOI: 10.1145/963770.963772)\nGroupLens research papers.", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0097", "text": "GroupLens research papers.\nContent-Boosted Collaborative Filtering for Improved Recommendations. Prem Melville, Raymond J. Mooney, and Ramadass Nagarajan. Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI-2002), pp. 187\u2013192, Edmonton, Canada, July 2002.\nA collection of past and present \"information filtering\" projects (including collaborative filtering) at MIT Media Lab", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0098", "text": "Eigentaste: A Constant Time Collaborative Filtering Algorithm. Ken Goldberg, Theresa Roeder, Dhruv Gupta, and Chris Perkins. Information Retrieval, 4(2), 133\u2013151. July 2001.\nA Survey of Collaborative Filtering Techniques Su, Xiaoyuan and Khoshgortaar, Taghi. M", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0099", "text": "Google News Personalization: Scalable Online Collaborative Filtering Abhinandan Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. International World Wide Web Conference, Proceedings of the 16th international conference on World Wide Web\nFactor in the Neighbors: Scalable and Accurate Collaborative Filtering Yehuda Koren, Transactions on Knowledge Discovery from Data (TKDD) (2009)\nRating Prediction Using Collaborative Filtering", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Collaborative_filtering_0100", "text": "Rating Prediction Using Collaborative Filtering\nRecommender Systems Archived 11 February 2013 at the Wayback Machine\nBerkeley Collaborative Filtering", "title": "Collaborative filtering", "url": "https://en.wikipedia.org/wiki/Collaborative_filtering"}, {"id": "rec_Retrievalaugmented_generation_0000", "text": "Retrieval-augmented generation (RAG) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources. With RAG, LLMs first refer to a specified set of documents, then respond to user queries. These documents supplement information from the LLM's pre-existing training data. This allows LLMs to use domain-specific and/or updated information that is not available in the training data", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0001", "text": ". For example, this helps LLM-based chatbots access internal company data or generate responses based on authoritative sources.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0002", "text": "RAG improves large language models (LLMs) by incorporating information retrieval before generating responses. Unlike LLMs that rely on static training data, RAG pulls relevant text from databases, uploaded documents, or web sources", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0003", "text": ". According to Ars Technica, \"RAG is a way of improving LLM performance, in essence by blending the LLM process with a web search or other document look-up process to help LLMs stick to the facts.\" This method helps reduce AI hallucinations, which have caused chatbots to describe policies that don't exist, or recommend nonexistent legal cases to lawyers that are looking for citations to support their arguments.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0004", "text": "RAG also reduces the need to retrain LLMs with new data, saving on computational and financial costs. Beyond efficiency gains, RAG also allows LLMs to include sources in their responses, so users can verify the cited sources. This provides greater transparency, as users can cross-check retrieved content to ensure accuracy and relevance.\nThe term RAG was first introduced in a 2020 research paper.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0005", "text": "RAG and LLM limitations", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0006", "text": "LLMs can provide incorrect information. For example, when Google first demonstrated its LLM tool \"Google Bard\" (later re-branded to Gemini), the LLM provided incorrect information about the James Webb Space Telescope. This error contributed to a $100 billion decline in Google's\u2019s stock value. RAG is used to prevent these errors, but it does not solve all the problems. For example, LLMs can generate misinformation even when pulling from factually correct sources if they misinterpret the context", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0007", "text": ". MIT Technology Review gives the example of an AI-generated response stating, \"The United States has had one Muslim president, Barack Hussein Obama.\" The model retrieved this from an academic book rhetorically titled Barack Hussein Obama: America\u2019s First Muslim President? The LLM did not \"know\" or \"understand\" the context of the title, generating a false statement.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0008", "text": "LLMs with RAG are programmed to prioritize new information. This technique has been called \"prompt stuffing.\" Without prompt stuffing, the LLM's input is generated by a user; with prompt stuffing, additional relevant context is added to this input to guide the model\u2019s response. This approach provides the LLM with key information early in the prompt, encouraging it to prioritize the supplied data over pre-existing training knowledge.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0009", "text": "Process", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0010", "text": "Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating an information-retrieval mechanism that allows models to access and utilize additional data beyond their original training set. Ars Technica notes that \"when new information becomes available, rather than having to retrain the model, all that\u2019s needed is to augment the model\u2019s external knowledge base with the updated information\" (\"augmentation\")", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0011", "text": ". IBM states that \"in the generative phase, the LLM draws from the augmented prompt and its internal representation of its training data to synthesize\" an answer.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0012", "text": "RAG key stages\nTypically, the data to be referenced is converted into LLM embeddings, numerical representations in the form of a large vector space. RAG can be used on unstructured (usually text), semi-structured, or structured data (for example knowledge graphs). These embeddings are then stored in a vector database to allow for document retrieval.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0013", "text": "Given a user query, a document retriever is first called to select the most relevant documents that will be used to augment the query. This comparison can be done using a variety of methods, which depend in part on the type of indexing used.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0014", "text": "The model feeds this relevant retrieved information into the LLM via prompt engineering of the user's original query. Newer implementations (as of 2023) can also incorporate specific augmentation modules with abilities such as expanding queries into multiple domains and using memory and self-improvement to learn from previous retrievals.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0015", "text": "Finally, the LLM can generate output based on both the query and the retrieved documents. Some models incorporate extra steps to improve output, such as the re-ranking of retrieved information, context selection, and fine-tuning.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0016", "text": "Improvements\nImprovements to the basic process above can be applied at different stages in the RAG flow.\n\nEncoder\nThese methods focus on the encoding of text as either dense or sparse vectors. Sparse vectors, which encode the identity of a word, are typically dictionary-length and contain mostly zeros. Dense vectors, which encode meaning, are more compact and contain fewer zeros. Various enhancements can improve the way similarities are calculated in the vector stores (databases).", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0017", "text": "Performance improves by optimizing how vector similarities are calculated. Dot products enhance similarity scoring, while approximate nearest neighbor (ANN) searches improve retrieval efficiency over K-nearest neighbors (KNN) searches.\nAccuracy may be improved with Late Interactions, which allow the system to compare words more precisely after retrieval. This helps refine document ranking and improve search relevance.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0018", "text": "Hybrid vector approaches may be used to combine dense vector representations with sparse one-hot vectors, taking advantage of the computational efficiency of sparse dot products over dense vector operations.\nOther retrieval techniques focus on improving accuracy by refining how documents are selected. Some retrieval methods combine sparse representations, such as SPLADE, with query expansion strategies to improve search accuracy and recall.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0019", "text": "Retriever-centric methods\nThese methods aim to enhance the quality of document retrieval in vector databases:", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0020", "text": "Pre-training the retriever using the Inverse Cloze Task (ICT), a technique that helps the model learn retrieval patterns by predicting masked text within documents.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0021", "text": "Supervised retriever optimization aligns retrieval probabilities with the generator model\u2019s likelihood distribution. This involves retrieving the top-k vectors for a given prompt, scoring the generated response\u2019s perplexity, and minimizing KL divergence between the retriever\u2019s selections and the model\u2019s likelihoods to refine retrieval.\nReranking techniques can refine retriever performance by prioritizing the most relevant retrieved documents during training.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0022", "text": "Language model", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0023", "text": "By redesigning the language model with the retriever in mind, a 25-time smaller network can get comparable perplexity as its much larger counterparts. Because it is trained from scratch, this method (Retro) incurs the high cost of training runs that the original RAG scheme avoided. The hypothesis is that by giving domain knowledge during training, Retro needs less focus on the domain and can devote its smaller weight resources only to language semantics", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0024", "text": ". The redesigned language model is shown here.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0025", "text": "It has been reported that Retro is not reproducible, so modifications were made to make it so.  The more reproducible version is called Retro++ and includes in-context RAG.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0026", "text": "Chunking\nChunking involves various strategies for breaking up the data into vectors so the retriever can find details in it.\n\nThree types of chunking strategies are:", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0027", "text": "Fixed length with overlap. This is fast and easy. Overlapping consecutive chunks helps to maintain semantic context across chunks.\nSyntax-based chunks can break the document up into sentences. Libraries such as spaCy or NLTK can also help.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0028", "text": "File format-based chunking. Certain file types have natural chunks built in, and it's best to respect them. For example, code files are best chunked and vectorized as whole functions or classes. HTML files should leave <table> or base64 encoded <img> elements intact. Similar considerations should be taken for pdf files. Libraries such as Unstructured or Langchain can assist with this method.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0029", "text": "Hybrid search\nSometimes vector database searches can miss key facts needed to answer a user's question. One way to mitigate this is to do a traditional text search, add those results to the text chunks linked to the retrieved vectors from the vector search, and feed the combined hybrid text into the language model for generation.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0030", "text": "The adoption of RAG in consumer-facing web search products has given rise to new content optimization disciplines, as practitioners have noted that content retrievability in RAG systems depends on factors like semantic structure, passage-level authority signals, and entity clarity rather than traditional search ranking signals such as backlinks.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0031", "text": "Evaluation and benchmarks\nRAG systems are commonly evaluated using benchmarks designed to test retrievability, retrieval accuracy and generative quality. Popular datasets include BEIR, a suite of information retrieval tasks across diverse domains, and Natural Questions or Google QA for open-domain QA.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0032", "text": "Challenges\nRAG does not prevent hallucinations in LLMs. According to Ars Technica, \"It is not a direct solution because the LLM can still hallucinate around the source material in its response.\"", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0033", "text": "While RAG improves the accuracy of large language models (LLMs), it does not eliminate all challenges. One limitation is that while RAG reduces the need for frequent model retraining, it does not remove it entirely. Additionally, LLMs may struggle to recognize when they lack sufficient information to provide a reliable response. Without specific training, models may generate answers even when they should indicate uncertainty", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0034", "text": ". According to IBM, this issue can arise when the model lacks the ability to assess its own knowledge limitations.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0035", "text": "RAG poisoning", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0036", "text": "RAG systems may retrieve factually correct but misleading sources, leading to errors in interpretation. In some cases, an LLM may extract statements from a source without considering its context, resulting in an incorrect conclusion. Additionally, when faced with conflicting information, RAG models may struggle to determine which source is accurate", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0037", "text": ". The worst case outcome of this limitation is that the model may combine details from multiple sources producing responses that merge outdated and updated information in a misleading manner. According to the MIT Technology Review, these issues occur because RAG systems may misinterpret the data they retrieve.", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Retrievalaugmented_generation_0038", "text": "== References ==", "title": "Retrieval-augmented generation", "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}, {"id": "rec_Transfer_learning_0000", "text": "Transfer learning (TL) is a technique in machine learning (ML) in which knowledge learned from a task is re-used in order to boost performance on a related task. For example, for image classification, knowledge gained while learning to recognize cars could be applied when trying to recognize trucks. This topic is related to the psychological literature on transfer of learning, although practical ties between the two fields are limited", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0001", "text": ". Reusing or transferring information from previously learned tasks to new tasks has the potential to significantly improve learning efficiency.", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0002", "text": "Since transfer learning makes use of training with multiple objective functions it is related to cost-sensitive machine learning and multi-objective optimization.", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0003", "text": "History\nIn 1976, Bozinovski and Fulgosi published a paper addressing transfer learning in neural network training. The paper gives a mathematical and geometrical model of the topic. In 1981, a report considered the application of transfer learning to a dataset of images representing letters of computer terminals, experimentally demonstrating positive and negative transfer learning.\nIn 1992, Lorien Pratt formulated the discriminability-based transfer (DBT) algorithm.", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0004", "text": "By 1998, the field had advanced to include multi-task learning, along with more formal theoretical foundations. Influential publications on transfer learning include the book Learning to Learn in 1998, a 2009 survey and a 2019 survey.\nNg said in his NIPS 2016 tutorial that TL would become the next driver of machine learning commercial success after supervised learning.", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0005", "text": "In the 2020 paper, \"Rethinking Pre-Training and self-training\", Zoph et al. reported that pre-training can hurt accuracy, and advocate self-training instead.", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0006", "text": "Definition\nThe definition of transfer learning is given in terms of domains and tasks. A domain \n  \n    \n      \n        \n          \n            D\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}}\n  \n consists of: a feature space \n  \n    \n      \n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {X}}}\n  \n and a marginal probability distribution \n  \n    \n      \n        P\n        (\n        X\n        )", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0007", "text": "(\n        X\n        )\n      \n    \n    {\\displaystyle P(X)}\n  \n, where \n  \n    \n      \n        X\n        =\n        {\n        \n          x\n          \n            1\n          \n        \n        ,\n        .\n        .\n        .\n        ,\n        \n          x\n          \n            n\n          \n        \n        }\n        \u2208\n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle X=\\{x_{1},...,x_{n}\\}\\in {\\mathcal {X}}}\n  \n. Given a specific domain,", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0008", "text": ". Given a specific domain, \n  \n    \n      \n        \n          \n            D\n          \n        \n        =\n        {\n        \n          \n            X\n          \n        \n        ,\n        P\n        (\n        X\n        )\n        }\n      \n    \n    {\\displaystyle {\\mathcal {D}}=\\{{\\mathcal {X}},P(X)\\}}\n  \n, a task consists of two components: a label space \n  \n    \n      \n        \n          \n            Y\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {Y}}}", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0009", "text": "{\\displaystyle {\\mathcal {Y}}}\n  \n and an objective predictive function \n  \n    \n      \n        f\n        :\n        \n          \n            X\n          \n        \n        \u2192\n        \n          \n            Y\n          \n        \n      \n    \n    {\\displaystyle f:{\\mathcal {X}}\\rightarrow {\\mathcal {Y}}}\n  \n. The function \n  \n    \n      \n        f\n      \n    \n    {\\displaystyle f}\n  \n is used to predict the corresponding label \n  \n    \n      \n        f\n        (\n        x\n        )", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0010", "text": "f\n        (\n        x\n        )\n      \n    \n    {\\displaystyle f(x)}\n  \n of a new instance \n  \n    \n      \n        x\n      \n    \n    {\\displaystyle x}\n  \n. This task, denoted by \n  \n    \n      \n        \n          \n            T\n          \n        \n        =\n        {\n        \n          \n            Y\n          \n        \n        ,\n        f\n        (\n        x\n        )\n        }\n      \n    \n    {\\displaystyle {\\mathcal {T}}=\\{{\\mathcal {Y}},f(x)\\}}", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0011", "text": ", is learned from the training data consisting of pairs \n  \n    \n      \n        {\n        \n          x\n          \n            i\n          \n        \n        ,\n        \n          y\n          \n            i\n          \n        \n        }\n      \n    \n    {\\displaystyle \\{x_{i},y_{i}\\}}\n  \n, where \n  \n    \n      \n        \n          x\n          \n            i\n          \n        \n        \u2208\n        \n          \n            X\n          \n        \n      \n    \n    {\\displaystyle x_{i}\\in {\\mathcal {X}}}", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0012", "text": "{\\displaystyle x_{i}\\in {\\mathcal {X}}}\n  \n and \n  \n    \n      \n        \n          y\n          \n            i\n          \n        \n        \u2208\n        \n          \n            Y\n          \n        \n      \n    \n    {\\displaystyle y_{i}\\in {\\mathcal {Y}}}\n  \n.\nGiven a source domain \n  \n    \n      \n        \n          \n            \n              D\n            \n          \n          \n            S\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}_{S}}\n  \n and learning task", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0013", "text": "and learning task \n  \n    \n      \n        \n          \n            \n              T\n            \n          \n          \n            S\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {T}}_{S}}\n  \n, a target domain \n  \n    \n      \n        \n          \n            \n              D\n            \n          \n          \n            T\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}_{T}}\n  \n and learning task \n  \n    \n      \n        \n          \n            \n              T", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0014", "text": "T\n            \n          \n          \n            T\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {T}}_{T}}\n  \n, where \n  \n    \n      \n        \n          \n            \n              D\n            \n          \n          \n            S\n          \n        \n        \u2260\n        \n          \n            \n              D\n            \n          \n          \n            T", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0015", "text": "T\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}_{S}\\neq {\\mathcal {D}}_{T}}\n  \n, or \n  \n    \n      \n        \n          \n            \n              T\n            \n          \n          \n            S\n          \n        \n        \u2260\n        \n          \n            \n              T\n            \n          \n          \n            T\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {T}}_{S}\\neq {\\mathcal {T}}_{T}}", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0016", "text": ", transfer learning aims to help improve the learning of the target predictive function \n  \n    \n      \n        \n          f\n          \n            T\n          \n        \n        (\n        \u22c5\n        )\n      \n    \n    {\\displaystyle f_{T}(\\cdot )}\n  \n in \n  \n    \n      \n        \n          \n            \n              D\n            \n          \n          \n            T\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}_{T}}\n  \n using the knowledge in", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0017", "text": "using the knowledge in \n  \n    \n      \n        \n          \n            \n              D\n            \n          \n          \n            S\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {D}}_{S}}\n  \n and \n  \n    \n      \n        \n          \n            \n              T\n            \n          \n          \n            S\n          \n        \n      \n    \n    {\\displaystyle {\\mathcal {T}}_{S}}\n  \n.", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0018", "text": "Applications\nAlgorithms for transfer learning are available in Markov logic networks and Bayesian networks. Transfer learning has been applied to cancer subtype discovery, building utilization, general game playing, text classification, digit recognition, medical imaging and spam filtering.", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0019", "text": "In 2020, it was discovered that, due to their similar physical natures, transfer learning is possible between electromyographic (EMG) signals from the muscles and classifying the behaviors of electroencephalographic (EEG) brainwaves, from the gesture recognition domain to the mental state recognition domain. It was noted that this relationship worked in both directions, showing that electroencephalographic can likewise be used to classify EMG", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0020", "text": ". The experiments noted that the accuracy of neural networks and convolutional neural networks were improved through transfer learning both prior to any learning (compared to standard random weight distribution) and at the end of the learning process (asymptote). That is, results are improved by exposure to another domain. Moreover, the end-user of a pre-trained model can change the structure of fully-connected layers to improve performance.", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Transfer_learning_0021", "text": "See also\nCrossover (genetic algorithm)\nDomain adaptation\nGeneral game playing\nMulti-task learning\nMultitask optimization\nTransfer of learning in educational psychology\nZero-shot learning\nFeature learning\nexternal validity\n\nReferences\nSources\nThrun, Sebastian; Pratt, Lorien (6 December 2012). Learning to Learn. Springer Science & Business Media. ISBN 978-1-4615-5529-2.", "title": "Transfer learning", "url": "https://en.wikipedia.org/wiki/Transfer_learning"}, {"id": "rec_Data_augmentation_0000", "text": "Data augmentation is a statistical technique which allows maximum likelihood estimation from incomplete data. Data augmentation has important applications in Bayesian analysis, and the technique is widely used in machine learning to reduce overfitting when training machine learning models, achieved by training models on several slightly-modified copies of existing data.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0001", "text": "Synthetic oversampling techniques for traditional machine learning", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0002", "text": "Synthetic Minority Over-sampling Technique (SMOTE) is a method used to address imbalanced datasets in machine learning. In such datasets, the number of samples in different classes varies significantly, leading to biased model performance. For example, in a medical diagnosis dataset with 90 samples representing healthy individuals and only 10 samples representing individuals with a particular disease, traditional algorithms may struggle to accurately classify the minority class", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0003", "text": ". SMOTE rebalances the dataset by generating synthetic samples for the minority class. For instance, if there are 100 samples in the majority class and 10 in the minority class, SMOTE can create synthetic samples by randomly selecting a minority class sample and its nearest neighbors, then generating new samples along the line segments joining these neighbors. This process helps increase the representation of the minority class, improving model performance.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0004", "text": "Data augmentation for image classification", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0005", "text": "When convolutional neural networks grew larger in mid-1990s, there was a lack of data to use, especially considering that some part of the overall dataset should be spared for later testing. It was proposed to perturb existing data with affine transformations to create new examples with the same labels, which were complemented by so-called elastic distortions in 2003, and the technique was widely used as of 2010s", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0006", "text": ". Data augmentation can enhance CNN performance and acts as a countermeasure against CNN profiling attacks.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0007", "text": "Data augmentation has become fundamental in image classification, enriching training dataset diversity to improve model generalization and performance. The evolution of this practice has introduced a broad spectrum of techniques, including geometric transformations, color space adjustments, and noise injection.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0008", "text": "Geometric Transformations\nGeometric transformations alter the spatial properties of images to simulate different perspectives, orientations, and scales. Common techniques include:", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0009", "text": "Affine Transformation\nRotation: Rotating images by a specified degree to help models recognize objects at various angles.\nReflection: Reflecting images horizontally or vertically to introduce variability in orientation.\nTranslation: Shifting images in different directions to teach models positional invariance.\nScaling\nShear Mapping\nCropping: Removing sections of the image to focus on particular features or simulate closer views.\nElastic Distortion", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0010", "text": "Elastic Distortion\nMorphing within the same class: Generating new samples by applying morphing techniques between two images belonging to the same class, thereby increasing intra-class diversity.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0011", "text": "Color Space Transformations\nColor space transformations modify the color properties of images, addressing variations in lighting, color saturation, and contrast. Techniques include:", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0012", "text": "Brightness Adjustment: Varying the image's brightness to simulate different lighting conditions.\nContrast Adjustment: Changing the contrast to help models recognize objects under various clarity levels.\nSaturation Adjustment: Altering saturation to prepare models for images with diverse color intensities.\nColor Jittering: Randomly adjusting brightness, contrast, saturation, and hue to introduce color variability.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0013", "text": "Noise Injection\nInjecting noise into images simulates real-world imperfections, teaching models to ignore irrelevant variations. Techniques involve:\n\nGaussian Noise: Adding Gaussian noise mimics sensor noise or graininess.\nSalt and Pepper Noise: Introducing black or white pixels at random simulates sensor dust or dead pixels.\n\nData augmentation for signal processing\nResidual or block bootstrap can be used for time series augmentation.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0014", "text": "Biological signals", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0015", "text": "Synthetic data augmentation is of paramount importance for machine learning classification, particularly for biological data, which tend to be high dimensional and scarce. The applications of robotic control and augmentation in disabled and able-bodied subjects still rely mainly on subject-specific analyses. Data scarcity is notable in signal processing problems such as for Parkinson's Disease Electromyography signals, which are difficult to source - Zanini, et al", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0016", "text": ". noted that it is possible to use a generative adversarial network (in particular, a DCGAN) to perform style transfer in order to generate synthetic electromyographic signals that corresponded to those exhibited by sufferers of Parkinson's Disease.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0017", "text": "The approaches are also important in electroencephalography (brainwaves). Wang, et al. explored the idea of using deep convolutional neural networks for EEG-Based Emotion Recognition, results show that emotion recognition was improved when data augmentation was used. \nA common approach is to generate synthetic signals by re-arranging components of real data. Lotte proposed a method of \"Artificial Trial Generation Based on Analogy\" where three data examples \n  \n    \n      \n        \n          x", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0018", "text": "x\n          \n            1\n          \n        \n        ,\n        \n          x\n          \n            2\n          \n        \n        ,\n        \n          x\n          \n            3\n          \n        \n      \n    \n    {\\displaystyle x_{1},x_{2},x_{3}}\n  \n provide examples and an artificial \n  \n    \n      \n        \n          x\n          \n            s\n            y\n            n\n            t\n            h\n            e\n            t\n            i\n            c", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0019", "text": "t\n            i\n            c\n          \n        \n      \n    \n    {\\displaystyle x_{synthetic}}\n  \n is formed which is to \n  \n    \n      \n        \n          x\n          \n            3\n          \n        \n      \n    \n    {\\displaystyle x_{3}}\n  \n what \n  \n    \n      \n        \n          x\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle x_{2}}\n  \n is to \n  \n    \n      \n        \n          x\n          \n            1", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0020", "text": "1\n          \n        \n      \n    \n    {\\displaystyle x_{1}}\n  \n. A transformation is applied to \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n      \n    \n    {\\displaystyle x_{1}}\n  \n to make it more similar to \n  \n    \n      \n        \n          x\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle x_{2}}\n  \n, the same transformation is then applied to \n  \n    \n      \n        \n          x\n          \n            3", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0021", "text": "x\n          \n            3\n          \n        \n      \n    \n    {\\displaystyle x_{3}}\n  \n which generates \n  \n    \n      \n        \n          x\n          \n            s\n            y\n            n\n            t\n            h\n            e\n            t\n            i\n            c\n          \n        \n      \n    \n    {\\displaystyle x_{synthetic}}\n  \n. This approach was shown to improve performance of a Linear Discriminant Analysis classifier on three different datasets.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0022", "text": "Current research shows great impact can be derived from relatively simple techniques. For example, Freer observed that introducing noise into gathered data to form additional data points improved the learning ability of several models which otherwise performed relatively poorly. Tsinganos et al", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0023", "text": ". Tsinganos et al. studied the approaches of magnitude warping, wavelet decomposition, and synthetic surface EMG models (generative approaches) for hand gesture recognition, finding classification performance increases of up to +16% when augmented data was introduced during training", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0024", "text": ". More recently, data augmentation studies have begun to focus on the field of deep learning, more specifically on the ability of generative models to create artificial data which is then introduced during the classification model training process. In 2018, Luo et al. observed that useful EEG signal data could be generated by Conditional Wasserstein Generative Adversarial Networks (GANs) which was then introduced to the training set in a classical train-test learning framework", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0025", "text": ". The authors found classification performance was improved when such techniques were introduced.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0026", "text": "Mechanical  signals", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0027", "text": "The prediction of mechanical signals based on data augmentation brings a new generation of technological innovations, such as new energy dispatch, 5G communication field, and robotics control engineering. In 2022, Yang et al", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0028", "text": ". In 2022, Yang et al. integrate constraints, optimization and control into a deep network framework based on data augmentation and data pruning with spatio-temporal data correlation, and improve the interpretability, safety and controllability of deep learning in real industrial projects through explicit mathematical programming equations and analytical solutions.", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Data_augmentation_0029", "text": "See also\nOversampling and undersampling in data analysis\nSurrogate data\nGenerative adversarial network\nVariational autoencoder\nData pre-processing\nConvolutional neural network\nRegularization (mathematics)\nData preparation\nData fusion\n\n\n== References ==", "title": "Data augmentation", "url": "https://en.wikipedia.org/wiki/Data_augmentation"}, {"id": "rec_Selfsupervised_learning_0000", "text": "Self-supervised learning (SSL) is a paradigm in machine learning where a model is trained on a task using the data itself to generate supervisory signals, rather than relying on externally-provided labels. In the context of neural networks, self-supervised learning aims to leverage inherent structures or relationships within the input data to create meaningful training signals. SSL tasks are designed so that solving them requires capturing essential features or relationships in the data", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0001", "text": ". The input data is typically augmented or transformed in a way that creates pairs of related samples, where one sample serves as the input, and the other is used to formulate the supervisory signal. This augmentation can involve introducing noise, cropping, rotation, or other transformations. Self-supervised learning more closely imitates the way humans learn to classify objects.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0002", "text": "During SSL, the model learns in two steps. First, the task is solved based on an auxiliary or pretext classification task using pseudo-labels, which help to initialize the model parameters. Next, the actual task is performed with supervised or unsupervised learning.\nSelf-supervised learning has produced promising results in recent years, and has found practical application in fields such as audio processing, and is being used by Facebook and others for speech recognition.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0003", "text": "Pseudo-labels\nPseudo-labels are automatically generated labels that a model assigns to unlabeled data based on its own predictions. They are widely used in self-supervised and semi-supervised learning, where ground-truth annotations are limited or unavailable. By treating predicted labels as surrogate ground truth, learning algorithms can make use of large quantities of unlabeled data in the training process.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0004", "text": "Pseudo-labeling also plays an important role in systems that must adapt to concept drift, where the statistical properties of the data change over time. In these scenarios, the model may detect that an incoming instance deviates from previously learned behavior. The system then generates a classification result for that instance, and this predicted class is used as a pseudo-label for updating or retraining model components that are becoming outdated", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0005", "text": ". This approach enables continuous adaptation in dynamic environments without requiring manual annotation.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0006", "text": "In many adaptive learning pipelines, pseudo-labels are chosen when the classifier produces sufficiently confident predictions, reducing the risk of propagating errors. These pseudo-labeled instances are then incorporated into training to refresh or evolve the model's understanding of emerging data patterns, particularly when existing components show signs of \u201caging\u201d due to drift or distributional shifts", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0007", "text": ". This strategy reduces reliance on manual labeling while helping maintain long-term model performance.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0008", "text": "Types\nAutoassociative self-supervised learning\nAutoassociative self-supervised learning is a specific category of self-supervised learning where a neural network is trained to reproduce or reconstruct its own input data. In other words, the model is tasked with learning a representation of the data that captures its essential features or structure, allowing it to regenerate the original input.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0009", "text": "The term \"autoassociative\" comes from the fact that the model is essentially associating the input data with itself. This is often achieved using autoencoders, which are a type of neural network architecture used for representation learning. Autoencoders consist of an encoder network that maps the input data to a lower-dimensional representation (latent space), and a decoder network that reconstructs the input from this representation.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0010", "text": "The training process involves presenting the model with input data and requiring it to reconstruct the same data as closely as possible. The loss function used during training typically penalizes the difference between the original input and the reconstructed output (e.g. mean squared error). By minimizing this reconstruction error, the autoencoder learns a meaningful representation of the data in its latent space.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0011", "text": "Contrastive self-supervised learning", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0012", "text": "For a binary classification task, training data can be divided into positive examples and negative examples. Positive examples are those that match the target. For example, if training a classifier to identify birds, the positive training data would include images that contain birds. Negative examples would be images that do not. Contrastive self-supervised learning uses both positive and negative examples", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0013", "text": ". The loss function in contrastive learning is used to minimize the distance between positive sample pairs, while maximizing the distance between negative sample pairs.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0014", "text": "An early example uses a pair of 1-dimensional convolutional neural networks to process a pair of images and maximize their agreement.\nContrastive Language-Image Pre-training (CLIP) allows joint pretraining of a text encoder and an image encoder, such that a matching image-text pair have image encoding vector and text encoding vector that span a small angle (having a large cosine similarity).", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0015", "text": "InfoNCE (Noise-Contrastive Estimation) is a method to optimize two models jointly, based on Noise Contrastive Estimation (NCE). Given a set \n  \n    \n      \n        X\n        =\n        \n          {\n          \n            \n              x\n              \n                1\n              \n            \n            ,\n            \u2026\n            \n              x\n              \n                N\n              \n            \n          \n          }", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0016", "text": "}\n        \n      \n    \n    {\\displaystyle X=\\left\\{x_{1},\\ldots x_{N}\\right\\}}\n  \n of \n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}\n  \n random samples containing one positive sample from \n  \n    \n      \n        p\n        \n          (\n          \n            \n              x\n              \n                t\n                +\n                k\n              \n            \n            \u2223\n            \n              c\n              \n                t", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0017", "text": "t\n              \n            \n          \n          )\n        \n      \n    \n    {\\displaystyle p\\left(x_{t+k}\\mid c_{t}\\right)}\n  \n and \n  \n    \n      \n        N\n        \u2212\n        1\n      \n    \n    {\\displaystyle N-1}\n  \n negative samples from the 'proposal' distribution \n  \n    \n      \n        p\n        \n          (\n          \n            x\n            \n              t\n              +\n              k\n            \n          \n          )", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0018", "text": ")\n        \n      \n    \n    {\\displaystyle p\\left(x_{t+k}\\right)}\n  \n, it minimizes the following loss function:", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0019", "text": "L\n            \n          \n          \n            \n              N\n            \n          \n        \n        =\n        \u2212\n        \n          \n            E\n          \n          \n            X\n          \n        \n        \n          [\n          \n            log\n            \u2061\n            \n              \n                \n                  \n                    f\n                    \n                      k", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0020", "text": "k\n                    \n                  \n                  \n                    (\n                    \n                      \n                        x\n                        \n                          t\n                          +\n                          k\n                        \n                      \n                      ,\n                      \n                        c\n                        \n                          t", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0021", "text": ")\n                  \n                \n                \n                  \n                    \u2211\n                    \n                      \n                        x\n                        \n                          j\n                        \n                      \n                      \u2208\n                      X\n                    \n                  \n                  \n                    f", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0022", "text": "f\n                    \n                      k\n                    \n                  \n                  \n                    (\n                    \n                      \n                        x\n                        \n                          j\n                        \n                      \n                      ,\n                      \n                        c\n                        \n                          t", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0023", "text": ")\n                  \n                \n              \n            \n          \n          ]\n        \n      \n    \n    {\\displaystyle {\\mathcal {L}}_{\\mathrm {N} }=-\\mathbb {E} _{X}\\left[\\log {\\frac {f_{k}\\left(x_{t+k},c_{t}\\right)}{\\sum _{x_{j}\\in X}f_{k}\\left(x_{j},c_{t}\\right)}}\\right]}", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0024", "text": "Non-contrastive self-supervised learning\nNon-contrastive self-supervised learning (NCSSL) uses only positive examples. Counterintuitively, NCSSL converges on a useful local minimum rather than reaching a trivial solution, with zero loss. For the example of binary classification, it would trivially learn to classify each example as positive. Effective NCSSL requires an extra predictor on the online side that does not back-propagate on the target side.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0025", "text": "Joint-Embedding and Predictive Architectures", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0026", "text": "A major class of self-supervised learning moves beyond contrastive pairs, instead maximizing the agreement between views while preventing collapse through statistical constraints. Rooted in Deep Canonical Correlation Analysis (Deep CCA), this approach includes Joint-Embedding Architectures (JEA) like Barlow Twins and VICReg, which enforce covariance constraints to learn invariant representations without negative sampling", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0027", "text": ". Deep Latent Variable Path Modelling (DLVPM) generalizes this to multimodal systems, using path models to enforce correlation and orthogonality across diverse data types.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0028", "text": "In 2022, this framework evolved into Joint-Embedding Predictive Architectures (JEPA) when Yann LeCun introduced JEPA as a step towards decision making, reasoning, and autonomous human intelligence in machines, including self-improvement through autonomous learning. Founded in representation learning, LeCun established the concept of a \u201cworld model\u201d in JEPA which aims to enable machines to replicate human intellect by providing machines with a concept for the world in which they exist.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0029", "text": "Unlike autoencoders, JEPAs operate entirely in latent space, avoiding pixel-level noise to focus on semantic structure, a key step toward autonomous world models. Rather than just learning invariance, JEPAs learn by predicting masked latent representations from visible context. Recently, there have been several new applications of JEPA for things like image analysis, audio processing, and even motion in images and video.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0030", "text": "Comparison with other forms of machine learning\nSSL belongs to supervised learning methods insofar as the goal is to generate a classified output from the input. At the same time, however, it does not require the explicit use of labeled input-output pairs. Instead, correlations, metadata embedded in the data, or domain knowledge present in the input are implicitly and autonomously extracted from the data. These supervisory signals, extracted from the data, can then be used for training.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0031", "text": "SSL is similar to unsupervised learning in that it does not require labels in the sample data. Unlike unsupervised learning, however, learning is not done using inherent data structures.\nSemi-supervised learning combines supervised and unsupervised learning, requiring only a small portion of the learning data be labeled.\nIn transfer learning, a model designed for one task is reused on a different task.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0032", "text": "Training an autoencoder intrinsically constitutes a self-supervised process, because the output pattern needs to become an optimal reconstruction of the input pattern itself. However, in current jargon, the term 'self-supervised' often refers to tasks based on a pretext-task training setup. This involves the (human) design of such pretext task(s), unlike \nthe case of fully self-contained autoencoder training.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0033", "text": "In reinforcement learning, self-supervising learning from a combination of losses can create abstract representations where only the most important information about the state are kept in a compressed way.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0034", "text": "Examples\nSelf-supervised learning is particularly suitable for speech recognition. For example, Facebook developed wav2vec, a self-supervised algorithm, to perform speech recognition using two deep convolutional neural networks that build on each other.\nGoogle's Bidirectional Encoder Representations from Transformers (BERT) model is used to better understand the context of search queries.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0035", "text": "OpenAI's GPT-3 is an autoregressive language model that can be used in language processing. It can be used to translate texts or answer questions, among other things.\nBootstrap Your Own Latent (BYOL) is a NCSSL that produced excellent results on ImageNet and on transfer and semi-supervised benchmarks.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0036", "text": "The Yarowsky algorithm is an example of self-supervised learning in natural language processing. From a small number of labeled examples, it learns to predict which word sense of a polysemous word is being used at a given point in text.\nDirectPred is a NCSSL that directly sets the predictor weights instead of learning it via typical gradient descent.\nSelf-GenomeNet is an example of self-supervised learning in genomics.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0037", "text": "Self-supervised learning continues to gain prominence as a new approach across diverse fields. Its ability to leverage unlabeled data effectively opens new possibilities for advancement in machine learning, especially in data-driven application domains.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0038", "text": "References\nFurther reading\nBalestriero, Randall; Ibrahim, Mark; Sobal, Vlad; Morcos, Ari; Shekhar, Shashank; Goldstein, Tom; Bordes, Florian; Bardes, Adrien; Mialon, Gregoire; Tian, Yuandong; Schwarzschild, Avi; Wilson, Andrew Gordon; Geiping, Jonas; Garrido, Quentin; Fernandez, Pierre (24 April 2023). \"A Cookbook of Self-Supervised Learning\". arXiv:2304.12210 [cs.LG].", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0039", "text": "External links\nDoersch, Carl; Zisserman, Andrew (October 2017). \"Multi-task Self-Supervised Visual Learning\". 2017 IEEE International Conference on Computer Vision (ICCV). pp. 2070\u20132079. arXiv:1708.07860. doi:10.1109/ICCV.2017.226. ISBN 978-1-5386-1032-9. S2CID 473729.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0040", "text": "Doersch, Carl; Gupta, Abhinav; Efros, Alexei A. (December 2015). \"Unsupervised Visual Representation Learning by Context Prediction\". 2015 IEEE International Conference on Computer Vision (ICCV). pp. 1422\u20131430. arXiv:1505.05192. doi:10.1109/ICCV.2015.167. ISBN 978-1-4673-8391-2. S2CID 9062671.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0041", "text": "Zheng, Xin; Wang, Yong; Wang, Guoyou; Liu, Jianguo (1 April 2018). \"Fast and robust segmentation of white blood cell images by self-supervised learning\". Micron. 107: 55\u201371. doi:10.1016/j.micron.2018.01.010. ISSN 0968-4328. PMID 29425969. S2CID 3796689.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Selfsupervised_learning_0042", "text": "Yarowsky, David (1995). \"Unsupervised Word Sense Disambiguation Rivaling Supervised Methods\". Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics. Cambridge, MA: Association for Computational Linguistics: 189\u2013196. doi:10.3115/981658.981684. Retrieved 1 November 2022.", "title": "Self-supervised learning", "url": "https://en.wikipedia.org/wiki/Self-supervised_learning"}, {"id": "rec_Federated_learning_0000", "text": "Federated learning (also known as collaborative learning) is a machine learning technique in a setting where multiple entities (often called clients) collaboratively train a model while keeping their data decentralized, rather than centrally stored. A defining characteristic of federated learning is data heterogeneity. Because client data is decentralized, data samples held by each client may not be independently and identically distributed.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0001", "text": "Federated learning is generally concerned with and motivated by issues such as data privacy, data minimization, and data access rights. Its applications involve a variety of research areas including defence, telecommunications, the Internet of things, and pharmaceuticals.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0002", "text": "Definition\nFederated learning aims at training a machine learning algorithm, for instance deep neural networks, on multiple local datasets contained in local nodes without explicitly exchanging data samples. The general principle consists in training local models on local data samples and exchanging parameters (e.g. the weights and biases of a deep neural network) between these local nodes at some frequency to generate a global model shared by all nodes.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0003", "text": "The main difference between federated learning and distributed learning lies in the assumptions made on the properties of the local datasets, as distributed learning originally aims at parallelizing computing power where federated learning originally aims at training on heterogeneous datasets", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0004", "text": ". While distributed learning also aims at training a single model on multiple servers, a common underlying assumption is that the local datasets are independent and identically distributed (i.i.d.) and roughly have the same size. None of these hypotheses are made for federated learning; instead, the datasets are typically heterogeneous and their sizes may span several orders of magnitude", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0005", "text": ". Moreover, the clients involved in federated learning may be unreliable as they are subject to more failures or drop out since they commonly rely on less powerful communication media (i.e. Wi-Fi) and battery-powered systems (i.e. smartphones and IoT devices) compared to distributed learning where nodes are typically datacenters that have powerful computational capabilities and are connected to one another with fast networks.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0006", "text": "Mathematical formulation\nThe objective function for federated learning is as follows:", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0007", "text": "f\n        (\n        \n          \n            x\n          \n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \n            x\n          \n          \n            K\n          \n        \n        )\n        =\n        \n          \n            \n              1\n              K\n            \n          \n        \n        \n          \u2211\n          \n            i\n            =\n            1\n          \n          \n            K", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0008", "text": "K\n          \n        \n        \n          f\n          \n            i\n          \n        \n        (\n        \n          \n            x\n          \n          \n            i\n          \n        \n        )\n      \n    \n    {\\displaystyle f(\\mathbf {x} _{1},\\dots ,\\mathbf {x} _{K})={\\dfrac {1}{K}}\\sum _{i=1}^{K}f_{i}(\\mathbf {x} _{i})}", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0009", "text": "where \n  \n    \n      \n        K\n      \n    \n    {\\displaystyle K}\n  \n is the number of nodes, \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n are the weights of model as viewed by node \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n, and \n  \n    \n      \n        \n          f\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle f_{i}}\n  \n is node", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0010", "text": "{\\displaystyle f_{i}}\n  \n is node \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n's local objective function, which describes how model weights \n  \n    \n      \n        \n          \n            x\n          \n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n conforms to node \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \n's local dataset.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0011", "text": "{\\displaystyle i}\n  \n's local dataset.\nThe goal of federated learning is to train a common model on all of the nodes' local datasets, in other words:", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0012", "text": "Optimizing the objective function \n  \n    \n      \n        f\n        (\n        \n          \n            x\n          \n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \n            x\n          \n          \n            K\n          \n        \n        )\n      \n    \n    {\\displaystyle f(\\mathbf {x} _{1},\\dots ,\\mathbf {x} _{K})}\n  \n.\nAchieving consensus on \n  \n    \n      \n        \n          \n            x\n          \n          \n            i", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0013", "text": "i\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{i}}\n  \n. In other words, \n  \n    \n      \n        \n          \n            x\n          \n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          \n            x\n          \n          \n            K\n          \n        \n      \n    \n    {\\displaystyle \\mathbf {x} _{1},\\dots ,\\mathbf {x} _{K}}\n  \n converge to some common \n  \n    \n      \n        \n          x", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0014", "text": "x\n        \n      \n    \n    {\\displaystyle \\mathbf {x} }\n  \n at the end of the training process.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0015", "text": "Centralized federated learning\nIn the centralized federated learning setting, a central server is used to orchestrate the different steps of the algorithms and coordinate all the participating nodes during the learning process. The server is responsible for the nodes selection at the beginning of the training process and for the aggregation of the received model updates. Since all the selected nodes have to send updates to a single entity, the server may become a bottleneck of the system.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0016", "text": "Decentralized federated learning\nIn the decentralized federated learning setting, the nodes are able to coordinate themselves to obtain the global model. This setup prevents single point failures as the model updates are exchanged only between interconnected nodes without the orchestration of the central server. Nevertheless, the specific network topology may affect the performances of the learning process. See blockchain-based federated learning and the references therein.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0017", "text": "Personalized federated learning", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0018", "text": "In many application domains, not all clients are the same. Clients may share the same model architecture, but each have a different data distribution\u2014which is known as Personalized federated learning (PFL), or even have different model architectures, like mobile phones and IoT devices\u2014which is known as Heterogeneous federated learning. The key challenge in training a PFL model is to keep each client different from others, and at the same time benefit from generalizing across clients.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0019", "text": "Heterogeneous federated learning", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0020", "text": "Most of the existing federated learning strategies assume that local models share the same global model architecture. Recently, a new federated learning framework named HeteroFL was developed to address heterogeneous clients equipped with very different computation and communication capabilities", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0021", "text": ". The HeteroFL technique can enable the training of heterogeneous local models with dynamically varying computation and non-IID data complexities while still producing a single accurate global inference model.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0022", "text": "Main features\nIterative learning", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0023", "text": "The iterative process of federated learning is composed of a series of fundamental client-server interactions, each of which is known as a federated learning round. Each round of this process consists in transmitting the current global model state to participating nodes, training local models on these local nodes to produce a set of potential model updates at each node, and then aggregating and processing these local updates into a single global update and applying it to the global model.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0024", "text": "In the methodology below, a central server is used for aggregation, while local nodes perform local training depending on the central server's orders. However, other strategies lead to the same results without central servers, in a peer-to-peer approach, using gossip or consensus methodologies.\nAssuming a federated round composed by one iteration of the learning process, the learning procedure can be summarized as follows:", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0025", "text": "Initialization: according to the server inputs, a machine learning model (e.g., linear regression, neural network, boosting) is chosen to be trained on local nodes and initialized. Then, nodes are activated and wait for the central server to give the calculation tasks.\nClient selection: a fraction of local nodes are selected to start training on local data. The selected nodes acquire the current statistical model while the others wait for the next federated round.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0026", "text": "Configuration: the central server orders selected nodes to undergo training of the model on their local data in a pre-specified fashion (e.g., for some mini-batch updates of gradient descent).", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0027", "text": "Reporting: each selected node sends its local model to the server for aggregation. The central server aggregates the received models and sends back the model updates to the nodes. It also handles failures for disconnected nodes or lost model updates. The next federated round is started returning to the client selection phase.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0028", "text": "Termination: once a pre-defined termination criterion is met (e.g., a maximum number of iterations is reached or the model accuracy is greater than a threshold) the central server aggregates the updates and finalizes the global model.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0029", "text": "The procedure considered before assumes synchronized model updates. Recent federated learning developments introduced novel techniques to tackle asynchronicity during the training process, or training with dynamically varying models", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0030", "text": ". Compared to synchronous approaches where local models are exchanged once the computations have been performed for all layers of the neural network, asynchronous ones leverage the properties of neural networks to exchange model updates as soon as the computations of a certain layer are available. These techniques are also commonly referred to as split learning and they can be applied both at training and inference time regardless of centralized or decentralized federated learning settings.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0031", "text": "Non-IID data", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0032", "text": "In most cases, the assumption of independent and identically distributed samples across local nodes does not hold for federated learning setups. Under this setting, the performances of the training process may vary significantly according to the unbalanced local data samples as well as the particular probability distribution of the training examples (i.e., features and labels) stored at the local nodes", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0033", "text": ". To further investigate the effects of non-IID data, the following description considers the main categories presented in the preprint by Peter Kairouz et al. from 2019.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0034", "text": "The description of non-IID data relies on the analysis of the joint probability between features and labels for each node.\nThis allows decoupling of each contribution according to the specific distribution available at the local nodes.\nThe main categories for non-iid data can be summarized as follows:", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0035", "text": "Covariate shift: local nodes may store examples that have different statistical distributions compared to other nodes. An example occurs in natural language processing datasets where people typically write the same digits/letters with different stroke widths or slants.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0036", "text": "Prior probability shift: local nodes may store labels that have different statistical distributions compared to other nodes. This can happen if datasets are regional and/or demographically partitioned. For example, datasets containing images of animals vary significantly from country to country.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0037", "text": "Concept drift (same label, different features): local nodes may share the same labels but some of them correspond to different features at different local nodes. For example, images that depict a particular object can vary according to the weather condition in which they were captured.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0038", "text": "Concept shift (same features, different labels): local nodes may share the same features but some of them correspond to different labels at different local nodes. For example, in natural language processing, the sentiment analysis may yield different sentiments even if the same text is observed.\nUnbalanced: the amount of data available at the local nodes may vary significantly in size.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0039", "text": "The loss in accuracy due to non-iid data can be bounded through using more sophisticated means of doing data normalization, rather than batch normalization.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0040", "text": "Algorithmic hyper-parameters\nNetwork topology\nThe way the statistical local outputs are pooled and the way the nodes communicate with each other can change from the centralized model explained in the previous section. This leads to a variety of federated learning approaches: for instance no central orchestrating server, or stochastic communication.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0041", "text": "In particular, orchestrator-less distributed networks are one important variation. In this case, there is no central server dispatching queries to local nodes and aggregating local models. Each local node sends its outputs to several randomly-selected others, which aggregate their results locally. This restrains the number of transactions, thereby sometimes reducing training time and computing cost.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0042", "text": "Federated learning parameters\nOnce the topology of the node network is chosen, one can control different parameters of the federated learning process (in addition to the machine learning model's own hyperparameters) to optimize learning:\n\nNumber of federated learning rounds: \n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}\n  \n\nTotal number of nodes used in the process: \n  \n    \n      \n        K\n      \n    \n    {\\displaystyle K}", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0043", "text": "Fraction of nodes used at each iteration for each node: \n  \n    \n      \n        C\n      \n    \n    {\\displaystyle C}\n  \n\nLocal batch size used at each learning iteration: \n  \n    \n      \n        B\n      \n    \n    {\\displaystyle B}\n  \n\nOther model-dependent parameters can also be tinkered with, such as:\n\nNumber of iterations for local training before pooling: \n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0044", "text": "Local learning rate: \n  \n    \n      \n        \u03b7\n      \n    \n    {\\displaystyle \\eta }", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0045", "text": "Those parameters have to be optimized depending on the constraints of the machine learning application (e.g., available computing power, available memory, bandwidth). For instance, stochastically choosing a limited fraction \n  \n    \n      \n        C\n      \n    \n    {\\displaystyle C}\n  \n of nodes for each iteration diminishes computing cost and may prevent overfitting, in the same way that stochastic gradient descent can reduce overfitting.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0046", "text": "Limitations\nTechnical", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0047", "text": "Federated learning requires frequent communication between nodes during the learning process. Thus, it requires not only enough local computing power and memory, but also high bandwidth connections to be able to exchange parameters of the machine learning model. However, the technology also avoids data communication, which can require significant resources before starting centralized machine learning", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0048", "text": ". Nevertheless, the devices typically employed in federated learning are communication-constrained, for example IoT devices or smartphones are generally connected to Wi-Fi networks, thus, even if the models are commonly less expensive to be transmitted compared to raw data, federated learning mechanisms may not be suitable in their general form.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0049", "text": "Federated learning raises several statistical challenges:", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0050", "text": "Heterogeneity between the different local datasets: each node may have some bias with respect to the general population, and the size of the datasets may vary significantly;\nTemporal heterogeneity: each local dataset's distribution may vary with time;\nInteroperability of each node's dataset is a prerequisite;\nEach node's dataset may require regular curations;\nHiding training data might allow attackers to inject backdoors into the global model;", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0051", "text": "Lack of access to global training data makes it harder to identify unwanted biases entering the training e.g. age, gender, sexual orientation;\nPartial or total loss of model updates due to node failures affecting the global model;\nLack of annotations or labels on the client side.\nHeterogeneity between processing platforms", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0052", "text": "Governance", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0053", "text": "Governance considerations may also limit the applicability of federated learning, in particular when applied in cross-organisational settings. Most federated learning frameworks rely on a central coordinating server, which introduces questions regarding infrastructure control, model ownership, and decision-making authority among collaborators", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0054", "text": ". In multi-party deployments, especially those spanning competitive or collaborative data markets, the absence of agreed-upon governance structures can create uncertainty about responsibilities for validating model updates, auditing system behaviour, and managing security incidents. These organisational issues may deter participants from contributing to shared models and can complicate regulatory compliance, thereby constraining adoption.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0055", "text": "A further governance-related issue concerns the distribution of value among participants as multi-party federated learning systems scale. As more clients contribute to training, model accuracy may approach a performance threshold, after which additional contributions yield diminishing marginal utility. This dynamic can create tensions within collaborative arrangements, as early participants may perceive later contributions as offering reduced value while still benefiting from the shared model", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0056", "text": ". Such asymmetries may affect incentives to join or remain in a consortium and complicate the design of fair participation, ownership, or reward structures. Establishing transparent rules for contribution assessment and benefit allocation is therefore an important organisational challenge and represents a direction for future research.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0057", "text": "Algorithms\nA number of different algorithms for federated optimization have been proposed.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0058", "text": "Federated stochastic gradient descent (FedSGD)\nStochastic gradient descent is an approach used in deep learning, where gradients are computed on a random subset of the total dataset and then used to make one step of the gradient descent..", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0059", "text": "Federated stochastic gradient descent is the analog of this algorithm to the federated setting, but uses a random subset of the nodes, each node using all its data. The server averages the gradients in proportion to the number of training data on each node, and uses the average to make a gradient descent step.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0060", "text": "Federated averaging (FedAvg)", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0061", "text": "Federated averaging (FedAvg) is a generalization of FedSGD which allows nodes to do more than one batch update on local data and exchange updated weights rather than gradients. This reduces communication and is equivalent to averaging the weights if all nodes start with the same weights. It does not seem to hurt the resulting averaged model's performance compared to FedSGD. FedAvg variations have been proposed based on adaptive optimizers such as ADAM and AdaGrad, and tend to outperform FedAvg.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0062", "text": "Federated Proximal (FedProx)\nFederated Proximal (FedProx) extends FedAvg by adding a proximal term to the local objective in the form of \n  \n    \n      \n        \n          \n            \u03bc\n            2\n          \n        \n        \u2016\n        w\n        \u2212\n        \n          w\n          \n            t\n          \n        \n        \n          \u2016\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle {\\frac {\\mu }{2}}\\|w-w^{t}\\|^{2}}\n  \n, where \n  \n    \n      \n        \u03bc", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0063", "text": ", where \n  \n    \n      \n        \u03bc\n      \n    \n    {\\displaystyle \\mu }\n  \n is the FedProx hyper parameter, in order to address data heterogeneity across clients.  The proximal term constrains local updates, helping reduce client drift when client data are non-IID.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0064", "text": "Federated learning with dynamic regularization (FedDyn)", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0065", "text": "Federated learning methods suffer when node datasets are distributed heterogeneously, because then minimizing the node losses is not the same as minimizing the global loss. In 2021, Acar et al. introduced a solution called FedDyn, which dynamically regularizes each node loss function so that they converge to the global loss. Since the local losses are aligned, FedDyn is robust to the different heterogeneity levels and so it can safely perform full minimization in each device", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0066", "text": ". In theory, FedDyn converges to the optimal (a stationary point for nonconvex losses) by being agnostic to the heterogeneity levels. These claims are verified with extensive experiments on various datasets.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0067", "text": "Besides reducing communication, it is also beneficial to reduce local computation. To do this, FedDynOneGD  modifies FedDyn to calculate only one gradient per node per round, regularizes it and updates the global model with it. Hence, the computational complexity is linear in local dataset size. Moreover, gradient computation can be parallelized on each node, unlike successive SGD steps. In theory, FedDynOneGD achieves the same convergence guarantees as in FedDyn with less local computation.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0068", "text": "Personalized federated learning by pruning (Sub-FedAvg)", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0069", "text": "Federated learning methods have poor global performance under non-IID settings. This motivates clients to yield personalized models in federation. To change this, Vahidian et al. recently introduced the algorithm Sub-FedAvg which does hybrid pruning (structured and unstructured pruning) with averaging on the intersection of clients' drawn subnetworks. This simultaneously addresses communication efficiency, resource constraints and personalized models accuracies.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0070", "text": "Sub-FedAvg also extends the \"lottery ticket hypothesis\" of centrally trained neural networks to federated learning, with the research question: \"Do winning tickets exist for clients' neural networks being trained in federated learning? If so, how to effectively draw the personalized subnetworks for each client?\" Sub-FedAvg experimentally answers \"yes\" and proposes two algorithms to effectively draw the personalized subnetworks.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0071", "text": "Dynamic aggregation - inverse distance aggregation\nIDA (Inverse Distance Aggregation) is a novel adaptive weighting approach for federated learning nodes based on meta-information which handles unbalanced and non-iid data. It uses the distance of the model parameters as a strategy to minimize the effect of outliers and improve the model's convergence rate.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0072", "text": "Hybrid federated dual coordinate ascent (HyFDCA)", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0073", "text": "Very few methods for hybrid federated learning, where clients only hold subsets of both features and samples, exist. Yet, this scenario is very important in practical settings. Hybrid Federated Dual Coordinate Ascent (HyFDCA) is a novel algorithm proposed in 2024 that solves convex problems in the hybrid FL setting. This algorithm extends CoCoA, a primal-dual distributed optimization algorithm introduced by Jaggi et al. (2014) and Smith et al", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0074", "text": ". (2014) and Smith et al. (2017), to the case where both samples and features are partitioned across clients.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0075", "text": "HyFDCA claims several improvement over existing algorithms:", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0076", "text": "HyFDCA is a provably convergent primal-dual algorithm for hybrid FL in at least the following settings.\nHybrid Federated Setting with Complete Client Participation\nHorizontal Federated Setting with Random Subsets of Available Clients\nThe authors show HyFDCA enjoys a convergence rate of O(1\u2044t) which matches the convergence rate of FedAvg (see below).\nVertical Federated Setting with Incomplete Client Participation", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0077", "text": "The authors show HyFDCA enjoys a convergence rate of O(log(t)\u2044t) whereas FedBCD exhibits a slower O(1\u2044sqrt(t)) convergence rate and requires full client participation.\nHyFDCA provides the privacy steps that ensure privacy of client data in the primal-dual setting. These principles apply to future efforts in developing primal-dual algorithms for FL.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0078", "text": "HyFDCA empirically outperforms HyFEM and FedAvg in loss function value and validation accuracy across a multitude of problem settings and datasets (see below for more details). The authors also introduce a hyperparameter selection framework for FL with competing metrics using ideas from multiobjective optimization.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0079", "text": "There is only one other algorithm that focuses on hybrid FL, HyFEM proposed by Zhang et al. (2020). This algorithm uses a feature matching formulation that balances clients building accurate local models and the server learning an accurate global model. This requires a matching regularizer constant that must be tuned based on user goals and results in disparate local and global models", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0080", "text": ". Furthermore, the convergence results provided for HyFEM only prove convergence of the matching formulation not of the original global problem. This work is substantially different than HyFDCA's approach which uses data on local clients to build a global model that converges to the same solution as if the model was trained centrally. Furthermore, the local and global models are synchronized and do not require the adjustment of a matching parameter between local and global models", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0081", "text": ". However, HyFEM is suitable for a vast array of architectures including deep learning architectures, whereas HyFDCA is designed for convex problems like logistic regression and support vector machines.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0082", "text": "HyFDCA is empirically benchmarked against the aforementioned HyFEM as well as the popular FedAvg in solving convex problem (specifically classification problems) for several popular datasets (MNIST, Covtype, and News20). The authors found HyFDCA converges to a lower loss value and higher validation accuracy in less overall time in 33 of 36 comparisons examined and 36 of 36 comparisons examined with respect to the number of outer iterations", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0083", "text": ". Lastly, HyFDCA only requires tuning of one hyperparameter, the number of inner iterations, as opposed to FedAvg (which requires tuning three) or HyFEM (which requires tuning four). In addition to FedAvg and HyFEM being quite difficult to optimize hyperparameters in turn greatly affecting convergence, HyFDCA's single hyperparameter allows for simpler practical implementations and hyperparameter selection methodologies.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0084", "text": "Current research topics", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0085", "text": "Federated learning has started to emerge as an important research topic in 2015 and 2016, with the first publications on federated averaging in telecommunication settings. Before that, in a thesis work titled \"A Framework for Multi-source Prefetching Through Adaptive Weight\", an approach to aggregate predictions from multiple models trained at three location of a request response cycle with was proposed", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0086", "text": ".  Another important aspect of active research is the reduction of the communication burden during the federated learning process. In 2017 and 2018, publications have emphasized the development of resource allocation strategies, especially to reduce communication requirements between nodes with gossip algorithms as well as on the characterization of the robustness to differential privacy attacks", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0087", "text": ". Other research activities focus on the reduction of the bandwidth during training through sparsification and quantization methods, where the machine learning models are sparsified and/or compressed before they are shared with other nodes. Developing ultra-light DNN architectures is essential for device-/edge- learning and recent work recognises both the energy efficiency requirements  for future federated learning and the need to compress deep learning, especially during learning.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0088", "text": "Recent research advancements are starting to consider real-world propagating channels as in previous implementations ideal channels were assumed. Another active direction of research is to develop Federated learning for training heterogeneous local models with varying computation complexities and producing a single powerful global inference model.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0089", "text": "A learning framework named Assisted learning was recently developed to improve each agent's learning capabilities without transmitting private data, models, and even learning objectives. Compared with Federated learning that often requires a central controller to orchestrate the learning and optimization, Assisted learning aims to provide protocols for the agents to optimize and learn among themselves without a global model.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0090", "text": "Use cases\nFederated learning typically applies when individual actors need to train models on larger datasets than their own, but cannot afford to share the data in itself with others (e.g., for legal, strategic or economic reasons). The technology yet requires good connections between local servers and minimum computational power for each node.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0091", "text": "Transportation: self-driving cars", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0092", "text": "Self-driving cars encapsulate many machine learning technologies to function: computer vision for analyzing obstacles, machine learning for adapting their pace to the environment (e.g., bumpiness of the road). Due to the potential high number of self-driving cars and the need for them to quickly respond to real world situations, traditional cloud approach may generate safety risks", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0093", "text": ". Federated learning can represent a solution for limiting volume of data transfer and accelerating learning processes.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0094", "text": "Industry 4.0: smart manufacturing", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0095", "text": "In Industry 4.0, there is a widespread adoption of machine learning techniques to improve the efficiency and effectiveness of industrial process while guaranteeing a high level of safety. Nevertheless, privacy of sensitive data for industries and manufacturing companies is of paramount importance. Federated learning algorithms can be applied to these problems as they do not disclose any sensitive data", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0096", "text": ". In addition, FL also implemented for PM2.5 prediction to support Smart city sensing applications.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0097", "text": "Medicine: digital health", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0098", "text": "Federated learning seeks to address the problem of data governance and privacy by training algorithms collaboratively without exchanging the data itself. Today's standard approach of centralizing data from multiple centers comes at the cost of critical concerns regarding patient privacy and data protection. To solve this problem, the ability to train machine learning models at scale across multiple medical institutions without moving the data is a critical technology", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0099", "text": ". Nature Digital Medicine published the paper \"The Future of Digital Health with Federated Learning\" in September 2020, in which the authors explore how federated learning may provide a solution for the future of digital health, and highlight the challenges and considerations that need to be addressed. Recently, a collaboration of 20 different institutions around the world validated the utility of training AI models using federated learning", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0100", "text": ". In a paper published in Nature Medicine \"Federated learning for predicting clinical outcomes in patients with COVID-19\", they showcased the accuracy and generalizability of a federated AI model for the prediction of oxygen needs in patients with COVID-19 infections", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0101", "text": ". Furthermore, in a published paper \"A Systematic Review of Federated Learning in the Healthcare Area: From the Perspective of Data Properties and Applications\", the authors trying to provide a set of challenges on FL challenges on medical data-centric perspective.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0102", "text": "A coalition from industry and academia has developed MedPerf, an open source platform that enables validation of medical AI models in real world data. The platform relies technically on federated evaluation of AI models aiming to alleviate concerns of patient privacy and conceptually on diverse benchmark committees to build the specifications of neutral clinically impactful benchmarks.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0103", "text": "Robotics", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0104", "text": "Robotics includes a wide range of applications of machine learning methods: from perception and decision-making to control. As robotic technologies have been increasingly deployed from simple and repetitive tasks (e.g. repetitive manipulation) to complex and unpredictable tasks (e.g. autonomous navigation), the need for machine learning grows. Federated Learning provides a solution to improve over conventional machine learning training methods", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0105", "text": ". In the paper, mobile robots learned navigation over diverse environments using the FL-based method, helping generalization. In the paper, Federated Learning is applied to improve multi-robot navigation under limited communication bandwidth scenarios, which is a current challenge in real-world learning-based robotic tasks. In the paper, Federated Learning is used to learn Vision-based navigation, helping better sim-to-real transfer.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0106", "text": "Biometrics", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0107", "text": "Federated Learning (FL) is transforming biometric recognition by enabling collaborative model training across distributed data sources while preserving privacy. By eliminating the need to share sensitive biometric templates like fingerprints, facial images, and iris scans, FL addresses privacy concerns and regulatory constraints, allowing for improved model accuracy and generalizability", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0108", "text": ". It mitigates challenges of data fragmentation by leveraging scattered datasets, making it particularly effective for diverse biometric applications such as facial and iris recognition. However, FL faces challenges, including model and data heterogeneity, computational overhead, and vulnerability to security threats like inference attacks", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0109", "text": ". Future directions include developing personalized FL frameworks, enhancing system efficiency, and expanding FL applications to biometric presentation attack detection (PAD) and quality assessment, fostering innovation and robust solutions in privacy-sensitive environments.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0110", "text": "See also\nDistributed artificial intelligence", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Federated_learning_0111", "text": "References\nExternal links\n\"Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016\" at eur-lex.europa.eu. Retrieved October 18, 2019.\n\"Data minimisation and privacy-preserving techniques in AI systems\" Archived 2020-07-23 at the Wayback Machine at UK Information Commissioners Office. Retrieved July 22, 2020\n\"Realising the Potential of Data Whilst Preserving Privacy with EyA and Conclave from R3\" at eya.global. Retrieved March 31, 2022.", "title": "Federated learning", "url": "https://en.wikipedia.org/wiki/Federated_learning"}, {"id": "rec_Explainable_artificial_intelligence_0000", "text": "Within artificial intelligence (AI), explainable AI (XAI), generally overlapping with interpretable AI or explainable machine learning (XML), is a field of research that explores methods that provide humans with the ability of intellectual oversight over AI algorithms. The main focus is on the reasoning behind the decisions or predictions made by the AI algorithms, to make them more understandable and transparent", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0001", "text": ". This addresses users' requirement to assess safety and scrutinize the automated decision making in applications. XAI counters the \"black box\" tendency of machine learning, where even the AI's designers cannot explain why it arrived at a specific decision.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0002", "text": "XAI hopes to help users of AI-powered systems perform more effectively by improving their understanding of how those systems reason. XAI may be an implementation of the social right to explanation. Even if there is no such legal right or regulatory requirement, XAI can improve the user experience of a product or service by helping end users trust that the AI is making good decisions", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0003", "text": ". XAI aims to explain what has been done, what is being done, and what will be done next, and to unveil which information these actions are based on. This makes it possible to confirm existing knowledge, challenge existing knowledge, and generate new assumptions.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0004", "text": "Background\nMachine learning (ML) algorithms used in AI can be categorized as white-box or black-box. White-box models provide results that are understandable to experts in the domain. Black-box models, on the other hand, are extremely hard to explain and may not be understood even by domain experts. XAI algorithms follow the three principles of transparency, interpretability, and explainability.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0005", "text": "A model is transparent \"if the processes that extract model parameters from training data and generate labels from testing data can be described and motivated by the approach designer.\"\nInterpretability describes the possibility of comprehending the ML model and presenting the underlying basis for decision-making in a way that is understandable to humans.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0006", "text": "Explainability is a concept that is recognized as important, but a consensus definition is not yet available; one possibility is \"the collection of features of the interpretable domain that have contributed, for a given example, to producing a decision (e.g., classification or regression)\".", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0007", "text": "In summary, Interpretability refers to the user's ability to understand model outputs, while Model Transparency includes Simulatability (reproducibility of predictions), Decomposability (intuitive explanations for parameters), and Algorithmic Transparency (explaining how algorithms work). Model Functionality focuses on textual descriptions, visualization, and local explanations, which clarify specific outputs or instances rather than entire models", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0008", "text": ". All these concepts aim to enhance the comprehensibility and usability of AI systems.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0009", "text": "If algorithms fulfill these principles, they provide a basis for justifying decisions, tracking them and thereby verifying them, improving the algorithms, and exploring new facts.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0010", "text": "Sometimes it is also possible to achieve a high-accuracy result with white-box ML algorithms. These algorithms have an interpretable structure that can be used to explain predictions. Concept Bottleneck Models, which use concept-level abstractions to explain model reasoning, are examples of this and can be applied in both image and text prediction tasks", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0011", "text": ". This is especially important in domains like medicine, defense, finance, and law, where it is crucial to understand decisions and build trust in the algorithms. Many researchers argue that, at least for supervised machine learning, the way forward is symbolic regression, where the algorithm searches the space of mathematical expressions to find the model that best fits a given dataset.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0012", "text": "AI systems optimize behavior to satisfy a mathematically specified goal system chosen by the system designers, such as the command \"maximize the accuracy of assessing how positive film reviews are in the test dataset.\" The AI may learn useful general rules from the test set, such as \"reviews containing the word \"horrible\" are likely to be negative.\" However, it may also learn inappropriate rules, such as \"reviews containing 'Daniel Day-Lewis' are usually positive\"; such rules may be undesirable", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0013", "text": "usually positive\"; such rules may be undesirable if they are likely to fail to generalize outside the training set, or if people consider the rule to be \"cheating\" or \"unfair.\" A human can audit rules in an XAI to get an idea of how likely the system is to generalize to future real-world data outside the test set.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0014", "text": "Goals", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0015", "text": "Cooperation between agents \u2013 in this case, algorithms and humans \u2013 depends on trust. If humans are to accept algorithmic prescriptions, they need to trust them. Incompleteness in formal trust criteria is a barrier to optimization. Transparency, interpretability, and explainability are intermediate goals on the road to these more comprehensive trust criteria", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0016", "text": ". This is particularly relevant in medicine, especially with clinical decision support systems (CDSS), in which medical professionals should be able to understand how and why a machine-based decision was made in order to trust the decision and augment their decision-making process.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0017", "text": "AI systems sometimes learn undesirable tricks that do an optimal job of satisfying explicit pre-programmed goals on the training data but do not reflect the more nuanced implicit desires of the human system designers or the full complexity of the domain data. For example, a 2017 system tasked with image recognition learned to \"cheat\" by looking for a copyright tag that happened to be associated with horse pictures rather than learning how to tell if a horse was actually pictured", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0018", "text": ". In another 2017 system, a supervised learning AI tasked with grasping items in a virtual world learned to cheat by placing its manipulator between the object and the viewer in a way such that it falsely appeared to be grasping the object.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0019", "text": "One transparency project, the DARPA XAI program, aims to produce \"glass box\" models that are explainable to a \"human-in-the-loop\" without greatly sacrificing AI performance. Human users of such a system can understand the AI's cognition (both in real-time and after the fact) and can determine whether to trust the AI. Other applications of XAI are knowledge extraction from black-box models and model comparisons", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0020", "text": ". In the context of monitoring systems for ethical and socio-legal compliance, the term \"glass box\" is commonly used to refer to tools that track the inputs and outputs of the system in question, and provide value-based explanations for their behavior. These tools aim to ensure that the system operates in accordance with ethical and legal standards, and that its decision-making processes are transparent and accountable", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0021", "text": ". The term \"glass box\" is often used in contrast to \"black box\" systems, which lack transparency and can be more difficult to monitor and regulate.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0022", "text": "The term is also used to name a voice assistant that produces counterfactual statements as explanations.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0023", "text": "Explainability and interpretability techniques\nThere is a subtle difference between the terms explainability and interpretability in the context of AI.\n\nSome explainability techniques don't involve understanding how the model works, and may work across various AI systems. Treating the model as a black box and analyzing how marginal changes to the inputs affect the result sometimes provides a sufficient explanation.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0024", "text": "Explainability\nExplainability is useful for ensuring that AI models are not making decisions based on irrelevant or otherwise unfair criteria. For classification and regression models, several popular techniques exist:", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0025", "text": "Partial dependency plots show the marginal effect of an input feature on the predicted outcome.\nSHAP (SHapley Additive exPlanations) enables visualization of the contribution of each input feature to the output. It works by calculating Shapley values, which measure the average marginal contribution of a feature across all possible combinations of features.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0026", "text": "Feature importance estimates how important a feature is for the model. It is usually done using permutation importance, which measures the performance decrease when the feature value is randomly shuffled across all samples.\nLIME (Local Interpretable Model-Agnostic Explanations method) approximates locally a model's outputs with a simpler, interpretable model.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0027", "text": "Multitask learning provides a large number of outputs in addition to the target classification. These other outputs can help developers deduce what the network has learned.\nFor images, saliency maps highlight the parts of an image that most influenced the result.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0028", "text": "Systems that are expert or knowledge based are software systems that are made by experts. This system consists of a knowledge based encoding for the domain knowledge. This system is usually modeled as production rules, and someone uses this knowledge base which the user can question the system for knowledge. In expert systems, the language and explanations are understood with an explanation for the reasoning or a problem solving activity.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0029", "text": "However, these techniques are not very suitable for language models like generative pretrained transformers. Since these models generate language, they can provide an explanation, but which may not be reliable", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0030", "text": ". Other techniques include attention analysis (examining how the model focuses on different parts of the input), probing methods (testing what information is captured in the model's representations), causal tracing (tracing the flow of information through the model) and circuit discovery (identifying specific subnetworks responsible for certain behaviors). Explainability research in this area overlaps significantly with interpretability and alignment research.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0031", "text": "Interpretability\nScholars sometimes use the term \"mechanistic interpretability\" to refer to the process of reverse-engineering artificial neural networks to understand their internal decision-making mechanisms and components, similar to how one might analyze a complex machine or computer program.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0032", "text": "Studying the interpretability of the most advanced foundation models often involves searching for an automated way to identify \"features\" in generative pretrained transformers. In a neural network, a feature is a pattern of neuron activations that corresponds to a concept. A compute-intensive technique called \"dictionary learning\" makes it possible to identify features to some degree", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0033", "text": ". Enhancing the ability to identify and edit features is expected to significantly improve the safety of frontier AI models.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0034", "text": "For convolutional neural networks, DeepDream can generate images that strongly activate a particular neuron, providing a visual hint about what the neuron is trained to identify.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0035", "text": "Knowledge localization\nLarge language models (LLMs), such as transformer-based models (GPT), appear to use general knowledge when generating output (completing inference). This capability raises the question of how, exactly, such knowledge is stored within the model.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0036", "text": "Research has suggested that the model's MLP component (the feed-forward layers) is the main site in which knowledge is stored, encoding information through associative links that function as key\u2013value memories: each key corresponds to textual patterns in the training data, while each value induces a distribution over the output vocabulary.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0037", "text": "A 2022 study, aimed at locating where knowledge resides in the model, employed a technique known as Causal Tracing. In tasks requiring general knowledge, the researchers injected noise into the hidden activations of the model, preventing it from completing the task. They then restored the clean activation values (taken from a noise-free run) to a different part of the model each time, observing when the model regained its ability to produce the correct answer", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0038", "text": ". Based on these results, the authors concluded that factual knowledge is stored primarily in the MLP components of the model's middle layers. They further proposed that model editing would be most effective in those regions, though this claim was later called into question.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0039", "text": "Later studies suggest that, in most cases, factual information is distributed across the model rather than localized within a single layer. According to one version of this view, different layers encode different aspects of the same association. For example, a question about the capital of Japan may activate representations related to \"Japan\" in one layer and representations corresponding to \"capital cities\" in another; the combination of these representations yields the concept of \"Tokyo\".", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0040", "text": "History and methods", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0041", "text": "During the 1970s to 1990s, symbolic reasoning systems, such as MYCIN, GUIDON, SOPHIE, and PROTOS could represent, reason about, and explain their reasoning for diagnostic, instructional, or machine-learning (explanation-based learning) purposes. MYCIN, developed in the early 1970s as a research prototype for diagnosing bacteremia infections of the bloodstream, could explain which of its hand-coded rules contributed to a diagnosis in a specific case", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0042", "text": ". Research in intelligent tutoring systems resulted in developing systems such as SOPHIE that could act as an \"articulate expert\", explaining problem-solving strategy at a level the student could understand, so they would know what action to take next. For instance, SOPHIE could explain the qualitative reasoning behind its electronics troubleshooting, even though it ultimately relied on the SPICE circuit simulator", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0043", "text": ". Similarly, GUIDON added tutorial rules to supplement MYCIN's domain-level rules so it could explain the strategy for medical diagnosis. Symbolic approaches to machine learning relying on explanation-based learning, such as PROTOS, made use of explicit representations of explanations expressed in a dedicated explanation language, both to explain their actions and to acquire new knowledge.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0044", "text": "In the 1980s through the early 1990s, truth maintenance systems (TMS) extended the capabilities of causal-reasoning, rule-based, and logic-based inference systems. A TMS explicitly tracks alternate lines of reasoning, justifications for conclusions, and lines of reasoning that lead to contradictions, allowing future reasoning to avoid these dead ends", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0045", "text": ". To provide an explanation, they trace reasoning from conclusions to assumptions through rule operations or logical inferences, allowing explanations to be generated from the reasoning traces. As an example, consider a rule-based problem solver with just a few rules about Socrates that concludes he has died from poison:", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0046", "text": "By just tracing through the dependency structure the problem solver can construct the following explanation: \"Socrates died because he was mortal and drank poison, and all mortals die when they drink poison. Socrates was mortal because he was a man and all men are mortal. Socrates drank poison because he held dissident beliefs, the government was conservative, and those holding conservative dissident beliefs under conservative governments must drink poison.\"", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0047", "text": "By the 1990s researchers began studying whether it is possible to meaningfully extract the non-hand-coded rules being generated by opaque trained neural networks. Researchers in clinical expert systems who created neural network-powered decision support for clinicians sought to develop dynamic explanations that allow these technologies to be more trusted and trustworthy in practice", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0048", "text": ". In the 2010s public concerns about racial and other bias in the use of AI for criminal sentencing decisions and findings of creditworthiness may have led to increased demand for transparent artificial intelligence. As a result, many academics and organizations are developing tools to help detect bias in their systems.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0049", "text": "Marvin Minsky et al. raised the issue that AI can function as a form of surveillance, with the biases inherent in surveillance, suggesting HI (Humanistic Intelligence) as a way to create a more fair and balanced \"human-in-the-loop\" AI.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0050", "text": "Explainable AI has been recently a new topic researched amongst the context of modern deep learning. Modern complex AI techniques, such as deep learning, are naturally opaque. To address this issue, methods have been developed to make new models more explainable and interpretable", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0051", "text": ". This includes layerwise relevance propagation (LRP), a technique for determining which features in a particular input vector contribute most strongly to a neural network's output, although this technique has been shown to suffer from several important issues. Other techniques explain some particular prediction made by a (nonlinear) black-box model, a goal referred to as \"local interpretability\"", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0052", "text": ". We still today cannot explain the output of today's DNNs without the new explanatory mechanisms, we also can't by the neural network, or external explanatory components  There is also research on whether the concepts of local interpretability can be applied to a remote context, where a model is operated by a third-party.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0053", "text": "There has been work on making glass-box models which are more transparent to inspection. This includes decision trees, Bayesian networks, sparse linear models, and more. The Association for Computing Machinery Conference on Fairness, Accountability, and Transparency (ACM FAccT) was established in 2018 to study transparency and explainability in the context of socio-technical systems, many of which include artificial intelligence.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0054", "text": "Some techniques allow visualisations of the inputs to which individual software neurons respond to most strongly. Several groups found that neurons can be aggregated into circuits that perform human-comprehensible functions, some of which reliably arise across different networks trained independently.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0055", "text": "There are various techniques to extract compressed representations of the features of given inputs, which can then be analysed by standard clustering techniques. Alternatively, networks can be trained to output linguistic explanations of their behaviour, which are then directly human-interpretable", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0056", "text": ". Model behaviour can also be explained with reference to training data\u2014for example, by evaluating which training inputs influenced a given behaviour the most, or by approximating its predictions using the most similar instances from the training data.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0057", "text": "The use of explainable artificial intelligence (XAI) in pain research, specifically in understanding the role of electrodermal activity for automated pain recognition: hand-crafted features and deep learning models in pain recognition, highlighting the insights that simple hand-crafted features can yield comparative performances to deep learning models and that both traditional feature engineering and deep feature learning approaches rely on simple characteristics of the input time-series data.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0058", "text": "Regulation", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0059", "text": "As regulators, official bodies, and general users come to depend on AI-based dynamic systems, clearer accountability will be required for automated decision-making processes to ensure trust and transparency. The first global conference exclusively dedicated to this emerging discipline was the 2017 International Joint Conference on Artificial Intelligence: Workshop on Explainable Artificial Intelligence (XAI)", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0060", "text": ". It has evolved over the years, with various workshops organised and co-located to many other international conferences, and it has now a dedicated global event, \"The world conference on eXplainable Artificial Intelligence\", with its own proceedings.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0061", "text": "The European Union introduced a right to explanation in the General Data Protection Regulation (GDPR) to address potential problems stemming from the rising importance of algorithms. The implementation of the regulation began in 2018. However, the right to explanation in GDPR covers only the local aspect of interpretability. In the United States, insurance companies are required to be able to explain their rate and coverage decisions", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0062", "text": ". In France the Loi pour une R\u00e9publique num\u00e9rique (Digital Republic Act) grants subjects the right to request and receive information pertaining to the implementation of algorithms that process data about them.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0063", "text": "Limitations\nDespite ongoing endeavors to enhance the explainability of AI models, they persist with several inherent limitations.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0064", "text": "Adversarial parties\nBy making an AI system more explainable, we also reveal more of its inner workings. For example, the explainability method of feature importance identifies features or variables that are most important in determining the model's output, while the influential samples method identifies the training samples that are most influential in determining the output, given a particular input. Adversarial parties could take advantage of this knowledge.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0065", "text": "For example, competitor firms could replicate aspects of the original AI system in their own product, thus reducing competitive advantage. An explainable AI system is also susceptible to being \"gamed\"\u2014influenced in a way that undermines its intended purpose. One study gives the example of a predictive policing system; in this case, those who could potentially \"game\" the system are the criminals subject to the system's decisions", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0066", "text": ". In this study, developers of the system discussed the issue of criminal gangs looking to illegally obtain passports, and they expressed concerns that, if given an idea of what factors might trigger an alert in the passport application process, those gangs would be able to \"send guinea pigs\" to test those triggers, eventually finding a loophole that would allow them to \"reliably get passports from under the noses of the authorities\".", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0067", "text": "Adaptive integration and explanation", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0068", "text": "Many approaches that it uses provides explanation in general, it doesn't take account for the diverse backgrounds and knowledge level of the users. This leads to challenges with accurate comprehension for all users. Expert users can find the explanations lacking in depth, and are oversimplified, while a beginner user may struggle understanding the explanations as they are complex", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0069", "text": ". This limitation downplays the ability of the XAI techniques to appeal to their users with different levels of knowledge, which can impact the trust from users and who uses it. The quality of explanations can be different amongst their users as they all have different expertise levels, including different situation and conditions.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0070", "text": "Technical complexity", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0071", "text": "A fundamental barrier to making AI systems explainable is the technical complexity of such systems. End users often lack the coding knowledge required to understand software of any kind. Current methods used to explain AI are mainly technical ones, geared toward machine learning engineers for debugging purposes, rather than toward the end users who are ultimately affected by the system, causing \"a gap between explainability in practice and the goal of transparency\"", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0072", "text": ". Proposed solutions to address the issue of technical complexity include either promoting the coding education of the general public so technical explanations would be more accessible to end users, or providing explanations in layperson terms.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0073", "text": "The solution must avoid oversimplification. It is important to strike a balance between accuracy \u2013 how faithfully the explanation reflects the process of the AI system \u2013 and explainability \u2013 how well end users understand the process. This is a difficult balance to strike, since the complexity of machine learning makes it difficult for even ML engineers to fully understand, let alone non-experts.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0074", "text": "Understanding versus trust", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0075", "text": "The goal of explainability to end users of AI systems is to increase trust in the systems, even \"address concerns about lack of 'fairness' and discriminatory effects\". However, even with a good understanding of an AI system, end users may not necessarily trust the system. In one study, participants were presented with combinations of white-box and black-box explanations, and static and interactive explanations of AI systems", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0076", "text": ". While these explanations served to increase both their self-reported and objective understanding, it had no impact on their level of trust, which remained skeptical.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0077", "text": "This outcome was especially true for decisions that impacted the end user in a significant way, such as graduate school admissions. Participants judged algorithms to be too inflexible and unforgiving in comparison to human decision-makers; instead of rigidly adhering to a set of rules, humans are able to consider exceptional cases as well as appeals to their initial decision. For such decisions, explainability will not necessarily cause end users to accept the use of decision-making algorithms", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0078", "text": ". We will need to either turn to another method to increase trust and acceptance of decision-making algorithms, or question the need to rely solely on AI for such impactful decisions in the first place.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0079", "text": "However, some emphasize that the purpose of explainability of artificial intelligence is not to merely increase users' trust in the system's decisions, but to calibrate the users' level of trust to the correct level. According to this principle, too much or too little user trust in the AI system will harm the overall performance of the human-system unit", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0080", "text": ". When the trust is excessive, the users are not critical of possible mistakes of the system and when the users do not have enough trust in the system, they will not exhaust the benefits inherent in it.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0081", "text": "Criticism\nSome scholars have suggested that explainability in AI should be considered a goal secondary to AI effectiveness, and that encouraging the exclusive development of XAI may limit the functionality of AI more broadly. Critiques of XAI rely on developed concepts of mechanistic and empiric reasoning from evidence-based medicine to suggest that AI technologies can be clinically validated even when their function cannot be understood by their operators.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0082", "text": "Some researchers advocate the use of inherently interpretable machine learning models, rather than using post-hoc explanations in which a second model is created to explain the first. This is partly because post-hoc models increase the complexity in a decision pathway and partly because it is often unclear how faithfully a post-hoc explanation can mimic the computations of an entirely separate model", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0083", "text": ". However, another view is that what is important is that the explanation accomplishes the given task at hand, and whether it is pre or post-hoc doesn't matter. If a post-hoc explanation method helps a doctor diagnose cancer better, it is of secondary importance whether it is a correct/incorrect explanation.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0084", "text": "The goals of XAI amount to a form of lossy compression that will become less effective as AI models grow in their number of parameters. Along with other factors this leads to a theoretical limit for explainability.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0085", "text": "Explainability in social choice\nExplainability was studied also in social choice theory. Social choice theory aims at finding solutions to social decision problems, that are based on well-established axioms. Ariel D. Procaccia explains that these axioms can be used to construct convincing explanations to the solutions. This principle has been used to construct explanations in various subfields of social choice.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0086", "text": "Voting\nCailloux and Endriss present a method for explaining voting rules using the axioms that characterize them. They exemplify their method on the Borda voting rule.\nPeters, Procaccia, Psomas and Zhou present an algorithm for explaining the outcomes of the Borda rule using O(m2) explanations, and prove that this is tight in the worst case.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0087", "text": "Participatory budgeting", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0088", "text": "Yang, Hausladen, Peters, Pournaras, Fricker and Helbing present an empirical study of explainability in participatory budgeting", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0089", "text": ". They compared the greedy and the equal shares rules, and three types of explanations: mechanism explanation (a general explanation of how the aggregation rule works given the voting input), individual explanation (explaining how many voters had at least one approved project, at least 10000 CHF in approved projects), and group explanation (explaining how the budget is distributed among the districts and topics)", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0090", "text": ". They compared the perceived trustworthiness and fairness of greedy and equal shares, before and after the explanations. They found out that, for MES, mechanism explanation yields the highest increase in perceived fairness and trustworthiness; the second-highest was Group explanation. For Greedy, Mechanism explanation increases perceived trustworthiness but not fairness, whereas Individual explanation increases both perceived fairness and trustworthiness", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0091", "text": ". Group explanation decreases the perceived fairness and trustworthiness.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0092", "text": "Payoff allocation", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0093", "text": "Nizri, Azaria and Hazon present an algorithm for computing explanations for the Shapley value. Given a coalitional game, their algorithm decomposes it to sub-games, for which it is easy to generate verbal explanations based on the axioms characterizing the Shapley value. The payoff allocation for each sub-game is perceived as fair, so the Shapley-based payoff allocation for the given game should seem fair as well", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0094", "text": ". An experiment with 210 human subjects shows that, with their automatically generated explanations, subjects perceive Shapley-based payoff allocation as significantly fairer than with a general standard explanation.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0095", "text": "See also\nAccumulated local effects \u2013 Machine learning method\nAlgorithmic transparency \u2013 Transparency of decisions by algorithms\nRight to explanation \u2013 Right to have an algorithm explained", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0096", "text": "References\nExternal links\n\"Leaders today face a new paradox: deploying powerful AI solutions\". 31 October 2025.\n\"the World Conference on eXplainable Artificial Intelligence\".\n\"ACM Conference on Fairness, Accountability, and Transparency (FAccT)\".\nMazumdar, Dipankar; Neto, M\u00e1rio Popolin; Paulovich, Fernando V. (2021). \"Random Forest similarity maps: A Scalable Visual Representation for Global and Local Interpretation\". Electronics. 10 (22): 2862. doi:10.3390/electronics10222862.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Explainable_artificial_intelligence_0097", "text": "\"Explainable AI: Making machines understandable for humans\". Explainable AI: Making machines understandable for humans. Retrieved 2017-11-02.\n\"Explaining How End-to-End Deep Learning Steers a Self-Driving Car\". Parallel Forall. 2017-05-23. Retrieved 2017-11-02.\nKnight, Will (2017-03-14). \"DARPA is funding projects that will try to open up AI's black boxes\". MIT Technology Review. Retrieved 2017-11-02.", "title": "Explainable artificial intelligence", "url": "https://en.wikipedia.org/wiki/Explainable_artificial_intelligence"}, {"id": "rec_Prompt_engineering_0000", "text": "Prompt engineering is the process of structuring natural language inputs (known as prompts) to produce specified outputs from a generative artificial intelligence (GenAI) model. Context engineering is the related area of software engineering that focuses on the management of non-prompt contexts supplied to the GenAI model, such as metadata, API tools, and tokens.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0001", "text": "It can also be defined as the practice of designing and refining input instructions given to a generative AI model in order to produce more accurate, relevant, or useful outputs. Effective prompt engineering involves understanding how a model interprets language, and may include techniques such as few-shot prompting, chain-of-thought prompting, and role assignment. It is increasingly considered a key skill for working with large language models (LLMs) in both research and professional contexts", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0002", "text": ".", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0003", "text": "During the 2020s AI boom, prompt engineering became regarded as an important business capability across corporations and industries. Employees with the title prompt engineer were hired to create prompts that would increase productivity and efficacy, although the individual title has since lost traction in light of AI models that produce better prompts than humans and corporate training in prompting for general employees.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0004", "text": "Common prompting techniques include multi-shot, chain-of-thought, and tree-of-thought prompting, as well as the use of assigning roles to the model. Automated prompt generation methods, such as retrieval-augmented generation (RAG), provide for greater accuracy and a wider scope of functions for prompt engineers. Prompt injection is a type of cybersecurity attack that targets machine learning models through malicious prompts.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0005", "text": "Terminology", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0006", "text": "The Oxford English Dictionary defines prompt engineering as \"The action or process of formulating and refining prompts for an artificial intelligence program, algorithm, etc., in order to optimize its output or to achieve a desired outcome; the discipline or profession concerned with this.\" In 2023, prompt (\"an instruction given to an artificial intelligence program, algorithm, etc., which determines or influences the content it generates\") was the runner-up to Oxford's word of the year.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0007", "text": "Prompt", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0008", "text": "A prompt is some natural language text that describes and prescribes the task that an artificial intelligence (AI) should perform. A prompt for a text-to-text language model can be a query, a command, or a longer statement referencing context, instructions, and conversation history", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0009", "text": ". The process of prompt engineering may involve designing clear queries, refining wording, providing relevant context, specifying the style of output, and assigning a character for the AI to mimic in order to guide the model toward more accurate, useful, and consistent responses.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0010", "text": "When communicating with a text-to-image or a text-to-audio model, a typical prompt contains a description of a desired output such as \"a high-quality photo of an astronaut riding a horse\" or \"Lo-fi slow BPM electro chill with organic samples\". Prompt engineering may be applied to text-to-image models to achieve a desired subject, style, layout, lighting, and aesthetic.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0011", "text": "Techniques\nCommon terms used to describe various specific prompt engineering techniques include chain-of-thought, tree-of-thought, and retrieval-augmented generation (RAG). A 2024 survey of the field identified over 50 distinct text-based prompting techniques, 40 multimodal variants, and a vocabulary of 33 terms used across prompting research, highlighting a present lack of standardised terminology for prompt engineering.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0012", "text": "Vibe coding is an AI-assisted software development method where a user prompts an LLM with a description of what they want and lets it generate or edit the code. In 2025, \"vibe coding\" was the Collins Dictionary word of the year.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0013", "text": "Context engineering", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0014", "text": "Context engineering is a related process that focuses on the context elements that accompany user prompts, which include system instructions, retrieved knowledge, tool definitions, conversation summaries, and task metadata. Context engineering is performed to improve reliability, provenance and token efficiency in production LLM systems", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0015", "text": ". The concept emphasises operational practices such as token budgeting, provenance tags, versioning of context artifacts, observability (logging which context was supplied), and context regression tests to ensure that changes to supplied context do not silently alter system behaviour.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0016", "text": "Rationale\nResearch has found that the performance of large language models (LLMs) is highly sensitive to choices such as the ordering of examples, the quality of demonstration labels, and even small variations in phrasing. In some cases, reordering examples in a prompt produced accuracy shifts of more than 40 percent.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0017", "text": "In-context learning", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0018", "text": "A model's ability to temporarily learn from prompts is known as in-context learning. In-context learning is an emergent ability of large language models. It is an emergent property of model scale, meaning that breaks in scaling laws occur, leading to its efficacy increasing at a different rate in larger models than in smaller models. Unlike training and fine-tuning, which produce lasting changes, in-context learning is temporary", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0019", "text": ". Training models to perform in-context learning can be viewed as a form of meta-learning, or \"learning to learn\".", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0020", "text": "Prompting to estimate model sensitivity", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0021", "text": "Research consistently demonstrates that LLMs are highly sensitive to subtle variations in prompt formatting, structure, and linguistic properties. Some studies have shown up to 76 accuracy points across formatting changes in few-shot settings. Linguistic features significantly influence prompt effectiveness\u2014such as morphology, syntax, and lexico-semantic changes\u2014which meaningfully enhance task performance across a variety of tasks", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0022", "text": ". Clausal syntax, for example, improves consistency and reduces uncertainty in knowledge retrieval. This sensitivity persists even with larger model sizes, additional few-shot examples, or instruction tuning.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0023", "text": "To address sensitivity of models and make them more robust, several evaluative methods have been proposed. FormatSpread facilitates systematic analysis by evaluating a range of plausible prompt formats, offering a more comprehensive performance interval. Similarly, PromptEval estimates performance distributions across diverse prompts, enabling robust metrics such as performance quantiles and accurate evaluations under constrained budgets.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0024", "text": "Prompting techniques\nMulti-shot\nA prompt may include a few examples for a model to learn from in context, an approach called few-shot learning. For example, the prompt may ask the model to complete \"maison \u2192 house, chat \u2192 cat, chien \u2192\", with the expected response being dog.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0025", "text": "Chain-of-thought", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0026", "text": "Chain-of-thought (CoT) prompting is a technique that allows large language models (LLMs) to solve a problem as a series of intermediate steps before giving a final answer. In 2022, Google Brain reported that chain-of-thought prompting improves reasoning ability by inducing the model to answer a multi-step problem with steps of reasoning that mimic a train of thought", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0027", "text": ". Chain-of-thought techniques were developed to help LLMs handle multi-step reasoning tasks, such as arithmetic or commonsense reasoning questions.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0028", "text": "When applied to PaLM, a 540 billion parameter language model, according to Google, CoT prompting significantly aided the model, allowing it to perform comparably with task-specific fine-tuned models on several tasks, achieving state-of-the-art results at the time on the GSM8K mathematical reasoning benchmark. It is possible to fine-tune models on CoT reasoning datasets to enhance this capability further and stimulate better interpretability.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0029", "text": "As originally proposed by Google, each CoT prompt is accompanied by a set of input/output examples\u2014called exemplars\u2014to demonstrate the desired model output, making it a few-shot prompting technique. However, according to a later paper from researchers at Google and the University of Tokyo, simply appending the words \"Let's think step-by-step\" was also effective, which allowed for CoT to be employed as a zero-shot technique.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0030", "text": "Self-consistency\nSelf-consistency performs several chain-of-thought rollouts, then selects the most commonly reached conclusion out of all the rollouts.\n\nTree-of-thought\nTree-of-thought prompting generalizes chain-of-thought by generating multiple lines of reasoning in parallel, with the ability to backtrack or explore other paths. It can use tree search algorithms like breadth-first, depth-first, or beam.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0031", "text": "Text-to-image prompting\nIn 2022, text-to-image models like DALL-E 2, Stable Diffusion, and Midjourney were released to the public. These models take text prompts as input and use them to generate images. Early text-to-image models typically do not understand negation, grammar and sentence structure in the same way as large language models, and may thus require a different set of prompting techniques. The prompt \"a party with no cake\" may produce an image including a cake.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0032", "text": "A text-to-image prompt commonly includes a description of the subject of the art, the desired medium (such as digital painting or photography), style (such as hyperrealistic or pop-art), lighting (such as rim lighting or crepuscular rays), color, and texture. Word order also affects the output of a text-to-image prompt. Words closer to the start of a prompt may be emphasized more heavily.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0033", "text": "Artist styles\nSome text-to-image models are capable of imitating the style of particular artists by name. For example, the phrase in the style of Greg Rutkowski has been used in Stable Diffusion and Midjourney prompts to generate images in the distinctive style of Polish digital artist Greg Rutkowski. Famous artists such as Vincent van Gogh and Salvador Dal\u00ed have also been used for styling and testing.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0034", "text": "Textual inversion and embeddings\nFor text-to-image models, textual inversion performs an optimization process to create a new word embedding based on a set of example images. This embedding vector acts as a \"pseudo-word\" which can be included in a prompt to express the content or style of the examples.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0035", "text": "Image prompting\nIn 2023, Meta's AI research released Segment Anything, a computer vision model that can perform image segmentation by prompting. As an alternative to text prompts, Segment Anything can accept bounding boxes, segmentation masks, and foreground/background points.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0036", "text": "Limitations\nThe process of writing and refining a prompt for an LLM or generative AI shares some parallels with an iterative engineering design process, such as by discovering reusable best practices through reproducible experimentation. But the techniques that improve performance depend heavily on the specific model being used. Such patterns are also volatile and exhibit significantly different results from seemingly insignificant prompt changes.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0037", "text": "Automated prompt generation\nRecent research has explored automated prompt engineering, using optimization algorithms to generate or refine prompts without human intervention. These automated approaches aim to identify effective prompt patterns by analyzing model gradients, reinforcement feedback, or evolutionary processes, reducing the need for manual experimentation.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0038", "text": "Retrieval-augmented generation (RAG)\nRetrieval-augmented generation is a technique that enables GenAI models to retrieve and incorporate new information. It modifies interactions with an LLM so that the model responds to user queries with reference to a specified set of documents, using this information to supplement information from its pre-existing training data. This allows LLMs to use domain-specific and/or updated information.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0039", "text": "RAG improves large language models by incorporating information retrieval before generating responses. Unlike traditional LLMs that rely on static training data, RAG pulls relevant text from databases, uploaded documents, or web sources. By dynamically retrieving information, RAG enables AI to generate more accurate responses and fewer AI hallucinations without frequent retraining.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0040", "text": "Graph retrieval-augmented generation (GraphRAG)\nGraphRAG (coined by Microsoft Research) is a technique that extends RAG with the use of a knowledge graph to allow the model to connect disparate pieces of information, synthesize insights, and understand summarized semantic concepts over large data collections. It was shown to be effective on datasets like the Violent Incident Information from News Articles.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0041", "text": "Using language models to generate prompts\nLLMs themselves can be used to compose prompts for LLMs. The automatic prompt engineer algorithm uses one LLM to beam search over prompts for another LLM:", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0042", "text": "There are two LLMs. One is the target LLM, and another is the prompting LLM.\nPrompting LLM is presented with example input-output pairs, and asked to generate instructions that could have caused a model following the instructions to generate the outputs, given the inputs.\nEach of the generated instructions is used to prompt the target LLM, followed by each of the inputs. The log-probabilities of the outputs are computed and added. This is the score of the instruction.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0043", "text": "The highest-scored instructions are given to the prompting LLM for further variations.\nRepeat until some stopping criteria is reached, then output the highest-scored instructions.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0044", "text": "CoT examples can be generated by LLM themselves. In \"auto-CoT\", a library of questions are converted to vectors by a model such as BERT. The question vectors are clustered. Questions close to the centroid of each cluster are selected, in order to have a subset of diverse questions. An LLM does zero-shot CoT on each selected question. The question and the corresponding CoT answer are added to a dataset of demonstrations", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0045", "text": ". These diverse demonstrations can then added to prompts for few-shot learning.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0046", "text": "Automatic prompt optimization\nAutomatic prompt optimization techniques refine prompts for large language models by automatically searching over alternative prompt strings using evaluation datasets and task-specific metrics:", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0047", "text": "MIPRO (Multi-prompt Instruction Proposal Optimizer) optimizes the instructions and few-shot demonstrations of multi-stage language model programs, proposing small changes to module prompts and retaining those that improve a downstream performance metric without access to module-level labels or gradients.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0048", "text": "GEPA (Genetic-Pareto) is a reflective prompt optimizer for compound AI systems that combines language-model-based analysis of execution traces and textual feedback with a Pareto-based evolutionary search over a population of candidate systems; across four tasks, GEPA reports average gains of about 10% over reinforcement-learning-based Group Relative Policy Optimization (GRPO) and over 10% over the MIPROv2 prompt optimizer, while using up to 35 times fewer rollouts than GRPO.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0049", "text": "Open-source frameworks such as DSPy and Opik expose these and related optimizers, allowing prompt search to be expressed as part of a programmatic pipeline rather than through manual trial and error.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0050", "text": "Using gradient descent to search for prompts\nIn \"prefix-tuning\", \"prompt tuning\", or \"soft prompting\", floating-point vectors are searched directly by gradient descent to maximize the log-likelihood on outputs. An earlier result uses the same idea of gradient descent search, but is designed for masked language models like BERT, and searches only over token sequences, rather than numerical vectors. Formally, it searches for \n  \n    \n      \n        arg\n        \u2061\n        \n          max", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0051", "text": "\u2061\n        \n          max\n          \n            \n              \n                X\n                ~\n              \n            \n          \n        \n        \n          \u2211\n          \n            i\n          \n        \n        log\n        \u2061\n        P\n        r\n        [\n        \n          Y\n          \n            i\n          \n        \n        \n          |\n        \n        \n          \n            \n              X\n              ~\n            \n          \n        \n        \u2217\n        \n          X", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0052", "text": "\u2217\n        \n          X\n          \n            i\n          \n        \n        ]\n      \n    \n    {\\displaystyle \\arg \\max _{\\tilde {X}}\\sum _{i}\\log Pr[Y^{i}|{\\tilde {X}}\\ast X^{i}]}\n  \n where \n  \n    \n      \n        \n          \n            \n              X\n              ~\n            \n          \n        \n      \n    \n    {\\displaystyle {\\tilde {X}}}\n  \n ranges over token sequences of a specified length.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0053", "text": "History", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0054", "text": "Early precedents of structured user interaction with ruled-based AI systems can be found in enterprise automation software from 1990s. For example, The Intelligent Filling Manager (1999), developed by Krishna C. Mukherjee, used a dynamic Q&A interface driven by rule-based expert system to collect user inputs for generating regulatory filings automatically across jurisdictions", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0055", "text": ". While not involving neural networks, such systems featured prompt-like workflows that influenced later human-in-the-loop AI designs. In 2018, researchers first proposed that all previously separate tasks in natural language processing (NLP) could be cast as question-answer problems over a context", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0056", "text": ". In addition, they trained a first single, joint, multi-task model that would answer any task-related question like \"What is the sentiment\" or \"Translate this sentence to German\" or \"Who is the president?\"", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0057", "text": "The AI boom saw an increased focus within academic literature and professional practice on applying prompting techniques to get the model to output the desired outcome and avoid nonsensical output, a process characterized by trial-and-error", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0058", "text": ". After the release of ChatGPT in 2022, prompt engineering was soon seen as an important business skill; companies began hiring dedicated prompt engineers, although, given advances in AI's ability to generate prompts better than humans, the employment market for prompt engineers has faced uncertainty", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0059", "text": ". According to The Wall Street Journal in 2025, the job of prompt engineer was one of the hottest in 2023, but has become obsolete due to models that better intuit user intent and to company trainings.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0060", "text": "A repository for prompts reported that over 2,000 public prompts for around 170 datasets were available in February 2022. In 2022, the chain-of-thought prompting technique was proposed by Google researchers. In 2023, several text-to-text and text-to-image prompt databases were made publicly available. The Personalized Image-Prompt (PIP) dataset, a generated image-text dataset that has been categorized by 3,115 users, has also been made available publicly in 2024.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0061", "text": "Prompt injection", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0062", "text": "Prompt injection is a cybersecurity exploit in which adversaries craft inputs that appear legitimate but are designed to cause unintended behavior in machine learning models, particularly large language models. This attack takes advantage of the model's inability to distinguish between developer-defined prompts and user inputs, allowing adversaries to bypass safeguards and influence model behaviour", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0063", "text": ". While LLMs are designed to follow trusted instructions, they can be manipulated into carrying out unintended responses through carefully crafted inputs.", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Prompt_engineering_0064", "text": "== References ==", "title": "Prompt engineering", "url": "https://en.wikipedia.org/wiki/Prompt_engineering"}, {"id": "rec_Finetuning_deep_learning_0000", "text": "Fine-tuning (in deep learning) is the process of adapting a model trained for one task (the upstream task) to perform a different, usually more specific, task (the downstream task). It is considered a form of transfer learning, as it reuses knowledge learned from the original training objective.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0001", "text": "Fine-tuning involves applying additional training (e.g., on new data) to the parameters of a neural network that have been pre-trained. Many variants exist.  The additional training can be applied to the entire neural network, or to only a subset of its layers, in which case the layers that are not being fine-tuned are \"frozen\" (i.e., not changed during backpropagation)", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0002", "text": ". A model may also be augmented with \"adapters\"\u2014lightweight modules inserted into the model's architecture that nudge the embedding space for domain adaptation. These contain far fewer parameters than the original model and can be fine-tuned in a parameter-efficient way by tuning only their weights and leaving the rest of the model's weights frozen.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0003", "text": "For some architectures, such as convolutional neural networks, it is common to keep the earlier layers (those closest to the input layer) frozen, as they capture lower-level features, while later layers often discern high-level features that can be more related to the task that the model is trained on.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0004", "text": "Models that are pre-trained on large, general corpora are usually fine-tuned by reusing their parameters as a starting point and adding a task-specific layer trained from scratch. Fine-tuning the full model is also common and often yields better results, but is more computationally expensive.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0005", "text": "Fine-tuning is typically accomplished via supervised learning, but there are also techniques to fine-tune a model using weak supervision. Fine-tuning can be combined with a reinforcement learning from human feedback-based objective to produce language models such as ChatGPT (a fine-tuned version of GPT models) and Sparrow.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0006", "text": "Robustness\nFine-tuning can degrade a model's robustness to distribution shifts. One mitigation is to linearly interpolate a fine-tuned model's weights with the weights of the original model, which can greatly increase out-of-distribution performance while largely retaining the in-distribution performance of the fine-tuned model.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0007", "text": "Variants\nLow-rank adaptation", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0008", "text": "Low-rank adaptation (LoRA) is an adapter-based technique for efficiently fine-tuning models. The basic idea is to design a low-rank matrix that is then added to the original matrix. An adapter, in this context, is a collection of low-rank matrices which, when added to a base model, produces a fine-tuned model. It allows for performance that approaches full-model fine-tuning with lower space requirements", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0009", "text": ". A language model with billions of parameters may be LoRA fine-tuned with only several millions of parameters.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0010", "text": "LoRA-based fine-tuning has become popular in the Stable Diffusion community. Support for LoRA was integrated into the diffusers library from Hugging Face. Support for LoRA and similar techniques is also available for a wide range of other models through Hugging Face's parameter-efficient fine-tuning (PEFT) package.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0011", "text": "Representation fine-tuning\nRepresentation fine-tuning (ReFT) is a technique developed by researchers at Stanford University aimed at fine-tuning large language models (LLMs) by modifying less than 1% of their representations. Unlike parameter-efficient fine-tuning (PEFT) methods, which mainly focus on updating weights, ReFT targets representations, suggesting that modifying representations might be a more effective strategy than updating weights.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0012", "text": "ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations and train interventions that manipulate a small fraction of model representations to steer model behaviors towards solving downstream tasks at inference time. One specific method within the ReFT family is low-rank linear subspace ReFT (LoReFT), which intervenes on hidden representations in the linear subspace spanned by a low-rank projection matrix", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0013", "text": ". LoReFT can be seen as the representation-based equivalent of low-rank adaptation (LoRA).", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0014", "text": "Applications\nNatural language processing\nFine-tuning is common in natural language processing (NLP), especially in the domain of language modeling. Large language models like OpenAI's series of GPT foundation models can be fine-tuned on data for specific downstream NLP tasks (tasks that use a pre-trained model) to improve performance over the unmodified pre-trained model.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0015", "text": "Platforms such as Semrush's AI Visibility Toolkit and Enterprise AIO exemplify how fine-tuned models are being used for entity-level monitoring; tracking how named entities are referenced and represented within responses generated by large-language-model-based answer engines.", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0016", "text": "Commercial models\nCommercially-offered large language models can sometimes be fine-tuned if the provider offers a fine-tuning API. As of June 19, 2023, language model fine-tuning APIs are offered by OpenAI and Microsoft Azure's Azure OpenAI Service for a subset of their models, as well as by Google Cloud Platform for some of their PaLM models, and by others.\n\nSee also\nCatastrophic forgetting\nContinual learning\nDomain adaptation\nFoundation model\nHyperparameter optimization\nOverfitting", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Finetuning_deep_learning_0017", "text": "== References ==", "title": "Fine-tuning (deep learning)", "url": "https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)"}, {"id": "rec_Feature_machine_learning_0000", "text": "In machine learning and pattern recognition, a feature is an individual measurable property or characteristic of a data set. Choosing informative, discriminating, and independent features is crucial to producing effective algorithms for pattern recognition, classification, and regression tasks. Features are usually numeric, but other types such as strings and graphs are used in syntactic pattern recognition, after some pre-processing step such as one-hot encoding", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0001", "text": ". The concept of \"features\" is related to that of explanatory variables used in statistical techniques such as linear regression.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0002", "text": "Feature types\nIn feature engineering, two types of features are commonly used: numerical and categorical.\nNumerical features are continuous values that can be measured on a scale. Examples of numerical features include age, height, weight, and income. Numerical features can be used in machine learning algorithms directly.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0003", "text": "Categorical features are discrete values that can be grouped into categories. Examples of categorical features include gender, color, and zip code. Categorical features typically need to be converted to numerical features before they can be used in machine learning algorithms. This can be done using a variety of techniques, such as one-hot encoding, label encoding, and ordinal encoding.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0004", "text": "The type of feature that is used in feature engineering depends on the specific machine learning algorithm that is being used. Some machine learning algorithms, such as decision trees, can handle both numerical and categorical features. Other machine learning algorithms, such as linear regression, can only handle numerical features.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0005", "text": "Classification\nA numeric feature can be conveniently described by a feature vector. One way to achieve binary classification is using a linear predictor function (related to the perceptron) with a feature vector as input. The method consists of calculating the scalar product between the feature vector and a vector of weights, qualifying those observations whose result exceeds a threshold.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0006", "text": "Algorithms for classification from a feature vector include nearest neighbor classification, neural networks, and statistical techniques such as Bayesian approaches.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0007", "text": "Examples\nIn character recognition, features may include histograms counting the number of black pixels along horizontal and vertical directions, number of internal holes, stroke detection and many others.\nIn speech recognition, features for recognizing phonemes can include noise ratios, length of sounds, relative power, filter matches and many others.\nIn spam detection algorithms, features may include the presence or absence of certain email headers,", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0008", "text": "the email structure, the language, the frequency of specific terms, the grammatical correctness of the text.\nIn computer vision, there are a large number of possible features, such as edges and objects.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0009", "text": "Feature vectors", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0010", "text": "In pattern recognition and machine learning, a feature vector is an n-dimensional vector of numerical features that represent some object. Many algorithms in machine learning require a numerical representation of objects, since such representations facilitate processing and statistical analysis. When representing images, the feature values might correspond to the pixels of an image, while when representing texts the features might be the frequencies of occurrence of textual terms", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0011", "text": ". Feature vectors are equivalent to the vectors of explanatory variables used in statistical procedures such as linear regression.  Feature vectors are often combined with weights using a dot product in order to construct a linear predictor function that is used to determine a score for making a prediction.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0012", "text": "The vector space associated with these vectors is often called the feature space. In order to reduce the dimensionality of the feature space, a number of dimensionality reduction techniques can be employed.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0013", "text": "Higher-level features can be obtained from already available features and added to the feature vector; for example, for the study of diseases the feature 'Age' is useful and is defined as Age = 'Year of death' minus 'Year of birth' . This process is referred to as feature construction. Feature construction is the application of a set of constructive operators to a set of existing features resulting in construction of new features", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0014", "text": ". Examples of such constructive operators include checking for the equality conditions {=, \u2260}, the arithmetic operators {+,\u2212,\u00d7, /}, the array operators {max(S), min(S), average(S)} as well as other more sophisticated operators, for example count(S, C) that counts the number of features in the feature vector S satisfying some condition C or, for example, distances to other recognition classes generalized by some accepting device", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0015", "text": ". Feature construction has long been considered a powerful tool for increasing both accuracy and understanding of structure, particularly in high-dimensional problems. Applications include studies of disease and emotion recognition from speech.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0016", "text": "Selection and extraction\nThe initial set of raw features can be redundant and large enough that estimation and optimization is made difficult or ineffective. Therefore, a preliminary step in many applications of machine learning and pattern recognition consists of selecting a subset of features, or constructing a new and reduced set of features to facilitate learning, and to improve generalization and interpretability.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0017", "text": "Extracting or selecting features is a combination of art and science; developing systems to do so is known as feature engineering. It requires the experimentation of multiple possibilities and the combination of automated techniques with the intuition and knowledge of the domain expert. Automating this process is feature learning, where a machine not only uses features for learning, but learns the features itself.", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_machine_learning_0018", "text": "See also\nCovariate\nDimensionality reduction\nFeature engineering\nHashing trick\nStatistical classification\nExplainable artificial intelligence\n\n\n== References ==", "title": "Feature (machine learning)", "url": "https://en.wikipedia.org/wiki/Feature_(machine_learning)"}, {"id": "rec_Feature_engineering_0000", "text": "Feature engineering is a preprocessing step in supervised machine learning and statistical modeling which transforms raw data into a more effective set of inputs. Each input comprises several attributes, known as features. By providing models with relevant information, feature engineering significantly enhances their predictive accuracy and decision-making capability.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0001", "text": "Beyond machine learning, the principles of feature engineering are applied in various scientific fields, including physics. For example, physicists construct dimensionless numbers such as the Reynolds number in fluid dynamics, the Nusselt number in heat transfer, and the Archimedes number in sedimentation. They also develop first approximations of solutions, such as analytical solutions for the strength of materials in mechanics.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0002", "text": "Clustering", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0003", "text": "One of the applications of feature engineering has been clustering of feature-objects or sample-objects in a dataset. Especially, feature engineering based on matrix decomposition has been extensively used for data clustering under non-negativity constraints on the feature coefficients. These include Non-Negative Matrix Factorization (NMF), Non-Negative Matrix-Tri Factorization (NMTF), Non-Negative Tensor Decomposition/Factorization (NTF/NTD), etc", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0004", "text": ". The non-negativity constraints on coefficients of the feature vectors mined by the above-stated algorithms yields a part-based representation, and different factor matrices exhibit natural clustering properties. Several extensions of the above-stated feature engineering methods have been reported in literature, including orthogonality-constrained factorization for hard clustering, and manifold learning to overcome inherent issues with these algorithms.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0005", "text": "Other classes of feature engineering algorithms include leveraging a common hidden structure across multiple inter-related datasets to obtain a consensus (common) clustering scheme. An example is Multi-view Classification based on Consensus Matrix Decomposition (MCMD), which mines a common clustering scheme across multiple datasets. MCMD is designed to output two types of class labels (scale-variant and scale-invariant clustering), and:", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0006", "text": "is computationally robust to missing information,\ncan obtain shape- and scale-based outliers,\nand can handle high-dimensional data effectively.\nCoupled matrix and tensor decompositions are popular in multi-view feature engineering.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0007", "text": "Predictive modelling", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0008", "text": "Feature engineering in machine learning and statistical modeling involves selecting, creating, transforming, and extracting data features", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0009", "text": ". Key components include feature creation from existing data, transforming and imputing missing or invalid features, reducing data dimensionality through methods like Principal Components Analysis (PCA), Independent Component Analysis (ICA), and Linear Discriminant Analysis (LDA), and selecting the most relevant features for model training based on importance scores and correlation matrices.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0010", "text": "Features vary in significance. Even relatively insignificant features may contribute to a model. Feature selection can reduce the number of features to prevent a model from becoming too specific to the training data set (overfitting).\nFeature explosion occurs when the number of identified features is too large for effective model estimation or optimization. Common causes include:", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0011", "text": "Feature templates - implementing feature templates instead of coding new features\nFeature combinations - combinations that cannot be represented by a linear system\nFeature explosion can be limited via techniques such as regularization, kernel methods, and feature selection.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0012", "text": "Automation\nAutomation of feature engineering is a research topic that dates back to the 1990s. Machine learning software that incorporates automated feature engineering has been commercially available since 2016. Related academic literature can be roughly separated into two types:\n\nMulti-relational Decision Tree Learning (MRDTL) uses a supervised algorithm that is similar to a decision tree.\nDeep Feature Synthesis uses simpler methods.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0013", "text": "Multi-relational Decision Tree Learning (MRDTL)\nMulti-relational Decision Tree Learning (MRDTL) extends traditional decision tree methods to relational databases, handling complex data relationships across tables. It innovatively uses selection graphs as decision nodes, refined systematically until a specific termination criterion is reached.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0014", "text": "Most MRDTL studies base implementations on relational databases, which results in many redundant operations. These redundancies can be reduced by using techniques such as tuple id propagation.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0015", "text": "Open-source implementations\nThere are a number of open-source libraries and tools that automate feature engineering on relational data and time series:", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0016", "text": "featuretools is a Python library for transforming time series and relational data into feature matrices for machine learning.\nMCMD: An open-source feature engineering algorithm for joint clustering of multiple datasets.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0017", "text": "OneBM or One-Button Machine combines feature transformations and feature selection on relational data with feature selection techniques. OneBM helps data scientists reduce data exploration time allowing them to try and error many ideas in short time. On the other hand, it enables non-experts, who are not familiar with data science, to quickly extract value from their data with a little effort, time, and cost.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0018", "text": "getML community is an open source tool for automated feature engineering on time series and relational data. It is implemented in C/C++ with a Python interface. It has been shown to be at least 60 times faster than tsflex, tsfresh, tsfel, featuretools or kats.\ntsfresh is a Python library for feature extraction on time series data. It evaluates the quality of the features using hypothesis testing.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0019", "text": "tsflex is an open source Python library for extracting features from time series data. Despite being 100% written in Python, it has been shown to be faster and more memory efficient than tsfresh, seglearn or tsfel.\nseglearn is an extension for multivariate, sequential time series data to the scikit-learn Python library.\ntsfel is a Python package for feature extraction on time series data.\nkats is a Python toolkit for analyzing time series data.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0020", "text": "Deep feature synthesis\nThe deep feature synthesis (DFS) algorithm beat 615 of 906 human teams in a competition.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0021", "text": "Feature stores", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0022", "text": "The feature store is where the features are stored and organized for the explicit purpose of being used to either train models (by data scientists) or make predictions (by applications that have a trained model)", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0023", "text": ". It is a central location where you can either create or update groups of features created from multiple different data sources, or create and update new datasets from those feature groups for training models or for use in applications that do not want to compute the features but just retrieve them when it needs them to make predictions.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0024", "text": "A feature store includes the ability to store code used to generate features, apply the code to raw data, and serve those features to models upon request. Useful capabilities include feature versioning and policies governing the circumstances under which features can be used.\nFeature stores can be standalone software tools or built into machine learning platforms.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0025", "text": "Alternatives", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0026", "text": "Feature engineering can be a time-consuming and error-prone process, as it requires domain expertise and often involves trial and error. Deep learning algorithms may be used to process a large raw dataset without having to resort to feature engineering. However, deep learning algorithms still require careful preprocessing and cleaning of the input data", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0027", "text": ". In addition, choosing the right architecture, hyperparameters, and optimization algorithm for a deep neural network can be a challenging and iterative process.", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Feature_engineering_0028", "text": "See also\nCovariate\nData transformation\nFeature extraction\nFeature learning\nHashing trick\nInstrumental variables estimation\nKernel method\nList of datasets for machine learning research\nScale co-occurrence matrix\nSpace mapping\n\nReferences\n\n\n== Further reading ==", "title": "Feature engineering", "url": "https://en.wikipedia.org/wiki/Feature_engineering"}, {"id": "rec_Bayesian_network_0000", "text": "A Bayesian network (also known as a Bayes network, Bayes net, belief network, or decision network) is a probabilistic graphical model that represents a set of variables and their conditional dependencies via a directed acyclic graph (DAG). While it is one of several forms of causal notation, causal networks are special cases of Bayesian networks", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0001", "text": ". Bayesian networks are ideal for taking an event that occurred and predicting the likelihood that any one of several possible known causes was the contributing factor.  For example, a Bayesian network could represent the probabilistic relationships between diseases and symptoms. Given symptoms, the network can be used to compute the probabilities of the presence of various diseases.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0002", "text": "Efficient algorithms can perform inference and learning in Bayesian networks. Bayesian networks that model sequences of variables (e.g. speech signals or protein sequences) are called dynamic Bayesian networks. Generalizations of Bayesian networks that can represent and solve decision problems under uncertainty are called influence diagrams.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0003", "text": "Graphical model", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0004", "text": "Formally, Bayesian networks are directed acyclic graphs (DAGs) whose nodes represent variables in the Bayesian sense: they may be observable quantities, latent variables, unknown parameters or hypotheses. Each edge represents a direct conditional dependency. Any pair of nodes that are not connected (i.e. no path connects one node to the other) represent variables that are conditionally independent of each other", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0005", "text": ". Each node is associated with a probability function that takes, as input, a particular set of values for the node's parent variables, and gives (as output) the probability (or probability distribution, if applicable) of the variable represented by the node. For example, if", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0006", "text": "m\n      \n    \n    {\\displaystyle m}\n  \n parent nodes represent \n  \n    \n      \n        m\n      \n    \n    {\\displaystyle m}\n  \n Boolean variables, then the probability function could be represented by a table of \n  \n    \n      \n        \n          2\n          \n            m\n          \n        \n      \n    \n    {\\displaystyle 2^{m}}\n  \n entries, one entry for each of the \n  \n    \n      \n        \n          2\n          \n            m", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0007", "text": "m\n          \n        \n      \n    \n    {\\displaystyle 2^{m}}\n  \n possible parent combinations. Similar ideas may be applied to undirected, and possibly cyclic, graphs such as Markov networks.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0008", "text": "Example", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0009", "text": "Suppose we want to model the dependencies between three variables: the sprinkler (or more appropriately, its state - whether it is on or not), the presence or absence of rain and whether the grass is wet or not. Observe that two events can cause the grass to become wet: an active sprinkler or rain. Rain has a direct effect on the use of the sprinkler (namely that when it rains, the sprinkler usually is not active). This situation can be modeled with a Bayesian network (shown to the right)", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0010", "text": ". Each variable has two possible values, T (for true) and F (for false).", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0011", "text": "The joint probability function is, by the chain rule of probability,", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0012", "text": "Pr\n        (\n        G\n        ,\n        S\n        ,\n        R\n        )\n        =\n        Pr\n        (\n        G\n        \u2223\n        S\n        ,\n        R\n        )\n        Pr\n        (\n        S\n        \u2223\n        R\n        )\n        Pr\n        (\n        R\n        )\n      \n    \n    {\\displaystyle \\Pr(G,S,R)=\\Pr(G\\mid S,R)\\Pr(S\\mid R)\\Pr(R)}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0013", "text": "where G = \"Grass wet (true/false)\", S = \"Sprinkler turned on (true/false)\", and R = \"Raining (true/false)\".\nThe model can answer questions about the presence of a cause given the presence of an effect (so-called inverse probability) like \"What is the probability that it is raining, given the grass is wet?\" by using the conditional probability formula and summing over all nuisance variables:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0014", "text": "Pr\n        (\n        R\n        =\n        T\n        \u2223\n        G\n        =\n        T\n        )\n        =\n        \n          \n            \n              Pr\n              (\n              G\n              =\n              T\n              ,\n              R\n              =\n              T\n              )\n            \n            \n              Pr\n              (\n              G\n              =\n              T\n              )\n            \n          \n        \n        =", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0015", "text": "=\n        \n          \n            \n              \n                \u2211\n                \n                  x\n                  \u2208\n                  {\n                  T\n                  ,\n                  F\n                  }\n                \n              \n              Pr\n              (\n              G\n              =\n              T\n              ,\n              S\n              =\n              x\n              ,\n              R\n              =\n              T", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0016", "text": "R\n              =\n              T\n              )\n            \n            \n              \n                \u2211\n                \n                  x\n                  ,\n                  y\n                  \u2208\n                  {\n                  T\n                  ,\n                  F\n                  }\n                \n              \n              Pr\n              (\n              G\n              =\n              T\n              ,\n              S\n              =\n              x", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0017", "text": "S\n              =\n              x\n              ,\n              R\n              =\n              y\n              )\n            \n          \n        \n      \n    \n    {\\displaystyle \\Pr(R=T\\mid G=T)={\\frac {\\Pr(G=T,R=T)}{\\Pr(G=T)}}={\\frac {\\sum _{x\\in \\{T,F\\}}\\Pr(G=T,S=x,R=T)}{\\sum _{x,y\\in \\{T,F\\}}\\Pr(G=T,S=x,R=y)}}}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0018", "text": "Using the expansion for the joint probability function \n  \n    \n      \n        Pr\n        (\n        G\n        ,\n        S\n        ,\n        R\n        )\n      \n    \n    {\\displaystyle \\Pr(G,S,R)}\n  \n and the conditional probabilities from the conditional probability tables (CPTs) stated in the diagram, one can evaluate each term in the sums in the numerator and denominator. For example,", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0019", "text": "Pr\n                (\n                G\n                =\n                T\n                ,\n                S\n                =\n                T\n                ,\n                R\n                =\n                T\n                )\n              \n              \n                \n                =\n                Pr\n                (\n                G\n                =\n                T\n                \u2223", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0020", "text": "T\n                \u2223\n                S\n                =\n                T\n                ,\n                R\n                =\n                T\n                )\n                Pr\n                (\n                S\n                =\n                T\n                \u2223\n                R\n                =\n                T\n                )\n                Pr\n                (\n                R\n                =\n                T\n                )", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0021", "text": ")\n              \n            \n            \n              \n              \n                \n                =\n                0.99\n                \u00d7\n                0.01\n                \u00d7\n                0.2\n              \n            \n            \n              \n              \n                \n                =\n                0.00198.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0022", "text": "{\\displaystyle {\\begin{aligned}\\Pr(G=T,S=T,R=T)&=\\Pr(G=T\\mid S=T,R=T)\\Pr(S=T\\mid R=T)\\Pr(R=T)\\\\&=0.99\\times 0.01\\times 0.2\\\\&=0.00198.\\end{aligned}}}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0023", "text": "Then the numerical results (subscripted by the associated variable values) are", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0024", "text": "Pr\n        (\n        R\n        =\n        T\n        \u2223\n        G\n        =\n        T\n        )\n        =\n        \n          \n            \n              \n                0.00198\n                \n                  T\n                  T\n                  T\n                \n              \n              +\n              \n                0.1584\n                \n                  T\n                  F\n                  T", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0025", "text": "0.00198\n                \n                  T\n                  T\n                  T\n                \n              \n              +\n              \n                0.288\n                \n                  T\n                  T\n                  F\n                \n              \n              +\n              \n                0.1584\n                \n                  T\n                  F\n                  T", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0026", "text": "T\n                \n              \n              +\n              \n                0.0\n                \n                  T\n                  F\n                  F\n                \n              \n            \n          \n        \n        =\n        \n          \n            891\n            2491\n          \n        \n        \u2248\n        35.77\n        %\n        .", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0027", "text": "35.77\n        %\n        .\n      \n    \n    {\\displaystyle \\Pr(R=T\\mid G=T)={\\frac {0.00198_{TTT}+0.1584_{TFT}}{0.00198_{TTT}+0.288_{TTF}+0.1584_{TFT}+0.0_{TFF}}}={\\frac {891}{2491}}\\approx 35.77\\%.}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0028", "text": "To answer an interventional question, such as \"What is the probability that it would rain, given that we wet the grass?\" the answer is governed by the post-intervention joint distribution function", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0029", "text": "Pr\n        (\n        S\n        ,\n        R\n        \u2223\n        \n          do\n        \n        (\n        G\n        =\n        T\n        )\n        )\n        =\n        Pr\n        (\n        S\n        \u2223\n        R\n        )\n        Pr\n        (\n        R\n        )\n      \n    \n    {\\displaystyle \\Pr(S,R\\mid {\\text{do}}(G=T))=\\Pr(S\\mid R)\\Pr(R)}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0030", "text": "obtained by removing the factor \n  \n    \n      \n        Pr\n        (\n        G\n        \u2223\n        S\n        ,\n        R\n        )\n      \n    \n    {\\displaystyle \\Pr(G\\mid S,R)}\n  \n from the pre-intervention distribution. The do operator forces the value of G to be true. The probability of rain is unaffected by the action:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0031", "text": "Pr\n        (\n        R\n        \u2223\n        \n          do\n        \n        (\n        G\n        =\n        T\n        )\n        )\n        =\n        Pr\n        (\n        R\n        )\n        .\n      \n    \n    {\\displaystyle \\Pr(R\\mid {\\text{do}}(G=T))=\\Pr(R).}\n  \n\nTo predict the impact of turning the sprinkler on:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0032", "text": "Pr\n        (\n        R\n        ,\n        G\n        \u2223\n        \n          do\n        \n        (\n        S\n        =\n        T\n        )\n        )\n        =\n        Pr\n        (\n        R\n        )\n        Pr\n        (\n        G\n        \u2223\n        R\n        ,\n        S\n        =\n        T\n        )\n      \n    \n    {\\displaystyle \\Pr(R,G\\mid {\\text{do}}(S=T))=\\Pr(R)\\Pr(G\\mid R,S=T)}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0033", "text": "with the term \n  \n    \n      \n        Pr\n        (\n        S\n        =\n        T\n        \u2223\n        R\n        )\n      \n    \n    {\\displaystyle \\Pr(S=T\\mid R)}\n  \n removed, showing that the action affects the grass but not the rain.\nThese predictions may not be feasible given unobserved variables, as in most policy evaluation problems. The effect of the action \n  \n    \n      \n        \n          do\n        \n        (\n        x\n        )\n      \n    \n    {\\displaystyle {\\text{do}}(x)}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0034", "text": "{\\displaystyle {\\text{do}}(x)}\n  \n can still be predicted, however, whenever the back-door criterion is satisfied. It states that, if a set Z of nodes can be observed that d-separates (or blocks) all back-door paths from X to Y then", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0035", "text": "Pr\n        (\n        Y\n        ,\n        Z\n        \u2223\n        \n          do\n        \n        (\n        x\n        )\n        )\n        =\n        \n          \n            \n              Pr\n              (\n              Y\n              ,\n              Z\n              ,\n              X\n              =\n              x\n              )\n            \n            \n              Pr\n              (\n              X\n              =\n              x\n              \u2223\n              Z", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0036", "text": "x\n              \u2223\n              Z\n              )\n            \n          \n        \n        .\n      \n    \n    {\\displaystyle \\Pr(Y,Z\\mid {\\text{do}}(x))={\\frac {\\Pr(Y,Z,X=x)}{\\Pr(X=x\\mid Z)}}.}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0037", "text": "A back-door path is one that ends with an arrow into X. Sets that satisfy the back-door criterion are called \"sufficient\" or \"admissible.\" For example, the set Z = R is admissible for predicting the effect of S = T on G, because R d-separates the (only) back-door path S \u2190 R \u2192 G. However, if S is not observed, no other set d-separates this path and the effect of turning the sprinkler on (S = T) on the grass (G) cannot be predicted from passive observations", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0038", "text": ". In that case P(G | do(S = T)) is not \"identified\". This reflects the fact that, lacking interventional data, the observed dependence between S and G is due to a causal connection or is spurious", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0039", "text": "(apparent dependence arising from a common cause, R). (see Simpson's paradox)\nTo determine whether a causal relation is identified from an arbitrary Bayesian network with unobserved variables, one can use the three rules of \"do-calculus\" and test whether all do terms can be removed from the expression of that relation, thus confirming that the desired quantity is estimable from frequency data.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0040", "text": "Using a Bayesian network can save considerable amounts of memory over exhaustive probability tables, if the dependencies in the joint distribution are sparse. For example, a naive way of storing the conditional probabilities of 10 two-valued variables as a table requires storage space for \n  \n    \n      \n        \n          2\n          \n            10\n          \n        \n        =\n        1024\n      \n    \n    {\\displaystyle 2^{10}=1024}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0041", "text": "{\\displaystyle 2^{10}=1024}\n  \n values. If no variable's local distribution depends on more than three parent variables, the Bayesian network representation stores at most \n  \n    \n      \n        10\n        \u22c5\n        \n          2\n          \n            3\n          \n        \n        =\n        80\n      \n    \n    {\\displaystyle 10\\cdot 2^{3}=80}\n  \n values.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0042", "text": "{\\displaystyle 10\\cdot 2^{3}=80}\n  \n values.\nOne advantage of Bayesian networks is that it is intuitively easier for a human to understand (a sparse set of) direct dependencies and local distributions than complete joint distributions.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0043", "text": "Inference and learning\nBayesian networks perform three main inference tasks:\n\nInferring unobserved variables\nParameter learning for the probability distributions of each node in the network\nStructure learning of the graphical network", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0044", "text": "Inferring unobserved variables", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0045", "text": "Because a Bayesian network is a complete model for its variables and their relationships, it can be used to answer probabilistic queries about them. For example, the network can be used to update knowledge of the state of a subset of variables when other variables (the evidence variables) are observed. This process of computing the posterior distribution of variables given evidence is called probabilistic inference", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0046", "text": ". The posterior gives a universal sufficient statistic for detection applications, when choosing values for the variable subset that minimize some expected loss function, for instance the probability of decision error. A Bayesian network can thus be considered a mechanism for automatically applying Bayes' theorem to complex problems.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0047", "text": "The most common exact inference methods are: variable elimination, which eliminates (by integration or summation) the non-observed non-query variables one by one by distributing the sum over the product; clique tree propagation, which caches the computation so that many variables can be queried at one time and new evidence can be propagated quickly; and recursive conditioning and AND/OR search, which allow for a space\u2013time tradeoff and match the efficiency of variable elimination when enough", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0048", "text": "efficiency of variable elimination when enough space is used", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0049", "text": ". All of these methods have complexity that is exponential in the network's treewidth. The most common approximate inference algorithms are importance sampling, stochastic MCMC simulation, mini-bucket elimination, loopy belief propagation, generalized belief propagation and variational methods.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0050", "text": "Parameter learning", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0051", "text": "In order to fully specify the Bayesian network and thus fully represent the joint probability distribution, it is necessary to specify for each node X the probability distribution for X conditional upon X's parents. The distribution of X conditional upon its parents may have any form. It is common to work with discrete or Gaussian distributions since that simplifies calculations", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0052", "text": ". Sometimes only constraints on distribution are known; one can then use the principle of maximum entropy to determine a single distribution, the one with the greatest entropy given the constraints. (Analogously, in the specific context of a dynamic Bayesian network, the conditional distribution for the hidden state's temporal evolution is commonly specified to maximize the entropy rate of the implied stochastic process.)", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0053", "text": "Often these conditional distributions include parameters that are unknown and must be estimated from data, e.g., via the maximum likelihood approach. Direct maximization of the likelihood (or of the posterior probability) is often complex given unobserved variables", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0054", "text": ". A classical approach to this problem is the expectation-maximization algorithm, which alternates computing expected values of the unobserved variables conditional on observed data, with maximizing the complete likelihood (or posterior) assuming that previously computed expected values are correct. Under mild regularity conditions, this process converges on maximum likelihood (or maximum posterior) values for parameters.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0055", "text": "A more fully Bayesian approach to parameters is to treat them as additional unobserved variables and to compute a full posterior distribution over all nodes conditional upon observed data, then to integrate out the parameters. This approach can be expensive and lead to large dimension models, making classical parameter-setting approaches more tractable.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0056", "text": "Structure learning\nIn the simplest case, a Bayesian network is specified by an expert and is then used to perform inference. In other applications, the task of defining the network is too complex for humans. In this case, the network structure and the parameters of the local distributions must be learned from data.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0057", "text": "Automatically learning the graph structure of a Bayesian network (BN) is a challenge pursued within machine learning. The basic idea goes back to a recovery algorithm developed by Rebane and Pearl and rests on the distinction between the three possible patterns allowed in a 3-node DAG:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0058", "text": "The first 2 represent the same dependencies (\n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n and \n  \n    \n      \n        Z\n      \n    \n    {\\displaystyle Z}\n  \n are independent given \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n) and are, therefore, indistinguishable. The collider, however, can be uniquely identified, since \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n and \n  \n    \n      \n        Z\n      \n    \n    {\\displaystyle Z}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0059", "text": "Z\n      \n    \n    {\\displaystyle Z}\n  \n are marginally independent and all other pairs are dependent. Thus, while the skeletons (the graphs stripped of arrows) of these three triplets are identical, the directionality of the arrows is partially identifiable. The same distinction applies when \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n and \n  \n    \n      \n        Z\n      \n    \n    {\\displaystyle Z}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0060", "text": "Z\n      \n    \n    {\\displaystyle Z}\n  \n have common parents, except that one must first condition on those parents. Algorithms have been developed to systematically determine the skeleton of the underlying graph and, then, orient all arrows whose directionality is dictated by the conditional independences observed.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0061", "text": "An alternative method of structural learning uses optimization-based search. It requires a scoring function and a search strategy. A common scoring function is posterior probability of the structure given the training data, like the BIC or the BDeu. The time requirement of an exhaustive search returning a structure that maximizes the score is superexponential in the number of variables. A local search strategy makes incremental changes aimed at improving the score of the structure", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0062", "text": ". A global search algorithm like Markov chain Monte Carlo (MCMC) can avoid getting trapped in local minima. Finding a structure that maximizes the mutual information, typically by restricting the parent candidate set to k nodes or by finding an optimal k on a node-per-node basis, is a technique that consistently achieves high scores on benchmark datasets.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0063", "text": "A particularly fast method for exact BN learning is to cast the problem as an optimization problem, and solve it using integer programming. Acyclicity constraints are added to the integer program (IP) during solving in the form of cutting planes. Such method can handle problems with up to 100 variables.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0064", "text": "In order to deal with problems with thousands of variables, a different approach is necessary. One is to first sample one ordering, and then find the optimal BN structure with respect to that ordering. This implies working on the search space of the possible orderings, which is convenient as it is smaller than the space of network structures. Multiple orderings are then sampled and evaluated", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0065", "text": ". This method has been proven to be the best available in literature when the number of variables is huge.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0066", "text": "Another method consists of focusing on the sub-class of decomposable models, for which the MLE have a closed form. It is then possible to discover a consistent structure for hundreds of variables.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0067", "text": "Learning Bayesian networks with bounded treewidth is necessary to allow exact, tractable inference, since the worst-case inference complexity is exponential in the treewidth k (under the exponential time hypothesis). Yet, as a global property of the graph, it considerably increases the difficulty of the learning process. In this context it is possible to use K-tree for effective learning.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0068", "text": "Statistical introduction\nGiven data \n  \n    \n      \n        x\n        \n        \n      \n    \n    {\\displaystyle x\\,\\!}\n  \n and parameter \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }\n  \n, a simple Bayesian analysis starts with a prior probability (prior) \n  \n    \n      \n        p\n        (\n        \u03b8\n        )\n      \n    \n    {\\displaystyle p(\\theta )}\n  \n and likelihood \n  \n    \n      \n        p\n        (\n        x\n        \u2223\n        \u03b8\n        )", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0069", "text": "\u2223\n        \u03b8\n        )\n      \n    \n    {\\displaystyle p(x\\mid \\theta )}\n  \n to compute a posterior probability \n  \n    \n      \n        p\n        (\n        \u03b8\n        \u2223\n        x\n        )\n        \u221d\n        p\n        (\n        x\n        \u2223\n        \u03b8\n        )\n        p\n        (\n        \u03b8\n        )\n      \n    \n    {\\displaystyle p(\\theta \\mid x)\\propto p(x\\mid \\theta )p(\\theta )}\n  \n.\nOften the prior on \n  \n    \n      \n        \u03b8\n      \n    \n    {\\displaystyle \\theta }", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0070", "text": "{\\displaystyle \\theta }\n  \n depends in turn on other parameters \n  \n    \n      \n        \u03c6\n      \n    \n    {\\displaystyle \\varphi }\n  \n that are not mentioned in the likelihood. So, the prior \n  \n    \n      \n        p\n        (\n        \u03b8\n        )\n      \n    \n    {\\displaystyle p(\\theta )}\n  \n must be replaced by a likelihood \n  \n    \n      \n        p\n        (\n        \u03b8\n        \u2223\n        \u03c6\n        )\n      \n    \n    {\\displaystyle p(\\theta \\mid \\varphi )}\n  \n, and a prior", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0071", "text": ", and a prior \n  \n    \n      \n        p\n        (\n        \u03c6\n        )\n      \n    \n    {\\displaystyle p(\\varphi )}\n  \n on the newly introduced parameters \n  \n    \n      \n        \u03c6\n      \n    \n    {\\displaystyle \\varphi }\n  \n is required, resulting in a posterior probability", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0072", "text": "p\n        (\n        \u03b8\n        ,\n        \u03c6\n        \u2223\n        x\n        )\n        \u221d\n        p\n        (\n        x\n        \u2223\n        \u03b8\n        )\n        p\n        (\n        \u03b8\n        \u2223\n        \u03c6\n        )\n        p\n        (\n        \u03c6\n        )\n        .\n      \n    \n    {\\displaystyle p(\\theta ,\\varphi \\mid x)\\propto p(x\\mid \\theta )p(\\theta \\mid \\varphi )p(\\varphi ).}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0073", "text": "This is the simplest example of a hierarchical Bayes model.\nThe process may be repeated; for example, the parameters \n  \n    \n      \n        \u03c6\n      \n    \n    {\\displaystyle \\varphi }\n  \n may depend in turn on additional parameters \n  \n    \n      \n        \u03c8\n        \n        \n      \n    \n    {\\displaystyle \\psi \\,\\!}\n  \n, which require their own prior. Eventually the process must terminate, with priors that do not depend on unmentioned parameters.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0074", "text": "Introductory examples\nGiven the measured quantities \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n        \n        \n      \n    \n    {\\displaystyle x_{1},\\dots ,x_{n}\\,\\!}\n  \neach with normally distributed errors of known standard deviation \n  \n    \n      \n        \u03c3\n        \n        \n      \n    \n    {\\displaystyle \\sigma \\,\\!}\n  \n,", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0075", "text": "x\n          \n            i\n          \n        \n        \u223c\n        N\n        (\n        \n          \u03b8\n          \n            i\n          \n        \n        ,\n        \n          \u03c3\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle x_{i}\\sim N(\\theta _{i},\\sigma ^{2})}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0076", "text": "Suppose we are interested in estimating the \n  \n    \n      \n        \n          \u03b8\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\theta _{i}}\n  \n. An approach would be to estimate the \n  \n    \n      \n        \n          \u03b8\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\theta _{i}}\n  \n using a maximum likelihood approach; since the observations are independent, the likelihood factorizes and the maximum likelihood estimate is simply", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0077", "text": "\u03b8\n          \n            i\n          \n        \n        =\n        \n          x\n          \n            i\n          \n        \n        .\n      \n    \n    {\\displaystyle \\theta _{i}=x_{i}.}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0078", "text": "However, if the quantities are related, so that for example the individual \n  \n    \n      \n        \n          \u03b8\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\theta _{i}}\n  \nhave themselves been drawn from an underlying distribution, then this relationship destroys the independence and suggests a more complex model, e.g.,", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0079", "text": "x\n          \n            i\n          \n        \n        \u223c\n        N\n        (\n        \n          \u03b8\n          \n            i\n          \n        \n        ,\n        \n          \u03c3\n          \n            2\n          \n        \n        )\n        ,\n      \n    \n    {\\displaystyle x_{i}\\sim N(\\theta _{i},\\sigma ^{2}),}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0080", "text": "\u03b8\n          \n            i\n          \n        \n        \u223c\n        N\n        (\n        \u03c6\n        ,\n        \n          \u03c4\n          \n            2\n          \n        \n        )\n        ,\n      \n    \n    {\\displaystyle \\theta _{i}\\sim N(\\varphi ,\\tau ^{2}),}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0081", "text": "with improper priors \n  \n    \n      \n        \u03c6\n        \u223c\n        \n          flat\n        \n      \n    \n    {\\displaystyle \\varphi \\sim {\\text{flat}}}\n  \n, \n  \n    \n      \n        \u03c4\n        \u223c\n        \n          flat\n        \n        \u2208\n        (\n        0\n        ,\n        \u221e\n        )\n      \n    \n    {\\displaystyle \\tau \\sim {\\text{flat}}\\in (0,\\infty )}\n  \n. When \n  \n    \n      \n        n\n        \u2265\n        3\n      \n    \n    {\\displaystyle n\\geq 3}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0082", "text": "{\\displaystyle n\\geq 3}\n  \n, this is an identified model (i.e. there exists a unique solution for the model's parameters), and the posterior distributions of the individual \n  \n    \n      \n        \n          \u03b8\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle \\theta _{i}}\n  \n will tend to move, or shrink away from the maximum likelihood estimates towards their common mean. This shrinkage is a typical behavior in hierarchical Bayes models.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0083", "text": "Restrictions on priors\nSome care is needed when choosing priors in a hierarchical model, particularly on scale variables at higher levels of the hierarchy such as the variable \n  \n    \n      \n        \u03c4\n        \n        \n      \n    \n    {\\displaystyle \\tau \\,\\!}\n  \n in the example. The usual priors such as the Jeffreys prior often do not work, because the posterior distribution will not be normalizable and estimates made by minimizing the expected loss will be inadmissible.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0084", "text": "Definitions and concepts\nSeveral equivalent definitions of a Bayesian network have been offered. For the following, let G = (V,E) be a directed acyclic graph (DAG) and let X = (Xv), v \u2208 V be a set of random variables indexed by V.\n\nFactorization definition\nX is a Bayesian network with respect to G if its joint probability density function (with respect to a product measure) can be written as a product of the individual density functions, conditional on their parent variables:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0085", "text": "p\n        (\n        x\n        )\n        =\n        \n          \u220f\n          \n            v\n            \u2208\n            V\n          \n        \n        p\n        \n          (\n          \n            \n              x\n              \n                v\n              \n            \n            \n            \n              \n                |\n              \n            \n            \n            \n              x\n              \n                pa\n                \u2061\n                (", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0086", "text": "\u2061\n                (\n                v\n                )\n              \n            \n          \n          )\n        \n      \n    \n    {\\displaystyle p(x)=\\prod _{v\\in V}p\\left(x_{v}\\,{\\big |}\\,x_{\\operatorname {pa} (v)}\\right)}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0087", "text": "where pa(v) is the set of parents of v (i.e. those vertices pointing directly to v via a single edge).\nFor any set of random variables, the probability of any member of a joint distribution can be calculated from conditional probabilities using the chain rule (given a topological ordering of X) as follows:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0088", "text": "P\n        \u2061\n        (\n        \n          X\n          \n            1\n          \n        \n        =\n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          X\n          \n            n\n          \n        \n        =\n        \n          x\n          \n            n\n          \n        \n        )\n        =\n        \n          \u220f\n          \n            v\n            =\n            1\n          \n          \n            n", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0089", "text": "1\n          \n          \n            n\n          \n        \n        P\n        \u2061\n        \n          (\n          \n            \n              X\n              \n                v\n              \n            \n            =\n            \n              x\n              \n                v\n              \n            \n            \u2223\n            \n              X\n              \n                v\n                +\n                1\n              \n            \n            =\n            \n              x", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0090", "text": "=\n            \n              x\n              \n                v\n                +\n                1\n              \n            \n            ,\n            \u2026\n            ,\n            \n              X\n              \n                n\n              \n            \n            =\n            \n              x\n              \n                n\n              \n            \n          \n          )", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0091", "text": ")\n        \n      \n    \n    {\\displaystyle \\operatorname {P} (X_{1}=x_{1},\\ldots ,X_{n}=x_{n})=\\prod _{v=1}^{n}\\operatorname {P} \\left(X_{v}=x_{v}\\mid X_{v+1}=x_{v+1},\\ldots ,X_{n}=x_{n}\\right)}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0092", "text": "Using the definition above, this can be written as:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0093", "text": "P\n        \u2061\n        (\n        \n          X\n          \n            1\n          \n        \n        =\n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          X\n          \n            n\n          \n        \n        =\n        \n          x\n          \n            n\n          \n        \n        )\n        =\n        \n          \u220f\n          \n            v\n            =\n            1\n          \n          \n            n", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0094", "text": "1\n          \n          \n            n\n          \n        \n        P\n        \u2061\n        (\n        \n          X\n          \n            v\n          \n        \n        =\n        \n          x\n          \n            v\n          \n        \n        \u2223\n        \n          X\n          \n            j\n          \n        \n        =\n        \n          x\n          \n            j\n          \n        \n        \n           for each \n        \n        \n          X\n          \n            j", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0095", "text": "j\n          \n        \n        \n        \n           that is a parent of \n        \n        \n          X\n          \n            v\n          \n        \n        \n        )\n      \n    \n    {\\displaystyle \\operatorname {P} (X_{1}=x_{1},\\ldots ,X_{n}=x_{n})=\\prod _{v=1}^{n}\\operatorname {P} (X_{v}=x_{v}\\mid X_{j}=x_{j}{\\text{ for each }}X_{j}\\,{\\text{ that is a parent of }}X_{v}\\,)}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0096", "text": "The difference between the two expressions is the conditional independence of the variables from any of their non-descendants, given the values of their parent variables.\n\nLocal Markov property\nX is a Bayesian network with respect to G if it satisfies the local Markov property: each variable is conditionally independent of its non-descendants given its parent variables:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0097", "text": "X\n          \n            v\n          \n        \n        \u22a5\n        \n        \n        \n        \u22a5\n        \n          X\n          \n            V\n            \n            \u2216\n            \n            de\n            \u2061\n            (\n            v\n            )\n          \n        \n        \u2223\n        \n          X\n          \n            pa\n            \u2061\n            (\n            v\n            )\n          \n        \n        \n        \n          for all \n        \n        v", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0098", "text": "for all \n        \n        v\n        \u2208\n        V\n      \n    \n    {\\displaystyle X_{v}\\perp \\!\\!\\!\\perp X_{V\\,\\smallsetminus \\,\\operatorname {de} (v)}\\mid X_{\\operatorname {pa} (v)}\\quad {\\text{for all }}v\\in V}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0099", "text": "where de(v) is the set of descendants and V \\ de(v) is the set of non-descendants of v.\nThis can be expressed in terms similar to the first definition, as", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0100", "text": "P\n                \u2061\n                (\n                \n                  X\n                  \n                    v\n                  \n                \n                =\n                \n                  x\n                  \n                    v\n                  \n                \n                \u2223\n                \n                  X\n                  \n                    i", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0101", "text": "i\n                  \n                \n                =\n                \n                  x\n                  \n                    i\n                  \n                \n                \n                   for each \n                \n                \n                  X\n                  \n                    i\n                  \n                \n                \n                   that is not a descendant of \n                \n                \n                  X", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0102", "text": "X\n                  \n                    v\n                  \n                \n                \n                )\n              \n            \n            \n              \n                =", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0103", "text": "P\n                (\n                \n                  X\n                  \n                    v\n                  \n                \n                =\n                \n                  x\n                  \n                    v\n                  \n                \n                \u2223\n                \n                  X\n                  \n                    j\n                  \n                \n                =", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0104", "text": "=\n                \n                  x\n                  \n                    j\n                  \n                \n                \n                   for each \n                \n                \n                  X\n                  \n                    j\n                  \n                \n                \n                   that is a parent of \n                \n                \n                  X\n                  \n                    v", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0105", "text": ")\n              \n            \n          \n        \n      \n    \n    {\\displaystyle {\\begin{aligned}&\\operatorname {P} (X_{v}=x_{v}\\mid X_{i}=x_{i}{\\text{ for each }}X_{i}{\\text{ that is not a descendant of }}X_{v}\\,)\\\\[6pt]={}&P(X_{v}=x_{v}\\mid X_{j}=x_{j}{\\text{ for each }}X_{j}{\\text{ that is a parent of }}X_{v}\\,)\\end{aligned}}}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0106", "text": "The set of parents is a subset of the set of non-descendants because the graph is acyclic.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0107", "text": "Marginal independence structure", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0108", "text": "In general, learning a Bayesian network from data is known to be NP-hard. This is due in part to the combinatorial explosion of enumerating DAGs as the number of variables increases", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0109", "text": ". Nevertheless, insights about an underlying Bayesian network can be learned from data in polynomial time by focusing on its marginal independence structure: while the conditional independence statements of a distribution modeled by a Bayesian network are encoded by a DAG (according to the factorization and Markov properties above), its marginal independence statements\u2014the conditional independence statements in which the conditioning set is empty\u2014are encoded by a simple undirected graph with", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0110", "text": "encoded by a simple undirected graph with special properties such as equal intersection and independence numbers.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0111", "text": "Developing Bayesian networks\nDeveloping a Bayesian network often begins with creating a DAG G such that X satisfies the local Markov property with respect to G. Sometimes this is a causal DAG. The conditional probability distributions of each variable given its parents in G are assessed. In many cases, in particular in the case where the variables are discrete, if the joint distribution of X is the product of these conditional distributions, then X is a Bayesian network with respect to G.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0112", "text": "Markov blanket", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0113", "text": "The Markov blanket of a node is the set of nodes consisting of its parents, its children, and any other parents of its children. The Markov blanket renders the node independent of the rest of the network; the joint distribution of the variables in the Markov blanket of a node is sufficient knowledge for calculating the distribution of the node. X is a Bayesian network with respect to G if every node is conditionally independent of all other nodes in the network, given its Markov blanket.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0114", "text": "d-separation\nThis definition can be made more general by defining the \"d\"-separation of two nodes, where d stands for directional. We first define the \"d\"-separation of a trail and then we will define the \"d\"-separation of two nodes in terms of that.\nLet P be a trail from node u to v. A trail is a loop-free, undirected (i.e. all edge directions are ignored) path between two nodes. Then P is said to be d-separated by a set of nodes Z if any of the following conditions holds:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0115", "text": "P contains (but does not need to be entirely) a directed chain, \n  \n    \n      \n        u\n        \u22ef\n        \u2190\n        m\n        \u2190\n        \u22ef\n        v\n      \n    \n    {\\displaystyle u\\cdots \\leftarrow m\\leftarrow \\cdots v}\n  \n or \n  \n    \n      \n        u\n        \u22ef\n        \u2192\n        m\n        \u2192\n        \u22ef\n        v\n      \n    \n    {\\displaystyle u\\cdots \\rightarrow m\\rightarrow \\cdots v}\n  \n, such that the middle node m is in Z,\nP contains a fork, \n  \n    \n      \n        u\n        \u22ef\n        \u2190", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0116", "text": "u\n        \u22ef\n        \u2190\n        m\n        \u2192\n        \u22ef\n        v\n      \n    \n    {\\displaystyle u\\cdots \\leftarrow m\\rightarrow \\cdots v}\n  \n, such that the middle node m is in Z, or\nP contains an inverted fork (or collider), \n  \n    \n      \n        u\n        \u22ef\n        \u2192\n        m\n        \u2190\n        \u22ef\n        v\n      \n    \n    {\\displaystyle u\\cdots \\rightarrow m\\leftarrow \\cdots v}\n  \n, such that the middle node m is not in Z and no descendant of m is in Z.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0117", "text": "The nodes u and v are d-separated by Z if all trails between them are d-separated. If u and v are not d-separated, they are d-connected.\nX is a Bayesian network with respect to G if, for any two nodes u, v:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0118", "text": "X\n          \n            u\n          \n        \n        \u22a5\n        \n        \n        \n        \u22a5\n        \n          X\n          \n            v\n          \n        \n        \u2223\n        \n          X\n          \n            Z\n          \n        \n      \n    \n    {\\displaystyle X_{u}\\perp \\!\\!\\!\\perp X_{v}\\mid X_{Z}}\n  \n\nwhere Z is a set which d-separates u and v. (The Markov blanket is the minimal set of nodes which d-separates node v from all other nodes.)", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0119", "text": "Causal networks\nAlthough Bayesian networks are often used to represent causal relationships, this need not be the case: a directed edge from u to v does not require that Xv be causally dependent on Xu. This is demonstrated by the fact that Bayesian networks on the graphs:", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0120", "text": "a\n        \u2192\n        b\n        \u2192\n        c\n        \n        \n          and\n        \n        \n        a\n        \u2190\n        b\n        \u2190\n        c\n      \n    \n    {\\displaystyle a\\rightarrow b\\rightarrow c\\qquad {\\text{and}}\\qquad a\\leftarrow b\\leftarrow c}", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0121", "text": "are equivalent: that is they impose exactly the same conditional independence requirements.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0122", "text": "A causal network is a Bayesian network with the requirement that the relationships be causal. The additional semantics of causal networks specify that if a node X is actively caused to be in a given state x (an action written as do(X = x)), then the probability density function changes to that of the network obtained by cutting the links from the parents of X to X, and setting X to the caused value x", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0123", "text": ". Using these semantics, the impact of external interventions from data obtained prior to intervention can be predicted.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0124", "text": "Inference complexity and approximation algorithms", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0125", "text": "In 1990, while working at Stanford University on large bioinformatic applications, Cooper proved that exact inference in Bayesian networks is NP-hard. This result prompted research on approximation algorithms with the aim of developing a tractable approximation to probabilistic inference. In 1993, Paul Dagum and Michael Luby proved two surprising results on the complexity of approximation of probabilistic inference in Bayesian networks", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0126", "text": ". First, they proved that no tractable deterministic algorithm can approximate probabilistic inference to within an absolute error \u025b < 1/2. Second, they proved that no tractable randomized algorithm can approximate probabilistic inference to within an absolute error \u025b < 1/2 with confidence probability greater than 1/2.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0127", "text": "At about the same time, Roth proved that exact inference in Bayesian networks is in fact #P-complete (and thus as hard as counting the number of satisfying assignments of a conjunctive normal form formula (CNF)) and that approximate inference within a factor 2n1\u2212\u025b for every \u025b > 0, even for Bayesian networks with restricted architecture, is NP-hard.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0128", "text": "In practical terms, these complexity results suggested that while Bayesian networks were rich representations for AI and machine learning applications, their use in large real-world applications would need to be tempered by either topological structural constraints, such as na\u00efve Bayes networks, or by restrictions on the conditional probabilities", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0129", "text": ". The bounded variance algorithm developed by Dagum and Luby was the first provable fast approximation algorithm to efficiently approximate probabilistic inference in Bayesian networks with guarantees on the error approximation. This powerful algorithm required the minor restriction on the conditional probabilities of the Bayesian network to be bounded away from zero and one by", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0130", "text": "1\n        \n          /\n        \n        p\n        (\n        n\n        )\n      \n    \n    {\\displaystyle 1/p(n)}\n  \n where \n  \n    \n      \n        p\n        (\n        n\n        )\n      \n    \n    {\\displaystyle p(n)}\n  \n was any polynomial of the number of nodes in the network, \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0131", "text": "Software\nNotable software for Bayesian networks include:\n\nOpenBUGS \u2013 Open-source development of WinBUGS.\nSPSS Modeler \u2013 Commercial software that includes an implementation for Bayesian networks.\nStan (software) \u2013 Stan is an open-source package for obtaining Bayesian inference using the No-U-Turn sampler (NUTS), a variant of Hamiltonian Monte Carlo.\nWinBUGS \u2013 One of the first computational implementations of MCMC samplers. No longer maintained.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0132", "text": "History\nThe term Bayesian network was coined by Judea Pearl in 1985 to emphasize:\n\nthe often subjective nature of the input information\nthe reliance on Bayes' conditioning as the basis for updating information\nthe distinction between causal and evidential modes of reasoning\nIn the late 1980s Pearl's Probabilistic Reasoning in Intelligent Systems and Neapolitan's Probabilistic Reasoning in Expert Systems summarized their properties and established them as a field of study.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0133", "text": "See also\nNotes\nReferences\nFurther reading\nConrady S, Jouffe L (2015-07-01). Bayesian Networks and BayesiaLab \u2013 A practical introduction for researchers. Franklin, Tennessee: Bayesian USA. ISBN 978-0-9965333-0-0.\nCharniak E (Winter 1991). \"Bayesian networks without tears\" (PDF). AI Magazine.\nKruse R, Borgelt C, Klawonn F, Moewes C, Steinbrecher M, Held P (2013). Computational Intelligence A Methodological Introduction. London: Springer-Verlag. ISBN 978-1-4471-5012-1.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0134", "text": "Borgelt C, Steinbrecher M, Kruse R (2009). Graphical Models \u2013 Representations for Learning, Reasoning and Data Mining (Second ed.). Chichester: Wiley. ISBN 978-0-470-74956-2.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0135", "text": "External links\nAn Introduction to Bayesian Networks and their Contemporary Applications\nOn-line Tutorial on Bayesian nets and probability\nWeb-App to create Bayesian nets and run it with a Monte Carlo method\nContinuous Time Bayesian Networks\nBayesian Networks: Explanation and Analogy\nA live tutorial on learning Bayesian networks", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Bayesian_network_0136", "text": "A live tutorial on learning Bayesian networks\nA hierarchical Bayes Model for handling sample heterogeneity in classification problems, provides a classification model taking into consideration the uncertainty associated with measuring replicate samples.\nHierarchical Naive Bayes Model for handling sample uncertainty Archived 2007-09-28 at the Wayback Machine, shows how to perform classification and learning with continuous and discrete variables with replicated measurements.", "title": "Bayesian network", "url": "https://en.wikipedia.org/wiki/Bayesian_network"}, {"id": "rec_Hidden_Markov_model_0000", "text": "A hidden Markov model (HMM) is a Markov model in which the observations are dependent on a latent (or hidden) Markov process (referred to as \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n). An HMM requires that there be an observable process \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n whose outcomes depend on the outcomes of \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n in a known way. Since \n  \n    \n      \n        X", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0001", "text": "X\n      \n    \n    {\\displaystyle X}\n  \n cannot be observed directly, the goal is to learn about state of \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n by observing \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n. By definition of being a Markov model, an HMM has an additional requirement that the outcome of \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n at time \n  \n    \n      \n        t\n        =\n        \n          t", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0002", "text": "t\n        =\n        \n          t\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle t=t_{0}}\n  \n must be \"influenced\" exclusively by the outcome of \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n at \n  \n    \n      \n        t\n        =\n        \n          t\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle t=t_{0}}\n  \n and that the outcomes of \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n and", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0003", "text": "{\\displaystyle X}\n  \n and \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n at \n  \n    \n      \n        t\n        <\n        \n          t\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle t<t_{0}}\n  \n must be conditionally independent of \n  \n    \n      \n        Y\n      \n    \n    {\\displaystyle Y}\n  \n at \n  \n    \n      \n        t\n        =\n        \n          t\n          \n            0", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0004", "text": "0\n          \n        \n      \n    \n    {\\displaystyle t=t_{0}}\n  \n given \n  \n    \n      \n        X\n      \n    \n    {\\displaystyle X}\n  \n at time \n  \n    \n      \n        t\n        =\n        \n          t\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle t=t_{0}}\n  \n. Estimation of the parameters in an HMM can be performed using maximum likelihood estimation. For linear chain HMMs, the Baum\u2013Welch algorithm can be used to estimate parameters.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0005", "text": "Hidden Markov models are known for their applications to thermodynamics, statistical mechanics, physics, chemistry, economics, finance, signal processing, information theory, pattern recognition\u2014such as speech recognition, handwriting recognition, gesture recognition, part-of-speech tagging, musical score following, partial discharges and bioinformatics.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0006", "text": "Definition\nLet \n  \n    \n      \n        \n          X\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle X_{n}}\n  \n and \n  \n    \n      \n        \n          Y\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle Y_{n}}\n  \n be discrete-time stochastic processes and \n  \n    \n      \n        n\n        \u2265\n        1\n      \n    \n    {\\displaystyle n\\geq 1}\n  \n. The pair \n  \n    \n      \n        (\n        \n          X\n          \n            n", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0007", "text": "X\n          \n            n\n          \n        \n        ,\n        \n          Y\n          \n            n\n          \n        \n        )\n      \n    \n    {\\displaystyle (X_{n},Y_{n})}\n  \n is a hidden Markov model if", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0008", "text": "X\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle X_{n}}\n  \n is a Markov process whose behavior is not directly observable (\"hidden\");", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0009", "text": "P\n          \n        \n        \u2061\n        \n          \n            (\n          \n        \n        \n          Y\n          \n            n\n          \n        \n        \u2208\n        A\n         \n        \n          \n            |\n          \n        \n         \n        \n          X\n          \n            1\n          \n        \n        =\n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          X", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0010", "text": ",\n        \n          X\n          \n            n\n          \n        \n        =\n        \n          x\n          \n            n\n          \n        \n        \n          \n            )\n          \n        \n        =\n        \n          \n            P\n          \n        \n        \u2061\n        \n          \n            (\n          \n        \n        \n          Y\n          \n            n\n          \n        \n        \u2208\n        A\n         \n        \n          \n            |", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0011", "text": "|\n          \n        \n         \n        \n          X\n          \n            n\n          \n        \n        =\n        \n          x\n          \n            n\n          \n        \n        \n          \n            )\n          \n        \n      \n    \n    {\\displaystyle \\operatorname {\\mathbf {P} } {\\bigl (}Y_{n}\\in A\\ {\\bigl |}\\ X_{1}=x_{1},\\ldots ,X_{n}=x_{n}{\\bigr )}=\\operatorname {\\mathbf {P} } {\\bigl (}Y_{n}\\in A\\ {\\bigl |}\\ X_{n}=x_{n}{\\bigr )}}\n  \n,\nfor every \n  \n    \n      \n        n", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0012", "text": ",\nfor every \n  \n    \n      \n        n\n        \u2265\n        1\n      \n    \n    {\\displaystyle n\\geq 1}\n  \n, \n  \n    \n      \n        \n          x\n          \n            1\n          \n        \n        ,\n        \u2026\n        ,\n        \n          x\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle x_{1},\\ldots ,x_{n}}\n  \n, and every Borel set \n  \n    \n      \n        A\n      \n    \n    {\\displaystyle A}\n  \n.\nLet \n  \n    \n      \n        \n          X\n          \n            t", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0013", "text": "X\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle X_{t}}\n  \n and \n  \n    \n      \n        \n          Y\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle Y_{t}}\n  \n be continuous-time stochastic processes. The pair \n  \n    \n      \n        (\n        \n          X\n          \n            t\n          \n        \n        ,\n        \n          Y\n          \n            t\n          \n        \n        )", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0014", "text": ")\n      \n    \n    {\\displaystyle (X_{t},Y_{t})}\n  \n is a hidden Markov model if", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0015", "text": "X\n          \n            t\n          \n        \n      \n    \n    {\\displaystyle X_{t}}\n  \n is a Markov process whose behavior is not directly observable (\"hidden\");", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0016", "text": "P\n          \n        \n        \u2061\n        (\n        \n          Y\n          \n            \n              t\n              \n                0\n              \n            \n          \n        \n        \u2208\n        A\n        \u2223\n        {\n        \n          X\n          \n            t\n          \n        \n        \u2208\n        \n          B\n          \n            t\n          \n        \n        \n          }\n          \n            t\n            \u2264", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0017", "text": "t\n            \u2264\n            \n              t\n              \n                0\n              \n            \n          \n        \n        )\n        =\n        \n          \n            P\n          \n        \n        \u2061\n        (\n        \n          Y\n          \n            \n              t\n              \n                0\n              \n            \n          \n        \n        \u2208\n        A\n        \u2223\n        \n          X\n          \n            \n              t\n              \n                0", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0018", "text": "t\n              \n                0\n              \n            \n          \n        \n        \u2208\n        \n          B\n          \n            \n              t\n              \n                0\n              \n            \n          \n        \n        )\n      \n    \n    {\\displaystyle \\operatorname {\\mathbf {P} } (Y_{t_{0}}\\in A\\mid \\{X_{t}\\in B_{t}\\}_{t\\leq t_{0}})=\\operatorname {\\mathbf {P} } (Y_{t_{0}}\\in A\\mid X_{t_{0}}\\in B_{t_{0}})}\n  \n,\nfor every \n  \n    \n      \n        \n          t", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0019", "text": ",\nfor every \n  \n    \n      \n        \n          t\n          \n            0\n          \n        \n      \n    \n    {\\displaystyle t_{0}}\n  \n, every Borel set \n  \n    \n      \n        A\n      \n    \n    {\\displaystyle A}\n  \n, and every family of Borel sets \n  \n    \n      \n        {\n        \n          B\n          \n            t\n          \n        \n        \n          }\n          \n            t\n            \u2264\n            \n              t\n              \n                0", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0020", "text": "0\n              \n            \n          \n        \n      \n    \n    {\\displaystyle \\{B_{t}\\}_{t\\leq t_{0}}}\n  \n.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0021", "text": "Terminology\nThe states of the process \n  \n    \n      \n        \n          X\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle X_{n}}\n  \n (resp. \n  \n    \n      \n        \n          X\n          \n            t\n          \n        \n        )\n      \n    \n    {\\displaystyle X_{t})}\n  \n are called hidden states, and \n  \n    \n      \n        \n          \n            P\n          \n        \n        \u2061\n        \n          \n            (\n          \n        \n        \n          Y", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0022", "text": "Y\n          \n            n\n          \n        \n        \u2208\n        A\n        \u2223\n        \n          X\n          \n            n\n          \n        \n        =\n        \n          x\n          \n            n\n          \n        \n        \n          \n            )\n          \n        \n      \n    \n    {\\displaystyle \\operatorname {\\mathbf {P} } {\\bigl (}Y_{n}\\in A\\mid X_{n}=x_{n}{\\bigr )}}\n  \n (resp. \n  \n    \n      \n        \n          \n            P", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0023", "text": "P\n          \n        \n        \u2061\n        \n          \n            (\n          \n        \n        \n          Y\n          \n            t\n          \n        \n        \u2208\n        A\n        \u2223\n        \n          X\n          \n            t\n          \n        \n        \u2208\n        \n          B\n          \n            t\n          \n        \n        \n          \n            )\n          \n        \n        )", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0024", "text": ")\n      \n    \n    {\\displaystyle \\operatorname {\\mathbf {P} } {\\bigl (}Y_{t}\\in A\\mid X_{t}\\in B_{t}{\\bigr )})}\n  \n is called emission probability or output probability.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0025", "text": "Examples\nDrawing balls from hidden urns\nIn its discrete form, a hidden Markov process can be visualized as a generalization of the urn problem with replacement (where each item from the urn is returned to the original urn before the next step).", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0026", "text": "Consider this example\nIn a room that is not visible to an observer there is a genie. The room contains urns X1, X2, X3, ... each of which contains a known mix of balls, with each ball having a unique label y1, y2, y3, ... . The genie chooses an urn in that room and randomly draws a ball from that urn. It then puts the ball onto a conveyor belt, where the observer can observe the sequence of the balls but not the sequence of urns from which they were drawn.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0027", "text": "The genie has some procedure to choose urns:", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0028", "text": "The choice of the urn for the n-th ball depends only upon a random number and the choice of the urn for the (n \u2212 1)-th ball.\nThe choice of urn does not directly depend on the urns chosen before this single previous urn.\nTherefore, this is called a Markov process. It can be described by the upper part of Figure 1.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0029", "text": "Markov process", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0030", "text": "The Markov process cannot be observed, only the sequence of labeled balls, thus this arrangement is called a hidden Markov process. This is illustrated by the lower part of the diagram shown in Figure 1, where one can see that balls y1, y2, y3, y4 can be drawn at each state. Even if the observer knows the composition of the urns and has just observed a sequence of three balls, e.g", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0031", "text": ". y1, y2 and y3 on the conveyor belt, the observer still cannot be sure which urn (i.e., at which state) the genie has drawn the third ball from. However, the observer can work out other information, such as the likelihood that the third ball came from each of the urns.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0032", "text": "Weather guessing game", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0033", "text": "Consider two friends, Alice and Bob, who live far apart from each other and who talk together daily over the telephone about what they did that day. Bob is only interested in three activities: walking in the park, shopping, and cleaning his apartment. The choice of what to do is determined exclusively by the weather on a given day. Alice has no definite information about the weather, but she knows general trends", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0034", "text": ". Based on what Bob tells her he did each day, Alice tries to guess what the weather must have been like.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0035", "text": "Alice believes that the weather operates as a discrete Markov chain. There are two states, \"Rainy\" and \"Sunny\", but she cannot observe them directly, that is, they are hidden from her. On each day, there is a certain chance that Bob will perform one of the following activities, depending on the weather: \"walk\", \"shop\", or \"clean\". Since Bob tells Alice about his activities, those are the observations. The entire system is that of a hidden Markov model (HMM).", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0036", "text": "Alice knows the general weather trends in the area, and what Bob likes to do on average. In other words, the parameters of the HMM are known. They can be represented as follows in Python:", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0037", "text": "In this piece of code, start_probability represents Alice's belief about which state the HMM is in when Bob first calls her (all she knows is that it tends to be rainy on average). The particular probability distribution used here is not the equilibrium one, which is (given the transition probabilities) approximately {'Rainy': 0.57, 'Sunny': 0.43}. The transition_probability represents the change of the weather in the underlying Markov chain", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0038", "text": ". In this example, there is only a 30% chance that tomorrow will be sunny if today is rainy. The emission_probability represents how likely Bob is to perform a certain activity on each day. If it is rainy, there is a 50% chance that he is cleaning his apartment; if it is sunny, there is a 60% chance that he is outside for a walk.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0039", "text": "A similar example is further elaborated in the Viterbi algorithm page.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0040", "text": "Structural architecture\nThe diagram below shows the general architecture of an instantiated HMM. Each oval shape represents a random variable that can adopt any of a number of values. The random variable x(t) is the hidden state at time t (with the model from the above diagram, x(t) \u2208 {\u200dx1, x2, x3\u200d}). The random variable y(t) is the observation at time t (with y(t) \u2208 {\u200dy1, y2, y3, y4\u200d}). The arrows in the diagram (often called a trellis diagram) denote conditional dependencies.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0041", "text": "From the diagram, it is clear that the conditional probability distribution of the hidden variable x(t) at time t, given the values of the hidden variable x at all times, depends only on the value of the hidden variable x(t \u2212 1); the values at time t \u2212 2 and before have no influence. This is called the Markov property. Similarly, the value of the observed variable y(t) depends on only the value of the hidden variable x(t) (both at time t).", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0042", "text": "In the standard type of hidden Markov model considered here, the state space of the hidden variables is discrete, while the observations themselves can either be discrete (typically generated from a categorical distribution) or continuous (typically from a Gaussian distribution). The parameters of a hidden Markov model are of two types, transition probabilities and emission probabilities (also known as output probabilities)", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0043", "text": ". The transition probabilities control the way the hidden state at time t is chosen given the hidden state at time t \u2212 1.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0044", "text": "The hidden state space is assumed to consist of one of N possible values, modelled as a categorical distribution. (See the section below on extensions for other possibilities.) This means that for each of the N possible states that a hidden variable at time t can be in, there is a transition probability from this state to each of the N possible states of the hidden variable at time t + 1, for a total of \n  \n    \n      \n        \n          N\n          \n            2", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0045", "text": "2\n          \n        \n      \n    \n    {\\displaystyle N^{2}}\n  \n transition probabilities. The set of transition probabilities for transitions from any given state must sum to 1. Thus, the N \u00d7 N matrix of transition probabilities is a Markov matrix. Because any transition probability can be determined once the others are known, there are a total of N(N \u2212 1) transition parameters.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0046", "text": "In addition, for each of the N possible states, there is a set of emission probabilities governing the distribution of the observed variable at a particular time given the state of the hidden variable at that time. The size of this set depends on the nature of the observed variable", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0047", "text": ". For example, if the observed variable is discrete with M possible values, governed by a categorical distribution, there will be M \u2212 1 separate parameters, for a total of N(M \u2212 1) emission parameters over all hidden states. On the other hand, if the observed variable is an M-dimensional vector distributed according to an arbitrary multivariate Gaussian distribution, there will be M parameters controlling the means and", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0048", "text": "M\n          (\n          M\n          +\n          1\n          )\n        \n        \n          /\n        \n        2\n      \n    \n    {\\displaystyle {M(M+1)}/2}\n  \n parameters controlling the covariance matrix, for a total of \n  \n    \n      \n        N\n        \n          (\n          \n            M\n            +\n            \n              M\n              (\n              M\n              +\n              1\n              )\n            \n            \n              /", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0049", "text": "/\n            \n            \n              2\n            \n          \n          )\n        \n        =\n        \n          N\n          M\n          (\n          M\n          +\n          3\n          )\n        \n        \n          /\n        \n        2\n        =\n        O\n        (\n        N\n        \n          M\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle N\\left(M+{M(M+1)}/{2}\\right)={NM(M+3)}/2=O(NM^{2})}", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0050", "text": "emission parameters. (In such a case, unless the value of M is small, it may be more practical to restrict the nature of the covariances between individual elements of the observation vector, e.g. by assuming that the elements are independent of each other, or less restrictively, are independent of all but a fixed number of adjacent elements.)", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0051", "text": "Inference\nSeveral inference problems are associated with hidden Markov models, as outlined below.\n\nProbability of an observed sequence\nThe task is to compute in a best way, given the parameters of the model, the probability of a particular output sequence. This requires summation over all possible state sequences:\nThe probability of observing a sequence", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0052", "text": "Y\n        =\n        y\n        (\n        0\n        )\n        ,\n        y\n        (\n        1\n        )\n        ,\n        \u2026\n        ,\n        y\n        (\n        L\n        \u2212\n        1\n        )\n        ,\n      \n    \n    {\\displaystyle Y=y(0),y(1),\\dots ,y(L-1),}\n  \n\nof length L is given by", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0053", "text": "of length L is given by\n\n  \n    \n      \n        P\n        (\n        Y\n        )\n        =\n        \n          \u2211\n          \n            X\n          \n        \n        P\n        (\n        Y\n        \u2223\n        X\n        )\n        P\n        (\n        X\n        )\n        ,\n      \n    \n    {\\displaystyle P(Y)=\\sum _{X}P(Y\\mid X)P(X),}\n  \n\nwhere the sum runs over all possible hidden-node sequences", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0054", "text": "X\n        =\n        x\n        (\n        0\n        )\n        ,\n        x\n        (\n        1\n        )\n        ,\n        \u2026\n        ,\n        x\n        (\n        L\n        \u2212\n        1\n        )\n        .\n      \n    \n    {\\displaystyle X=x(0),x(1),\\dots ,x(L-1).}\n  \n\nApplying the principle of dynamic programming, this problem, too, can be handled efficiently using the forward algorithm.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0055", "text": "Probability of the latent variables\nA number of related tasks ask about the probability of one or more of the latent variables, given the model's parameters and a sequence of observations \n  \n    \n      \n        y\n        (\n        1\n        )\n        ,\n        \u2026\n        ,\n        y\n        (\n        t\n        )\n      \n    \n    {\\displaystyle y(1),\\dots ,y(t)}\n  \n.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0056", "text": "Filtering\nThe task is to compute, given the model's parameters and a sequence of observations, the distribution over hidden states of the last latent variable at the end of the sequence, i.e. to compute \n  \n    \n      \n        P\n        (\n        x\n        (\n        t\n        )\n        \u2223\n        y\n        (\n        1\n        )\n        ,\n        \u2026\n        ,\n        y\n        (\n        t\n        )\n        )\n      \n    \n    {\\displaystyle P(x(t)\\mid y(1),\\dots ,y(t))}", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0057", "text": ". This task is used when the sequence of latent variables is thought of as the underlying states that a process moves through at a sequence of points in time, with corresponding observations at each point. Then, it is natural to ask about the state of the process at the end.\nThis problem can be handled efficiently using the forward algorithm. An example is when the algorithm is applied to a Hidden Markov Network to determine \n  \n    \n      \n        \n          P", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0058", "text": "P\n        \n        \n          \n            (\n          \n        \n        \n          h\n          \n            t\n          \n        \n        \u2223\n        \n          v\n          \n            1\n            :\n            t\n          \n        \n        \n          \n            )\n          \n        \n      \n    \n    {\\displaystyle \\mathrm {P} {\\big (}h_{t}\\mid v_{1:t}{\\big )}}\n  \n.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0059", "text": "Smoothing\nThis is similar to filtering but asks about the distribution of a latent variable somewhere in the middle of a sequence, i.e. to compute \n  \n    \n      \n        P\n        (\n        x\n        (\n        k\n        )\n        \u2223\n        y\n        (\n        1\n        )\n        ,\n        \u2026\n        ,\n        y\n        (\n        t\n        )\n        )\n      \n    \n    {\\displaystyle P(x(k)\\mid y(1),\\dots ,y(t))}\n  \n for some \n  \n    \n      \n        k\n        <\n        t", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0060", "text": "k\n        <\n        t\n      \n    \n    {\\displaystyle k<t}\n  \n. From the perspective described above, this can be thought of as the probability distribution over hidden states for a point in time k in the past, relative to time t.\nThe forward-backward algorithm is a good method for computing the smoothed values for all hidden state variables.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0061", "text": "Most likely explanation", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0062", "text": "The task, unlike the previous two, asks about the joint probability of the entire sequence of hidden states that generated a particular sequence of observations (see illustration on the right). This task is generally applicable when HMM's are applied to different sorts of problems from those for which the tasks of filtering and smoothing are applicable", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0063", "text": ". An example is part-of-speech tagging, where the hidden states represent the underlying parts of speech corresponding to an observed sequence of words. In this case, what is of interest is the entire sequence of parts of speech, rather than simply the part of speech for a single word, as filtering or smoothing would compute.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0064", "text": "This task requires finding a maximum over all possible state sequences, and can be solved efficiently by the Viterbi algorithm.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0065", "text": "Statistical significance", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0066", "text": "For some of the above problems, it may also be interesting to ask about statistical significance", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0067", "text": ". What is the probability that a sequence drawn from some null distribution will have an HMM probability (in the case of the forward algorithm) or a maximum state sequence probability (in the case of the Viterbi algorithm) at least as large as that of a particular output sequence?  When an HMM is used to evaluate the relevance of a hypothesis for a particular output sequence, the statistical significance indicates the false positive rate associated with failing to reject the hypothesis for the", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0068", "text": "with failing to reject the hypothesis for the output sequence.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0069", "text": "Learning", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0070", "text": "The parameter learning task in HMMs is to find, given an output sequence or a set of such sequences, the best set of state transition and emission probabilities. The task is usually to derive the maximum likelihood estimate of the parameters of the HMM given the set of output sequences. No tractable algorithm is known for solving this problem exactly, but a local maximum likelihood can be derived efficiently using the Baum\u2013Welch algorithm or the Baldi\u2013Chauvin algorithm", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0071", "text": ". The Baum\u2013Welch algorithm is a special case of the expectation-maximization algorithm.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0072", "text": "If the HMMs are used for time series prediction, more sophisticated Bayesian inference methods, like Markov chain Monte Carlo (MCMC) sampling are proven to be favorable over finding a single maximum likelihood model both in terms of accuracy and stability. Since MCMC imposes significant computational burden, in cases where computational scalability is also of interest, one may alternatively resort to variational approximations to Bayesian inference, e.g", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0073", "text": ". Indeed, approximate variational inference offers computational efficiency comparable to expectation-maximization, while yielding an accuracy profile only slightly inferior to exact MCMC-type Bayesian inference.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0074", "text": "Applications\nHMMs can be applied in many fields where the goal is to recover a data sequence that is not immediately observable (but other data that depend on the sequence are). Applications include:", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0075", "text": "Computational finance\nSingle-molecule kinetic analysis\nNeuroscience\nCryptanalysis\nSpeech recognition, including Siri\nSpeech synthesis\nPart-of-speech tagging\nDocument separation in scanning solutions\nMachine translation\nPartial discharge\nGene prediction\nHandwriting recognition\nAlignment of bio-sequences\nTime series analysis\nActivity recognition\nProtein folding\nSequence classification\nMetamorphic virus detection\nSequence motif discovery (DNA and proteins)\nDNA hybridization kinetics", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0076", "text": "DNA hybridization kinetics\nChromatin state discovery\nTransportation forecasting\nSolar irradiance variability", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0077", "text": "History\nHidden Markov models were described in a series of statistical papers by Leonard E. Baum and other authors in the second half of the 1960s. One of the first applications of HMMs was speech recognition, starting in the mid-1970s. From the linguistics point of view, hidden Markov models are equivalent to stochastic regular grammar.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0078", "text": "In the second half of the 1980s, HMMs began to be applied to the analysis of biological sequences, in particular DNA. Since then, they have become ubiquitous in the field of bioinformatics.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0079", "text": "Extensions\nGeneral state spaces", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0080", "text": "In the hidden Markov models considered above, the state space of the hidden variables is discrete, while the observations themselves can either be discrete (typically generated from a categorical distribution) or continuous (typically from a Gaussian distribution). Hidden Markov models can also be generalized to allow continuous state spaces", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0081", "text": ". Examples of such models are those where the Markov process over hidden variables is a linear dynamical system, with a linear relationship among related variables and where all hidden and observed variables follow a Gaussian distribution", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0082", "text": ". In simple cases, such as the linear dynamical system just mentioned, exact inference is tractable (in this case, using the Kalman filter); however, in general, exact inference in HMMs with continuous latent variables is infeasible, and approximate methods must be used, such as the extended Kalman filter or the particle filter.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0083", "text": "Nowadays, inference in hidden Markov models is performed in nonparametric settings, where the dependency structure enables  identifiability of the model and the learnability limits are still under exploration.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0084", "text": "Bayesian modeling of the transitions probabilities", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0085", "text": "Hidden Markov models are generative models, in which the joint distribution of observations and hidden states, or equivalently both the prior distribution of hidden states (the transition probabilities) and conditional distribution of observations given states (the emission probabilities), is modeled. The above algorithms implicitly assume a uniform prior distribution over the transition probabilities", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0086", "text": ". However, it is also possible to create hidden Markov models with other types of prior distributions. An obvious candidate, given the categorical distribution of the transition probabilities, is the Dirichlet distribution, which is the conjugate prior distribution of the categorical distribution. Typically, a symmetric Dirichlet distribution is chosen, reflecting ignorance about which states are inherently more likely than others", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0087", "text": ". The single parameter of this distribution (termed the concentration parameter) controls the relative density or sparseness of the resulting transition matrix. A choice of 1 yields a uniform distribution. Values greater than 1 produce a dense matrix, in which the transition probabilities between pairs of states are likely to be nearly equal", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0088", "text": ". Values less than 1 result in a sparse matrix in which, for each given source state, only a small number of destination states have non-negligible transition probabilities. It is also possible to use a two-level prior Dirichlet distribution, in which one Dirichlet distribution (the upper distribution) governs the parameters of another Dirichlet distribution (the lower distribution), which in turn governs the transition probabilities", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0089", "text": ". The upper distribution governs the overall distribution of states, determining how likely each state is to occur; its concentration parameter determines the density or sparseness of states", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0090", "text": ". Such a two-level prior distribution, where both concentration parameters are set to produce sparse distributions, might be useful for example in unsupervised part-of-speech tagging, where some parts of speech occur much more commonly than others; learning algorithms that assume a uniform prior distribution generally perform poorly on this task", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0091", "text": ". The parameters of models of this sort, with non-uniform prior distributions, can be learned using Gibbs sampling or extended versions of the expectation-maximization algorithm.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0092", "text": "An extension of the previously described hidden Markov models with Dirichlet priors uses a Dirichlet process in place of a Dirichlet distribution. This type of model allows for an unknown and potentially infinite number of states. It is common to use a two-level Dirichlet process, similar to the previously described model with two levels of Dirichlet distributions. Such a model is called a hierarchical Dirichlet process hidden Markov model, or HDP-HMM for short", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0093", "text": ". It was originally described under the name \"Infinite Hidden Markov Model\" and was further formalized in \"Hierarchical Dirichlet Processes\".", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0094", "text": "Discriminative approach", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0095", "text": "A different type of extension uses a discriminative model in place of the generative model of standard HMMs. This type of model directly models the conditional distribution of the hidden states given the observations, rather than modeling the joint distribution. An example of this model is the so-called maximum entropy Markov model (MEMM), which models the conditional distribution of the states using logistic regression (also known as a \"maximum entropy model\")", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0096", "text": ". The advantage of this type of model is that arbitrary features (i.e. functions) of the observations can be modeled, allowing domain-specific knowledge of the problem at hand to be injected into the model", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0097", "text": ". Models of this sort are not limited to modeling direct dependencies between a hidden state and its associated observation; rather, features of nearby observations, of combinations of the associated observation and nearby observations, or in fact of arbitrary observations at any distance from a given hidden state can be included in the process used to determine the value of a hidden state", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0098", "text": ". Furthermore, there is no need for these features to be statistically independent of each other, as would be the case if such features were used in a generative model. Finally, arbitrary features over pairs of adjacent hidden states can be used rather than simple transition probabilities", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0099", "text": ". The disadvantages of such models are: (1) The types of prior distributions that can be placed on hidden states are severely limited; (2) It is not possible to predict the probability of seeing an arbitrary observation. This second limitation is often not an issue in practice, since many common usages of HMM's do not require such predictive probabilities.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0100", "text": "A variant of the previously described discriminative model is the linear-chain conditional random field. This uses an undirected graphical model (aka Markov random field) rather than the directed graphical models of MEMM's and similar models. The advantage of this type of model is that it does not suffer from the so-called label bias problem of MEMM's, and thus may make more accurate predictions. The disadvantage is that training can be slower than for MEMM's.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0101", "text": "Other extensions\nYet another variant is the factorial hidden Markov model, which allows for a single observation to be conditioned on the corresponding hidden variables of a set of \n  \n    \n      \n        K\n      \n    \n    {\\displaystyle K}\n  \n independent Markov chains, rather than a single Markov chain. It is equivalent to a single HMM, with \n  \n    \n      \n        \n          N\n          \n            K\n          \n        \n      \n    \n    {\\displaystyle N^{K}}\n  \n states (assuming there are", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0102", "text": "states (assuming there are \n  \n    \n      \n        N\n      \n    \n    {\\displaystyle N}\n  \n states for each chain), and therefore, learning in such a model is difficult: for a sequence of length \n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}\n  \n, a straightforward Viterbi algorithm has complexity \n  \n    \n      \n        O\n        (\n        \n          N\n          \n            2\n            K\n          \n        \n        \n        T\n        )", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0103", "text": "T\n        )\n      \n    \n    {\\displaystyle O(N^{2K}\\,T)}\n  \n. To find an exact solution, a junction tree algorithm could be used, but it results in an \n  \n    \n      \n        O\n        (\n        \n          N\n          \n            K\n            +\n            1\n          \n        \n        \n        K\n        \n        T\n        )\n      \n    \n    {\\displaystyle O(N^{K+1}\\,K\\,T)}", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0104", "text": "{\\displaystyle O(N^{K+1}\\,K\\,T)}\n  \n complexity. In practice, approximate techniques, such as variational approaches, could be used.\nAll of the above models can be extended to allow for more distant dependencies among hidden states, e.g. allowing for a given state to be dependent on the previous two or three states rather than a single previous state; i.e. the transition probabilities are extended to encompass sets of three or four adjacent states (or in general", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0105", "text": "K\n      \n    \n    {\\displaystyle K}\n  \n adjacent states). The disadvantage of such models is that dynamic-programming algorithms for training them have an \n  \n    \n      \n        O\n        (\n        \n          N\n          \n            K\n          \n        \n        \n        T\n        )\n      \n    \n    {\\displaystyle O(N^{K}\\,T)}\n  \n running time, for \n  \n    \n      \n        K\n      \n    \n    {\\displaystyle K}\n  \n adjacent states and \n  \n    \n      \n        T", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0106", "text": "T\n      \n    \n    {\\displaystyle T}\n  \n total observations (i.e. a length-\n  \n    \n      \n        T\n      \n    \n    {\\displaystyle T}\n  \n Markov chain). This extension has been widely used in bioinformatics, in the modeling of DNA sequences.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0107", "text": "Another recent extension is the triplet Markov model, in which an auxiliary underlying process is added to model some data specificities. Many variants of this model have been proposed. One should also mention the interesting link that has been established between the theory of evidence and the triplet Markov models and which allows to fuse data in Markovian context and to model nonstationary data", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0108", "text": ". Alternative multi-stream data fusion strategies have also been proposed in recent literature, e.g.,", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0109", "text": "Finally, a different rationale towards addressing the problem of modeling nonstationary data by means of hidden Markov models was suggested in 2012. It consists in employing a small recurrent neural network (RNN), specifically a reservoir network, to capture the evolution of the temporal dynamics in the observed data. This information, encoded in the form of a high-dimensional vector, is used as a conditioning variable of the HMM state transition probabilities", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0110", "text": ". Under such a setup, eventually is obtained a nonstationary HMM, the transition probabilities of which evolve over time in a manner that is inferred from the data, in contrast to some unrealistic ad-hoc model of temporal evolution.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0111", "text": "In 2023, two innovative algorithms were introduced for the Hidden Markov Model. These algorithms enable the computation of the posterior distribution of the HMM without the necessity of explicitly modeling the joint distribution, utilizing only the conditional distributions", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0112", "text": ". Unlike traditional methods such as the Forward-Backward and Viterbi algorithms, which require knowledge of the joint law of the HMM and can be computationally intensive to learn, the Discriminative Forward-Backward and Discriminative Viterbi algorithms circumvent the need for the observation's law. This breakthrough allows the HMM to be applied as a discriminative model, offering a more efficient and versatile approach to leveraging Hidden Markov Models in various applications.", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0113", "text": "The model suitable in the context of longitudinal data is named latent Markov model. The basic version of this model has been extended to include individual covariates, random effects and to model more complex data structures such as multilevel data. A complete overview of the latent Markov models, with special attention to the model assumptions and  to their practical use is provided in", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0114", "text": "Measure theory\nGiven a Markov transition matrix and an invariant distribution on the states, a probability measure can be imposed on the set of subshifts. For example, consider the Markov chain given on the right on the states \n  \n    \n      \n        A\n        ,\n        \n          B\n          \n            1\n          \n        \n        ,\n        \n          B\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle A,B_{1},B_{2}}\n  \n, with invariant distribution", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0115", "text": ", with invariant distribution \n  \n    \n      \n        \u03c0\n        =\n        (\n        2\n        \n          /\n        \n        7\n        ,\n        4\n        \n          /\n        \n        7\n        ,\n        1\n        \n          /\n        \n        7\n        )\n      \n    \n    {\\displaystyle \\pi =(2/7,4/7,1/7)}\n  \n. By ignoring the distinction between \n  \n    \n      \n        \n          B\n          \n            1\n          \n        \n        ,\n        \n          B\n          \n            2", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0116", "text": "B\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle B_{1},B_{2}}\n  \n, this space of subshifts is projected on \n  \n    \n      \n        A\n        ,\n        \n          B\n          \n            1\n          \n        \n        ,\n        \n          B\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle A,B_{1},B_{2}}\n  \n into another space of subshifts on \n  \n    \n      \n        A\n        ,\n        B\n      \n    \n    {\\displaystyle A,B}", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0117", "text": "B\n      \n    \n    {\\displaystyle A,B}\n  \n, and this projection also projects the probability measure down to a probability measure on the subshifts on \n  \n    \n      \n        A\n        ,\n        B\n      \n    \n    {\\displaystyle A,B}\n  \n.\nThe curious thing is that the probability measure on the subshifts on \n  \n    \n      \n        A\n        ,\n        B\n      \n    \n    {\\displaystyle A,B}\n  \n is not created by a Markov chain on \n  \n    \n      \n        A\n        ,\n        B", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0118", "text": "A\n        ,\n        B\n      \n    \n    {\\displaystyle A,B}\n  \n, not even multiple orders. Intuitively, this is because if one observes a long sequence of \n  \n    \n      \n        \n          B\n          \n            n\n          \n        \n      \n    \n    {\\displaystyle B^{n}}\n  \n, then one would become increasingly sure that the \n  \n    \n      \n        Pr\n        (\n        A\n        \u2223\n        \n          B\n          \n            n\n          \n        \n        )\n        \u2192", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0119", "text": ")\n        \u2192\n        \n          \n            2\n            3\n          \n        \n      \n    \n    {\\displaystyle \\Pr(A\\mid B^{n})\\to {\\frac {2}{3}}}\n  \n, meaning that the observable part of the system can be affected by something infinitely in the past.\nConversely, there exists a space of subshifts on 6 symbols, projected to subshifts on 2 symbols, such that any Markov measure on the smaller subshift has a preimage measure that is not Markov of any order (example 2.6).", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0120", "text": "See also\nReferences\nExternal links\nConcepts\nTeif, V. B.; Rippe, K. (2010). \"Statistical\u2013mechanical lattice models for protein\u2013DNA binding in chromatin\". J. Phys.: Condens. Matter. 22 (41) 414105. arXiv:1004.5514. Bibcode:2010JPCM...22O4105T. doi:10.1088/0953-8984/22/41/414105. PMID 21386588. S2CID 103345.\nA Revealing Introduction to Hidden Markov Models by Mark Stamp, San Jose State University.\nFitting HMM's with expectation-maximization \u2013 complete derivation", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Hidden_Markov_model_0121", "text": "A step-by-step tutorial on HMMs Archived 2017-08-13 at the Wayback Machine (University of Leeds)\nHidden Markov Models (an exposition using basic mathematics)\nHidden Markov Models (by Narada Warakagoda)\nHidden Markov Models: Fundamentals and Applications Part 1, Part 2 (by V. Petrushin)\nLecture on a Spreadsheet by Jason Eisner, Video and interactive spreadsheet", "title": "Hidden Markov model", "url": "https://en.wikipedia.org/wiki/Hidden_Markov_model"}, {"id": "rec_Cosine_similarity_0000", "text": "In data analysis, cosine similarity is a measure of similarity between two non-zero vectors defined in an inner product space. Cosine similarity is the cosine of the angle between the vectors; that is, it is the dot product of the vectors divided by the product of their lengths. It follows that the cosine similarity does not depend on the magnitudes of the vectors, but only on their angle. The cosine similarity always belongs to the interval \n  \n    \n      \n        [\n        \u2212\n        1", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0001", "text": "[\n        \u2212\n        1\n        ,\n        +\n        1\n        ]\n        .\n      \n    \n    {\\displaystyle [-1,+1].}\n  \n For example, two proportional vectors have a cosine similarity of +1, two orthogonal vectors have a similarity of 0, and two opposite vectors have a similarity of \u22121. In some contexts, the component values of the vectors cannot be negative, in which case the cosine similarity is bounded in \n  \n    \n      \n        [\n        0\n        ,\n        1\n        ]", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0002", "text": "[\n        0\n        ,\n        1\n        ]\n      \n    \n    {\\displaystyle [0,1]}\n  \n.\nFor example, in information retrieval and text mining, each word is assigned a different coordinate and a document is represented by the vector of the numbers of occurrences of each word in the document. Cosine similarity then gives a useful measure of how similar two documents are likely to be, in terms of their subject matter, and independently of the length of the documents.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0003", "text": "The technique is also used to measure cohesion within clusters in the field of data mining.\nOne advantage of cosine similarity is its low complexity, especially for sparse vectors: only the non-zero coordinates need to be considered.\nOther names for cosine similarity include Orchini similarity and Tucker coefficient of congruence; the Otsuka\u2013Ochiai similarity (see below) is cosine similarity applied to binary data.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0004", "text": "Definition\nThe cosine of two non-zero vectors can be derived by using the Euclidean dot product formula:", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0005", "text": "A\n        \n        \u22c5\n        \n          B\n        \n        =\n        \n          \u2016\n          \n            A\n          \n          \u2016\n        \n        \n          \u2016\n          \n            B\n          \n          \u2016\n        \n        cos\n        \u2061\n        \u03b8\n      \n    \n    {\\displaystyle \\mathbf {A} \\cdot \\mathbf {B} =\\left\\|\\mathbf {A} \\right\\|\\left\\|\\mathbf {B} \\right\\|\\cos \\theta }", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0006", "text": "Given two n-dimensional vectors of attributes, A and B, the cosine similarity, cos(\u03b8), is represented using a dot product and magnitude as", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0007", "text": "cosine similarity\n        \n        =\n        \n          S\n          \n            C\n          \n        \n        (\n        A\n        ,\n        B\n        )\n        :=\n        cos\n        \u2061\n        (\n        \u03b8\n        )\n        =\n        \n          \n            \n              \n                A\n              \n              \u22c5\n              \n                B\n              \n            \n            \n              \u2016\n              \n                A", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0008", "text": "A\n              \n              \u2016\n              \u2016\n              \n                B\n              \n              \u2016\n            \n          \n        \n        =\n        \n          \n            \n              \n                \u2211\n                \n                  i\n                  =\n                  1\n                \n                \n                  n\n                \n              \n              \n                \n                  A", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0009", "text": "A\n                  \n                    i\n                  \n                \n                \n                  B\n                  \n                    i\n                  \n                \n              \n            \n            \n              \n                \n                  \n                    \u2211\n                    \n                      i\n                      =\n                      1\n                    \n                    \n                      n", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0010", "text": "n\n                    \n                  \n                  \n                    \n                      A\n                      \n                        i\n                      \n                      \n                        2\n                      \n                    \n                  \n                \n              \n              \u22c5\n              \n                \n                  \n                    \u2211\n                    \n                      i", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0011", "text": "i\n                      =\n                      1\n                    \n                    \n                      n\n                    \n                  \n                  \n                    \n                      B\n                      \n                        i\n                      \n                      \n                        2", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0012", "text": ",\n      \n    \n    {\\displaystyle {\\text{cosine similarity}}=S_{C}(A,B):=\\cos(\\theta )={\\mathbf {A} \\cdot \\mathbf {B}  \\over \\|\\mathbf {A} \\|\\|\\mathbf {B} \\|}={\\frac {\\sum \\limits _{i=1}^{n}{A_{i}B_{i}}}{{\\sqrt {\\sum \\limits _{i=1}^{n}{A_{i}^{2}}}}\\cdot {\\sqrt {\\sum \\limits _{i=1}^{n}{B_{i}^{2}}}}}},}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0013", "text": "where \n  \n    \n      \n        \n          A\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle A_{i}}\n  \n and \n  \n    \n      \n        \n          B\n          \n            i\n          \n        \n      \n    \n    {\\displaystyle B_{i}}\n  \n are the \n  \n    \n      \n        i\n      \n    \n    {\\displaystyle i}\n  \nth components of vectors \n  \n    \n      \n        \n          A\n        \n      \n    \n    {\\displaystyle \\mathbf {A} }\n  \n and \n  \n    \n      \n        \n          B", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0014", "text": "B\n        \n      \n    \n    {\\displaystyle \\mathbf {B} }\n  \n, respectively.\nThe resulting similarity ranges from \u22121 meaning exactly opposite, to +1 meaning exactly the same, with 0 indicating orthogonality or decorrelation, while in-between values indicate intermediate similarity or dissimilarity.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0015", "text": "For text matching, the attribute vectors A and B are usually the term frequency vectors of the documents. Cosine similarity can be seen as a method of normalizing document length during comparison. In the case of information retrieval, the cosine similarity of two documents will range from \n  \n    \n      \n        0\n        \u2192\n        1\n      \n    \n    {\\displaystyle 0\\to 1}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0016", "text": "{\\displaystyle 0\\to 1}\n  \n, since the term frequencies cannot be negative. This remains true when using TF-IDF weights. The angle between two term frequency vectors cannot be greater than 90\u00b0.\nIf the attribute vectors are normalized by subtracting the vector means (e.g., \n  \n    \n      \n        A\n        \u2212\n        \n          \n            \n              A\n              \u00af\n            \n          \n        \n      \n    \n    {\\displaystyle A-{\\bar {A}}}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0017", "text": "{\\displaystyle A-{\\bar {A}}}\n  \n), the measure is called the centered cosine similarity and is equivalent to the Pearson correlation coefficient. For an example of centering,", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0018", "text": "if\n        \n        \n        A\n        =\n        [\n        \n          A\n          \n            1\n          \n        \n        ,\n        \n          A\n          \n            2\n          \n        \n        \n          ]\n          \n            T\n          \n        \n        ,\n        \n           then \n        \n        \n          \n            \n              A\n              \u00af\n            \n          \n        \n        =\n        \n          \n            [", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0019", "text": "[\n            \n              \n                \n                  \n                    (\n                    \n                      A\n                      \n                        1\n                      \n                    \n                    +\n                    \n                      A\n                      \n                        2\n                      \n                    \n                    )\n                  \n                  2", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0020", "text": "2\n                \n              \n              ,\n              \n                \n                  \n                    (\n                    \n                      A\n                      \n                        1\n                      \n                    \n                    +\n                    \n                      A\n                      \n                        2\n                      \n                    \n                    )\n                  \n                  2", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0021", "text": "2\n                \n              \n            \n            ]\n          \n          \n            T\n          \n        \n        ,\n      \n    \n    {\\displaystyle {\\text{if}}\\,A=[A_{1},A_{2}]^{T},{\\text{ then }}{\\bar {A}}=\\left[{\\frac {(A_{1}+A_{2})}{2}},{\\frac {(A_{1}+A_{2})}{2}}\\right]^{T},}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0022", "text": "so \n        \n        A\n        \u2212\n        \n          \n            \n              A\n              \u00af\n            \n          \n        \n        =\n        \n          \n            [\n            \n              \n                \n                  \n                    (\n                    \n                      A\n                      \n                        1\n                      \n                    \n                    \u2212", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0023", "text": "\u2212\n                    \n                      A\n                      \n                        2\n                      \n                    \n                    )\n                  \n                  2\n                \n              \n              ,\n              \n                \n                  \n                    (\n                    \u2212\n                    \n                      A\n                      \n                        1", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0024", "text": "1\n                      \n                    \n                    +\n                    \n                      A\n                      \n                        2\n                      \n                    \n                    )\n                  \n                  2\n                \n              \n            \n            ]\n          \n          \n            T\n          \n        \n        .", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0025", "text": ".\n      \n    \n    {\\displaystyle {\\text{ so }}A-{\\bar {A}}=\\left[{\\frac {(A_{1}-A_{2})}{2}},{\\frac {(-A_{1}+A_{2})}{2}}\\right]^{T}.}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0026", "text": "Cosine distance\nWhen the distance between two unit-length vectors is defined to be the length of their vector difference then", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0027", "text": "dist\n        \u2061\n        (\n        \n          A\n        \n        ,\n        \n          B\n        \n        )\n        =\n        \n          \n            (\n            \n              A\n            \n            \u2212\n            \n              B\n            \n            )\n            \u22c5\n            (\n            \n              A\n            \n            \u2212\n            \n              B\n            \n            )\n          \n        \n        =", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0028", "text": "=\n        \n          \n            \n              A\n            \n            \u22c5\n            \n              A\n            \n            \u2212\n            2\n            (\n            \n              A\n            \n            \u22c5\n            \n              B\n            \n            )\n            +\n            \n              B\n            \n            \u22c5\n            \n              B\n            \n          \n        \n        =\n        \n          \n            2\n            (\n            1\n            \u2212", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0029", "text": "(\n            1\n            \u2212\n            \n              S\n              \n                C\n              \n            \n            (\n            \n              A\n            \n            ,\n            \n              B\n            \n            )\n            )\n          \n        \n        \n        .", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0030", "text": ".\n      \n    \n    {\\displaystyle \\operatorname {dist} (\\mathbf {A} ,\\mathbf {B} )={\\sqrt {(\\mathbf {A} -\\mathbf {B} )\\cdot (\\mathbf {A} -\\mathbf {B} )}}={\\sqrt {\\mathbf {A} \\cdot \\mathbf {A} -2(\\mathbf {A} \\cdot \\mathbf {B} )+\\mathbf {B} \\cdot \\mathbf {B} }}={\\sqrt {2(1-S_{C}(\\mathbf {A} ,\\mathbf {B} ))}}\\,.}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0031", "text": "Nonetheless the cosine distance is often defined without the square root or factor of 2:", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0032", "text": "cosine distance\n        \n        =\n        \n          D\n          \n            C\n          \n        \n        (\n        A\n        ,\n        B\n        )\n        :=\n        1\n        \u2212\n        \n          S\n          \n            C\n          \n        \n        (\n        A\n        ,\n        B\n        )\n        \n        .\n      \n    \n    {\\displaystyle {\\text{cosine distance}}=D_{C}(A,B):=1-S_{C}(A,B)\\,.}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0033", "text": "It is important to note that, by virtue of being proportional to squared Euclidean distance, the cosine distance is not a true distance metric; it does not exhibit the triangle inequality property \u2014 or, more formally, the Schwarz inequality \u2014 and it violates the coincidence axiom.  To repair the triangle inequality property while maintaining the same ordering, one can convert to Euclidean distance \n  \n    \n      \n        \n          \n            2\n            (\n            1\n            \u2212", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0034", "text": "(\n            1\n            \u2212\n            \n              S\n              \n                C\n              \n            \n            (\n            A\n            ,\n            B\n            )\n            )\n          \n        \n      \n    \n    {\\textstyle {\\sqrt {2(1-S_{C}(A,B))}}}\n  \n or angular distance \u03b8 = arccos(SC(A, B)). Alternatively, the triangular inequality that does work for angular distances can be expressed directly in terms of the cosines; see below.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0035", "text": "Angular distance and similarity\nThe normalized angle, referred to as angular distance, between any two vectors \n  \n    \n      \n        A\n      \n    \n    {\\displaystyle A}\n  \n and \n  \n    \n      \n        B\n      \n    \n    {\\displaystyle B}\n  \n is a formal distance metric and can be calculated from the cosine similarity. The complement of the angular distance metric can then be used to define angular similarity function bounded between 0 and 1, inclusive.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0036", "text": "When the vector elements may be positive or negative:", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0037", "text": "angular distance\n        \n        =\n        \n          D\n          \n            \u03b8\n          \n        \n        :=\n        \n          \n            \n              arccos\n              \u2061\n              (\n              \n                cosine similarity\n              \n              )\n            \n            \u03c0\n          \n        \n        =\n        \n          \n            \u03b8\n            \u03c0", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0038", "text": "\u03c0\n          \n        \n      \n    \n    {\\displaystyle {\\text{angular distance}}=D_{\\theta }:={\\frac {\\arccos({\\text{cosine similarity}})}{\\pi }}={\\frac {\\theta }{\\pi }}}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0039", "text": "angular similarity\n        \n        =\n        \n          S\n          \n            \u03b8\n          \n        \n        :=\n        1\n        \u2212\n        \n          angular distance\n        \n        =\n        1\n        \u2212\n        \n          \n            \u03b8\n            \u03c0\n          \n        \n      \n    \n    {\\displaystyle {\\text{angular similarity}}=S_{\\theta }:=1-{\\text{angular distance}}=1-{\\frac {\\theta }{\\pi }}}\n  \n\nOr, if the vector elements are always positive:", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0040", "text": "angular distance\n        \n        =\n        \n          D\n          \n            \u03b8\n          \n        \n        :=\n        \n          \n            \n              2\n              \u22c5\n              arccos\n              \u2061\n              (\n              \n                cosine similarity\n              \n              )\n            \n            \u03c0\n          \n        \n        =\n        \n          \n            \n              2\n              \u03b8\n            \n            \u03c0", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0041", "text": "\u03b8\n            \n            \u03c0\n          \n        \n      \n    \n    {\\displaystyle {\\text{angular distance}}=D_{\\theta }:={\\frac {2\\cdot \\arccos({\\text{cosine similarity}})}{\\pi }}={\\frac {2\\theta }{\\pi }}}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0042", "text": "angular similarity\n        \n        =\n        \n          S\n          \n            \u03b8\n          \n        \n        :=\n        1\n        \u2212\n        \n          angular distance\n        \n        =\n        1\n        \u2212\n        \n          \n            \n              2\n              \u03b8\n            \n            \u03c0\n          \n        \n      \n    \n    {\\displaystyle {\\text{angular similarity}}=S_{\\theta }:=1-{\\text{angular distance}}=1-{\\frac {2\\theta }{\\pi }}}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0043", "text": "Unfortunately, computing the inverse cosine (arccos) function is slow, making the use of the angular distance more computationally expensive than using the more common (but not metric) cosine distance above.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0044", "text": "L2-normalized Euclidean distance\nAnother effective proxy for cosine distance can be obtained by \n  \n    \n      \n        \n          L\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle L_{2}}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0045", "text": "normalisation of the vectors, followed by the application of normal Euclidean distance. Using this technique each term in each vector is first divided by the magnitude of the vector, yielding a vector of unit length", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0046", "text": ". Then the Euclidean distance over the end-points of any two vectors is a proper metric which gives the same ordering as the cosine distance (a monotonic transformation of Euclidean distance; see below) for any comparison of vectors, and furthermore avoids the potentially expensive trigonometric operations required to yield a proper metric", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0047", "text": ". Once the normalisation has occurred, the vector space can be used with the full range of techniques available to any Euclidean space, notably standard dimensionality reduction techniques. This normalised form distance is often used within many deep learning algorithms.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0048", "text": "Otsuka\u2013Ochiai coefficient\nIn biology, there is a similar concept known as the Otsuka\u2013Ochiai coefficient named after Yanosuke Otsuka (also spelled as \u014ctsuka, Ootsuka or Otuka, Japanese: \u5927\u585a \u5f25\u4e4b\u52a9) and Akira Ochiai (Japanese: \u843d\u5408 \u660e), also known as the Ochiai\u2013Barkman or Ochiai coefficient, which can be represented as:", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0049", "text": "K\n        =\n        \n          \n            \n              \n                |\n              \n              A\n              \u2229\n              B\n              \n                |\n              \n            \n            \n              \n                |\n              \n              A\n              \n                |\n              \n              \u00d7\n              \n                |\n              \n              B\n              \n                |", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0050", "text": "|\n              \n            \n          \n        \n      \n    \n    {\\displaystyle K={\\frac {|A\\cap B|}{\\sqrt {|A|\\times |B|}}}}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0051", "text": "Here, \n  \n    \n      \n        A\n      \n    \n    {\\displaystyle A}\n  \n and \n  \n    \n      \n        B\n      \n    \n    {\\displaystyle B}\n  \n are sets, and \n  \n    \n      \n        \n          |\n        \n        A\n        \n          |\n        \n      \n    \n    {\\displaystyle |A|}\n  \n is the number of elements in \n  \n    \n      \n        A\n      \n    \n    {\\displaystyle A}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0052", "text": "A\n      \n    \n    {\\displaystyle A}\n  \n. If sets are represented as bit vectors, the Otsuka\u2013Ochiai coefficient can be seen to be the same as the cosine similarity. It is identical to the score introduced by Godfrey Thomson.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0053", "text": "In a recent book, the coefficient is tentatively misattributed to another Japanese researcher with the family name Otsuka. The confusion arises because in 1957 Akira Ochiai attributes the coefficient only to Otsuka (no first name mentioned) by citing an article by Ikuso Hamai (Japanese: \u6d5c\u4e95 \u751f\u4e09), who in turn cites the original 1936 article by Yanosuke Otsuka.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0054", "text": "Properties\nThe most noteworthy property of cosine similarity is that it reflects a relative, rather than absolute, comparison of the individual vector dimensions. For any positive constant \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n and vector \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n, the vectors \n  \n    \n      \n        V\n      \n    \n    {\\displaystyle V}\n  \n and \n  \n    \n      \n        a\n        V\n      \n    \n    {\\displaystyle aV}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0055", "text": "V\n      \n    \n    {\\displaystyle aV}\n  \n are maximally similar. The measure is thus most appropriate for data where frequency is more important than absolute values; notably, term frequency in documents.\nHowever more recent metrics with a grounding in information theory, such as Jensen\u2013Shannon, SED, and triangular divergence have been shown to have improved semantics in at least some contexts.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0056", "text": "Cosine similarity is related to Euclidean distance as follows. Denote Euclidean distance by the usual \n  \n    \n      \n        \u2016\n        A\n        \u2212\n        B\n        \u2016\n      \n    \n    {\\displaystyle \\|A-B\\|}\n  \n, and observe that", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0057", "text": "\u2016\n        A\n        \u2212\n        B\n        \n          \u2016\n          \n            2\n          \n        \n        =\n        (\n        A\n        \u2212\n        B\n        )\n        \u22c5\n        (\n        A\n        \u2212\n        B\n        )\n        =\n        \u2016\n        A\n        \n          \u2016\n          \n            2\n          \n        \n        +\n        \u2016\n        B\n        \n          \u2016\n          \n            2\n          \n        \n        \u2212\n        2\n        (\n        A\n        \u22c5\n        B", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0058", "text": "2\n        (\n        A\n        \u22c5\n        B\n        )\n         \n      \n    \n    {\\displaystyle \\|A-B\\|^{2}=(A-B)\\cdot (A-B)=\\|A\\|^{2}+\\|B\\|^{2}-2(A\\cdot B)\\ }\n  \n (polarization identity)\nby expansion. When A and B are normalized to unit length, \n  \n    \n      \n        \u2016\n        A\n        \n          \u2016\n          \n            2\n          \n        \n        =\n        \u2016\n        B\n        \n          \u2016\n          \n            2\n          \n        \n        =\n        1", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0059", "text": "=\n        1\n      \n    \n    {\\displaystyle \\|A\\|^{2}=\\|B\\|^{2}=1}\n  \n so this expression is equal to", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0060", "text": "2\n        (\n        1\n        \u2212\n        cos\n        \u2061\n        (\n        A\n        ,\n        B\n        )\n        )\n        .\n      \n    \n    {\\displaystyle 2(1-\\cos(A,B)).}\n  \n\nIn short, the cosine distance can be expressed in terms of Euclidean distance as", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0061", "text": "D\n          \n            C\n          \n        \n        (\n        A\n        ,\n        B\n        )\n        =\n        \n          \n            \n              \u2016\n              A\n              \u2212\n              B\n              \n                \u2016\n                \n                  2\n                \n              \n            \n            2\n          \n        \n        \n        \n          w\n          h\n          e\n          n\n        \n        \n        \u2016\n        A", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0062", "text": "n\n        \n        \n        \u2016\n        A\n        \n          \u2016\n          \n            2\n          \n        \n        =\n        \u2016\n        B\n        \n          \u2016\n          \n            2\n          \n        \n        =\n        1\n      \n    \n    {\\displaystyle D_{C}(A,B)={\\frac {\\|A-B\\|^{2}}{2}}\\quad \\mathrm {when} \\quad \\|A\\|^{2}=\\|B\\|^{2}=1}\n  \n.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0063", "text": ".\nThe Euclidean distance is called the chord distance (because it is the length of the chord on the unit circle) and it is the Euclidean distance between the vectors which were normalized to unit sum of squared values within them.\nNull distribution: For data which can be negative as well as positive, the null distribution for cosine similarity is the distribution of the dot product of two independent random unit vectors. This distribution has a mean of zero and a variance of", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0064", "text": "1\n        \n          /\n        \n        n\n      \n    \n    {\\displaystyle 1/n}\n  \n (where \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n is the number of dimensions), and although the distribution is bounded between \u22121 and +1, as \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0065", "text": "n\n      \n    \n    {\\displaystyle n}\n  \n grows large the distribution is increasingly well-approximated by the normal distribution. Other types of data such as bitstreams, which only take the values 0 or 1, the null distribution takes a different form and may have a nonzero mean.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0066", "text": "Triangle inequality for cosine similarity\nThe ordinary triangle inequality for angles (i.e., arc lengths on a unit hypersphere) gives us that", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0067", "text": "|\n        \n         \n        \u2220\n        \n          A\n          C\n        \n        \u2212\n        \u2220\n        \n          C\n          B\n        \n         \n        \n          |\n        \n        \u2264\n         \n        \u2220\n        \n          A\n          B\n        \n         \n        \u2264\n         \n        \u2220\n        \n          A\n          C\n        \n         \n        +\n         \n        \u2220\n        \n          C\n          B\n        \n         \n        .", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0068", "text": ".\n      \n    \n    {\\displaystyle |~\\angle {AC}-\\angle {CB}~|\\leq ~\\angle {AB}~\\leq ~\\angle {AC}~+~\\angle {CB}~.}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0069", "text": "Because the cosine function decreases as an angle in [0, \u03c0] radians increases, the sense of these inequalities is reversed when we take the cosine of each value:", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0070", "text": "cos\n        \u2061\n        (\n        \u2220\n        \n          A\n          C\n        \n        \u2212\n        \u2220\n        \n          C\n          B\n        \n        )\n        \u2265\n        cos\n        \u2061\n        (\n        \u2220\n        \n          A\n          B\n        \n        )\n        \u2265\n        cos\n        \u2061\n        (\n        \u2220\n        \n          A\n          C\n        \n        +\n        \u2220\n        \n          C\n          B\n        \n        )\n        .", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0071", "text": ")\n        .\n      \n    \n    {\\displaystyle \\cos(\\angle {AC}-\\angle {CB})\\geq \\cos(\\angle {AB})\\geq \\cos(\\angle {AC}+\\angle {CB}).}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0072", "text": "Using the cosine addition and subtraction formulas, these two inequalities can be written in terms of the original cosines,", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0073", "text": "cos\n        \u2061\n        (\n        A\n        ,\n        C\n        )\n        \u22c5\n        cos\n        \u2061\n        (\n        C\n        ,\n        B\n        )\n        +\n        \n          \n            \n              (\n              \n                1\n                \u2212\n                cos\n                \u2061\n                (\n                A\n                ,\n                C\n                \n                  )\n                  \n                    2", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0074", "text": "2\n                  \n                \n              \n              )\n            \n            \u22c5\n            \n              (\n              \n                1\n                \u2212\n                cos\n                \u2061\n                (\n                C\n                ,\n                B\n                \n                  )\n                  \n                    2\n                  \n                \n              \n              )\n            \n          \n        \n        \u2265", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0075", "text": "\u2265\n        cos\n        \u2061\n        (\n        A\n        ,\n        B\n        )\n        ,\n      \n    \n    {\\displaystyle \\cos(A,C)\\cdot \\cos(C,B)+{\\sqrt {\\left(1-\\cos(A,C)^{2}\\right)\\cdot \\left(1-\\cos(C,B)^{2}\\right)}}\\geq \\cos(A,B),}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0076", "text": "cos\n        \u2061\n        (\n        A\n        ,\n        B\n        )\n        \u2265\n        cos\n        \u2061\n        (\n        A\n        ,\n        C\n        )\n        \u22c5\n        cos\n        \u2061\n        (\n        C\n        ,\n        B\n        )\n        \u2212\n        \n          \n            \n              (\n              \n                1\n                \u2212\n                cos\n                \u2061\n                (\n                A\n                ,\n                C", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0077", "text": "C\n                \n                  )\n                  \n                    2\n                  \n                \n              \n              )\n            \n            \u22c5\n            \n              (\n              \n                1\n                \u2212\n                cos\n                \u2061\n                (\n                C\n                ,\n                B\n                \n                  )\n                  \n                    2", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0078", "text": ")\n            \n          \n        \n        .\n      \n    \n    {\\displaystyle \\cos(A,B)\\geq \\cos(A,C)\\cdot \\cos(C,B)-{\\sqrt {\\left(1-\\cos(A,C)^{2}\\right)\\cdot \\left(1-\\cos(C,B)^{2}\\right)}}.}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0079", "text": "This form of the triangle inequality can be used to bound the minimum and maximum similarity of two objects A and B if the similarities to a reference object C is already known. This is used for example in metric data indexing, but has also been used to accelerate spherical k-means clustering the same way the Euclidean triangle inequality has been used to accelerate regular k-means.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0080", "text": "Soft cosine measure\nA soft cosine or (\"soft\" similarity) between two vectors considers similarities between pairs of features. The traditional cosine similarity considers the vector space model (VSM) features as independent or completely different, while the soft cosine measure proposes considering the similarity of features in VSM, which help generalize the concept of cosine (and soft cosine) as well as the idea of (soft) similarity.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0081", "text": "For example, in the field of natural language processing (NLP) the similarity among features is quite intuitive. Features such as words, n-grams, or syntactic n-grams can be quite similar, though formally they are considered as different features in the VSM. For example, words \"play\" and \"game\" are different words and thus mapped to different points in VSM; yet they are semantically related", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0082", "text": ". In case of n-grams or syntactic n-grams, Levenshtein distance can be applied (in fact, Levenshtein distance can be applied to words as well).", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0083", "text": "For calculating soft cosine, the matrix s is used to indicate similarity between features. It can be calculated through Levenshtein distance, WordNet similarity, or other similarity measures. Then we just multiply by this matrix.\nGiven two N-dimension vectors \n  \n    \n      \n        a\n      \n    \n    {\\displaystyle a}\n  \n and \n  \n    \n      \n        b\n      \n    \n    {\\displaystyle b}\n  \n, the soft cosine similarity is calculated as follows:", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0084", "text": "s\n                    o\n                    f\n                    t\n                    _\n                    c\n                    o\n                    s\n                    i\n                    n\n                    e\n                  \n                  \n                    1\n                  \n                \n                \u2061\n                (\n                a", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0085", "text": "(\n                a\n                ,\n                b\n                )\n                =\n                \n                  \n                    \n                      \n                        \u2211\n                        \n                          i\n                          ,\n                          j\n                        \n                        \n                          N\n                        \n                      \n                      \n                        s", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0086", "text": "s\n                        \n                          i\n                          j\n                        \n                      \n                      \n                        a\n                        \n                          i\n                        \n                      \n                      \n                        b\n                        \n                          j", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0087", "text": "\u2211\n                            \n                              i\n                              ,\n                              j\n                            \n                            \n                              N\n                            \n                          \n                          \n                            s", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0088", "text": "s\n                            \n                              i\n                              j\n                            \n                          \n                          \n                            a\n                            \n                              i\n                            \n                          \n                          \n                            a\n                            \n                              j", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0089", "text": "\u2211\n                            \n                              i\n                              ,\n                              j\n                            \n                            \n                              N", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0090", "text": "s\n                            \n                              i\n                              j\n                            \n                          \n                          \n                            b\n                            \n                              i\n                            \n                          \n                          \n                            b", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0091", "text": "j\n                            \n                          \n                        \n                      \n                    \n                  \n                \n                ,", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0092", "text": "{\\displaystyle {\\begin{aligned}\\operatorname {soft\\_cosine} _{1}(a,b)={\\frac {\\sum \\nolimits _{i,j}^{N}s_{ij}a_{i}b_{j}}{{\\sqrt {\\sum \\nolimits _{i,j}^{N}s_{ij}a_{i}a_{j}}}{\\sqrt {\\sum \\nolimits _{i,j}^{N}s_{ij}b_{i}b_{j}}}}},\\end{aligned}}}", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0093", "text": "where sij = similarity(featurei, featurej).\nIf there is no similarity between features (sii = 1, sij = 0 for i \u2260 j), the given equation is equivalent to the conventional cosine similarity formula.\nThe time complexity of this measure is quadratic, which makes it applicable to real-world tasks. Note that the complexity can be reduced to subquadratic. An efficient implementation of such soft cosine similarity is included in the Gensim open source library.", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Cosine_similarity_0094", "text": "See also\nS\u00f8rensen\u2013Dice coefficient\nHamming distance\nCorrelation\nJaccard index\nSimRank\nInformation retrieval\n\nReferences\nExternal links\nWeighted cosine measure\nA tutorial on cosine similarity using Python", "title": "Cosine similarity", "url": "https://en.wikipedia.org/wiki/Cosine_similarity"}, {"id": "rec_Euclidean_distance_0000", "text": "In mathematics, the Euclidean distance between two points in a Euclidean space is the length of the line segment between them. It can be calculated from the Cartesian coordinates of the points using the Pythagorean theorem, and therefore is occasionally called the Pythagorean distance.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0001", "text": "These names come from the ancient Greek mathematicians Euclid and Pythagoras. In the Greek deductive geometry exemplified by Euclid's Elements, distances were not represented as numbers but line segments of the same length, which were considered \"equal\". The notion of distance is inherent in the compass tool used to draw a circle, whose points all have the same distance from a common center point", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0002", "text": ". The connection from the Pythagorean theorem to distance calculation was not made until the 18th century.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0003", "text": "The distance between two objects that are not points is usually defined to be the smallest distance among pairs of points from the two objects. Formulas are known for computing distances between different types of objects, such as the distance from a point to a line. In advanced mathematics, the concept of distance has been generalized to abstract metric spaces, and other distances than Euclidean have been studied", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0004", "text": ". In some applications in statistics and optimization, the square of the Euclidean distance is used instead of the distance itself.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0005", "text": "Distance formulas\nOne dimension\nThe distance between any two points on the real line is the absolute value of the numerical difference of their coordinates, their absolute difference. Thus if \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n and \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n are two points on the real line, then the distance between them is given by:", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0006", "text": "d\n        (\n        p\n        ,\n        q\n        )\n        =\n        \n          |\n        \n        p\n        \u2212\n        q\n        \n          |\n        \n        .\n      \n    \n    {\\displaystyle d(p,q)=|p-q|.}\n  \n\nA more complicated formula, giving the same value, but generalizing more readily to higher dimensions, is:", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0007", "text": "d\n        (\n        p\n        ,\n        q\n        )\n        =\n        \n          \n            (\n            p\n            \u2212\n            q\n            \n              )\n              \n                2\n              \n            \n          \n        \n        .\n      \n    \n    {\\displaystyle d(p,q)={\\sqrt {(p-q)^{2}}}.}\n  \n\nIn this formula, squaring and then taking the square root leaves any positive number unchanged, but replaces any negative number by its absolute value.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0008", "text": "Two dimensions\nIn the Euclidean plane, let point \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n have Cartesian coordinates \n  \n    \n      \n        (\n        \n          p\n          \n            1\n          \n        \n        ,\n        \n          p\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle (p_{1},p_{2})}\n  \n and let point \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n have coordinates \n  \n    \n      \n        (", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0009", "text": "(\n        \n          q\n          \n            1\n          \n        \n        ,\n        \n          q\n          \n            2\n          \n        \n        )\n      \n    \n    {\\displaystyle (q_{1},q_{2})}\n  \n. Then the distance between \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n and \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n is given by:", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0010", "text": "d\n        (\n        p\n        ,\n        q\n        )\n        =\n        \n          \n            (\n            \n              p\n              \n                1\n              \n            \n            \u2212\n            \n              q\n              \n                1\n              \n            \n            \n              )\n              \n                2\n              \n            \n            +\n            (\n            \n              p\n              \n                2", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0011", "text": "p\n              \n                2\n              \n            \n            \u2212\n            \n              q\n              \n                2\n              \n            \n            \n              )\n              \n                2\n              \n            \n          \n        \n        .\n      \n    \n    {\\displaystyle d(p,q)={\\sqrt {(p_{1}-q_{1})^{2}+(p_{2}-q_{2})^{2}}}.}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0012", "text": "This can be seen by applying the Pythagorean theorem to a right triangle with horizontal and vertical sides, having the line segment from \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n to \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0013", "text": "q\n      \n    \n    {\\displaystyle q}\n  \n as its hypotenuse. The two squared formulas inside the square root give the areas of squares on the horizontal and vertical sides, and the outer square root converts the area of the square on the hypotenuse into the length of the hypotenuse. In terms of the Pythagorean addition operation \n  \n    \n      \n        \u2295\n      \n    \n    {\\displaystyle \\oplus }\n  \n, available in many software libraries as hypot, the same formula can be expressed as:", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0014", "text": "d\n        (\n        p\n        ,\n        q\n        )\n        =\n        (\n        \n          p\n          \n            1\n          \n        \n        \u2212\n        \n          q\n          \n            1\n          \n        \n        )\n        \u2295\n        (\n        \n          p\n          \n            2\n          \n        \n        \u2212\n        \n          q\n          \n            2\n          \n        \n        )\n        =\n        \n          \n            h\n            y\n            p", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0015", "text": "h\n            y\n            p\n            o\n            t\n          \n        \n        (\n        \n          p\n          \n            1\n          \n        \n        \u2212\n        \n          q\n          \n            1\n          \n        \n        ,\n        \n          p\n          \n            2\n          \n        \n        \u2212\n        \n          q\n          \n            2\n          \n        \n        )\n        .", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0016", "text": ")\n        .\n      \n    \n    {\\displaystyle d(p,q)=(p_{1}-q_{1})\\oplus (p_{2}-q_{2})={\\mathsf {hypot}}(p_{1}-q_{1},p_{2}-q_{2}).}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0017", "text": "It is also possible to compute the distance for points given by polar coordinates. If the polar coordinates of \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n are \n  \n    \n      \n        (\n        r\n        ,\n        \u03b8\n        )\n      \n    \n    {\\displaystyle (r,\\theta )}\n  \n and the polar coordinates of \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n are \n  \n    \n      \n        (\n        s\n        ,\n        \u03c8\n        )\n      \n    \n    {\\displaystyle (s,\\psi )}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0018", "text": "{\\displaystyle (s,\\psi )}\n  \n, then their distance is given by the law of cosines:", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0019", "text": "d\n        (\n        p\n        ,\n        q\n        )\n        =\n        \n          \n            \n              r\n              \n                2\n              \n            \n            +\n            \n              s\n              \n                2\n              \n            \n            \u2212\n            2\n            r\n            s\n            cos\n            \u2061\n            (\n            \u03b8\n            \u2212\n            \u03c8\n            )\n          \n        \n        .", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0020", "text": ".\n      \n    \n    {\\displaystyle d(p,q)={\\sqrt {r^{2}+s^{2}-2rs\\cos(\\theta -\\psi )}}.}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0021", "text": "When \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n and \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n are expressed as complex numbers in the complex plane, the same formula for one-dimensional points expressed as real numbers can be used, although here the absolute value sign indicates the complex norm:", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0022", "text": "d\n        (\n        p\n        ,\n        q\n        )\n        =\n        \n          |\n        \n        p\n        \u2212\n        q\n        \n          |\n        \n        .\n      \n    \n    {\\displaystyle d(p,q)=|p-q|.}\n\nHigher dimensions\nIn three dimensions, for points given by their Cartesian coordinates, the distance is", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0023", "text": "d\n        (\n        p\n        ,\n        q\n        )\n        =\n        \n          \n            (\n            \n              p\n              \n                1\n              \n            \n            \u2212\n            \n              q\n              \n                1\n              \n            \n            \n              )\n              \n                2\n              \n            \n            +\n            (\n            \n              p\n              \n                2", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0024", "text": "p\n              \n                2\n              \n            \n            \u2212\n            \n              q\n              \n                2\n              \n            \n            \n              )\n              \n                2\n              \n            \n            +\n            (\n            \n              p\n              \n                3\n              \n            \n            \u2212\n            \n              q\n              \n                3", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0025", "text": "3\n              \n            \n            \n              )\n              \n                2\n              \n            \n          \n        \n        .\n      \n    \n    {\\displaystyle d(p,q)={\\sqrt {(p_{1}-q_{1})^{2}+(p_{2}-q_{2})^{2}+(p_{3}-q_{3})^{2}}}.}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0026", "text": "In general, for points given by Cartesian coordinates in \n  \n    \n      \n        n\n      \n    \n    {\\displaystyle n}\n  \n-dimensional Euclidean space, the distance is", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0027", "text": "d\n        (\n        p\n        ,\n        q\n        )\n        =\n        \n          \n            (\n            \n              p\n              \n                1\n              \n            \n            \u2212\n            \n              q\n              \n                1\n              \n            \n            \n              )\n              \n                2\n              \n            \n            +\n            (\n            \n              p\n              \n                2", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0028", "text": "p\n              \n                2\n              \n            \n            \u2212\n            \n              q\n              \n                2\n              \n            \n            \n              )\n              \n                2\n              \n            \n            +\n            \u22ef\n            +\n            (\n            \n              p\n              \n                n\n              \n            \n            \u2212\n            \n              q\n              \n                n", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0029", "text": "q\n              \n                n\n              \n            \n            \n              )\n              \n                2\n              \n            \n          \n        \n        .\n      \n    \n    {\\displaystyle d(p,q)={\\sqrt {(p_{1}-q_{1})^{2}+(p_{2}-q_{2})^{2}+\\cdots +(p_{n}-q_{n})^{2}}}.}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0030", "text": "The Euclidean distance may also be expressed more compactly in terms of the Euclidean norm of the Euclidean vector difference:\n\n  \n    \n      \n        d\n        (\n        p\n        ,\n        q\n        )\n        =\n        \u2016\n        p\n        \u2212\n        q\n        \u2016\n        .\n      \n    \n    {\\displaystyle d(p,q)=\\|p-q\\|.}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0031", "text": "Objects other than points\nFor pairs of objects that are not both points, the distance can most simply be defined as the smallest distance between any two points from the two objects, although more complicated generalizations from points to sets such as Hausdorff distance are also commonly used. Formulas for computing distances between different types of objects include:", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0032", "text": "The distance from a point to a line, in the Euclidean plane\nThe distance from a point to a plane in three-dimensional Euclidean space\nThe distance between two lines in three-dimensional Euclidean space\nThe distance from a point to a curve can be used to define its parallel curve, another curve all of whose points have the same distance to the given curve.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0033", "text": "Properties\nThe Euclidean distance is the prototypical example of the distance in a metric space, and obeys all the defining properties of a metric space:", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0034", "text": "It is symmetric, meaning that for all points \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n and \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n, \n  \n    \n      \n        d\n        (\n        p\n        ,\n        q\n        )\n        =\n        d\n        (\n        q\n        ,\n        p\n        )\n      \n    \n    {\\displaystyle d(p,q)=d(q,p)}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0035", "text": "{\\displaystyle d(p,q)=d(q,p)}\n  \n. That is (unlike road distance with one-way streets) the distance between two points does not depend on which of the two points is the start and which is the destination.\nIt is positive, meaning that the distance between every two distinct points is a positive number, while the distance from any point to itself is zero.\nIt obeys the triangle inequality: for every three points \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n,", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0036", "text": "{\\displaystyle p}\n  \n, \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n, and \n  \n    \n      \n        r\n      \n    \n    {\\displaystyle r}\n  \n, \n  \n    \n      \n        d\n        (\n        p\n        ,\n        q\n        )\n        +\n        d\n        (\n        q\n        ,\n        r\n        )\n        \u2265\n        d\n        (\n        p\n        ,\n        r\n        )\n      \n    \n    {\\displaystyle d(p,q)+d(q,r)\\geq d(p,r)}\n  \n. Intuitively, traveling from", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0037", "text": ". Intuitively, traveling from \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n to \n  \n    \n      \n        r\n      \n    \n    {\\displaystyle r}\n  \n via \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n cannot be any shorter than traveling directly from \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n to \n  \n    \n      \n        r\n      \n    \n    {\\displaystyle r}\n  \n.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0038", "text": "r\n      \n    \n    {\\displaystyle r}\n  \n.\nAnother property, Ptolemy's inequality, concerns the Euclidean distances among four points \n  \n    \n      \n        p\n      \n    \n    {\\displaystyle p}\n  \n, \n  \n    \n      \n        q\n      \n    \n    {\\displaystyle q}\n  \n, \n  \n    \n      \n        r\n      \n    \n    {\\displaystyle r}\n  \n, and \n  \n    \n      \n        s\n      \n    \n    {\\displaystyle s}\n  \n. It states that", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0039", "text": "d\n        (\n        p\n        ,\n        q\n        )\n        \u22c5\n        d\n        (\n        r\n        ,\n        s\n        )\n        +\n        d\n        (\n        q\n        ,\n        r\n        )\n        \u22c5\n        d\n        (\n        p\n        ,\n        s\n        )\n        \u2265\n        d\n        (\n        p\n        ,\n        r\n        )\n        \u22c5\n        d\n        (\n        q\n        ,\n        s\n        )\n        .", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0040", "text": "s\n        )\n        .\n      \n    \n    {\\displaystyle d(p,q)\\cdot d(r,s)+d(q,r)\\cdot d(p,s)\\geq d(p,r)\\cdot d(q,s).}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0041", "text": "For points in the plane, this can be rephrased as stating that for every quadrilateral, the products of opposite sides of the quadrilateral sum to at least as large a number as the product of its diagonals. However, Ptolemy's inequality applies more generally to points in Euclidean spaces of any dimension, no matter how they are arranged. For points in metric spaces that are not Euclidean spaces, this inequality may not be true", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0042", "text": ". Euclidean distance geometry studies properties of Euclidean distance such as Ptolemy's inequality, and their application in testing whether given sets of distances come from points in a Euclidean space.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0043", "text": "According to the Beckman\u2013Quarles theorem, any transformation of the Euclidean plane or of a higher-dimensional Euclidean space that preserves unit distances must be an isometry, preserving all distances.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0044", "text": "Squared Euclidean distance\nIn many applications, and in particular when comparing distances, it may be more convenient to omit the final square root in the calculation of Euclidean distances, as the square root does not change the order (\n  \n    \n      \n        \n          d\n          \n            1\n          \n          \n            2\n          \n        \n        >\n        \n          d\n          \n            2\n          \n          \n            2", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0045", "text": "2\n          \n        \n      \n    \n    {\\displaystyle d_{1}^{2}>d_{2}^{2}}\n  \n if and only if \n  \n    \n      \n        \n          d\n          \n            1\n          \n        \n        >\n        \n          d\n          \n            2\n          \n        \n      \n    \n    {\\displaystyle d_{1}>d_{2}}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0046", "text": "). The value resulting from this omission is the square of the Euclidean distance, and is called the squared Euclidean distance. For instance, the Euclidean minimum spanning tree can be determined using only the ordering between distances, and not their numeric values. Comparing squared distances produces the same result but avoids an unnecessary square-root calculation and sidesteps issues of numerical precision. As an equation, the squared distance can be expressed as a sum of squares:", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0047", "text": "d\n          \n            2\n          \n        \n        (\n        p\n        ,\n        q\n        )\n        =\n        (\n        \n          p\n          \n            1\n          \n        \n        \u2212\n        \n          q\n          \n            1\n          \n        \n        \n          )\n          \n            2\n          \n        \n        +\n        (\n        \n          p\n          \n            2\n          \n        \n        \u2212\n        \n          q", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0048", "text": "\u2212\n        \n          q\n          \n            2\n          \n        \n        \n          )\n          \n            2\n          \n        \n        +\n        \u22ef\n        +\n        (\n        \n          p\n          \n            n\n          \n        \n        \u2212\n        \n          q\n          \n            n\n          \n        \n        \n          )\n          \n            2\n          \n        \n        .", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0049", "text": ".\n      \n    \n    {\\displaystyle d^{2}(p,q)=(p_{1}-q_{1})^{2}+(p_{2}-q_{2})^{2}+\\cdots +(p_{n}-q_{n})^{2}.}", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0050", "text": "Beyond its application to distance comparison, squared Euclidean distance is of central importance in statistics, where it is used in the method of least squares, a standard method of fitting statistical estimates to data by minimizing the average of the squared distances between observed and estimated values, and as the simplest form of divergence to compare probability distributions", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0051", "text": ". The addition of squared distances to each other, as is done in least squares fitting, corresponds to an operation on (unsquared) distances called Pythagorean addition. In cluster analysis, squared distances can be used to strengthen the effect of longer distances.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0052", "text": "Squared Euclidean distance does not form a metric space, as it does not satisfy the triangle inequality. However it is a smooth, strictly convex function of the two points, unlike the distance, which is non-smooth (near pairs of equal points) and convex but not strictly convex. The squared distance is thus preferred in optimization theory, since it allows convex analysis to be used", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0053", "text": ". Since squaring is a monotonic function of non-negative values, minimizing squared distance is equivalent to minimizing the Euclidean distance, so the optimization problem is equivalent in terms of either, but easier to solve using squared distance.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0054", "text": "The collection of all squared distances between pairs of points from a finite set may be stored in a Euclidean distance matrix, and is used in this form in distance geometry.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0055", "text": "Generalizations", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0056", "text": "In more advanced areas of mathematics, when viewing Euclidean space as a vector space, its distance is associated with a norm called the Euclidean norm, defined as the distance of each vector from the origin. One of the important properties of this norm, relative to other norms, is that it remains unchanged under arbitrary rotations of space around the origin", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0057", "text": ". By Dvoretzky's theorem, every finite-dimensional normed vector space has a high-dimensional subspace on which the norm is approximately Euclidean; the Euclidean norm is the", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0058", "text": "only norm with this property. It can be extended to infinite-dimensional vector spaces as the L2 norm or L2 distance. The Euclidean distance gives Euclidean space the structure of a topological space, the Euclidean topology, with the open balls (subsets of points at less than a given distance from a given point) as its neighborhoods.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0059", "text": "Other common distances in real coordinate spaces and function spaces:", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0060", "text": "Chebyshev distance (L\u221e distance), which measures distance as the maximum of the distances in each coordinate.\nTaxicab distance (L1 distance), also called Manhattan distance, which measures distance as the sum of the distances in each coordinate.\nMinkowski distance (Lp distance), a generalization that unifies Euclidean distance, taxicab distance, and Chebyshev distance.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0061", "text": "For points on surfaces in three dimensions, the Euclidean distance should be distinguished from the geodesic distance, the length of a shortest curve that belongs to the surface", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0062", "text": ". In particular, for measuring great-circle distances on the Earth or other spherical or near-spherical surfaces, distances that have been used include the haversine distance giving great-circle distances between two points on a sphere from their longitudes and latitudes, and Vincenty's formulae also known as \"Vincent distance\" for distance on a spheroid.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0063", "text": "History", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0064", "text": "Euclidean distance is the distance in Euclidean space. Both concepts are named after ancient Greek mathematician Euclid, whose Elements became a standard textbook in geometry for many centuries. Concepts of length and distance are widespread across cultures, can be dated to the earliest surviving \"protoliterate\" bureaucratic documents from Sumer in the fourth millennium BC (far before Euclid), and have been hypothesized to develop in children earlier than the related concepts of speed and time", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0065", "text": ". But the notion of a distance, as a number defined from two points, does not actually appear in Euclid's Elements. Instead, Euclid approaches this concept implicitly, through the congruence of line segments, through the comparison of lengths of line segments, and through the concept of proportionality.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0066", "text": "The Pythagorean theorem is also ancient, but it could only take its central role in the measurement of distances after the invention of Cartesian coordinates by Ren\u00e9 Descartes in 1637. The distance formula itself was first published in 1731 by Alexis Clairaut. Because of this formula, Euclidean distance is also sometimes called Pythagorean distance", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0067", "text": ". Although accurate measurements of long distances on the Earth's surface, which are not Euclidean, had again been studied in many cultures since ancient times (see history of geodesy), the idea that Euclidean distance might not be the only way of measuring distances between points in mathematical spaces came even later, with the 19th-century formulation of non-Euclidean geometry", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0068", "text": ". The definition of the Euclidean norm and Euclidean distance for geometries of more than three dimensions also first appeared in the 19th century, in the work of Augustin-Louis Cauchy.", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}, {"id": "rec_Euclidean_distance_0069", "text": "== References ==", "title": "Euclidean distance", "url": "https://en.wikipedia.org/wiki/Euclidean_distance"}]