Anthropology, Development, and Beyond | For More Conversations with Economists and Other Scholars

Question: What is the relationship between anthropology and economy in the field of development studies and what it should be in face of the grave problems today?

1.The field of development studies exemplifies the changes of economic anthropology and the relationship between economy and anthropology | In a special issue about the relations between anthropology and economics, it is mandatory to mention economic anthropology as a disciplinary niche that has represented anthropologists' visions on economic issues. | From the 1960s to the mid-1980s, Marxism as an influential theory and a common ground facilitated crossfertilizations among anthropologists, economists, historians, and geographers. | As Marxism has declined after the end of the Cold War, many of the issues discussed by economic anthropologists slowly retreated to the back of practitioners' minds or became a part of research areas identified by distinct labels and engagements with topics such as development, globalization, infrastructure, financialization, climate change, and the Anthropocene. | It is possible to find subjects dear to economic anthropology in different areas of contemporary research, but this does not mean that it currently exists in a rebranded fashion. First, economic anthropology has kept its own heuristic space. Second, the insertion in a new field of inquiry calls for new research, interlocutions, recalibrations, or creations of concepts and interpretations, and it generates genuine and original knowledge and theories. | The field of development studies allows to refer to scholarly, professional, and political relations and subjects of common interest of economists and anthropologists. | Conclusion: The field of development studies exemplifies the changes of economic anthropology and the relationship between economy and anthropology. | 2.Review the involvement of anthropologists in development since the 1940s. | Since the 1940s, development has been a discourse specialized on issues of global wealth or poverty and inequalities and on defining models and actions to solve the problems of so-called underdeveloped countries, developing countries, or emerging | economies. Economists' prominence is especially visible in places such as the World Bank (WB), the global governance institution once thought to be the mecca of development, and where they also have interacted with development anthropologists who entered this scenario as specialists on local, "backward," "exotic," "poor," and "Third World" peoples. | The involvement of anthropologists in development may be subdivided into two trends: development anthropology and the anthropology of development. Development anthropology is mainly concerned with the impacts of development on local people. The anthropology of development is mostly dedicated to a critique of "development" as a Western discourse. The study of what was called "development projects" and development agencies or agents is included in this last area of anthropological concerns. | In the 1970s the academic production on development started to blossom. By studying the impact of development projects based on comparisons of different local responses, anthropologists tried to find constant structures that would allow them to predict individual and group behavior in a vein similar to economists, whose professional practices often involve risk analysis and how to avoid contingencies. At the same time, anthropologists became hire options for multilateral and national bureaucracies and civil society agencies related to the development industry. But Whether or not the practice of "development anthropologists" had positive effects on the institutions they have worked for and on the lives of local populations is a hot issue among anthropologists that resonates with debates related to the applied or academic divide in the discipline. | In the 1990s, there is a postdevelopment critique. According to Escobar (2005), the main point of the postdevelopment debate was not to propose another version of development but to answer the question of "why, by means of which historical processes and with what consequences, Asia, Africa and Latin America were 'thought' as the 'Third World' by development discourses and practices" . One important research effort within the interdisciplinary and diverse field of postdevelopment was the study of development in concrete settings, its agents, agencies, and processes. | Conclusion: The roles anthropologists play in the field of development studies are changing. | 3.The Development Field in the Wider Juncture in the Late 1980s and 1990s | The 1990s was a particularly important decade for the involvement of anthropologists in global development policies, a professional environment usually hegemonized by economists. | Macro political, economic, and sociological dynamics that affected the world system by the end of the 1980s and beginning of the 1990s: The bipolar world of the Cold War was finished, opening a period of crisis of socialist utopian discourses and projects. This gap was filled by other utopian discourses that had been gaining momentum since the 1970s and reached their climax in 1988 and 1992. The early 1990s was a propitious juncture for a series of experiments under the banner of the newly accepted global consensus: sustainable development. | The impact of sustainable development was particularly felt in the global development agencies and in the aid industry. The window was open for a greater participation of anthropologists in these processes within multilateral agencies, state organizations, and NGOs. But development is a power field in which anthropologists and economists converge and diverge. It is made up of several social, ethnic, political, technical, professional, organizational, financial, state, and private agencies and agents. | The development field has spurred the mingling of economists and anthropologists in working environments. Their relationships are dependent on the institution's hierarchy and internal politics and not on some supposed ontological superiority of one discipline over the other. Moreover, The impact of wider political, ideological, and utopian discourses on the development field cannot be underestimated and | may open avenues of cooperation and conflict between anthropologists and economists. Generally speaking, Anthropologists will usually emphasize the value of diversity, cultural relativist approaches (such as different conceptions of nature and the good life), and the empowerment of local people. Economists will usually emphasize the value of models, numbers, rational choices, and market forces as modes of solving development problems. | 4.Changes in Academia and in the World System | The high time of development anthropology and anthropology of development is behind us, some of the subjects encompassed by the development debate of the 1990s and early 2000s are currently part of other domains of academic and public interest such as the growing field of interactions and exchanges among different epistemic communities called Anthropocene. | Limits of Anthropocene: The larger discursive and institutional field in which the Anthropocene locates itself is made up by agencies and agents that are not as identifiable as those of the development field. Moreover, in certain interpretations of the Anthropocene, there is a tendency to think that subjects of complex political fields can hardly be distinguished and that the establishment of powerful collective counterhegemonic subjects is no longer possible. Such visions obliterate the mapping of the power field now in action and make it harder to establish political priorities for scholars and citizens. | The relationship between Anthropocene and sustainable development: Anthropocene as a discourse shares similarities with sustainable development. It makes up a field of contentions where different political forces may meet, oppose, or ally themselves. There are some notable differences though. Sustainable development is a utopia, whereas the Anthropocene is a dystopia. And more importantly, sustainable development, as an heir to the many meanings and adjectives associated with "development," generated its own organizational and political field, institutional and otherwise, that, despite its normalization, has also been useful for the promotion of the notion and political field of the Anthropocene itself. | Issue for future: How each of these discursive fields, with their agents and agencies, may establish productive and virtuous exchanges between them and with other sets of progressive discourses capable of constructing new formulations to win the hearts and minds of the many millions of people who know there cannot be a future without environmental and social justice. | New challenges: The world system has changed since the 1990s, especially after the end of the Cold War, the development power field has also changed in the past two decades. The most notorious novelty is the role of China in international development. | 5.Anthropologists and Economists Are a Diverse Group of Professionals Facing New Interdisciplinary Challenges | "Heterodox economists," who are mostly oriented by political economy or Marxist theories and not by the prevailing neoclassical economics school, are aware that historical, sociological, cultural, racial, gender, and environmental dynamics are not "externalities" that may be ignored. | Anthropologists and economists make up a diverse group of professionals. Their approximations and detachments are based on both disciplinary and political visions. The kinds of specific disciplinary education and worldviews embedded in economics and anthropology are not enough to explain all divergences and convergences in the relationships between these professionals. | Conclusion: Ultimately, what is at stake when economists/anthropologists discuss development is how wealth is produced and partaken and how it affects the human | and nonhuman worlds, currently, these questions come together with major challenges. it is time for anthropologists to critically and more consistently engage with notions and problems concerning humankind. And the expectation of a generalized catastrophe caused by global warming calls for the integration of diverse cultural, political, and scientific standpoints beyond the realms of economics and anthropology.

1. Does the answer include the perspective that the field of development studies exemplifies the changes in economic anthropology and the relationship between economy and anthropology? | 2. Does the answer review the involvement of anthropologists in development since the 1940s? | 3. Does the answer discuss the development field in the wider juncture in the late 1980s and 1990s? | 4. Does the answer discuss how anthropologists and economists should act in response to new interdisciplinary challenges? | 5. Does the answer discuss the content of changes in academia and in the world system?

1.The world system has changed since the 1990s, when the tremendous power of social and environmental destruction deployed by triumphant neoliberal capitalist globalization became the norm. The West no longer controls the development discourse as completely as it used to. | 2.Heterodox economists are aware that historical, sociological, cultural, racial, gender, and environmental dynamics are not “externalities” that may be ignored, and anthropologists and economists make up a diverse group of professionals. | 3.How wealth is produced and partaken and how it affects the human and nonhuman worlds, which is what at stake when economists/anthropologists discuss development, currently come together with major challenges, such as the increased importance of conservative and ultraconservative politics, new imperialist wars, and climate change.

1.development:A multidimensional process involving economic growth, social progress, and improvement in the quality of life of a population. It is understood as a complex, context-dependent transformation of societies aiming at reducing poverty, inequality, and vulnerability while expanding opportunities and freedoms for all individuals. | 2.development anthropology:A subfield of anthropology that studies the social, cultural, political, and economic dimensions of development processes and projects. It seek to understand the impacts of development from the perspectives of the people involved, aiming to improve development practices by promoting culturally sensitive, participatory, and sustainable approaches. 3.anthropology of development:A specialized area within anthropology that critically analyzes the concepts, practices, and impacts of development as a global phenomenon. It examines power dynamics, inequalities, and the unintended consequences of development, often questioning dominant development paradigms and advocating for more inclusive, locally grounded approaches. 4.Anthropocene: A proposed geological epoch that marks the period during which human activity has become the dominant influence on the Earth’s climate, ecosystems, and geology. It signifies a new era where humans significantly alter natural processes, leading to large-scale environmental changes such as climate change, biodiversity loss, pollution, and landscape transformation. | 5.Capitalocene: A critical term used to describe the current historical epoch, emphasizing that the environmental crises we face are primarily driven by the capitalist mode of production rather than humanity as a whole. It highlights how capitalism’s focus on endless economic growth, exploitation of labor, and natural resources leads to systemic environmental degradation and social inequalities.

1.Review the involvement of anthropologists in development since the 1940s.2.Review the development field in the wider juncture in the late 1980s and 1990s, and discuss the relationship between anthropology and economy in the field of development studies.3.Disscuss the Anthropocene/Capitalocene debates under the context of current major challenges , and advocate new interdisciplinary exchanges.


A Lower Bound for Light Spanners in General Graphs

Question: Determine the optimal lightness of spanners in graphs, specifically focusing on the dependence on the parameter \eps.

Introduction | | Spanners are fundamental graph sparsifiers with applications in networking, chip design, flow algorithms, and more. | | Theorem 1 | For all positive integers n, k, every n-node graph has a (2k − 1)-spanner with O(n^(1+1/k)) edges. | | Conjecture 2 (Girth Conjecture) | There exists a family of n-node graphs with Ω(n^(1+1/k)) edges and girth greater than 2k. The girth of a graph is the smallest number of edges in any of its cycles. | | If any edge (u, v) is removed from such a graph, there is no alternate u–v path of length ≤ 2k − 1, otherwise a short cycle would exist. Therefore, such a graph has no (2k − 1)-spanner other than itself, giving a matching lower bound to Theorem 1. The conjecture is confirmed for k in {1, 2, 3, 5, Ω(log n)}, but proving or refuting it for other k is an open problem. | | Theorem 3 | For all positive integers n, k, and all ε > 0, every n-node graph G has a (1 + ε)(2k − 1)-spanner H with lightness | ℓ(H | G) ≤ O(ε^(-1) · n^(1/k)). | | Currently, little is known about lower bounds for lightness. A girth conjecture graph implies a lower bound of Ω(n^(1/k)) for (2k − 1)-spanners, but this does not use edge weights. No known lower bounds leverage weighted edges, so it is conceivable that no ε-dependence is needed and that sparsity and lightness bounds match. | | The main result shows that ε-dependence in lightness bounds is indeed necessary. | | Theorem 4 (Main Result) | Let k ≥ 2 be constant and assume the girth conjecture with parameter k − 1. There exists a family of n-node weighted graphs G where, for ε := Θ(n^(−1/(2k−1))), any spanner H with stretch (1 + ε)(2k − 1) has | ℓ(H | G) ≥ Ω(ε^(−1/k) · n^(1/k)). | | Thus, ε-dependence is necessary in general. This lower bound also extends to mildly super-constant k with extra technical work. | | Corollary 5 | For any n and constant k ≥ 2, assuming the girth conjecture with parameter k − 1, there exists an n-node weighted graph G where any (2k − 1)-spanner H has | ℓ(H | G) ≥ Ω(n^(k / (1 + k(2k − 1)))). | | ⸻ | | Relationship to the Weighted Girth Conjecture | | Definition 3 (Weighted Girth) | For a cycle C in a weighted graph G = (V, E, w), its normalized weight is | w*(C) := w(C) / max_{e∈C} w(e). | | The weighted girth of G is the minimum w*(C) over all cycles C in G. | | Theorem 6 | For all t ≥ 1, every graph has a t-spanner with weighted girth > t + 1. Moreover, any graph with weighted girth > t + 1 has no nontrivial t-spanner. | | Thus, the question of lightness for t-spanners is equivalent to the maximum possible lightness over all graphs of weighted girth > t + 1. This reframing is used in our lower bound construction: we aim to construct a graph H with weighted girth > (1 + ε)·2k and | ℓ(H) ≥ Ω(ε^(−1/k) · n^(1/k)). | | Conjecture 7 (Weighted Girth Conjecture) | For all n, g, there exists an unweighted n-node graph that maximizes lightness among all graphs with weighted girth exactly g. | | Our new lower bound does not refute this conjecture but does refute certain stronger variants, which may guide future attacks on the problem. | | Corollary 8 (Refutation of “Strong” Weighted Girth Conjectures) | The conjecture is false if we instead consider maximum lightness over graphs with weighted girth strictly larger than g, or allow g to be non-integral. | | Example: For g = 4, the densest unweighted girth-4 graph has Θ(n^2) edges, while girth > 4 graphs have Θ(n^(3/2)) edges. Our lower bound gives graphs with weighted girth > 4 and lightness Θ(n^(2/3)), exceeding the latter bound but not the former. This pattern holds for any even g ≥ 4. | | ⸻ | | Limitations to Further Progress | | The ε-dependence in our lower bound is far from the current upper bound’s dependence. Improving it for all ε or matching the upper bound’s ε-dependence is difficult because our bound relies on the girth conjecture with parameter k − 1. Either improvement would imply the girth conjecture for parameter k. | | Theorem 9 (Technical Limitations) | If Theorem 4 could be improved as follows— | • Holds for all 0 < ε < 1, or | • Has ε^(−1) dependence instead of ε^(−1/k) in the lower bound— | then the girth conjecture would be true for all constant k. | | Given the longstanding difficulty of the girth conjecture, these barriers are significant. | | ⸻ | | Lower Bound for Light Spanners | | Structure of Girth Conjecture Graphs | | Lemma 10 | If the girth conjecture holds for k, it can be satisfied by graphs that are: | 1. Bipartite | 2. Approximately regular (all degrees Θ(d) for some d) | 3. Have at most O(n^((k+2c−1)/k)) cycles of length (2k + 2c) containing any given edge. | | The proof constructs such graphs from arbitrary girth-conjecture graphs while preserving edge count and girth properties. | | ⸻ | | Lower Bound Construction | • Let G be an n-node graph satisfying the girth conjecture with parameter k − 1 and Lemma 10’s properties. | • Let C be a cycle on N = 4k ε^(−1) n nodes with all edges weight 1 (the spanning cycle, SC). | • Partition C into alternating clusters (k ε^(−1) nodes) and spacers (3k ε^(−1) nodes), n of each. | • Map each node v in G to a cluster X_v in C. | • For each edge (u, v) in G, add one edge of weight ε^(−1) between random nodes in X_u and X_v. | • Delete all non-SC edges in cycles X with normalized weight ≤ (1 + ε)(2k). | | Let H be the final graph. | | ⸻ | | Cycle Analysis | | Lemma 12 | If ε < 1/2, any cycle in H′ (before deletions) with normalized weight ≤ 2k(1 + ε) contains between 2k and 2k(1 + ε) non-SC edges. | | Lemma 13 | For c ≥ 0, the probability that a (2k + 2c)-cycle in G maps to a cycle in H′ with normalized weight ≤ (1 + ε)2k is at most Θ(ε)^(2k+2c) / (2k + 2c)!. | | Lemma 14 | If ε ≤ O(k · n^(−(k−1)(2^(k+2)^c + 2c))) with a small enough constant, then for each non-SC edge e, the expected number of small-normalized-weight cycles in H′ containing e is ≤ 4^c / 4. | | ⸻ | | Proof Wrap-up | | Choosing ε according to Lemma 14 and bounding c via Lemma 12 ensures that each edge survives the final deletion step with constant probability, meeting Lemma 11’s conditions and completing the proof of Theorem 4. | | For constant k, this simplifies to ε = Θ(k · N^(−2k / (2k² − k))) and | ℓ(H) = Ω_k(N^(1/k) · ε^(−1/k)). | | ⸻ | | Proof of Technical Lower Bound | | Let γ(n, 2k) be the maximum number of edges in an n-node graph with girth > 2k. The girth conjecture for k states: | γ(n, 2k) ≥ Ω(n^(1+1/k)). | | Theorem 15 | For all n, k, ε > 0, every n-node graph has a (1 + ε)(2k − 1)-spanner with | ℓ(H | G) ≤ O(ε^(−1) · γ(n, 2k)). | | Proof of Theorem 9 | Using Theorem 4 (conditional on k − 1) and induction on k, improving Theorem 4 to hold for all ε or with ε^(−1) dependence would imply γ(n, 2k) ≥ Ω(n^(1+1/k)), proving the girth conjecture for k. This shows the inherent difficulty in further improving the bound.

1. Correctly define the free energy function related to factor graphs.

2. Correctly understand and apply the lemma stating that any graph satisfying the girth conjecture can be transformed into a bipartite, approximately regular graph in which every edge participates in at most O(n^((k+2c−1)/k)) cycles of length (2k+2c).

3. Correctly state and prove the main lower bound theorem showing that for constant k and ε := Θ(n^(−1/(2k−1))), any spanner with stretch (1 + ε)(2k − 1) must have lightness at least Ω(ε^(−1/k) · n^(1/k)), thereby establishing the necessity of ε-dependence.

4. Correctly understand and apply the theorem stating that every graph admits a t-spanner of weighted girth greater than t + 1, and that any graph with weighted girth greater than t + 1 has no nontrivial t-spanner, thus linking the light spanner problem to the weighted girth conjecture.

5. Correctly state and prove the technical limitations theorem showing that if the lower bound in Theorem 4 could be improved either to hold for all 0 < ε < 1 or to achieve ε^(−1) dependence, then the girth conjecture would follow for all constant k.

6. Correctly explain the necessity of ε-dependence in lightness bounds, and analyze how the new results improve upon previous lower bounds.

#### Definitions and Key Concepts | | - **Spanners**: Given an input graph \( G \), a \( k \)-spanner is a subgraph \( H \) such that the distance between any two nodes \( s \) and \( t \) in \( H \) is at most \( k \) times the distance between \( s \) and \( t \) in \( G \). Formally, \( \dist_H(s, t) \le k \cdot \dist_G(s, t) \) for all nodes \( s, t \). | | - **Sparsity**: The size of a spanner \( H \) is often measured by its sparsity, which is the number of edges \( |E(H)| \). The goal is to minimize this number while maintaining the desired stretch \( k \). | | - **Lightness**: Another way to measure the size of a spanner is by its total edge weight \( w(H) \). Since the weight of a spanner can be unbounded, it is normalized by the weight of a minimum spanning tree (MST) of the input graph \( G \). The lightness of a spanner \( H \) is defined as: | \[ | \ell(H \mid G) := \frac{w(H)}{w(\mst(G))}. | \]

## Weighted Girth | | **Definition (Weighted Girth)** | | For a cycle \( C \) in a weighted graph \( G = (V, E, w) \), its normalized weight is the quantity | \[ | w^*(C) := \frac{w(C)}{\max_{e \in C} w(e)}. | \] | The weighted girth of \( G \) is the minimum value of \( w^*(C) \) over all cycles \( C \) in \( G \).

## Girth Conjecture Graphs | | Start with a girth conjecture graph \( G \) with parameter \( k - 1 \), ensuring it is bipartite, approximately regular, and has a bounded number of cycles. | | ## Spanning Cycle (SC) | | Construct a large cycle \( C \) with \( N = 4k\epsilon^{-1}n \) nodes, partitioned into clusters and spacers. | | ## Mapping Nodes to Clusters | | Map nodes of \( G \) to clusters of \( C \), adding high-weight edges between clusters corresponding to edges in \( G \). | | ## Random Edges and Cycle Removal | | Add random edges between clusters and remove cycles with normalized weight below a threshold to ensure the resulting graph \( H \) has the desired weighted girth.


Tight Streaming Lower Bounds for Deterministic Approximate Counting

Question: What is the streaming complexity of the k-counter approximate counting problem, please provide a lower bound.

Streaming Complexity of the k-Counter Approximate Counting Problem | | Computation Model: Read-once Branching Programs (ROBP) | | A read-once branching program over a finite alphabet Σ of length n consists of n+1 layers of vertices | V₀, V₁, …, Vₙ, with edges between consecutive layers labeled by distinct symbols from Σ. | • The program starts at a single vertex in V₀. | • On input x ∈ Σⁿ, the label of each symbol determines the edge taken at each step. | • Vertices in the last layer Vₙ are labeled with outputs, and the output of the program is the label of the vertex reached. | • The width is the maximum number of vertices in any layer. | • Every vertex must be reachable from some input. | | ⸻ | | Problem Definition: Approximate Counting | | {0,1}-Approximate Counting | | For n ≥ 1 and error bound Δ ≥ 0, the problem ApproxCount[n, Δ] takes input x ∈ {0,1}ⁿ and outputs a value Ŝ such that | |Ŝ − ∑ᵢ xᵢ| ≤ Δ. | | k-Counter Approximate Counting | | For integers k ≥ 2, n ≥ 1, and Δ ≥ 0, the problem ApproxCountₖ-counter[n, Δ] takes input x ∈ [k]ⁿ and outputs a k-tuple (Ŝ₁, …, Ŝₖ) such that for all j ∈ [k]: | |Ŝⱼ − #{i : xᵢ = j}| ≤ Δ. | | k-Parallel Approximate Counting | | For integers k ≥ 1, n ≥ 1, and Δ ≥ 0, the problem ApproxCountₖ-parallel[n, Δ] takes input x ∈ ({0,1}ᵏ)ⁿ and outputs (Ŝ₁, …, Ŝₖ) such that for all j ∈ [k]: | |Ŝⱼ − #{i : (xᵢ)ⱼ = 1}| ≤ Δ. | | ⸻ | | Lower Bound for k-Counter Approximate Counting | | Theorem 1 (k-Counter Lower Bound) | Let k ≥ 2, n ≥ 1, w ≥ 1, and 0 ≤ Δ ≤ n / (2(k−1)). Suppose there exists a length-n, width-w ROBP P over [k] computing ApproxCountₖ-counter[n, Δ]. | Let m be the largest integer with m ≤ n−1 and C(m+k−1, k−1) ≤ w. Then: | | C(m, k) + (n − m − 1)·w ≥ n − 2(k−1)Δ + k / (k−1). | | This bound shows that for small error Δ, the width w must be large, scaling polynomially with n for fixed k. | | ⸻ | | Corollaries | 1. Large Error Regime | If Δ = 3(kⁿ − 1), then | w ≥ Ω((n/k)^(k−1)). | 2. Width–Error Tradeoff | For 0 ≤ Δ ≤ n / (2(k−1)): | Δ ≥ n / (2(k−1)) − (√(k!)) / (2(k−1)) · √(n / w). | 3. Small Error Regime | If 1 ≤ Δ ≤ n / (10(k−1)) and n ≥ 10k: | w ≥ C(n + k − 1 − 2(kᵏ − 1)Δ, k − 1) − O(√(nΔ), k − 1). | | ⸻ | | Lower Bound for k-Parallel Approximate Counting | | Theorem 2 | For k ≥ 1, n ≥ 3k, and Δ = n/3, if a length-n, width-w ROBP P over ({0,1}ᵏ) computes ApproxCountₖ-parallel[n, Δ], then: | w ≥ n^(Ω(k)). | | This shows exponential (in k) width requirement in the parallel setting, even for constant-factor errors. | | ⸻ | | Intuition Behind the Proof | | The proof framework uses: | • Rectangle labeling: Assigns to each vertex in the ROBP a hyper-rectangle in ℕᵏ⁻¹ (or ℕᵏ in the parallel case) describing possible counts for each symbol up to that point. | • Potential function: Measures how much “uncertainty” remains in the count at each layer. | • Growth Lemma: Shows that potential grows by at least a certain amount each layer, unless width is large enough to track many distinct states. | • Final Bound: Relates the maximum possible potential at the last layer (bounded by Δ) to the accumulated growth, yielding a lower bound on w. | | ⸻ | | Final takeaway: | In the streaming model with width-limited ROBPs (equivalent to space-limited streaming algorithms), the k-counter approximate counting problem requires polynomial width (space) for small error, and even super-polynomial width in the k-parallel setting. Theorem 1 provides a general quantitative tradeoff between width, error Δ, and problem parameters.

1. Verify that the Read-Once Branching Program (ROBP) model is explicitly chosen as the computation framework for analyzing streaming complexity.

2. Confirm that the key components of the ROBP model are clearly described, including the finite alphabet, layered graph structure, start state, transition rules, and the definition of width.

3. Check that the proof introduces **interval labeling** for the {0,1}-approximate counting case and **rectangle labeling** for the general k-counter case, which characterize the range of possible counts at each vertex.

4. Ensure that a potential function is formally defined to measure the accumulated uncertainty in ROBP states as inputs are processed.

5. Confirm that the properties of the potential function are rigorously analyzed, including its guaranteed growth when the ROBP width is small and its boundedness under approximate counting constraints.

6. Verify that the base case k=2 ({0,1}-approximate counting) is analyzed in detail, establishing that any ROBP computing ApproxCount[n, Δ] with Δ = n/10 requires width Ω(n).

7. Check that the analysis is generalized from the k=2 case to the general k-counter case via rectangle labeling and multidimensional potential functions, yielding polynomial width lower bounds.

8. Confirm that the main conclusion is clearly stated: computing k-counter approximate counting in the streaming model requires Ω(k log(n/k)) bits of space, and equivalently, width Ω((n/k)^(k−1)) in the ROBP model for error Δ = n/(3(k−1)).

- In computer science, the streaming model is a computational model for processing large-scale data, where data arrives as a stream and algorithms need to process the data with limited storage space. In the context of streaming, exact counting of the frequency of each element may require substantial storage space, making the study of approximate counting problems of great significance. Approximate counting problems allow the algorithm to output an approximate frequency value as long as the error between this value and the true frequency is within a certain range. | - The k - counter approximate counting problem is a classic problem in streaming. Given an input string of length n, where each character of the string is from a character set of size k, the goal is to approximate the frequency of each character in the string, with the requirement that the error between the approximate frequency and the true frequency for each character does not exceed a given threshold. For example, when k = 2, the problem is equivalent to estimating the frequency of 1 (i.e., the Hamming weight) in a 01 - string. This kind of problem appears as a basic module in many complex algorithms, so studying its complexity is a fundamental question in theoretical computer science. | - In the streaming model, for the k - counter approximate counting problem, the best-known lower bound before was trivial, i.e., using O(klogn) bits of space to exactly count the frequency of each character. However, in the regime where n is much larger than k, it was previously unclear whether less space could be used to output an approximation of the frequency. In addition, for some other problems related to approximate counting, such as the heavy hitters problem and the quantile sketch problem, there are also similar issues of storage space complexity. For example, the Misra - Gries algorithm is a well - known algorithm for solving the heavy hitters problem. It uses O(k(log(n/k)+log(U/k))) bits of space in the streaming model to output the heavy hitters and their frequency estimates, but it was previously unknown whether the space complexity of this algorithm was already optimal.

## 1. {0, 1}-Approximate Counting | For any integer n ≥ 1 and real number ∆ ≥ 0, define the problem ApproxCount[n, ∆] as follows: on any input x ∈ {0, 1}^n, a valid output of x is a real number ̂S such that | ̂S − ∑_{i=1}^n x_i | ≤ ∆. | | ## 2. k-counter Approximate Counting | For any integers k ≥ 2, n ≥ 1, and real number ∆ ≥ 0, define the problem ApproxCountk-counter[n, ∆] as follows: on any input x ∈ [k]^n, a valid output of x is a k-tuple of real numbers ( ̂S_1, · · · , ̂S_k ) such that | ̂S_j − #{i ∈ [n] : x_i = j} | ≤ ∆ for all j ∈ [k]. | | ## 3. Rectangle Labeling of the k - counter Case | For a length - n ROBP P over alphabet [k] that attempts to compute ApproxCountk - counter[n, ∆], label each vertex v ∈ Vt (0 ≤ t ≤ n) with a (k - 1) - dimensional rectangle R(v). R(v) is defined as [a1, b1] × · · · × [ak−1, bk−1], where for each j ∈ [k - 1], aj, bj are the minimal and maximal elements in {c ∈ N: ∃(x1, · · · , xt) ∈ [k]t such that (x1, · · · , xt) reaches v in P and #{i ∈ [t]: xi = j} = c}. For each 0 ≤ t ≤ n, let Rt := {R(v): v ∈ Vt} be the collection of all rectangle labels in layer Vt. | | ## 4. Potential Function | Consider a length - n ROBP P over alphabet [k] that attempts to compute ApproxCountk - counter[n, ∆]. For each 0 ≤ t ≤ n, define Tt,k := {(x1, · · · , xk−1) ∈ Nk−1: x1 + · · · + xk−1 ≤ t}, which is the set of possible numbers of 1’s, 2’s, · · · , (k - 1)’s in the first t input words. Then, for each 0 ≤ t ≤ n and (x1, · · · , xk−1) ∈ Tt,k, define φt(x1, · · · , xk−1) := maxR=[a1,b1]×···×[ak−1,bk−1]: R∈Rt, (x1,··· ,xk−1)∈R{min{b1 + · · · + bk−1, t} - x1 - · · · - xk−1}. Furthermore, for each 0 ≤ t ≤ n, define the total potential of layer Vt as Φt := ∑(x1,··· ,xk−1)∈Tt,kφt(x1, · · · , xk−1). | | ## 5. Rectangle Labeling of the k - parallel Case | For a length - n ROBP P over alphabet {0, 1}k that attempts to compute ApproxCountk - parallel[n, ∆], label each vertex v ∈ Vt (0 ≤ t ≤ n) with a k - dimensional rectangle R(v). R(v) is defined as [a1, b1] × · · · × [ak, bk], where for each j ∈ [k], aj, bj are the minimal and maximal elements in {c ∈ N: ∃(x1, · · · , xt) ∈ {0, 1}k^t, (x1, · · · , xt) reaches v and #{i ∈ [t]: (xi)j = 1} = c}. For each 0 ≤ t ≤ n, let Rt := {R(v): v ∈ Vt} be the collection of all rectangle labels in layer Vt.

# Methodology | | The methodology employed to establish tight streaming lower bounds for deterministic approximate counting involves the following steps: | | 1. **Model Selection**: Utilize the read-once branching program (ROBP) model to represent streaming algorithms. In this model, a streaming algorithm is depicted as a layered multigraph where vertices correspond to memory states, and edges labeled with input symbols dictate transitions between states. | | | 2. **Interval and Rectangle Labeling**: For the k=2 case, label vertices in the ROBP with intervals representing possible counts of a particular symbol. Generalize this approach to higher dimensions using rectangle labeling for the k-counter case, where vertices are labeled with multidimensional rectangles representing possible counts of each symbol. | | 3. **Potential Function Analysis**: Define a potential function based on the interval or rectangle labels. This function quantifies the growth of uncertainty in the counts as the ROBP processes the input, with its growth rate tied to the width of the ROBP (corresponding to the space used by the streaming algorithm). | | 4.**Proof Strategy**: Demonstrate that if the space (or width) is too small, the potential function grows excessively, contradicting the requirement that the final approximation must stay within specified error bounds. This contradiction establishes the lower bound on the space required. | | | 5. **Implications**: Apply the results to derive lower bounds for related streaming problems, such as heavy hitters and quantile sketch, by showing that solutions to these problems can be used to approximate counts, thereby inheriting the lower bounds established for approximate counting. | | 6.**Algorithm Development**: Devise non-trivial algorithms for approximate counting that nearly match the lower bounds. These algorithms demonstrate the tightness of the results and the difficulty of improving upon them.


A Refutation of the Pach-Tardos Conjecture for 0-1 Matrices

Question: Refute the Pach-Tardos conjecture for 0-1 matrices.

Refuting the Pach–Tardos Conjecture for 0–1 Matrices | | The Pach–Tardos conjecture predicted an upper bound on the extremal function Ex(M, n)—the maximum number of 1s in an n × n 0–1 matrix that avoids a given forbidden submatrix M. | The work here constructs explicit 0–1 matrices with far more 1s than the conjecture would allow, thereby disproving it for certain patterns. | | ⸻ | | Core Construction | | We define a base matrix A[b, m]: | • Rows: indexed by [m] × [m]^b | • Columns: indexed by [m]^b × {0,1}^b | • Entry rule: | | A((s, r), (c, i)) = 1 if r = c + s · i | 0 otherwise | | | • Arithmetic is done coordinatewise over integers, with i treated as a 0–1 indicator vector. | | Key property (Lemma 2): | Each column has on average at least m/(b+1) - 1 ones, so | ‖A[b, m]‖₁ ≥ Ω(m/b) × (#columns). | | ⸻ | | Pattern Avoidance | | The forbidden patterns of interest are: | • S₀ and S₁: small fixed 0–1 matrices that arise in the conjecture. | • Covering patterns: defined to have a “distinguished” row with 1s at the extremes, plus certain row–interval covering properties. | | Lemma 4 gives structural constraints on the positions of 1s in A[b, m], relating the order of row/column indices to coordinate differences in the [m]^b part. | | ⸻ | | Main Lower Bound Result | | Theorem 1 | For S₀ and S₁, | | Ex(S₀, n), Ex(S₁, n) ≥ n² · √(log n − O(log log n)) | | Reasoning: | 1. Show A[b, m] contains no S₀ or S₁: | • Assume S₀ is embedded in A. | • Define difference vectors x, y, z from certain column pairs. | • Lemma 4 forces their first nonzero coordinates to be incompatible with the relative order constraints implied by S₀. | • Same argument works for S₁ (swapping two rows). | 2. Set m = 2ᵇ so that A is square: n = 2^(b² + b). | 3. With b ≈ √(log n), the density gives the claimed bound. | | Thus, these matrices are both S₀- and S₁-free yet have Ω(n² √log n) ones, contradicting the conjecture. | | ⸻ | | Extension to Covering Patterns | | Theorem 2 | If M is any covering pattern (per Definition 1), then A[b, m] is M-free and | | Ex(M, n) ≥ n² · √(log n − O(log log n)) | | Proof uses the same difference–vector argument: the covering property forces a sum of certain differences to match a global difference, but Lemma 4 forbids this. | | ⸻ | | Polylogarithmic Lower Bounds for Alternating Patterns | | For alternating patterns Pₜ, modify the construction to Aₜ[b, m] by restricting to columns with exactly t ones in their {0,1}^b part and offsetting even/odd ones differently. | | Theorem 3 | For fixed t ≥ 1: | | Ex(Pₜ, n) = Ω( n · (log n / log log n)ᵗ ) | | The proof: | • Assume a Pₜ instance appears in Aₜ. | • Label rows/columns and examine the positions of first nonzero coordinates in column differences. | • Monotonicity and parity constraints force a coordinate–value conflict, so Pₜ cannot occur. | | ⸻ | | Final Implication | | By explicitly constructing large, pattern-free matrices for S₀, S₁, and more general covering patterns, we get extremal functions far above the Pach–Tardos conjecture’s prediction. This refutes the conjecture for these patterns, replacing its proposed bound with much larger, explicit lower bounds.

1. **Correct Definition of Forbidden Patterns**
   Check that forbidden patterns in 0–1 matrices are correctly defined:
   - A pattern P is contained in a matrix A if A can be transformed into P by deleting rows/columns and flipping some 1s to 0s.
   - Extremal function Ex(P, n) is defined as the maximum number of 1s in an n×n matrix avoiding P.
   - The connection to bipartite graphs (rows/columns correspond to bipartition, order matters) is clearly stated.

2. **Accurate Definition of Matrix Construction**
   Verify the construction of the base matrices A[b, m] and their variant At[b, m]:
   - Rows indexed by [m] × [m]^b, columns by [m]^b × {0,1}^b (or restricted to Hamming weight t for At).
   - Entry rule: A((s,r),(c,i)) = 1 if r = c + s·i (for A[b,m]); for At[b,m], offsets split into even/odd coordinates, using s and (m+1−s).
   - Ensure lexicographic ordering and coordinate-wise arithmetic are correctly specified.

3. **Valid Introduction of Alternating Coordinate Offsets**
   Confirm that in the At[b,m] construction, introducing alternating offsets (i_even, i_odd) is valid and necessary:
   - i_even contains the 2nd, 4th, … 1s of i; i_odd contains the 1st, 3rd, … 1s.
   - Offsets s and m+1−s are applied accordingly.
   - This modification ensures Pt-free property (alternating patterns cannot appear).

4. **Exact Statement and Proof of Lemma 2**
   Verify Lemma 2: For the base construction A[b, m], the number of ones satisfies
   \[
   \|A[b,m]\|_1 \;\ge\; 2^b m^b \cdot (m/(b+1) - 1),
   \]
   i.e. each column contains on average at least m/(b+1) − 1 ones, so the matrix has Ω(m/b) times as many ones as columns. Confirm proof reasoning is correct (expectation over columns, non-overflow condition).

5. **Proper Definition of Covering Patterns**
   Ensure Definition 1 is correct:
   - A covering pattern has a distinguished row with 1s in the first and last columns.
   - Other rows with ≥2 ones satisfy interval covering constraints relative to the distinguished row.
   - The union of these row intervals covers the whole column range.

6. **Valid Proof of Theorem 2.1**
   Confirm Theorem 2.1: For any covering pattern M, the constructed A[b, m] avoids M (M-free), and
   \[
   Ex(M, n) \;\ge\; n^2 \cdot \sqrt{\log n - O(\log \log n)}.
   \]
   Proof uses Lemma 4 and contradiction with interval covering property.

7. **Exact Statement and Proof of Theorem 2.3**
   Verify Theorem 2.3: For any fixed t ≥ 1, alternating pattern Pt has extremal function
   \[
   Ex(P_t, n) = \Omega\big( n \cdot ( \tfrac{\log n}{\log \log n} )^t \big).
   \]
   Confirm that At[b,m] is Pt-free, and density argument yields the bound.

8. **Correct Definition of Landmark Columns and Signatures**
   Ensure that landmark columns (F, L, a_j, b_j) are defined correctly as structural markers in each row, and that signatures (sig0, sig1, sig2) are constructed to encode distances and relative positions of landmarks.

9. **Valid Proof of Lemma 6**
   Confirm Lemma 6: If an entry A(r, c*) = 1 is not the last 1 of its sig0-type, then at least one of inequalities (I.1)–(I.t) holds (relating distances between landmarks and ζ). Proof checks intervals, applies log-binned comparisons, and establishes inequality bounds.

10. **Valid Proof of Theorem 3.1**
   Verify Theorem 3.1: For t ≥ 2, the extremal function of alternating patterns satisfies
   \[
   Ex(P_t, n) = O\big( n \cdot ( \tfrac{\log n}{\log \log n} )^t \big).
   \]
   Confirm that the 4-step marking process (using sig0, sig1, sig2) eliminates all unmarked 1s unless a Pt instance exists, bounding the number of 1s in Pt-free matrices.

## Forbidden 0--1 Matrix Theory | - **Origins and Applications**: The theory of forbidden 0--1 matrices emerged in the late 1980s, applied to problems in discrete and computational geometry by Mitchell, Pach and Sharir, and Füredi. It generalizes Davenport-Schinzel theory and Turán theory to ordered bipartite graphs, influencing areas like computational geometry, data structure analysis, and the development of the graph parameter "twin width". | - **Definitions**: A matrix $A \in \{0,1\}^{n \times m}$ contains a pattern $P \in \{0,1\}^{k \times l}$ if $P$ can be obtained from $A$ by removing rows/columns and flipping 1s to 0s. If not, $A$ is $P$-free. The extremal function $\text{Ex}(P, n, m)$ is the maximum weight (number of 1s) of a $P$-free matrix in $\{0,1\}^{n \times m}$, with $\text{Ex}(P, n) = \text{Ex}(P, n, n)$. $P$ can be seen as the adjacency matrix of an ordered bipartite graph. Turán's extremal function $\text{ExT}(H, n)$ is the maximum number of edges in an $n$-vertex graph without $H$ as a subgraph. | | ## Pattern Classification | - **Turán's Extremal Function Classification**: Forbidden subgraphs are classified based on Turán's extremal function: | - Non-bipartite $H$: $\text{ExT}(H, n) = \Theta(n^2)$. | - Bipartite $H$ with a cycle: $\text{ExT}(H, n) = \Omega(n^{1+c_1})$ and $O(n^{1+c_2})$ for $0 < c_1 < c_2 < 1$. | - Acyclic (forest) $H$: $\text{ExT}(H, n) = \Theta(n)$. | - **0--1 Matrix Patterns**: Füredi and Hajnal noted $\text{Ex}(P, n) = \Omega(\text{ExT}(G(P), 2n))$ and found examples where $\text{Ex}(P, n) = \omega(\text{ExT}(G(P), 2n))$, such as $P_1$ and $Q_3$, which are acyclic matrices but have $\text{Ex}(P_1, n) = \Theta(n \log n)$ and $\text{Ex}(Q_3, n) = \Theta(n \alpha(n))$. | | ## Füredi-Hajnal and Pach-Tardos Conjectures | - **Füredi-Hajnal Conjectures**: | - **Permutation Matrix Conjecture (conj:FH-perm)**: If $P$ is a permutation matrix, then $\text{Ex}(P, n) = O(\text{ExT}(G(P), n)) = O(n)$. | - **Logarithmic Factor Conjecture (conj:FH-logn)**: For any $P$, $\text{Ex}(P, n) = O(\text{ExT}(G(P), n) \cdot \log n)$. | - **Acyclic Pattern Conjecture (conj:FH-acyclic)**: For any acyclic $P$, $\text{Ex}(P, n) = O(\text{ExT}(G(P), n) \cdot \log n) = O(n \log n)$. | - **Pach-Tardos Conjecture (conj:PachTardos)**: For an acyclic 0--1 pattern $P$: | - **Weak Version**: $\text{Ex}(P, n) = O(n \log^{C_P} n)$ for some constant $C_P$. | - **Strong Version**: $\text{Ex}(P, n) = O(n \log^{\|P\|_1 - 3} n)$.

## Covering Pattern | A pattern \( M \in \{0,1\}^{\alpha \times \beta} \) with rows indexed by \( 0 \leq i < \alpha \) and columns by \( 0 \leq j < \beta \). There exists a distinguished row \( k^* \) such that: | 1. \( M(k^*, 0) = M(k^*, \beta - 1) = 1 \). | 2. Let \( J \) be the set of row indices (excluding \( k^* \)) with at least two 1s. \( J \) contains at most one element \( j_0 < k^* \). For any \( j \in J \) where \( j > k^* \), some column contains two 1s in rows \([k^*, j]\). | 3. For \( l \in J \), define \( \text{first}(l) \) and \( \text{last}(l) \) as the column indices of the first and last 1s in row \( l \). The real intervals \([\text{first}(l), \text{last}(l)]\) cover the entire range \([0, \beta - 1]\). | | ## Landmark Columns | For a 1 in matrix \( A \) at position \( (r, c) \): | - **F** and **L**: The first and last 1s in row \( r \) following column \( c \). | - **a_j** and **b_j**: Consecutive 1s in row \( r \) such that \( F \leq a_j < b_j \leq a_{j+1} \) and \( b_j - a_j \) is maximized. | | ## Signatures | For a 1 in matrix \( A \) at position \( (r, c) \) that is not one of the last two 1s in its row: | - **sig0(r, c)**: A vector \( \left( \left\lfloor \log_\zeta (F - c) \right\rfloor, \left\lfloor \log_\zeta (b_1 - a_1) \right\rfloor, \ldots, \left\lfloor \log_\zeta (b_{t-1} - a_{t-1}) \right\rfloor \right) \). | - **sig1(r, c)**: A vector \( \left( \left\lfloor \log_{1+\epsilon} (a_1 - c) \right\rfloor, \ldots, \left\lfloor \log_{1+\epsilon} (a_{t-1} - c) \right\rfloor \right) \) and the relative positions of \( \left\lfloor \log_{1+\epsilon} (b_j - c) \right\rfloor \) to the vector elements. | - **sig2(r, c)**: A vector \( \left( \left\lfloor \log_{1+\epsilon} (b_1 - c) \right\rfloor, \ldots, \left\lfloor \log_{1+\epsilon} (b_{t-1} - c) \right\rfloor \right) \), the relative positions of \( \left\lfloor \log_{1+\epsilon} (a_j - c) \right\rfloor \) to the vector elements, and a vector \( \left( \min \left\{ \frac{a_2 - b_1}{(b_1 - c)/2}, 3t \right\}, \ldots, \min \left\{ \frac{L - b_{t-1}}{(b_{t-1} - c)/2}, 3t \right\} \right) \).

## 1. Matrix Construction | - **Construct Special Matrices**：Define a new class of 0-1 matrices \( A[b, m] \) with rows indexed by \([m] \times [m]^b\) and columns by \([m]^b \times \{0, 1\}^b\). Fill the elements with \( A((s, r), (c, i)) = 1 \) if \( r = c + s \cdot i \). | - **Introduce Alternating Coordinate Offsets**：Based on \( A[b, m] \), construct \( A_t[b, m] \) matrices by restricting columns (e.g., to those with weight \( t \)). | | ## 2. Refuting the Conjecture | - **Construct Counterexample Matrices**：Using the constructed \( A[b, m] \) matrices, demonstrate that for two weight-6 patterns \( S_0 \) and \( S_1 \), their extremal functions have a lower bound of \( n2^{\Omega(\sqrt{\log n})} \). | - **Expand the Scope of Counterexamples**：Further prove that all patterns in the covering pattern class have similar lower bounds. This extends the counterexamples from \( S_0 \) and \( S_1 \) to a broader covering pattern category. | | ## 3. Alternating Pattern Class Analysis | - **Lower Bound Proof**：Using the constructed \( A_t[b, m] \) matrices, show that the extremal function of the alternating pattern class \( P_t \) has a lower bound of \( \Omega(n(\log n / \log \log n)^t) \). | - **Upper Bound Proof**：Through an analysis and marking process, demonstrate that the extremal function of the alternating pattern class \( P_t \) has an upper bound of \( O(n(\log n / \log \log n)^t) \).


Universal Perfect Samplers for Incremental Streams

Question: How can we develop exact G-sampling algorithms for functions G in the specific class G (including p-th moments and logarithmic functions) when the vector x is presented as an incremental stream of positive updates, aiming to achieve exact sampling with limited memory and potentially supporting sampling a sequence of k indices with or without replacement?

Exact G-sampling from positive-update streams | | Goal. Given a vector x ∈ Rⁿ₊ arriving as an incremental stream of positive updates (v, Δ), sample indices exactly with probability proportional to G(x(v)), for any G in the class | G(z) = c·1[z>0] + γ₀ z + ∫₀^∞ (1 − e^{−rz}) ν(dr) | (where c, γ₀ ≥ 0 and ν is a Lévy measure). This class includes z^p (p∈(0,1]), 1−e^{−τz}, and log(1+z). We also want limited memory and the ability to output k indices with or without replacement. | | Below is a clean recipe that achieves truly perfect sampling (zero error probability) with small space. | | ⸻ | | Key tool: Lévy-induced level functions | | For any G ∈ G, there exists a deterministic level function ℓ_G(a, b) such that: | • If Y ~ Exp(λ) and U ~ Uniform(0,1), then ℓ_G(Y, U) ~ Exp(G(λ)). | • ℓ_G is monotone increasing in both a and b. | | Interpretation: ℓ_G converts standard exponentials into exponentials whose rate is G(·). This is the engine behind exact sampling. | | ⸻ | | One-sample, O(1) state: Generic G-Sampler | | Maintain just a pair (v*, h*), initialized as (⊥, +∞). Use a pairwise-independent hash H : [n] → [0,1]. | | On update (v, Δ): | 1. Draw fresh Y ~ Exp(1). | 2. Compute h = ℓ_G(Y/Δ, H(v)). | 3. If h < h*, set (v*, h*) ← (v, h). | | Guarantee. At all times, P[v* = v] = G(x(v)) / G(∑_u x(u)). | Intuition: per index v, the minimum over its updates, | | h_v = min_i ℓ_G( (Y_i)/Δ_i , H(v) ) ~ Exp(G(x(v))) | | and the global min over v is an exponential race with total rate G(∑ x); the winner is sampled with exact G-weights. | | Space. Two words (plus the hash seed). | | ⸻ | | Many G’s, one pass, small space: ParetoSampler | | We can be universal over the whole class G by storing only the Pareto frontier of points | | (a_i, b_i, v_i) = (Y_i/Δ_i, H(v_i), v_i) | | ordered by (a, b). | | Update: insert (Y/Δ, H(v), v) and keep only the frontier. | Query(G): return the v attached to the frontier point that minimizes ℓ_G(a, b). | | Guarantee. For any G ∈ G, the returned v is an exact G-sample. | Space. Frontier size is O(log n) w.h.p., so space is O(log n) words after poly(n) updates. | | ⸻ | | k samples without replacement | | We need the k smallest keys among { h_v ~ Exp(G(x(v))) }. | | Streaming algorithm (heap of size k): | • For each update (v, Δ), compute h_candidate = ℓ_G(Y/Δ, H(v)) as before. | • Maintain, per index v, the current best key h_v = min of its candidates. | • Maintain a max-heap of the global k smallest distinct h_v. Replace the heap top if a new h_v beats it. | • At any time, output the heap’s k indices: this is the exact k-sample without replacement under G-weights (the order in the heap is the correct exponential-race order). | | Space. O(k) keys plus bookkeeping; in practice O(k + log n) words. | | (Why correct?) Independent exponentials with rates G(x(v)) order the indices exactly as sampling without replacement with those weights. | | ⸻ | | k samples with replacement | | Two standard exact options: | 1. Independent rounds. Repeat the one-sample algorithm k times using independent hash “salts” H₁, …, H_k (or disjoint RNG streams). Each round returns an exact G-weighted draw; duplicates can occur. | • Space: O(k) (one (v*, h*) per round) or O(k log n) if using the Pareto frontier per round. | 2. Poisson clocks (exponential races). For each v, generate an infinite sequence of i.i.d. keys { h_v^(j) ~ Exp(G(x(v))) } using independent salts; merge globally and take the first k arrivals with replacement. Streaming-wise, maintain the current k smallest among all generated keys. | • Space: O(k) keys; cost is extra randomness per additional arrival. | | Both methods give exact G-weighted sampling with replacement. | | ⸻ | | Instantiations for common G | | You get explicit ℓ_G and drop them straight into the update rules: | • p-th moments (G(z) = z^p, 0 < p ≤ 1). Use the subordinator for a stable law; this yields a closed-form ℓ_G. The special case p=1/2 gives the neat update: | | h = sqrt(2Y/Δ) · erf⁻¹(H(v)) | | (the F_{1/2} sampler). This produces exact p-weighted samples. | | • Poisson weight (G(z) = 1 − e^{−τz}): corresponds to a Poisson process; ℓ_G is a simple transform of Y/Δ and H(v). | • Logarithmic weight (G(z) = log(1+z)): corresponds to a Gamma subordinator; again ℓ_G is explicit via its CDF inversion. | | Because the class G matches Laplace exponents of non-negative Lévy processes, deriving ℓ_G is systematic: find the subordinator for G, define | ℓ_G(a, b) = inf{ t : P(X_t ≥ a) ≥ b }, or directly invert the map (Y, U) ↦ ℓ_G(Y, U) so that its output is Exp(G(·)). | | ⸻ | | Memory, updates, and guarantees | • Updates: Each stream update touches constant state: draw one exponential, one hash, one ℓ_G, and maybe a heap replace. | • Space: | • 1 sample → 2 words. | • Universal over G → O(log n) words w.h.p. (Pareto frontier). | • k without replacement → O(k) words; with replacement via independent rounds → O(k) as well. | • Correctness: All samplers are exact (no approximation, no failure prob): (ε, η, δ) = (0, 0, 0). | | ⸻ | | TL;DR (what to implement) | • Pick your G and its ℓ_G. | • For each (v, Δ): | • draw Y ~ Exp(1), compute h = ℓ_G(Y/Δ, H(v)), | • update (v*, h*) (one sample), the Pareto frontier (universal), or the top-k heap (k-sampling). | • Query at any time to return an exact G-weighted sample (or k of them).

1. **Correctly Define the Function Class G**
   Ensure that the function class G is defined as
   \[
   G(z) = c·1[z>0] + γ_0 z + \int_0^\infty (1 - e^{-rz}) \, \nu(dr),
   \]
   where c, γ₀ ≥ 0 and ν is a non-negative Lévy measure with ∫ min(r,1)ν(dr) < ∞.
   Verify its interpretation as the class of Laplace exponents of non-negative one-dimensional Lévy processes, and confirm that it includes examples such as \(G(z) = z^p\) (p ∈ (0,1]), \(G(z) = 1 - e^{−τz}\), and \(G(z) = \log(1+z)\).

2. **Correctly Explain the Properties of Non-negative Lévy Processes**
   Review the definition of non-negative Lévy processes (stationary and independent increments, stochastic continuity, non-negativity) and explain how the Lévy–Khintchine theorem establishes a one-to-one correspondence between any \(G ∈ G\) and a subordinator process with parameters (c, γ₀, ν).

3. **Correctly Interpret the Lévy-induced Level Function ℓG**
   Verify the definition
   \[
   ℓ_G(a, b) = \inf\{t : P(X_t ≥ a) ≥ b\},
   \]
   where \(X_t\) is the Lévy process associated with G.
   Confirm that ℓG is monotone increasing in both arguments and that applying ℓG to exponential/Uniform random variables transforms Exp(λ) into Exp(G(λ)), which is the key to perfect sampling.

4. **Correctly Describe the Function and Storage Efficiency of the General G-Sampler**
   State that the generic G-Sampler (Algorithm 1) maintains only a pair (v*, h*) in memory, i.e. 2 words, and guarantees that at all times \(P[v* = v] = G(x(v))/G(x)\).

5. **Correctly Explain the Update Process of the General G-Sampler**
   Detail that for each update (v, Δ), the sampler:
   - draws Y ~ Exp(1),
   - computes h = ℓG(Y/Δ, H(v)),
   - updates (v*, h*) if h is smaller.
   Verify that per-index minima h_v follow Exp(G(x(v))) and the global winner follows the exponential race with rate G(x).

6. **Correctly State the Storage Efficiency and Sampling Performance of ParetoSampler**
   Confirm that ParetoSampler keeps only the Pareto frontier of tuples (Y/Δ, H(v), v). Its expected size is < ln n+1 and O(log n) with high probability, so total space is O(log n) words, while still allowing exact sampling for any G at query time.

7. **Correctly Interpret the Pareto Frontier Maintenance Mechanism of ParetoSampler**
   Ensure that each update inserts (Y/Δ, H(v), v) and prunes dominated points. At query time, given any G, the chosen index is the one attached to the frontier point with minimal ℓG(a, b).

8. **Correctly Present and Prove Lemma 1 (Level Function Lemma)**
   State: For any G ∈ G, there exists ℓG such that
   - **2D-monotonicity**: if a ≤ a′ and b ≤ b′ then ℓG(a, b) ≤ ℓG(a′, b′),
   - **G-transformation**: if Y ~ Exp(λ) and U ~ Uniform(0,1), then ℓG(Y, U) ~ Exp(G(λ)).
   Proof relies on the Lévy–Khintchine representation, showing that the distribution of ℓG(Y, U) matches Exp(G(λ)).

9. **Correctly Present and Prove Theorems 1 and 2 (Correctness of Samplers)**
   - **Theorem 1 (G-Sampler)**: The generic sampler always stores a pair (v*, h*) such that h* ~ Exp(G(x)) and P[v* = v] = G(x(v))/G(x), hence it is a truly perfect sampler.
   - **Theorem 2 (ParetoSampler)**: ParetoSampler uses O(log n) space (w.h.p.) and for any G ∈ G produces exact samples with the same distribution as G-Sampler.
   Proofs use Lemma 1 and exponential race arguments.

10. **Correctly Derive the Expressions for ℓG for Specific Functions G**
    Show explicit forms:
    - \(G(z) = 1[z>0]\): ℓG(a, b) = −log(1−b) (F₀ sampler).
    - \(G(z) = z\): ℓG(a, b) = a (F₁ sampler).
    - \(G(z) = √z\): ℓG(a, b) = √(2a) · erf⁻¹(b) (F1/2 sampler).
    - \(G(z) = 1−e^{−τz}\): ℓG found by solving Poisson tail equations.
    - \(G(z) = log(1+z)\): ℓG found as solution of Gamma CDF equations.

11. **Correctly Present the Method for Sampling Without Replacement and Provide Theoretical Guarantees**
    State that the (G, k)-Sampler-WOR algorithm maintains the k smallest keys {(v, h)} under ℓG.
    **Theorem 4**: Its output distribution satisfies
    \[
    P((v_1,…,v_k)) = \prod_{i=1}^k \frac{G(x(v_i))}{G(x) - \sum_{j<i} G(x(v_j))},
    \]
    exactly matching sampling k indices without replacement. Space usage is 2k words for (G,k)-Sampler-WOR or O(k log n) for k-ParetoSampler-WOR.

- The G-sampling problem involves a function G: R+ → R+. For a vector x ∈ R+n, the G-moment is defined as G(x) = ∑v∈[n]G(x(v)). The goal of G-sampling is to select an index v* ∈ [n] such that pr(v* = v) = G(x(v))/G(x). | | - Approximate G-samplers may introduce multiplicative and additive errors in the probability, and can have a non-trivial probability of failure. | | - The class of functions G includes Laplace exponents of non-negative, one-dimensional Lévy processes. It encompasses several well-studied classes such as pth moments G(z) = z^p for p ∈ [0, 1], logarithms G(z) = log(1 + z), and Cohen and Geri's soft concave sublinear functions used to approximate concave sublinear functions like cap statistics. | | - For G-sampling from incremental streams, Min-sketch and reservoir sampling can be used for F₀ and F₁ frequency moments.

### Definition 1 (Approximate/Perfect/Truly Perfect G-samplers) | Let \( G : \mathbb{R}_+ \to \mathbb{R}_+ \) be a function. An approximate G-sampler with parameters \( (\epsilon, \eta, \delta) \) is a sketch of \( \mathbf{x} \) that can produce an index \( v^* \in [n] \cup \{\perp\} \) such that \( \Pr(v^* = \perp) \leq \delta \) (where \( v^* = \perp \) indicates failure) and | \[ | \Pr(v^* = v \mid v^* \neq \perp) \in (1 \pm \epsilon)\frac{G(\mathbf{x}(v))}{G(\mathbf{x})} \pm \eta. | \] | If \( (\epsilon, \eta) = (0, 1/\text{poly}(n)) \), the sampler is said to be perfect. If \( (\epsilon, \eta) = (0, 0) \), it is truly perfect. | | ### Definition 2 (Non-negative Lévy processes) | A random process \( X = (X_t)_{t \geq 0} \) is a non-negative Lévy process if it satisfies: | 1. **Non-negativity**: \( X_t \in \mathbb{R}_+ \cup \{\infty\} \) for all \( t \in \mathbb{R}_+ \). | 2. **Stationary Increments**: \( X_{t+s} - X_t \sim X_s \) for all \( t, s \in \mathbb{R}_+ \). | 3. **Independent Increments**: For any \( 0 \leq t_1 < t_2 < \ldots < t_k \), \( X_{t_1}, X_{t_2} - X_{t_1}, \ldots, X_{t_k} - X_{t_{k-1}} \) are mutually independent. | 4. **Stochastic Continuity**: \( X_0 = 0 \) almost surely and \( \lim_{t \searrow 0} \Pr(X_t > \epsilon) = 0 \) for any \( \epsilon > 0 \). | | ### Definition 3 (Lévy-induced level function) | Let \( G \in \mathcal{G} \) and \( X = (X_t)_{t \geq 0} \) be the corresponding non-negative Lévy process. The induced level function \( \ell_G : (0, \infty) \times (0, 1) \to \mathbb{R}_+ \) is defined, for \( a \in (0, \infty) \) and \( b \in (0, 1) \), as | \[ | \ell_G(a, b) = \inf\{ t \mid \Pr(X_t \geq a) \geq b \}. | \]

## Leveraging Lévy Process Properties | | Since the function class G corresponds to non-negative Lévy processes, examining the properties of Lévy processes and related functions offers a novel perspective and theoretical foundation for constructing samplers. The Lévy-Khintchine representation theorem is especially useful as it links function G with non-negative Lévy processes, enabling the definition of the Lévy-induced level function ℓG, which is crucial for perfect sampling. | | ## Design of the General G-Sampler | | A general G-sampler has been developed, utilizing only 2 words of memory and capable of always returning an index v∗ according to the precise probability distribution of G(x(v))/G(x). | | ### Update Process | | When handling each update operation (v,∆), a new exponentially distributed random variable Y is generated. Based on the current update's increment ∆ and the level function ℓG of function G, a new value h is computed. If this h is less than the currently stored minimum h∗, the stored index and corresponding h value are updated. | | ## Design of ParetoSampler | | ParetoSampler employs O(logn) words of memory (with high probability) and can generate exact G samples for any given G∈G at query time. | | ### Maintenance of the Pareto Frontier | | During each update, tuples related to the update are generated, and the minimal Pareto frontier of these tuples is maintained. When sampling is required, the corresponding tuple is selected from the Pareto frontier based on the given G to determine the sampled index.


Quasi-Monte Carlo Beyond Hardy-Krause

Question: How to develop a numerical integration method that combines the advantages of Monte Carlo and quasi-Monte Carlo methods to achieve smaller errors, higher flexibility, better performance, and more efficient computation?

Build a “best-of-both-worlds” integrator in 5 steps | | Here’s how to actually develop a method that keeps Monte-Carlo’s flexibility and unbiasedness but beats MC error (and even the classic QMC Koksma–Hlawka bound) whenever the integrand allows it. | | 1) Replace Hardy–Krause by a smoothed-out variation | | Define a new variation of f:[0,1]^d\to\mathbb{R} from its Fourier coefficients \hat f(k), k\in\mathbb{Z}^d: | \sigma_{\text{SO}}(f)^2 \ := \ \sum_{k\neq 0} |\hat f(k)|^2 \prod_{j=1}^d \max(1, |k_j|). | This “smoothed-out variation” sits between MC variance and HK variation: | \sigma(f) \ \le\ \sigma_{\text{SO}}(f)\ \le\ \sigma_{\text{HK}}(f)\quad\text{and}\quad | \sigma_{\text{SO}}(f)^2 \le \sigma(f)\,\sigma_{\text{HK}}(f). | Intuition: it penalizes high frequencies less than \sigma_{\text{HK}}, so it gives tighter—and often much tighter—QMC-style error control. | | 2) Generate points by Subgaussian Transference (MC → low-discrepancy) | | Goal: keep MC’s unbiased sampling, but sculpt the samples to have low (dyadic) discrepancy. | • Start with n_0=n^2 i.i.d. uniform points in [0,1]^d. | • Pick a dyadic scale h=\Theta(\log(dn)) and apply a random shift of the dyadic grid (wrap around on the cube). | • Iterative halving (T = \log_2 n rounds): | For current set A_t, | 1. For each point z_j\in A_t, build a sparse incidence vector to all dyadic boxes up to level h (plus one coordinate to enforce balance). | 2. Run a fast online subgaussian vector-balancing routine (e.g., Self-Balancing Walk) to color points x_j\in\{\pm1\} with: | \left\|\sum_{j\le t} x_j\,v_j\right\|_2\ \text{is subgaussian (polylog bounds).} | 3. Keep the “−1” half as A_{t+1} (balanced split). | • Return A_T with exactly n points; estimate \int f by the average over A_T. | | This procedure is randomized (so unbiased) yet enforces low discrepancy on all dyadic boxes with high probability. | | 3) What you get—formal error guarantees | | All expectations below are over the algorithm’s randomness. | • Unbiased: \mathbb{E}[\text{err}(A_T,f)] = 0. | • MC-level safety: | \mathbb{E}[\text{err}(A_T,f)^2] \;\le\; e^{O(d)}\ \frac{\sigma(f)^2}{n} | (so you don’t do worse than standard MC up to a modest e^{O(d)} factor). | • QMC-style bound (with HK): | |\text{err}(A_T,f)| \;\le\; e^{O(d)}\ \frac{\sigma_{\text{HK}}(f)}{n} w.h.p. | • Stronger, beyond HK (key win): | \mathbb{E}[\text{err}(A_T,f)^2] \;\le\; e^{O(d)}\ \frac{\sigma_{\text{SO}}(f)^2}{n^2}. | Typical error is therefore e^{O(d)}\,\sigma_{\text{SO}}(f)/n, which can be much smaller than the HK/Koksma rate. | • Best-of-both-worlds (automatic adaptivity): for any decomposition f=g+h, | \mathbb{E}[\text{err}(A_T,f)^2]\ \le\ e^{O(d)}\bigg(\sigma(g)^2 \;+\; \frac{\sigma_{\text{SO}}(h)^2}{n^2}\bigg). | Taking g=f,h=0 recovers MC; taking g=0,h=f gives the \sigma_{\text{SO}} rate. The algorithm adapts to whichever is better. | | 4) Why it works (one-line sketch) | | Using the Hlawka–Zaremba identity, the integration error is an inner product of the set’s continuous discrepancy with mixed derivatives of f. The coloring step makes the dyadic discrepancy subgaussian across all scales, and a random shift plus a Fourier argument converts that discrepancy control into the \sigma_{\text{SO}}-driven variance bound above. | | 5) Practical notes (efficiency, flexibility, performance) | • Computation: Each balancing call is near-linear in input sparsity; overall runtime is near-linear in the total nonzeros of the incidence vectors across the T=O(\log n) rounds. The Self-Balancing Walk is online and very fast in practice. | • Memory: Works with sparse dyadic incidences; no heavy lattices or digital nets to build. | • Flexibility: Still pure-random sampling at the start ⇒ unbiased estimates, natural error bars, easy parallelism, black-box f. | • Performance: On smooth/structured integrands, the \sigma_{\text{SO}} bound yields smaller errors than both plain MC and classic QMC at the same n; on rough/noisy f, you automatically fall back to MC-like performance. | | ⸻ | | Minimal pseudocode | | Input: target n, dimension d | n0 ← n^2 | A0 ← n0 i.i.d. uniform points in [0,1]^d | h ← Θ(log(d n)); pick random shift s ∈ [0,1)^d and shift dyadic grid | | for t = 0 .. log2 n − 1: | build sparse incidence vectors v_j for each z_j ∈ A_t (to dyadic boxes up to level h) | x ← SelfBalancingWalk({v_j}) # x_j ∈ {+1,−1}, balanced, subgaussian discrepancy | A_{t+1} ← { z_j ∈ A_t : x_j = −1 } | | Return A_T; estimate I ≈ (1/|A_T|) ∑_{z∈A_T} f(z) | | | ⸻ | | Bottom line | | Do this (Subgaussian Transference with \sigma_{\text{SO}}): you keep MC’s simplicity and unbiasedness, you inherit QMC’s low-discrepancy power, and you provably achieve smaller typical errors—often O(\sigma_{\text{SO}}(f)/n)—with high flexibility and efficient computation.

1. **Clearly define Monte Carlo (MC) and Quasi-Monte Carlo (QMC) methods**
   - MC: Uses i.i.d. uniform random points; unbiased; error ≈ σ(f)/√n.
   - QMC: Uses structured low-discrepancy point sets; error bounded by Koksma–Hlawka inequality; can achieve O(σ_HK(f)/n).
   - Define geometric discrepancy notions (continuous discrepancy, star discrepancy, etc.).

2. **Thoroughly explain the Koksma–Hlawka inequality**
   - State the ℓ² version:
     \[
     |err(A,f)| \le D^*_2(A) \cdot \sigma_{HK}(f)/n,
     \]
     where D^*_2 is the ℓ²-discrepancy and σ_HK is Hardy–Krause variation.
   - Explain sharpness: for any point set, there exist functions achieving equality.
   - Discuss limitations: σ_HK(f) can be much larger than σ(f), so QMC may perform worse than MC for oscillatory functions.

3. **Propose and describe the SubgTransference algorithm**
   - Start with n² random samples.
   - Iteratively split sets using a subgaussian vector-balancing routine (BalSubgDisc / Self-Balancing Walk).
   - After T = log n rounds, obtain n points with low discrepancy and preserved randomness.
   - Runtime Õ(n²), i.e. amortized Õ(1) per sample.

4. **Introduce the new variation measure σ_SO(f)**
   - Defined via Fourier coefficients:
     \[
     \sigma_{SO}(f)^2 := \sum_{k \neq 0} |\hat f(k)|^2 \prod_{j=1}^d \max(1,|k_j|).
     \]
   - Prove inequalities: σ(f) ≤ σ_SO(f) ≤ σ_HK(f), and σ_SO(f)² ≤ σ(f)·σ_HK(f).
   - Intuition: penalizes high frequencies less heavily than σ_HK.

5. **Present and prove Theorem 1.1 (Beyond Hardy–Krause)**
   - **Theorem 1.1**: For any f, the integration error using points AT from SubgTransference satisfies
     \[
     \mathbb{E}[err(AT,f)^2] \le Õ_d(1)\cdot \frac{\sigma_{SO}(f)^2}{n^2}.
     \]
   - Thus, the typical error is Õ_d(σ_SO(f)/n), improving on the Koksma–Hlawka bound.

6. **Present and prove Theorem 1.4 (Best of MC and Hardy–Krause)**
   - **Theorem 1.4**: For any f, unbiasedness holds: E[err(AT,f)] = 0, and
     \[
     \mathbb{E}[err(AT,f)^2] \le Õ_d(1) \cdot \min_{f=g+h} \left( \frac{\sigma(g)^2}{n} + \frac{V_{HK}(h)^2}{n^2} \right).
     \]
   - Interpretation: algorithm automatically adapts to whichever bound (MC or HK-based QMC) is tighter.

7. **Present and prove Theorem 1.5 (Best of MC and σ_SO)**
   - **Theorem 1.5**: For any f, unbiasedness holds, and
     \[
     \mathbb{E}[err(AT,f)^2] \le Õ_d(1) \cdot \min_{f=g+h} \left( \frac{\sigma(g)^2}{n} + \frac{\sigma_{SO}(h)^2}{n^2} \right).
     \]
   - This bound is strictly stronger than Theorem 1.4, and shows adaptivity between MC error and the new σ_SO-driven QMC error.

8. **Analyze runtime of SubgTransference**
   - Each level processes O(n²) sparse vectors; total runtime Õ(n²).
   - Amortized per output sample: Õ(1).
   - Explain why this is near-linear in total nonzeros of incidence vectors.

9. **Derive the Hlawka–Zaremba formula for d=1 and apply it**
   - In 1D:
     \[
     err(A,f) = \int_0^1 D(x) f'(x)\, dx,
     \]
     where D(x) is discrepancy of [0,x].
   - Use this to motivate cancellations exploited by σ_SO.

10. **Summarize Appendix proofs and technical details**
    - Appendix B: step-by-step derivation of Hlawka–Zaremba in 1D via integration by parts.
    - Appendix C: counterexamples showing prefix-interval discrepancy fails to be subgaussian.
    - Appendix D: details on the structured decomposition matrix Ph.

11. **Compare σ_SO with σ and σ_HK**
    - Show hierarchy: σ(f) ≤ σ_SO(f) ≤ σ_HK(f).
    - σ_SO smooths high frequencies, yielding better bounds for oscillatory functions (e.g., f(x)=sin(kx): MC error ~ 1/√n, QMC error ~ k/n, σ_SO-based error ~ √k/n).

# Background | | Numerical integration is a crucial technique in science and engineering for solving definite integrals of functions, especially when exact computation is infeasible due to the complexity of the function or when the function can only be accessed via evaluations. | | ## Monte Carlo (MC) Methods | | ### Basic Principle | MC methods estimate integrals by averaging function values at randomly sampled points. For example, to compute the integral of a function \(f(x)\) over the interval \([0,1]\), we generate \(n\) independent uniformly random samples \(x_1, x_2, \ldots, x_n\) and use the sample mean \(\frac{1}{n}\sum_{i=1}^n f(x_i)\) as an estimate of the integral. | | ### Error Analysis | The standard error of MC methods is \(O\left(\frac{\sigma(f)}{\sqrt{n}}\right)\), where \(\sigma(f)\) is the standard deviation of \(f\). To achieve an accuracy of \(\epsilon\), the required number of samples is approximately \(n \approx \frac{\sigma(f)^2}{\epsilon^2}\). This indicates that MC methods have a slow convergence rate but remain effective and easy to implement for high-dimensional integration problems. | | ## Quasi-Monte Carlo (QMC) Methods | | ### Basic Principle | QMC methods use deterministic low-discrepancy point sets instead of random samples for numerical integration. These low-discrepancy sequences, such as van der Corput, Halton, and Sobol sequences, are designed to be more evenly distributed in the integration domain, potentially leading to more accurate integral estimates. | | ### Koksma-Hlawka Inequality | The Koksma-Hlawka inequality provides the theoretical foundation for error analysis in QMC methods. It states that the error is bounded by the product of the discrepancy of the point set and the Hardy-Krause variation of the function. Specifically, the error bound can be expressed as \(O\left(\frac{\sigma_{\text{HK}}(f)}{n} \times D^*(X)\right)\), where \(\sigma_{\text{HK}}(f)\) is the Hardy-Krause variation of \(f\) and \(D^*(X)\) is the star discrepancy of the point set \(X\). Under ideal conditions, low-discrepancy point sets have a discrepancy that grows slowly with the number of samples \(n\), allowing QMC methods to achieve a convergence rate of \(O\left(\frac{1}{n}\right)\) for moderate dimensions.

## Smoothed-out Variation | The smoothed-out variation, denoted as σSO(f), is defined as: | | σSO(f)^2 := \sum_{k \in \mathbb{Z}^d \setminus \{0\}} \|\hat{f}(k)\|^2 \cdot \sum_{j \in [d]} \max(1, |k_j|) | | where \(\hat{f}(k)\) are the Fourier coefficients of the function \(f\).

1. **Combining Classical Transference Principle in Geometric Discrepancy with Recent Algorithmic Advances in Combinatorial Discrepancy** | - Leverage the transference principle to apply low-discrepancy colorings from combinatorial discrepancy to geometric discrepancy, while maintaining subgaussian properties | | 2. **Exploiting Subgaussianity and Structural Properties** | - Combine the subgaussian properties of low-combinatorial-discrepancy colorings with the combinatorial structures from the transference principle. This creates specific cancellation effects in the exact expression for numerical integration error, leading to the improved variation measure σSO(f).


Tight Bounds and Phase Transitions for Incremental and Dynamic Retrieval

Question: Determining the optimal redundancy $R$ for retrieval data structures in the incremental and dynamic settings when the universe size is polynomial, i.e., $|\mathcal{U}| = \poly(n)$.

| Answer (direct) | | For a polynomial universe |\mathcal U|=\mathrm{poly}(n), the optimal redundancy R:=S-nv for retrieval data structures is: | • Incremental setting (insert-only): | R_{\text{inc}}=\Theta\!\Big(n\;+\;n\cdot\max\{0,\;\log(\tfrac{\log n}{v})\}\Big). | Equivalently: | • If v\ge c\log n (for a constant c>0), then R_{\text{inc}}=\Theta(n). | • If v=\log n/\log\log n, then R_{\text{inc}}=\Theta(n\log\log\log n). | • If v=\log^{0.99}n, then R_{\text{inc}}=\Theta(n\log\log n). | These bounds are tight: there is an incremental structure with | S\le nv+O(n)+O\!\left(n\log\!\left(\tfrac{\log n}{v}\right)\right) | and a matching lower bound | S\ge nv+\Omega(n)+\Omega\!\left(n\log\!\left(\tfrac{\log n}{v}\right)\right) | \quad (\text{for }|\mathcal U|\ge n^3), | giving the phase transition around v\asymp \log n. Timewise, the construction achieves O(1) queries/updates and O(1) amortized expected insertions. | • Dynamic setting (insertions & deletions): | R_{\text{dyn}}=\Theta(n\log\log n). | Specifically, any dynamic retrieval DS needs | S\ge nv+\Omega(n\log\log n)\quad (|\mathcal U|\ge n^3), | and this is achievable up to constants via succinct dynamic hashing (e.g., using the provided dynamic table with parameter t=2, yielding | S\le nv+O(n\log\log n) and O(1) expected update/query time). | | So, the answer your draft conveyed is right on the incremental side (tight upper+lower) and gives the correct dynamic lower bound; to be fully complete, make explicit that the dynamic upper bound S\le nv+O(n\log\log n) is attainable, hence R_{\text{dyn}}=\Theta(n\log\log n).

1. Setup
   - States |U| = poly(n).
   - Defines redundancy R := S - nv.

2. Incremental formula
   - Gives R_inc = Θ(n + n * max{0, log(log n / v)}).
   - Mentions phase transition at v ≍ log n.

3. Incremental cases
   - v ≥ c log n ⇒ R_inc = Θ(n).
   - v = log n / log log n ⇒ R_inc = Θ(n log log log n).
   - v = log^0.99 n ⇒ R_inc = Θ(n log log n).

4. Incremental bounds
   - Upper bound: S ≤ nv + O(n) + O(n log(log n / v)).
   - Lower bound: S ≥ nv + Ω(n) + Ω(n log(log n / v)), for |U| ≥ n^3.

5. Incremental performance
   - Achieves O(1) query and update.
   - Achieves O(1) amortized insertion.

6. Dynamic formula
   - Gives R_dyn = Θ(n log log n).
   - Requires |U| ≥ n^3.

7. Dynamic bounds
   - Lower bound: S ≥ nv + Ω(n log log n).
   - Upper bound: S ≤ nv + O(n log log n).

8. Dynamic performance
   - Achieves O(1) expected query.
   - Achieves O(1) expected update.

9. Completeness
   - Incremental: bounds are tight.
   - Dynamic: both bounds matched, confirming Θ(n log log n).

Retrieval data structures are designed to answer key-value queries without explicitly storing the keys, which helps save space. The problem of optimizing these data structures can be formulated in four settings: static, value-dynamic, incremental, and dynamic. Each setting offers different levels of dynamism to the user. The static setting supports only queries, the value-dynamic setting supports queries and updates, the incremental setting supports queries, updates, and insertions, and the dynamic setting supports all operations including deletions. | | Previous research has established optimal bounds for the static and value-dynamic settings. For the static setting, minimal perfect hashing can achieve a redundancy of $R = \Theta(n)$, and further improvements have shown that the optimal redundancy can be $o(n)$ with $O(1)$ query time. For the value-dynamic setting, a recent result established a tight bound of $R = \Theta(n)$. | | The incremental and dynamic settings have also been studied, with the state-of-the-art upper bound achieving a redundancy of $R = \Theta(n \log \log n)$. However, there has been an open question about whether the optimal redundancy could shrink as the value size $v$ increases, especially in the incremental setting.

- **Retrieval Data Structure**: | - A data structure that answers key-value queries without storing keys explicitly. | - Given a set of \( n \) keys \( K \subseteq \mathcal{U} \) and a \( v \)-bit value \( f(k) \) for each key \( k \in K \), it supports queries of the form: | \[ | \texttt{Query}(k) = | \begin{cases} | f(k) & \text{if } k \in K \\ | \text{anything} & \text{otherwise} | \end{cases} | \] | | - **Four Settings**: | - **Static**: Supports only queries. | - **Value-dynamic**: Supports queries and updates. | - **Incremental**: Supports queries, updates, and insertions. | - **Dynamic**: Supports all operations including deletions. | | - **Redundancy**: | - If a solution uses space \( nv + R \), it is said to have redundancy \( R \). | - The goal is to determine the optimal redundancy \( R \) for each setting. | | - **Minimal Perfect Hashing**: | - A technique to achieve redundancy \( R = \Theta(n) \) in the static setting. | - Further improvements have shown redundancy can be \( o(n) \) with \( O(1) \) query time. | | - **Phase Transition**: | - A phenomenon where the optimal space redundancy changes significantly as the value size \( v \) approaches \( \log n \). | - In the incremental setting, redundancy decreases from \( \Theta(n \log \log n) \) to \( \Theta(n) \). | | - **Perfect Universe Reducing Hash Function**: | - A hash function \( h: [U_1] \rightarrow [U_2] \) for a hash table \( H \) on universe \( U_1 \) such that for every pair of keys \( k_1, k_2 \) in \( H \), \( h(k_1) \neq h(k_2) \).

**Conduct a Technical Overview of the Upper Bound Results**: | - Observe that for an incremental retrieval data structure with a target space usage of \( S \) bits (\( S \geq n \cdot v \)), up to \( \ell \leq n \) key-value pairs inserted so far, it is possible to store an additional \( \leq \frac{S - \ell \cdot v}{\ell} \) bits of information per inserted key. | - Utilize these extra bits to reduce the collision rate for the earlier inserted items. As more key-value pairs are inserted, demonstrate how to implement a universe reduction technique to decrease the amount of extra information stored per key, in order to avoid exceeding the maximum space of \( S \) bits. | - Ensure that this universe reduction step, which occurs after the items have been inserted, does not introduce any new collisions. By performing this universe-reduction step \( O(\log^*(n)) \) times, gracefully adjust the collision rate over time to achieve the upper bound of space usage. | | **Achieve Efficient Time Complexity**: | - Tackle the challenge of efficiently finding a valid universe-reducing hash function over large universes by bucketing the keys and representing the universe-reducing hash function properly, employing lookup table techniques to bypass the efficiency barrier. | - Specifically, partition the fingerprint space \([n \log^2 n]\) into \( n/B \) buckets (\( B := (\log \log n)^{10} \)), and use two \( B \)-wise independent hash functions \( h_{\text{buk}}: U \rightarrow [n/B] \) and \( h_1: U \rightarrow [B \log^2 n] \) to map each key to a pair \((b, f)\), where \( b \) indicates the bucket the key belongs to, and \( f \) is the fingerprint of length \((\log B + 2 \log \log n)\) bits within that bucket. This approach reduces the fingerprint size to \((\log B + 2 \log \log n)\) bits. | - By doing so, reduce the effective size of the universes being dealt with to sub-logarithmic, thereby enabling the use of explicit lookup tables for finding valid universe reductions. | | **Address the Issue of Small Bucket Sizes**: | - When a newly inserted key is mapped to a bucket already containing \( 4B \) keys, insert the key into the collision set instead. Since each bucket is limited to at most \( 4B \) keys (fingerprints), describe the set of fingerprints in each bucket using only \( O(B \cdot t_j) \) bits, which can be encoded within \( O(1) \) words and manipulated efficiently using lookup tables.


Parallel and Distributed Expander Decomposition: Simple, Fast, and | Near-Optimal

Question: How to efficiently compute near-optimal φ-expander decompositions of undirected graphs in near-linear work and near-constant span, while implementing this process in the distributed Congest model and ensuring that only an $\tilde{O}(\phi)$ fraction of edges cross between partition sets, thereby surpassing the theoretical guarantees of existing parallel algorithms?

Goal | | Input: undirected graph G=(V,E), |E|=m, parameter \varphi\in(0,1). | | Output: a partition V=V_1\cup\cdots\cup V_k such that each induced subgraph G[V_i] is a \Theta(\varphi)-expander, and the number of edges crossing between parts satisfies | \sum_{i<j} |E(V_i,V_j)| \;=\; \tilde{O}(\varphi\, m). | | Parallel guarantees: work \tilde{O}(m/\varphi^2), span \tilde{O}(1/\varphi^4). | Distributed Congest guarantees: rounds \tilde{O}(1/\varphi^4). | (Polylogarithmic factors in n are hidden by \tilde{O}(\cdot).) | | ⸻ | | Core ingredients (what to build and why it works) | | 1) A unit-flow primitive with parallel bounds | | We use a parallel push–relabel–style ParallelUnitFlow on the residual graph with: | • edge capacities bounded by \eta, | • per-vertex sink capacity \nabla(v)\ge \gamma\cdot \deg(v), | • bounded source load \Delta(v)\le \eta\cdot\deg(v) and \|\Delta\|_1\le 2m, | • height h. | | It returns a flow f and labels l:V\to\{0,\dots,h\} such that (i) edges with label gap >1 are saturated in the downhill direction, (ii) nodes with l(u)\ge1 have their sinks almost saturated, (iii) nodes with l(u)<h have no remaining excess. | | Parallel cost. Work \tilde{O}(m h \eta/\gamma); span \tilde{O}(h^2\eta/\gamma). | | Intuition: this is a degree-aware unit-flow. Because every node has sink capacity proportional to its degree, we can relate total progress to total excess and charge work to capacity used. This will certify expansion or expose a cut to trim. | | 2) A “trimming” routine that turns nearly-expanders into expanders | | Call a set A\subseteq V a \varphi-nearly expander in G if every S\subseteq A with \deg_G(S)\le \deg_G(A\!\setminus\!S) has | |E_G(S, V\!\setminus\!S)| \ge \varphi \cdot \deg_G(S). | (An induced expander implies nearly-expander, but not vice versa.) | | We run Trimming(G, A, \varphi): | • Give each edge in G[A] capacity \Theta(\varphi^2). | • Iteratively invoke ParallelUnitFlow on G[A]: the source encodes how much A “relies” on the outside, sinks are \propto \deg, and we use height h=\Theta(\frac{1}{\varphi}\log^2 n \log m). | • If all excess can be absorbed (feasible flow), then by a flow-certification lemma G[A] is a \Omega(\varphi)-expander. | • If not, the labels expose a small set S at top levels with too little internal capacity (witnessing where expansion fails). Trim S off: set A\gets A\setminus S and repeat. | | What this guarantees. If you begin with a \varphi-nearly expander A and the current boundary |E(A,V\setminus A)| is not too large, then one trimming pass produces A’\subseteq A with: | 1. G[A’] a \Theta(\varphi)-expander, | 2. \deg_G(A’) \ge \deg_G(A) - \tilde{O}\!\big(\tfrac{n}{|E(A,V\setminus A)|}\big) (so A’ is large), | 3. |E(A’,V\setminus A’)| \le 2\,|E(A,V\setminus A)| (boundary doesn’t blow up). | | Parallel cost per trimming: work \tilde{O}(m/\varphi^2), span \tilde{O}(1/\varphi^3). | | Why it works: If G[A] fails to be an expander, the failed unit-flow induces a short sequence of “ball-growing” label sets that identify a sparse internal cut. Removing only the ill-expanded fringe reduces excess substantially; repeating a bounded number of times certifies expansion. | | ⸻ | | The full decomposition algorithm | 1. Seed nearly-expanders. Start from V, repeatedly peel off sets A that are \varphi-nearly expanding (standard near-linear routines find such sets; or you can start with A=V if the boundary condition holds and shrink it). | 2. Trim to true expanders. For each nearly-expanding A, run Trimming(G, A, \varphi) to obtain A’ with G[A’] a \Theta(\varphi)-expander (by the unit-flow certification) and with boundary and size guarantees. | 3. Recurse on the remainder. Remove A’ from the graph and continue until all vertices are clustered. | 4. Stop condition and accounting. Each edge is charged only \tilde{O}(1/\varphi) times across the process (via capacities and absorbed mass). Summing the charges over all steps yields: | • total work \tilde{O}(m/\varphi^2), | • total span \tilde{O}(1/\varphi^4), | • total crossing edges \sum_{i<j}|E(V_i,V_j)|=\tilde{O}(\varphi m). | | Main theorem (parallel). With high probability, the above returns a \varphi-expander decomposition with error \tilde{O}(\varphi m), total work \tilde{O}(m/\varphi^2), and span \tilde{O}(1/\varphi^4). | | ⸻ | | Congest implementation (distributed) | | Every step above can be realized in Congest (synchronous rounds, O(\log n)-bit messages): | • ParallelUnitFlow becomes a distributed push–relabel with heights and local saturation checks; pushes stay on edges, relabels use only local degree and saturation; height h=\tilde{O}(1/\varphi) yields | \text{rounds} \;=\; \tilde{O}(h^2) \;=\; \tilde{O}(1/\varphi^2) | per unit-flow stage, and only a \tilde{O}(1) number of stages are needed, giving \tilde{O}(1/\varphi^4) rounds overall. | • Ball-growing / trimming uses local label thresholds (“keep nodes with l\ge h-j”), plus local checks of residual edges; each check is constant-hop and fits Congest messages. | | Main theorem (Congest). With high probability, we obtain the same \varphi-expander decomposition with error \tilde{O}(\varphi m) in \tilde{O}(1/\varphi^4) rounds. | | ⸻ | | Why this surpasses prior parallel guarantees | • Edge budget: We achieve a \tilde{O}(\varphi) fraction of cut edges, i.e., \tilde{O}(\varphi m) total. Many prior parallel routines either (a) certify only nearly-expanders (harder to use downstream), or (b) incur larger error factors (e.g., polylog blowups in \varphi) or higher span. | • Quality of parts: Every part is a true expander (not just nearly-expanding), simplifying subsequent applications (spectral routines, flow, sparsification). | • Parallel cost profile: The algorithm is near-linear work and near-constant span (polylog hidden), which improves on earlier constructions with larger depth/round complexity for comparable error. | | ⸻ | | Minimal, actionable recipe (what to implement) | 1. ParallelUnitFlow(G, c, Δ, ∇, h): | • Capacities: c(e)\le \eta, sinks \nabla(v)\ge \gamma\deg(v), sources \|\Delta\|_1\le 2m, \Delta(v)\le \eta\deg(v). | • Push from level j to j-1 until edges saturate or no excess; relabel nodes whose outgoing to j-1 fully saturated; repeat top-down; stage-doubling trick on sinks to halve unsettled excess. | • Return f,l. | Costs: work \tilde{O}(m h \eta/\gamma), span \tilde{O}(h^2\eta/\gamma). | 2. Trimming(G, A, φ): | • Set c\equiv \Theta(\varphi^2), h=\Theta((1/\varphi)\log^2 n\log m). | • Iterate: run ParallelUnitFlow on G[A] with sources from external-dependency gap and sinks \propto \deg. | • If all excess absorbed → certify G[A] as \Theta(\varphi)-expander; return A. | • Else use high-label set sequence to pick S with small internal cut; set A\leftarrow A\setminus S and continue. | • Guarantees: expander, size-loss bound, boundary growth bound. | 3. Decompose all of V: | • Find a large nearly-expanding region A (or start with V), call Trimming, remove its A’, recurse. | • Bookkeeping ensures at most \tilde{O}(1/\varphi) charging per edge → total inter-part edges \tilde{O}(\varphi m). | 4. Congest port: | • Implement pushes, relabels, residual checks with O(\log n)-bit messages; pipeline across levels. | • Achieve \tilde{O}(1/\varphi^4) rounds end-to-end. | | ⸻ | | Final guarantees (precise statements) | • Parallel expander decomposition. With high probability, | \text{work}=\tilde{O}(m/\varphi^2),\quad \text{span}=\tilde{O}(1/\varphi^4),\quad | \sum_{i<j}|E(V_i,V_j)|=\tilde{O}(\varphi m), | and every G[V_i] is a \Theta(\varphi)-expander. | • Distributed Congest. With high probability, the same decomposition is computed in \tilde{O}(1/\varphi^4) rounds. | | That’s the full method: a degree-aware unit-flow to certify expansion, a trim-if-needed loop to convert nearly-expanders into true expanders with controlled boundary growth, and a recursive cover to decompose the whole graph—all with the desired parallel and distributed complexities and the \tilde{O}(\varphi) cut-edge budget.

1. Problem setup
   - Input: undirected graph G=(V,E), |E|=m, parameter φ∈(0,1).
   - Output: partition V=⋃V_i, each G[V_i] is Θ(φ)-expander.
   - Crossing edges: ∑_{i<j}|E(V_i,V_j)| = Õ(φ m).

2. Parallel guarantees
   - Work = Õ(m / φ²).
   - Span = Õ(1 / φ⁴).
   - Polylog(n) factors hidden in Õ(·).

3. Distributed guarantees
   - Congest model implementation.
   - Rounds = Õ(1 / φ⁴).

4. Unit-flow primitive
   - Parallel push–relabel style with bounded capacities.
   - Guarantees saturation and excess elimination by labels.
   - Cost: work Õ(m h η / γ), span Õ(h² η / γ).

5. Trimming routine
   - Defines φ-nearly expander.
   - Iteratively runs unit-flow with height h=Θ((1/φ) log²n log m).
   - Guarantees: Θ(φ)-expander, large size retained, boundary growth ≤2×.
   - Cost: work Õ(m / φ²), span Õ(1 / φ³).

6. Decomposition algorithm
   - Seed nearly-expanders, then trim to true expanders.
   - Recurse on remainder until all vertices clustered.
   - Accounting: each edge charged Õ(1/φ) times.

7. Parallel theorem
   - With high probability:
     • Work Õ(m / φ²).
     • Span Õ(1 / φ⁴).
     • Crossing edges Õ(φ m).
     • Each part a Θ(φ)-expander.

8. Congest theorem
   - With high probability: Õ(1 / φ⁴) rounds.
   - Same guarantees on cut size and expansion.

9. Improvements over prior work
   - Cut edges reduced to Õ(φ m).
   - True expanders obtained (not just nearly-expanders).
   - Parallel cost near-linear work and small span.

10. Minimal recipe
   - Step 1: ParallelUnitFlow with degree-aware sinks/sources.
   - Step 2: Trimming to certify or cut sparse fringe.
   - Step 3: Recursive decomposition with charging.
   - Step 4: Congest implementation via local pushes/relabels.

- Expander decompositions have become a central framework in graph algorithms, offering a way to partition graphs into well-connected subgraphs (expanders) with few crossing edges. | | - Formally, for an undirected graph \( G=(V,E) \), a near-optimal \( \phi \)-expander decomposition partitions the vertex set \( V \) into \( V_1, V_2, \ldots, V_k \), such that each subgraph \( G[V_i] \) is a \( \phi \)-expander. A \( \phi \)-expander ensures that for any subset \( S \subseteq V \), the number of edges leaving \( S \) is large relative to the volume of the smaller side of the cut, i.e., \( |E_G(S, V \setminus S)| \geq \phi \cdot \min\{\vol_G(S), \vol_G(V \setminus S)\} \). When \( \phi \approx 1 \), the graph is highly connected with low diameter and spectral properties similar to a complete graph, while every simple connected graph is at least a \( \phi = 1/|V|^2 \) expander. | | - Recent practical implementations have demonstrated that expander decompositions can be computed for medium-sized graphs. However, to handle very large graphs with hundreds of millions or billions of edges, there is a need for either significantly improved algorithms or the utilization of parallelization techniques.

- **Nearly Expander**: Given a graph \( G = (V, E) \) and a subset \( A \subseteq V \), the subgraph \( G[A] \) is considered a φ-nearly expander in \( G \) if for every subset \( S \subseteq A \) where \( \text{deg}_G(S) \leq \text{deg}_G(A \setminus S) \), the number of edges leaving \( S \) is at least \( \phi \cdot \text{deg}_G(S) \).

- **Flow-based Techniques**: Employe a new parallel algorithm , leveraging flow-based techniques to enhance both the quality and efficiency of expander decomposition. The sequential algorithm framework is extended, utilizing a flow-based approach for trimming subgraphs to ensure each subgraph is a \(\phi\)-expander. | - **Parallelization Design**:developed a Parallel Unit Flow algorithm is , accelerating the solution of flow problems through multiple rounds of push and relabel operations. | - **Trimming Algorithm Optimization**: Introduce an optimized trimming algorithm, capable of converting a nearly expander into an expander within logarithmic iterations. By dynamically adjusting sink capacities, the number of flow problems to be solved is reduced, leading to a significant improvement in the algorithm's convergence speed.