From Errors to Proofs: Minimal-Core-Guided Repair for Neuro-Symbolic Constraint Solving
Paper • 2608.14771 • Published
AI & ML interests
None defined yet.
Recent Activity
View all activity
dipankarsarkar
authored 3
papers 18 days ago
dipankarsarkar
submitted a
paper to Daily Papers 18 days ago
dipankarsarkar
posted an update 18 days ago
Post
375
Privacy is moving ad decisions onto the device. The auction runs locally, next to the context it scores.
The budget still lives on a server.
So every device bids against the balance it saw at its last sync. Between syncs, campaigns spend money they no longer have.
I simulated how much that costs: 50 devices, 36 campaigns, about 300,000 auctions per run, 30 seeds per cell, budgets frozen before evaluation. Even pacing, second-score payment:
- Sync every tick: spend lands 18% over budget (32% with iPinYou-calibrated values).
- Sync every 10 ticks: 5.8 times the budget.
- Sync every 50 ticks: 17.7 times the budget (10.1 times with iPinYou-calibrated values).
Zero-lag sync stays within about 1%. The lag alone does the damage.
The incentive side runs the other way. With second-score payment and zero lag, 98% of auctions are manipulable (92% with iPinYou values). Switch the payment rule to critical-bid and that drops to 0 at every sync interval, in both setups.
Sync fixes the budget. The payment rule, not the sync interval, is what removes manipulability.
All 12 result sets are on the Hub, plus the 50,808 context labels every run samples from.
Paper: When Privacy Moves ML-Mediated Decisions On Device: Information and Incentive Misalignment in Auctions (2609.33312)
Dataset: skelfresearch/on-device-auction-audit
Code: https://github.com/sarkar-dipankar/on-device-auction-audit
If your ads stack moves on device, how often does the device learn its budget?
The budget still lives on a server.
So every device bids against the balance it saw at its last sync. Between syncs, campaigns spend money they no longer have.
I simulated how much that costs: 50 devices, 36 campaigns, about 300,000 auctions per run, 30 seeds per cell, budgets frozen before evaluation. Even pacing, second-score payment:
- Sync every tick: spend lands 18% over budget (32% with iPinYou-calibrated values).
- Sync every 10 ticks: 5.8 times the budget.
- Sync every 50 ticks: 17.7 times the budget (10.1 times with iPinYou-calibrated values).
Zero-lag sync stays within about 1%. The lag alone does the damage.
The incentive side runs the other way. With second-score payment and zero lag, 98% of auctions are manipulable (92% with iPinYou values). Switch the payment rule to critical-bid and that drops to 0 at every sync interval, in both setups.
Sync fixes the budget. The payment rule, not the sync interval, is what removes manipulability.
All 12 result sets are on the Hub, plus the 50,808 context labels every run samples from.
Paper: When Privacy Moves ML-Mediated Decisions On Device: Information and Incentive Misalignment in Auctions (2609.33312)
Dataset: skelfresearch/on-device-auction-audit
Code: https://github.com/sarkar-dipankar/on-device-auction-audit
If your ads stack moves on device, how often does the device learn its budget?
dipankarsarkar
authored 2
papers 20 days ago
dipankarsarkar
submitted 2
papers to Daily Papers 21 days ago
dipankarsarkar
posted an update 21 days ago
Post
3,178
I audited one of my own evaluations. The ranking did not hold up the way I expected.
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Paper: How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure (2609.30074)
Dataset: dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit
How many of the leaderboard rankings you rely on would survive re-running the same calls?
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Paper: How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure (2609.30074)
Dataset: dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit
How many of the leaderboard rankings you rely on would survive re-running the same calls?
- 2 replies
dipankarsarkar
authored a
paper 21 days ago
dipankarsarkar
posted an update 4 months ago
Post
888
Your issue tracker is in the wrong place.
It lives on a server. Your code lives in git. So every time an agent picks up work it makes an API call, burns a token, fights a rate limit, and still cannot see what the other agent just did.
Move the issues into the repo. Append-only event log in git refs. Branches when you branch, merges when you merge, CRDT so two agents never conflict. No server, no database.
The coordination signal that PR-level telemetry misses lives before the pull request. The paper, and a live demo running the real tool:
Before the Pull Request: Mining Multi-Agent Coordination (2606.19616)
https://huggingface.co/spaces/neullabs/grite
If your agents share a repo, where does their shared state actually live right now?
It lives on a server. Your code lives in git. So every time an agent picks up work it makes an API call, burns a token, fights a rate limit, and still cannot see what the other agent just did.
Move the issues into the repo. Append-only event log in git refs. Branches when you branch, merges when you merge, CRDT so two agents never conflict. No server, no database.
The coordination signal that PR-level telemetry misses lives before the pull request. The paper, and a live demo running the real tool:
Before the Pull Request: Mining Multi-Agent Coordination (2606.19616)
https://huggingface.co/spaces/neullabs/grite
If your agents share a repo, where does their shared state actually live right now?
- 14 replies
dipankarsarkar
authored 8
papers 4 months ago
The Correctness Illusion in LLM-Generated GPU Kernels
Paper • 2606.20128 • Published • 1
Before the Pull Request: Mining Multi-Agent Coordination
Paper • 2606.19616 • Published
Epistral Network: Revolutionizing Media Curation and Consumption through Decentralization
Paper • 2402.04881 • Published
Navigating the Knowledge Sea: Planet-scale answer retrieval using LLMs
Paper • 2402.05318 • Published
FairFlow Protocol: Equitable Maximal Extractable Value (MEV) mitigation in Ethereum
Paper • 2312.12654 • Published
Decentralized Deepfake Detection Blockchain Network using Dynamic Algorithm management
Paper • 2311.18545 • Published
Centralized Intermediation in a Decentralized Web3 Economy: Value Accrual and Extraction
Paper • 2311.08234 • Published
Towards Universal Atomic Composability: A Formal Model for Multi-Rollup Environments on Ethereum
Paper • 2311.00422 • Published