The Embedder's Dilemma: LLMs Are Better, but at What Cost?
Paper • 2608.12875 • Published • 10
AI & ML interests
A one-year long research workshop on large language models: the Summer of Language Models 21 🌸
Recent Activity
View all activity
Muennighoff
submitted a
paper to Daily Papers 1 day ago
Security test - please ignore
#294 opened 5 days ago
by 
hackeronetestt
BramVanroy
posted an update 13 days ago
Post
2630
**I benchmarked HF buckets against https access for Common Crawl.**
Took me a while to get round to do this but I benchmarked access to Common Crawl via https vs hf buckets. Both experiments were run at night in Europe. I do not think other hardware problems were impacting the speeds since CPU processing time of the non-download pipeline components were highly similar (within 2% identical) and below only the WarcReader speeds of datatrove are used.
Experiment: selected 5 disjoint samples of 64 files each (randomly from the latest crawl; 20,499 docs/file). Those five batches were then processed by 32 single-core tasks with 4GB/core (five batches to calculate CIs). Paired experiment between using https and hf bucket.
- https: 40.0 [39.3-40.6] (seconds per WARC file)
- hf bucket: 172.0 [122.7-221.2]
That is a difference of about 4x in streaming speed. You'll see that https is also more stable (smaller CI).
I also ran raw throughput tests to the endpoints to measure rate limiting (64MiB transfer at 8/32/128/256 concurrent readers) and rate limiting seems not an issue for either: at any of those parallel reader numbers, their respective speeds stay about the same.
Note that, given CC scale, this is still a small test. Rate limiting may become more obvious when processing a full crawl. I do not know whether the https endpoint vs HF bucket will shut you out earlier with which limits.
Took me a while to get round to do this but I benchmarked access to Common Crawl via https vs hf buckets. Both experiments were run at night in Europe. I do not think other hardware problems were impacting the speeds since CPU processing time of the non-download pipeline components were highly similar (within 2% identical) and below only the WarcReader speeds of datatrove are used.
Experiment: selected 5 disjoint samples of 64 files each (randomly from the latest crawl; 20,499 docs/file). Those five batches were then processed by 32 single-core tasks with 4GB/core (five batches to calculate CIs). Paired experiment between using https and hf bucket.
- https: 40.0 [39.3-40.6] (seconds per WARC file)
- hf bucket: 172.0 [122.7-221.2]
That is a difference of about 4x in streaming speed. You'll see that https is also more stable (smaller CI).
I also ran raw throughput tests to the endpoints to measure rate limiting (64MiB transfer at 8/32/128/256 concurrent readers) and rate limiting seems not an issue for either: at any of those parallel reader numbers, their respective speeds stay about the same.
Note that, given CC scale, this is still a small test. Rate limiting may become more obvious when processing a full crawl. I do not know whether the https endpoint vs HF bucket will shut you out earlier with which limits.
Fixed CO2 metadata in README.md
1
#293 opened 26 days ago
by 
Croc-Prog-HF
why mt0-large is 1.3B while mt5-large is 780M?
1
#6 opened almost 2 years ago
by 
tansq
Geração de Texto
#63 opened 9 months ago
by 
alcidesmoreira1963
Adding Evaluation Results
#61 opened almost 3 years ago
by 
leaderboard-pr-bot
Hosted inference API: 500 Internal Server Error returned
1
#4 opened almost 4 years ago
by 
MarkDeSouza
Adding Evaluation Results
#41 opened almost 3 years ago
by leaderboard-pr-bot
Adding Evaluation Results
#42 opened about 2 years ago
by leaderboard-pr-bot
Add evaluation results on the mathemakitten--winobias_antistereotype_test config and test split of mathemakitten/winobias_antistereotype_test
1
#32 opened over 3 years ago
by 
autoevaluator
System Requirements
3
#38 opened over 3 years ago
by 
Robo0890
Request: DOI
3
#43 opened almost 2 years ago
by 
ovv4thewin
w11wo
authored 2
papers 3 months ago
espejelomar
posted an update 3 months ago
Post
4753
Sharing WorldForge with @abdelstark
It's an open-source Python project for evaluating and replaying robotics and world-model workflows.
The useful part is not only calling a model. WorldForge records the run, validates action shapes, translates outputs into actions, and keeps replay artifacts you can inspect later.
The current demo uses LeRobot + LeWorldModel on PushT through the official loader:
The harness also has replay-only paths for Cosmos-Policy and GR00T-style outputs, so you can inspect the provider contract from saved artifacts without keeping a GPU server online.
Try it:
Repo: https://github.com/AbdelStark/worldforge
Docs: https://abdelstark.github.io/worldforge/
Pre-1.0, MIT, and actively looking for contributors. Good areas:
- robotics provider adapters
- replay artifacts
- eval flows
- docs & first-run demos
Good first issues: https://github.com/AbdelStark/worldforge/contribute
If you're building robot policy evals or model adapters, would love a PR — or an issue describing what's missing.
It's an open-source Python project for evaluating and replaying robotics and world-model workflows.
The useful part is not only calling a model. WorldForge records the run, validates action shapes, translates outputs into actions, and keeps replay artifacts you can inspect later.
The current demo uses LeRobot + LeWorldModel on PushT through the official loader:
stable_worldmodel.policy.AutoCostModel("pusht/lewm")The harness also has replay-only paths for Cosmos-Policy and GR00T-style outputs, so you can inspect the provider contract from saved artifacts without keeping a GPU server online.
Try it:
pip install worldforge-aiuv run --extra harness worldforge-harness --flow robotics-compareRepo: https://github.com/AbdelStark/worldforge
Docs: https://abdelstark.github.io/worldforge/
Pre-1.0, MIT, and actively looking for contributors. Good areas:
- robotics provider adapters
- replay artifacts
- eval flows
- docs & first-run demos
Good first issues: https://github.com/AbdelStark/worldforge/contribute
If you're building robot policy evals or model adapters, would love a PR — or an issue describing what's missing.
w11wo
authored a
paper 3 months ago
pbloem
submitted a
paper to Daily Papers 4 months ago
odegiber
authored 2
papers 5 months ago