Rechtspraak
🌖
Comparison of Rechtspraak data processed by GPT-NL
Edwin Rijgersberg
Rijgersberg
AI & ML interests
None yet
Recent Activity
updated a Space about 15 hours ago
AIStudioDelta/Rechtspraak updated a dataset about 16 hours ago
Rijgersberg/YouTube-Commons-nl-audio published a Space 3 days ago
AIStudioDelta/RechtspraakOrganizations
updated a
Space about 15 hours ago
updated a
dataset about 16 hours ago
published a
Space 3 days ago
liked a
dataset 3 days ago
liked a
model 3 days ago
liked a
model 15 days ago
License?
3
#2 opened about 1 year ago
by Rijgersberg
liked a
dataset 20 days ago
liked a
model 20 days ago
upvoted an article 27 days ago
Article
Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers
tomaarsen
• • 144 liked a
model 28 days ago
liked a
model 29 days ago
liked a
model about 1 month ago
reacted to BramVanroy's post with 👍 about 1 month ago
Post
2762
**I benchmarked HF buckets against https access for Common Crawl.**
Took me a while to get round to do this but I benchmarked access to Common Crawl via https vs hf buckets. Both experiments were run at night in Europe. I do not think other hardware problems were impacting the speeds since CPU processing time of the non-download pipeline components were highly similar (within 2% identical) and below only the WarcReader speeds of datatrove are used.
Experiment: selected 5 disjoint samples of 64 files each (randomly from the latest crawl; 20,499 docs/file). Those five batches were then processed by 32 single-core tasks with 4GB/core (five batches to calculate CIs). Paired experiment between using https and hf bucket.
- https: 40.0 [39.3-40.6] (seconds per WARC file)
- hf bucket: 172.0 [122.7-221.2]
That is a difference of about 4x in streaming speed. You'll see that https is also more stable (smaller CI).
I also ran raw throughput tests to the endpoints to measure rate limiting (64MiB transfer at 8/32/128/256 concurrent readers) and rate limiting seems not an issue for either: at any of those parallel reader numbers, their respective speeds stay about the same.
Note that, given CC scale, this is still a small test. Rate limiting may become more obvious when processing a full crawl. I do not know whether the https endpoint vs HF bucket will shut you out earlier with which limits.
Took me a while to get round to do this but I benchmarked access to Common Crawl via https vs hf buckets. Both experiments were run at night in Europe. I do not think other hardware problems were impacting the speeds since CPU processing time of the non-download pipeline components were highly similar (within 2% identical) and below only the WarcReader speeds of datatrove are used.
Experiment: selected 5 disjoint samples of 64 files each (randomly from the latest crawl; 20,499 docs/file). Those five batches were then processed by 32 single-core tasks with 4GB/core (five batches to calculate CIs). Paired experiment between using https and hf bucket.
- https: 40.0 [39.3-40.6] (seconds per WARC file)
- hf bucket: 172.0 [122.7-221.2]
That is a difference of about 4x in streaming speed. You'll see that https is also more stable (smaller CI).
I also ran raw throughput tests to the endpoints to measure rate limiting (64MiB transfer at 8/32/128/256 concurrent readers) and rate limiting seems not an issue for either: at any of those parallel reader numbers, their respective speeds stay about the same.
Note that, given CC scale, this is still a small test. Rate limiting may become more obvious when processing a full crawl. I do not know whether the https endpoint vs HF bucket will shut you out earlier with which limits.
liked a
model about 2 months ago
upvoted a paper about 2 months ago
liked a
dataset about 2 months ago
liked 2
models 2 months ago
liked a
dataset 2 months ago