red-ml/TankieV2a-8B
8B • Updated • 2
RedML
government
AI & ML interests
None defined yet.
Recent Activity
Fizzarolli
updated 2
models about 1 month ago
Fizzarolli
published a
model about 1 month ago
Fizzarolli
updated a
model about 1 month ago
Fizzarolli
published a
model about 1 month ago
Fizzarolli
updated a
bucket about 1 month ago
Fizzarolli
published a
model about 2 months ago
Fizzarolli
published a
bucket about 2 months ago
[bot] Conversion to Parquet
#1 opened about 2 months ago
by 
parquet-converter
Fizzarolli
updated 2
datasets about 2 months ago
Fizzarolli
published 2
datasets about 2 months ago
Fizzarolli
updated a
Space about 2 months ago
Fizzarolli
published a
Space about 2 months ago
Fizzarolli
posted an update about 2 years ago
Post
3071
hi everyone!
i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into HuggingFaceFW/fineweb-edu could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
check it out! Fizzarolli/phi3-4x4b-v1
i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into HuggingFaceFW/fineweb-edu could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
check it out! Fizzarolli/phi3-4x4b-v1
Fizzarolli
posted an update about 2 years ago
Post
3487
Is anyone looking into some sort of decentralized/federated dataset generation or classification by humans instead of synthetically?
From my experience with trying models, a *lot* of modern finetunes are trained on what amounts to, in essence, GPT-4 generated slop that makes everything sound like a rip-off GPT-4 (refer to i.e. the Dolphin finetunes). I have a feeling that this is a lot of the reason people haven't been quite as successful as Meta's instruct tunes of Llama 3.
From my experience with trying models, a *lot* of modern finetunes are trained on what amounts to, in essence, GPT-4 generated slop that makes everything sound like a rip-off GPT-4 (refer to i.e. the Dolphin finetunes). I have a feeling that this is a lot of the reason people haven't been quite as successful as Meta's instruct tunes of Llama 3.