Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
AbstractPhil 
posted an update about 9 hours ago
Post
46
Say hello to the AlephLLM: Mini-Beatrix - in her huggingface space!
She is currently stepped at 8000 steps in the first couple datasets, so she's not very smart yet.
AbstractPhil/alephllm-chat

Be warned, whatever you say WILL be recorded in a public cache, WHEN the chat version works. For now she records nothing. The idea is to help debug the K/V cache, and I would rather the data accumulated be shared. If you wish to speak to her in private I will include a toggle, that way you'll see that nothing is recorded when you speak and you can still have a private chat with her. For now she's simply auto-completing, so have fun with her.

The AlephLLM prototype is currently in full training with SDPA attention.
AbstractPhil/alephllm-mini-beatrix-training

https://github.com/AbstractEyes/alephllm Here's the model code and training code for the prototype.
As the training progresses, the AlephLLM will become more coherent and communicative, the tensorboard will consist of a large series of useful and useless analysis, and each checkpoint recorded at around 2000 steps unless the train crashes or the system faults.

It will take about 9 hours for the first few datasets to converge, then I'll train a chat AMOE expert cluster to see if she wants to speak yet. Until then, she's learning.

Yes I know it's early, but there isn't much more I could think of to analyze the AlephLM directly currently. The only way train the AlephLLM, is to train the full AlephLLM prototype. The bigger training has to run, otherwise the analysis won't matter. As it progresses, the analysis and huge amount of tensorboard statistics will flood out. Everything is transparent through the process from start to finish, everything recorded.
AbstractPhil
•
edited about 4 hours ago

Alright the prelim went well. We're at about 3.4 billion tokens learned.

Model very stable as the prelim tests showed. The structure is not collapsing and the model is in fact learning useful pathways of information. So far it hasn't established legitimate ground-form informational segmentation pathology, the model requires many more tokens for this.

Model learns at around 100k tokens per second scaled to the RTX 6000 PRO card available on colab, so the bf16 training is scaling nicely to the benefits from the blackwell hardware.

Each is packed with an fp8 variation for inference as well, which is substantially faster for inference, but they aren't very smart yet. They ARE available. The results show the correctly aligned fp8 variants are roughly the same accuracy on inference, but training they essentially collapse the attention in less than a thousand steps.

There is a full pretrain lineup ready. We're looking at maybe 80 billion tokens or so, and with that the chat AMOE cluster will be ran after each major finetune. We'll be drawing the conversation out of the model each major train, and with each train we'll determine if the model even needs to be finetuned with chat to allow the model to behave.

AMOE arm training planning begins already, as each layer can have direct integrated AMOE arms into the core for couple/decouple purposes. The structure itself will be learning chat through finetuning arms with chat directly onto the trunk. Based on the responses, we will continue pretraining the trunk until the construction of the AMOE chat conforms to the behavior.

So far it's showing the model likely does not require the chat to be baked into the core in finetune state. Prelim shows the chat is likely corrupting the core of the data compendium by learning it atop the core system. The AMOE could very well handle the first-class behavior, but there's no guarantees for this just yet.

wikitext 103 and fineweb main + extended are the preliminary.

These two are targeting;

  [  done  ] warmup_wikitext    wikitext-103   0.300/0.30B
  [  done  ] fineweb_main       fineweb-edu    3.000/3.00B
  [planned ] fineweb_extended   fineweb-edu    0.000/12.00B

So roughly 30 hours until fineweb's larger structure is trained. After this our first chat arms will be trained, which ought to let us have a conversation with her. See how well she took to the information.
In this post