U4RASD

/

NeoAraBERT

Like 10

Follow

Digital Research Unit - Arab Center For Research and Policy Studies 33

Fix compatibility with Transformers 5 and make xFormers an optional dependency.

Includes changes to model.py, rotary.py, README.md, and a new swiglu.py module.

Thank you for the PR, this is really cool! I'll take a closer look soon.

Before we can merge, could you check a few things?

• Make sure it works across Transformers 5.x, not just 5.20, and let us know the minimum version it supports.
• Benchmark the speed before and after your changes.
• Run the Muradif benchmark before and after your changes. Both the embeddings and the final scores should be identical.

Thanks for quick response, I don't have access to linux GPU right now, so I won't be able to use "original" version of your model (xformers is not published for macOS). But I can benchmark the model using monkey-patched version of SwiGLU, like I did in PR.