Share onnx conversion script
Can you share the script to convert to onnx, this is very useful for anyone finetuning this model as existing libraries do not support onnx conversion for this yet.
Should be supported by normal ONNX, its just a model in HF format.
We didn't run the conversion ourselves, but I can check.
Last I checked optimum does not support this yet (what I usually use for MoE), using standard ONNX converts it to a dense model and you lose the speed benefits.
Anyone have solution for this?
We didn't run the conversion ourselves, but I can check.
Hi @mihaimaruseac ,
I'm running openai/privacy-filter via the published ONNX artifacts and are planning to fine-tune on my data to reduce false positive. I'm stuck on re-exporting ONNX from a fine-tuned checkpoint, so if you have any guidance or tooling to share, i would really appreciate it. Thank you!
@rViper I made my own export script for this and made it public https://github.com/jalbrethsen-highflame/privacy-filter-onnx, you should be able to run this on your own finetunes. I also found a performance bug in their onnx, they implement the banded attention as a mask on global attention which gives the same logit parity but scales o(n^2) instead of the intended o(n). I also simplified the attention graph and made a few more optimizations for inference speed, its all in the README.
I reexported the base privacy-filter models jalbrethsen-highflame/privacy-filter-onnx.