haydso/tiny-GptOssForCausalLM-ONNX

tiny-GptOssForCausalLM (ONNX)

This is an ONNX version of trl-internal-testing/tiny-GptOssForCausalLM. It was converted using the ONNX Runtime GenAI builder.

Conversion Details

• Precision: fp16

• Execution Provider: cpu

• Base Model: trl-internal-testing/tiny-GptOssForCausalLM

Usage with ONNX Runtime GenAI

import onnxruntime_genai as og



# Load the model

model = og.Model("haydso/tiny-GptOssForCausalLM-ONNX")

tokenizer = og.Tokenizer(model)



# Generate text

prompt = "Your prompt here"

tokens = tokenizer.encode(prompt)

params = og.GeneratorParams(model)

params.set_search_options(max_length=200)

params.input_ids = tokens



generator = og.Generator(model, params)

while not generator.is_done():

    generator.compute_logits()

    generator.generate_next_token()



output_tokens = generator.get_sequence(0)

text = tokenizer.decode(output_tokens)

print(text)

---

Original Model Card

Tiny GptOssForCausalLM

This is a minimal model built for unit tests in the TRL library.