Upload ONNX model converted with ONNX Runtime GenAI

6d9af73 verified 4 months ago

1.39 kB

library_name: onnxruntime
tags:
  - trl
  - onnx
  - onnxruntime-genai
base_model:
  - trl-internal-testing/tiny-GptOssForCausalLM
pipeline_tag: text-generation

tiny-GptOssForCausalLM (ONNX)

This is an ONNX version of trl-internal-testing/tiny-GptOssForCausalLM. It was converted using the ONNX Runtime GenAI builder.

Conversion Details

Precision: fp16
Execution Provider: cpu
Base Model: trl-internal-testing/tiny-GptOssForCausalLM

Usage with ONNX Runtime GenAI


import onnxruntime_genai as og



# Load the model

model = og.Model("haydso/tiny-GptOssForCausalLM-ONNX")

tokenizer = og.Tokenizer(model)



# Generate text

prompt = "Your prompt here"

tokens = tokenizer.encode(prompt)

params = og.GeneratorParams(model)

params.set_search_options(max_length=200)

params.input_ids = tokens



generator = og.Generator(model, params)

while not generator.is_done():

    generator.compute_logits()

    generator.generate_next_token()



output_tokens = generator.get_sequence(0)

text = tokenizer.decode(output_tokens)

print(text)

Original Model Card

Tiny GptOssForCausalLM

This is a minimal model built for unit tests in the TRL library.