metadata
base_model: AquilaX-AI/ai_scanner
tags:
- text-generation-inference
- transformers
- unsloth
- qwen2
- gguf
license: apache-2.0
language:
- en
Uploaded model
- Developed by: AquilaX-AI
- License: apache-2.0
- Finetuned from model : AquilaX-AI/ai_scanner
This qwen2 model was trained 2x faster with Unsloth and Huggingface's TRL library.
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
import torch
import json
model_id = "AquilaX-AI/AI-Scanner-Quantized"
filename = "unsloth.Q8_0.gguf"
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model.to(device)
sys_prompt = """<|im_start|>system\nYou are Securitron, an AI assistant specialized in detecting vulnerabilities in source code. Analyze the provided code and provide a structured report on any security issues found.<|im_end|>"""
user_prompt = """
CODE FOR SCANNING
"""
prompt = f"""{sys_prompt}
<|im_start|>user
{user_prompt}<|im_end|>
<|im_start|>assistant
"""
encodeds = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.to(device)
text_streamer = TextStreamer(tokenizer, skip_prompt=True)
response = model.generate(
input_ids=encodeds,
streamer=text_streamer,
max_new_tokens=4096,
use_cache=True,
pad_token_id=151645,
eos_token_id=151645,
num_return_sequences=1
)
output = json.loads(tokenizer.decode(response[0]).split('<|im_start|>assistant')[-1].split('<|im_end|>')[0].strip())