Instructions to use fdtn-ai/Foundation-Sec-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use fdtn-ai/Foundation-Sec-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="fdtn-ai/Foundation-Sec-8B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("fdtn-ai/Foundation-Sec-8B")
model = AutoModelForCausalLM.from_pretrained("fdtn-ai/Foundation-Sec-8B")

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use fdtn-ai/Foundation-Sec-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "fdtn-ai/Foundation-Sec-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fdtn-ai/Foundation-Sec-8B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/fdtn-ai/Foundation-Sec-8B

SGLang

How to use fdtn-ai/Foundation-Sec-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "fdtn-ai/Foundation-Sec-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fdtn-ai/Foundation-Sec-8B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "fdtn-ai/Foundation-Sec-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fdtn-ai/Foundation-Sec-8B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use fdtn-ai/Foundation-Sec-8B with Docker Model Runner:
```
docker model run hf.co/fdtn-ai/Foundation-Sec-8B
```

Good Contribution to the Community

by Kapenge - opened May 8, 2025

Discussion

Kapenge

May 8, 2025

Some results, below:

MPS device found. Using GPU (Metal Performance Shaders).
Loading model and tokenizer...
Loading checkpoint shards: 100%|██████████████████████████████████████████████| 4/4 [00:22<00:00, 5.71s/it]
Model and tokenizer loaded and moved to mps in 68.12 seconds.

--- Running Inference on mps ---

Q (Combined): Please answer the following two questions:

Question 1: What is the primary purpose of a SIEM system?

Question 2: An employee reports receiving an email with a suspicious attachment. What are the immediate steps an analyst should take?

Provide your answers clearly, addressing each question.
A (Combined): ## Answers:

Question 1: The primary purpose of a SIEM system is to collect, analyze, and correlate security event data from various sources within an organization's IT infrastructure. It helps in identifying potential security threats, monitoring system activity, and providing real-time alerts to security teams.

Question 2: The immediate steps an analyst should take upon receiving a report of a suspicious email with an attachment include:

Isolating the affected system from the network to prevent potential malware spread.
Analyzing the email and attachment using sandboxing tools to determine if they are malicious.
Searching for any related indicators of compromise (IOCs) in the SIEM system to identify if similar threats have been detected elsewhere.
Reporting the incident to the security team for further investigation and response.

Generated Tokens: 167
Tokenization Duration: 0.0062s
Generation Duration: 61.0173s (on mps)
Decoding Duration: 0.0045s
Total Q&A Duration: 61.04s

Maybe for future consideration, considering Google Gemma3 models with QAT could help in terms of performance.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment