Instructions to use InternScience/Agents-A1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use InternScience/Agents-A1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="InternScience/Agents-A1")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("InternScience/Agents-A1")
model = AutoModelForMultimodalLM.from_pretrained("InternScience/Agents-A1")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use InternScience/Agents-A1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "InternScience/Agents-A1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InternScience/Agents-A1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/InternScience/Agents-A1

SGLang

How to use InternScience/Agents-A1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "InternScience/Agents-A1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InternScience/Agents-A1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "InternScience/Agents-A1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InternScience/Agents-A1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use InternScience/Agents-A1 with Docker Model Runner:
```
docker model run hf.co/InternScience/Agents-A1
```

Software Development Benchmarks please! and Congrats! Too good to be true?

#23

by darkmatter2222 - opened about 13 hours ago

Discussion

darkmatter2222

about 13 hours ago

•

edited about 13 hours ago

The existing benchmarks suggest this is likely a great Software Development model. Can we get those benchmarks please?

Software Engineering Benchmark Reference

Tier 1 — Most Important (Real-World Software Engineering)

These are generally considered the strongest indicators of how well a model performs as a software engineer.

Benchmark	What It Measures	Typical Tasks
SWE-Bench Verified	Fixing real GitHub issues	Modify existing repositories to resolve actual bugs
SWE-Bench Pro	Harder version of SWE-Bench	Multi-file, complex repository modifications
LiveCodeBench	General coding ability using newly released problems	Prevents benchmark contamination
Aider Polyglot	Real git commits	Edit repositories using realistic developer workflows
Terminal-Bench	Command-line software engineering	Shell commands, builds, debugging, testing

Tier 2 — Strong Engineering Benchmarks

Benchmark	What It Measures	Typical Tasks
BigCodeBench	Large-scale programming tasks	Multi-function implementations
RepoBench	Repository understanding	Navigate and modify existing projects
CRUXEval	Code reasoning	Understand code execution and behavior
DS-1000	Data science programming	NumPy, Pandas, ML workflows
EvalPlus	Improved HumanEval/MBPP evaluation	More comprehensive correctness testing

Tier 3 — Traditional Coding Benchmarks

These are still widely reported but are generally easier for modern frontier models.

Benchmark	What It Measures	Typical Tasks
HumanEval	Python function generation	Complete small functions
HumanEval+	Harder HumanEval	More robust unit tests
MBPP	Basic Python programming	Introductory programming tasks
MultiPL-E	Multilingual coding	HumanEval translated into many languages
APPS	Competitive programming	Interview and contest-style problems
CodeContests	Programming competitions	Google Code Jam style challenges

Agentic Software Engineering Benchmarks

These evaluate autonomous agents rather than code generation alone.

Benchmark	Focus
GAIA	Long-horizon autonomous problem solving
AgentBench	Tool use and multi-step reasoning
BrowseComp	Web research and information gathering
TauBench (τ-Bench)	Long-running business and API workflows
IFBench	Instruction following
SciCode	Scientific software engineering
Seal-0	Agent planning and execution

Specialized Benchmarks

Benchmark	Domain
MathArena-Code	Mathematical programming
CodeArena	Head-to-head coding evaluation
OpenHands Evaluation	Autonomous coding agents
Mini-SWE-Agent	Lightweight SWE agent evaluation
MLE-Bench	Machine learning engineering
BrowserArena	Browser automation agents

Benchmark Categories

Category	Best Benchmarks
Repository bug fixing	SWE-Bench Verified, SWE-Bench Pro
Autonomous software engineering	SWE-Bench Pro, OpenHands Evaluation
General coding	LiveCodeBench, BigCodeBench
Python function generation	HumanEval, HumanEval+
Repository navigation	RepoBench
Git-based engineering	Aider Polyglot
Terminal workflows	Terminal-Bench
Tool use	AgentBench
Long-horizon agents	GAIA
Web research	BrowseComp
Data science	DS-1000
Competitive programming	APPS, CodeContests
Code reasoning	CRUXEval

Current Industry Consensus

The benchmarks below are the ones most frequently cited in papers and model releases for evaluating software engineering capability.

Importance	Benchmark
Very High	SWE-Bench Verified
Very High	SWE-Bench Pro
Very High	LiveCodeBench
High	Aider Polyglot
High	Terminal-Bench
High	BigCodeBench
High	RepoBench
High	CRUXEval
Medium	HumanEval+
Medium	DS-1000
Medium	HumanEval
Medium	APPS
Medium	MBPP

Recommended Benchmark Suite

If you are comparing coding models (GPT, Claude, Qwen, Gemma, DeepSeek, Kimi, Agents-A1, Tess, etc.), this suite provides broad coverage across the software engineering lifecycle.

Benchmark	Why Include It
SWE-Bench Verified	Repository bug fixing
SWE-Bench Pro	Advanced software engineering
LiveCodeBench	General coding ability with contamination resistance
BigCodeBench	Large programming tasks
Aider Polyglot	Real-world git workflows
Terminal-Bench	CLI engineering skills
RepoBench	Repository comprehension
CRUXEval	Code reasoning
HumanEval+	Function generation
DS-1000	Data science engineering

What Each Benchmark Tests

Capability	Primary Benchmark
Bug fixing	SWE-Bench Verified
Multi-file refactoring	SWE-Bench Pro
Repository understanding	RepoBench
Code generation	HumanEval+
Real-world coding	LiveCodeBench
Long-context coding	BigCodeBench
Git workflows	Aider Polyglot
Shell/CLI usage	Terminal-Bench
Tool calling	AgentBench
Autonomous agents	GAIA
Research & browsing	BrowseComp
Data science	DS-1000
Competitive programming	APPS
Code reasoning	CRUXEval
Scientific programming	SciCode

Coverage Matrix

Benchmark	Functions	Multi-file	Repository	Terminal	Tool Use	Agentic	Web	Long Horizon
HumanEval	✓
HumanEval+	✓
MBPP	✓
LiveCodeBench	✓	✓
BigCodeBench	✓	✓	✓
RepoBench		✓	✓
SWE-Bench Verified		✓	✓	✓	✓	✓
SWE-Bench Pro		✓	✓	✓	✓	✓
Aider Polyglot		✓	✓	✓		✓
Terminal-Bench				✓	✓	✓
AgentBench				✓	✓	✓
GAIA				✓	✓	✓	✓	✓
BrowseComp					✓	✓	✓	✓
DS-1000	✓	✓
CRUXEval	✓
SciCode	✓	✓				✓

darkmatter2222

about 12 hours ago

@Sentdex Whats your vibe on this model? Based on your latest vid, this might be one you want to benchmark.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment