Instructions to use HuggingFaceM4/VLM_WebSight_finetuned with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HuggingFaceM4/VLM_WebSight_finetuned with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="HuggingFaceM4/VLM_WebSight_finetuned", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("HuggingFaceM4/VLM_WebSight_finetuned", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use HuggingFaceM4/VLM_WebSight_finetuned with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HuggingFaceM4/VLM_WebSight_finetuned"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/VLM_WebSight_finetuned",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/HuggingFaceM4/VLM_WebSight_finetuned

SGLang

How to use HuggingFaceM4/VLM_WebSight_finetuned with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HuggingFaceM4/VLM_WebSight_finetuned" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/VLM_WebSight_finetuned",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HuggingFaceM4/VLM_WebSight_finetuned" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/VLM_WebSight_finetuned",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use HuggingFaceM4/VLM_WebSight_finetuned with Docker Model Runner:
```
docker model run hf.co/HuggingFaceM4/VLM_WebSight_finetuned
```

HugoLaurencon commited on Jan 12, 2024

Commit

2f09a11

verified ·

1 Parent(s): 4746d53

Update README.md

Browse files

Files changed (1) hide show

README.md +4 -4

README.md CHANGED Viewed

@@ -1,7 +1,7 @@
 ---
 license: mit
 datasets:
-- HuggingFaceM4/img2html
 language:
 - en
 tags:
@@ -15,7 +15,7 @@ tags:
 This model converts screenshots of website components into HTML/CSS codes.
-It is based on a very early checkpoint of our forthcoming vision-language foundation model, which has been fine-tuned using the [img2html](https://huggingface.co/datasets/HuggingFaceM4/img2html) dataset.
 This is very much an alpha version. The goal is to kick off an effort to develop improved models capable of converting a website screenshot into actual code.
@@ -96,10 +96,10 @@ print(generated_text)
 - **Parent Models:** [SigLIP](https://github.com/huggingface/transformers/pull/26522) and [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
 - **Resources for more information:**
     <!-- - [GitHub Repo](https://github.com/huggingface/m4/) -->
-    - img2html dataset: [Dataset card](https://huggingface.co/datasets/HuggingFaceM4/img2html)
 # License
 The model is built on top of two pre-trained models: [SigLIP](https://github.com/huggingface/transformers/pull/26522) and [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1). As such, users should comply with the licenses of these models.
-The two pre-trained models are connected to each other with newly initialized parameters that we train. These are not based on any of the two base frozen models forming the composite model. We release the additional weights we trained under an MIT license.

 ---
 license: mit
 datasets:
+- HuggingFaceM4/WebSight
 language:
 - en
 tags:
 This model converts screenshots of website components into HTML/CSS codes.
+It is based on a very early checkpoint of our forthcoming vision-language foundation model, which has been fine-tuned using the [Websight](https://huggingface.co/datasets/HuggingFaceM4/Websight) dataset.
 This is very much an alpha version. The goal is to kick off an effort to develop improved models capable of converting a website screenshot into actual code.
 - **Parent Models:** [SigLIP](https://github.com/huggingface/transformers/pull/26522) and [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
 - **Resources for more information:**
     <!-- - [GitHub Repo](https://github.com/huggingface/m4/) -->
+    - Websight dataset: [Dataset card](https://huggingface.co/datasets/HuggingFaceM4/Websight)
 # License
 The model is built on top of two pre-trained models: [SigLIP](https://github.com/huggingface/transformers/pull/26522) and [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1). As such, users should comply with the licenses of these models.
+The two pre-trained models are connected to each other with newly initialized parameters that we train. These are not based on any of the two base frozen models forming the composite model. We release the additional weights we trained under an MIT license.