Update README.md
Browse files
README.md
CHANGED
|
@@ -1,6 +1,9 @@
|
|
| 1 |
---
|
| 2 |
library_name: transformers
|
| 3 |
-
tags:
|
|
|
|
|
|
|
|
|
|
| 4 |
base_model: Qwen/Qwen2.5-VL-3B-Instruct
|
| 5 |
license: cc-by-nc-4.0
|
| 6 |
---
|
|
@@ -28,19 +31,21 @@ license: cc-by-nc-4.0
|
|
| 28 |
- **Language(s) (NLP):** English
|
| 29 |
- **License:** CC-BY-NC-4.0
|
| 30 |
- **Finetuned from model:** Qwen/Qwen2.5-VL-3B-Instruct
|
| 31 |
-
-
|
| 32 |
|
| 33 |
### Model Sources
|
| 34 |
|
| 35 |
<!-- Provide the basic links for the model. -->
|
| 36 |
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
|
|
|
|
|
|
| 44 |
|
| 45 |
## Uses
|
| 46 |
|
|
@@ -56,8 +61,9 @@ This includes a CLI tool for running the model, as well as a streamlit app.
|
|
| 56 |
<img src="https://raw.githubusercontent.com/convergence-ai/proxy-lite/refs/heads/main/assets/demo.gif" alt="Proxy Lite Demo" />
|
| 57 |
</div>
|
| 58 |
|
|
|
|
| 59 |
|
| 60 |
-
### Direct Use
|
| 61 |
|
| 62 |
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
|
| 63 |
|
|
@@ -75,6 +81,8 @@ The tool arguments are **very important** for parsing the tool calls from the mo
|
|
| 75 |
|
| 76 |
> **Important:** Qwen-2.5-VL Support in `transformers` is not yet available in the latest release so be sure to install from source.
|
| 77 |
|
|
|
|
|
|
|
| 78 |
When it comes to using and prompting Proxy Lite, please refer to the [repository](https://github.com/convergence-ai/proxy-lite) for more information, but the model expects a message history of the form:
|
| 79 |
|
| 80 |
```python
|
|
@@ -101,6 +109,8 @@ This would then build up the message history, alternating between the assistant
|
|
| 101 |
|
| 102 |
> **Context-Window Management:** When making calls to the model, all the observations other than the current one are discarded in order to reduce the large number of image tokens required. Since the model responses include reflection on the observations and are all included in the message history, the model is still aware of the entire history when planning new actions.
|
| 103 |
|
|
|
|
|
|
|
| 104 |
You should also pass the `Tools` that the model has access to, these will define the action space available to the model. You can do this with `transformers`:
|
| 105 |
|
| 106 |
```python
|
|
|
|
| 1 |
---
|
| 2 |
library_name: transformers
|
| 3 |
+
tags:
|
| 4 |
+
- agent
|
| 5 |
+
- action
|
| 6 |
+
- vlm
|
| 7 |
base_model: Qwen/Qwen2.5-VL-3B-Instruct
|
| 8 |
license: cc-by-nc-4.0
|
| 9 |
---
|
|
|
|
| 31 |
- **Language(s) (NLP):** English
|
| 32 |
- **License:** CC-BY-NC-4.0
|
| 33 |
- **Finetuned from model:** Qwen/Qwen2.5-VL-3B-Instruct
|
| 34 |
+
- [Running the agent](https://github.com/convergence-ai/proxy-lite)
|
| 35 |
|
| 36 |
### Model Sources
|
| 37 |
|
| 38 |
<!-- Provide the basic links for the model. -->
|
| 39 |
|
| 40 |
+
**Repository:** [convergence-ai/proxy-lite](https://github.com/convergence-ai/proxy-lite) to run Proxy-lite on a browser:
|
| 41 |
+
|
| 42 |
+
```
|
| 43 |
+
git clone https://github.com/convergence-ai/proxy-lite.git
|
| 44 |
+
make proxy
|
| 45 |
+
proxy "Find some markets near Kings Cross and tell me their ratings."
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
([endpoint](https://huggingface.co/spaces/convergence-ai/demo-api) for small-scale testing)
|
| 49 |
|
| 50 |
## Uses
|
| 51 |
|
|
|
|
| 61 |
<img src="https://raw.githubusercontent.com/convergence-ai/proxy-lite/refs/heads/main/assets/demo.gif" alt="Proxy Lite Demo" />
|
| 62 |
</div>
|
| 63 |
|
| 64 |
+
---
|
| 65 |
|
| 66 |
+
#### Direct Use
|
| 67 |
|
| 68 |
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
|
| 69 |
|
|
|
|
| 81 |
|
| 82 |
> **Important:** Qwen-2.5-VL Support in `transformers` is not yet available in the latest release so be sure to install from source.
|
| 83 |
|
| 84 |
+
#### Message History
|
| 85 |
+
|
| 86 |
When it comes to using and prompting Proxy Lite, please refer to the [repository](https://github.com/convergence-ai/proxy-lite) for more information, but the model expects a message history of the form:
|
| 87 |
|
| 88 |
```python
|
|
|
|
| 109 |
|
| 110 |
> **Context-Window Management:** When making calls to the model, all the observations other than the current one are discarded in order to reduce the large number of image tokens required. Since the model responses include reflection on the observations and are all included in the message history, the model is still aware of the entire history when planning new actions.
|
| 111 |
|
| 112 |
+
#### Tools
|
| 113 |
+
|
| 114 |
You should also pass the `Tools` that the model has access to, these will define the action space available to the model. You can do this with `transformers`:
|
| 115 |
|
| 116 |
```python
|