fix typo
Browse files- Introduction_to_Jarvis.md +36 -36
Introduction_to_Jarvis.md
CHANGED
|
@@ -1,30 +1,30 @@
|
|
| 1 |
-
## Introduction to
|
| 2 |
|
| 3 |
-
|
| 4 |
|
| 5 |
|
| 6 |
|
| 7 |
-
To allow for extended the lifespan of
|
| 8 |
|
| 9 |
|
| 10 |
**Notice:**
|
| 11 |
-
[The code interpreter of
|
| 12 |
|
| 13 |
|
| 14 |
|
| 15 |
-
### Structure of
|
| 16 |
|
| 17 |
-
The high level structure of
|
| 18 |
|
| 19 |
-
To start with, the input of
|
| 20 |
|
| 21 |
-
Before running
|
| 22 |
|
| 23 |
-
Entering
|
| 24 |
|
| 25 |
-
Let's cut things short, here is an illustration of the
|
| 26 |
```
|
| 27 |
-
goal (input data), memory_files (configured before running
|
| 28 |
|
|
| 29 |
v
|
| 30 |
+-------------------+
|
|
@@ -101,7 +101,7 @@ Here is the structure of the `CtrlExMemFlow`:
|
|
| 101 |
|
| 102 |
```
|
| 103 |
|
| 104 |
-
Now, the `Executor` of
|
| 105 |
```
|
| 106 |
+-------------------+
|
| 107 |
| Branching |
|
|
@@ -115,12 +115,12 @@ Coder ask_user re_plan update_plan intermediate_answer final_answer
|
|
| 115 |
|
| 116 |
```
|
| 117 |
|
| 118 |
-
In fact, this whole structure is defined as [AbstractBoss](https://huggingface.co/Tachi67/AbstractBossFlowModule),
|
| 119 |
inherits from this structure. for example, [Coder](https://huggingface.co/Tachi67/CoderFlowModule), [ExtendLibrary](https://huggingface.co/Tachi67/ExtendLibraryFlowModule).
|
| 120 |
|
| 121 |
-
#### Basic ideas and components of
|
| 122 |
-
Let's go through some basic ideas of
|
| 123 |
-
- memory:
|
| 124 |
- memory_files: A dictionary: `{memory_name: memory_file_path}`
|
| 125 |
- [MemoryReadingFlow](https://huggingface.co/Tachi67/MemoryReadingFlowModule): Reads in the content of the memory files.
|
| 126 |
- [PlanWriter](https://huggingface.co/Tachi67/PlanWriterFlowModule): Writes a step-by-step plan to achieve the goal in an interactive fashion. During the process of writing a plan, the user has the chance to provide feedback to or directly edit the plan just written. The PlanWriter can comprehend the user's action and make adjustments to the plan accordingly.
|
|
@@ -148,9 +148,9 @@ We also applied a sliding window for history chat messages as a workaround for t
|
|
| 148 |
- Whenever the chat that utilizes the sliding window is called, we inject a new system prompt with the newest memory (plan, logs, for `Coder` and some other flows, `code_library` is also injected) into the chat history.)
|
| 149 |
- We crop the chat history with a tailing window of length 3, that will include the assistant's previous call, the updated system prompt, the the user's new response.
|
| 150 |
|
| 151 |
-
As a result,
|
| 152 |
|
| 153 |
-
### An Example run of
|
| 154 |
**Before reading this section**:
|
| 155 |
- This section is quite lengthy.
|
| 156 |
- Please run `run_Jarvis.py` when reading this section and follow along, in this way it's much easier to understand what's going on.
|
|
@@ -158,17 +158,17 @@ As a result, Jarvis is able to run for quite a long time, and it's able to use t
|
|
| 158 |
|
| 159 |
Here we go!
|
| 160 |
- memory files configuration:
|
| 161 |
-
- memory files for
|
| 162 |
- memory files for Coder: `plan_coder.txt`, `logs_coder.txt`, `library.py`
|
| 163 |
- memory files for ExtendLibrary: `plan_extlib.txt`, `logs_extlib.txt`, `library.py`
|
| 164 |
- memory files for [InteractiveCoder](https://huggingface.co/Tachi67/InteractiveCodeGenFlowModule): `library.py`
|
| 165 |
- memory files for [TestCode](https://huggingface.co/Tachi67/TestCodeFlowModule): `library.py`
|
| 166 |
- user input goal = "Download tesla's stock prices from 2022-01-01 to 2022-06-01 from yfinance, plot the prices."
|
| 167 |
-
- run
|
| 168 |
- MemoryReadingFlow reads in the memory files, for now, nothing is in the files.
|
| 169 |
-
- Entering PlanWriter of
|
| 170 |
|
| 171 |
-
> Plan (
|
| 172 |
> 1. Write and run code to download Tesla's stock prices from 2022-01-01 to 2022-06-01 using python package yfinance.
|
| 173 |
> 2. Write and run code to plot the downloaded stock prices using matplotlib.
|
| 174 |
> 3. Give a final answer to the user.
|
|
@@ -275,11 +275,11 @@ Here we go!
|
|
| 275 |
- MemoryReadingFlow of Coder reads in the memory, here, new logs will override the old logs.
|
| 276 |
- The Controller of CtrlExMemFlow_Coder finishes the Coder flow and return result to the JarvisFlow.
|
| 277 |
- When finishing, the CoderFlow resets, setting its flow_state and the flow_state of its subflows an an empty dict. The core takeaway here is that the chat history of the Controller of CtrlExMemFlow_Coder and CtrlExMemFlow_ExtLib (as Coder's subflow) is erased.
|
| 278 |
-
- MemoryWritingFlow of
|
| 279 |
-
- MemoryReadingFlow of
|
| 280 |
- The Controller of CtrlExMemFlow_Jarvis calls update_plan to mark step 1 of the plan as done.
|
| 281 |
-
- MemoryWritingFlow of
|
| 282 |
-
- MemoryReadingFlow of
|
| 283 |
- The Controller of CtrlExMemFlow_Jarvis calls Coder with the goal of the second step of plan.
|
| 284 |
- The PlanWriter of Coder makes a plan:
|
| 285 |
|
|
@@ -359,21 +359,21 @@ Here we go!
|
|
| 359 |
- The Controller of CtrlExMemFlow_Coder updates the rest of the plan as done.
|
| 360 |
- The MemoryWritingFlow of Coder updates the logs of Coder, it records the action of update_plan.
|
| 361 |
- The MemoryReadingFlow of Coder reads in the memory, here, new logs will override the old logs.
|
| 362 |
-
- The Controller of CtrlExMemFlow_Coder finishes the CoderFlow, reset the flow and go back to
|
| 363 |
-
- The MemoryWritingFlow of
|
| 364 |
-
- The MemoryReadingFlow of
|
| 365 |
- The Controller of CtrlExMemFlow_Jarvis updates the plan to mark the second step of plan as done.
|
| 366 |
-
- The MemoryWritingFlow of
|
| 367 |
-
- The MemoryReadingFlow of
|
| 368 |
- The Controller of CtrlExMemFlow_Jarvis calls final_ans to provide the final answer.
|
| 369 |
- The user provides a positive feedback, for example, "lgtm".
|
| 370 |
-
- The Controller of CtrlExMemFlow_Jarvis finishes the
|
| 371 |
|
| 372 |
|
| 373 |
### Future Improvements
|
| 374 |
-
**We are currently running on V0 of
|
| 375 |
|
| 376 |
-
Up until the publishing of V0
|
| 377 |
|
| 378 |
- When running code tests in [TestCodeFlow](https://huggingface.co/Tachi67/TestCodeFlowModule) we should show the test results to the user and fetch for feedback before exiting the flow, and add the part of user feedback to the result returned to the controller.
|
| 379 |
- This can be done by appending a subflow to ask for human feedback (like [this](https://huggingface.co/Tachi67/JarvisFlowModule/blob/main/IntermediateAns_Jarvis.py)) to [TestCodeFlow](https://huggingface.co/Tachi67/TestCodeFlowModule/blob/main/TestCodeFlow.py).
|
|
@@ -404,6 +404,6 @@ Up until the publishing of V0 Jarvis, there are several things that we would lik
|
|
| 404 |
|
| 405 |
|
| 406 |
### Acknowledgement
|
| 407 |
-
|
| 408 |
|
| 409 |
-
|
|
|
|
| 1 |
+
## Introduction to JARVIS
|
| 2 |
|
| 3 |
+
JARVIS is a general purpose agent built upon `aiflows`, empowered by a hierarchical structure of large language models and tools including a code interpreter. At a high level, JARVIS takes in tasks in natural language, and achieve the task by making plans, writing and executing code.
|
| 4 |
|
| 5 |
|
| 6 |
|
| 7 |
+
To allow for extended the lifespan of JARVIS (constrained by token limitations of LLM APIs) and enable JARVIS to be able to utilize previously accomplished results, we implemented some interesting [memory management mechanisms](#memory-management-mechanisms).
|
| 8 |
|
| 9 |
|
| 10 |
**Notice:**
|
| 11 |
+
[The code interpreter of JARVIS]((https://huggingface.co/Tachi67/InterpreterFlowModule)) relies on **open-interpreter** (https://github.com/KillianLucas/open-interpreter) (version v0.1.15). We are extracting the specific code from open-interpreter because the `litellm` version of open-interpreter is not compatible with that of the current version of aiflows (v.0.1.7).
|
| 12 |
|
| 13 |
|
| 14 |
|
| 15 |
+
### Structure of JARVIS
|
| 16 |
|
| 17 |
+
The high level structure of JARVIS is a sequence of flows.
|
| 18 |
|
| 19 |
+
To start with, the input of JARVIS (from the user) is `goal`, which is the task described in natural language.
|
| 20 |
|
| 21 |
+
Before running JARVIS, we configure the `memory_files` for JARVIS, memory files are external memories for a flow (a careful reader may have already discovered that we also configure memory files to some other flows, please refer to [Memory Management Mechanisms](#memory-management-mechanisms) for more details). We inject the corresponding memory to the system prompts of the respective LLM chats.
|
| 22 |
|
| 23 |
+
Entering JARVIS, we first read in the memory from the memory files with `MemoryReadingFlow`. After that, we invoke the `PlanWriter`, to draft a step-by-step plan to achieve the goal given by the user. Finally, the goal, the plan, and other memory, along with the memory_files are input to the `CtrlExMemFlow`, who executes the plan in a controller-executor fashion.
|
| 24 |
|
| 25 |
+
Let's cut things short, here is an illustration of the JARVIS architecture, for detailed information on all the modules involved down below, find them in the [Basic ideas and components of JARVIS](#basic-ideas-and-components-of-jarvis) section.
|
| 26 |
```
|
| 27 |
+
goal (input data), memory_files (configured before running JARVIS)
|
| 28 |
|
|
| 29 |
v
|
| 30 |
+-------------------+
|
|
|
|
| 101 |
|
| 102 |
```
|
| 103 |
|
| 104 |
+
Now, the `Executor` of JARVIS. Remember that it is a "branching" flow with several branches, for detailed information about each branch, find them in the [Basic ideas and components of JARVIS](#basic-ideas-and-components-of-jarvis) section.
|
| 105 |
```
|
| 106 |
+-------------------+
|
| 107 |
| Branching |
|
|
|
|
| 115 |
|
| 116 |
```
|
| 117 |
|
| 118 |
+
In fact, this whole structure is defined as [AbstractBoss](https://huggingface.co/Tachi67/AbstractBossFlowModule), JARVIS inherits from this abstract flow. There are other components that
|
| 119 |
inherits from this structure. for example, [Coder](https://huggingface.co/Tachi67/CoderFlowModule), [ExtendLibrary](https://huggingface.co/Tachi67/ExtendLibraryFlowModule).
|
| 120 |
|
| 121 |
+
#### Basic ideas and components of JARVIS
|
| 122 |
+
Let's go through some basic ideas of JARVIS and its components now.
|
| 123 |
+
- memory: JARVIS should remember certain important memory, they include the plan (`plan`), and the history of subflows execution results (`logs`). To do this, we put them in external files, and inject them into the system prompts to controllers and other corresponding roles. There are other classes that inherits from `AbstractBoss` and have other memories, e.g. `Coder` takes the memory of `code_libarary`, which is the function signatures and docstrings of the code written by the Coder and saved in the code library.
|
| 124 |
- memory_files: A dictionary: `{memory_name: memory_file_path}`
|
| 125 |
- [MemoryReadingFlow](https://huggingface.co/Tachi67/MemoryReadingFlowModule): Reads in the content of the memory files.
|
| 126 |
- [PlanWriter](https://huggingface.co/Tachi67/PlanWriterFlowModule): Writes a step-by-step plan to achieve the goal in an interactive fashion. During the process of writing a plan, the user has the chance to provide feedback to or directly edit the plan just written. The PlanWriter can comprehend the user's action and make adjustments to the plan accordingly.
|
|
|
|
| 148 |
- Whenever the chat that utilizes the sliding window is called, we inject a new system prompt with the newest memory (plan, logs, for `Coder` and some other flows, `code_library` is also injected) into the chat history.)
|
| 149 |
- We crop the chat history with a tailing window of length 3, that will include the assistant's previous call, the updated system prompt, the the user's new response.
|
| 150 |
|
| 151 |
+
As a result, JARVIS is able to run for quite a long time, and it's able to use the code written before.
|
| 152 |
|
| 153 |
+
### An Example run of JARVIS
|
| 154 |
**Before reading this section**:
|
| 155 |
- This section is quite lengthy.
|
| 156 |
- Please run `run_Jarvis.py` when reading this section and follow along, in this way it's much easier to understand what's going on.
|
|
|
|
| 158 |
|
| 159 |
Here we go!
|
| 160 |
- memory files configuration:
|
| 161 |
+
- memory files for JARVIS: `plan_jarvis.txt`, `logs_jarvis.txt`
|
| 162 |
- memory files for Coder: `plan_coder.txt`, `logs_coder.txt`, `library.py`
|
| 163 |
- memory files for ExtendLibrary: `plan_extlib.txt`, `logs_extlib.txt`, `library.py`
|
| 164 |
- memory files for [InteractiveCoder](https://huggingface.co/Tachi67/InteractiveCodeGenFlowModule): `library.py`
|
| 165 |
- memory files for [TestCode](https://huggingface.co/Tachi67/TestCodeFlowModule): `library.py`
|
| 166 |
- user input goal = "Download tesla's stock prices from 2022-01-01 to 2022-06-01 from yfinance, plot the prices."
|
| 167 |
+
- run JARVIS.
|
| 168 |
- MemoryReadingFlow reads in the memory files, for now, nothing is in the files.
|
| 169 |
+
- Entering PlanWriter of JARVIS, plan of JARVIS is generated, the user is able to edit the plan directly or give feedback.
|
| 170 |
|
| 171 |
+
> Plan (JARVIS):
|
| 172 |
> 1. Write and run code to download Tesla's stock prices from 2022-01-01 to 2022-06-01 using python package yfinance.
|
| 173 |
> 2. Write and run code to plot the downloaded stock prices using matplotlib.
|
| 174 |
> 3. Give a final answer to the user.
|
|
|
|
| 275 |
- MemoryReadingFlow of Coder reads in the memory, here, new logs will override the old logs.
|
| 276 |
- The Controller of CtrlExMemFlow_Coder finishes the Coder flow and return result to the JarvisFlow.
|
| 277 |
- When finishing, the CoderFlow resets, setting its flow_state and the flow_state of its subflows an an empty dict. The core takeaway here is that the chat history of the Controller of CtrlExMemFlow_Coder and CtrlExMemFlow_ExtLib (as Coder's subflow) is erased.
|
| 278 |
+
- MemoryWritingFlow of JARVIS updates the logs of JARVIS, it records the action of Coder.
|
| 279 |
+
- MemoryReadingFlow of JARVIS reads in the memory, here, new logs will override the old logs.
|
| 280 |
- The Controller of CtrlExMemFlow_Jarvis calls update_plan to mark step 1 of the plan as done.
|
| 281 |
+
- MemoryWritingFlow of JARVIS updates the logs of JARVIS, it records the action of update_plan.
|
| 282 |
+
- MemoryReadingFlow of JARVIS reads in the memory, here, new logs will override the old logs.
|
| 283 |
- The Controller of CtrlExMemFlow_Jarvis calls Coder with the goal of the second step of plan.
|
| 284 |
- The PlanWriter of Coder makes a plan:
|
| 285 |
|
|
|
|
| 359 |
- The Controller of CtrlExMemFlow_Coder updates the rest of the plan as done.
|
| 360 |
- The MemoryWritingFlow of Coder updates the logs of Coder, it records the action of update_plan.
|
| 361 |
- The MemoryReadingFlow of Coder reads in the memory, here, new logs will override the old logs.
|
| 362 |
+
- The Controller of CtrlExMemFlow_Coder finishes the CoderFlow, reset the flow and go back to JARVIS.
|
| 363 |
+
- The MemoryWritingFlow of JARVIS updates the logs of JARVIS, it records the action of Coder.
|
| 364 |
+
- The MemoryReadingFlow of JARVIS reads in the memory, here, new logs will override the old logs.
|
| 365 |
- The Controller of CtrlExMemFlow_Jarvis updates the plan to mark the second step of plan as done.
|
| 366 |
+
- The MemoryWritingFlow of JARVIS updates the logs of JARVIS, it records the action of update_plan.
|
| 367 |
+
- The MemoryReadingFlow of JARVIS reads in the memory, here, new logs will override the old logs.
|
| 368 |
- The Controller of CtrlExMemFlow_Jarvis calls final_ans to provide the final answer.
|
| 369 |
- The user provides a positive feedback, for example, "lgtm".
|
| 370 |
+
- The Controller of CtrlExMemFlow_Jarvis finishes the JARVIS flow.
|
| 371 |
|
| 372 |
|
| 373 |
### Future Improvements
|
| 374 |
+
**We are currently running on V0 of JARVIS, so any contributions are more than welcome!**
|
| 375 |
|
| 376 |
+
Up until the publishing of V0 JARVIS, there are several things that we would like to improve:
|
| 377 |
|
| 378 |
- When running code tests in [TestCodeFlow](https://huggingface.co/Tachi67/TestCodeFlowModule) we should show the test results to the user and fetch for feedback before exiting the flow, and add the part of user feedback to the result returned to the controller.
|
| 379 |
- This can be done by appending a subflow to ask for human feedback (like [this](https://huggingface.co/Tachi67/JarvisFlowModule/blob/main/IntermediateAns_Jarvis.py)) to [TestCodeFlow](https://huggingface.co/Tachi67/TestCodeFlowModule/blob/main/TestCodeFlow.py).
|
|
|
|
| 404 |
|
| 405 |
|
| 406 |
### Acknowledgement
|
| 407 |
+
JARVIS is done independently as a semester project by [Haolong Li](https://github.com/Tachi-67) ([haolong.li@epfl.ch](mailto:haolong.li@epfl.ch)) at EPFL. Very special and strong thanks to his mentor: Martin Josifoski, and semi-mentor: Nicolas Baldwin for their powerful and long-lasting help and support during the design and implementation of JARVIS, JARVIS is not possible without you guys, love you!
|
| 408 |
|
| 409 |
+
JARVIS is built under the framework of [aiflows](https://github.com/epfl-dlab/aiflows), it's cool -- try it out!
|