Tachi67 commited on
Commit
ffd5b8d
·
verified ·
1 Parent(s): 4610241
Files changed (1) hide show
  1. Introduction_to_Jarvis.md +36 -36
Introduction_to_Jarvis.md CHANGED
@@ -1,30 +1,30 @@
1
- ## Introduction to Jarvis
2
 
3
- Jarvis is a general purpose agent built upon `aiflows`, empowered by a hierarchical structure of large language models and tools including a code interpreter. At a high level, Jarvis takes in tasks in natural language, and achieve the task by making plans, writing and executing code.
4
 
5
 
6
 
7
- To allow for extended the lifespan of Jarvis (constrained by token limitations of LLM APIs) and enable Jarvis to be able to utilize previously accomplished results, we implemented some interesting [memory management mechanisms](#memory-management-mechanisms).
8
 
9
 
10
  **Notice:**
11
- [The code interpreter of Jarvis]((https://huggingface.co/Tachi67/InterpreterFlowModule)) relies on **open-interpreter** (https://github.com/KillianLucas/open-interpreter) (version v0.1.15). We are extracting the specific code from open-interpreter because the `litellm` version of open-interpreter is not compatible with that of the current version of aiflows (v.0.1.7).
12
 
13
 
14
 
15
- ### Structure of Jarvis
16
 
17
- The high level structure of Jarvis is a sequence of flows.
18
 
19
- To start with, the input of Jarvis (from the user) is `goal`, which is the task described in natural language.
20
 
21
- Before running Jarvis, we configure the `memory_files` for Jarvis, memory files are external memories for a flow (a careful reader may have already discovered that we also configure memory files to some other flows, please refer to [Memory Management Mechanisms](#memory-management-mechanisms) for more details). We inject the corresponding memory to the system prompts of the respective LLM chats.
22
 
23
- Entering Jarvis, we first read in the memory from the memory files with `MemoryReadingFlow`. After that, we invoke the `PlanWriter`, to draft a step-by-step plan to achieve the goal given by the user. Finally, the goal, the plan, and other memory, along with the memory_files are input to the `CtrlExMemFlow`, who executes the plan in a controller-executor fashion.
24
 
25
- Let's cut things short, here is an illustration of the Jarvis architecture, for detailed information on all the modules involved down below, find them in the [Basic ideas and components of Jarvis](#basic-ideas-and-components-of-jarvis) section.
26
  ```
27
- goal (input data), memory_files (configured before running Jarvis)
28
  |
29
  v
30
  +-------------------+
@@ -101,7 +101,7 @@ Here is the structure of the `CtrlExMemFlow`:
101
 
102
  ```
103
 
104
- Now, the `Executor` of Jarvis. Remember that it is a "branching" flow with several branches, for detailed information about each branch, find them in the [Basic ideas and components of Jarvis](#basic-ideas-and-components-of-jarvis) section.
105
  ```
106
  +-------------------+
107
  | Branching |
@@ -115,12 +115,12 @@ Coder ask_user re_plan update_plan intermediate_answer final_answer
115
 
116
  ```
117
 
118
- In fact, this whole structure is defined as [AbstractBoss](https://huggingface.co/Tachi67/AbstractBossFlowModule), Jarvis inherits from this abstract flow. There are other components that
119
  inherits from this structure. for example, [Coder](https://huggingface.co/Tachi67/CoderFlowModule), [ExtendLibrary](https://huggingface.co/Tachi67/ExtendLibraryFlowModule).
120
 
121
- #### Basic ideas and components of Jarvis
122
- Let's go through some basic ideas of Jarvis and its components now.
123
- - memory: Jarvis should remember certain important memory, they include the plan (`plan`), and the history of subflows execution results (`logs`). To do this, we put them in external files, and inject them into the system prompts to controllers and other corresponding roles. There are other classes that inherits from `AbstractBoss` and have other memories, e.g. `Coder` takes the memory of `code_libarary`, which is the function signatures and docstrings of the code written by the Coder and saved in the code library.
124
  - memory_files: A dictionary: `{memory_name: memory_file_path}`
125
  - [MemoryReadingFlow](https://huggingface.co/Tachi67/MemoryReadingFlowModule): Reads in the content of the memory files.
126
  - [PlanWriter](https://huggingface.co/Tachi67/PlanWriterFlowModule): Writes a step-by-step plan to achieve the goal in an interactive fashion. During the process of writing a plan, the user has the chance to provide feedback to or directly edit the plan just written. The PlanWriter can comprehend the user's action and make adjustments to the plan accordingly.
@@ -148,9 +148,9 @@ We also applied a sliding window for history chat messages as a workaround for t
148
  - Whenever the chat that utilizes the sliding window is called, we inject a new system prompt with the newest memory (plan, logs, for `Coder` and some other flows, `code_library` is also injected) into the chat history.)
149
  - We crop the chat history with a tailing window of length 3, that will include the assistant's previous call, the updated system prompt, the the user's new response.
150
 
151
- As a result, Jarvis is able to run for quite a long time, and it's able to use the code written before.
152
 
153
- ### An Example run of Jarvis
154
  **Before reading this section**:
155
  - This section is quite lengthy.
156
  - Please run `run_Jarvis.py` when reading this section and follow along, in this way it's much easier to understand what's going on.
@@ -158,17 +158,17 @@ As a result, Jarvis is able to run for quite a long time, and it's able to use t
158
 
159
  Here we go!
160
  - memory files configuration:
161
- - memory files for Jarvis: `plan_jarvis.txt`, `logs_jarvis.txt`
162
  - memory files for Coder: `plan_coder.txt`, `logs_coder.txt`, `library.py`
163
  - memory files for ExtendLibrary: `plan_extlib.txt`, `logs_extlib.txt`, `library.py`
164
  - memory files for [InteractiveCoder](https://huggingface.co/Tachi67/InteractiveCodeGenFlowModule): `library.py`
165
  - memory files for [TestCode](https://huggingface.co/Tachi67/TestCodeFlowModule): `library.py`
166
  - user input goal = "Download tesla's stock prices from 2022-01-01 to 2022-06-01 from yfinance, plot the prices."
167
- - run Jarvis.
168
  - MemoryReadingFlow reads in the memory files, for now, nothing is in the files.
169
- - Entering PlanWriter of Jarvis, plan of Jarvis is generated, the user is able to edit the plan directly or give feedback.
170
 
171
- > Plan (Jarvis):
172
  > 1. Write and run code to download Tesla's stock prices from 2022-01-01 to 2022-06-01 using python package yfinance.
173
  > 2. Write and run code to plot the downloaded stock prices using matplotlib.
174
  > 3. Give a final answer to the user.
@@ -275,11 +275,11 @@ Here we go!
275
  - MemoryReadingFlow of Coder reads in the memory, here, new logs will override the old logs.
276
  - The Controller of CtrlExMemFlow_Coder finishes the Coder flow and return result to the JarvisFlow.
277
  - When finishing, the CoderFlow resets, setting its flow_state and the flow_state of its subflows an an empty dict. The core takeaway here is that the chat history of the Controller of CtrlExMemFlow_Coder and CtrlExMemFlow_ExtLib (as Coder's subflow) is erased.
278
- - MemoryWritingFlow of Jarvis updates the logs of Jarvis, it records the action of Coder.
279
- - MemoryReadingFlow of Jarvis reads in the memory, here, new logs will override the old logs.
280
  - The Controller of CtrlExMemFlow_Jarvis calls update_plan to mark step 1 of the plan as done.
281
- - MemoryWritingFlow of Jarvis updates the logs of Jarvis, it records the action of update_plan.
282
- - MemoryReadingFlow of Jarvis reads in the memory, here, new logs will override the old logs.
283
  - The Controller of CtrlExMemFlow_Jarvis calls Coder with the goal of the second step of plan.
284
  - The PlanWriter of Coder makes a plan:
285
 
@@ -359,21 +359,21 @@ Here we go!
359
  - The Controller of CtrlExMemFlow_Coder updates the rest of the plan as done.
360
  - The MemoryWritingFlow of Coder updates the logs of Coder, it records the action of update_plan.
361
  - The MemoryReadingFlow of Coder reads in the memory, here, new logs will override the old logs.
362
- - The Controller of CtrlExMemFlow_Coder finishes the CoderFlow, reset the flow and go back to Jarvis.
363
- - The MemoryWritingFlow of Jarvis updates the logs of Jarvis, it records the action of Coder.
364
- - The MemoryReadingFlow of Jarvis reads in the memory, here, new logs will override the old logs.
365
  - The Controller of CtrlExMemFlow_Jarvis updates the plan to mark the second step of plan as done.
366
- - The MemoryWritingFlow of Jarvis updates the logs of Jarvis, it records the action of update_plan.
367
- - The MemoryReadingFlow of Jarvis reads in the memory, here, new logs will override the old logs.
368
  - The Controller of CtrlExMemFlow_Jarvis calls final_ans to provide the final answer.
369
  - The user provides a positive feedback, for example, "lgtm".
370
- - The Controller of CtrlExMemFlow_Jarvis finishes the Jarvis flow.
371
 
372
 
373
  ### Future Improvements
374
- **We are currently running on V0 of Jarvis, so any contributions are more than welcome!**
375
 
376
- Up until the publishing of V0 Jarvis, there are several things that we would like to improve:
377
 
378
  - When running code tests in [TestCodeFlow](https://huggingface.co/Tachi67/TestCodeFlowModule) we should show the test results to the user and fetch for feedback before exiting the flow, and add the part of user feedback to the result returned to the controller.
379
  - This can be done by appending a subflow to ask for human feedback (like [this](https://huggingface.co/Tachi67/JarvisFlowModule/blob/main/IntermediateAns_Jarvis.py)) to [TestCodeFlow](https://huggingface.co/Tachi67/TestCodeFlowModule/blob/main/TestCodeFlow.py).
@@ -404,6 +404,6 @@ Up until the publishing of V0 Jarvis, there are several things that we would lik
404
 
405
 
406
  ### Acknowledgement
407
- Jarvis is done independently as a semester project by [Haolong Li](https://github.com/Tachi-67) ([haolong.li@epfl.ch](mailto:haolong.li@epfl.ch)) at EPFL. Very special and strong thanks to his mentor: Martin Josifoski, and semi-mentor: Nicolas Baldwin for their powerful and long-lasting help and support during the design and implementation of Jarvis, Jarvis is not possible without you guys, love you!
408
 
409
- Jarvis is built under the framework of [aiflows](https://github.com/epfl-dlab/aiflows), it's cool -- try it out!
 
1
+ ## Introduction to JARVIS
2
 
3
+ JARVIS is a general purpose agent built upon `aiflows`, empowered by a hierarchical structure of large language models and tools including a code interpreter. At a high level, JARVIS takes in tasks in natural language, and achieve the task by making plans, writing and executing code.
4
 
5
 
6
 
7
+ To allow for extended the lifespan of JARVIS (constrained by token limitations of LLM APIs) and enable JARVIS to be able to utilize previously accomplished results, we implemented some interesting [memory management mechanisms](#memory-management-mechanisms).
8
 
9
 
10
  **Notice:**
11
+ [The code interpreter of JARVIS]((https://huggingface.co/Tachi67/InterpreterFlowModule)) relies on **open-interpreter** (https://github.com/KillianLucas/open-interpreter) (version v0.1.15). We are extracting the specific code from open-interpreter because the `litellm` version of open-interpreter is not compatible with that of the current version of aiflows (v.0.1.7).
12
 
13
 
14
 
15
+ ### Structure of JARVIS
16
 
17
+ The high level structure of JARVIS is a sequence of flows.
18
 
19
+ To start with, the input of JARVIS (from the user) is `goal`, which is the task described in natural language.
20
 
21
+ Before running JARVIS, we configure the `memory_files` for JARVIS, memory files are external memories for a flow (a careful reader may have already discovered that we also configure memory files to some other flows, please refer to [Memory Management Mechanisms](#memory-management-mechanisms) for more details). We inject the corresponding memory to the system prompts of the respective LLM chats.
22
 
23
+ Entering JARVIS, we first read in the memory from the memory files with `MemoryReadingFlow`. After that, we invoke the `PlanWriter`, to draft a step-by-step plan to achieve the goal given by the user. Finally, the goal, the plan, and other memory, along with the memory_files are input to the `CtrlExMemFlow`, who executes the plan in a controller-executor fashion.
24
 
25
+ Let's cut things short, here is an illustration of the JARVIS architecture, for detailed information on all the modules involved down below, find them in the [Basic ideas and components of JARVIS](#basic-ideas-and-components-of-jarvis) section.
26
  ```
27
+ goal (input data), memory_files (configured before running JARVIS)
28
  |
29
  v
30
  +-------------------+
 
101
 
102
  ```
103
 
104
+ Now, the `Executor` of JARVIS. Remember that it is a "branching" flow with several branches, for detailed information about each branch, find them in the [Basic ideas and components of JARVIS](#basic-ideas-and-components-of-jarvis) section.
105
  ```
106
  +-------------------+
107
  | Branching |
 
115
 
116
  ```
117
 
118
+ In fact, this whole structure is defined as [AbstractBoss](https://huggingface.co/Tachi67/AbstractBossFlowModule), JARVIS inherits from this abstract flow. There are other components that
119
  inherits from this structure. for example, [Coder](https://huggingface.co/Tachi67/CoderFlowModule), [ExtendLibrary](https://huggingface.co/Tachi67/ExtendLibraryFlowModule).
120
 
121
+ #### Basic ideas and components of JARVIS
122
+ Let's go through some basic ideas of JARVIS and its components now.
123
+ - memory: JARVIS should remember certain important memory, they include the plan (`plan`), and the history of subflows execution results (`logs`). To do this, we put them in external files, and inject them into the system prompts to controllers and other corresponding roles. There are other classes that inherits from `AbstractBoss` and have other memories, e.g. `Coder` takes the memory of `code_libarary`, which is the function signatures and docstrings of the code written by the Coder and saved in the code library.
124
  - memory_files: A dictionary: `{memory_name: memory_file_path}`
125
  - [MemoryReadingFlow](https://huggingface.co/Tachi67/MemoryReadingFlowModule): Reads in the content of the memory files.
126
  - [PlanWriter](https://huggingface.co/Tachi67/PlanWriterFlowModule): Writes a step-by-step plan to achieve the goal in an interactive fashion. During the process of writing a plan, the user has the chance to provide feedback to or directly edit the plan just written. The PlanWriter can comprehend the user's action and make adjustments to the plan accordingly.
 
148
  - Whenever the chat that utilizes the sliding window is called, we inject a new system prompt with the newest memory (plan, logs, for `Coder` and some other flows, `code_library` is also injected) into the chat history.)
149
  - We crop the chat history with a tailing window of length 3, that will include the assistant's previous call, the updated system prompt, the the user's new response.
150
 
151
+ As a result, JARVIS is able to run for quite a long time, and it's able to use the code written before.
152
 
153
+ ### An Example run of JARVIS
154
  **Before reading this section**:
155
  - This section is quite lengthy.
156
  - Please run `run_Jarvis.py` when reading this section and follow along, in this way it's much easier to understand what's going on.
 
158
 
159
  Here we go!
160
  - memory files configuration:
161
+ - memory files for JARVIS: `plan_jarvis.txt`, `logs_jarvis.txt`
162
  - memory files for Coder: `plan_coder.txt`, `logs_coder.txt`, `library.py`
163
  - memory files for ExtendLibrary: `plan_extlib.txt`, `logs_extlib.txt`, `library.py`
164
  - memory files for [InteractiveCoder](https://huggingface.co/Tachi67/InteractiveCodeGenFlowModule): `library.py`
165
  - memory files for [TestCode](https://huggingface.co/Tachi67/TestCodeFlowModule): `library.py`
166
  - user input goal = "Download tesla's stock prices from 2022-01-01 to 2022-06-01 from yfinance, plot the prices."
167
+ - run JARVIS.
168
  - MemoryReadingFlow reads in the memory files, for now, nothing is in the files.
169
+ - Entering PlanWriter of JARVIS, plan of JARVIS is generated, the user is able to edit the plan directly or give feedback.
170
 
171
+ > Plan (JARVIS):
172
  > 1. Write and run code to download Tesla's stock prices from 2022-01-01 to 2022-06-01 using python package yfinance.
173
  > 2. Write and run code to plot the downloaded stock prices using matplotlib.
174
  > 3. Give a final answer to the user.
 
275
  - MemoryReadingFlow of Coder reads in the memory, here, new logs will override the old logs.
276
  - The Controller of CtrlExMemFlow_Coder finishes the Coder flow and return result to the JarvisFlow.
277
  - When finishing, the CoderFlow resets, setting its flow_state and the flow_state of its subflows an an empty dict. The core takeaway here is that the chat history of the Controller of CtrlExMemFlow_Coder and CtrlExMemFlow_ExtLib (as Coder's subflow) is erased.
278
+ - MemoryWritingFlow of JARVIS updates the logs of JARVIS, it records the action of Coder.
279
+ - MemoryReadingFlow of JARVIS reads in the memory, here, new logs will override the old logs.
280
  - The Controller of CtrlExMemFlow_Jarvis calls update_plan to mark step 1 of the plan as done.
281
+ - MemoryWritingFlow of JARVIS updates the logs of JARVIS, it records the action of update_plan.
282
+ - MemoryReadingFlow of JARVIS reads in the memory, here, new logs will override the old logs.
283
  - The Controller of CtrlExMemFlow_Jarvis calls Coder with the goal of the second step of plan.
284
  - The PlanWriter of Coder makes a plan:
285
 
 
359
  - The Controller of CtrlExMemFlow_Coder updates the rest of the plan as done.
360
  - The MemoryWritingFlow of Coder updates the logs of Coder, it records the action of update_plan.
361
  - The MemoryReadingFlow of Coder reads in the memory, here, new logs will override the old logs.
362
+ - The Controller of CtrlExMemFlow_Coder finishes the CoderFlow, reset the flow and go back to JARVIS.
363
+ - The MemoryWritingFlow of JARVIS updates the logs of JARVIS, it records the action of Coder.
364
+ - The MemoryReadingFlow of JARVIS reads in the memory, here, new logs will override the old logs.
365
  - The Controller of CtrlExMemFlow_Jarvis updates the plan to mark the second step of plan as done.
366
+ - The MemoryWritingFlow of JARVIS updates the logs of JARVIS, it records the action of update_plan.
367
+ - The MemoryReadingFlow of JARVIS reads in the memory, here, new logs will override the old logs.
368
  - The Controller of CtrlExMemFlow_Jarvis calls final_ans to provide the final answer.
369
  - The user provides a positive feedback, for example, "lgtm".
370
+ - The Controller of CtrlExMemFlow_Jarvis finishes the JARVIS flow.
371
 
372
 
373
  ### Future Improvements
374
+ **We are currently running on V0 of JARVIS, so any contributions are more than welcome!**
375
 
376
+ Up until the publishing of V0 JARVIS, there are several things that we would like to improve:
377
 
378
  - When running code tests in [TestCodeFlow](https://huggingface.co/Tachi67/TestCodeFlowModule) we should show the test results to the user and fetch for feedback before exiting the flow, and add the part of user feedback to the result returned to the controller.
379
  - This can be done by appending a subflow to ask for human feedback (like [this](https://huggingface.co/Tachi67/JarvisFlowModule/blob/main/IntermediateAns_Jarvis.py)) to [TestCodeFlow](https://huggingface.co/Tachi67/TestCodeFlowModule/blob/main/TestCodeFlow.py).
 
404
 
405
 
406
  ### Acknowledgement
407
+ JARVIS is done independently as a semester project by [Haolong Li](https://github.com/Tachi-67) ([haolong.li@epfl.ch](mailto:haolong.li@epfl.ch)) at EPFL. Very special and strong thanks to his mentor: Martin Josifoski, and semi-mentor: Nicolas Baldwin for their powerful and long-lasting help and support during the design and implementation of JARVIS, JARVIS is not possible without you guys, love you!
408
 
409
+ JARVIS is built under the framework of [aiflows](https://github.com/epfl-dlab/aiflows), it's cool -- try it out!