Spaces:
Running
Running
Adam McQuilkin commited on
Commit ·
6033d24
1
Parent(s): 451b0cb
Default readme remote options to closed (#28)
Browse files
README.md
CHANGED
|
@@ -14,6 +14,7 @@
|
|
| 14 |
To install the library, simply run `pip install repo2vec`!
|
| 15 |
|
| 16 |
## Prerequisites
|
|
|
|
| 17 |
`repo2vec` performs two steps:
|
| 18 |
|
| 19 |
1. Indexes your codebase (requiring an embdder and a vector store)
|
|
@@ -23,6 +24,7 @@ To install the library, simply run `pip install repo2vec`!
|
|
| 23 |
<summary><strong>:computer: Running locally</strong></summary>
|
| 24 |
|
| 25 |
1. To index the codebase locally, we use the open-source project <a href="https://github.com/marqo-ai/marqo">Marqo</a>, which is both an embedder and a vector store. To bring up a Marqo instance:
|
|
|
|
| 26 |
```
|
| 27 |
docker rm -f marqo
|
| 28 |
docker pull marqoai/marqo:latest
|
|
@@ -33,6 +35,7 @@ To install the library, simply run `pip install repo2vec`!
|
|
| 33 |
|
| 34 |
- Head over to [ollama.com](https://ollama.com) to download the appropriate binary for your machine.
|
| 35 |
- Pull the desired model, e.g. `ollama pull llama3.1`.
|
|
|
|
| 36 |
</details>
|
| 37 |
|
| 38 |
<details>
|
|
@@ -50,6 +53,7 @@ To install the library, simply run `pip install repo2vec`!
|
|
| 50 |
```
|
| 51 |
export ANTHROPIC_API_KEY=...
|
| 52 |
```
|
|
|
|
| 53 |
</details>
|
| 54 |
|
| 55 |
<br>
|
|
@@ -59,6 +63,7 @@ If you are planning on indexing GitHub issues in addition to the codebase, you w
|
|
| 59 |
export GITHUB_TOKEN=...
|
| 60 |
|
| 61 |
## Running it
|
|
|
|
| 62 |
<details open>
|
| 63 |
<summary><strong>:computer: Running locally</strong></summary>
|
| 64 |
<p>To index the codebase:</p>
|
|
@@ -77,7 +82,7 @@ If you are planning on indexing GitHub issues in addition to the codebase, you w
|
|
| 77 |
--llm-model=llama3.1
|
| 78 |
</details>
|
| 79 |
|
| 80 |
-
<details
|
| 81 |
<summary><strong>:cloud: Using external providers</strong></summary>
|
| 82 |
<p>To index the codebase:</p>
|
| 83 |
|
|
@@ -98,25 +103,35 @@ To get a public URL for your chat app, set `--share=true`.
|
|
| 98 |
</details>
|
| 99 |
|
| 100 |
## Additional features
|
|
|
|
| 101 |
- **Control which files get indexed** based on their extension. You can whitelist or blacklist extensions by passing a file with one extension per line (in the format `.ext`):
|
| 102 |
-
|
|
|
|
| 103 |
```
|
| 104 |
index ... --include=/path/to/extensions/file
|
| 105 |
```
|
| 106 |
-
|
|
|
|
|
|
|
| 107 |
```
|
| 108 |
index ... --exclude=/path/to/extensions/file
|
| 109 |
```
|
|
|
|
| 110 |
- **Index open GitHub issues** (remember to `export GITHUB_TOKEN=...`):
|
| 111 |
-
|
|
|
|
| 112 |
```
|
| 113 |
index ... --index-issues
|
| 114 |
```
|
| 115 |
-
|
|
|
|
|
|
|
| 116 |
```
|
| 117 |
index ... --index-issues --index-issue-comments
|
| 118 |
```
|
| 119 |
-
|
|
|
|
|
|
|
| 120 |
```
|
| 121 |
index ... --index-issues --no-index-repo
|
| 122 |
```
|
|
@@ -153,4 +168,4 @@ If you're the maintainer of an OSS repo and would like a dedicated page on Code
|
|
| 153 |
|
| 154 |
We built the code purposefully modular so that you can plug in your desired embeddings, LLM and vector stores providers by simply implementing the relevant abstract classes.
|
| 155 |
|
| 156 |
-
Feel free to send feature requests to [founders@storia.ai](mailto:founders@storia.ai) or make a pull request!
|
|
|
|
| 14 |
To install the library, simply run `pip install repo2vec`!
|
| 15 |
|
| 16 |
## Prerequisites
|
| 17 |
+
|
| 18 |
`repo2vec` performs two steps:
|
| 19 |
|
| 20 |
1. Indexes your codebase (requiring an embdder and a vector store)
|
|
|
|
| 24 |
<summary><strong>:computer: Running locally</strong></summary>
|
| 25 |
|
| 26 |
1. To index the codebase locally, we use the open-source project <a href="https://github.com/marqo-ai/marqo">Marqo</a>, which is both an embedder and a vector store. To bring up a Marqo instance:
|
| 27 |
+
|
| 28 |
```
|
| 29 |
docker rm -f marqo
|
| 30 |
docker pull marqoai/marqo:latest
|
|
|
|
| 35 |
|
| 36 |
- Head over to [ollama.com](https://ollama.com) to download the appropriate binary for your machine.
|
| 37 |
- Pull the desired model, e.g. `ollama pull llama3.1`.
|
| 38 |
+
|
| 39 |
</details>
|
| 40 |
|
| 41 |
<details>
|
|
|
|
| 53 |
```
|
| 54 |
export ANTHROPIC_API_KEY=...
|
| 55 |
```
|
| 56 |
+
|
| 57 |
</details>
|
| 58 |
|
| 59 |
<br>
|
|
|
|
| 63 |
export GITHUB_TOKEN=...
|
| 64 |
|
| 65 |
## Running it
|
| 66 |
+
|
| 67 |
<details open>
|
| 68 |
<summary><strong>:computer: Running locally</strong></summary>
|
| 69 |
<p>To index the codebase:</p>
|
|
|
|
| 82 |
--llm-model=llama3.1
|
| 83 |
</details>
|
| 84 |
|
| 85 |
+
<details>
|
| 86 |
<summary><strong>:cloud: Using external providers</strong></summary>
|
| 87 |
<p>To index the codebase:</p>
|
| 88 |
|
|
|
|
| 103 |
</details>
|
| 104 |
|
| 105 |
## Additional features
|
| 106 |
+
|
| 107 |
- **Control which files get indexed** based on their extension. You can whitelist or blacklist extensions by passing a file with one extension per line (in the format `.ext`):
|
| 108 |
+
- To only index a whitelist of files:
|
| 109 |
+
|
| 110 |
```
|
| 111 |
index ... --include=/path/to/extensions/file
|
| 112 |
```
|
| 113 |
+
|
| 114 |
+
- To index all code except a blacklist of files:
|
| 115 |
+
|
| 116 |
```
|
| 117 |
index ... --exclude=/path/to/extensions/file
|
| 118 |
```
|
| 119 |
+
|
| 120 |
- **Index open GitHub issues** (remember to `export GITHUB_TOKEN=...`):
|
| 121 |
+
- To index GitHub issues without comments:
|
| 122 |
+
|
| 123 |
```
|
| 124 |
index ... --index-issues
|
| 125 |
```
|
| 126 |
+
|
| 127 |
+
- To index GitHub issues with comments:
|
| 128 |
+
|
| 129 |
```
|
| 130 |
index ... --index-issues --index-issue-comments
|
| 131 |
```
|
| 132 |
+
|
| 133 |
+
- To index GitHub issues, but not the codebase:
|
| 134 |
+
|
| 135 |
```
|
| 136 |
index ... --index-issues --no-index-repo
|
| 137 |
```
|
|
|
|
| 168 |
|
| 169 |
We built the code purposefully modular so that you can plug in your desired embeddings, LLM and vector stores providers by simply implementing the relevant abstract classes.
|
| 170 |
|
| 171 |
+
Feel free to send feature requests to [founders@storia.ai](mailto:founders@storia.ai) or make a pull request!
|