Update README.md

README.md

@@ -1,3 +1,92 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+---
+
+
+# embedfile
+
+
+Experimental CLI tool for generating and searching text embeddings, built on 
+[llamafile](https://github.com/Mozilla-Ocho/llamafile),
+[`sqlite-vec`](https://github.com/asg017/sqlite-vec), 
+[`sqlite-lembed`](https://github.com/asg017/sqlite-lembed), 
+[the SQLite CLI](https://www.sqlite.org/cli.html), 
+and a few other SQLite extensions.
+
+
+| Model                                                                                                   | embedfile                                                                                                                                         | Size (f16 quant) |
+| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
+| [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | [`all-MiniLM-L6-v2.f16.embedfile`](https://huggingface.co/asg017/embedfile/resolve/main/all-MiniLM-L6-v2.f16.embedfile)                           | `56MB`           |
+| [mixedbread-ai/mxbai-embed-xsmall-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-xsmall-v1)       | [`mxbai-embed-xsmall-v1-f16.embedfile`](https://huggingface.co/asg017/embedfile/resolve/main/mxbai-embed-xsmall-v1-f16.embedfile)                 | `61MB`           |
+| [nomic-ai/nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5)                 | [`nomic-embed-text-v1.5.f16.embedfile`](https://huggingface.co/asg017/embedfile/resolve/main/nomic-embed-text-v1.5.f16.embedfile)                 | `273MB`          |
+| [snowflake-arctic-embed-m-v1.5](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5)         | [`snowflake-arctic-embed-m-v1.5-f16.embedfile`](https://huggingface.co/asg017/embedfile/resolve/main/snowflake-arctic-embed-m-v1.5-f16.embedfile) | `221MB`          |
+| -                                                                                                       | [`embedfile`](https://huggingface.co/asg017/embedfile/resolve/main/embedfile) (no embedded model)                                                 | `12MB`           |
+
+
+embedfiles run on Linux, Mac, and Windows computers in the same file, thanks to [cosmopolitan](https://github.com/jart/cosmopolitan). 
+You can embed data from CSVs, JSON, NDJSON, and txt files from the CLI, or "eject" to the `sqlite3` CLI at any time.
+
+
+Here's an example, using MixedBread's xsmall model:
+
+```
+$ wget https://huggingface.co/asg017/embedfile/resolve/main/mxbai-embed-xsmall-v1-f16.embedfile
+$ chmod u+x mxbai-embed-xsmall-v1-f16.embedfile 
+$ ./mxbai-embed-xsmall-v1-f16.embedfile --version
+embedfile 0.0.1-alpha.1, llamafile 0.8.16, SQLite 3.47.0, sqlite-vec=v0.1.6, sqlite-lembed=v0.0.1-alpha.8
+
+```
+
+This executable file already has `sqlite-vec`, `sqlite-lembed`, and the embeddings model pre-configured. Test that embeddings work with:
+
+
+```
+./mxbai-embed-xsmall-v1-f16.embedfile embed 'hello!'
+[-0.058174,0.043776,0.030660,...]
+```
+
+You can embed data from CSV, JSON, NDJSON, and .txt files and save the results to a SQLite database. Here we are embedding the `text` column in the `dbpedia.min.csv` file, outputting to a `dbpedia.db` database.
+
+```
+$ ./mxbai-embed-xsmall-v1-f16.embedfile import --embed text dbpedia.min.csv dbpedia.db
+INSERT INTO vec_items SELECT rowid, lembed("text") FROM temp.source;
+100%|████████████████████| 10000/10000 [02:00<00:00, 83/s]
+✔ dbpedia.min.csv imported into dbpedia.db, 10000 items
+```
+
+That was 10,000 rows with 820,604 tokens. I got 83 embeddings per second on my older 2019 Intel Macbook. On my M1 Mac Mini I get 173 embbedings/second, and I'm sure it's faster on newer macs. 
+
+Once indexed, you can search with the `search` command:
+
+```
+$ ./mxbai-embed-xsmall-v1-f16.embedfile search dbpedia.db 'global warming'
+3240 0.852299 Attribution of recent climate change is the effort to scientifically ascertain mechanisms ...
+6697 0.904844 The global warming controversy concerns the public debate over whether global warming is occurring, how ...
+...
+```
+
+
+At any point, if you want to "eject" and run SQL scripts yourself, the `sh` command will fire up the `sqlite3` CLI with all extensions and embeddings models pre-configured.
+
+```
+$ ./mxbai-embed-xsmall-v1-f16.embedfile sh
+SQLite version 3.47.0 2024-10-21 16:30:22
+Enter ".help" for usage hints.
+Connected to a transient in-memory database.
+Use ".open FILENAME" to reopen on a persistent database.
+sqlite> .mode qbox
+sqlite> select sqlite_version(), vec_version(), lembed_version();
+┌──────────────────┬───────────────┬──────────────────┐
+│ sqlite_version() │ vec_version() │ lembed_version() │
+├──���───────────────┼───────────────┼──────────────────┤
+│ '3.47.0'         │ 'v0.1.6'      │ 'v0.0.1-alpha.8' │
+└──────────────────┴───────────────┴──────────────────┘
+sqlite> select vec_to_json(vec_slice(lembed('hello!'), 0, 8)) as sample;
+┌──────────────────────────────────────────────────────────────┐
+│                            sample                            │
+├──────────────────────────────────────────────────────────────┤
+│ '[-0.058174,0.043776,0.030660,0.047412,-0.059377,-0.036267,0 │
+│ .038117,0.005184]'                                           │
+└──────────────────────────────────────────────────────────────┘
+```
+