ILM v2.0.1

README.md

@@ -2,7 +2,23 @@
 license: apache-2.0
 language:
 - en
+- de
 pipeline_tag: question-answering
+datasets:
+- Muennighoff/babi
+- marcov/scan_simple_promptsource
+- marcov/scan_length_promptsource
+- marcov/scan_addprim_jump_promptsource
+- marcov/scan_addprim_turn_left_promptsource
+- tasksource/proofwriter
+- rlhf-and-friends/proofwriter-source
+- tasksource/lonli
+- CLUTRR/v1
+- amueller/syntactic_transformations
+- Lots-of-LoRAs/task151_tomqa_find_location_easy_clean
+- Lots-of-LoRAs/task152_tomqa_find_location_easy_noise
+- Lots-of-LoRAs/task153_tomqa_find_location_hard_clean
+- wikimedia/wikipedia
 tags:
 - cpu
 - counting
@@ -17,12 +33,12 @@ model-index:
       type: question-answering
       name: Question answering over a story
     dataset:
-      name: bAbI, 20 tasks, 1,000 examples each
+      name: bAbI, 20 tasks, 1,000 examples each (18,903 of 19,399 read questions; 601 test questions with the story of a lesson are not read)
       type: Muennighoff/babi
       split: test
     metrics:
     - type: accuracy
-      value: 99.78
+      value: 97.44
       name: Accuracy
   - task:
       type: text2text-generation
@@ -78,7 +94,7 @@ model-index:
       split: test
     metrics:
     - type: accuracy
-      value: 100.0
+      value: 99.34
       name: Accuracy
   - task:
       type: question-answering
@@ -90,7 +106,7 @@ model-index:
       split: test
     metrics:
     - type: accuracy
-      value: 100.0
+      value: 99.70
       name: Accuracy
   - task:
       type: text-classification
@@ -100,65 +116,72 @@ model-index:
       type: tasksource/lonli
     metrics:
     - type: accuracy
-      value: 94.9
+      value: 98.18
       name: Accuracy
   - task:
       type: question-answering
-      name: Kinship over a story written by people
+      name: Kinship over told relations
     dataset:
-      name: CLUTRR, 2 to 10 steps
+      name: CLUTRR, graph form, 2 to 10 relations
       type: CLUTRR/v1
       config: gen_train234_test2to10
       split: test
     metrics:
     - type: accuracy
-      value: 72.3
+      value: 99.14
       name: Accuracy
+  - task:
+      type: text2text-generation
+      name: A sentence into a question or into the passive
+    dataset:
+      name: Syntactic transformations, test and generalization sets of 4 parts
+      type: amueller/syntactic_transformations
+    metrics:
+    - type: exact_match
+      value: 100
+      name: Whole sentence right
   - task:
       type: question-answering
-      name: Facts from encyclopedia text
+      name: Belief over a story
     dataset:
-      name: LAMA, T-REx part
-      type: facebook/lama
-      config: trex
+      name: ToM-QA, easy, 4 kinds (650 test questions)
+      type: Lots-of-LoRAs/task151_tomqa_find_location_easy_clean
+      split: test
     metrics:
-    - type: precision_at_1
-      value: 64.0
-      name: Precision at 1
+    - type: accuracy
+      value: 100
+      name: Accuracy
   - task:
       type: question-answering
-      name: Word problems
+      name: Belief over a story
     dataset:
-      name: SVAMP, standard setting (300 test problems)
-      type: ChilleD/SVAMP
+      name: ToM-QA, easy with noise, 4 kinds (650 test questions)
+      type: Lots-of-LoRAs/task152_tomqa_find_location_easy_noise
       split: test
     metrics:
     - type: accuracy
-      value: 38.3
+      value: 100
       name: Accuracy
   - task:
-      type: text-generation
-      name: Text from a list of facts
+      type: question-answering
+      name: Belief over a story
     dataset:
-      name: E2E NLG
-      type: tuetschek/e2e_nlg
+      name: ToM-QA, hard, 4 kinds (650 test questions)
+      type: Lots-of-LoRAs/task153_tomqa_find_location_hard_clean
       split: test
     metrics:
-    - type: bleu
-      value: 65.37
-      name: BLEU
-    - type: nist
-      value: 8.29
-      name: NIST
+    - type: accuracy
+      value: 100
+      name: Accuracy
 ---
 
 # 🧮 ILM: Integer Language Model
 
-⚡ **CPU only.** ILM is a language model that needs no GPU, to learn or to answer. On one desktop CPU it learns each benchmark below from its lessons in seconds, then answers its test questions.
+⚡ **CPU only.** ILM is a language model that needs no GPU, to learn or to answer. On one desktop CPU it learns each benchmark below from its lessons, then answers its test questions.
 
-🤔 **Why no GPU?** A neural network, such as a transformer, learns by changing millions of decimal weights. It repeats billions of multiplications over many passes through the data, and a GPU is built for that work. ILM has no weights. It stores whole-number counts: how often each pattern in a text came with each answer. Counting is the addition of whole numbers, and a normal CPU does it fast.
+🤔 **Why no GPU?** A neural network, such as a transformer, learns by changing millions of decimal weights. It repeats billions of multiplications over many passes through the data, and a GPU is built for that work. ILM has no weights. What it learns is lines of English: rules, question forms and facts. It keeps a line when the line answers the lessons right, and it finds this by counting. Counting is the addition of whole numbers, and a normal CPU does it fast.
 
-🧠 ILM learns from lessons: a short text, a question and the exact answer (for two tasks below, also the labels that the set gives). It keeps a pattern only when its counts show that the pattern is reliable. So what it learned is a set of rules that you can read, and it gives the same answers on every run and for every number of threads.
+🧠 ILM learns from lessons: a short text, a question and the exact answer. What it learned is a set of lines that you can read, and each answer names the lines that gave it, with a proof tree when steps prove it. ILM gives the same answers on every run and for every number of threads.
 
 - 💻 Source code, documentation and the commands that give each number below: https://github.com/Datamine-Crypto/ILM
 - 📦 Crate: https://crates.io/crates/datamine-ilm
@@ -175,24 +198,26 @@ model-index:
 | `LICENSE` | the Apache License 2.0 |
 | `THIRD_PARTY_NOTICES.md` | the licenses of the code of others that the programs hold |
 
-ILM has no weights to download: it learns each task from the lessons of that task when you run it. The program reads the descriptions of the sets and the list of sources from the GitHub repository, so run it from a clone of that repository.
+ILM has no weights to download: it learns each task from the lessons of that task when you run it. The program reads its library of texts, the descriptions of the sets and the list of sources from the GitHub repository, so run it from a clone of that repository.
 
 ## 📊 Results
 
-Each benchmark is read with frozen code on its test data: the official test split, or for LoNLI, LAMA and SVAMP the part that [docs/RESULTS.md](https://github.com/Datamine-Crypto/ILM/blob/main/docs/RESULTS.md) names. The runs use one desktop CPU (Intel Core i9-12900K, 24 threads, no GPU). The time is for the whole run, learning from all lessons and answering the test questions, as the best of three or more runs.
+Each benchmark is read with frozen code on its test data: the official test split, or for LoNLI the part that [docs/RESULTS.md](https://github.com/Datamine-Crypto/ILM/blob/main/docs/RESULTS.md) names. The runs use one desktop CPU (Intel Core i9-12900K, 24 threads, no GPU). The time is for the whole run, learning from all lessons and answering the test questions, as the best of three or more runs.
 
 | Benchmark | ILM | Published, for comparison | Time |
 |---|---|---|---|
-| bAbI, 20 tasks, 1,000 examples each | 20 of 20 tasks pass, 99.78% right | best published learner from answers only: 3 tasks fail (Universal Transformer) | 5.9 s |
-| SCAN, 4 splits | 100% on every split | Transformer: length 0.0, add jump 1.0 | 0.8 s |
-| ProofWriter, depth 5 | 100% (40,063 of 40,063; closed depth 5 also learns from the formal forms of the set) | ProofWriter, T5-11B: 99.6 to 99.7 | 4.0 s |
-| LoNLI, 17 groups of logic in NLI | 94.9% on its sealed part (15 groups; 10 of 16 learned) | tuned on MNLI, not on LoNLI: 59 to 71; RoBERTa-large tuned on 2% of LoNLI: near 100 | 0.7 s |
-| CLUTRR, 2 to 10 steps | 72.3% (learns from the step labels of the set) | GPT-4o with a solver: 67.7; Faithful CoT with votes: 71.9 | 15.9 s |
-| LAMA T-REx | 64.0% | BERT-large: 32.3; MegatronLM 11B with trained prompts, on a subset: 64.2 | 11.6 s |
-| SVAMP | 38.3% | Graph2Tree: 36.5; with RoBERTa: 43.8; GPT-3 with chain of thought: 68.9 | 27.6 s |
-| E2E NLG | BLEU 65.37 | TGen: 65.93; GPT-2 large with LoRA: 70.4 | 9.8 s |
+| bAbI, 20 tasks, 1,000 examples each | 19 of 20 tasks pass, 97.44% right (18,903 of 19,399; 601 test questions with the story of a lesson are not read) | the best published learner from answers only that we found: 3 tasks fail (Universal Transformer) | 14.0 s |
+| SCAN, 4 splits | 100% (17,016 of 17,016) | Transformer: length 0.0, add jump 1.0 | 1.5 s |
+| ProofWriter, depth 5, open and closed form | 99.52% (39,870 of 40,063) | ProofWriter (T5-11B): 99.6 to 99.7 | 0.4 s |
+| LoNLI, 17 groups of logic in NLI | 98.18% on its sealed part (15 groups) | tuned on MNLI, not on LoNLI: 59 to 71; RoBERTa-large tuned on 2% of LoNLI: near 100 | 2.5 s |
+| CLUTRR, graph form, 2 to 10 steps | 99.14% (1,036 of 1,045) | trained on 2 to 4 steps, read at 10 steps: Transformer 70, NCRL 97, R5 98 | 1.4 s |
+| Sentence transformations: question and passive, English and German | 100% whole sentences right (80,000 of 80,000) | Transformer from scratch: whole sentences near 0% on the new builds | 1.6 s |
+| PCFG SET | 100% (31,229 of 31,229) | Transformer: productivity 50, systematicity 72 | 8.4 s |
+| ToM-QA, 12 kinds | 100% (1,950 of 1,950) | best published for each form: 88.6 to 100 | 2.1 s |
+
+The papers give some of these as fractions; they are shown here as percentages.
 
-The settings, the published numbers with their sources, and notes on each benchmark are in [docs/RESULTS.md](https://github.com/Datamine-Crypto/ILM/blob/main/docs/RESULTS.md) of the GitHub repository.
+The settings, the counts, the published numbers with their sources, how ILM stands against transformers with the evidence for each claim, and notes on each benchmark are in [docs/RESULTS.md](https://github.com/Datamine-Crypto/ILM/blob/main/docs/RESULTS.md) of the GitHub repository.
 
 ## 🚀 How to use
 
@@ -208,15 +233,15 @@ chmod +x ./ilm
 ./ilm measure cases/final/src_babi
 ```
 
-On macOS, take `ilm-macos-arm64` in the same way, and also run `xattr -d com.apple.quarantine ./ilm`, because the program is not signed with an Apple developer certificate. On an Intel Mac, build the program from source. On Windows, copy `ilm-windows-x86_64.exe` into the clone as `ilm.exe` and run `.\ilm.exe fetch data/sources.txt sets=src_babi`, and so on. To build the program yourself in place of a download: `cargo build --release` (Rust 1.90 or later and a C compiler), then call `./target/release/ilm`. `fetch` gets the raw files of a set from the Hugging Face hub or GitHub at a fixed revision and checks their SHA-256. `export` writes a case file for each kind of the set. `measure` learns each kind from its lessons and reads its test cases. [docs/REPRODUCE.md](https://github.com/Datamine-Crypto/ILM/blob/main/docs/REPRODUCE.md) gives the commands for every benchmark.
+On macOS, take `ilm-macos-arm64` in the same way, and also run `xattr -d com.apple.quarantine ./ilm`, because the program is not signed with an Apple developer certificate. On an Intel Mac, build the program from source. On Windows, copy `ilm-windows-x86_64.exe` into the clone as `ilm.exe` and run `.\ilm.exe fetch data/sources.txt sets=src_babi`, and so on. To build the program yourself in place of a download: `cargo build --release` (Rust 1.97 or later and a C compiler), then call `./target/release/ilm`. `fetch` gets the raw files of a set from the Hugging Face hub or GitHub at a fixed revision and checks their SHA-256. `export` writes a case file for each kind of the set. `measure` learns each kind from its lessons and reads its test cases. `prove` prints the answer to a story and a question with its proof tree. [docs/REPRODUCE.md](https://github.com/Datamine-Crypto/ILM/blob/main/docs/REPRODUCE.md) gives the commands for every benchmark.
 
 ## ⚙️ How it works
 
-ILM has seventeen tools, each named for its way of reading: patterns, state, amount, order, link, rule, parts, together, list, requests, frames, table, arrange, graph, grid, space and ranks. For each kind of task, every tool learns from two thirds of the lessons and reads the last third. The tool with the most right answers learns again from all lessons. No person chooses a tool, and no setting changes between tasks. Beside the tools, ILM has a fact reader for encyclopedia text, a question reader, a writer that turns given facts into a text, and one command (`ilm ask`) that routes each request and prints a trace of its steps.
+ILM has one tool, the explain tool, and a library of texts in plain English: rules, told facts, question forms with their answers, and lines that write a sentence again. The engine chains the rules over the told sentences of a case and answers with the first answer whose conditions hold. For each kind of task, the tool learns lines of the library from the lessons: the wording of the questions, rules discovered from the lessons alone, the ways of joining of a language of commands, and the rewrites of a sentence into another build. The facts step reads facts of the library from plain text that people wrote (the capitals of the library come from a plain-text copy of the English Wikipedia). The tool tries its ways on two thirds of the lessons and judges them on the last third; no person chooses a way for a kind, and no setting changes between tasks. Each answer is marked proof, default, guess or none, and `ilm prove` prints its proof tree.
 
 ## 🚧 Limits
 
-ILM is not a general chat model. It learns one kind of task at a time from lessons of that kind, and each lesson needs the exact answer. On a board of 25 public sets (240 kinds in 112 tasks), it learns 61 tasks: every kind of the task at 95% right or more on held cases (90% where people wrote the text). On SVAMP and E2E, pre-trained transformers are ahead.
+ILM is not a general chat model. It learns one kind of task at a time from lessons of that kind, and each lesson needs the exact answer. On a board of 25 public sets (240 kinds in 112 tasks), it learns 79 tasks: every kind of the task at 95% right or more on held cases (90% where people wrote the text). It learns no task whose text people wrote: the stories of CLUTRR, the word problems (MAWPS, ASDiv, SVAMP), the descriptions of regular expressions of NL-RX, the rules in the words of people of ParaRules, and the intents and slots of ATIS and SNIPS. On these, pre-trained transformers lead. It gets no result on StepGame, SpartQA, FLD and BoardgameQA. On bAbI, task 14 (time) fails on its test split, 519 of 1,000 right (51.9%), although ILM reads its development data at 100%: learned from 943 or more of its lessons, it reads "before" in the order of the story, and the stories of that task tell the times out of order. On ProofWriter, rules of the library read the told rules: that number measures the engine, not rules that ILM learned, and it is level with ProofWriter (T5-11B), not ahead. The ToM-QA number is not a read of unseen cases: the board read that test split during development (shares only, never a wrong case). During development the board read 527 of the 1,045 CLUTRR graph test graphs (shares only, never a wrong case).
 
 ## 📜 License
 

SHA256SUMS.txt

@@ -1,3 +1,3 @@
-43d58e15f5f533b77c83d5161248c401b0c83c9da344e59047e40094d5b9a2eb  ilm-linux-x86_64
-45b6f3adf0a996eca86cce81a06ca4bb53b88b7336e1c88a864c61b7197a2f73  ilm-macos-arm64
-951195d2088a1c9e50d6542b118f910565a55239577b2292fda1b58c0a5bb3cd  ilm-windows-x86_64.exe
+c7ed927a00904f1779fff3e9af54b43fb4d26e83f764b838ef6d2c2d6cc6e3e8  ilm-linux-x86_64
+3d131556c3a5691d350444717f8ee98f10e5caf22a6e286c80146d5eb7d2d142  ilm-macos-arm64
+ca6cf46adf22921b0ee4eae2a3edfabdfedb5d13edfd0760018fc29790b7e775  ilm-windows-x86_64.exe

THIRD_PARTY_NOTICES.md

@@ -123,7 +123,7 @@ SOFTWARE.
 
 ## Unicode Character Database
 
-The tables of the kinds of signs, of decimal digits and of the decompositions of letters in crates/ilm-core/src/know/letters.rs, crates/ilm-core/src/know/signs.rs, crates/ilm-core/src/write/signs.rs and crates/ilm-data/src/sets/textmatch.rs come from the Unicode Character Database.
+The tables of the kinds of signs, of decimal digits and of the decompositions of letters in crates/ilm-core/src/base/signs.rs, crates/ilm-data/src/sets/knowledge/letters.rs and crates/ilm-data/src/sets/textmatch.rs come from the Unicode Character Database.
 
 ```
 UNICODE LICENSE V3
@@ -166,3 +166,41 @@ not be used in advertising or otherwise to promote the sale, use or other
 dealings in these Data Files or Software without prior written
 authorization of the copyright holder.
 ```
+
+## WordNet 3.0
+
+The repository also holds data of others that the program reads from a clone and does not hold: the pairs of words in data/explain/knowledge/antonyms.txt below its note on WordNet come from WordNet 3.0 (Princeton University). The license is also in data/explain/knowledge/WORDNET_LICENSE.
+
+```
+WordNet Release 3.0
+
+This software and database is being provided to you, the LICENSEE, by
+Princeton University under the following license.  By obtaining, using
+and/or copying this software and database, you agree that you have
+read, understood, and will comply with these terms and conditions.:
+
+Permission to use, copy, modify and distribute this software and
+database and its documentation for any purpose and without fee or
+royalty is hereby granted, provided that you agree to comply with
+the following copyright notice and statements, including the disclaimer,
+and that the same appear on ALL copies of the software, database and
+documentation, including modifications that you make for internal
+use or for distribution.
+
+WordNet 3.0 Copyright 2006 by Princeton University.  All rights reserved.
+
+THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" AND PRINCETON
+UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
+IMPLIED.  BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON
+UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES OF MERCHANT-
+ABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE
+OF THE LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL NOT
+INFRINGE ANY THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR
+OTHER RIGHTS.
+
+The name of Princeton University or Princeton may not be used in
+advertising or publicity pertaining to distribution of the software
+and/or database.  Title to copyright in this software, database and
+any associated documentation shall at all times remain with
+Princeton University and LICENSEE agrees to preserve same.
+```

ilm-linux-x86_64

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:43d58e15f5f533b77c83d5161248c401b0c83c9da344e59047e40094d5b9a2eb
-size 12485128
+oid sha256:c7ed927a00904f1779fff3e9af54b43fb4d26e83f764b838ef6d2c2d6cc6e3e8
+size 9076064

ilm-macos-arm64

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:45b6f3adf0a996eca86cce81a06ca4bb53b88b7336e1c88a864c61b7197a2f73
-size 10326448
+oid sha256:3d131556c3a5691d350444717f8ee98f10e5caf22a6e286c80146d5eb7d2d142
+size 7437552

ilm-windows-x86_64.exe

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:951195d2088a1c9e50d6542b118f910565a55239577b2292fda1b58c0a5bb3cd
-size 12482048
+oid sha256:ca6cf46adf22921b0ee4eae2a3edfabdfedb5d13edfd0760018fc29790b7e775
+size 9210880