ExeaLM / results /sweep /ft_ctrl.log
AvnehBhatia's picture
Research checkpoints of the 17M experiments (checkpoints/17m), all result files (results/), and an updated card: both memory skills at 17M, equal-compute results, a correction to the swarm-on-text claim
2315c01 verified
Raw History Blame Contribute Delete
9.57 kB
{"model": "exealm", "params": 18080796, "library_chunks": 483270, "eval_library_chunks": 53467, "train_articles": 14209, "eval_articles": 1578, "args": {"model": "exealm", "preset": "small", "search": 0, "search_every": 250, "search_until": 0.5, "search_mix": "text:3,wiki:1,relay:1,lookup:1", "wiki_oracle": 0.0, "init_ckpt": "results/full_small/exealm_skillsA_seed0_ckpt.pt", "root": "data/tokpack", "toks": "bpe32k", "train": "train_en_big,train_ja_big,train_code_big", "mix": "0.5,0.3,0.2", "p_kinds": "0.7,0.15,0.075,0.075,0", "warm_steps": 0, "warm_kinds": "0.2,0,0.4,0.4", "schedule": "", "facts": 1024, "grow": 16, "dense": 1, "relay_chunk": 8, "relay_facts": 4, "grow_every": 150, "d": 320, "slots": 10, "steps": 4000, "batch": 32, "seq": 512, "chunk": 64, "min_tokens": 1024, "lr": 0.001, "warmup": 200, "refresh_every": 1000, "eval_every": 2000, "eval_batches": 12, "save_every": 4000, "key_dim": 128, "lexical": 128, "gen_tokens": 60, "prompts": "The history of the city begins||Photosynthesis is the process by which||\u65e5\u672c\u306e\u9996\u90fd\u306f||def fibonacci(n):", "tokenizer": "results/tokenizer/bpe32k/tokenizer.json", "lib_on": "wiki", "board_chunk": 8, "pool": "mean", "tag": "ft_ctrl", "stop_at": 0, "seed": 0, "smoke": 0, "device": "auto", "out": "results/irl"}}
{"step": 2000, "secs": 1037.0, "tokens": 32768000, "train_loss": 3.5145, "batches": {"text": 1409, "wiki": 277, "relay": 153, "lookup": 161, "pair": 0}, "val_en": {"removed": 1.3737, "attached": 1.3742}, "val_ja_wiki": {"removed": 1.4952, "attached": 1.4954}, "val_ja_web": {"removed": 1.3772, "attached": 1.3786}, "val_code": {"removed": 1.709, "attached": 1.7092}, "swarm": {"shared": 1.3771, "cut": 1.3773, "board_gain": 0.0003}, "knowledge": {"removed": 5.063, "attached": 5.0976, "retrieval_gain": -0.0346, "same_article_hits": 0.2598}, "relay": {"shared": 0.9993, "cut": 0.0088}, "lookup": {"attached": 0.6232, "removed": 0.0135}}
{"step": 4000, "secs": 2145.6, "tokens": 65536000, "train_loss": 4.2056, "batches": {"text": 2814, "wiki": 581, "relay": 311, "lookup": 294, "pair": 0}, "val_en": {"removed": 1.3191, "attached": 1.3203}, "val_ja_wiki": {"removed": 1.4145, "attached": 1.4159}, "val_ja_web": {"removed": 1.3114, "attached": 1.3127}, "val_code": {"removed": 1.6188, "attached": 1.6238}, "swarm": {"shared": 1.3226, "cut": 1.323, "board_gain": 0.0004}, "knowledge": {"removed": 4.8482, "attached": 5.0719, "retrieval_gain": -0.2236, "same_article_hits": 0.2591}, "relay": {"shared": 1.0, "cut": 0.0133}, "lookup": {"attached": 0.9273, "removed": 0.0067}}
{"samples": {"free": [{"prompt": "The history of the city begins", "text": " in 1900, when the city came to the north with the city of Geskowo, and is the site of the city. It is known for its significance as the city of Geskowo.\n\nGovernment and politics\n\nGeskowo is in the"}, {"prompt": "Photosynthesis is the process by which", "text": " any of these functions are made to support them, as many as 10% of the time they become.\n\nMost modern development models are in this way are the ability to provide an alternative for the development of a new design, although this would be the case for the development of the original system."}, {"prompt": "日本の首都は", "text": "、南アメリカで起きた。\n\nアメリカで起きたのは1857年に、首都であるマグロアにある。\n\n1945年にポーグの都市であるリチャードに生まれる。\n\n人口\n\n人口\n\n政治\n\n政治\nリチャードは、ミララ"}, {"prompt": "def fibonacci(n):", "text": " sx.p.a.g. h. g. t.d. h. z. g. t.\nKathurik Kirri (Ratarikat) is the second name of the Lotu-chuku (Jewishish meaning)."}], "retrieval": [{"article_start": "Boonsboro is a town in Washington County, Maryland, United States, located at the foot of South Mountain. It nearly borders Frederick County and is proximate to the Antietam National Battlefield. The population was 3,336", "prompt": " body of the Town. In Boonsboro, from 1831 through 1939, Mayors (originally called Burgesses) were elected annually. From 1940 through 1975, they served two-year terms. Since 1976, Mayors have been chosen for four-year terms, except for the previous mayor, who had been in office from 1988 through 2016.\n\nBoonsboro's current Mayor is Howard W. Long.\n\nPrevious Mayors include:\n\n 1831–1835 Jonathan Shafer\n 1835–1836 David Brookhart\n 1836–1837 Lewis F", "with_library": ". W. H. Wins\n 3rd Lie: Dave Matt\n 3rd Lie: Bedford H. D. H. Bennett\n 3rd Lie: Richard H. Kicky\n 3rd Lie: Buck D. H. W. Bennett", "without_library": ". W. Whitman\n 1836–1843 James D. Whitman\n 1837–1838 William Brookhart\n 1838–1842 William Brookhart\n 1839–1845 William Brookhart\n 1839–1849 William Brookhart\n 1831–", "truth": "letcher\n 1837–1838 Joseph O'neal\n 1838–1839 Anthony McBride\n 1839–1840 Joseph Knox\n 1840–1841 Charles Perry\n 1841–1843 Jacob Smith\n 1843–1845 James Chambers\n"}, {"article_start": "A macrobiotic diet (or macrobiotics) is a fad diet based on ideas about types of food drawn from Zen Buddhism. The diet tries to balance the supposed yin and yang elements of food and cookware.", "prompt": " and nut butters, seasonings, sweeteners, fruits, and beverages may be enjoyed occasionally, two to three times per week. Other naturally-raised animal products may be included if needed during dietary transition or according to individual needs.\n\nKitchenware \nCooking utensils should be made from certain materials such as wood or glass, while some materials including plastic, copper, and non-stick coatings are to be avoided. Electric ovens should not be used.\n\nJapanese popularity and influence\nThe macrobiotic way of eating was developed and popularized by the Japanese. During", "with_library": " the mid-1920s and early decades of the Japanese economy, the use of a plastic, or plastic is often the most common use. In the United States, a plastic bag or metal mask, the use of a plastic bag or an electric mask, may be used as a way to reduce the", "without_library": " the early 20th century, when the use of a raw material to produce a variety of materials, the use of the raw material used to make a solid and solid material would be impossible. The traditional method of producing a variety of food, such as broccoli, kayak, chin", "truth": " the Edo period in Japan peasants had a diet based on staples of rice and soybeans. According to some macrobiotic advocates, a majority of the world population in the past ate a diet based primarily on grains, vegetables, and other plants. Because the macrobiotic diet was"}, {"article_start": "The BNC connector (initialism of \"Bayonet Neill–Concelman\") is a miniature quick connect/disconnect radio frequency connector used for coaxial cable. It is designed to maintain the same", "prompt": " not mate with standard interface connectors. The SHV connector is a high-voltage BNC variant that uses this reverse polarity configuration.\n\nSmaller versions of the BNC connector, called Mini BNC and High Density BNC (HD BNC), are manufactured by Amphenol. While retaining the electrical characteristics of the original specification, they have smaller footprints giving a higher packing density on circuit boards and equipment backplanes. These connectors have true 75 ohm impedance making them suitable for HD video applications.\n\nCompatibility\nThe different", "with_library": " CNC applications are called Mini BNC and Mini BNC. Mini BNC is an Amphenol. Mini BNC is a BNC and Mini BNC and Mini BNC is an Amphenol. Mini BNC is a BNC vari", "without_library": " systems of the DC and the BNC are:\n Amphenol is a non-magnetic (magnetic) (magnetic) or magnetic resonance (magnetic).\n Amphenol is a non-magnetic-borean (magnetic), a non-m", "truth": " versions are designed to mate with each other,\nand a 75 ohm and a 50 ohm BNC connector which both comply with the 2007 IEC standard, IEC 61169-8, will mate non-destructively. At least one manufacturer claims"}, {"article_start": "Salvation (from Latin: salvatio, from salva, 'safe, saved') is the state of being saved or protected from harm or a dire situation. In religion and theology, salvation generally refers to", "prompt": " of Jesus Christ of Latter-day Saints, the plan of salvation is God's plan to save, redeem, and exalt all humankind who chose, either in this life, or in the world of spirits of the dead, to accept the grace of Jesus Christ by exercising faith in Him, repenting of their sins, and by making and keeping sacred covenants (including baptism). Since the vast majority of God's children depart this life without that opportunity, Christ's gospel is preached to the unbelieving spirits in spirit prison (1 Peter 3: 19) so that they", "with_library": " do not receive the blessings of the dead, that the God of God is the Lord, and will not be the God of God. He is not all God, nor God, but God, and God and the Lord.\nIn the world of the earth, God is God, in the", "without_library": " do not receive the blessings of God's will. The God of God's will, and the will is revealed in Christ's life. He is said to have made a living, and the Lord will be a living.\nThe will of the dead are not so much God's will in the", "truth": " might be judged by the same standards as the living and live by following God in their spirit form (1 Peter 4: 6). If they accept Christ, sincerely repent of their sins, and accept ordinances done on their behalf, they can, by the grace of Christ"}]}}
WROTE results/irl/full_small_exealm_ft_ctrl_seed0.json