Hugging Face's logo

Spaces:
ASTERIZER
/
luna-300m-program
Running

App Files Community
luna-300m-program
File size: 6,828 Bytes
0944ada
 
 
 
 
 
9b01565
71fafcd
9b01565
 
71fafcd
9b01565
 
71fafcd
9b01565
 
 
 
71fafcd
9b01565
 
 
 
 
 
 
71fafcd
9b01565
 
 
 
 
 
71fafcd
9b01565
 
 
 
71fafcd
 
9b01565
 
 
0944ada
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>LUNA-300M Program</title>
<style>
  :root{--bg:#0b1020;--panel:#131a30;--panel2:#182139;--line:#26304d;--text:#e8ecf8;--muted:#9aa7c7;--accent:#8b97ad;--accent2:#c3cbd9;--green:#34d399;}
  *{box-sizing:border-box;margin:0;padding:0;}
  body{background:var(--bg);color:var(--text);font-family:system-ui,-apple-system,'Segoe UI',Roboto,sans-serif;line-height:1.55;}
  a{color:var(--accent2);text-decoration:none;} a:hover{text-decoration:underline;}
  .wrap{max-width:1080px;margin:0 auto;padding:32px 20px 64px;}
  header.hero{padding:40px 0 28px;border-bottom:1px solid var(--line);}
  .tag{display:inline-block;font-size:12px;letter-spacing:1.5px;text-transform:uppercase;color:var(--accent2);font-weight:700;}
  h1{font-size:34px;margin:8px 0 6px;} h1 span{color:var(--accent);}
  .sub{color:var(--muted);max-width:760px;}
  .links{margin-top:14px;} .links a{margin-right:14px;font-weight:600;}
  .cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:14px;margin:26px 0 8px;}
  .card{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:16px 18px;}
  .card .v{font-size:22px;font-weight:800;color:var(--accent);} .card .l{font-size:12px;color:var(--muted);margin-top:4px;text-transform:uppercase;letter-spacing:.5px;}
  .tabs{display:flex;flex-wrap:wrap;gap:8px;margin:28px 0 18px;}
  .tab{background:var(--panel);border:1px solid var(--line);color:var(--muted);padding:10px 18px;border-radius:999px;cursor:pointer;font-weight:600;font-size:14px;}
  .tab.active{background:var(--accent);color:#fff;border-color:var(--accent);}
  .panel{display:none;} .panel.active{display:block;}
  .grid2{display:grid;grid-template-columns:1fr 1fr;gap:16px;}
  @media(max-width:760px){.grid2{grid-template-columns:1fr;}}
  table{width:100%;border-collapse:collapse;background:var(--panel);border-radius:12px;overflow:hidden;font-size:14px;}
  th,td{text-align:left;padding:10px 14px;border-bottom:1px solid var(--line);}
  th{background:var(--panel2);color:var(--muted);font-size:12px;text-transform:uppercase;letter-spacing:.6px;}
  tr:last-child td{border-bottom:none;}
  .h2{font-size:20px;margin:22px 0 12px;}
  .note{color:var(--muted);font-size:13px;margin:6px 0 16px;}
  .filters{display:flex;gap:10px;margin:14px 0;flex-wrap:wrap;}
  select{background:var(--panel2);color:var(--text);border:1px solid var(--line);border-radius:8px;padding:9px 12px;font-size:14px;}
  .sample{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:16px 18px;margin-bottom:12px;}
  .sample h4{font-size:15px;margin-bottom:6px;}
  .sample .i{color:var(--muted);font-size:13px;margin-bottom:8px;white-space:pre-wrap;}
  .sample .o{font-size:14px;white-space:pre-wrap;}
  .meta{display:inline-block;background:var(--panel2);border:1px solid var(--line);color:var(--accent2);font-size:11px;border-radius:6px;padding:2px 8px;margin:2px 6px 2px 0;}
  .pill{display:inline-block;background:rgba(139,151,173,.12);color:var(--accent);border:1px solid rgba(139,151,173,.3);font-size:11px;border-radius:6px;padding:2px 8px;margin:2px 6px 2px 0;}
  .ok{color:var(--green);font-weight:700;}
  code{background:var(--panel2);padding:1px 5px;border-radius:5px;font-size:13px;}
  footer{margin-top:44px;padding-top:20px;border-top:1px solid var(--line);color:var(--muted);font-size:13px;}
</style>
</head>
<body>
<div class="wrap">
  <header class="hero">
    <div class="tag">ASTERIZER &middot; LUNA Model Family</div>
    <h1>LUNA-300M <span>Program</span></h1>
    <p class="sub">A ~303M-parameter, English-first causal language model — pretrained <b>from scratch</b> on the 4.5B-token <b>LUNA_PreTrain</b> corpus. The scaled-up sibling of <a href="https://huggingface.co/ASTERIZER/LUNA-100M">LUNA-100M</a>.</p>
    <div class="links">
      <a href="https://huggingface.co/ASTERIZER/LUNA-300M">Model &middot; LUNA-300M</a>
      <a href="https://huggingface.co/ASTERIZER/LUNA-100M">Model &middot; LUNA-100M</a>
      <a href="https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain">Dataset &middot; LUNA_PreTrain</a>
      <a href="https://huggingface.co/spaces/ASTERIZER/LUNA">Workspace</a>
      <a href="https://huggingface.co/collections/ASTERIZER/luna-300m-program-6a91195ab75671026f83ac94">Collection</a>
    </div>
  </header>

  <div class="cards">
    <div class="card"><div class="v">≈303M</div><div class="l">Parameters</div></div>
    <div class="card"><div class="v">31,052</div><div class="l">Pretrain steps</div></div>
    <div class="card"><div class="v">4.5B</div><div class="l">Tokens trained</div></div>
    <div class="card"><div class="v">1,024</div><div class="l">Context window</div></div>
  </div>

  <h2>Status</h2>
  <p class="note"><span class="ok">&#10003; TRAINED (pretraining complete)</span> — final fp32 weights available at <code>pretrained/final/lit_model.pth</code>. Instruction tuning (RAG + MCP SFT, mirroring LUNA-100M) is planned on top of this checkpoint.</p>

  <div class="grid2">
    <div>
      <h2>Architecture</h2>
      <table><thead><tr><th>Property</th><th>Value</th></tr></thead><tbody>
        <tr><td>Layers</td><td>20</td></tr>
        <tr><td>Hidden size</td><td>1024</td></tr>
        <tr><td>Heads</td><td>16</td></tr>
        <tr><td>Context</td><td>1,024 tokens</td></tr>
        <tr><td>Vocab</td><td>50,304 (Pythia-160m tokenizer)</td></tr>
        <tr><td>Precision</td><td>bf16 / fp16</td></tr>
      </tbody></table>
    </div>
    <div>
      <h2>Training</h2>
      <table><thead><tr><th>Property</th><th>Value</th></tr></thead><tbody>
        <tr><td>Corpus</td><td>LUNA_PreTrain (4,515,286,950 tokens)</td></tr>
        <tr><td>Optimizer</td><td>AdamW (wd 0.1, clip 1.0)</td></tr>
        <tr><td>LR</td><td>3e-4 &rarr; 3e-5 cosine</td></tr>
        <tr><td>Global batch</td><td>120</td></tr>
        <tr><td>Steps</td><td>31,052</td></tr>
      </tbody></table>
    </div>
  </div>

  <h2>Checkpoints</h2>
  <table><thead><tr><th>File</th><th>Description</th></tr></thead><tbody>
    <tr><td><code>pretrained/final/lit_model.pth</code></td><td>Final fp32 weights (1.2 GB) — recommended</td></tr>
    <tr><td><code>pretrained/step-00031052/lit_model.pth</code></td><td>Final logged step</td></tr>
    <tr><td><code>pretrained/step-00031000/lit_model.pth</code></td><td>Milestone checkpoint</td></tr>
    <tr><td><code>pretrained/latest.pt</code></td><td>Weights + optimizer state (3.6 GB)</td></tr>
  </tbody></table>

  <footer>
    Data &amp; models by <a href="https://huggingface.co/ASTERIZER">ASTERIZER</a> &middot; Static program page &middot; No analytics
  </footer>
</div>
</body>
</html>