[{"content": "# Ablation Panel Guide\n\n## What Is Ablation?\n\nAblation is an experiment where you **remove (disable) specific attention heads** from the model and observe how the output changes. If removing a head causes the prediction to change significantly, that head was important for this particular input. If the prediction barely changes, the head was either redundant or not relevant to this context.\n\nThe term comes from neuroscience, where \"ablation\" means removing part of the brain to study its function.\n\n## How to Use the Ablation Panel\n\nThe ablation panel is found in the **Investigation Panel** at the bottom of the dashboard, under the \"Ablation\" tab.\n\n### Step-by-Step\n\n1. **Run an analysis first**: You need to have clicked \"Analyze\" on a prompt before ablation is available.\n2. **Select a generation for comparison**: Click \"Select for Comparison\" on one of the generated sequences. This gives the ablation experiment a full generation to compare against.\n3. **Choose a head to ablate**: Use the **Layer** and **Head** dropdowns to pick a specific attention head (e.g., Layer 0, Head 3). Click the **+** button to add it.\n4. **Add more heads (optional)**: You can add multiple heads from different layers. Each appears as a chip (e.g., \"L0-H3\") with an \u00d7 button to remove it.\n5. **Run the experiment**: Click \"Run Ablation Experiment.\"\n6. **View results**: The panel shows a side-by-side comparison of the original vs. ablated generation, plus the probability change for the immediate next token.\n\n### Understanding the Results", "source_file": "ablation_panel_guide.md", "chunk_index": 0, "embedding": [0.02053725, 0.03211044, 0.01849644, 0.02764133, -0.01125028, 0.003703809, -0.022875138, 0.025109695, 0.019658927, 0.034022085, 0.027486332, -0.05450767, -0.002268462, 0.011999438, 0.026440095, -0.006671377, 0.052311864, -0.0054862876, 0.02327555, 0.087470606, -0.015861474, 0.011650692, -0.0069167907, 0.03404792, 0.0076659485, -0.018380193, 0.05920928, 0.017411456, 0.03908536, 0.010907993, 0.0077111563, -0.010352583, -0.026130099, -0.0013828737, -0.06349756, -0.037587043, 0.02002059, 0.009351552, -0.028106324, 0.010740078, 0.016081054, -0.036321227, -0.02335305, 0.05853762, -0.016068138, 0.033143762, 0.01813478, -0.0038168286, -0.016752712, 0.034099586, -0.009390302, 0.05605765, -0.022371395, -0.029992135, -0.019142266, 0.04724859, 0.019297265, 0.004359322, 0.026943838, -0.016959377, -0.01420816, 0.042495318, 0.033169597, 0.040170345, -0.017049793, -0.028571319, -0.037277047, 0.03603706, 0.010501122, -0.024748033, 0.041152, -0.015822724, -0.0066326275, 0.023701794, -0.030224632, -0.026000934, 0.0028448612, -0.03133545, -0.03337626, -0.012070478, 0.049160235, 0.028545486, 0.030431297, -0.020679332, -0.0055702445, -0.039266188, -0.057297636, -0.00577368, 0.0039233896, -0.041203666, -0.0128519265, 0.03712205, -0.029217144, 0.038827028, -0.011670066, 0.04492362, -0.0038878694, -0.010313833, -0.0019665386, 0.042701982, 0.030612128, -0.0040913043, -0.019336015, -0.04544028, 0.0041688033, -0.026117181, -0.051304374, 0.0119477715, -0.016093971, 0.024773866, -0.0047597336, -0.03355709, -0.043580305, 0.03174878, -0.008092193, -0.009035098, -0.036398724, -0.007472201, 0.023766378, -0.026414262, -0.067217514, 0.0066326275, 0.030302132, -0.00066035654, -0.012238393, -0.010107169, 0.028364655, -0.030896291, 0.01261943, -0.0034809993, 0.026091348, 0.030431297, -0.03893036, -0.078945704, -0.02468345, 0.034435414, 0.0062451325, 0.030353798, -0.07884237, 0.02138974, 0.015887307, 0.0232368, -0.011431111, 0.00104543, -0.06272257, 0.013768999, 0.051046047, 0.054817665, -0.007975944, 0.05853762, -0.0009477489, 0.03198128, 0.005579932, -0.03851703, -0.023727627, -0.039550353, -0.005221499, -0.006871583, 0.01191548, -0.08767727, 0.037018716, 0.0010422008, -0.0034616245, -0.017824784, -0.042211153, 0.045646943, -0.012684013, 0.018083114, -0.030560462, 0.00984238, -0.03725121, -0.027176335, -0.0025542397, -0.010597996, -0.0034874575, 0.017385622, -0.059415944, 0.0008476459, -0.017514788, -0.04851441, -0.0062451325, 0.007833863, -0.05357768, 0.03944702, -0.028312989, -0.036605388, -0.008886559, -0.017915199, -0.06473754, -0.040945336, 0.03185211, 0.0056735766, 0.010488206, 0.019516846, -0.01954268, -0.0006865932, 0.053112686, 0.010417165, 0.025587605, -0.024179706, 0.023030136, 0.014879819, 0.01480232, 0.018238112, 0.019374764, 0.031542115, -0.015590227, 0.013090882, 0.0072138705, -0.032394607, 0.015977722, -0.059829272, 0.017385622, -0.007833863, -0.035055406, -0.037716206, -0.03670872, 0.048307747, 0.03304043, 0.020408085, 0.01757937, -0.017321039, 0.040015347, 0.06050093, -0.026375512, 0.026233431, 0.039653685, -0.013187756, 0.004207553, -0.034022085, 0.009073848, 0.032885432, -0.001343317, 0.042960312, -0.0041914075, -0.067785844, 0.0004290703, -0.019284349, 0.031774614, -0.044536125, 0.020111006, 0.0072590783, 0.03603706, 0.0010979034, 0.028803816, 0.008034069, 0.0071622045, -0.094755515, 0.016959377, 0.050761882, 0.05450767, 0.02960464, -0.023779294, 0.0012206101, 0.0006361381, 0.022616807, -0.00725262, -0.031206287, -0.026633842, -0.035856232, -0.02083433, -0.0026010622, -0.033247095, 0.008150318, -0.03815537, 0.004485258, 0.003497145, 0.024270121, -0.06752751, -0.027718829, -0.02353388, -0.013588168, 0.013459003, 0.0023637211, -0.03353126, 0.007129913, 0.010455915, -0.004230157, -0.04004118, 0.0014902423, -0.010042586, -0.015486895, 0.014634405, 0.0030176195, -0.026918005, 0.008415106, -0.01946518, -0.005825346, -0.0009735819, 0.006190237, 0.0028367883, -0.037277047, 0.092740536, 0.0055411826, -0.024179706, 0.0007035461, 0.008169692, 0.01480232, -0.027279668, 0.031645447, -0.0044239047, -0.02420554, 0.054146007, -0.02349513, 0.030043801, 0.0005755919, 0.016894795, 0.00973259, -0.014492324, 0.024360538, -0.0031354828, -0.00425599, -0.00018587665, -0.013768999, -0.030147133, -0.008434481, 0.024903031, -0.022733057, -0.017527703, -0.028080491, -0.025432607, -0.07284912, -0.0022813785, 0.011527984, -0.008763852, 0.025329275, -0.009157805, 0.01099195, 0.012625888, -0.005877012, 0.059364278, -0.016223136, -0.0018583628, -0.0044013006, -0.017811866, 0.013936914, -0.040351175, 0.037302878, -0.012219018, 0.01676563, -0.018625606, 0.020821413, -0.04688693, -0.0074463678, -0.023030136, 0.036114562, -0.01054633, -0.010966117, 0.045827776, 0.013129632, -0.023340132, 0.01716604, -0.036941215, -0.037509542, 0.02908798, 0.075587414, -0.008208442, -0.028571319, 0.039472852, -0.007149288, -0.010378416, 0.028803816, 0.002357263, 0.03105129, -0.00043351034, -0.014195244, -0.036837883, 0.023792211, 0.022113064, -0.020227253, 0.023210967, -0.043580305, 0.003778079, -0.039292023, -0.06923249, -0.026065515, 0.0015007369, -0.008408648, -0.025820103, 0.007950111, 0.007556158, -0.026143014, -0.02301722, -0.06463421, 0.05383601, 0.0148927355, -0.016830212, 0.0061450293, -0.06143092, 0.04218532, 0.053009354, -0.012419224, 0.017088542, -0.0018196133, -0.011114656, 0.055230994, -0.02823549, 0.051020212, 0.043916132, 0.011553817, -0.009054474, -0.008847809, -0.022448894, -0.026168847, 0.023391798, 0.0104107065, -0.0103848735, 0.009080307, 0.032472104, 0.024050541, 0.062309243, -0.008931766, 0.03149045, -0.0028884544, 0.02157057, -0.013717333, 0.032213774, 0.055592656, 0.029165478, -0.0077176145, -0.00377485, -0.046680264, 0.003305012, -0.029397976, -0.00083230756, -0.007129913, -0.0021635154, 0.0070653306, 0.04141033, -0.052156866, -0.0074140765, 0.037767872, 0.024166789, -0.036217894, -0.032730434, -0.019891424, 0.005799513, 0.06308423, -0.06447922, 0.015590227, 0.023740545, 0.0019035706, 0.018276861, 0.0010462373, -0.08059902, -0.0039847433, -0.02361138, -0.05357768, -0.004100992, -0.03911119, -0.017605202, -0.0027544457, -0.029914636, 0.07388243, 0.013730249, -0.07496742, 0.059932604, 0.0050051473, 0.062154245, -0.0017840929, 0.037432045, -0.033350427, -0.0032113672, 0.06995582, -0.041875325, 0.036579557, 0.018289778, -0.013446086, -0.007556158, 0.0038588073, 0.002815799, -0.004362551, 0.009144889, -0.011121115, -0.03893036, -0.03141295, 0.016236052, -0.017643953, 0.019374764, -0.057297636, 0.006610024, -0.004381926, 0.028287156, 0.006309715, 0.03918869, 0.06887083, -0.006187008, -0.014117745, 0.006723043, -0.008046986, 0.0018809667, -0.0164298, 0.014104828, -0.013730249, 0.0493669, -0.041126166, -0.035701234, -0.008867184, 0.01998184, 0.005602536, 0.047300257, 0.024877198, 0.009196554, 0.040842004, 0.010662579, -0.020898912, 0.0301213, 0.026840506, -0.008279483, 0.048385244, -0.014673155, -0.010236334, 0.011140489, -0.005805971, -0.014750654, 0.011928396, 0.010468831, -0.021660987, 0.025833018, 0.007782197, -0.013768999, -0.015086483, -0.021790152, 0.023185134, 0.034616247, 0.0111017395, -0.024515536, -0.027357167, -0.05647098, -0.005757534, -0.00069628056, -0.050296888, -0.012503182, 0.042211153, -0.033867087, -0.036398724, 0.018251028, 0.022552226, -0.050012726, -0.0071169967, 0.0232368, 0.012025271, -0.004966398, -0.022229312, 0.016597714, -0.0035294362, 0.041100334, -0.023585547, 0.02250056, 0.008783227, -0.010765911, -0.029191311, 0.018083114, -0.025225943, -0.027253835, 0.021751402, -0.028984647, -0.045130283, 0.0070136646, -0.030508796, 0.0045853606, -0.027486332, -0.016068138, -0.042572815, 0.0038975568, 0.022345562, -0.015667725, 0.0026398117, -0.0164298, -0.008137401, 0.0013957903, -0.010727162, 0.013239422, -0.023120552, 0.026918005, 0.014324409, -0.0016379749, 0.010223418, -0.0153448135, -0.0133556705, -0.019478096, -0.020072255, -0.045311116, 0.03381542, -0.003248502, -0.042391986, -0.0018680502, -0.000648651, -0.042831145, -0.016119804, 0.006923249, -0.030224632, 0.03725121, 0.038982026, -0.0075884494, 0.019245598, 0.024140956, 0.024334705, 0.014014413, 0.008815518, 0.027537998, 0.00016882282, -0.0045789024, 0.048591908, -0.011521527, -0.007355952, -0.035649568, 0.013601084, 0.013536502, 0.0077628223, 0.015964806, -0.012793803, -0.05605765, 0.0016856045, -0.016403968, 0.04448446, 0.013872331, 0.018858103, 0.03851703, -0.014608572, 0.0015620904, -0.00041494286, -0.030302132, 0.0138335815, 0.02272014, 0.02031767, -0.008957599, -0.00517952, 0.003978285, 0.0021974212, -0.0689225, 0.035907898, 0.02264264, 0.018806437, -0.0012189955, 0.03208461, -0.0013352442, -0.016520215, 0.011967146, 0.01339442, -0.0021554425, -0.0012480577, 0.017346872, -0.012593597, -0.011121115, 0.005221499, 0.024063457, -0.02002059, -0.009790714, -0.017941032, 0.009571133, -0.032885432, 0.047429424, 0.013510669, 0.024450952, 0.006768251, 9.3039234e-05, -0.0020133608, 0.0028254865, 0.0054539964, -0.04236615, 0.034926243, 0.003038609, -0.030612128, 0.013562335, 0.009823006, -0.0045240074, 0.0028287156, 0.0037102674, -4.7982816e-05, -0.03725121, 0.009080307, 0.007575533, 0.0046725473, -0.02138974, 0.05672931, 0.011883189, -0.0028400174, -0.023792211, 0.03497791, -0.013484836, 0.024709282, -0.031516284, 0.02841632, -0.0056154523, 0.01247089, 0.01032675, -0.0069426238, -0.003577873, -0.011786315, 0.009319262, -0.0219064, -0.015551478, -0.012057561, -0.020730998, 0.025561772, -0.022319729, 0.03908536, -0.0044884873, -0.08721227, -0.012038187, -0.007375327, 0.017450204, 0.0020036735, -0.015835641, 0.005608994, 0.011779857, 0.049263567, 0.015641892, -0.02994047, -0.032730434, 0.018586857, -0.014711904, 0.020924745, -0.030276299, 0.051304374, -0.026995504, 0.028803816, -0.019568512, -0.047300257, -0.04696443, 0.041461997, -0.047584422, -0.0013513898, 0.049392734, -0.021712653, -0.0044239047, 0.008221358, -0.022745972, -0.0051762913, 0.0153706465, 0.0011406892, -0.0050342097, -0.04184949, 0.008776768, 0.035701234, 0.0022668475, -0.023262633, 0.013090882, -0.022745972, 0.026763007, 0.02242306, 0.034022085, 0.010462373, 0.0031128789, 0.02638843, -0.009196554, -0.0042172405, -0.053371016, -0.016817296, 0.0034035002, 0.010417165, -0.010933826, -0.00012744962, 0.017191874, 0.03037963, -0.009965087, -0.022526393, 0.01813478, 0.009203013, 0.013820665, -0.010036128, 0.02686634, 0.007963028, -0.028338822, -0.029785471, 0.03105129, -0.0013336296, 0.021105576, 0.03446125, -0.006490546, 0.010494664, 0.037328713, 0.018225195, 0.015008984, -0.011482777, 0.04750692, -0.024450952, -0.010688412, 0.026918005, -0.016378134, 0.0047532753, -0.042727813, 0.001761489, -0.0101975845, -0.03397042, -0.031387117, -0.014427741, -0.035313737, -0.019633094, -0.008654062, 0.025161361, 0.012457973, -0.018754771, 0.0124515155, 0.01450524, 0.0071363715, -0.07419243, -0.0014967006, -0.01013946, -0.018121863, -0.036088727, -0.016946461, 0.02815799, 0.0036198518, 0.03319543, 0.0041688033, 0.0010801431, -0.031542115, 0.008847809, -0.012457973, 0.027925493, -0.03327293, 0.010507581, -0.0133815035, 0.03996368, -0.020756831, -0.040067013, 0.01624897, -0.020162672, -0.0014482636, -0.044794455, 0.02756383, -0.03867203, 0.025251776, 0.01058508, 0.03363459, -0.025393859, -0.0038071412, -0.03815537, 0.018586857, -0.02105391, 0.026556343, 0.004937336, 0.0012932655, 3.612586e-05, 0.0014175869, 0.014673155, -0.013471919, -0.0035197488, -0.012625888, 0.023469297, -0.004178491, -0.01728229, 0.013446086, -0.014053162, 0.019839758, 0.0108886175, -0.011682983, 0.008137401, 0.015473979, 0.016701046, -0.036760386, 0.005935136, 0.011586109, 0.003282408, 0.018483525, 0.04458779, 0.05424934, 0.035727065, -0.002150599, -0.019658927, 0.0108886175, 0.010985491, 0.00034067294, -0.05024522, -0.031619616, -0.019865591, 0.03378959, -0.024063457, -0.007782197, 0.019361848, 0.017114375, 0.03859453, 0.028312989, 0.0143631585, -0.014711904, 0.018780604, -0.027202168, -0.04022201, 0.010830494, -0.016403968, 0.030792959, -0.016520215, 0.014595656, 0.013730249, 0.054197673, 0.01535773, -0.011837981, 0.018548107, 0.023185134, -0.037716206, -0.012302975, 0.012580681, 0.012374016, 0.043761134, 0.0013925611, 0.00078629254, -0.007627199, 0.00065833837, -0.015771057, -0.016533133, -0.0021958067, -0.005111709, -0.028261323, -0.0206535, -0.007246162, 0.045207784, 0.045491945, -0.039395355, 0.010300917, -0.015977722, 0.011818606, 0.050451886, 0.008983432, 0.0004783145, -0.010778828, 0.006723043, 0.016610632, -0.0023524193, -0.013265255, 0.032420438, 0.025703853, -0.011340695, -0.021932233, 0.002953037, 0.033763755, 0.016068138, -0.0023927833, -0.03918869, -0.0016460477, -0.0026801757, -0.008428022, -0.01039779, 0.011967146, -0.022113064, 0.00973259, -0.0206535, -0.008279483, -0.0019600803, -0.005999719, -0.06856083, -0.014414825, 0.0014539146, -0.012561305, -0.028028825, 0.027847994, 0.0178377, 0.04339947, -0.016210219, -0.003064442, 0.005835033, 0.033350427, 0.03500374, -0.019865591, -0.041126166, 0.039524518, 0.009093223, 0.0016581569, 0.027331334, 0.0046111937, -0.00035722222, 0.025845936, -0.019710593, 0.0055411826, -0.0019778404, 0.006903874, -0.017450204, 0.0068844995, 0.00836344, 0.025561772, 0.040273678, 0.031387117, -0.024450952, -0.02401179, -0.038827028, 0.009648632, -0.0016274801, 0.025639271, -0.024967613, -0.026181765, -0.031154621, -0.008873642, 0.015125233, -0.012748595, 0.009971545, -0.022913888, 0.016326468, 0.016442716, 0.0026801757, -0.007704698, -0.020149754, 0.0053829555, -0.0219839, 0.0191681, -0.051562704, -0.032678768, -0.032187942, 0.056935973, 0.015667725, -0.013768999, 0.017863533, 0.029837137, 0.009952171, -0.0011067834, -0.024748033, 0.0013965976, -0.03490041, 0.009306345, 0.0035068323, 0.019413514, -0.008537813, 0.02212598, -0.006871583, -0.0128777595, -0.005405559, -0.017631035, -0.04303781, -0.053784344, -0.018147696, 0.016378134, 0.01394983, 0.024618868, 0.025174277, -0.0072074123, 0.004989002, 0.015280231, -0.02357263, -0.009700298, -0.015267314, 0.006645544, -0.029811304, 0.03936952, 0.007950111, -0.0017711764, 0.0133815035, 0.0076401155, 0.044200294, -0.004840462, -0.037948705, -0.009390302, -0.014195244, -0.024812615, 0.030663794, 0.019594345, 0.012509639, -0.04040284, -0.038207036, -0.027822161, 0.0036682887, -0.008395731, 0.012283601, -0.0047791083, 0.03236877, -0.013291088, 0.035442904, 0.0033776672, -0.0438903, 0.031025456, -0.001887425, -0.04492362, 0.01664938, 0.005980344, -0.0356754, -0.022616807, 0.022345562, -0.0031290245, 0.029036313, 0.016029388, -0.014944402, -0.00891885, -0.0046467143, -0.00041857563, 0.0024396058, 0.0018438317, 0.010662579, -0.03371209, 0.041384496, 0.033505425, -0.015525645, 0.01668813, -0.0073365774, -0.019155184, 0.0021990358, 0.012625888, -0.0026656447, -0.019439347, -0.025484273, 0.029837137, 0.019155184, -0.016416883, -0.012651721, 0.02593635, -0.029139645, 0.0073172026, 0.0020230482, -0.037199546, 0.027383, -0.023818044, -0.02005934, -0.014104828, -0.011192155, -0.006574503, 0.003548811, -0.044200294, -0.022655558, 0.028442154, -0.012774428, 0.036321227, 0.017941032, 0.022435976, -0.015254398, -0.029811304, 0.0075367833, -0.041797824, -0.01365275, -0.033143762, -0.016713964, 0.036088727, -0.00943551, 0.008079276, -0.02079558, -0.028106324, -0.005928678, 0.0026753321, -0.023068886, -0.021493072, -0.004559528, 0.009112597, -0.024024708, 0.003093504, 0.012296517, 0.023921376, -0.026608009, -0.015693558, 0.015422313, 0.0033163137, -0.040428676, -0.00016741008, -0.02250056, 0.011476318, -0.0013748009, -0.010765911, 0.024825532, -0.03482291, 0.024308871, 0.016106887, 0.06809584, -0.056367647, -0.018638523, -0.02994047, 0.0133298375, 0.0006296798, -0.006723043, -0.0046047354, -0.032807935, -0.007149288, -0.028390488, 0.03482291, 0.012005895, 0.012755053, -0.008544271, 0.01757937, -0.032187942, 0.012580681, 0.028700484, 0.0032258981, 0.014169411, 0.045982774, -0.0153706465, 0.0077886553, 0.032627102, 0.005473371, 0.018741855, 0.032704603, -0.012529015, 0.009196554, -0.006768251, -0.08028902, 0.021015162, -0.005880241, -0.018703105, -0.0013126403, 0.04528528, 0.0059609693, -0.00880906, -0.017476037, -0.010740078, 0.027383, 0.002076329, -0.012645263, -0.01591314, 0.013174839, -0.012800261, 0.013717333, 0.015254398, -0.026582176, -0.0006926478, 0.0098811295, -0.00832469, -0.0020117464, 0.030353798, -0.017230624, 0.0044174464, 0.028855482, 0.004927648, -0.0131683815, -0.010979033, 0.048824407, -0.017463122, 0.0072913696, 0.0037199548, -0.023365965, -0.043089475, 0.0046886927, -0.009810089, 0.0015637049, -0.02934631, -0.017256457, 0.040532008, -0.0068134586, 0.010165293, 0.015073567, -0.009370928, -0.03725121, 0.037509542, -0.007975944, -5.0127157e-05, -0.030069634, -0.022733057, -0.019581428, -0.028777983, -0.0077951136, 0.023081802, 0.0057704505, -0.060965925, 0.026892172, -0.0038329742, 0.0052118115, 0.027383, 0.03988618, 0.027512165, -0.0027512165, -0.030534629, -0.024283038, 0.02653051, 0.00013199059, 0.01946518, -0.005805971, -0.024037624, -0.010255708, -0.018483525, 0.02194515, -0.022952637, -0.0036198518, -0.004029951, -0.0030612128, 0.019026019, -0.019064767, -0.01809603, -0.008608853, -0.025187194, 0.011941313, 0.00610628, 0.03526207, -0.021828901, -0.012955259, 0.000919494, -0.027512165, 0.0056703477, -0.018987268, -0.00429151, -0.013077966, 0.03045713, 0.00028759416, -0.016920628, 0.007853238, 0.010023211, 0.0022297124, 0.036502056, -0.026168847, -0.022293895, -0.007601366, -0.006923249, -0.008118026, -0.028726317, 0.04339947, -0.020098088, 0.02272014, -0.012819636, -0.049315233, 0.012038187, 0.04156533, -0.0014902423, 0.0038814112, 0.018328527, -0.019529762, -0.02764133, 0.008647603, 0.008557187, -0.019064767, 0.012929426, -0.04766192, -0.02383096, 0.022087231, 0.0033066263, 0.0067617926, 0.017411456, 0.0034842284, -0.034616247, -0.0019084143, 0.013239422, 0.0032646477, -0.03688955, -0.001589538, 0.03918869, 0.01994309, -0.009661549, 0.020214338, 0.00091787946, -0.013704416, 0.0069297072, -0.015900223, -0.0032775642, 0.0101717515, -0.01672688, 0.005931907, -0.001838988, 0.002649499, 0.00067609857, -0.042469483, 0.002520334, 0.029656306, -0.027537998, -0.011185697, 0.006258049, 0.013510669, 0.0066907518, -0.021557655, 0.018289778, 0.007840321, 0.03714788, 0.026763007, -0.0037716208, 0.0059868023, 0.0046564015, -0.0018793521, 0.00910614, 0.0019665386, -0.0076724067, -0.022926804, -0.01424691, 0.023779294, 0.013975663, -0.0070072063, -0.017863533, 0.01424691, -0.010940284, 0.0037102674, -0.01591314, -0.0081826085, 0.01565481, 0.011353612, 0.012942342, 0.0009372542, -0.0049050446, -0.0014700602, -0.01480232, 0.008344065, 0.030276299, -0.008608853, -0.021867651, 0.015008984, -0.023508048, -0.03234294, -0.022293895, -0.037457876, -0.0060643014, -0.0088801, -0.040428676, -0.0024896571, -0.03988618, 0.025213027, -0.005302227, 0.002856163, -0.000345113, 0.019258516, 0.025471358, 0.026737174, -3.1408308e-05, 0.0017631035, -0.005111709, 0.037535377, 0.009532384, -0.03986035, -0.010475289, 0.0017388851, 0.009448427, 0.016933544, -0.008098652, 0.017514788, -0.022926804, -0.004327031, 0.026168847, 0.0056154523, -0.031877946, 0.01813478, -0.047455255, -0.0016726879, 0.00477265, 0.006748876, 0.012606514, 0.039472852, -0.022616807, 0.00041494286, -0.02498053, 0.020769747, -0.008570104, -0.026814673, -0.03696705, -0.016223136, 0.024903031, 0.054301005, 0.0050826464, 0.012167352, -0.0131167155, -0.018522274, 0.02375346, -0.0060804468, 0.018561024, -0.036372893, -0.0028448612, 0.018031448, 0.006361381, 0.022771806, 0.0015556321, -0.0058447206, -0.0034454789, 0.022410143, -0.011366528, 0.009299886, 0.000444005, 0.02079558, -0.0064485674, -0.02934631, 0.006383985, 0.006261278, -0.040506173, -0.011037157, 0.013207131, 0.0001873903, 0.01728229, 0.03063796, -0.0030918894, 0.012535472, -0.016029388, -0.021221826, -0.0020666416, -0.019762259, -0.026711341, -0.0073882435, 0.034926243, 0.005521808, 0.028002992, -0.012406307, -0.016145637, -0.01861269, 0.01565481, 0.00499546, -0.015435229, 0.006035239, 0.02131224, -0.014001496, -0.006109509, 0.03327293, 0.011812148, -0.030586295, -0.0019552365, 0.023327217, 0.033143762, 0.042727813, -0.024567202, -0.017191874, -0.008705728, -0.006923249, 0.029294644, -0.009222387, -0.009293429, 0.003987972, -0.006490546, 0.018522274, 0.013730249, 0.01960726, -0.016675213, 0.018896854, -0.006061072, 0.01960726, -0.023430549, -0.004214011, 0.004482029, -0.012806719, -0.0065841908, -0.030250465, -0.021828901, 0.0018551337, 0.027718829, -0.033841256, -0.03626956, 0.024528451, -0.0054346216, 0.026078433, -0.021493072, 0.029242977, -0.0077692806, -0.030947957, -0.039472852, -0.0057607633, -0.009784256, 0.0036456848, 0.03593373, -0.04673193, -0.027718829, -0.0043238015, 0.0074399095, -0.01561606, 0.016274802, 0.0034293332, 0.018767688, -0.011663608, 0.015189815, -0.027150502, 0.029656306, -0.0053603514, 0.008305316, 0.018418942, -0.012025271], "content_hash": "b535398474b391abdc4dcb7d8526648c"}, {"content": "Each appears as a chip (e.g., \"L0-H3\") with an \u00d7 button to remove it. 5. **Run the experiment**: Click \"Run Ablation Experiment.\" 6. **View results**: The panel shows a side-by-side comparison of the original vs. ablated generation, plus the probability change for the immediate next token. ### Understanding the Results\n\n- **Full Generation Comparison**: Shows the original generated text alongside what the model generates with those heads removed. If the text changed, the ablated heads were important for that generation.\n- **Probability Change**: Shows the immediate next-token probability before and after ablation (e.g., \"72.3% \u2192 45.1% (-27.2%)\"). A large drop means the head was important.\n- **No change?** If ablating a head has no effect, it may be redundant, or it may serve a function that isn't relevant to this specific prompt. Try a different prompt or a different head.\n\n### Tips\n\n- Start by ablating heads from the **Previous-Token** category -- these often have noticeable effects.\n- Try ablating heads from the **Other** category for comparison -- these often have less impact.\n- Use the **head categories** in the Attention stage (Stage 3) to pick interesting heads to ablate.\n- You can ablate heads from multiple layers simultaneously to see compound effects.", "source_file": "ablation_panel_guide.md", "chunk_index": 1, "embedding": [0.03795369, 0.019715818, 0.05737392, 0.0118088, -0.021297222, -0.02503643, 0.009015479, 0.026292684, 0.023750614, 0.038160603, 0.049304325, -0.030002333, -0.052053306, 0.026942981, -0.001459473, -0.024282675, 0.042594444, -0.028819975, 0.009200223, 0.04912697, -0.015991392, 0.003135096, 0.0049585137, 0.043392535, 0.036327947, -0.003935035, 0.031302925, -0.0010696645, 0.048949618, 0.015518447, -0.016833821, -0.018415224, -0.030194465, -0.003484261, -0.05793554, 0.0016562248, 0.01715897, 0.003467634, -0.0042490987, -0.017528456, 0.026750848, -0.026248345, -0.034702204, 0.028790416, -0.021385899, 0.042801358, 0.039490756, -0.05226022, -0.030682188, 0.049836386, -0.001244247, 0.07194648, -0.015917493, 0.0122374045, -0.036387064, -0.005313221, -0.00899331, -0.018060517, 0.015237638, -0.03322426, -0.016094847, 0.011084606, 0.022361344, 0.035293385, 0.000695559, -0.027652396, -0.037598982, 0.025893638, 0.009370186, 0.008091762, 0.034347497, -0.011934426, 2.3439323e-05, -0.0035249046, -0.031214248, -0.015503668, 0.025154665, -0.015503668, -0.0035249046, -0.013153732, 0.051875953, 0.012119168, 0.034140587, -0.02444525, -0.007929188, -0.036150593, -0.03431794, -0.018296989, -0.028864313, -0.075197965, -0.02558327, 0.029943215, -0.024622604, 0.01432131, -0.023277672, 0.049806826, 0.009377576, -0.009392356, 0.0040828297, 0.023248112, 0.03378588, -0.01574014, -0.025775403, -0.014262193, 0.0045964164, -0.011306298, -0.05867451, 0.0064734095, -0.030416157, 0.00873467, -0.012466487, -0.035382062, -0.02837659, 0.026750848, -0.005538608, -0.035263825, -0.029352035, -0.0028044053, 0.042742237, -0.041559882, -0.04495916, -0.0036098866, -0.009096766, 0.005708572, 0.0002279272, -0.0013671013, 0.05226022, -0.047087405, 0.019789716, -0.010981149, 0.01770581, 0.021385899, -0.019804494, -0.026809966, -0.045609456, 0.019021183, 0.01714419, 0.04924521, -0.07419296, 0.038810898, 0.0096066585, -0.023839291, -0.026603054, 0.010729898, -0.08016387, 0.011764461, 0.012850753, 0.060004666, -0.025080768, 0.052408013, -0.0020100086, -0.005254103, 0.00016084225, -0.014712966, -0.016124407, -0.04835844, -0.015562786, 0.0065103583, -0.03455441, -0.06603469, 0.029159902, -0.001665462, -0.018636918, -0.024489589, -0.019612363, 0.03322426, -0.0063847327, 0.00068586, -0.05648715, -0.02864262, -0.041796353, -0.036948685, -2.4435205e-05, -0.046555344, -0.055482145, -0.0016414453, -0.07046853, -0.019301994, -0.01770581, -0.028553944, 0.014978997, 0.030534394, -0.025760623, 0.048181087, -0.020380896, -0.050634477, 0.013249799, -0.008586874, -0.057787742, -0.023543702, 0.021681488, -0.0039165607, 0.06130526, -0.010478647, -0.012193066, -0.02809578, 0.084538594, 0.014749915, 0.004518824, 0.008365182, 0.014070059, 0.03479088, 0.0069241836, 0.0074968883, 0.0051543415, 0.0022169212, -0.011919646, 0.014565172, -0.0065583917, -0.0028561333, 0.044249747, -0.036682654, 0.01773537, 0.004012627, -0.04717608, -0.022420462, -0.03428838, 0.055334352, 0.029839758, 0.01486815, 0.03485, -0.009857909, 0.050398007, 0.07082324, -0.024386132, 0.031716753, 0.043481212, -0.016523452, 0.00077222753, -0.023942748, 0.032692198, 0.029248578, -0.020839058, -0.009776622, -0.022538697, -0.055836853, 0.029352035, -0.01859258, 0.031687193, -0.04859491, -0.033815436, -0.0066138147, 0.057699066, 0.009880078, 0.032130577, 0.033519845, -0.03505691, -0.069759116, 0.02468172, 0.038012806, 0.04886094, 0.008343013, -0.03597324, -0.0157697, 0.00632192, -0.0049548186, -0.015799258, 0.006909404, -0.0056088106, -0.010027873, -0.0053280005, -0.0008941582, -0.039963696, 0.02500687, -0.042890035, -0.008978531, -0.0022723442, -0.0015943358, -0.0516986, -0.03482044, -0.030046672, -0.010552544, 0.036889568, 0.018784711, -0.05143257, -0.0056790127, -0.013087224, 0.0027249656, -0.035648093, 0.031096013, -0.008343013, 0.005837892, 0.018326547, -0.0045040445, -0.039815903, 0.0023037505, 0.00024824898, 0.018459562, 0.010338242, -0.016671246, 0.012281743, -0.050782274, 0.08217388, -0.0224648, -0.032189693, 0.0064992737, 0.011904866, 0.02305598, -0.04779682, -0.00012805029, 0.024386132, -0.033253815, 0.03597324, -0.0030390294, 0.019893171, 0.014727746, 0.018370885, 0.022922965, -0.019922731, 0.0122374045, 0.007530142, 0.014801643, 0.007921798, -0.024356574, -0.029588507, -0.026809966, 0.03334249, -0.024193998, 0.0025679336, -0.022509139, 0.0019601278, -0.044811364, 0.009155884, 0.013382814, -0.016493892, 0.010286514, -0.00057547574, 0.0039239503, 0.011018098, -0.0081361, 0.03963855, -0.002948505, 0.018356107, 0.02051391, -0.017025955, 0.0004202913, -0.04522519, 0.047323875, 0.0147646945, 0.03514559, -0.056634944, 0.00252729, -0.0027249656, 0.00844647, -0.008003085, 0.026765628, 0.002667695, 0.018134415, 0.035352502, 0.02837659, -0.01910986, 0.0029244884, -0.012399979, -0.03730339, 0.054950085, 0.0680447, 0.006909404, -0.022361344, 0.043096945, 0.01265123, -0.0011186214, 0.014395208, 0.021297222, 0.02329245, 0.024371352, -0.028007103, -0.045905046, 0.027593277, 0.033312935, -0.014639069, -0.038958695, -0.020750381, 0.019228095, -0.00899331, -0.08282417, -0.006658153, -0.024341794, 0.003085215, -0.019745378, 0.034465734, 0.0054979646, -0.036121033, -0.041205175, -0.045402545, 0.04490004, 0.01431392, 0.0030741305, 0.010426919, -0.057492156, 0.017025955, 0.039195165, -0.009473643, -0.00021464877, -0.004851362, -0.02670651, 0.07732621, -0.04020017, 0.056605387, 0.02332201, 0.055511706, 0.004334081, 0.01628698, -0.045993723, -0.02389841, 0.046466667, 0.030209245, 0.017055513, -0.006713576, 0.030238803, 0.034909118, 0.05134389, 0.025997095, 0.03597324, -0.014070059, 0.022050975, -0.004832888, 0.043776803, 0.03006145, 0.02587886, 0.013079834, -0.028036661, -0.07638032, -0.017425, -0.04466357, -0.014698187, -0.020912956, -0.0047700754, -0.004101304, 0.045727693, -0.04948168, -0.039195165, 0.036150593, 0.040052373, -0.036948685, -0.0064032073, -0.040643554, 0.0048957006, 0.077089734, -0.042298853, -0.029086005, 0.015444551, 0.002024788, 0.02641092, -0.019139418, -0.07561179, 0.008076983, -0.014557782, -0.059147455, -0.013641454, -0.07348354, -0.03680089, -0.0018843829, -0.030889101, 0.06366997, 0.012991157, -0.06272409, 0.07626209, -0.005228239, 0.09648041, -0.00022654163, 0.021903181, -0.045609456, 0.0029300307, 0.048949618, -0.020898176, 0.05196463, 0.0065657813, -0.0084243, 0.03452485, 0.0021300917, 0.0005556159, -0.0047368216, -0.012015712, -0.023514144, -0.00660273, -0.007700106, 0.026632613, -0.007751834, 0.014195685, -0.052408013, 0.01630176, -0.026617832, 0.059029218, 0.00870511, -0.02638136, 0.09748541, 0.0009652844, -0.040407084, 0.020262659, -0.0026048822, 0.011350636, -0.020839058, 0.0162722, -0.0185778, 0.063315265, -0.01293943, 0.0037151903, 0.015873155, 0.021711048, -0.030209245, 0.035263825, 0.015444551, -0.003935035, 0.04776726, 0.0074599395, -0.02193274, 0.06916794, 0.05427023, -0.005261493, 0.056694064, -0.02638136, -0.033933673, -0.0047848546, -0.0007260417, -0.022982081, 0.01884383, 0.0013744911, 0.01884383, 0.020292219, 0.006410597, -0.006521443, 0.0075966497, -0.008062203, 0.012555163, -0.0023277672, -0.017262425, -0.02756372, -0.04525475, -0.041825913, -0.01770581, 0.028923431, -0.0325444, -0.004160422, 0.036357507, -0.042535327, -0.031214248, 0.011505821, 0.019228095, -0.039727226, 0.00344177, 0.022834288, 0.026218787, 0.022124873, -0.006654458, 0.01178663, 0.0020100086, 0.04835844, -0.01823787, 0.015828816, -0.007411906, 0.013700573, -0.0059007052, 0.025997095, -0.034731764, -0.020765161, -0.002301903, -0.026676951, -0.010057433, -0.00688354, -0.019760156, 0.021755386, -0.024696501, -0.0077370545, -0.020395674, -0.013375424, 0.025627607, 0.013168511, 0.0331947, 0.0061371764, -0.007378652, -0.00033900418, -0.014513443, 0.02982498, -0.040643554, 0.013744911, -0.0049843774, 0.0030999947, 0.022597816, 0.0026732374, 0.0016109627, 0.006802253, -0.027755853, -0.021755386, 0.03493868, -0.024223559, -0.014247413, 0.023824513, 0.012821194, -0.03647574, 0.0050287163, 0.016641688, -0.028790416, 0.041944146, 0.046850935, -0.021563252, 0.017927501, 0.03257396, -0.008823346, -0.013331085, -0.019065522, 0.010353021, 0.0076853265, -0.023987086, 0.03570721, -0.008756839, 0.02135634, -0.044988718, 0.00012354717, 0.0073306193, 0.000119967764, 0.010101771, -0.013944434, -0.053767726, -0.0009121707, -0.001240552, 0.029322477, 0.02021832, 0.006155651, 0.027120335, -0.027475042, 0.0012627214, 0.00034985787, -0.016449554, 0.0057603, 0.009865299, 0.01630176, -4.6258024e-06, -0.021445017, 0.018474342, 0.005095224, -0.04750123, 0.03452485, 0.01628698, 0.015126792, -0.010877693, 0.023750614, 0.024400912, 0.0038722223, 0.0001995229, -0.018607358, 0.00463706, -0.0030224025, 0.000555154, 0.0027748463, -0.046850935, 0.019568024, 0.017794486, -0.0054684053, -0.019139418, -0.02555371, 0.006750525, -0.017262425, 0.057462595, 0.004851362, 0.0010530375, -0.03647574, 0.00259934, -0.019656701, -0.003131401, -0.011609277, -0.014291751, 0.021267664, -0.03768766, -0.030682188, 0.0033069074, 0.025908418, 0.0016423691, 0.007951357, -0.013035496, 0.0036690044, -0.033017345, 0.016626908, 0.033667643, 0.010545155, -0.011550159, 0.048713148, -0.033549406, -0.03485, -0.01630176, 0.018311769, -0.008025254, 0.0058637564, -0.032455724, -0.007053504, 0.011601887, 0.0032681113, 0.017883163, -0.010877693, -0.000369025, 0.0035415315, -0.018208312, -0.009673166, 0.003262569, -0.009340628, -0.012200456, 0.0021467186, -0.016050508, 0.044338424, -0.0020469571, -0.054122433, -0.0010881388, -0.022568258, 0.020853838, -0.0031849768, -0.0021134648, 8.4520114e-05, 0.0050841393, 0.05370861, 0.027282909, -0.01236303, -0.010803795, 0.015518447, -0.0097175045, 0.01970104, -0.031066455, 0.03520471, -0.016390437, -0.017336324, -0.008313454, -0.050723154, -0.008468639, 0.023425467, -0.022272667, -0.011040268, 0.006802253, -0.010035262, 0.0055792513, -0.03875178, -0.024888635, -0.0062517175, 0.012030492, 0.021873621, 0.0065879505, -0.028494826, -0.0012137643, 0.014225244, -0.0155332275, -0.011055047, 0.0055718618, -0.021223325, 0.012525604, -0.014151347, 0.018932506, 0.019257655, -0.000927412, -0.0009130944, -0.00788485, 0.016198304, -0.02024788, -0.025967536, -0.021001633, 0.014971607, -0.004947429, -0.005560777, 0.0275785, 0.015067674, -0.0010798253, -0.01993751, -0.0039128656, 0.030829983, 0.018282209, -0.0049326494, 0.007197604, 0.02364716, -0.0076631573, -0.019257655, 0.013722742, 0.00954754, 0.010478647, 0.02330723, -0.010234786, 0.008062203, 0.041619, 0.0033069074, 0.032987785, -0.031332485, 0.050723154, -0.013382814, -0.034968235, 0.011010708, -0.015503668, -8.919644e-05, -0.063847326, 0.0044264523, -0.016686026, -0.020085305, -0.01265123, -0.021297222, -0.021178987, -0.027489822, 0.0124590965, 0.014424766, 0.009473643, -0.027061217, -0.000398353, 0.0018363496, -0.0043045217, -0.10942723, 0.003523057, 0.013050276, -0.041559882, -0.016745145, -0.019715818, 0.019050742, 0.0027009489, 0.015932273, 0.0021430238, 0.008483418, -0.0147646945, -0.0021614982, -0.030800425, 0.028893871, -0.044574894, 0.012473877, -0.031243809, 0.03703736, -0.0465849, -0.027016878, 0.008535147, -0.0023092928, -0.0146612385, -0.054595377, 0.007190214, -0.037628543, 0.0008973912, -0.00591179, 0.022065755, -0.021548472, 0.012252184, -0.01939067, 0.018666476, -0.024001867, 0.0037004107, -0.009444084, -0.008003085, 0.004552078, 0.011350636, 0.03399279, -0.029337255, 0.0056974874, -0.004467096, 0.0034657866, -0.017897943, -0.020528689, -0.0021319392, 0.026588274, -0.0013532456, 0.00462967, -0.000829498, 0.0058822306, 0.038042367, 0.014188295, -0.03851531, 0.0005505354, -0.0019545855, 0.0052171545, 0.023854071, 0.026307464, 0.05920657, 0.016050508, -0.011697954, 0.0065140533, -0.014587341, -0.01208222, 0.017425, -0.038042367, -0.048269764, -0.029026886, 0.028982548, -0.010256955, 0.006717271, 0.03597324, 0.030150127, 0.04014105, 0.03091866, 0.0022704967, 0.00869772, 0.02555371, -0.039076928, -0.011106775, 0.019301994, -0.0068983193, 0.028287914, 0.02332201, 0.0025291375, 0.0100500425, 0.034199703, 0.031066455, -0.042564884, -0.0032052984, 0.04306739, -0.051048305, -0.020912956, 0.02979542, 0.015503668, 0.016819041, 0.0055496925, -0.004980683, -0.00086506107, -0.007530142, -0.030889101, -0.015488889, 0.015208079, 0.0029595897, 0.014816423, -0.017085072, -0.007448855, 0.032987785, 0.0155332275, -0.019257655, 0.040318407, -0.021873621, 0.022819508, 0.029086005, -0.018385665, -0.0185778, -0.03148028, -0.00013359259, 0.038929135, -0.012887701, -0.007563396, 0.0056272848, 0.027977545, -0.014247413, 0.008535147, -0.0026566104, 0.016730364, 0.0145430025, -0.0071865194, -0.029721523, -0.010818575, -0.0064512403, -0.011010708, -0.0034306855, 0.00079347304, -0.009207612, 0.014018331, -0.02553893, -0.0014419224, 0.03118469, -0.009118935, -0.0556595, -0.038929135, -0.00281549, -0.012266964, -0.031923663, -0.0069537424, 0.011424533, 0.01856302, 0.003526752, -0.00023878088, 0.009917027, 0.015681023, 0.04522519, -0.008867685, -0.0058600614, 0.048240203, 0.010234786, 0.021415457, 0.034909118, 0.010574713, -0.00017885472, 0.0041973707, -0.03508647, 0.0036911736, -0.012525604, 0.006521443, -0.03207146, 0.0041493373, 0.004962208, 0.011010708, 0.034436174, 0.019035963, -0.008897243, -0.012392589, -0.012828584, 0.028125338, -0.006828117, 0.0155332275, -0.01885861, -0.021563252, -0.0030408767, -0.016523452, 0.018888168, 0.021149427, 0.022583036, -0.02134156, -0.0092889, 0.026292684, -0.0032958228, 0.031864546, -0.020011408, -0.011927036, 0.017439779, 0.013981382, -0.055600382, -0.035500295, -0.04634843, 0.066744104, 0.015681023, -0.007267806, 0.026100552, 0.028184457, -0.03313558, 0.010183058, 0.005985687, 0.023750614, -0.028982548, 0.019789716, -0.006092838, 0.00505458, -0.019553244, 0.007833121, -0.022154432, 0.00060688215, 0.013138953, -0.012045272, -0.048269764, -0.037273835, -0.015961831, 0.033963233, -0.012156118, -0.0023536312, 0.026529156, 0.004973293, 0.0073306193, 0.016656468, -0.031657632, -0.026928201, 0.0034824135, -0.0010151651, -0.022893405, 0.046259753, -0.025420696, -0.011624056, -0.0063995123, 0.012540384, 0.037746776, -0.01234086, -0.02839137, -0.021267664, -0.008121321, -0.020440012, 0.022612596, 0.01435087, 0.0071458756, -0.031302925, -0.021193765, -0.016922498, 0.02414966, -0.019981848, 0.018947287, -0.0040311017, 0.039993256, 0.016626908, 0.039756786, 0.022021417, -0.035766326, 0.024888635, 0.010005704, -0.044781808, 0.007563396, 0.011971374, -0.024327014, -0.013013327, 0.03875178, -0.018208312, 0.029026886, 0.052378457, 0.007005471, 0.021105088, -0.0019638226, -0.010818575, 0.017321544, -0.0022741915, 0.015030725, -0.009229781, 0.034909118, 0.027652396, -0.014794254, 0.008490808, -0.0058046384, -0.023735836, -0.00702764, -0.009650997, 0.0027840834, 0.0016885549, -0.00547949, 0.019597583, 0.009333238, 0.012407368, -0.00591179, 0.021755386, 0.0068724556, 0.013848367, 0.018755153, -0.052614927, 0.0032958228, -0.00047386688, 0.03349029, -0.03898825, -0.0185778, 0.0022520223, 0.032455724, -0.05001374, 0.004688788, 0.0049511236, -0.009665776, 0.04218062, -0.0052688825, 0.017336324, -0.02555371, -0.016715584, 0.012725127, -0.013966603, -0.004496655, -0.013641454, -0.023395907, 0.018178754, -0.00067292794, 0.030534394, -0.008845516, -0.02979542, 0.0010207074, -0.0036948684, -0.002723118, -3.022287e-05, 0.016804261, 0.0015998781, -0.014358259, 0.007833121, 0.0015167436, 0.020144423, -0.031036895, 0.0037872402, 0.008971141, -0.0126807885, -0.027608057, 0.026691731, -0.026676951, 0.0058674514, 0.013331085, 0.00085536204, -0.0055718618, -0.02191796, 0.023780175, 0.003011318, 0.08442035, -0.05873363, -0.03405191, -0.028494826, 0.004851362, 0.014395208, -0.0019379585, -0.01770581, -0.052969635, -0.014151347, -0.03260352, 0.02049913, 0.007175435, 0.017203309, 0.009030258, 0.020912956, -0.026529156, 0.0035489213, 0.034613527, -0.00239058, 0.017380662, 0.0404662, -0.022612596, 0.012606892, 0.017070293, 0.011491041, 0.012784245, 0.033608522, 0.009081987, 0.02165193, -0.014306531, -0.042210177, -0.0056088106, -0.016331319, -0.025893638, -0.0052356287, 0.011003318, 0.018651696, -0.024208779, -0.023100318, -0.02558327, 0.023794953, 0.016700806, -0.00758187, 0.009333238, -0.012259574, 0.0039165607, 0.002752677, 0.005257798, -0.025139885, -0.014860761, -0.00591179, -0.02055825, -0.012496046, 0.03993414, -0.04218062, 0.0031942138, 0.02836181, -0.0010456478, -0.008224777, -0.010988539, 0.035648093, -0.014136567, 0.0033124497, -0.013308916, -0.047915056, -0.025923198, 0.024666943, -0.003820494, 0.015089843, -0.028923431, -0.020129643, 0.029248578, -0.013464101, 0.007249332, -0.0016756229, -0.018503902, -0.029647624, 0.039195165, 0.009229781, 0.0030611986, -0.03742163, -0.030948218, -0.02160759, -0.0133384755, -0.011978764, 0.026647393, 0.011638836, -0.03544118, -0.0055127437, -0.018666476, 0.011594498, 0.055157, 0.04977727, 0.0089415815, -0.021415457, -0.019981848, -0.014025721, 0.013249799, -0.02245002, 0.013316306, -0.011077216, -0.010367801, -0.013737521, -0.020114865, -0.017173748, -0.044515777, 0.024873855, 0.008579485, 0.01152799, -5.9868416e-05, -0.008963751, -0.004496655, 0.012525604, -0.008246946, 0.003493498, -0.0049548186, 0.058112893, -0.025346799, -0.040613994, 0.010766847, -0.0030094704, 0.0037447494, 0.003063046, 0.0064771045, -0.012303912, 0.022878626, -0.0013320001, 0.0019379585, -0.00450035, 0.011195452, -0.0024903414, 0.023809733, -0.003258874, -0.014949438, -0.01712941, -0.01630176, 0.0011934425, -0.017927501, 0.029322477, -0.006325615, 0.01967148, -0.02107553, -0.04832888, 0.007323229, 0.030889101, 0.0157697, -0.00081748964, 0.010619052, -0.02553893, -0.034170143, -0.00052928994, 0.02160759, 0.0029577422, 0.010330852, -0.025139885, -0.023011642, 0.0027009489, 0.027327247, 0.012318691, 0.010212617, 0.0013707962, -0.021696268, -0.005368644, 0.0057713846, -0.011084606, -0.034702204, -0.0054942695, 0.036150593, -0.00589701, -0.009429305, 0.009739674, -0.014188295, -0.007810952, 0.014070059, -0.017750148, 0.0038020196, 0.039461195, -0.0056974874, 0.021297222, -0.009798791, -0.012326081, 0.0024663247, -0.018814271, -0.01040475, 0.016922498, -0.018784711, -0.0043710293, -0.0065694763, 0.013678404, 0.012126558, -0.024193998, 0.0010825965, 0.0086164335, 0.023277672, 0.036357507, -0.010818575, 0.012828584, 0.031864546, -0.012466487, 0.005542303, -0.010626442, -0.009089377, -0.012399979, -0.02698732, 0.027312469, 0.004330386, -0.018282209, -0.001982297, 0.0057898588, 0.004703568, 0.004814414, -0.0074082115, -0.009074597, 0.01964192, -0.0149863865, 0.00898592, -0.007190214, 0.0005362178, -0.008764228, -0.01628698, 0.017617133, 0.03913605, 0.02363238, -0.024844296, 0.00047525245, -0.008165659, -0.011077216, -0.015902715, -0.007057199, 0.010072212, -0.02414966, -0.030297922, 0.0040015425, 0.0020728211, 0.020040967, -0.031421162, -0.019981848, 0.0090598175, 0.020765161, 0.010227396, 0.0208095, -0.013959213, 0.022612596, -0.023248112, 0.057758186, -0.0032902805, -0.014890321, -0.015858376, -0.008719889, 0.031716753, 0.006913099, -0.020750381, 0.00083781144, -0.031243809, -0.02781497, 0.008217388, -0.0041715065, -0.026337022, 0.0059709074, -0.05370861, -0.008527757, -0.011151114, -0.007892239, 0.011217621, 0.027223792, -0.0016395979, -0.0009015479, -0.031568956, 0.058910985, -0.012946819, -0.0015546159, -0.019154198, -0.00788485, 0.017691031, 0.03505691, 0.00449296, 0.012666009, -0.008904633, -0.023957528, 0.003820494, -0.029352035, 0.007833121, -0.01179402, -0.0071532656, 0.02836181, 0.014107008, 0.013840977, 0.017173748, -0.011734903, -0.0030999947, 0.03201234, 0.0041973707, 0.015060284, -0.022996861, 0.0036930211, 0.006635984, -0.02391319, -0.012991157, -0.0101534985, -0.0371556, -0.014143957, 0.009488422, -0.007951357, 0.010515596, 0.025819741, -0.01378186, 0.02417922, 0.003347551, -0.018356107, -0.0072012986, -0.023558483, -0.011904866, 0.008091762, 0.021415457, 0.028406149, 0.017957062, -0.015858376, -0.039431635, -0.010301294, -0.0112471795, 0.023100318, -0.023543702, 0.0012525604, 0.0017439779, -0.0123482505, -0.021504134, 0.03594368, 0.0024829516, -0.022036197, -0.004829193, 0.036889568, 0.032160137, -0.001763376, -0.0048957006, -0.011956595, 0.0024847991, -0.027785411, 0.0224648, -0.019848833, -0.022095313, -0.0013079834, 0.00843908, 0.002854286, 0.012702958, 0.032189693, -0.008793787, 0.02809578, -0.018932506, 0.002314835, 0.0018853067, -0.0014557781, 0.0028616756, 0.006351479, 0.004515129, -0.0044818753, -0.0006932497, -0.011978764, 0.008771618, -0.054595377, -0.031864546, -0.0149863865, 0.012577333, 0.026883863, -0.016656468, 0.019819275, -0.007618819, -0.033549406, -0.0208095, 0.005667928, -0.02614489, 0.01856302, 0.027962765, -0.019346332, -0.014528223, -0.0073195347, -0.00899331, -0.026499597, 0.009998314, -0.028347032, 0.029189462, -0.021548472, 0.006096533, -0.01909508, 0.03597324, 0.0014816422, -0.00046255134, 0.021312002, -0.01207483], "content_hash": "65f446fd27f72a314995f8ed45a20c90"}, {"content": "# The Attention Mechanism\n\n## What Is Attention?\n\nAttention is the core innovation that makes Transformers powerful. It allows the model to look at **all tokens in the input simultaneously** and figure out which ones are relevant to each other. When processing any token, the model can \"attend to\" (focus on) other tokens that provide useful context.\n\n## An Intuitive Analogy\n\nImagine you're reading a sentence and you come across the word \"it.\" To understand what \"it\" refers to, you look back at earlier words -- maybe \"the cat\" or \"the ball.\" Your brain is doing something like attention: scanning the context and focusing on the most relevant parts.\n\nThe model does this mathematically by computing **attention scores** between every pair of tokens.\n\n## How It Works: Queries, Keys, and Values\n\nAttention uses three concepts (don't worry about the math -- the intuition is what matters):\n\n- **Query (Q)**: \"What am I looking for?\" -- each token asks a question\n- **Key (K)**: \"What do I contain?\" -- each token advertises what information it has\n- **Value (V)**: \"Here's my actual information\" -- the content that gets passed along\n\nThe model computes how well each Query matches each Key. High matches mean strong attention. The final output for each token is a weighted combination of all the Values, weighted by these attention scores.\n\n## Multi-Head Attention\n\nInstead of computing attention once, Transformers compute it multiple times in parallel using different **attention heads**. Each head learns to look for different kinds of relationships:\n\n- One head might track which word a pronoun refers to\n- Another might focus on the immediately preceding token\n- Another might spread attention broadly across the sentence\n\nGPT-2 has **12 attention heads per layer** (144 heads total across 12 layers). The dashboard categorizes these heads to help you understand their roles.\n\n## What You See in the Dashboard\n\nIn **Stage 3 (Attention)** of the pipeline, you can see:", "source_file": "attention_mechanism.md", "chunk_index": 0, "embedding": [0.009518963, 0.054300353, 0.027301572, 0.02299943, 0.010200241, -0.040523406, 0.0064973705, 0.04413165, -0.03320598, 0.036460973, 0.024690008, -0.035375975, -0.02717541, -0.018293567, -0.012395469, -0.029471567, -0.007986088, -0.009323411, 0.033862025, 0.029622963, -0.031565864, -0.0015044882, 0.04771467, 0.055158257, 0.012849654, 0.0005424989, -0.032575164, 0.029042615, 0.014988109, -0.013512007, 0.039943058, -0.018369265, -0.048496876, -0.008743064, 0.022229837, 0.029622963, -0.021573793, 0.03191912, -0.021409782, 0.020627573, 0.020400481, 0.0147484, -0.036814228, 0.053089194, -0.0274782, -0.0033243827, 0.016022641, -0.018104324, -0.045822233, 0.07842263, -0.011733116, 0.047336183, -0.0416841, 0.034543302, 0.040018756, 0.02417274, -0.048421178, 0.03585539, 0.0122882305, -0.012792881, -0.010446258, -0.0074372813, 0.037697364, 0.039034687, -0.0115943365, 0.00025311357, -0.050061293, 0.008944923, -0.021813503, -0.01582078, 0.05054071, 0.0579843, -0.00072267477, 0.006938939, 0.039186083, -0.015215201, -0.02535867, 0.033962954, -0.044030722, -0.0070083286, 0.024462914, -0.042668168, 0.019857982, -0.029168777, -0.02485402, -0.032650862, -0.032877956, -0.067068, -0.04039724, -0.052534077, -0.018091707, 0.028260406, -0.014483458, 0.004131823, 0.014861946, 0.041431773, 0.004368378, 0.010723815, -0.010698583, 0.022848034, 0.062374756, 0.0056583895, -0.013032589, -0.0173095, 0.031364005, -0.018798217, 0.028411802, -0.0025122114, -0.008881842, -0.037293643, -0.07287148, -0.013688635, -0.008263646, 0.016010026, 0.02616611, -0.009588352, -0.00194448, 0.013940959, -0.00025311357, -0.010767972, -0.03272656, -0.023756405, 0.044156887, 0.028058548, 0.01286227, -0.01245855, 0.033483535, -0.01606049, 0.005570076, 0.023340069, 0.003955195, 0.019479495, 0.020097692, -0.022608325, 0.010938291, -0.04299619, -0.022419082, 0.025762388, -0.042542007, 0.0020217546, 0.010093003, -0.018760368, -0.00012330415, 0.009821753, -0.04299619, 0.0024302057, -0.018810833, 0.054148957, -0.039615035, 0.016552525, -0.0006990193, -0.02128362, -0.069894046, -0.039640266, -0.0051600477, -0.011039222, -0.010799512, 0.06903613, 0.002042256, -0.023289604, 0.036385275, 0.0012395469, -0.0114114005, -0.029572498, -0.041532706, 0.045872696, -0.02299943, 0.0032234527, -0.024425065, -0.0559657, -0.039514106, -0.026292272, 0.0041255145, -0.011070762, 0.04236538, 0.00398989, -0.06666428, -0.02053926, -0.0007605235, -0.033306908, 0.0035672456, -0.016527291, -0.014369912, 0.019580426, -0.01952996, -0.011480791, -0.019277634, 0.04150747, -0.026267039, 0.026418434, -0.03131354, 0.03835341, 0.04128038, 0.028411802, -0.019921064, -0.030783657, 0.036107715, 0.01942903, 0.047033392, 0.0066550737, 0.040043987, 0.02990052, 0.071559384, -0.022469547, 0.054552678, 0.03643574, -0.005683622, -0.002745612, -0.054401282, -0.043046657, 0.024059195, -0.07044916, 0.028765056, 0.013587705, -0.03161633, -0.012376544, -0.062273826, 0.0549564, 0.0077526877, 0.041835494, 0.02047618, 0.012786573, 0.014319447, 0.033407837, -0.045923162, 0.04657921, 0.03623388, -0.03565353, 0.015543224, -0.022204606, 0.017233802, 0.010332711, 0.011367244, -0.018861298, -0.04801746, -0.056419883, 0.03883283, -0.031439703, 0.056722675, -0.02959773, -0.015467526, -0.009518963, -0.013852646, 0.029622963, 0.014420377, 0.021561177, -0.010887827, 0.00017298065, 0.06070941, 0.041028056, 0.048295017, 0.003292842, -0.06267755, 0.007967164, -0.019681355, 0.025106344, -0.014899795, 0.058287088, 0.0024507071, 0.038126316, -0.014597005, 0.023491463, -0.03797492, -0.045140952, -0.012174685, -0.01680485, -0.0012758186, -0.009216173, -0.05344245, -0.031565864, -0.01767537, 0.026519364, 0.017347349, 0.0040624337, -0.046049323, -0.029168777, -0.002786615, 0.01585863, -0.0042579854, 0.002042256, -0.014041889, -0.011013989, 0.049581874, 0.031944353, -0.007178648, -0.00038775266, -0.013915727, -0.012893811, -0.0035735536, -0.01626235, 0.035073183, -0.038630966, 0.064090565, -0.0017173875, -0.007954547, 8.422335e-05, 0.0044535375, 0.01706979, 0.026620295, -0.010427333, 0.035577834, -0.038630966, -0.0011654263, -0.011657418, 0.0014327333, 0.017423045, -0.01505119, -0.008269954, 0.0071471073, -0.018558508, 0.008534895, -0.03164156, 0.011354628, -0.011310471, -0.009449573, 0.0065036784, 0.03694039, 0.01293166, -0.014130203, 0.013057822, -0.036057252, -0.035880625, 0.00013424482, 0.004084512, 0.03827771, -0.009165708, -0.031187378, 0.011417709, -0.007721147, -0.039110385, 0.07388078, -0.025068495, 0.00045970472, -0.0249171, -0.024866635, -0.0056142327, 0.015543224, 0.045544673, -0.026948316, 0.027730525, -0.020286934, -0.0109004425, -0.06333359, -0.018962229, -0.052534077, 0.05324059, 0.019353332, -0.012363928, 0.012408085, -0.011329396, -0.0091467835, -0.023668092, 0.025812853, -0.04006922, -0.00956312, 0.025093727, -0.030960286, -0.009014313, 0.00029313075, -0.01791508, -0.01579555, 0.004223291, 0.04912769, -0.000117291725, -0.0021731495, -0.025030646, -0.03532551, -0.0009367568, -0.010818437, -0.001696886, 0.037217945, 0.050666872, 0.01595956, -0.014496075, -0.016186653, -0.011070762, 0.017587056, -0.0028197325, -0.030329473, 0.021763036, 0.022482162, 0.023390533, -0.033988185, -0.020085076, 0.033534, 0.039640266, -0.0008216335, -0.001696886, -0.066361494, -0.019101007, -0.0264689, -0.040321544, -0.025446983, 0.019012693, 0.008490738, 0.03482086, 0.019542577, 0.037899226, 0.04319805, 0.057126395, 0.031263076, 0.017952928, -0.041936424, -0.013537239, 0.025711924, -0.0012285076, -0.009582045, 0.011827737, 0.01807909, 0.008610593, 0.0752938, 0.007613909, 0.01868467, 0.007916699, -0.011285238, 0.018243102, 0.017713219, 0.0064342893, 0.012231457, 0.025081111, -0.051676173, -0.03191912, -0.038403876, 0.011323087, -0.07080241, 0.016022641, 0.057631046, -0.039211314, 0.024500763, -0.027024014, 0.0024775167, 0.047411878, 0.010244397, -0.003195066, 0.011083378, -0.0146600865, 0.048093155, 0.037394576, 0.010597653, -0.037571203, 0.026267039, -0.055915233, -0.018848682, -0.027856687, -0.03484609, 0.010383177, -0.025560528, -0.051474314, -0.0007605235, -0.033811558, 0.02299943, 0.047462344, 0.024298903, 0.020324783, -0.03777306, -0.0057088546, 0.04559514, 0.041154217, 0.028386569, -0.019555192, 0.017233802, -0.0076454496, -0.016035257, 0.007866234, -0.017498743, -0.021258386, 0.0015107963, -0.032348074, 0.03090982, -0.009317103, 0.0022709256, -0.005529073, 0.00204541, -0.021573793, 0.041431773, -0.031364005, 0.028159477, 0.010849978, -0.002039102, -0.05889267, 0.00516951, -0.009600969, -0.025093727, 0.02255786, 0.020577108, 0.061769176, -0.009689283, -0.021649491, 0.028008081, -0.031540632, -0.010711199, -0.03270133, 0.026519364, -0.012704567, 0.020425715, -0.01293166, -0.047033392, -0.037091784, -0.00986591, -0.013045206, 0.034770392, -0.005626849, 0.03302935, 0.074334964, -0.0028843908, 0.025573146, 0.049884666, 0.036713295, -0.0146979345, 0.024841402, -0.021081759, -0.009783904, 0.02532082, 0.032272376, -0.023832101, 0.017195953, 0.042314913, 0.02094298, 0.007853618, 0.0025579452, -0.0070903343, 7.914728e-05, -0.0011417709, -0.0025232506, 0.0037722597, -0.008604284, -0.050389316, -0.016994093, -0.06515033, -0.030834123, 0.0034789317, -0.018949613, 0.026695991, 0.018520659, -0.039211314, 0.010219165, 0.0021747267, -0.045973625, -0.060658947, -5.5639648e-05, 0.0026226037, 0.007134491, 0.02047618, 0.011941284, 0.015669387, 0.008068094, 0.021510711, -0.03471993, 0.0032187216, 0.049405247, 0.0012537402, -0.034063883, 0.012370236, -0.021018678, -0.023819486, -0.013524624, -0.009884834, -0.0118845105, 0.009727132, -0.025787622, 0.0136381695, 0.0013420539, -0.0036618675, -0.035905857, -0.03209575, 0.04017015, 0.009626201, -0.0064122104, 0.0004297411, -0.033559233, -0.013877878, -0.020816818, 0.011493406, -0.015896479, 0.020350017, 0.008055477, -0.0026005253, 0.024702623, -0.0074120485, 0.028310873, -0.032549933, -0.017044557, -0.034997486, -0.0046080863, -0.024248438, 0.004929801, -0.011991749, -0.011543872, -0.028437035, -0.004384148, 0.02511896, -0.029017381, 0.012029598, 0.020450946, -0.025888551, -0.015013341, 0.009077394, 0.005194742, -0.03000145, 0.03310505, 0.010029921, 0.022671407, -0.0070587937, -0.017839381, -0.017359965, 0.017687988, -0.024929715, 0.0012671449, 0.025282972, 0.011386168, -0.03636004, -0.034089115, -0.04201212, 0.00087130995, -0.05324059, 0.030834123, -0.037545968, 0.02969866, 0.004037201, 0.016047874, 0.015618921, -0.017952928, 0.018747753, 0.0064342893, 0.00956312, 0.0315154, 0.010736431, -0.039211314, -0.00909001, 0.014634853, -0.053038727, 0.012666718, 0.04049817, 0.00868629, -0.026014714, 0.020009378, 0.0013318033, -0.01414282, 0.04097759, -0.015139503, -0.021977512, 0.0074120485, -0.025446983, 0.011291547, -0.0050622714, 0.009455882, -0.01026963, 0.010263322, -0.016880546, -0.02017339, -0.036057252, -0.046680138, 0.067068, -0.00020166292, 0.0109004425, -0.008881842, -0.048774436, -0.008976464, 0.0014114433, 0.017826766, -0.040927123, 0.0031651026, -0.009619893, -0.03986736, -0.01966874, 0.009481114, 0.018508043, 0.018470194, 0.022633558, 0.009708207, -0.090635166, 0.03933748, -0.011562796, 0.02195228, 0.0249171, 0.010433641, 0.020097692, -0.043753166, -0.00378803, 0.049304318, -0.0036524052, -0.004440921, -0.04198689, -0.007128183, 0.021422397, 0.005532227, -0.0045765457, -0.023844719, 0.05172664, -0.0064185187, -0.004223291, -0.007191264, 0.023554545, -0.034164812, -0.027730525, -0.029269706, 0.015871247, 0.063585915, -0.017473511, -0.05980104, -0.019769669, -0.0039078845, -0.03164156, 0.011695267, -0.01811694, -0.01585863, 0.025888551, 0.0722659, 0.008989081, -0.041936424, -0.044838164, 0.030152844, 0.02255786, 0.024286287, 0.0056079244, 0.05023792, -0.033054583, -0.015328747, 0.0074498975, -0.018508043, -0.02518204, 0.023756405, -0.0067938524, 0.0027345729, 0.01993368, -0.016539907, -0.030581798, -0.008843994, -0.008364576, 0.027730525, 0.006491062, 0.007885158, -0.015013341, -0.07377985, -0.022343384, 0.022027979, -0.029446336, -0.0124333175, -0.03696562, -0.023491463, 0.0033496153, -0.005131661, 0.01384003, 0.023340069, 0.02154856, 0.025939016, -0.0030436711, 0.009878526, -0.037091784, -0.033887256, -0.013713867, 0.01006777, 0.016438978, 0.0013538817, -0.0073742, 0.002687262, 0.029421102, -0.014861946, 0.033407837, 0.0294968, -0.008471814, -0.033534, -0.043298982, 0.025838086, 0.0025153654, -0.037495505, 0.0021163765, -0.012710875, 0.032348074, 0.015139503, 0.0020469872, 0.0031714106, 0.029067848, -0.008944923, 0.007626525, -0.020968212, 0.042138286, -0.00983437, 0.0053587537, -0.017662754, 0.022267686, 0.0017851999, -0.022027979, -0.039917827, 0.0033212285, -0.009720823, -0.025989482, -0.016855314, -0.013171368, -0.0031635254, -0.02306251, 0.0008015263, 0.023075126, -0.026569828, -0.006118883, 0.008446582, 0.004727941, -0.07741333, -0.04044771, 0.050591175, -0.048295017, -0.0119223595, -0.03333214, 0.035047952, 0.0081501, 0.021069143, -0.02616611, 0.0037091784, -0.026822153, 0.008806145, -0.004825717, 0.0315154, -0.035451673, -0.010887827, 0.0036208646, 0.016237117, -0.002486979, -0.037621666, -0.011436634, 0.017486127, -0.011209541, -0.024828786, 0.014016657, -0.006131499, 0.012773956, 0.005825555, -0.0012852808, -0.02919401, 0.016325431, 0.0051064286, 0.04723525, 0.0036965623, 0.026796922, -0.0040182765, 0.043046657, 0.017359965, -0.015341364, -0.002999514, -0.024690008, 0.013751715, -0.030178078, 0.010730123, 0.01178358, 0.01178358, -0.019958913, 0.00617881, 0.035022717, -0.025636226, 0.01983275, -0.015063806, 0.014609621, 0.031061215, 0.02501803, -0.019757053, 0.0033780017, 0.009228789, -0.011871894, -0.023075126, -0.00035522637, 0.056318954, 0.007443589, 0.034013417, 0.010414717, 0.013587705, 0.032575164, -0.01862159, -0.024702623, -0.0006891628, 0.0032281838, 0.009828061, -0.011342011, -0.013877878, -0.027276339, 0.0038511113, 0.026494132, 0.0099857645, -0.021081759, 0.06232429, -0.014407761, -0.02094298, 0.021561177, 0.013575088, 0.02100606, -0.017233802, -0.010477798, 0.032171447, 0.03734411, -0.0035987862, -0.0029821668, 0.026267039, -0.0133479955, 0.008433965, 0.00818164, 0.022280304, 0.04186073, 0.07953286, 0.0039488873, 0.0610122, 0.01942903, -0.0042611393, -0.043021422, -0.053492915, 0.011386168, -0.02798285, -0.01761229, -0.0019097853, -0.015290899, -0.0017804687, 0.00044905976, 0.01888653, 0.0069957124, 0.015896479, 0.035905857, 0.035804927, 0.012313463, 0.023188673, -0.009746055, 0.0071597234, 0.0068190848, -0.03696562, -0.003191912, -0.013310147, -0.018873915, -0.0013065707, 0.033660162, 0.037091784, -0.011127535, 0.0056079244, 0.029925752, 0.019580426, -0.019794902, -0.012527939, 0.017006708, 0.026039947, 0.026569828, -0.011852969, 0.031767726, -0.039615035, -0.0048004845, 0.015290899, -0.0011938129, -0.017751068, -0.01791508, -0.0054439134, -0.043298982, -0.025674075, 0.021056527, 0.03310505, 0.005229437, 0.017246418, 0.04347561, -0.010143467, 0.03424051, -0.010093003, -0.042062588, -0.02205321, 0.027907152, 0.0078031523, 0.033407837, 0.061062664, 0.0073678917, -0.00051490084, -0.00609365, -0.020993445, 0.008610593, -0.019340716, 0.016956244, -0.029042615, 0.014268982, 0.04869874, 0.029875288, 0.025156809, -0.003138293, -0.010282246, -0.0016795387, 0.022116292, 0.021106992, -0.016968861, 0.024841402, 0.004980266, -0.0036555594, 0.023491463, 0.014218518, -0.010805821, 0.018356647, -0.011802505, -0.034997486, 0.024626926, 0.007084026, -0.022343384, 0.0023734325, -0.0336097, -0.021258386, 0.020993445, 0.0037154865, -0.00959466, 0.016552525, -0.0012356043, 0.026897851, 0.010919367, -0.015013341, -0.0013523046, 0.0063806702, 0.009216173, -0.016073106, 0.009714515, -0.013549856, -0.010881518, 0.005677314, -0.02124577, 0.029673427, -0.023150824, 0.005841325, 0.025610995, -0.023743788, 0.0011409824, -0.00568993, -0.053795703, -0.03532551, -0.023466231, -0.0122503815, -0.0004711382, 0.031061215, 0.017826766, -0.0066361492, -0.008875534, 0.004904568, -0.00946219, 0.0009564697, -0.010465182, 0.02535867, 0.0056205406, 0.0047594816, 0.004078204, -0.011796197, 0.038000155, 0.0046080863, 0.0073426594, 0.02798285, 0.009746055, 0.011342011, -0.001569935, -0.015543224, 0.045569908, 0.010048846, 0.0040561254, -0.01912624, -0.014483458, 0.0426934, 0.029320173, -0.00959466, 0.021119608, -0.0050654258, 0.026796922, 0.023327451, -0.0003221087, -0.013499391, 0.032802258, 0.022545245, 0.010919367, -0.03633481, 5.9532944e-05, -1.0453995e-05, -0.015126888, -0.020577108, 0.01609834, 0.0017915079, -0.034619, 0.0284875, -0.0073742, 0.00079955504, -0.0012805497, -0.0031114835, 0.003936271, -0.015404445, 0.019365948, -0.02118269, -0.021889199, -0.019567808, 0.0133479955, -0.0070020203, 0.008061786, -0.0041475934, -0.048572574, 0.0053902944, 0.01050303, -0.03090982, -0.026292272, 0.015694618, 0.0066550737, -0.038075853, -0.009449573, 0.0109004425, -0.03302935, 0.005734087, 0.0066992305, -0.04100282, 0.02562361, 0.011720499, 0.023112975, -0.0113735525, 0.0056047705, 0.00031974315, 0.011695267, -0.052988265, 0.016779616, 0.025989482, -0.009525271, 0.0073678917, -0.0076454496, 0.04700816, 0.007613909, -0.0055953083, 0.02148548, -0.02098083, 0.0061693476, -0.031162145, 0.0045639295, 0.04448491, -0.020728504, 0.00313987, -0.0043116044, -0.027806222, 0.025308203, -0.0027077633, 0.006623533, 0.018974844, 0.000684826, 0.02259571, 0.034164812, 0.016981477, -0.041835494, -0.028462267, -0.01916409, -0.028134244, 0.012477474, -0.013827413, 0.0072985026, -0.009310795, -0.020286934, -0.0029033152, 0.03482086, 0.00040115742, 0.009771288, -0.011619569, -0.0005452587, 0.012666718, 0.055662908, -0.02879029, -0.02400873, 0.008257338, -0.007109259, 0.024954949, -0.039816894, -0.005718317, -0.041305613, -0.036486205, -0.02687262, -0.0013783256, -0.017095024, -0.015139503, 0.00747513, -0.014420377, -0.0213467, 0.022343384, 0.033988185, -0.0034316208, -0.0022614633, 0.029446336, -0.03199482, 0.007096642, 0.0086358255, 0.012035905, 0.0068569337, 0.006159886, 0.02768006, -0.00027322074, -0.0012813382, -0.04751281, -0.0067812363, 0.0026699146, 0.0012419124, 0.030329473, -0.021207921, 0.014104971, -0.031364005, 0.015909094, 0.0062450455, -0.006579376, 0.0056173867, 0.00627974, -0.037041318, 0.0064059026, -0.01114646, 0.0063648997, -0.0336097, 0.0021652645, -0.028865987, 0.025989482, 0.016211886, -0.021397166, 0.029824823, -0.030329473, -0.0006607763, 0.033306908, -0.009531579, 0.0305061, -0.035375975, 0.024803553, -0.017826766, -0.0013018396, -0.022961581, -0.034391906, -0.0025516371, 0.009209865, -0.021119608, 0.0066361492, -0.0028370798, -0.01333538, 0.015278283, 0.007992396, 0.029723892, 0.0029616654, 0.013764332, 0.015946943, -0.0121873, 0.019101007, -0.028411802, 0.015467526, -0.030707961, 0.02899215, -0.022822801, -0.016640838, 0.016237117, -0.028083779, -0.04869874, 0.048774436, -0.012710875, 0.026721224, 0.030632263, 0.054199424, 0.018936995, -0.00777792, -0.045418512, -0.03199482, 0.023150824, -0.02185135, -0.008093326, -0.012805497, -0.039211314, 0.017864615, -0.023024661, -0.010282246, -0.02568669, 0.008251029, 0.016918395, -0.006623533, -0.009973148, -0.00022354424, -0.02148548, 0.002646259, 0.0073489673, 0.023049895, 0.010477798, 0.053089194, -0.021170072, -0.01807909, 0.023743788, -0.03141447, -0.00041278804, -0.0004289526, 0.0067118467, -0.007960856, 0.005147431, -0.003680792, -0.02350408, -0.017158104, 0.002690416, 0.04039724, 0.036990855, -0.040927123, 0.0026304887, 0.017044557, -0.0064721378, -0.0020028302, -0.008358268, 0.013019973, 0.010717507, 0.00936126, -0.011499715, 0.0003368934, 0.025270356, 0.0026557213, 0.024525996, 0.012736107, -0.030077148, 0.0022346538, 0.018104324, -0.013562473, 0.03565353, 0.010395792, 0.007033561, -0.039943058, -0.002540598, -0.009884834, -0.0110455295, -0.012521631, 0.022784954, 0.0006580165, -0.00479733, 0.0043052966, 0.023756405, 0.00066274754, -0.027074479, 0.009853294, 0.040220615, 0.008755679, 0.022065826, -0.0049550333, -0.034543302, 0.018142171, -0.0059075607, 0.006737079, 0.024525996, -0.013890495, -0.0120674465, 0.007815769, 0.00797978, 0.004138131, -0.03522458, -0.029976217, -0.0030105533, 0.03744504, -0.034417138, -0.010553496, -0.003232915, 0.025144193, -0.014773632, -0.05737872, 0.0084528895, 0.010944599, 0.015303515, 0.037192713, -0.004049817, 0.026115645, 0.021397166, 0.019782284, -0.005144277, -0.016350664, -0.0009288716, -0.021725187, -0.015883863, 0.030682728, -0.003841649, 0.010181316, 0.012868579, 0.030354705, 0.0012103717, -0.024097044, -0.028109012, -0.011638493, 0.009323411, 0.012130528, 0.015480143, 0.008301495, -0.0009454305, -0.0163002, 0.015530608, -0.0069452473, 0.026039947, -0.0060211066, -0.022444313, 0.038202014, -0.03149017, -0.008509663, -0.0013854223, -0.013398461, 0.0027487662, -0.025939016, 0.0067433873, 0.041658867, 0.0057403953, 0.0005357965, -0.016022641, -0.009910067, 0.012792881, 0.005932793, 0.008421349, 0.04723525, 0.0057246247, 0.024740472, -0.0027219567, 0.0067686196, 0.0044188425, 0.001316033, -0.037924457, 0.008219489, 0.010080386, 0.02255786, -0.034417138, -0.035073183, -0.021624258, -0.022015361, 0.0077526877, -0.0073426594, 0.026090411, -0.007468822, -0.030354705, 0.016073106, -0.025800237, 0.015089039, -0.0153918285, 0.03323121, -0.019239785, 0.01333538, -0.02909308, -0.00033511923, -0.008604284, -0.027957616, -0.00228985, 0.013713867, 0.020425715, 0.03310505, 0.011588029, 0.0066550737, -0.026241807, 0.005383986, -0.023478847, -0.014079738, 0.008301495, -0.01155018, 0.022507396, -0.009323411, 0.008320419, -0.0071029505, 0.011348319, -0.0253839, 0.009424341, 0.019101007, -0.021561177, -0.011770965, -0.011133843, 0.011878203, -0.012313463, -0.019403797, -0.018205253, 0.04440921, 0.007285886, -0.0026604524, 0.021535944, -0.008061786, 0.023529312, 0.0017710065, -0.012174685, 0.03898422, 0.022166757, -0.010925675, -0.0062198127, -0.03426574, -0.032070514, -0.003942579, -0.011272622, -0.013411077, -0.0029112005, -0.045670837, -0.0052105123, 0.03694039, 0.014420377, 0.02586332, 0.028159477, -0.03222191, 0.0620215, -0.009317103, 0.020148156, 0.004387302, 0.03202005, -0.020652806, 0.00747513, 0.038580503, 0.04511572, 0.0005058329, 0.004652243, 0.02023647, 0.014786249, 0.011808813, 0.023743788, -0.0239078, -0.011316779, -0.022961581, -0.05551151, 0.009544196, 0.019807518, 0.022974197, 0.019706588, -0.0018782447, -0.01918932, -0.0024081273, -0.009884834, 0.034593765, 0.0011520216, -0.02199013, -0.023377918, 0.027654827, 0.009979457, 0.010622885, -0.017221186, -0.03214621, -0.02521989, -0.008459197, -0.0035010101, -0.0019570962, -0.0055385353, 0.031338774, -0.0058192466, -0.034492835, -0.021977512, -0.006718155, 0.0049581877, 0.012363928, 0.022633558, 0.012244074, -0.03149017, -0.020148156, 0.00053461373, -0.021460246, 0.021573793, -0.025270356, 0.026317503, -0.043021422, 0.017511358, -0.011449249, 0.014609621, -0.009241405, -0.018369265, 0.052483615, 0.0028701976], "content_hash": "7df61a5f96fe8df2c86624bb74e00b8a"}, {"content": "- Another might spread attention broadly across the sentence GPT-2 has **12 attention heads per layer** (144 heads total across 12 layers). The dashboard categorizes these heads to help you understand their roles. ## What You See in the Dashboard In **Stage 3 (Attention)** of the pipeline, you can see:\n\n- **Head Categories**: Heads are grouped by pattern type -- Previous-Token, First/Positional, Bag-of-Words, Syntactic, and Other. Click each category to see which specific heads (like L0-H3) belong to it.\n- **BertViz Visualization**: An interactive attention map. Lines connect tokens on the left to the tokens they attend to on the right. Thicker, darker lines mean stronger attention. You can click on specific heads to focus on them.\n\n**Want more technical detail?** Ask the chatbot about Q/K/V dot products, softmax normalization, or how attention scores are computed mathematically.", "source_file": "attention_mechanism.md", "chunk_index": 1, "embedding": [-0.0076232045, 0.018918902, 0.033664517, 0.025512695, -0.020894257, -0.042261485, -0.0077692694, 0.0412599, 0.00879868, 0.013931823, 0.021200297, -0.008638704, -0.0023683405, -0.014070933, -0.011893868, -0.019252764, -0.006266886, 0.011615649, 0.0060164887, 0.0351669, -0.034415707, 0.00632253, 0.027390674, 0.06972171, 0.025206653, 0.035834625, -0.001527597, 0.02821142, 0.0057104477, 0.01585849, 0.005654804, -0.02325912, -0.0261526, -0.011031388, 0.0022083644, -0.0012685054, -0.011726936, 0.067217745, -0.032356884, 0.037003145, -0.011100943, -0.010363663, -0.029685982, 0.025554428, -0.034666106, -0.00036320637, 0.02049084, -0.019809203, -0.06326703, 0.07228133, -0.025540518, 0.053390253, -0.03661364, 0.027001167, 0.030993612, 0.009285564, -0.031911734, 0.021450695, 0.033358473, -0.017138299, 0.008290931, -0.0034794782, 0.01918321, 0.048159733, -0.014787347, -0.037281364, -0.086247936, -0.0026987256, 0.0050044665, 0.0077553582, 0.05144272, 0.054558773, -0.0056339377, 0.004576705, 0.03733701, 0.007991845, 0.014077888, 0.009633337, -0.0029247787, -0.039061967, 0.049940336, -0.027418496, -0.0070771994, -0.039729692, -0.032329064, -0.07361678, -0.029546872, -0.05714621, -0.0511645, -0.043875158, -0.030186776, 0.020337818, -0.017180031, 0.0009850697, 0.0070250332, 0.011483495, 0.05122014, -0.022813968, -0.016498394, -0.010780992, 0.03282986, 0.015635915, -0.045182787, -0.009382941, 0.017806025, -0.040397417, 0.025067544, 0.004110688, -0.019322319, -0.05216609, -0.08057226, 0.024385907, -0.005800869, 0.027599338, 0.068553194, 0.0034307898, 0.004747114, 0.0032499472, 0.020449106, -0.010168909, -0.012269464, -0.012408573, 0.034638282, 0.015538539, -0.01198429, 0.008367441, -0.0031716982, -0.029546872, 0.006840713, -0.020184798, -0.007456273, 0.017124388, 0.034860857, -0.01221382, 0.0128189465, -0.011733892, -0.017680826, 0.027557606, -0.040675636, -0.025651805, -0.0064477287, -0.019572716, 0.007588427, 0.035723336, -0.034415707, 0.018807614, -0.055449072, 0.059038103, -0.06454684, 0.038978502, -0.023926847, 0.006604227, -0.06410169, -0.031049255, -0.036557995, -0.015886312, -0.0070076445, 0.040759105, -0.018223353, -0.032941144, 0.028016668, -0.024232887, 0.027947113, 0.006520761, -0.028600927, 0.044375952, -0.027446318, 0.0003662494, -0.036585815, -0.06237673, -0.022285353, -0.027710626, 0.011622604, -0.029546872, 0.019864846, 0.008986478, -0.059538893, 0.021854114, -0.0050601107, -0.04242842, -0.0053000744, -0.013187587, 0.015496806, 0.005905201, -0.0735055, -0.011928646, 0.012763303, 0.018765882, -0.020003956, 0.024761504, -0.0033455852, 0.016053244, 0.06922092, 0.03650235, -0.016707059, -0.058370374, 0.050663706, 0.017639093, 0.0033299353, -0.0016962673, 0.023078278, 0.0231061, 0.0473529, -0.03853335, 0.0026361262, 0.003456873, 0.0019701393, 0.003025633, -0.038338598, -0.031299654, 0.028378353, -0.050051622, 0.025248386, 0.07183618, -0.078235224, 0.006159076, -0.06465813, 0.034443527, -0.0036203265, 0.032440353, 0.02862875, 0.0046879924, 0.05887117, 0.029825091, -0.06465813, 0.04838231, 0.056645416, -0.008513506, 0.015649827, -0.009076899, 0.03157787, 0.003947234, 0.011379163, -0.012547683, -0.023175655, -0.05714621, 0.027432406, -0.028183598, 0.018362463, -0.045878336, -0.00952205, 0.013208454, 0.022813968, 0.021325497, 0.0430405, 0.009821136, -0.0018831958, -0.03836642, 0.04593398, 0.013403207, 0.05480917, -0.01801469, -0.049495183, 0.004336741, -0.03180045, 0.010787947, -0.014084844, 0.030492818, 0.0012702442, -0.016651416, 0.021116832, 0.021311587, -0.05032984, -0.008423084, -0.0005042722, -0.054113623, -0.020908168, 0.014801258, -0.06571536, -0.047185965, -0.052972924, -8.335706e-05, 0.029964201, 0.010989656, -0.046045266, -0.00029930292, -0.0018866735, -0.01761127, -0.02164545, -0.0051261876, 0.004910568, 0.011838225, 0.04807627, 0.033914912, -0.0006068655, -0.013006745, -0.004162854, -0.011532183, -0.0006112127, -0.009695937, 0.041482475, -0.008680437, 0.067774184, -0.0062738415, -0.017847758, -0.0034968667, 0.030659748, 0.033024613, -0.0020727327, -0.00916732, 0.036418885, -0.05330679, 0.014286553, 0.007588427, 0.01903019, 0.034554817, 0.0231061, 0.018278997, -0.012561594, -0.051081035, -0.0120468885, -0.038004734, 0.033692338, -0.0038081245, -0.031160543, -0.039368007, 0.015566361, 0.033553228, -0.0004477589, -0.009862868, -0.016554039, -0.032746393, 0.012088621, 0.03196738, 0.009779403, -0.012109488, -0.022410551, 0.0349165, 0.004357607, -0.028155776, 0.06571536, 0.0136675155, -0.036251955, 0.018793702, -0.030576283, -0.02064386, -0.00074640976, 0.044069912, -0.03780998, 0.022076689, -0.060039688, -0.01918321, -0.04317961, -0.0039646225, -0.05130361, 0.059872758, 0.006990256, -0.016067155, 0.017527806, -0.017249586, -0.024399819, -0.024691949, 0.014940368, -0.02645864, 0.0015719382, 0.023926847, 0.0019562284, -0.002413551, 0.008443951, 0.013243231, -0.03068757, -0.015218587, 0.033497583, 0.0060582217, 0.014411751, -0.009682026, -0.024469374, -0.0270429, -0.014481306, -0.0054913503, 0.017666915, -0.010391485, -0.0025717882, -0.040091377, -0.04827102, -0.006426862, -0.0064929393, 0.024052044, -0.051804405, 0.029519051, -0.013876179, -0.0022883525, -0.0041558985, -0.0115113165, 0.032162134, 0.033330653, 0.006600749, 0.0083535295, -0.069833, 0.00040972116, -0.0045141056, -0.016762704, -0.013848358, 0.013041522, -0.0155107165, 0.022424463, -0.012603327, 0.030659748, 0.0012563333, 0.055755116, 0.015302053, -0.011692159, -0.023064366, -0.03152223, 0.041204255, -0.0033977511, -0.027084634, 0.004465417, 0.008423084, -0.036251955, 0.049050033, 0.0025822213, 0.010794902, -0.010419306, -0.027752358, 0.035278186, 0.015315964, 0.0027787136, 0.00021725002, 0.044598527, -0.00094507565, -0.04031395, -0.018237265, 0.0068859234, -0.032607283, 0.0008224853, 0.055643827, -0.028740037, 0.015371608, -0.022271441, -0.008555238, 0.0631001, 0.0016962673, -0.016707059, -0.0027700192, -0.022410551, 0.039173253, 0.04509932, -0.007915335, -0.0045036725, 0.022480106, -0.0319952, -0.021158565, -0.018918902, -0.048799638, -0.009271652, -0.02122812, -0.07066766, 0.033052433, -0.0520548, -0.0069972114, 0.028016668, -0.005654804, 0.04593398, -0.058370374, -0.028879147, 0.07912552, 0.0074214954, 0.038199488, -0.025623983, 0.015093388, 0.01903019, -0.030492818, 0.020323908, -0.044654172, 0.012568549, -0.0071363207, -0.03171698, 0.034860857, 0.008179642, -0.02239664, -0.0038602906, 0.018918902, -0.033720158, 0.0024448507, -0.017430428, 0.017180031, -0.013417118, -0.0034447007, -0.050190732, 0.03241253, -0.008986478, 0.0053661517, 0.025763093, 0.020421285, 0.06415733, -0.01628973, -0.007504961, 0.023231298, -0.006040833, -0.038644638, -0.018668504, 0.04551665, -0.016247997, 0.00879868, 0.0058078244, -0.035667695, -0.06298881, -0.027752358, -0.024052044, 0.021158565, -0.009202098, -0.0033403686, 0.018585037, 0.02559616, 0.007852735, 0.041232076, 0.021395052, -0.007240653, 0.024886701, -0.045154966, -0.019252764, 0.0021840201, 0.02051866, -0.0200596, -0.002401379, 0.045294076, 0.014842991, -0.0034099233, -0.012443351, -0.013862268, -0.010384529, -0.0022500972, 0.0039298455, -0.018807614, -0.02561007, -0.019544894, -0.015622005, -0.03024242, -0.022382729, 0.021450695, -0.05614462, 0.0055643828, -0.009869824, -0.07133539, 0.00070119917, 0.0030134611, -0.033553228, -0.025053633, 0.02296699, -0.007400629, 0.013027611, 0.026792502, 0.015900224, 0.0051922644, 0.0030204165, 0.023050455, -0.021325497, 0.004093299, 0.0631001, 0.0150516555, -0.021909757, 0.029407762, 0.0053244187, -0.051720936, -0.01859895, -0.027001167, -0.007824914, 0.003828991, -0.021673271, 0.05583858, -0.029825091, 0.015788935, -0.047603294, -0.018932812, 0.04070346, 0.013180631, 0.008541328, 0.025512695, -0.034972146, -0.0209499, -0.023926847, 0.018807614, -0.04782587, 0.008304841, -0.008889101, -0.014258731, 0.022076689, 0.00798489, 0.0115878275, -0.027223742, -0.032329064, -0.014550861, 0.0005912157, -0.008847369, -0.007852735, 0.026486462, -0.02367645, -0.04131554, 0.016595772, 0.03271857, -0.017472161, -0.013591005, 0.034137487, -0.027696716, -0.028030578, 0.014411751, -0.014564772, -0.025554428, 0.017082656, 0.0016423623, 0.0042741415, -0.029101722, -0.013319741, -0.01585849, 0.015149032, -0.027668893, -0.008332663, 0.0038602906, 0.01641493, -0.018515484, -0.018779792, -0.043958623, -0.026096955, -0.047881514, 0.020003956, -0.041204255, 0.001193734, 0.0069485228, 0.012394663, 0.014328285, -0.009668115, -0.003505561, -0.0050044665, 0.011866046, 0.030325886, 0.022313174, -0.032356884, -0.009341207, -0.0025665716, -0.037170075, 0.013750981, 0.0432909, -0.0015658521, -0.023523428, 0.004055044, -0.0091255875, 0.010655793, 0.048159733, -0.028141866, -0.0013024133, -0.01061406, -0.018000778, -0.03547294, 0.018835435, 0.0015823713, -0.01759736, -0.009779403, -0.017346963, -0.013848358, -0.01527423, -0.044181198, 0.045989625, 0.0019423174, 0.0041802423, -0.039145432, -0.038032558, -0.015107299, -0.0023196521, 0.022563571, -0.043207433, -0.0012615499, 0.0033281965, -0.045433186, -0.019322319, 0.022521839, 0.009035166, 0.011288742, 0.010495817, 0.010801858, -0.07689977, 0.050691526, 0.03536165, 0.022327086, 0.043068323, 0.004187198, -0.022507928, -0.05631155, -0.024038134, 0.027056811, -0.013486672, 0.024344174, -0.047603294, 0.0004864488, 0.03082668, 0.005202698, 0.02645864, -0.023147833, 0.025679626, 0.0046393042, 0.0032238644, -0.015009922, 0.033998378, -0.013500583, -0.017166121, 0.006259931, -0.01830682, 0.06326703, -0.012881546, -0.01773647, -0.020143066, -0.008673482, -0.015580271, -0.007908379, -0.0050149, -0.005418318, 0.026416907, 0.07261519, 0.010203687, -0.008624793, -0.03764305, 0.008423084, 0.0057800026, 0.0179034, -0.008749992, 0.03166134, -0.022507928, -0.0360572, 0.030131133, -0.019600539, -0.017513895, 0.0371979, -0.006280797, 0.00792229, 0.017138299, -0.02284179, -0.020838613, -0.046295665, -0.013938779, 0.017708648, 0.010148043, 0.039228898, -0.0029195622, -0.051943514, -0.009723759, 0.032217775, -0.040341776, 0.0010494078, -0.029463407, -0.041232076, 0.0340262, -0.022118421, -0.0034168789, 0.011427851, 0.019976133, 0.009320341, 0.0039576674, 0.0033925346, -0.036140665, -0.057368785, 0.0011815618, 0.012985879, 0.00038646377, 9.4496696e-05, -0.0030604105, 0.0071571874, 0.00915341, -0.025039723, 0.055616006, 0.026486462, 0.01309021, -0.003317763, -0.018849347, 0.038449883, 0.01801469, -0.04554447, -0.0020605605, 0.022104511, 0.0159976, 0.0100437105, 0.033942733, -0.013069344, 0.019141477, 0.006082566, 0.04201109, -0.045627937, 0.039201077, -0.014856902, 0.015594183, 0.0034777392, -0.018946724, -0.0046879924, -0.001133743, -0.035445116, 0.00290739, -0.0067711584, -0.022883523, -0.009925468, -0.0040654773, -0.021144655, -0.019906579, -0.019572716, 0.0012685054, -0.032217775, -0.010481906, -0.00040819964, 0.009334252, -0.10104919, -0.046573885, 0.0091255875, -0.030186776, -0.02092208, -0.009062989, 0.055643827, 0.011441762, 0.032301243, -0.020101333, 0.025832647, -0.007456273, -0.008339618, -0.008110088, 0.025234476, -0.050107267, 0.0057382695, 0.01832073, 0.028823502, -0.017472161, -0.03282986, 0.001287633, -0.012366841, 0.011344385, -0.039813157, 0.010523639, -0.0059191124, -0.0022605306, 0.023732092, 0.02239664, -0.019002367, 0.018487662, 0.00020018737, 0.032078665, 0.008416129, 0.023175655, -0.017708648, 0.009932423, 0.047853693, 0.0034064455, 0.0050879326, -0.031494405, -0.00632253, -0.02195149, 0.006249497, 0.009779403, -0.01221382, -0.04098168, 0.011810402, 0.0311049, -0.005345285, -0.0001862764, 0.0016554039, 0.025679626, 0.036001556, 0.012185998, -0.008256153, -0.0100298, 0.037837803, 0.00289174, -0.012672882, 0.003881157, 0.024552839, -0.010621016, 0.019141477, -0.007831869, -0.002321391, 0.04607309, -0.009529006, -0.0063712182, -0.03522254, -0.0063260077, 0.007574516, -0.005028811, 0.0031386595, -0.008270063, 0.012881546, 0.01773647, 0.021603717, -0.0070215557, 0.040953856, -0.024956256, -0.04479328, 0.021339407, 0.01287459, 0.014008334, 0.023467785, -0.033052433, 0.016985279, 0.019169299, -4.529212e-05, -0.01992049, -0.008673482, 0.024302442, -0.021715004, 0.0036307597, 0.020309998, 0.021311587, 0.06321139, 0.0018988457, 0.04854924, -0.0005916504, -0.022605306, -0.035584226, -0.056923635, -0.005439184, -0.00071989204, -0.009118632, -0.0009415979, -0.03622413, 0.008687393, 0.002253575, 0.024024222, 0.014523039, -0.005613071, 0.033720158, 0.077289276, 0.008242242, 0.0051853093, -0.012408573, 0.014091799, -0.0054739616, -0.027307209, 0.009452495, -0.0047088587, -0.0029960724, -0.023954667, 0.029407762, 0.0073936735, -0.019670093, -0.0050844545, 0.011073122, -0.007664937, 0.00059338927, 0.0007555388, 0.0051400987, 0.01845984, 0.005439184, -0.021742826, 0.033775803, -0.024942346, -0.011525228, 0.036724925, 0.020407373, -0.0136675155, -0.008492639, -0.002933473, -0.01830682, -0.01287459, 0.010071533, 0.031466585, 0.007477139, 0.016025422, 0.02630562, -0.0011128766, 0.028475728, 0.0036029378, -0.028044488, -0.018682415, 0.028545283, -0.0035281663, 0.030353708, 0.047853693, 0.00036494527, 0.010822725, 0.00719892, -0.025220565, 0.005776525, -0.029936379, 0.038143843, -0.054976102, 0.021826291, 0.04070346, 0.048771814, 0.002293569, 0.025554428, -0.006788547, 0.005856513, 0.034137487, 0.015093388, 0.002519622, 0.018084245, -0.02007351, 0.012039933, 0.048771814, 0.015079477, -0.007477139, 0.02021262, -0.01438393, -0.014787347, 0.01024542, 0.009515095, -0.018348552, 0.023732092, -0.014550861, -0.035389476, 0.03792127, 0.01127483, -0.020254353, 0.01761127, -0.0034690448, 0.01293719, 0.007191965, 0.00027126365, -0.0049175234, 0.024872791, -0.009563783, -0.0241216, 0.0077136257, -0.015455073, -0.008075311, 0.031605694, -0.021200297, 0.010913146, -0.015955867, 0.020699503, 0.0471025, 0.0048375353, 0.014147443, -0.0010433218, -0.046712995, -0.009292519, -0.03575116, 2.1953228e-05, 0.019364052, 0.02106119, 0.014982101, -0.025067544, -0.020421285, 0.011226142, -0.0068372353, 0.003053455, -0.0007225003, 0.012526817, 0.008513506, -0.019224942, 0.0060373554, 0.0002851746, 0.015162943, 0.008165732, 0.017374786, 0.02893479, 0.018432017, 0.0083535295, 0.00013084993, -0.022660948, 0.0115113165, -0.0024587617, -0.008332663, -0.02164545, 0.0008424823, 0.014912546, 0.02559616, -0.010154999, 0.00065338024, -0.0130484775, 0.034693927, 0.007880557, 0.015455073, 0.0045106276, 0.014286553, 0.048465773, -0.00018040772, -0.022869613, 0.007226742, -0.0013850096, -0.017180031, -0.0013824013, 0.03241253, 0.016623594, -0.043819513, 0.035862446, -0.0049140453, 0.00057165336, 0.000979853, 0.009848958, 0.035834625, -0.02876786, 0.021603717, -0.007331074, 0.0011772147, -0.0064442507, -0.0030099833, -0.011685204, 0.006816369, -0.0029021734, -0.043346543, -0.0072336975, 0.021200297, -0.02776627, -0.012325108, -0.009007345, 0.013333652, -0.012290331, -0.0070458995, 0.017541716, -0.009786358, 0.004319352, 0.022591393, -0.05060806, 0.018432017, 0.011615649, 0.027154189, -0.019155387, 0.0054044067, 0.011330474, 0.01775038, -0.044292487, 0.0029978112, 0.0058704237, 0.017027011, 0.014926457, -0.016247997, 0.033080254, 0.012700704, -0.004823624, 0.018793702, -0.020296086, 0.01322932, -0.016540127, 0.0056513264, 0.044542883, -0.029936379, -0.0008733473, 0.013104121, -0.026862057, 0.00509141, -0.008840413, 0.010711437, -0.011747803, 0.03505561, 0.015218587, 0.00021257681, 0.014801258, -0.06655002, -0.0239964, -0.003561205, -0.021395052, 0.0045627938, -0.020296086, 0.014161354, 0.007762314, -0.0054878723, -0.0048688347, 0.044542883, 0.0040724324, 0.01773647, -0.014884723, 0.0063433964, -0.0028743516, 0.05208262, -0.018292908, -0.026486462, 0.019739648, -0.007824914, 0.03722572, 0.009904602, -0.016623594, -0.03327501, -0.011191364, -0.006496417, 0.00045949628, 0.0052166088, -0.018098155, -0.0016693148, -0.010099354, -0.023147833, 0.0064616394, 0.04028613, -0.011469584, -0.0048966566, 0.016512306, -0.03736483, 0.019739648, 0.028461818, 0.016957456, 0.020017866, 0.004385429, 0.012283375, -0.0011285264, -0.0071641426, -0.048604883, 6.950044e-05, 0.017541716, -0.0046984255, 0.032495994, -0.007191965, 0.029797269, -0.010342796, 0.0023822514, 0.005150532, 0.021297675, 0.013716203, 0.012332063, -0.01583067, -0.005574816, 0.011080077, 0.014272641, -0.015135121, -0.028280975, -0.033052433, 0.019224942, 0.008040533, -0.0026969868, 0.023954667, -0.032440353, -0.017945135, 0.018515484, 0.00012519861, 0.028002756, -0.038171664, 0.0120468885, -0.0073936735, -0.0034064455, -0.0190441, -0.043791693, -0.004319352, -0.015343785, -0.012958056, 0.02776627, 0.011504361, -0.018098155, 0.0019040622, -0.011942556, 0.034054022, 0.0049662115, 0.00071597955, 0.0005699145, -0.009132544, 0.029491229, -0.019753559, 0.007070244, -0.03152223, -0.007657982, -0.005762614, -0.009647248, 0.024719771, -0.029324297, -0.033191543, 0.024358086, -0.003081277, 0.015733292, 0.010175865, 0.05786958, 0.018835435, -0.009382941, -0.008158776, -0.02442764, 0.011886912, -0.012380752, -0.0027039421, -0.031633515, -0.018418107, 0.01527423, -0.018390285, -0.016108887, -0.021575894, 0.016762704, 0.018376375, -0.010516684, 0.01453695, 0.009828091, -0.0040446105, 0.003161265, 0.008562194, 0.012352929, -0.0060616992, 0.046712995, -0.033692338, -0.0084300395, 0.029908556, -0.012561594, 0.020727325, 0.002613521, 0.0063329632, 0.0147456145, 0.0064477287, -0.01759736, -0.033219364, -0.046852104, -0.011212232, 0.0418998, 0.028044488, -0.01344494, -0.0058495575, 0.024399819, 0.003933323, -0.018710237, -0.03764305, 0.030325886, 0.022633126, 0.020546483, 0.011330474, -0.019322319, 0.021589804, -0.0125129055, 0.022327086, 0.012102532, -0.0128189465, -0.00054730923, -0.0030169387, -0.017499983, 0.0006864188, 0.02149243, 2.4738138e-05, -0.052110445, -0.006924179, -0.004823624, -0.029157365, -0.010940967, 0.023189565, -0.009570738, -0.02876786, -0.009396851, 0.024997989, 0.012575505, 0.0013250186, 0.0024744114, 0.013625782, 0.01583067, 0.033831447, -0.027488051, -0.026556017, 0.016011512, 0.023746002, -0.0015041222, 0.0041385097, 0.02108901, -0.010036755, 0.019419696, -0.030604105, 0.02051866, -0.031299654, -0.045878336, 0.010008933, 0.011956467, -0.0371979, -0.014098755, -0.0018292909, 0.023231298, 0.0061799427, -0.043430008, 0.0018153798, -0.013375385, 0.00090334273, 0.051720936, -0.0030638883, 0.04390298, 0.031077078, 0.002521361, -0.014606505, -0.013709248, -0.026333442, -0.0023752959, -0.032440353, 0.023662537, -0.009320341, -0.005745225, -0.0006825063, 0.0069276565, -0.0075258277, -0.016860079, -0.049161322, 0.009118632, 0.004041133, -0.0100506665, 0.01512121, 0.0041524204, 0.01024542, -0.04713032, 0.029602516, 0.00079118565, 0.035278186, -0.008339618, -0.015232498, 0.02558225, -0.020532573, -0.020838613, 0.00270742, -0.02515101, 0.010447129, -0.05583858, 0.01527423, 0.017666915, 0.01816771, 0.017388696, -0.02776627, -0.004823624, -0.006458162, 0.0018623293, 0.014189176, 0.044737637, -0.0119982, 0.017054833, -0.0022187976, 0.015218587, -0.011219187, 0.005094888, -0.026695127, 0.023231298, 0.016011512, 0.012172087, -0.037837803, -0.053891048, -0.02921301, -0.0074075847, 0.0014693448, 0.027641071, 0.013945734, -0.009626382, -0.06527021, 0.011379163, 0.0038150798, 0.0006764203, 0.018000778, 0.026055222, -0.011170498, 0.008221376, -0.03182827, 0.010760125, -0.02106119, 0.013855313, -0.0064199064, 0.013723159, 0.011302653, 0.060874347, 0.0033751458, 0.032162134, -0.021130743, 0.008158776, 0.020449106, -0.017040921, 0.020546483, -0.024010312, 0.009633337, 0.0018484184, -0.0016527956, 0.010224553, 0.034972146, -0.023787737, -0.012728525, 0.014084844, -0.014342196, -0.016957456, -0.009264697, 0.0116643375, -0.007873601, -0.010878368, -0.030798858, 0.024024222, -0.028461818, -0.017430428, 0.013827492, -0.016804436, 0.01046104, 0.024552839, 0.0038881125, 0.019795291, -0.0028360963, 0.0036168487, 0.0052444306, -0.022118421, -0.01818162, -0.004941867, 0.0015562883, 0.0134379845, 0.010996612, -0.029101722, 0.0038359463, 0.022507928, 0.0011041822, 0.017541716, 0.03806038, -0.0051122764, 0.043708228, -0.013159765, 0.0008446559, 0.0147456145, 0.03561205, -0.017138299, 0.009264697, 0.042233665, 0.049161322, -0.0015189026, 0.010920101, -0.008826502, -0.00073032524, 0.022452284, 0.023746002, -0.01889108, -0.022132332, -0.012081666, -0.03619631, -0.009779403, 0.006548583, 0.0351669, -0.018557217, -0.029046077, -0.0027022033, 0.008756948, -0.022299264, 0.007518872, -0.026709037, -0.012554638, 0.0016484484, 0.012492039, 0.008652615, 0.00981418, -0.020866435, -0.037142254, -0.041621584, -0.017291319, -0.003347324, 0.001921451, -0.021548072, 0.011803447, -0.018501572, -0.034276597, -0.009758536, 0.010001978, 0.002079688, 0.0073728072, 0.03806038, -0.001971878, -0.005400929, -0.011455673, -0.030715393, -0.027362851, 0.033914912, -0.015775025, 0.025373586, -0.02312001, 0.006823324, -0.009480317, 0.016915724, 0.0034186176, 0.0015388996, 0.04245624, 0.0038116022], "content_hash": "bbf918757a7cbeef7accb9f249d13fc6"}, {"content": "# Attribution Panel Guide\n\n## What Is Token Attribution?\n\nToken attribution measures **which input tokens had the most influence on the model's prediction**. It answers the question: \"Which parts of my input mattered most for this output?\"\n\nFor example, if your prompt is \"The capital of France is\" and the model predicts \"Paris,\" attribution will show that \"France\" and \"capital\" had the highest influence on that prediction.\n\n## How to Use the Attribution Panel\n\nThe attribution panel is in the **Investigation Panel** at the bottom of the dashboard, under the \"Token Attribution\" tab.\n\n### Step-by-Step\n\n1. **Run an analysis first**: Click \"Analyze\" on a prompt so the model has made a prediction.\n2. **Choose an attribution method**:\n   - **Integrated Gradients**: More accurate but slower. Computes attribution by gradually \"building up\" the input and measuring how each token contributes along the way.\n   - **Simple Gradient**: Faster but less precise. Takes a single gradient measurement to estimate token importance.\n3. **Choose a target token (optional)**: By default, attribution is computed for the model's top prediction. You can select a different token from the top-5 predictions dropdown to see which input tokens would drive *that* alternative prediction.\n4. **Click \"Compute Attribution\"**.\n\n### Reading the Results\n\nThe results have two visualizations:\n\n- **Color-coded token chips**: Your input tokens are displayed as colored boxes. Darker blue = higher influence. Hover over any chip to see the exact attribution score.\n- **Horizontal bar chart**: Shows the same information as bars, with attribution scores labeled. Longer bars = more influential tokens.\n\n### What Attribution Scores Mean", "source_file": "attribution_panel_guide.md", "chunk_index": 0, "embedding": [-0.0053097797, 0.009309273, -0.023023605, 0.0014335158, 0.028551767, -0.021625966, -0.015024618, 0.04143001, 0.0075871823, 0.011917368, -0.005727824, 0.01407622, -0.011599155, -0.0595494, -0.014750082, -0.02181315, 0.004065008, 0.0052473852, 0.004592242, 0.06993186, -0.039982457, -0.009384147, 0.04073119, 0.037811123, 0.009415344, -0.014063741, -0.018930519, -0.020839794, -0.010345023, 0.036188863, -0.00033654083, -0.016572004, -0.03544013, -0.04544822, -0.02630556, -0.044724446, -0.025095105, 0.0030292575, 0.029225627, -0.0043333047, -0.020615174, 0.030373689, -0.033218883, 0.030223941, -0.04559797, 0.0046203197, 0.0051756315, -0.0662381, -0.038984142, 0.06174569, -0.026480265, 0.043177057, -0.006021078, -0.0072876886, -0.024795612, 0.051413145, -0.017620234, 0.015848227, -0.024196623, -0.052810784, -0.015885664, -0.017021244, 0.04118043, 0.036313653, 0.034092404, -0.03197099, -0.04302731, 0.011156153, -0.006869645, -0.019766606, 0.05725328, 0.026180772, 0.04949139, 0.004951011, -0.037386842, -0.030248899, -0.024408765, 0.04187925, -0.032644853, -0.017595276, 0.0048262216, 0.0015941819, -0.019716691, -0.013477232, -0.00014789468, -0.027503537, -0.040531527, -0.02354772, -0.01606037, -0.07008161, -0.027453622, -0.0033443503, -0.02687959, 0.031471834, 0.018755814, 0.005250505, -0.027852947, -0.019254971, 0.004398819, -0.0053690546, 0.036413483, -0.028077567, 0.013102865, -0.01811939, 0.014725124, -0.02306104, 0.009839627, -0.021214161, -0.01073187, -0.004005733, -0.028327145, 0.038709607, -0.071030006, 0.051762555, -0.008504383, -0.03162158, -0.042752776, 0.04869274, 0.037436757, -0.04856795, -0.026155813, -0.028801344, -0.0015824828, 0.033992574, 0.011124956, -0.0035689708, -0.012616186, -0.028926134, -0.00565295, -0.018606069, -0.032170653, 0.0058682114, -0.034117363, -0.042278577, 0.0031244094, -0.032495104, 0.017245866, 0.039533213, -0.057652604, 0.0120920725, -0.0074249567, -0.018893084, -0.010326305, -0.03104755, -0.06474063, -0.014737603, -0.0032944346, 0.03266981, -0.035490043, 0.028476892, -0.0035720905, -0.012840807, -0.018481279, -0.039083973, -0.036762893, -0.03162158, 0.012553792, 0.03381787, -0.008211128, -0.06469072, -0.012372848, 0.030972676, -0.016646877, -0.020041144, -0.05805193, 0.029999321, -0.03349342, -0.011761381, -0.034541648, -0.013439795, -0.027952777, -0.041504882, 0.0083795935, -0.039158847, 0.030149067, -0.040656317, -0.0863042, -0.011667789, 0.011218548, -0.018992914, -0.030398646, 0.024733217, -0.013951431, -0.0011472806, -0.017570317, 0.0019482712, -0.018718379, 0.018618546, -5.288917e-05, -0.017882291, 0.04484923, 0.0013461633, 0.04821854, 0.019092746, -0.04085598, -0.019230014, 0.047270145, 0.016572004, 0.024882963, -0.002110497, 0.0041118036, 0.010594602, 0.0410806, -0.016796624, -0.018705899, 0.016996287, -0.011698986, -0.040431697, -0.039158847, -0.019654296, -0.0006013279, -0.046770986, 0.034092404, -0.0057902182, -0.07397503, -0.0066263056, -0.023260705, 0.038160533, 0.039782792, 0.033742998, 0.023210788, 0.016547047, 0.021987855, -0.033418544, -0.017520402, 0.03509072, 0.031022592, -0.019454634, 0.028776387, -0.03566475, 0.025931193, 0.027253957, -0.011355816, -0.020864751, -0.004355143, -0.037486672, 0.005300421, -0.01581079, 0.03324384, -0.06024822, 0.032869473, 0.013302527, 0.05121348, 0.01234165, 0.009072174, -0.004648397, -0.007225294, -0.03521551, 0.029450249, 0.037237093, 0.04292748, 0.0025441395, -0.032245524, 0.039208762, 0.023385493, 0.019217534, -0.006264417, 0.016097805, -0.0065514324, -0.022462053, 0.03406745, -0.02341045, -0.062294763, -0.020016186, -0.017932206, 0.011661549, -0.034916013, 0.007843001, -0.06983203, -0.037686333, 4.808771e-05, -0.009090892, 0.015062055, 0.0042896285, -0.02259932, -0.008473186, -0.024733217, 0.0058806906, -0.035739623, -0.005459527, 0.0018609187, 0.0018905562, 0.051313315, 0.03544013, -0.040406737, -0.017707586, 0.038010787, 0.04582259, 0.0138516, 0.004511129, 0.010008093, -0.03588937, 0.047345016, -0.0045204884, -0.021401346, 0.016359862, 0.01281585, 0.015087013, -0.019804044, 0.01038246, 0.008747722, -0.04327689, 0.0093467105, -0.04652141, 0.032969303, -0.02815244, 0.024209103, 0.028252272, 0.01719595, -0.030498477, -0.040756147, 0.003300674, -0.008117537, 0.033193924, -0.022087686, 0.0620951, 0.07018144, -0.011661549, 0.041030686, -0.0034192237, 0.013227654, -0.043127142, 0.015648564, 0.011106238, -0.002941905, -0.007930352, -0.014475546, 0.028576724, -0.011742663, -0.0042303535, -0.00015257428, 0.039907582, -0.014013826, 0.059599314, -0.05495716, -0.004763827, -0.0060304375, 0.0011020445, -0.0024302695, 0.040556487, -0.025456995, -0.032495104, -0.0102077555, -0.03069814, 0.012547553, -0.0011402612, 0.007899156, 0.04270286, 0.015224281, -0.030648224, 0.0135271475, 0.019641818, -0.0044736923, -0.04489915, 0.025706572, 0.026155813, -0.0069382787, 0.0059462045, 0.017632712, 0.0030136588, -0.027303874, 0.028726472, 0.035864413, 0.036213823, -0.0083172, -0.0507892, -0.023260705, -0.010981448, 0.010170319, -0.027902862, -0.020727484, 0.013327485, -0.027902862, -0.0442752, -0.037411798, -0.0088350745, 0.006532714, 0.012266777, -0.015374028, -0.01776998, 0.045048896, -0.057452943, 0.014450588, -0.026105898, 0.065639116, -0.00027746096, 0.024096793, -0.0053472165, -0.019517029, 0.030174026, -0.03034873, -0.008847553, 0.006021078, 0.034416858, -0.0034223434, 0.038135577, -0.00704435, 0.039957497, 0.05216188, 0.03429207, -0.02920067, -0.004592242, -0.04257807, 0.003104131, 0.037237093, 0.059150074, -0.008254805, 0.0031119303, 0.030598309, 0.030049236, 0.059749063, -0.053359855, 0.04245328, -0.053160194, 0.025257332, 0.03034873, 0.05355952, 0.014987181, 0.030797971, 0.024995275, -0.0033880265, -0.02840202, 0.013090386, -0.044824276, -0.030074194, 0.005865092, 0.030149067, -0.017682627, 0.023859693, -0.03359325, 0.030947719, 0.027428662, 0.07043102, -0.021264078, -0.007412478, -0.012372848, 0.062344678, 0.01222934, 0.0043676216, 0.0025207417, 0.016796624, -0.021588529, -0.010014332, -0.050814155, -0.065639116, -0.020390553, -0.02042799, -0.05530657, 0.022312306, 0.019878918, -0.010794264, 0.05585564, -0.021513656, 0.064940296, -0.05565598, -0.048493076, 0.021638446, 0.05146306, 0.044924106, -0.005840134, 0.051038776, 0.020278243, -0.012978075, 0.030947719, -0.0033911462, -0.018269137, -0.018518716, -0.017507924, 0.014500503, -0.00093981856, -0.010713152, -0.010033051, -0.009527654, -0.018156826, -0.004161719, -0.038534902, 0.018843167, -0.021850586, 0.0042303535, -0.040656317, 0.019279929, 0.012285495, -0.02792782, 0.0042615505, 0.005412731, 0.055556145, 0.01685902, -0.018443841, -0.009945698, -0.027977735, 0.0056997463, -0.054507915, 0.017532881, -0.012672342, 0.020615174, -0.041704547, -0.006407925, -0.012297974, 0.015723437, -0.0184688, 0.034491733, -0.013027991, -0.046995606, 0.061945353, 0.0066263056, 0.005050842, 0.061945353, 0.027079254, -0.029674869, 0.05241146, -0.05425834, -0.025182458, -0.019978749, -0.015548733, -0.036263738, 0.01164907, -0.024870485, 0.030623266, 0.02815244, 0.0066512637, 0.017408092, 0.03349342, -0.0148374345, -0.02677976, 0.021650923, -0.009721078, -0.008916187, -0.00080645015, -0.040207077, -0.024546033, 0.0050820396, -0.060198303, 0.0019373521, 0.022324786, -0.04085598, 0.013651937, -0.007281449, 0.012429003, -0.04547318, 0.0034566605, 0.041604716, -0.01582327, -0.016921414, 0.011667789, 0.0043894597, -9.003345e-05, 0.002225927, -0.04961618, -0.0046140803, 0.018980436, -0.00013171109, -0.036213823, 0.0057371827, -0.037661377, -0.016409779, -0.0145504195, 0.006251938, -0.027753115, -0.019941311, -0.04038178, 0.024558512, 0.0137517685, 0.016334904, -0.023984483, -0.009845867, 0.03299426, 0.038035743, 0.020989541, 0.01811939, 0.005493844, 0.019554466, 0.039433382, -0.00038489664, 0.022287348, 0.039832707, 0.0026377314, -0.018930519, 0.0062301, -0.001442875, 0.015760874, -0.015760874, 0.009721078, -0.01534907, 0.011736423, -0.03104755, -0.014338277, -0.0009959737, 0.008616693, -0.008254805, 0.010756828, 0.0024177905, -0.015723437, 0.01050101, 0.033069134, -0.0140387835, 0.040631358, 0.013514669, -0.020739963, -0.022786506, 0.016671835, 0.02827723, -0.040306907, -0.028826302, 0.0019576303, 0.015161886, 0.024321413, -0.019878918, 0.031147381, -0.012622426, 0.006311213, -0.008629172, -0.06938279, -0.014862392, 0.011243505, -0.009596288, 0.022561885, -0.012978075, 0.031921074, 0.009427823, -0.0030838528, 0.005066441, -0.011218548, -0.011836254, -0.0017517282, 0.0003184074, 0.0456978, 0.018056996, -0.019304886, 0.023585156, 3.5145706e-05, -0.039283637, 0.016596962, 0.029749742, 0.021176726, 0.01420101, 0.02283642, -0.0049416516, 0.008410791, -0.0009585369, -0.0038934227, 0.033418544, 0.030523434, -0.011611634, -0.017270824, 0.0019139542, -0.000911741, 0.019354803, 0.022287348, -0.030623266, -0.03174637, -0.014438109, -0.023023605, 0.01685902, -0.009153287, -9.517613e-06, -0.0049572503, 0.03187116, 0.022674195, -0.040656317, -0.0060179587, -0.0343919, -0.011642831, -0.02595615, -0.043526467, -0.02805261, 0.018518716, -0.002458347, 0.030223941, 0.036987513, -0.012029678, -0.10632039, 0.033742998, -0.008267283, 0.038010787, -0.0110064065, 0.016260032, 0.017782459, -0.033568293, -0.028726472, 0.054208424, -0.007811803, 0.009602528, -0.029050922, -0.01084418, 0.013776726, -0.005734063, 0.023959523, 0.01431332, 0.006270657, -0.039333552, -0.008441988, -0.010139122, 0.003403625, -0.0077369297, -0.04996559, 0.0070006736, 0.0143257985, 0.05630488, 0.0029029085, -0.059050243, -0.011954804, -0.018705899, -0.037386842, -0.02827723, -0.0050758, 0.023447888, -0.017632712, 0.017058682, 0.010045529, -0.041979082, -0.03177133, 0.012123269, 0.01338988, 0.00041960363, -0.023934565, 0.03553996, -0.014238446, 0.036912642, -0.014862392, -0.065738946, -0.034791224, 0.0059243664, -0.04799392, 0.021126809, 0.037861038, -0.027753115, -0.014562898, 0.008797638, -0.008248565, 0.030872844, -0.005340977, -0.01153052, -0.019928833, -0.040681276, -0.009914501, 0.010107924, 0.0022586843, 0.01084418, -0.0017174112, -0.007818042, 0.014176052, 0.0030713738, 0.011486845, -0.033767954, 0.038609773, 0.039707918, -0.04175446, 0.02550691, -0.029275544, -0.044574697, -0.0021214162, 0.009321752, 0.024283975, -0.018206742, 0.015486338, 0.0048168628, 0.009402866, -0.06329308, -0.008217368, 0.023148393, -0.008061381, 0.024945358, -0.0021338952, 0.007605901, -0.034841143, -0.022362223, 0.0487177, 0.025806405, -0.011337098, 0.02920067, -0.020615174, -0.007200336, 0.02655514, -0.015735917, 0.043751087, -0.014737603, 0.036138948, -0.0054657664, -0.022299828, 0.007780606, 0.011137435, 0.0030963316, -0.022911295, -0.0028951091, 0.01858111, -0.0054408084, 0.00854182, -0.029749742, 0.008098818, 0.0006418844, -0.020964583, -0.008354636, 0.006657503, -0.031471834, -0.003968296, 0.017957164, -0.0080801, -0.0671865, -0.03219561, 0.04467453, -0.027328832, -0.010681954, -0.017857332, 0.037711293, 0.0005677908, -0.026230687, 0.0022134483, -0.030024279, -0.0044268966, 0.0011667788, -0.0042896285, -0.031097466, -0.0102077555, -0.011698986, 0.006323692, 0.004639038, -0.007468633, -0.02319831, -0.005702866, 0.0032226807, -0.0118050575, -0.04717031, -0.009140808, -0.044000667, 0.021488698, 0.024296455, 0.01304047, -0.013951431, 0.036712978, 0.036837768, -0.01708364, 0.016260032, 0.040806063, 0.0035128156, 0.010463573, -0.016197637, -0.039882623, -0.006545193, -0.040681276, 0.008666609, -0.024633385, 0.082810104, -0.05760269, -0.018493757, -0.015735917, -0.02412175, 0.023160873, -0.016834062, 0.026180772, -0.001108284, -0.0070318705, 0.030548394, 0.015461381, 0.030997634, -0.016759187, -0.011505563, 0.022674195, -0.0067323768, 0.022000333, 0.0038684648, 0.018094432, 0.005612394, 0.02181315, -0.0040182117, -0.0046359184, -0.044025626, -0.023373015, -0.009134568, 0.024383808, 0.0010310706, -0.022898816, -0.03336863, 0.0038809436, 0.026954465, -0.007936592, -0.05226171, 0.010413658, 0.05919999, -0.0071940967, 0.011149914, 0.05033996, -0.005344097, 0.03346846, 0.0075996616, -0.013789205, -0.0068010106, 0.027528495, 0.047245186, -0.008710286, 0.0106070805, 0.00075653446, -0.0143257985, -0.01257875, 0.017857332, 0.049241815, 0.021975376, -0.015386507, -0.0067198975, 0.019404719, -0.05111365, -0.018793251, -0.008722764, -0.019292409, -0.018967956, -0.03197099, -0.022312306, -0.0031821243, 0.017969644, 0.043301847, -0.02447116, 0.00058884895, -0.016210115, 0.010438615, 0.034441814, 0.02598111, 0.025182458, -0.012366609, 0.020777399, 0.00095619715, -0.004096205, 0.0025534988, 0.0023444768, 0.039982457, -0.017720064, 0.0044019385, 0.01453794, 0.00900354, 0.029450249, 0.016322426, -0.00842951, -0.018144349, -0.046172, 0.0077244504, 0.0064141643, -0.013215175, -0.05390893, 0.006208262, -0.016247552, 0.029450249, 0.03012411, 0.015436422, -0.056754123, -0.009240639, -0.029375374, -0.04567284, 0.016522089, 0.017520402, 0.024483638, 0.03034873, 0.00083608757, 0.010869138, -0.033193924, 0.020552779, 0.025207415, -0.018294094, -0.0033849068, 0.040256992, 0.0023553958, 0.028576724, 0.02366003, 0.0021900502, 0.043801006, -0.017345697, 0.013764247, 0.009047216, 0.005241146, 0.015598648, -0.014700166, -0.010307587, 0.016696794, 0.021401346, 0.009820909, 0.028327145, -0.00022793525, 0.0051725116, -0.005340977, 0.034192238, -0.02665497, 0.019417198, -0.0221376, 0.012915681, -0.0032164413, -0.03301922, 0.000346095, -0.0066450243, -0.038509943, -0.013477232, 0.025544347, 0.024957838, 0.019367281, 0.028027652, -0.037811123, 0.009090892, 0.012029678, 0.007406238, -0.014987181, 0.014176052, -0.034366943, 0.0044393754, 0.027029337, 0.017620234, 0.024770653, 0.02412175, -0.015561212, -0.0010131322, 0.025294768, -0.010987688, -0.0044799317, 0.014587856, -0.03212074, 0.030997634, 0.00077291304, -0.008111297, -0.040556487, -0.006451601, -0.017819896, -0.0043239454, -0.038335238, -0.03256998, 0.004395699, 0.027977735, 0.0049416516, -0.004813743, 0.05206205, -0.015448901, 0.009989374, 0.0042178747, -0.017969644, -0.017819896, -0.014762561, 0.0014257164, -0.020228326, 0.008997301, 0.0038466267, -0.014263404, 0.034691393, -0.017645191, 0.03578954, 0.03212074, -0.0036968796, 0.007818042, -0.03346846, -0.03336863, 0.012329171, 0.0063642487, -0.014151094, 0.0013937393, 0.007293928, 0.07018144, 0.030723099, -0.042178746, 0.011380774, -0.008286002, 0.05600539, 0.013801684, 0.0065826294, 0.0073438436, 0.0052723433, 0.027253957, 0.018955477, -0.0074935905, 0.008816356, -0.012716018, -0.023223268, 0.0045298473, 0.011929846, 0.011917368, -0.017220907, 0.017882291, 0.0021136168, 0.01627251, -0.0023522761, -0.008697806, 0.0015590849, -0.034940973, -0.008760201, -0.010401179, -0.015486338, 0.024907922, -0.031322084, 0.011611634, 0.011274703, 0.015498817, 0.02191298, 0.0060616345, -0.022474533, 0.0020824196, 0.0011020445, 0.015136928, 0.018306574, 0.013901516, 0.01211079, 0.005759021, 0.0029075881, 0.007200336, -0.005843254, -0.023797298, 0.042727817, -0.0203032, -0.018830689, -0.012865765, 0.015785832, 0.015461381, 0.025706572, 0.00055219216, 0.003971416, 0.0019420317, -0.03222057, -0.0009047216, -0.031322084, 0.037486672, -0.013602021, -0.020939626, -0.016983809, -0.058551088, -0.010900335, -0.014263404, 0.0082048895, -0.004002613, 0.0016908934, 0.025007753, 0.011050083, -0.0140887, 0.014475546, 0.01200472, -0.008048902, 0.010806744, -0.019729171, -0.016309947, 0.011025124, -0.0019108345, -0.042727817, 0.012940639, -0.016572004, -0.019928833, 0.0038372674, -0.06269409, 0.01131214, 0.0052099484, -0.0002942295, -0.016896456, 0.02907588, 0.012510116, 0.02191298, -0.008123776, 0.020353116, 0.031322084, 0.0671865, -0.02294873, 0.02087723, 0.005150674, 0.0046951934, 0.026704885, -0.02294873, 0.017645191, -0.015636085, 0.011343337, 0.0069258, 0.040232033, 0.010557165, -0.015274197, -0.015611127, 0.015611127, 0.0074811117, -0.0025379, 0.04200404, 0.018793251, 0.00888499, 0.018418884, -0.0047139116, 0.002516062, 0.014375715, 0.0019342324, 0.0066075874, 0.019417198, 0.014413151, 0.008030185, 0.01187993, -0.028701512, 0.02815244, 0.01280337, -0.024209103, 0.021663403, 0.0042303535, 0.034466773, -0.006198903, -0.02331062, 0.012241819, 0.008048902, 0.010993928, 0.0015185284, -0.02712917, 0.016010452, -0.004327065, -0.016659357, -0.0020465427, -0.040556487, 0.003890303, -0.0015037097, -0.019654296, -0.02493288, 0.01604789, -8.20757e-06, 0.008448228, -0.016322426, -0.008011466, -0.019554466, 0.00028682014, 0.02343541, -0.037985828, 0.0042646704, 0.039433382, -0.033518374, -0.019404719, -0.0059867613, -0.0034597802, 0.0030323772, -0.028726472, -7.550721e-05, -0.026530182, 0.012179425, 0.023335578, 0.006751095, 0.001026391, -0.020702526, 0.024096793, -0.03426711, -0.016609441, -0.008573017, -0.027877904, 0.003587689, -0.042952437, -0.02655514, 0.031347044, -0.001984148, -0.025544347, 0.04627183, -0.0046764747, 0.008947385, 0.0039308593, 0.022898816, -0.0069757155, -0.005300421, -0.0022586843, 0.006563911, 0.0074998303, 0.0045984816, 0.0081487335, -0.024271498, -0.02932546, 0.012716018, 0.026929507, 0.013913995, -0.033892743, 0.0015512856, 0.012085833, -0.005746542, 0.052561205, -0.0024661464, 0.012728497, -0.0019669896, -0.020814836, 0.030298814, 0.0037592743, 0.044724446, -0.025706572, -0.024159187, 0.003843507, -0.005400252, 0.008710286, 0.0057153446, 0.019055309, 0.012765934, 0.02630556, 0.008417031, -0.04614704, 0.010369982, 0.02493288, 0.0136145, 0.031596623, -0.0029809016, -0.035490043, 0.0053472165, -0.00043598222, 0.006788532, 0.0043582623, 0.0456978, -0.0053160195, 0.024845527, -0.02238718, -0.039682962, 0.013489711, 0.012104551, 0.017283302, -0.00519123, 0.002838954, -0.00031080307, 0.0022571245, -0.016422257, 0.021039456, -0.015374028, -0.011199829, -0.042952437, -0.032170653, 0.004741989, -0.031696454, 0.017183471, 0.040531527, -0.009296794, -0.029924447, -0.0161602, 0.009396626, -0.005980522, -0.026280602, 0.02146374, 0.013065428, 0.016784146, -0.024371328, -0.01685902, -0.010132882, 0.010812983, -0.0062581776, -0.0318462, -0.0030604547, 0.003416104, -0.009309273, 0.0006535834, -0.012990555, -0.02757841, -0.010245192, -0.029450249, -0.026729845, 0.010082967, -0.017146034, -0.010164079, 0.011836254, -0.00031177799, -0.008404552, -0.05495716, -0.007836761, -0.006638785, 0.02782799, 0.04420033, -0.006033557, 0.0094652595, 0.014263404, 0.0019279929, -0.009746036, 0.003968296, 0.023273183, -0.029375374, 0.0024957838, 0.037511628, -0.017021244, -0.04524856, -0.008785158, 0.026505223, -0.007006913, 0.0037124783, -0.000380802, 0.012622426, 0.0071254624, -0.0064204037, -0.006401685, -0.012840807, 0.007375041, -0.016010452, -0.003909021, -0.008922427, -0.0019560705, -0.011461887, -0.019816523, -0.004277149, -0.012198143, -0.03601416, 0.03266981, -0.035490043, 0.005677908, -0.023260705, -0.02989949, 0.033992574, -0.004314586, 0.018144349, -0.04442495, 0.0034441815, 0.005053962, 0.025556825, -0.03139696, 0.020789878, -0.017807417, 0.027104212, 0.018531194, 0.013127822, 0.008154973, -0.009795951, 0.0010794265, -0.010644518, 0.03544013, -0.007886677, -0.027753115, -0.011199829, -0.019941311, -0.009664923, 0.02157605, 0.024895443, -0.0066512637, 0.010987688, -0.05850117, 0.015873184, 0.016422257, 0.0061801844, 0.008417031, 0.02538212, 0.00048667783, -0.02410927, -0.033069134, -0.0030495357, 0.002698566, -0.008966103, 0.0005042263, -0.015511296, 0.025107585, 0.059349738, 0.034017533, -0.02259932, -0.012266777, -0.018131869, -0.006070994, 0.026405392, -0.00013297847, -0.020602696, 0.004489291, 0.013102865, 0.008417031, 0.029350417, -0.021713318, -0.017033724, 0.0129531175, 0.030423604, 0.007911635, -0.017745022, -0.015174365, 0.02598111, 0.0135271475, -0.03464148, -0.01085042, 0.035190552, 0.014725124, -0.006969476, 0.044100497, -0.017245866, 0.0256317, 0.026180772, -0.011268464, 0.017932206, -0.025756488, -0.020340638, 0.021950418, -0.0047263904, -0.024508595, -0.008785158, 0.0025924954, -0.040905897, -0.0030292575, -0.010189037, -0.01235413, -0.018294094, 0.014512982, 0.003784232, 0.005169392, -0.004648397, 0.020253286, -0.0041274023, 0.0042428323, 0.043751087, 0.001119203, -0.0228489, 0.012516355, 0.031671494, 0.030448562, 0.033867784, 0.0023070402, -0.01073187, -0.030024279, 0.004779426, 0.024259018, -0.011262224, 0.0057309433, 0.02215008, 0.005122596, 0.019766606, 0.0014826516, 0.05251129, -0.044350076, -0.014113657, -0.018518716, -0.009727317, 0.00090316177, 0.037237093, -0.0071254624, 0.0010591482, -0.015211802, 0.017008767, 0.014238446, 0.008136255, -0.013864079, -0.029924447, -0.0064890375, -0.026180772, -0.005512562, 0.029175712, 0.0055593583, 0.038634732, -0.019779086, -0.052461375, -0.01638482, 0.0047326297, -0.004508009, -0.004096205, 0.019729171, -0.0113059, -0.006211382, 0.02747858, -0.0046702353, -0.009827148, 0.0017548479, -0.008666609, 0.037461713, 0.013639458, -0.0012861085, -0.0039558173, 0.044699486, -0.037811123, -0.039882623, 0.031122424, -0.016834062], "content_hash": "11e5dbff75922f24aa696d75de40c670"}, {"content": "token chips**: Your input tokens are displayed as colored boxes. Darker blue = higher influence. Hover over any chip to see the exact attribution score. - **Horizontal bar chart**: Shows the same information as bars, with attribution scores labeled. Longer bars = more influential tokens. ### What Attribution Scores Mean\n\n- **High score (near 1.0)**: This token was highly influential for the prediction. It strongly pushed the model toward the target token.\n- **Low score (near 0.0)**: This token had little influence on this particular prediction.\n- **Scores are normalized**: The highest-influence token gets a score of 1.0, and others are scaled relative to it.\n\n### Tips\n\n- Compare attribution for different target tokens. You might find that different input tokens drive different predictions.\n- Try Integrated Gradients for the most reliable results, especially when you want to draw conclusions.\n- Short prompts give cleaner attribution results since there are fewer tokens to compare.\n- Function words (like \"the\" or \"is\") often have low attribution; content words (nouns, verbs) tend to have higher attribution.", "source_file": "attribution_panel_guide.md", "chunk_index": 1, "embedding": [0.004626793, -0.0030052688, 0.001941866, 0.002912799, 0.04863912, -0.040158316, 0.0031357172, 0.0076221544, -0.025640557, -0.0037120024, 0.00970933, 0.01939224, -0.046763305, -0.046287745, -0.013751581, -0.00039650558, 0.0019666348, -0.040581036, -0.031413317, 0.034425188, -0.04285315, -0.0072786952, 0.045865025, 0.026908714, 0.020211259, 0.013540222, -0.023883631, -0.026076486, 0.010303779, 0.03949782, 0.0009998298, -0.014213931, -0.008143948, -0.046287745, -0.016301107, -0.021268057, -0.04372501, 0.010601004, 0.029775279, -0.004078579, -0.030937755, 0.028586382, -0.03450445, 0.0042734263, -0.045627248, -0.009597045, 0.024781909, -0.07540253, -0.04282673, 0.09099029, -0.056168806, 0.020369777, -0.021941764, 0.05344755, -0.027450323, 0.047370963, -0.015957648, 0.01882421, -0.027027605, -0.045706507, -0.023157082, -0.026882295, 0.04885048, 0.006350694, 0.027978722, -0.027688103, -0.059127837, 0.01453097, -0.0031621372, -0.016842715, 0.05336829, 0.01877137, 0.050171476, -0.016697405, -0.01955076, -0.014346031, -0.015772708, 0.021122746, -0.0026865783, -0.018309023, 0.008084503, 0.015851967, -0.027899463, -0.024821538, 0.009597045, -0.035746187, -0.03503285, -0.04855986, -0.023910051, -0.052232232, -0.03796546, -0.010528348, -0.027344644, 0.015429248, 0.0026073183, -0.0043493835, -0.035376307, -0.0065323315, -0.010422668, -0.013183553, 0.015217889, -0.020990647, 0.017357904, -0.022245593, -0.0030779236, -0.011664406, 0.001821325, -0.050964076, -0.005888345, -0.014781959, -0.0005094096, 0.051598154, -0.08982781, 0.050224315, -0.002926009, -0.03992054, -0.04282673, 0.0381504, 0.004098394, -0.028850581, -0.017133335, -0.045019586, 0.014121461, 0.03416099, 0.0060336553, 0.0007409969, 0.006112915, -0.0119682355, -0.008368518, -0.06784642, -0.017582474, 0.0019055386, -0.009649885, 0.008553457, -0.014081831, -0.02421388, 0.008850682, 0.04380427, -0.07693488, 0.022866463, -0.011532307, -0.02468944, -0.017450374, -0.0039398745, -0.048216403, 0.012351325, -0.027608844, 0.022787204, -0.037833363, 0.05902216, 0.017146545, -0.0051684016, -0.03759558, -0.005924673, -0.03416099, -0.022351274, 0.012972194, 0.008209998, -0.037278544, -0.051730253, -0.01438566, 0.022787204, -0.00056802883, -0.020832127, -0.037463482, 0.04031684, -0.03291925, -0.017899513, -0.040237576, -0.029326139, -0.036485944, -0.029563919, -0.015825547, -0.057754003, -0.002504941, -0.027582424, -0.050567776, -0.018705321, 0.020990647, -0.011651196, 0.0037780523, 0.02015842, -0.010653843, 0.004986765, -0.01462344, -0.0004916587, -0.030699978, 0.033130612, 0.016419996, -0.0018196738, 0.042958833, 0.013394913, 0.067000985, 0.01994706, -0.016882345, -0.05376459, 0.041135855, 0.032655053, 0.009649885, -0.0007537941, 0.011083167, 0.028242921, 0.02472907, -0.013183553, -0.032628633, -0.018533591, -0.0028203293, -0.04425341, -0.06763506, -0.00045161595, 0.017952353, -0.045415886, 0.028903421, -0.01449134, -0.09796516, 0.0044682734, -0.005627448, 0.04409489, 0.036485944, 0.016763456, -0.0034643153, 0.03479507, 0.04401563, -0.027820203, -0.05328903, 0.032549374, 0.02402894, -0.015970858, 0.050145056, -0.023434492, 0.02892984, 0.031360477, -0.0014861219, -0.057278443, -0.01463665, -0.023328813, 0.022192754, -0.0111360075, 0.04509885, -0.09030338, -0.009352661, -0.008038268, 0.051439635, 0.021571886, 0.00965649, 0.00965649, -0.022892883, -0.027371064, 0.0044319457, 0.008322283, 0.0385467, -0.0239761, -0.022866463, 0.022615474, 0.030515037, -0.017965563, -0.0062119896, 0.04797862, 0.047265284, -0.00982822, 0.0025842008, -0.0064960043, -0.07392301, -0.03323629, -0.01484801, -0.0016826203, -0.02451771, 0.0287449, -0.055006325, -0.028639222, -0.0050660246, -0.003107646, 0.03331555, -0.0054656263, -0.027767362, 0.0027707918, -0.034398768, 0.020211259, -0.040818814, -0.00032508918, -0.01435924, 0.0070343106, 0.059497718, 0.025997227, -0.037120022, -0.013910102, 0.026393525, 0.027767362, 0.005617541, -0.0033322156, 0.018388283, -0.031439736, 0.0394714, -0.035429146, -0.033077773, 0.015587768, 0.017740993, -0.0031984646, -0.022034235, -0.050065797, 0.0129986135, -0.040713135, -0.015165049, -0.023183502, 0.015640607, -0.007873144, 0.040079057, 0.042958833, 0.01957718, -0.04457045, -0.020567928, 0.015138629, 0.005686893, 0.02430635, -0.06589135, 0.040237576, 0.06721234, -0.019616809, 0.03437235, 0.022126704, -0.03756916, -0.055323366, 0.018414702, -0.00062540965, -0.0057067084, -0.011677616, -0.009980135, 0.017384324, 0.004306451, -0.020871757, 0.0019699372, 0.046340585, -0.001429154, 0.047450222, -0.07899564, -0.022853253, -0.027925882, 0.015627397, 0.005650566, 0.060713034, -0.031545416, -0.055164848, -0.0012846699, -0.01223904, 0.03408173, -0.001510065, -0.0031571835, 0.026961554, 0.056010284, -0.007919379, 0.016895555, 0.0071465955, 0.0028368416, -0.029880958, 0.016274687, 0.037833363, -0.011301132, -0.00016006772, -0.017001236, 0.013778002, -0.017106915, 0.005036302, 0.049220357, 0.017357904, 0.0086525325, -0.055112008, -0.025455618, 0.0075494996, 0.03286641, -0.035508405, -0.020700028, 0.009240376, -0.027556004, -0.051413216, -0.031149115, 0.020620767, -0.0006452246, 0.035508405, -0.021624725, -0.0012896236, 0.045653667, -0.056115966, 0.014332821, -0.017542845, 0.051941615, 0.009042227, 0.010898228, -0.004365896, -0.02431956, 0.027503163, -0.0012615524, -0.0078004887, -0.018956311, 0.032760732, -0.004451761, 0.04937888, 0.006254922, 0.043962788, 0.027582424, 0.04319661, -0.053949527, -0.023077823, -0.057331283, -0.017146545, 0.008163763, 0.06789926, -0.005769456, 0.013500593, 0.022734363, 0.062086873, 0.069854334, -0.032179493, 0.015759498, -0.041056596, 0.011268107, 0.021651145, 0.021783246, 0.021479417, 0.013256208, 0.040026218, -0.009313031, -0.056908563, -0.017661734, -0.059074998, -0.007291905, -0.020845337, 0.037120022, -0.011406812, 0.040184736, -0.03817682, 0.017542845, 0.031783193, 0.026168955, -0.036116067, -0.012575895, -0.026076486, 0.060395997, 0.023526961, -0.010171679, -0.0064794915, 0.018811, -0.015019739, -0.02383079, -0.06182267, -0.025032898, -0.02422709, -0.010713288, -0.0676879, 0.005383064, -0.0128004635, -0.028057983, 0.031360477, -0.0047721025, 0.075032644, -0.036512364, -0.04414773, 0.035006426, 0.016472837, 0.026050067, -0.008540248, 0.015997278, -0.012509844, -0.0032430484, 0.017371114, -4.8995582e-05, -0.0035270627, -0.027159704, -0.013593062, 0.03870522, 0.006426652, 0.007952403, -0.022443743, -0.002691532, -0.016023697, 0.025970807, -0.036565203, 0.032153074, -0.046684045, 0.031386897, -0.03471581, 0.0056901956, -0.0091677215, -0.04317019, -0.014015781, 0.0062780394, 0.05939204, 0.026921924, -0.027186124, -0.013348677, -0.012311694, -0.010581188, -0.04708034, 0.017265435, -0.01475554, 0.021413365, -0.028665641, 0.02409499, -0.014121461, 0.010039579, -0.03949782, 0.057701163, -0.02924688, -0.06647258, 0.038229663, -0.03519137, 0.0025016386, 0.06219255, 0.004441853, -0.04327587, 0.03928646, -0.03915436, -0.032153074, -0.02924688, -0.0067602037, -0.041928455, 0.003857312, 0.009431921, 0.037172865, -0.00180151, 0.006783321, 0.018454332, -0.0009213956, -0.012100335, -0.008639323, 0.009194141, -0.02414783, -0.011717246, -0.044940326, -0.046763305, -0.025587717, 0.014081831, -0.031836033, 0.021637935, 0.018665692, -0.06282663, 0.022390904, 0.0048249424, -0.010105629, -0.036987923, 0.011182242, 0.036882244, -0.014028992, 0.016169006, 0.0030349912, 0.00046152342, 0.010970882, -0.01976212, -0.0381504, -0.0020029622, 0.026855875, -0.016102957, -0.028586382, 0.03513853, -0.017344695, -0.003335518, -0.024121411, 0.030911336, -0.017120125, -0.016221847, -0.005264174, 0.035376307, -0.012575895, 0.016525676, 0.002389354, -0.035640508, 0.028057983, 0.049009, 0.03392321, 0.032707892, 0.0031109487, -0.011413417, 0.002222578, 0.020937808, -0.009636675, 0.016023697, -0.0024289838, -0.005845413, 0.013309048, 0.027767362, -0.013091083, -0.0076089446, 0.0108321775, -0.0141478805, 0.0016182216, -0.0013465916, 0.0040653693, 0.009352661, 0.021558676, 0.015244309, 0.03949782, 0.016710615, 0.0032579096, -0.0015108907, 0.016631356, -0.03466297, 0.045521565, 0.014055411, -0.0099999495, -0.011690826, 0.006509214, 0.01495369, -0.031281214, -0.01968286, -0.012628734, 0.008857287, 0.046155646, -0.022892883, 0.019867798, -0.0060567725, 0.015112209, -0.031492576, -0.044781808, -0.01996027, 0.0047754054, -0.0080316635, -0.0048909923, -0.013038243, 0.020739658, 0.021122746, -0.018573223, -0.026697354, -0.031281214, -0.006469584, -0.008725187, -0.012655154, 0.039233617, -0.0022044142, -0.01475554, 0.0056142383, 0.0031720446, -0.028295761, 0.0071730153, 0.026499206, 0.017410744, 0.020449039, 0.011888975, -0.00084254856, 0.01463665, -0.00055069075, -0.017727783, 0.030013058, 0.0054458114, -0.019828169, -0.01974891, 0.015165049, -0.00243724, -0.00054037047, 0.002765838, -0.026882295, -0.00997353, -0.022641893, -0.031783193, 0.010356619, -0.011162427, -0.018692112, -0.03949782, 0.021347316, 0.010508534, -0.0494053, -0.026089696, -0.01210694, 0.010594398, -0.036987923, -0.036248166, -0.01209373, -0.0038176822, -0.007661784, 0.032153074, 0.002452101, -0.011373786, -0.057754003, 0.017159754, 0.0078004887, 0.032734312, -0.0015835455, 0.0036525575, -0.016208638, -0.04858628, -0.031360477, 0.036644466, 0.0054953485, 0.015495298, -0.026684145, -0.050831977, 0.007866539, -0.017120125, 0.027503163, 0.008632718, 0.021492627, -0.029828118, -0.015112209, 0.023011772, 0.0069880756, -0.0015629049, -0.03450445, 0.00019907842, 0.013949731, 0.04951098, -0.011129403, 0.0071069654, 0.004600373, -0.0025891548, -0.021386946, -0.03402889, -0.020435829, 0.019828169, 0.011202057, 0.023038192, 0.008051478, -0.021479417, -0.01992064, 0.026763406, 0.012047495, 0.0037252123, -0.01471591, 0.030911336, 0.004567348, -0.0023051403, -0.014319611, -0.06684246, -0.016367156, 0.001345766, -0.041030176, 0.0025462222, 0.031281214, -0.016301107, -0.013262813, -0.023183502, -0.020435829, 0.0134147275, -0.010171679, 0.0018890261, -0.008890312, -0.02898268, 0.0014885988, 0.01475554, -0.03471581, 0.015759498, -0.011360576, -0.010224519, 0.000545737, -0.008513828, 0.013276023, -0.024980059, 0.023262762, 0.016380366, -0.030752817, 0.027265385, -0.012113545, -0.063724905, -0.018163713, 0.011162427, 0.032020975, -0.00044831345, 0.009114881, 0.0075759194, -0.008586482, -0.047793683, 0.002623831, 0.023130663, -0.016063327, 0.03754274, -0.016921975, -0.0015150188, -0.017912723, 0.005409484, -0.0030036175, 0.017252225, -0.025944388, 0.059550557, -0.0126947835, -0.015653817, 0.038335342, -0.016486047, 0.030964177, -0.01461023, 0.016221847, 0.021400155, -0.03342123, -0.0036129276, 0.00981501, -0.022219174, -0.023579802, 0.010997303, 0.010838782, -0.0045277183, 0.014161091, -0.030303678, 0.008434568, 0.008322283, -0.020079158, 0.02383079, 0.017846674, -0.037357803, -0.01998669, 0.008315678, -0.014900849, -0.10235087, -0.023091033, 0.057225604, -0.008566667, 0.008896917, -0.0028071192, 0.03455729, 0.006367207, -0.040263996, 0.03283999, -0.038731642, -0.008295863, -0.028586382, -0.0026964857, -0.0075494996, -0.01877137, 0.0015059368, -0.0028649129, -0.010085815, 0.011756876, -0.027397484, 0.002351375, 0.0055283736, 0.006895606, -0.036380265, 0.008005244, -0.03416099, 0.0064960043, -0.0025363148, 0.0044682734, -0.0107264975, 0.0014481433, 0.039682757, 0.01945829, 0.008731792, 0.010112234, -0.0035270627, 0.015085789, 0.0009469899, -0.023064613, 0.027265385, -0.024834748, 0.023091033, -0.015231099, 0.077886, -0.07693488, -0.025733028, -0.022747573, 0.00038350202, 0.012503239, -0.0038143797, 0.017252225, 0.014438501, 0.010779338, 0.0072192503, 0.01900915, 0.029114781, -0.035666928, -0.010541558, 0.009597045, -0.018295811, 0.013659112, 0.0015851967, 0.0020178233, 0.032655053, 0.00977538, -0.0026799731, 0.00600063, -0.026736986, -0.021651145, -0.006664431, 0.01896952, 0.008989386, -0.0021945068, -0.015600977, 0.021215217, 0.010350014, -0.0010402853, -0.054319408, 0.01939224, 0.056908563, -0.009293216, 0.016380366, 0.031994555, 0.014227141, 0.046129227, -0.0035336677, 0.0064596767, 0.0047588926, 0.017278645, 0.056221645, -0.005865228, 0.0018939798, 0.013870471, -0.021756826, -0.015548138, 0.035904706, 0.03756916, 0.022285225, -0.008467592, 0.013025033, -0.0009866199, -0.029537499, -0.01446492, -0.0095376, -0.017225804, -0.01997348, -0.04813714, -0.0192073, 0.0070475205, 0.003219931, 0.030145157, -0.0034676178, 0.013804422, -0.017833464, -0.002750977, -0.0022159729, 0.022377694, 0.027291805, -0.006611591, 0.0022721153, 0.011479466, -0.018916681, 0.0023943076, -0.009029017, 0.04858628, 0.0023216528, -0.008157158, 0.008527038, -0.014095041, 0.014887639, 0.005108957, -0.016353946, -0.020884966, -0.03331555, 0.005029697, 0.043090932, -0.029933799, -0.054794967, 0.020911386, -0.004425341, 0.05239075, 0.025759447, -0.009128091, -0.061135754, -0.016499257, -0.04036968, -0.027344644, 0.021003857, 0.013976151, 0.018480752, 0.014808379, -0.009200746, 0.014068621, -0.01438566, 0.0132165775, 0.020079158, -0.024359189, 0.0070805456, 0.063619226, 0.018256182, 0.030594297, 0.030092318, 0.008157158, 0.043090932, -0.014028992, -0.011294527, 0.020739658, 0.0067403885, 0.0021862504, -0.013011823, -0.016486047, 0.010871808, 0.050620615, 0.004293241, 0.020700028, 0.005941185, 0.011928606, -0.011287922, 0.023857212, -0.01177669, 0.018190132, -0.027556004, 0.0042470065, 0.030832076, -0.0042668213, 0.025482038, 0.0050726295, -0.01908841, -0.0046367003, 0.026750196, 0.027793784, 0.021386946, 0.046419844, -0.03289283, -0.024108201, 0.028903421, 0.0014299796, -0.0062053846, 0.025402777, -0.036036804, -0.011169032, 0.039656337, 0.020211259, 0.024966849, 0.02948466, -0.02853354, -0.012826883, 0.035799026, -0.0022737666, -0.023593012, 0.016697405, -0.0041809566, 0.022800414, -0.017582474, -0.00304655, -0.03405531, -0.0011492677, -0.0024933824, -0.004019134, -0.030937755, -0.017331485, 0.011545517, -0.0024141225, 0.009240376, -0.003910152, 0.046023544, -0.01435924, 0.018044824, 0.017239014, 0.0014052108, -0.005604331, 0.014068621, -0.003929967, -0.009412105, 0.009240376, 0.013672322, -0.004613583, 0.040924497, 0.0074174, 0.027740942, 0.037806943, 0.009715935, -0.013910102, -0.037622, -0.018295811, -0.004488088, 0.026340686, -0.011096377, -0.002389354, 0.0041215117, 0.055957444, 0.041030176, -0.046895403, -0.0066512213, -0.017674943, 0.0492732, 0.023659062, 0.026974764, 0.019828169, 0.016036907, 0.030515037, 0.011961631, -0.027503163, 0.007734439, -0.02453092, -0.01453097, 0.0039828066, -0.006955051, 0.022655103, -0.021915345, 0.031968135, 0.009180931, 0.026142536, 0.0026155745, -0.0034775254, -0.0012970542, -0.041188695, 0.009504575, 0.0028550054, -0.036644466, 0.0052212416, -0.032100234, 0.006948446, 0.023249552, 0.012569289, 0.0075230794, 0.0048447577, -0.013883681, 0.01472912, 0.0021565282, 0.030594297, 0.025349937, 0.0067569013, -0.0036063227, 0.017265435, 0.021043487, -0.0064497693, 0.007899564, -0.014808379, 0.036591623, 0.0071069654, -0.016129376, -0.018599642, 0.008216603, 0.028903421, 0.017582474, -0.020766078, 0.00712678, -0.01198805, -0.027133284, 0.013645902, -0.020673607, 0.032338012, 0.015297148, -0.020131998, -0.015231099, -0.052020874, -0.0064662816, -0.0038441021, 0.0031836035, 0.0015695099, 0.00961686, 0.027106864, 0.018850632, -0.0090686465, 0.00610631, 0.00010537017, -0.011618171, 0.010964277, -0.02441203, 0.0006307762, -0.0027410693, -0.0005222067, -0.038308922, 0.020937808, -0.0033454257, -0.007846724, -0.015310358, -0.054425087, 0.024847958, 0.024781909, 0.0048975972, -0.023024982, 0.036776565, 0.008692162, -0.0057001035, -0.0019121435, 0.0010865203, 0.0039530843, 0.07994676, -0.011202057, -0.0029573827, 0.00972254, -0.009544205, 0.033526912, -0.020198049, 0.0037483298, -0.03236443, -0.005264174, 0.013844051, 0.026287846, 0.03865238, -0.003850707, -0.023645852, 0.0010650541, 0.0074438197, -0.0018923286, 0.026697354, 0.0072126454, 0.026512416, 0.021651145, -0.005739733, 0.0074570295, -0.0054986514, -0.008824262, -0.0037516323, -0.0029904076, 0.020554718, 0.0071730153, 0.00019237022, -0.029828118, 0.00068774424, 0.0120673105, -0.021056697, 0.018335443, -0.032126654, 0.035508405, -0.035323467, -0.016737036, 0.017754203, 0.008157158, 0.00961686, 0.021743616, -0.018335443, -0.017569264, 0.01475554, -0.015019739, -0.0014431896, -0.011162427, -0.0064299544, -0.0017371115, -0.024610179, -0.025376357, 0.015534928, -0.007787279, 0.0053335265, 9.773316e-05, 0.0016784922, -0.02464981, -0.0049306224, 0.0062516197, -0.02903552, -0.008731792, 0.03899584, -0.027714523, 0.010944462, -0.015402828, 0.0012070612, 0.029880958, -0.006677641, -0.00480843, -0.020739658, 0.019590389, 0.021651145, -0.013857261, 0.005538281, 0.012945773, 0.0061756624, -0.028375022, -0.017674943, -0.025415989, -0.03503285, 0.0004041426, -0.035376307, -0.027186124, 0.039603498, 0.009768775, -0.015085789, 0.013738371, -0.015521718, -0.011314342, 0.035957545, 0.016314317, -0.001915446, -0.021241637, -0.0022754178, 0.010165074, 0.008474197, -0.00713999, 0.005571306, -0.017516425, -0.028375022, -0.0075561046, 0.016380366, 0.0107264975, -0.015984068, 0.012879724, 0.020779287, -0.010607609, 0.033976052, 0.00982822, 0.027820203, 0.019365821, 0.0017239015, 0.02404215, 0.012021075, 0.031783193, -0.046023544, -0.02882416, 0.0024917312, 0.002009567, 0.004425341, -0.00986785, 0.012648549, 0.018916681, 0.030884916, 0.0025759446, -0.03479507, -0.017476793, 0.026961554, 0.007813699, 0.04229833, 0.013711952, -0.003287632, 0.005115562, 0.0022671616, -0.0029161016, -0.002840144, 0.02379116, 0.0053203166, 0.016974814, -0.010178284, -0.044781808, 0.02916762, 0.0049768575, 0.018375073, 0.016155796, 0.0066908514, -0.002899589, -0.00081736705, -0.02488759, 0.015640607, 0.0069682607, -0.016301107, -0.026538836, -0.010951067, -0.0056340536, -0.015627397, 0.0145706, 0.01234472, -0.020277308, -0.009451736, -0.011849346, 0.026287846, -0.010072604, -0.050382838, 0.008018454, 0.011829531, 0.0029805002, -0.0021912043, -0.011056747, 0.004002622, 0.042034134, -0.0020723145, -0.01966965, 0.010250939, 0.017635314, -0.021624725, 0.011803111, -0.011710641, -0.026631305, 0.014425291, -0.007886354, -0.0046928427, 0.0100990245, -0.0032314896, -0.012470215, 0.028163662, 0.0018626062, -0.017886303, -0.06763506, -0.0036789775, 0.00044212127, 0.029114781, 0.04908826, -0.02972244, 0.010158469, 0.021439785, 0.01438566, -0.0009461643, 0.0054160887, 0.02464981, -0.020686818, -0.008196788, 0.017437164, -0.020581137, -0.03424025, 0.011010513, 0.007648574, 0.016684195, 0.00028545925, 0.0058289007, 0.0190752, 0.018692112, -0.016486047, -0.006889001, -0.014068621, 0.014319611, -0.014914059, 0.006046865, -0.010640633, 0.016578516, -0.017542845, -0.02850712, -0.017859884, 0.00059444876, -0.020198049, 0.040950917, -0.0011327552, 0.027291805, -0.026367106, -0.027014395, 0.01482159, 0.01246361, 0.016525676, -0.047370963, -0.008764817, 0.000552342, 0.031545416, -0.041717093, -0.0004293241, -0.014887639, 0.014980109, -0.012866514, 0.036485944, 0.012985404, -0.0053467364, -0.00048381527, -0.011743666, 0.02964318, 0.0023794463, -0.021532256, -0.021624725, -0.022100285, -0.03849386, 0.007252275, 0.014028992, 0.0045211134, 0.005693498, -0.07719908, 0.00019763358, 0.02354017, -0.010264149, 0.010118839, 0.013844051, -0.012324905, 0.012853304, -0.03381753, 0.0034841304, -0.016182216, -0.0020871756, -0.006231805, -0.021294476, 0.007846724, 0.03519137, 0.03326271, 0.0018824211, -0.0066512213, -0.03426667, 0.0013309048, 0.007787279, 0.0005973385, -0.013857261, 0.0095838355, 0.00022477594, 0.0046367003, 0.04229833, 0.0047688, -0.013091083, 0.0016495953, 0.015957648, 0.0015868479, -0.005970908, -0.02453092, 0.012562684, 0.01896952, -0.021083117, -0.0013276023, 0.018850632, 0.041267954, -0.0064134416, 0.023157082, 0.002691532, 0.02426672, 0.028639222, 0.017727783, 0.020581137, -0.022007816, 0.0043229638, 0.009729145, 0.00024748058, -0.016433205, -0.016129376, 0.00053995766, -0.026168955, 0.001553823, -0.008606297, -0.011789901, -0.0050495123, -0.00952439, -0.026776616, 0.009464946, 0.009577231, 0.022166334, -0.0019501222, 0.0013350329, 0.055534724, -0.008427963, 0.0036723725, 0.012833488, 0.0035666926, 0.026023647, -0.0009337799, 0.021439785, -0.022681523, -0.01204089, -0.0013886984, 0.008176973, -0.024557339, -0.014914059, 0.017080495, 0.009299821, -0.0015596024, 0.0021202005, 0.030594297, -0.017225804, -0.013203368, -0.037859783, -0.0144517105, 0.03426667, 0.03402889, 0.009088461, 0.0027377668, -0.0039167567, 0.027476743, 0.03373827, -0.016406786, -0.0082364185, -0.041875616, 0.010924648, -0.026459575, -0.01956397, 0.029775279, 0.016182216, 0.030699978, -0.015970858, -0.048242822, 0.004019134, 0.0033751482, -0.006822951, 0.008639323, 0.010904833, 0.0034511054, 0.0008359436, 0.036538783, -0.016697405, -0.0043196613, 0.00984143, -0.021360526, 0.03997338, 0.016419996, -0.011472861, -0.003454408, 0.022998562, -0.015627397, -0.04832208, 0.015125419, 0.013394913], "content_hash": "903501e06932bc125329a18df29e142e"}, {"content": "# Beam Search and Generation\n\n## What Is Text Generation?\n\nWhen you click \"Analyze\" in the dashboard, the model generates a continuation of your prompt. The simplest approach is **greedy decoding**: at each step, pick the single most likely next token. This is fast but can lead to repetitive or suboptimal text.\n\n## What Is Beam Search?\n\nBeam search is a smarter generation strategy that explores **multiple possible paths simultaneously** instead of committing to the single best token at each step.\n\n### How It Works\n\nImagine you're at a fork in the road. Greedy decoding always takes the path that looks best right now. Beam search keeps multiple paths open and evaluates them over several steps before deciding which one is actually best overall.\n\nMore specifically:\n1. At each step, the model considers the top candidates for the next token\n2. It keeps the **N best partial sequences** (where N is the number of beams)\n3. It extends each of those sequences by one more token\n4. It ranks all the possibilities and keeps the top N again\n5. This continues until the desired number of tokens have been generated\n\nThe result is that beam search can find better overall sequences, even if the individual tokens at each step aren't always the highest-probability choice.\n\n### Example\n\nWith the prompt \"The cat sat on the\" and 3 beams:\n- Beam 1 might generate: \"The cat sat on the mat and purred\"\n- Beam 2 might generate: \"The cat sat on the floor and looked\"\n- Beam 3 might generate: \"The cat sat on the couch and slept\"\n\nEach beam represents a different possible continuation.\n\n## Generation Controls in the Dashboard\n\nIn the generator section, you have two settings:\n\n- **Number of Generation Choices (Beams)**: How many parallel paths to explore (1-5). With 1 beam, you get greedy decoding. More beams = more diverse results but slower.\n- **Number of New Tokens**: How many tokens to generate beyond your prompt (1-20). Longer generations take more time.\n\n## How Generated Sequences Are Used", "source_file": "beam_search_and_generation.md", "chunk_index": 0, "embedding": [0.021287892, 0.009334653, 0.033566758, -0.0059562684, -0.0076997317, -0.030554706, -0.007604757, 0.030364757, 0.024164269, 0.036497407, 0.0012177109, -0.021233622, -0.031260233, -0.046103414, 0.009009025, -0.023811504, -0.035737608, -0.017597448, 0.02470698, 0.068598844, 0.009192191, -0.021884875, -0.007028125, 0.038234085, -0.01381203, -0.006332775, -0.0300934, 0.044800907, 0.015290921, -0.020229604, -0.013628865, -0.041327547, -0.022074826, -0.046781804, -0.05877575, 0.0027457853, -0.038125545, 0.009409276, 0.016498456, -0.020962264, -0.026172305, -0.010033395, -0.004365443, -0.0050675767, -0.011600477, 0.012570575, -0.020270307, -0.053185813, 0.014870319, 0.0383969, 0.031314503, 0.055899374, -0.023513013, -0.031612996, -0.0021589776, -0.015779361, -0.06637372, -0.02359442, -0.009368572, -0.02006679, 0.032291386, -0.0013347332, 0.01839795, 0.07777068, 0.025602456, -0.006482021, -0.025507482, 0.019049205, -0.015223082, 0.02508688, 0.0144768525, 0.019605484, -0.018899959, 0.0024031983, -0.045289345, 0.0061055142, 0.030554706, 0.010786409, -0.027895415, -0.018140161, 0.038505442, 0.0025812758, 0.0006639747, -0.048518486, -0.0120007275, -0.00894797, 0.02470698, -0.0016866482, -0.034462236, -0.045533568, -0.05796168, 0.020636637, -0.047731552, 0.05845012, 0.014001979, 0.009280382, -0.0060037556, -0.046998892, 0.0070688287, 0.008473096, 0.03622605, -0.022427589, -0.040622022, -0.01290977, 0.05299586, -0.010440429, -0.021369299, 0.0056984797, -0.024204971, -0.045804925, -0.05085215, -0.02229191, 0.005773103, 0.016010014, 0.022508996, 0.010467565, 0.0043790103, 0.0221698, 0.022956733, 0.008975105, -0.04227729, 0.006763553, -0.023404472, -0.016023582, -0.012536655, -0.0021216662, 0.024530599, -0.048409943, -0.013337157, 0.023689395, 0.033458218, -0.0036056454, 0.019117042, -0.04043207, -0.05920992, 0.02873662, -0.022251207, 0.00993842, -0.03760997, 0.037365746, -0.0013135335, -0.0081271175, -0.0048437077, -0.010956006, -0.051557675, -0.013907004, -0.0149517255, 0.024747685, -0.030934604, 0.015060268, 0.029333603, -0.032617014, -0.05877575, -0.040730562, 0.046103414, -0.016457753, -0.011159523, 0.05573656, -0.06262901, 0.004073735, -0.008194957, 0.033593897, -0.014409013, -0.009850229, -0.056062188, 0.026606474, -0.019551214, 0.032101437, -0.022061257, -0.022156233, 0.0158472, -0.021138646, 0.009219326, 0.0038159464, 0.04477377, -0.016498456, -0.084228955, -0.060729515, -0.024679845, -0.050580792, -0.014327606, 0.013432131, -0.030201944, 0.0010328494, 0.0053660683, -0.0048742355, 0.005606897, -0.0005906237, -0.005094712, 0.0024371177, 0.0021827214, 0.014069818, 0.074243054, -0.007815058, -0.040106446, -0.03541198, 0.06328026, 0.022658242, 0.01796378, -0.021789901, 0.05573656, -0.042114478, 0.023607988, 0.005064185, -0.024489895, 0.0336753, -0.031857215, -0.0012397586, -0.04187026, -0.02222407, 0.028763756, -0.02272608, 0.049332555, -0.0022980478, -0.043362718, -0.01659343, 0.026280846, 0.03467932, 0.002208161, 0.041408952, -0.0018401342, -0.0128962025, -0.004894587, 0.011037413, 0.018669305, 0.058178764, 0.0074758627, -0.035140626, 0.020704476, 0.014096954, 0.060512427, -0.010528619, 0.011722587, -0.021884875, 0.001067617, -0.028221043, -0.032345656, 0.009999475, 0.017244685, -0.02965923, -0.011478366, -0.008398474, 0.035819016, -0.00483014, -0.0112884175, 0.030744655, -0.07066115, -0.02941501, 0.012468817, 0.064257145, 0.015874337, 0.009965556, -0.008975105, 0.030066265, 0.018791415, -0.010379374, -0.035574794, 0.029007975, -0.026023058, -0.00668893, 0.008038927, 0.025290396, -0.03101601, -0.0038736095, -0.049305417, 0.009965556, -0.028193908, -0.077173695, -0.044692364, -0.035656203, -0.02440849, -0.033349674, -0.010223344, -0.017516041, -0.033973794, -0.051503405, 0.0032325306, -0.012034646, 0.0005274486, 0.0069331503, -0.034896404, -0.009606008, 0.049006928, 0.008751237, -0.010847463, 0.0058104144, -0.0018706616, 0.018479357, 0.03874966, -0.017814534, 0.0006974702, 0.00050158496, 0.0015882817, -0.006454885, -0.007082396, 0.040214986, -0.0005070969, -0.007048477, -0.035764746, 0.021206485, 0.029089382, -0.04515367, 0.02477482, -0.04024212, -0.007116316, 0.0028814634, -0.04045921, 0.021640655, -0.004484161, -0.014436149, -0.004619839, 0.034842134, -0.03771851, -0.07326617, 0.018954229, 0.0019537646, 0.051259182, -0.028899433, 0.034923542, -0.038478307, -0.0013771326, -0.015928607, 0.020541662, -0.02154568, -0.026307981, -0.0040058955, 0.008934402, -0.0092328945, -0.0035852937, -0.026321549, 0.022346182, 0.019239154, 0.025466777, 0.05573656, -0.036985848, -0.074351594, 0.011566557, 0.027108483, 0.020365281, 0.023445174, -0.024571301, -0.003768459, -0.016010014, -0.04347126, -0.013642432, -0.0020656988, -0.012292435, 0.02694567, 0.022373317, 0.006397222, -0.02712205, -0.011912537, -0.0032613622, -0.029767774, 0.039618004, 0.013683136, -0.0004189061, -0.023336632, -0.022346182, -0.009673848, -0.017516041, 0.040025037, 0.013201478, 0.023458742, 0.00017267157, -0.037501425, -0.009897716, -0.008011791, 0.020609502, 0.015453734, -0.02248186, 0.04298282, 0.0039550164, 0.018818552, -0.04409538, 0.013628865, -0.04173458, 0.0253311, -0.0011210402, 0.08878774, -0.00866983, -0.027366271, -0.055628017, -0.043525532, 0.020650206, 0.05568229, -0.0050981045, 0.0050200894, -0.036198914, -0.000110768444, -0.036497407, -0.0387768, 0.021749198, -0.011152739, 0.0019673323, 0.049658183, 0.0030239255, 0.029116519, 0.037935596, 0.030419027, 0.042195886, -0.025263261, -0.049685318, -0.011254498, -0.018099457, 0.0038532577, 0.018845687, 0.03820695, 0.04762301, 0.0021521938, 0.042820007, 0.032779828, 0.039156698, -0.04005217, 0.029170789, 0.036253184, 0.040214986, 0.04162604, -0.00047402532, 0.053728525, -0.021681359, -0.077065155, -0.03598183, -0.021884875, -0.038369764, 0.023892911, 0.0033749924, 0.0070688287, 0.019727595, -0.024137132, 0.0058782534, 0.044068243, 0.07896465, -0.009124352, -0.012713037, -0.0024422056, 0.058558665, 0.029116519, -0.004769085, -0.04290141, 0.016688405, -0.0063565187, -0.038613986, 0.0011507198, -0.032671284, 0.010738921, -0.03183008, -0.023974318, -0.004124614, -0.032128572, -0.021287892, 0.02321452, -0.021776333, 0.026267279, -0.04366121, 0.020175332, 0.064311415, -0.016484888, 0.04941396, 0.0024710372, 0.02898084, -0.013350724, 0.00039346647, -0.0072384262, -0.006875487, 0.02199342, 0.002839064, 0.019944679, 0.021206485, -0.007821842, 0.004752125, 0.016064286, -0.00032499142, -0.047161706, -0.020134628, -0.00040003838, 0.016579863, 0.005905389, 0.0021572816, -0.026986372, 0.019184882, 0.05790741, 0.03318686, 0.07608827, -0.017475339, 0.072560646, 0.01907634, -0.015928607, 0.018994933, -0.024042157, -0.012285651, -0.01883212, 0.023173818, -0.015942175, 0.010108017, -0.021681359, -0.032101437, 0.0241507, 0.01381203, 0.012875851, -0.0034665752, -0.001353389, -0.017841669, 0.03448937, 0.028845163, -0.02303814, 0.04284714, 0.039102428, 0.0030612368, 0.01468037, -0.019673323, -0.02583311, 0.011783642, 0.019483374, -0.008350986, 0.03394666, -0.01765172, 0.04501799, 0.007095964, 0.014205496, 0.017841669, 0.021572817, 0.014409013, 0.0052643097, 0.00038816655, -0.060512427, -0.053240083, -0.059861172, -0.034977812, 0.0002633003, 0.036334593, -0.043742616, 0.0054440834, 0.04105619, -0.06545111, -0.05269737, -0.011261282, 0.030364757, -0.012428113, -0.024883362, -0.027312, 0.021586385, -0.02291603, 0.02402859, -0.005094712, -0.0026966021, 0.013018313, 0.006129258, 0.0059087807, 0.007502998, 0.02291603, -0.037012983, 0.041571766, -0.03815268, -0.018520059, 0.020216035, -0.02006679, -0.022997437, 0.02463914, -0.0253311, -0.010304751, 0.0150467, 0.020595934, -0.003917705, -0.048762705, 0.016946193, -0.0029001192, 0.023743667, -0.009178623, 0.0097620385, -0.0042772517, -0.002067395, -0.0052337823, 0.042440105, 0.014341175, -0.013330372, 0.0057968465, 0.008907267, -0.024734115, 0.029550688, -0.016824083, -0.01683765, 0.0062344084, 0.012278867, -0.04284714, -0.0005261766, -0.0154265985, 0.035004947, 0.013601729, 0.008493449, 0.013113287, 0.004484161, 0.0038091624, 0.030066265, -0.03419088, -0.01021656, -0.02161352, -0.033349674, -0.025249694, -0.0079236, 0.024299946, -0.041517496, -0.047460195, 0.01591504, -0.02446276, 0.0044264975, -0.017366795, 0.022685377, 0.025670296, -0.021477843, -0.03372957, -0.0030137496, -0.030066265, -0.032752693, 0.0075640534, 0.011532637, -0.007855762, 0.0015407943, 0.027610492, -0.02538537, 0.025724566, 0.023580853, 0.023987886, -0.008914051, -2.5095147e-05, -0.012753741, 0.0008759717, -0.03128737, 0.011742939, 0.033241134, 0.0071705873, -0.011132387, 0.0099452045, -0.008262795, -0.0069467183, 0.0443396, 0.027746169, 0.02253613, -0.010426861, -0.0013703487, 0.012651982, 0.0062988554, -2.1848942e-05, -0.030826062, -0.020446688, 0.024218539, -0.0040296395, 0.017203981, -0.010942438, -0.034353692, -0.037447155, -0.028790891, 0.051177777, -0.022373317, 0.04732452, -0.01598288, -0.004026247, -0.009341436, 0.004684286, -0.017733127, -0.007502998, 0.011960024, -0.028465264, -0.051883303, -0.0075640534, 0.05839585, 0.009823094, 0.024544166, -0.0070145573, 0.0027271297, -0.040757697, 0.039726544, 0.0019741163, -0.004765693, -0.035683338, 0.0056645605, 0.010827112, -0.03541198, -0.0060139312, 0.009076864, -0.004192453, 0.00439597, -0.039618004, -0.02724416, 0.04873557, -0.009585657, 0.02606376, -0.004446849, 0.037040118, 0.018316543, 0.009253246, 0.0360361, -0.016322074, 0.01683765, 0.00033219933, -0.03864112, 0.0053423247, 0.03264415, 0.006892447, -0.077010885, -0.0423587, 0.009619577, 0.0069467183, -0.013194694, 0.0033834723, -0.021315029, -0.039835088, 0.03760997, -0.010915303, -0.0020996185, -0.010453997, -0.03535771, -0.0069161905, 0.018804984, 0.0190085, -0.007604757, -0.024720548, -0.017258253, 0.017176846, -0.0079168165, -0.036008965, 0.021057239, -0.036823034, -0.02965923, 0.0061224736, -0.025928084, 0.008568072, -0.0061496096, -0.039590865, 0.04892552, 0.030853197, -0.0037718508, 0.00035763896, 9.2801696e-05, -0.012523088, 0.010270831, -0.031043148, 0.00014786792, -0.010108017, -0.01579293, 0.014734641, 0.0064989803, 0.025195422, 0.0028271922, 0.0009395708, 0.001846918, 0.0035072786, 0.02341804, 0.012570575, -0.01164118, -0.0069467183, 0.011437663, -0.005288054, -0.04596774, 0.06463704, 0.018316543, 0.04732452, -0.0021928973, 0.0027440893, 0.03728434, 0.013506754, 0.0018248704, -0.03809841, 0.030229079, -0.0054440834, -0.023702962, 0.0025999313, 0.003122292, -0.007408024, -0.0004121222, -0.007631893, 0.0072859135, -5.596721e-05, -0.0006080074, 0.047351655, -0.028926568, 0.060349613, 0.0069467183, -0.050825015, 0.0016925841, 0.019971816, 0.020880857, -0.036958713, 0.008317067, -0.010426861, -0.0031748672, -0.0018113025, -0.028600942, -0.031368773, -0.026090898, -0.0051693353, -0.020948697, 0.049820997, -0.004819964, -0.021206485, 0.0049522505, 0.012312787, -0.112232916, -0.023105979, 0.009171839, 0.014042682, 0.0028967273, -0.028193908, 0.038668256, 0.015969312, -0.006159785, 0.0013389732, -0.0012024471, -0.016729109, 0.043335583, 0.0064752367, 0.0047182054, -0.039970767, -0.008622343, -0.007930384, 0.019103475, -0.007869329, -0.022142664, 0.01999895, -0.013303237, -0.009795958, -0.016145693, -0.020840155, -0.011159523, 0.004979386, 0.023987886, -0.031260233, -0.010677866, 0.07473149, -0.013567809, 0.018994933, 0.0069229747, 0.0014492116, -0.007469079, -0.0063768704, 0.00938214, -0.005684912, -0.020528095, -0.020962264, 0.021627087, -0.009836662, 0.0016434009, 0.0063158153, -0.004399362, 0.0055831536, 0.0027457853, 0.012808012, -0.00934822, 0.00028916393, -0.0026423307, 0.00084629207, 0.018180864, -0.0037888107, 0.01080676, 0.01405625, 0.035086356, 0.045614973, 0.010738921, 0.02192558, 0.031314503, -0.024734115, 0.011023845, -0.02718989, 0.010074099, 0.0018129984, -0.026077328, 0.009056512, -0.013201478, 0.010983141, -0.004046599, -0.0008183085, -0.00096840237, -0.009009025, 0.028600942, 0.0071366676, -0.0071705873, -0.008812292, 0.028628077, -0.010847463, -0.01653916, 0.031450182, -0.017488906, 0.0304733, -0.009857013, -0.008710533, 0.011980375, 0.019212017, 0.022644673, -0.05245315, -0.007184155, 0.0035988614, -0.049658183, -0.014856751, 0.0024031983, 0.020690909, 0.030256214, -0.0012270387, 0.026986372, 0.035004947, -0.031992894, -0.027393406, -0.0038736095, 0.016756244, -0.011220578, -0.003208787, -0.033539623, -0.011837914, 0.009477114, -0.0007890529, -0.0043111714, 0.013947708, -0.02649793, 0.033648167, 0.0411376, -0.016267803, 0.004911547, -0.010684649, 0.0241507, -0.017081872, -0.021749198, 0.006461669, 0.017380364, 0.01795021, -0.0006487109, -0.012828363, 0.0053558927, 0.007340185, -0.009158271, 0.007292697, 0.031612996, -0.0026609865, -0.020853722, -0.022115529, -0.0005244806, 0.007197723, -0.0443396, -0.010847463, -0.024951201, 0.003476751, 0.061055142, -0.0018910134, -0.00938214, -0.037257202, -0.023892911, -0.013757759, -0.027976822, 0.013642432, 0.022807486, -0.0020945305, -0.025114015, 0.027651194, -0.037772782, 0.02470698, -0.012855499, -0.03332254, -0.0038227302, 0.013649216, 0.03462505, 0.031097418, 0.050065216, 0.013669568, 0.01653916, -0.029930586, -0.009653497, 0.003751499, -0.0045214724, 0.00061479135, -0.026090898, 0.017570313, 0.015955744, 0.014653234, 0.04477377, -0.003927881, 0.022766784, 0.03386525, 0.009755255, 0.04254865, -0.0062004887, 0.0124484645, 0.0062785037, 0.02199342, -0.012862283, -0.024679845, -0.01758388, -0.009619577, -0.004280644, -0.029007975, 0.0072519938, 0.04043207, 0.012882635, 0.033539623, -0.030581841, 0.03134164, 0.032209978, -0.024544166, -0.027949687, -0.0005240566, -0.0064481013, 0.009056512, -0.013167559, 0.00019864121, -0.009402492, 0.021097943, -0.039265238, 0.035710473, 0.013947708, 0.010799976, -0.03931951, -0.027610492, -0.013404995, -0.003476751, -0.030934604, 0.010861031, 0.02025674, -0.014870319, -0.028139636, -0.02223764, -0.058938563, -0.044638094, 0.0040228553, 0.021342164, -0.021599952, -0.016077854, 0.015874337, 0.015887905, -0.038722526, 0.008744453, -0.023933616, -0.029876316, 0.024720548, -0.0055899373, 0.008052494, 0.04954964, -0.033838116, -0.020283874, -0.0019452847, 0.0001994892, 0.03535771, 0.027013509, 0.013520322, -0.021355731, -0.03202003, -0.024367785, -0.00024252459, -0.006404006, 0.007516566, -0.020338146, -0.02396075, 0.03942805, 0.027461246, -0.011559773, 0.005176119, -0.021586385, 0.015209514, 0.04444814, 0.0137645425, -0.00056899997, 0.007258778, 0.0043756184, -0.0047317734, -0.008771589, -0.007340185, -0.011207011, -0.002557532, 0.0010599851, 0.031667266, -0.020731613, -0.0057866704, 0.014883887, -0.009470331, 0.04105619, -0.025439642, 0.013404995, -0.0014568435, -0.018750712, 0.014137657, 0.0020165157, 0.009165055, 0.00089971535, 0.00046045752, -0.02941501, -0.01567082, -0.003354641, -0.019578349, 0.00411783, 0.018940661, -0.014788912, 0.021233622, 0.015114539, -0.020677341, -0.012970826, -0.027162753, -0.023974318, -0.012217812, 0.042005938, 0.007821842, -0.034787863, -0.028845163, -0.0025897555, 0.00340552, -0.01548087, 0.014911022, 0.009253246, 0.03096174, -0.03003913, -0.007163803, -0.0018740536, 0.0022488644, 0.030608978, -0.026728583, 0.011139171, -0.0021521938, -0.035330575, -0.011803994, 0.0025660119, 0.012190676, -0.019279856, -0.029143654, -0.01080676, -0.013493186, 0.031124555, 0.019049205, 0.008595207, 0.0017858628, 0.017190414, -0.0025151325, 0.023499446, 0.026484363, -0.022441156, 0.040974785, -0.012373842, -0.025683863, 0.010738921, -0.053484302, 0.0023726707, 0.02291603, -0.020473825, 0.013588161, -0.017855236, -0.014761777, 0.0050845365, 0.01913061, -0.00038477458, 0.01381203, 0.0032376184, 0.03590042, -0.0051455917, 0.07880183, -0.016566295, -0.019049205, -0.0066719702, -0.0092600295, -0.013947708, -0.032671284, -0.015413031, -0.014856751, -0.027447678, 0.00255414, 0.011525854, 0.017678855, -0.0042602923, 0.039238103, -0.019049205, 0.008941187, -0.006336167, 0.03454364, -0.016430616, 0.030744655, 0.020663774, 0.011966808, -0.011593693, 0.0025456601, 0.017203981, 0.02347231, 0.0037481072, 0.017176846, 0.03183008, 0.0038634336, -0.037799917, 0.004931899, 0.015887905, -0.017923076, -0.0014466676, 0.013710271, 0.021016536, -0.018140161, 0.0069467183, -0.020134628, -0.0018621818, 0.018045187, -0.0051354156, 0.03299691, 0.026348686, 0.020270307, -0.0046435827, 0.011715803, -0.04811145, -0.015901472, 0.0020758747, 0.008025359, -0.035819016, 0.026145168, -0.056225, 0.025616024, 0.023309495, -0.002491389, 0.0068585277, -0.03690444, 0.019863272, -0.04452955, 0.0017841669, -0.008683397, -0.0601868, -0.0023591027, 0.0016866482, -0.0049217227, 0.029279333, -0.0035988614, -0.018913526, 0.009158271, 0.019836137, 0.03231852, -0.0013940923, -0.0009811221, -0.013961275, -0.0048878035, 0.003910921, -0.017393932, -0.007977872, -0.031423047, -0.009789174, -0.0008246684, 0.012977609, 0.041571766, 0.024924066, -0.035276305, 0.025154717, -0.0324542, 0.0144768525, 0.043606937, 0.07934455, 0.0069297585, -0.0149517255, 0.0016145693, 0.0120007275, -0.006027499, -0.00863591, 0.02854667, -0.0050675767, -0.038613986, 0.014693937, -0.031504452, -0.027664762, -0.037148662, -0.0153994635, 0.008690182, 0.01207535, -0.021844173, 0.0150467, -0.012984393, -0.0027237376, -0.02440849, 0.010671082, -0.0036090373, 0.021884875, 0.002052131, -0.055031035, 0.02854667, 0.023078844, -0.0061665694, 0.012577359, 0.00885978, -0.026891397, 0.0014000284, -0.0013678047, 0.012536655, 0.010277615, -0.018818552, 0.012570575, 0.020243172, 0.020541662, 0.015887905, -0.010562539, -0.027447678, -0.019591916, 0.018899959, 0.035656203, -0.008011791, 0.008873347, -0.053050134, -0.021871308, 0.0097620385, 0.03584615, 0.008642694, 0.012428113, -0.00024634053, 0.030880334, -0.0137645425, -0.02680999, 0.023933616, -0.011783642, -0.003558158, -0.011166307, -0.0162271, 0.0154265985, -0.010643946, -0.01591504, 0.018574331, 0.00054398435, -0.035059217, -0.0028797674, 0.033213995, -0.005840942, -0.007604757, -0.008296715, -0.013201478, 0.034462236, -0.033892386, -0.00934822, -0.031667266, 0.0059121726, 0.012190676, -0.022766784, 0.007692948, 0.0008624039, 0.006037675, 0.02390648, 0.0051659434, 0.0013347332, -0.005969836, -0.014191928, -0.0006347191, 0.03549339, 0.006465061, 0.005335541, 0.00017881948, 0.03139591, 0.004979386, -0.06724206, -0.014870319, -0.0024676453, 0.00953817, 0.0047453414, -0.002335359, -0.018845687, 0.037420016, 0.033403944, -0.00941606, 0.026660744, 0.011783642, 0.033919524, -0.013723839, 0.007950736, -0.009992692, 0.014748208, 0.00049480103, -0.013147207, -0.027719034, -0.0018977973, -0.009443196, -0.0023981102, 0.038613986, -0.04954964, -0.0072723455, 0.015535141, -0.0020131236, -0.028166771, 0.03975368, 0.0039244886, -0.009511034, -0.0122110285, -0.027298432, 0.0044502416, 0.015141675, -0.02081302, 0.02538537, 0.014137657, 0.02093513, -0.0128962025, -0.009456763, 0.02291603, -0.009931636, -0.014449717, -0.016620567, -0.023621555, -0.0011507198, -0.003120596, -0.045777787, 0.03264415, -0.0035344143, 0.024544166, -0.008710533, 0.03652454, 0.0038430819, 0.0045248643, 0.005193079, -0.011396959, 0.022088394, 0.04496372, -0.021844173, -0.025263261, -0.025846677, -0.032101437, 0.0072655617, -0.016742676, -0.025968786, 0.02025674, -0.00015390983, -0.008208524, 0.0070891804, 0.00089377945, -0.008120334, 0.0044264975, 0.025480345, -0.04526221, -0.032915507, 0.047405925, 0.008527368, -0.014816048, -0.03128737, -0.028872298, 0.01709544, 0.0253311, 0.009137919, -0.0025778837, 0.0097620385, 0.031043148, -0.00042420602, 0.025900947, 0.00059231964, -0.010745705, 0.012400977, -0.005474611, 0.001418684, 0.015684387, -0.022020554, -0.025181854, 0.021844173, 0.01721755, 0.008133901, -0.0022251208, 0.012048215, 0.00882586, 0.004948858, 0.0019690283, 0.00067923847, 0.013527106, -0.030419027, -0.01652559, 0.0013347332, -0.0233502, -0.0018757496, -0.004538432, -0.020487392, 0.019890409, 0.017000465, -0.01721755, 0.044828042, -0.011790426, -0.015182379, 0.0065227244, 0.020324579, 0.011424095, 0.0066346587, -0.018859254, -0.039835088, -0.007313049, -0.04230443, 0.045913465, 0.027284864, -0.0060071475, 0.0344351, -0.02644366, -0.0094974665, -0.016335642, 0.0036361727, -0.0122110285, 0.029116519, 0.017678855, 0.022644673, -0.030988876, 0.009599225, -0.013689919, 0.014856751, -0.0061428254, 0.04322704, -0.014571827, 0.033024047, -0.011145955, -0.025100447, 0.0044400655, -0.015697954, 0.04249438, -0.046157688, 0.016308507, 0.011892185, -0.008215308, 0.02117935, 0.016023582, 0.013208263, -0.04859989, -0.008303499, -0.00441293, 0.012516304, 0.039672274, 0.02321452, -0.02235975, -0.041788854, -0.012590927, 0.019619051, -0.0007059501, 0.0119735915, 0.030581841, 0.009558521, -0.027366271, 0.012828363, 0.014788912, 0.023635123, 0.021477843, 0.02434065, -0.03264415, -0.0148974545, 0.011817561, -0.026213007, 0.0050743604, 0.0073537524, -0.043525532, 0.041788854, -0.028410992, 0.04477377, 0.0023777585, 0.06262901, -0.028193908, -0.03671449, 0.030364757, 0.0005927437], "content_hash": "5484ab5b8e0d34389726fd0c5cfcfd7e"}, {"content": "Generation Choices (Beams)**: How many parallel paths to explore (1-5). With 1 beam, you get greedy decoding. More beams = more diverse results but slower. - **Number of New Tokens**: How many tokens to generate beyond your prompt (1-20). Longer generations take more time. ## How Generated Sequences Are Used\n\nAfter generation, the dashboard displays the resulting sequences. You can:\n1. **Read them**: See how the model would continue your prompt\n2. **Select one for comparison**: Click \"Select for Comparison\" to use that sequence as a baseline in ablation experiments. This lets you see how removing attention heads changes the full generated sequence, not just the immediate next token.\n\n## Greedy vs. Beam Search\n\n| Feature | Greedy (1 beam) | Beam Search (2+ beams) |\n|---------|-----------------|------------------------|\n| Speed | Fast | Slower |\n| Quality | May miss better paths | Finds better overall sequences |\n| Diversity | Only one output | Multiple alternatives |\n| Best for | Quick exploration | Thorough analysis |", "source_file": "beam_search_and_generation.md", "chunk_index": 1, "embedding": [0.0410936, -0.0025848488, 0.06470555, -0.009679434, -0.010940693, -0.025797158, -0.01522311, 0.03317406, 0.014878463, 0.05218095, 0.02418392, -0.035608586, -0.03554992, -0.0116006555, -0.0039194375, -0.055730075, -0.013932519, -0.053852852, 0.02345063, 0.06546817, 0.008315514, 0.0024125257, 0.0010192738, 0.041592237, 0.0011604321, -0.00062833825, -0.0005018457, 0.038541745, 0.04584532, -0.028026361, -0.017408315, -0.04200288, -0.0086748265, -0.022776002, -0.054820795, -0.00119618, -0.030915525, 0.0059213205, -0.001807561, -0.021324087, 0.0018607245, -0.019681515, -0.022409357, 0.00056509196, 0.009312789, 0.009760096, -0.009312789, -0.056639355, 0.0058003278, 0.054087505, 0.0012878414, 0.07890205, -0.04549334, -0.0031403163, 0.006254968, -0.012817917, -0.043909434, -0.047605217, 0.009466779, -0.016279047, 0.040330976, -0.016147057, 0.026119806, 0.047077246, 0.034347326, -0.0041064266, -0.041944213, 0.030064909, -0.022732005, 0.037603136, -0.010515385, 0.003316306, -0.00896081, -0.0048763817, -0.020708123, 0.041269585, 0.035872567, 0.004627063, -0.009386118, -0.016455038, 0.043234807, 0.0049057133, -0.0118279755, -0.037485808, 0.0011384335, -0.024755886, 0.052650254, -0.040888276, -0.0250932, -0.040770948, -0.04678393, 0.007424566, -0.044496067, 0.02067879, 0.0126785925, 0.015355102, 0.019505527, -0.03642987, -0.002529852, 0.027865037, 0.024066593, -0.016117724, -0.028261015, -0.011160681, 0.045376014, -0.01900689, -0.04115226, 0.01133667, -0.015017789, -0.032323442, -0.047751874, -0.020561464, 0.0035931233, 0.019798843, 0.022145372, 0.0015078285, -0.023142647, 0.0060129818, 0.04660794, -0.004264084, -0.05314889, -0.018537583, -0.008704158, -0.0133605525, -0.030827532, 0.010684042, 0.051946297, -0.061889715, -0.009664768, 0.013690533, 0.028524999, 0.016543033, 0.019754846, -0.022761336, -0.048338506, 0.021881387, -0.038952388, 0.015531092, -0.062828325, 0.036224548, -0.0037434478, -0.022805333, -0.010933361, 0.0026600112, -0.04478938, 0.013257892, -0.037485808, 0.017701631, -0.042530846, 0.023861272, 0.008256851, -0.031326167, -0.05188763, -0.021368084, 0.016059062, -0.021177428, -0.023289304, 0.027513059, -0.054087505, -0.012656594, -0.012641927, 0.02673577, -0.017027004, -0.011292673, -0.044672053, 0.03464064, -0.044642724, 0.036840513, -0.028290346, -0.007417233, 0.028700989, 0.0052466933, -0.00035908318, -0.005034039, 0.037955113, -0.023186645, -0.10876164, -0.06200704, -0.04364545, -0.057225987, -0.010229402, 0.018816233, -0.03807244, 0.011783978, -0.023201311, 0.0011668485, 0.0024143588, -0.0077875447, 0.017994948, -0.0144971525, -0.004590398, 0.02302532, 0.0702199, -0.026471786, -0.046490613, -0.04989308, 0.07567558, 0.038952388, -0.015428431, -0.030856863, 0.025679832, -0.036048558, 0.011849973, -0.018640244, -0.0003668744, -0.00239236, -0.019270873, -0.004084428, -0.018478919, -0.004036764, 0.021514742, -0.02229203, 0.03100352, 0.008616162, -0.04080028, 0.0058736564, 0.004872715, 0.03147283, 0.0015105783, 0.03807244, 0.002520686, -0.030739537, -0.00080111984, 0.03678185, 0.008718824, 0.032880746, -0.0015857406, -0.03009424, 0.0080221975, -0.013668534, 0.032910075, 0.016440373, -0.0043740775, -0.010119408, -0.013397217, -0.006526285, -0.012165288, -0.004949711, 0.032059457, -0.066054806, -0.022233367, -0.02372928, 0.018156271, -0.012077294, -0.02827568, 0.033995345, -0.064118914, -0.039891, -0.016601697, 0.03499262, 0.0019670518, -0.015399099, 0.0066436115, 0.010471388, 0.015589755, -0.0046417285, -0.03261676, 0.02229203, -0.030211566, -0.009474113, -0.0066179466, 0.023567956, 0.012517268, -0.0135585405, -0.058956552, 0.032528765, -0.018625578, -0.048954472, -0.06887064, -0.054322157, -0.0055583417, -0.04382144, 0.00094777794, -0.0075088944, -0.048866477, -0.04182689, 0.012209286, -0.011351337, -0.02355329, -0.024022596, -0.048103854, 0.004003766, 0.036693856, 0.0038461084, -0.0011668485, -0.015457762, 0.033760693, 0.017936284, 0.01946153, -0.021778727, 0.0023960266, -0.009452114, 0.014922461, -0.03619522, -0.005836992, 0.034054007, 0.0359019, -0.006353962, -0.03801378, 0.0012456771, 0.021852056, -0.021500077, 0.032294113, -0.02403726, -0.007516227, 0.025225192, -0.039627016, 0.044320077, 0.0038791066, 0.0076995497, 0.008777487, 0.043762773, -0.025870487, -0.064412236, -0.0053456873, -0.01704167, 0.039040383, -0.05889789, 0.040946938, -0.057607297, -0.02526919, -0.037573803, -0.001122851, 0.008476838, -0.016381709, -0.014159839, -0.016000398, -0.023567956, 0.009305456, -0.023230642, 0.013617204, -0.009180796, 0.013477879, 0.032323442, -0.050685037, -0.04528802, -0.00039826837, 0.036840513, 0.027806373, 0.03261676, -0.037867118, -0.0007809543, -0.0038057775, -0.02138275, -0.028349008, -0.006104643, -0.0076482194, 0.040477633, 0.029346284, 0.01168865, -0.0013648368, -0.03915771, -0.011446664, -0.0039414363, 0.043616116, 0.012685925, 0.010867365, -0.0273664, -0.014783136, 0.018566914, -0.018801568, 0.03801378, -0.0011705148, 0.025665166, 0.004069762, -0.032851413, -0.017760295, 0.0133605525, 0.027630385, 0.0025280188, -0.041592237, 0.025899818, 0.0017314821, 0.01974018, -0.065526836, 0.025885154, -0.014504485, 0.013030572, -0.006258634, 0.06001249, -0.031795476, 0.003241144, -0.054644804, -0.059748508, 0.00672794, 0.05042105, -0.002295199, 0.011644652, -0.049981076, -0.028378341, -0.010698708, -0.013961851, 0.019798843, 0.005386018, -0.018156271, 0.049042467, 0.003525294, 0.038541745, 0.048162516, 0.04118159, 0.0353446, -0.054234162, -0.031208841, -0.027806373, -0.018904228, 0.005408017, 0.025181195, 0.020180153, 0.05147699, 0.01984284, 0.0438801, 0.0575193, 0.04874915, -0.043792106, 0.0015866573, 0.007611555, -0.00038956056, 0.03936303, 0.022878662, 0.054996785, -0.020928109, -0.06276967, -0.057724625, -0.017202994, -0.022907995, -0.009364119, -0.0050047073, -0.006251301, 0.04437874, -0.023890603, -0.015692417, 0.059484523, 0.05329555, -0.0128765805, 0.0012264283, 0.008806818, 0.057372645, 0.03329139, -0.015281773, -0.045258686, 0.02229203, -0.007728881, -0.016733687, -0.0032484767, -0.039803006, 0.006555617, -0.061713725, -0.0176723, -0.007875539, -0.020268148, -0.03968568, 0.00934212, -0.017129665, 0.027879704, -0.03062221, -0.004418075, 0.07784612, -0.035960563, 0.061479073, 0.002390527, 0.015355102, -0.040858943, -0.015384434, 0.0013960017, 0.014409157, 0.04405609, -0.0015692416, -0.007387901, 0.02600248, 0.020722788, 0.0032686421, -0.0018222268, -0.004707725, -0.027982363, -0.009232127, -0.007758213, 0.017276322, 0.024051927, 0.0029496609, -0.020180153, -0.017862955, 0.0444374, 0.034581978, 0.05479146, -0.037221823, 0.09086935, 0.043762773, -0.014636477, 0.015941735, -0.02873032, -0.014035179, -0.029888919, 0.026134472, -0.018420257, 0.017056337, -0.03100352, -0.052708916, 0.024550565, -0.017862955, 0.0037947781, 0.02022415, 0.008535501, -0.020488136, 0.032323442, 0.032470103, -0.041416246, 0.046842594, 0.036488533, -0.0040734285, 0.0068416, -0.02428658, -0.039744344, 0.0004954294, 0.011857307, -0.0135878725, 0.029918252, 0.0052320273, 0.025445178, 0.017364318, 0.025489176, 0.026589112, -0.007985533, -0.0033126397, 0.011754646, 0.016880346, -0.061948378, -0.04147491, -0.064236246, -0.013463213, -0.013507211, 0.033144727, -0.041005604, 0.003646287, 0.025474511, -0.08195254, -0.03818977, -0.00012912786, 0.016513702, -0.023699947, -0.005664669, -0.027058419, 0.019065551, 0.009532776, 0.010280732, 0.0015059953, -0.0071899127, 0.020238817, 0.009540109, 0.0037654466, 0.0008620746, 0.018346928, -0.03698717, 0.033379383, -0.03601923, -0.028671658, 0.0040734285, -0.02694109, -0.04074162, 0.03481663, -0.020664126, -0.010361394, 0.008271516, 0.037603136, 0.0024015263, -0.025738494, 0.007160581, -0.0003441882, -0.006232969, 0.0040660957, -0.023538623, -0.020590795, -0.015105783, 0.019828174, -0.0025500176, 0.00530169, 0.003956102, 0.003411634, 0.02981559, -0.016088393, 0.014599813, 0.005609672, -0.029243624, -0.007175247, -0.004484071, -0.03510995, -0.02236536, -0.019696182, 0.02663311, 0.010625379, 0.016997673, 0.029375616, 0.010764704, 0.0020312145, 0.006702275, -0.01928554, -0.01363187, -0.02873032, -0.03027023, -0.017965617, -0.013221227, 0.027219743, -0.025049202, -0.036165886, 0.0073475703, -0.03100352, -0.0016416541, -0.028363675, -0.0032374773, 0.025562506, -0.013661201, -0.020884112, -0.00042874576, -0.0091294665, -0.017246991, -0.008777487, 0.02862766, -0.015010456, 0.0016599863, 0.03898172, -0.012957242, 0.023641285, 0.017246991, 0.026251798, 0.025899818, 0.019270873, 0.010478721, 0.0038791066, -0.03437666, 0.008821485, 0.00547768, -0.006515286, -0.0028250015, -0.012766587, 0.0015949068, -0.008176189, 0.0427655, 0.014020514, 0.02827568, -0.013059903, -0.0024491902, -0.0042970823, 0.008557499, -0.008924145, -0.035931233, -0.02100144, 0.013045237, 0.011241343, -0.006460289, -0.0014482486, -0.03044622, -0.048837144, -0.019241542, 0.06722807, -0.02330397, 0.027806373, -0.023861272, -0.008396176, -0.022174703, -0.008638161, -0.017862955, 0.004718724, 0.00706892, -0.04279483, -0.047399893, 0.010508052, 0.051770307, 0.025181195, 0.0422082, -0.010544716, -0.016631028, -0.035931233, 0.05564208, 0.007728881, -0.008168856, -0.04349879, -0.023861272, -0.00090744696, -0.026691772, -0.002861666, 0.0133605525, -0.011014023, 0.0032136454, -0.04182689, -0.029830256, 0.047194574, 0.0054850127, 0.025049202, -0.004594065, 0.04528802, 0.012641927, -0.008462172, 0.036165886, 0.011102018, 0.029859588, 0.0165577, -0.029199626, 0.0019872172, 0.04097627, -0.018214935, -0.057049997, -0.028788984, 0.021177428, 0.016235052, -0.012304614, -0.010434723, -0.0031201509, -0.03733915, 0.03733915, -0.005595006, 0.0018845565, -0.014533817, -0.020942776, -0.0095841065, 0.027879704, 0.0024766885, -0.007970868, -0.024755886, -0.040360305, 0.005690334, -0.009488778, -0.012121291, 0.015384434, -0.03329139, -0.019094884, 0.00067737704, -0.0159564, 0.008880148, 0.007391568, -0.028011696, 0.03516861, 0.02130942, 0.0046783932, -0.012216619, 0.007860874, -0.008513502, 0.02390527, -0.02918496, -0.013067236, -0.013771195, -0.013477879, 0.027777042, 0.0025243524, 0.035872567, -0.0046343957, 0.0064749545, -0.0001907701, -0.0043154145, 0.010735372, 0.00021723729, -0.025151862, -0.0051183673, 0.014262499, 0.0006155057, -0.02330397, 0.0541755, 0.006922262, 0.029786259, -0.011622653, 0.010060745, 0.04223753, 0.011131349, -0.00054950954, -0.033496708, 0.028481001, -0.0004035389, -0.007369569, -0.016675025, -0.008161522, -0.004744389, 0.011380668, 0.0037086166, 0.014416491, 0.018962892, -0.004627063, 0.05188763, -0.035080615, 0.052122287, -0.014225835, -0.0496291, -0.005855324, 0.007142249, 0.013067236, -0.029947583, 0.0042604175, -0.025342518, 0.010801368, 0.00067554385, -0.02309865, -0.0008794902, -0.03437666, 0.0029734927, -0.014555816, 0.028495668, -0.003347471, -0.02100144, 0.009716098, 0.0010641877, -0.12647794, -0.022204036, 0.004971709, -0.0122239515, -0.0021173763, -0.02739573, 0.04291216, 0.015648419, -0.013917853, 0.0159564, -0.025474511, -0.031062184, 0.029947583, -0.0017424815, 0.021808058, -0.04200288, 0.0020000497, -0.018640244, 0.014907795, -0.02617847, -0.024799883, 0.010852699, -0.009422783, -0.013382551, -0.025870487, -0.000774538, 0.003145816, 0.009298123, 0.0013923353, -0.024315912, -0.015824407, 0.056756683, -0.025811825, 0.04115226, 0.0061339745, -0.008366844, -0.009217461, -0.02120676, 0.015516426, 0.004194421, -0.007123917, -0.023861272, 0.022658676, -0.0052430267, 0.0049790423, -0.0059873164, 0.00888748, 0.0035656248, 0.026838431, 0.023597287, -0.005972651, -0.004594065, -0.006973592, 0.00017770835, 0.022204036, -0.015017789, 0.0033438045, 0.015883071, 0.028583663, 0.010112075, 0.021250758, 0.03062221, 0.044818714, -0.033203393, 0.01141, -0.034669973, 0.024755886, 0.008689492, -0.016851015, 0.007765546, -0.016792351, 0.011373335, -0.016059062, 0.015325771, -0.007890205, 0.0124512715, 0.04549334, 0.029888919, -0.013529209, -0.01991617, 0.019696182, -0.039597686, -0.029302286, 0.034405988, 0.0061743055, 0.020004164, -0.017540308, -0.026207801, 0.00091615476, 0.029170295, 0.023245307, -0.027953032, -0.0004638062, 0.007065254, -0.059132542, -0.015941735, -0.0038131103, 0.013881189, 0.027014421, 0.004590398, 0.04534668, 0.02782104, -0.014152505, -0.039245706, 0.0052466933, 0.014599813, -0.0010806868, 0.0014289998, -0.017202994, -0.006672943, 0.017686965, 0.01949086, 0.0016746521, 0.008916812, -0.016117724, 0.029170295, 0.04531735, -0.0008354928, 0.0055876733, -0.011307339, 0.0025756827, -0.005122034, -0.00078828726, 0.014885796, 0.0122826155, 0.013089235, 0.012539267, -0.009613438, 0.010676709, 0.00092807075, -0.005276025, 0.016807018, 0.020532133, -0.001030273, -0.0048763817, -0.00059992325, 0.0025023536, -0.027043752, -0.070102565, 0.0047957194, -0.024887878, -0.0039267703, 0.053324882, -0.0068709315, -0.019270873, -0.03464064, -0.025166528, 0.0037837788, -0.012832583, 0.016748354, 0.01855225, -0.019813508, 0.0008749072, 0.016352378, -0.017877622, 0.023171978, 0.00019054094, -0.017056337, 0.0043007485, 0.018420257, 0.038512416, 0.021192094, 0.0541755, 0.007868207, 0.013221227, -0.0036169551, -0.0069442606, 3.242519e-05, 0.0071385824, 0.0038791066, -0.029551605, -0.017202994, 0.026603777, 0.027278405, 0.055143442, 0.0037874451, 0.017789626, 0.036693856, 0.007516227, 0.0507437, 0.012121291, 0.045786656, -0.0066252793, 0.030974189, 0.006185305, -0.016161721, -0.012656594, 0.005910321, 0.02113343, -0.01868424, -0.0070322556, 0.045962647, 0.007230244, 0.02953694, -0.003517961, 0.014783136, 0.029551605, -0.0199455, -0.03062221, -0.009386118, 0.017246991, 0.021338752, -0.0065372842, 0.011659319, 0.0117986435, 0.030827532, 0.0006530868, 0.035637915, 0.023010654, 0.04346946, -0.04546401, -0.02438924, -0.014482487, 0.0051696976, -0.03862974, 0.03173681, 0.010302731, -0.007028589, -0.025342518, 0.0012355944, -0.06288699, -0.030123573, -0.019329537, 0.003598623, -0.017173663, -0.024051927, 0.022614678, 0.020077493, -0.045757324, 0.012502602, -0.004707725, -0.018258933, 0.023318637, -0.01913888, -0.00650062, 0.027879704, -0.023978598, 0.012839916, -0.0038204433, -0.00073145726, 0.048133187, 0.001511495, 0.00199455, -0.008080861, -0.026618443, -0.028847646, -0.00071220833, -0.011930636, 0.011886639, -0.02113343, -0.013426548, 0.024447903, 0.026339794, -0.020180153, 0.013807859, -0.017408315, 0.0036444536, 0.022262698, 0.034669973, 0.0075895563, 0.0013547541, 0.030680872, 0.008667493, -0.024169253, 0.006493287, -0.014101176, -0.0068782647, 0.0046380623, 0.037984446, -0.029170295, 0.003550959, 0.025386516, -0.0030761536, 0.04382144, -0.010772036, 0.0077362144, 0.00014665809, -0.016997673, 0.02236536, -0.021984048, 0.0012355944, -0.011050687, -0.008447506, -0.023113316, -0.03153149, -0.010706041, 0.00027315068, 0.013939852, 0.027659716, 0.002350196, 0.018346928, 0.032880746, 0.0012291782, 0.0032448103, -0.020048162, -0.015883071, 0.0019835506, 0.024653226, 0.026985088, -0.026662441, -0.015267108, 0.0050890357, 0.009481446, -0.022893328, 0.013719864, 0.017554974, 0.039128378, -0.04150424, -0.015369768, -0.0017608138, -0.008183521, 0.053207554, -0.0124512715, 0.009481446, -0.0001843538, -0.0116006555, -0.012062628, -0.006493287, 0.00055730075, -0.008462172, -0.015868405, -0.01282525, -0.015765745, 0.033027403, 0.0012896745, -0.030020911, 0.00024152754, 0.0020385475, -0.0056573357, 0.025078533, 0.026339794, -0.023142647, 0.032000795, -0.012047962, -0.013485212, 0.022424022, -0.0467546, 0.0039267703, 0.006518952, -0.011651985, 0.008058862, -0.0017204827, -0.012568599, -0.006401626, 0.0071202503, 0.019637518, 0.0155457575, -0.010948027, 0.015311105, 0.004553734, 0.08746689, -0.014035179, -0.020664126, 0.002927662, -0.011952635, -0.0061669727, -0.026574446, -0.0056610024, -0.012187287, -0.01009741, -0.004480405, 0.021177428, 0.031150179, 0.0018772236, 0.02830501, -0.0093274545, 0.008850816, -0.015384434, 0.034523316, -0.008132191, 0.025063869, 0.04511203, -0.0032393106, 0.012407275, -0.0032741418, 0.024403907, 0.026266465, 0.004605064, 0.0066399453, 0.033056732, 0.002286033, -0.023494625, -0.010794035, 0.0009771095, -0.02012149, -0.0065666162, 0.0025555172, 0.031766143, -0.01718833, 0.004007432, -0.009312789, 0.00593232, 0.014438489, 0.0060753115, 0.022130705, 0.00020027996, 0.018508252, 0.019828174, 0.010390726, -0.032059457, -0.018640244, -0.003088986, -0.010060745, -0.019080218, 0.0433228, -0.07033722, 0.03208879, 0.024301246, 0.008594165, 0.0018057277, -0.04291216, 0.013140565, -0.03772046, 0.010896696, -0.0023465294, -0.08113126, -0.002748006, 0.0006081728, 0.0071459156, 0.044173416, -0.012341279, -0.016000398, 0.00883615, 0.019798843, 0.019784177, 0.009562108, 0.0060569793, -0.019886836, -0.0016966508, 0.01133667, -0.0072045787, -0.028935641, -0.035461925, -0.036107223, -0.021016104, 0.022482686, 0.04625596, 0.026017146, -0.026985088, 0.038952388, -0.03783779, 0.015516426, 0.04200288, 0.07778745, 0.0088728145, -0.01701234, -0.013609871, -0.0111166835, -0.0072485763, -0.0025426846, 0.032734085, 0.004480405, -0.020517467, 0.008484171, -0.04402676, -0.031326167, -0.029170295, -0.006493287, 0.019872172, 0.0029606603, -0.0072265775, 0.021192094, 0.015663084, 0.0098114265, -0.02418392, 0.02512253, 0.0039451024, 0.020136155, 0.0131039005, -0.044877376, 0.03437666, 0.024975874, -0.005650003, 0.004058763, 0.00087261567, -0.024858547, 0.0023483627, 0.009444781, 0.014753804, 0.0062733, -0.0052246945, 0.005631671, 0.014526484, 0.03173681, 0.034493983, -0.0017543974, -0.018185603, -0.018669575, 0.0070725866, 0.036869843, -0.020253483, 0.020722788, -0.04241352, -0.017408315, 0.010024081, 0.02257068, 0.014284498, -0.007457564, -0.011930636, 0.010478721, -0.013030572, -0.01855225, 0.014394492, -0.010750039, 0.015032454, -0.025738494, 0.00273884, 0.012957242, 0.007402567, -0.021338752, 0.030710204, 0.010852699, -0.02526919, -0.0021283755, 0.028363675, 0.006526285, -0.011351337, -0.017071001, -0.010647377, 0.032352775, -0.007948869, -0.018478919, -0.011959967, 0.008066195, 0.0041650897, -0.0135585405, -0.011439331, 0.0047370563, 0.010082744, 0.026559781, -0.0020147155, 0.011102018, 0.008572166, -0.013514543, -0.010764704, 0.0133605525, -0.002295199, -0.0017663134, 0.0035802906, 0.03572591, 0.0055876733, -0.061127093, -0.0026545115, -0.0013905021, 0.02236536, -0.012517268, -0.025943816, -0.010368727, 0.025958482, 0.023391966, -0.012069961, 0.017452313, -0.0016040729, 0.03769113, -0.03390735, 0.0039414363, -0.007109251, 0.02648645, -0.0033511375, -0.011549325, -0.016719023, -0.0030999854, -0.020312145, -0.017349653, 0.04244285, -0.04698925, 0.0021467078, 0.008821485, -0.016205719, -0.03437666, 0.014071844, -0.00067004416, 0.0073989006, 0.00031966882, -0.037925784, 0.0035124612, -0.00049497106, 0.0047700545, 0.024975874, 0.008630829, 0.0064492896, -0.014944459, -0.020605462, 0.01105802, -0.023142647, -0.0064162915, -0.03282208, -0.008491503, 0.0032264781, 0.019109549, -0.036107223, 0.011527326, 0.0021962048, 0.01739365, -0.027307736, 0.050127737, -0.007178914, -0.017598972, -0.0059433193, 0.004854383, 0.017100334, 0.020957442, -0.031766143, -0.020517467, -0.012113959, -0.03769113, -0.005496012, -0.005895655, -0.02691176, 0.01984284, -0.010691375, 0.002163207, 0.0046563945, -0.00021242508, -0.016088393, 0.0033896351, 0.012187287, -0.02145608, -0.012590597, 0.043381464, -0.010508052, 0.0017809792, -0.044173416, -0.025386516, 0.024579896, 0.02554784, -0.008491503, 0.010647377, -0.005968984, 0.019593522, 0.018141607, 0.011813309, -0.0055876733, -0.019901503, 0.017422982, 0.013015905, 0.004458406, 0.0119966315, -0.013675867, -0.018244267, 0.020341476, 0.032294113, 0.011717982, -0.007692217, -0.005122034, -0.01209196, 0.0059066545, -0.0091001345, -0.0034996287, 0.025987813, -0.03390735, -0.016381709, 0.012751921, -0.014225835, 0.0042970823, 0.010060745, -0.020444138, 0.034963287, 0.01966685, -0.0037727794, 0.035931233, -0.008117526, -0.012590597, 0.006485954, 0.032206118, 0.0205468, -0.004304415, 0.010207403, -0.037045833, 0.0015912403, -0.053970177, 0.034288663, 0.021045437, -0.0211041, 0.01855225, -0.017525641, -0.0033034736, -0.014951793, 0.019534858, -0.013551207, 0.03493396, 0.019270873, 0.007875539, -0.020414807, -0.00070258393, -0.022556014, 0.0115419915, -0.014724473, 0.042589508, -0.007838875, 0.016117724, -0.00752356, -0.022174703, -0.00951811, -0.02330397, 0.0393337, -0.04135758, 0.013353219, 0.003655453, -0.013903187, 0.019373534, 0.025841156, 0.015237776, -0.024931876, -0.0014180004, -0.00096611015, 0.009804093, 0.024755886, 0.020414807, -0.015252441, -0.037456475, -0.014225835, 0.035432596, 0.0005687584, 0.0063906265, 0.022805333, -0.006705941, -0.03238211, 0.0025445179, 0.013213894, 0.014907795, 0.037045833, 0.029551605, -0.028994305, -0.0051330333, 0.008198188, -0.021602737, -0.0022163703, 0.02229203, -0.01830293, 0.023259973, -0.030886194, 0.023010654, -0.00067004416, 0.044877376, -0.033232722, -0.0290823, 0.021294754, 0.004553734], "content_hash": "a391a36b69807fd98403083dcc620382"}, {"content": "# Dashboard Overview\n\n## What Is This Dashboard?\n\nThe Transformer Explanation Dashboard is an interactive web application that lets you look inside transformer-based language models as they process text. Instead of treating the model as a \"black box,\" you can see exactly what happens at each step when the model makes a prediction.\n\n## How to Navigate\n\nThe dashboard has several main sections, from top to bottom:\n\n### 1. Header and Sidebar\nAt the top is the dashboard title. On the left edge, there's a collapsible **sidebar** (click the hamburger menu icon). The sidebar contains advanced configuration options for selecting which internal model components to hook into. Most users won't need to change these settings.\n\n### 2. Generator Section\nThis is where you start. It contains:\n- **Model dropdown**: Choose which transformer model to load (e.g., GPT-2, Qwen2.5)\n- **Prompt input**: Type the text you want the model to analyze\n- **Generation settings**: Control beam search parameters (number of beams and tokens to generate)\n- **Analyze button**: Click to run the model and see results\n\n### 3. Generation Results\nAfter clicking Analyze, you'll see one or more **generated sequences** -- these are the model's continuations of your prompt. If you used beam search with multiple beams, you'll see multiple possible continuations. Click \"Select for Comparison\" on any sequence to use it in ablation experiments.\n\n### 4. Pipeline Visualization\nThis is the core educational section. It shows **5 expandable stages** that your text passes through:\n1. **Tokenization**: How your text is split into tokens\n2. **Embedding**: How tokens become number vectors\n3. **Attention**: How the model finds relationships between tokens\n4. **MLP**: How stored knowledge is retrieved\n5. **Output**: What the model predicts and how confident it is\n\nClick any stage to expand it and see detailed explanations and visualizations.", "source_file": "dashboard_overview.md", "chunk_index": 0, "embedding": [-0.0074431063, 0.029747594, 0.009833095, -0.010689766, -0.00012260329, -0.004956898, -0.01614018, 0.04390129, -0.017878354, 0.063666806, 0.011316749, -0.039580688, -0.017965263, -0.035409074, 0.0029347194, -0.008386686, -0.008368063, 0.032081712, 0.017381733, 0.097536355, 0.03558289, -0.038115658, 0.020137979, 0.0474273, -0.011329166, -0.0052300394, -0.011937526, 0.041865144, -0.019293724, -0.0070458096, 0.0078093642, -0.024483413, -0.033745393, -0.00080157723, 0.0028276355, -0.027314153, 0.003774319, 0.038786095, -0.011745085, 0.03084016, 0.0070458096, 0.002866434, -0.026594052, 0.06401444, -0.035359412, 0.04337984, -0.022534175, -0.021553349, -0.03819015, 0.029449621, 0.0072072116, 0.034937285, -0.032603163, -0.0044416534, -0.013123209, -0.008926761, -0.06356748, -0.008703282, -0.026842361, -0.033074956, 0.00640641, 0.005993594, 0.0048606773, 0.07434416, -0.010242807, -0.012651418, -0.023862636, 0.012055473, -0.035682216, -0.009764809, 0.044745546, -0.0028881612, -0.003805358, -0.0018157703, 0.005503181, -0.020733925, 0.00572666, 0.040648423, -0.04320602, -0.039481364, 0.002866434, -0.039531026, -0.022608668, -0.018312898, -0.0043299138, 0.00464651, -0.019318555, 0.018573623, 0.0017521407, -0.06912964, -0.015804961, -0.0022006514, -0.046086423, 0.020609768, 0.037395556, 0.021503687, -0.045962267, -0.007890065, 0.0017521407, 0.049835913, 0.049314458, -0.037321065, 0.02041112, -0.017158253, 0.037519712, -0.037097584, 0.013408765, -0.028307393, -0.0455898, -0.014786889, -0.08492218, -0.02773628, 0.0036780988, 0.0076231314, 0.022310697, 0.015345587, 0.015829792, -0.010801505, -0.011298127, -0.043181192, -0.034515154, 0.0007845059, 0.010609065, 0.006431241, -0.036104344, 0.024309594, 0.010180729, -0.012620379, -0.004252317, 0.025575979, -4.665521e-05, 0.006431241, -0.0065057343, -0.0488675, -0.02753763, 0.0038426046, -0.045142844, 0.0219134, -0.021863736, 0.030393202, 0.043007374, -0.001118949, -0.009944835, -0.00091176503, -0.06530565, -0.025973275, -0.011012569, 0.03081533, -0.05408202, 0.018921258, 0.012192044, -0.007480353, -0.030442864, -0.047129326, 0.01885918, -0.018772272, 0.013892971, 0.028183239, 0.0010235048, -0.05189689, 0.008194245, 0.034713805, -0.042187948, -0.011322957, -0.06192863, 0.016537478, -0.029822087, -0.005580778, -0.053088777, -0.026867192, -0.014041957, -0.024272349, 0.005965659, -0.0016698879, 0.009864134, -0.011006362, -0.067043826, -0.07657895, 0.0018437052, -0.041914806, -0.019380633, 0.010658727, -0.029052325, -0.0026724415, -0.03203205, -0.008119753, -0.020448368, 0.029350298, -0.009423383, 0.0028710898, -0.018561207, 0.048395712, 0.024595153, -0.0038301891, -0.0383888, 0.01026143, 0.057955664, 0.023043212, 0.036228497, 0.013284611, 0.04760112, 0.02582429, 0.029126817, -0.026469896, -0.010354547, 0.06942761, -0.014054373, 0.020249719, -0.01744381, -0.040449776, 0.03516076, -0.025898783, 0.021292623, 0.013979879, -0.03647681, -0.0206346, -0.04171616, 0.03195756, 0.03081533, 0.023875052, -0.019877253, 0.035483565, 0.007983182, -0.00918128, -0.008703282, 0.053039115, 0.012353446, -0.026867192, -0.0037929425, 0.043727472, 0.03203205, -0.004109538, -0.016040856, 0.010342131, 0.024247518, -0.07737354, -0.0040226295, -0.016425738, 0.022025138, 0.0006328037, -0.006636097, 0.016624385, 0.004143681, 0.018077003, 0.04623541, 0.022559006, -0.047030002, -0.0669445, 0.033819888, 0.052989453, 0.04171616, 0.022919057, -0.033397757, -0.0030914654, 0.002579325, 0.015581482, -0.04037528, 0.010571818, -0.020324212, 0.0005532668, 0.020423535, 0.005341779, -0.053237766, -0.021652672, -0.011124309, -0.0051462343, 0.0028245316, -0.028034251, -0.051151957, -0.044199266, 0.0029455828, -0.027636955, -0.0040443568, 0.03868677, -0.055919517, -0.03225553, 0.035011776, -0.005922205, 0.009212319, 0.017990094, -0.011614722, -0.03764387, 0.03729623, 0.031361613, -0.016165012, 0.009684108, 0.02701618, -0.008324608, 0.0102365995, -0.024570322, 0.026767869, 0.0038922667, 0.042014133, -0.02704101, -0.020435952, 0.018735025, 0.016810618, 0.021093974, 0.014029542, -0.006716798, 0.011353997, -0.048768178, 0.015047614, 0.01364466, 0.018672947, 0.01705893, -0.010745635, 0.0017924912, 0.012136174, -0.012061681, -0.034912452, -0.019169569, 0.026544388, -0.016227089, 0.015705638, -0.008343232, 0.041219536, 0.003985383, 0.026718207, -0.03647681, -0.027388645, -0.026743038, -0.015904285, -0.00995725, -0.019529618, 0.02074634, -0.012682457, 0.0041778237, -0.013272194, -0.004305083, 0.06451106, -0.021975476, 0.021205714, 0.011273296, -0.044671055, -0.033770222, 0.023316354, 0.03913373, -0.059197217, 0.042560413, -0.06570295, -0.01506003, -0.07528774, -0.04489453, -0.027810773, 0.03156026, -0.023887467, 0.03277698, 0.027761111, -0.0021773723, -0.027487969, -0.04228727, -0.01636366, -0.023949545, 0.03317428, 0.04534149, -0.042113457, 0.00884606, 0.010180729, -0.014501331, -0.01836256, 0.010801505, 0.02820807, 0.013930217, -0.034912452, -0.0061146454, -0.034068197, 0.009144033, 0.020137979, -0.022198956, 0.012787989, 0.007939727, 0.02915165, 0.004680652, -0.05711141, -0.040474605, 0.00865362, -0.028580535, -0.04933929, 0.04489453, 0.03488762, 0.0053541944, -0.044671055, -0.015953947, 0.0282329, 0.050407026, 0.0011492119, -0.0077472865, -0.036104344, 0.04745213, 0.017332071, -0.055373237, -0.0030418034, -0.029275805, -0.007629339, 0.029871749, 0.02584912, 0.0032528671, 0.05904823, 0.00089934946, 0.012949391, 0.0058570234, -0.0611837, -0.035185594, -0.0057142447, 0.011409866, 0.016748542, 0.01955445, 0.01542008, -0.024595153, 0.05477729, -0.03503661, 0.0066857594, -0.02481863, 0.008504634, 0.009454421, 0.01638849, 0.037097584, 0.034266844, 0.04077258, -0.03602985, -0.030020736, -0.043280516, 0.0009482356, -0.050009727, 0.047253482, 0.029002663, -0.027910097, -0.0051679616, -0.033074956, 0.02041112, 0.033074956, 0.04534149, 0.0013137176, 0.01314804, -0.061829306, 0.057260394, 0.019678604, 0.0026693377, 0.002917648, 0.020932572, -0.036923766, -0.004395095, -0.027785942, -0.045142844, 0.034515154, -0.012974222, -0.067639776, -0.026892023, -0.04648372, 0.02560081, 0.04313153, -0.0029905892, 0.00942959, -0.05293979, -0.030542187, 0.03846329, 0.012074097, 0.06664653, 0.021454025, 0.026072599, 0.04293288, -0.022136878, 0.0074927686, -0.02121813, -0.0018995751, 0.0003798374, -0.030169722, 0.014128866, -0.00718238, -0.021230545, -0.015817378, 0.005099676, -0.01245277, -0.015457327, -0.010404209, 0.03508627, -0.032603163, -0.00037091377, -0.04628507, -0.027388645, -0.0025622537, -0.022062385, 0.016674047, -0.004190239, 0.10548229, -0.058501948, -0.007120303, 0.005965659, -0.01469998, -0.0051772734, -0.020920157, 0.018077003, -0.012595548, 0.022894226, -0.007486561, -0.06252458, -0.013259779, -0.02099465, -0.003982279, 0.015109692, 0.020349042, -0.0136074135, 0.07459246, 0.031609923, 3.157229e-05, 0.019244062, 0.031932727, 0.013979879, 0.011602307, -0.052095536, -0.03602985, 0.00918128, 0.009702732, -0.011236049, 0.013359103, 0.013421181, 0.0047023795, 0.035135932, 0.013023884, 0.0050251833, -0.0156187285, -0.004196447, -0.023440508, -0.0002630539, -0.018623285, -0.053138442, -0.063368835, -0.045316663, -0.002138574, 0.0018902634, -0.028481212, -0.00027876731, 0.02468206, -0.076231316, -0.026966518, 0.018424638, -0.00829357, -0.03652647, -0.008231492, 0.018983336, 0.040275957, 0.019206814, -0.0050438065, 0.012620379, 0.01730724, -0.0051927925, -0.016736126, 0.01120501, 0.011099478, 0.025749795, -0.04645889, 0.030293878, -0.0227949, -0.0035880862, 0.010224184, -0.0039729676, -0.022372773, 0.002992141, -0.045838114, -0.0038426046, 0.014414423, 0.01827565, -0.014327514, -0.014128866, 0.020311797, -0.00051990006, -0.020709094, -0.026668545, -0.0044944193, 0.012825236, -0.00912541, -0.0016217778, 0.032876305, 0.0072506657, -0.011763709, -0.003755696, 0.01825082, -0.020758756, 0.01400471, -0.027214827, -0.045242168, -0.0211933, 0.021975476, -0.03342259, 0.040226296, -0.001223705, 0.011087063, -0.014836551, 0.00033696508, 0.029548947, 0.0010467839, -0.016798204, 0.016016025, -0.036104344, -0.02560081, 0.021665089, -0.010168314, -0.036923766, 0.0018173222, 0.0025995004, -0.040126972, -0.026345741, -0.012912145, 0.0075362227, -0.019231645, -0.011385035, 0.015295925, -0.008610166, -0.002791941, -0.018064586, -0.021776829, -0.03503661, -0.007008563, -0.010546987, 0.03908407, 0.002998349, 0.018598454, 0.04477038, -0.0126017565, -0.008020428, -0.03625333, 0.022621084, -0.037097584, 0.007939727, 0.0046837563, -0.005602505, -0.009243357, -0.04151751, 0.021503687, -0.037668698, 0.022360358, 0.053933036, -0.010391793, -0.0054007526, 0.057806678, 0.005763907, -0.015854623, 0.009851718, -0.02241002, 0.0038239812, -0.019417878, -0.023390846, -0.0022766967, 0.00776591, 0.018027341, 0.0016869593, 0.030293878, -0.038314305, -0.008976423, -0.049612433, -0.011993396, 0.06024012, 0.0068657845, 0.024595153, -0.002394644, 0.019703437, -0.016090518, -0.009342682, 0.008194245, -0.044944197, 0.017567966, -0.005940828, -0.027115503, -0.03560772, 0.04695551, 0.017468642, 0.025749795, 0.027463138, 0.012458978, -0.051598914, 0.021230545, -0.002998349, 0.019542035, 0.0050686374, -0.019157153, 0.022248618, -0.042634908, 0.006493319, 0.04936412, -0.01656231, -0.00385502, -0.03156026, 0.0282329, 0.03764387, 0.013334272, 0.007300328, -0.0052517666, 0.034763467, 0.017481057, -0.021441609, 0.008672243, 0.00068712165, -0.012595548, -0.02146644, -0.017071344, 0.001745933, 0.05641614, 0.010081405, -0.10786607, -0.014290268, -0.020212471, -0.022919057, -0.020510444, -0.011211217, -0.028034251, -0.005754595, 0.029772425, 0.028580535, -0.04626024, -0.028034251, 0.002841603, -0.0012462082, -0.017158253, -0.02421027, 0.03200722, -0.0113788275, 0.018002508, 0.00890193, -0.009951042, -0.01789077, 0.019852422, -0.038512953, -0.014327514, 0.005915997, -0.06361714, -0.004385784, -0.00069682125, -0.024284763, -0.0020687366, 0.016078103, 0.007368613, -0.018424638, -0.051847227, 0.028034251, 0.0035446319, 0.0010972219, 0.0027795252, -0.02773628, -0.021131221, -0.032578334, 0.025327668, -0.012074097, 0.022136878, -0.013222532, -0.013992295, -0.013210117, 0.034242015, -0.015842209, -0.026767869, 0.016065687, 0.009547537, -0.001745933, 0.00057189004, -0.004143681, 0.014389591, 0.049537938, 0.0011057575, 0.01944271, -8.905471e-06, 0.016798204, 0.0011065336, -0.018809518, 0.029524116, -0.007914896, -0.018735025, 0.021975476, 0.017741783, 0.016810618, -0.011142933, 0.02488071, -0.007840403, 0.026494727, -0.010056574, 0.04176582, -0.028282562, 0.008442556, 0.003594294, -0.01614018, 0.0044944193, -0.03178374, 0.013594998, -0.0027407268, -0.010180729, 0.009634446, -0.0052052084, -0.02074634, -0.011087063, -0.016736126, -0.01838739, -0.037047923, 0.01758038, 0.03893508, -0.010410417, -0.004584432, -0.018077003, 0.014575825, -0.10026777, -0.051847227, -0.0041033304, -0.008119753, 0.020001408, -0.041840315, 0.0052300394, -0.009417174, 0.008399102, -0.032901138, -0.013930217, -0.01656231, 0.021019481, 0.016599555, 0.02939996, -0.059594512, 0.03319911, -0.008337024, 0.036054682, -0.0022518656, -0.022633499, -0.0073313667, 0.0144516695, 0.01314804, -0.059693836, -0.008666036, -0.00771004, 0.021168469, 0.021652672, 0.010217976, -0.008802606, 0.02704101, -0.014575825, 0.0019119906, 0.004019526, 0.0048824046, -0.0056863097, -0.026221586, 0.014563409, -0.0030837057, -0.0035539437, -0.024173025, 0.023452925, -0.024135778, 0.019864837, 0.022049969, -0.016301582, -0.019455126, 0.003541528, 0.015718052, -0.0105904415, 0.019157153, -0.00029157082, 0.010658727, 0.017828692, 0.002109087, 0.0043081865, -0.009920003, 0.030467695, 0.0037153454, 0.01339635, 0.031907894, 0.016450569, 0.0040505645, 0.006896823, 0.024247518, 0.014414423, 0.016996851, -0.047625948, -0.016897528, -0.010702182, 0.00918128, -0.017567966, -0.0275128, -0.005217624, 0.028903339, 0.026171923, 0.013446012, 0.011142933, -0.020324212, 0.022174126, -0.013272194, -0.055869855, 0.030666344, 0.0046558212, 0.03173408, -0.012787989, -0.015395249, 0.009491668, 0.03242935, 0.0122479135, 0.0065181497, 0.0032808022, -0.024843462, -0.0184743, -0.003333568, -0.00021571972, 0.026494727, 0.044199266, -0.008808814, 0.03771836, 0.013843309, -0.044447575, -0.016102934, -0.028506042, 0.026544388, -0.020733925, -0.016512645, -0.05433033, -0.0017273097, -0.0067354217, 0.0067912913, 0.016065687, -0.00044501893, -0.0126576265, 0.03846329, 0.030368371, 0.032454178, 0.019852422, -0.022844564, 0.017605212, 0.010993946, -0.044050276, -0.0033118408, -0.005760803, 0.01888401, 0.01014969, -0.004100227, -0.0042305896, 0.01614018, 0.004823431, -0.005866335, 0.007815572, -0.0156187285, -0.009876549, 0.010938076, 0.00077480625, 0.024756555, -0.007834195, 0.004562705, -0.03913373, 0.031634755, 0.038984742, -0.006977524, -0.016326413, -0.0082377, -0.004820327, -0.035533227, -0.013768815, 0.0021059832, 0.042113457, 0.029375128, -0.020684263, 0.033770222, 0.01777903, 0.020137979, -0.021007067, -0.07186105, -0.03461448, 0.016152596, 0.024024038, 0.020696677, 0.016177427, -0.02562564, 0.011831994, -0.01350809, -0.03103881, 0.01627675, -0.034068197, 0.015705638, -0.011825786, 0.02560081, 0.01730724, -0.005673894, 0.014501331, 0.01921923, 0.011186386, 0.058005325, 0.017183084, 0.03749488, -0.031833403, 0.050407026, -0.0001377347, 0.0026274351, 0.015407665, -0.00879019, 0.0027205516, -0.0021975476, -0.02773628, -0.0003332792, 0.0029067844, 0.009559953, -0.010621481, 0.031162964, -0.038115658, -8.6714674e-05, 0.039059237, 0.0019337178, -0.033596408, -0.013706738, -0.018946089, 0.041592006, 0.022943888, -0.01730724, 0.0006747061, 0.031088471, -0.00848601, 0.01566839, 0.014513747, 0.010478701, -0.011583683, 0.0045782244, 0.0006448313, 0.013818477, -0.005388337, 0.0009994496, 0.03607951, -0.0041064345, 0.0069278623, -0.015879454, -0.039754506, -0.055174585, 0.007530015, -0.00015558203, -0.028977832, -0.0041405773, 0.0065802275, 0.019157153, -0.008684658, 0.010683558, 0.0011468839, -0.030244214, 0.026171923, 0.0014277853, 0.010391793, 0.031088471, 0.015358003, 0.004395095, 0.010602857, 0.007921104, 0.025948444, 0.007666586, -0.004066084, -0.006040152, -0.0111056855, 0.015792547, 0.036228497, 0.030964315, 0.0064746956, -0.02609743, -0.008802606, 0.039829, 0.025650471, -0.0054814536, 0.014104035, -0.006859577, 0.026221586, 0.033993702, -0.015233847, -6.741241e-05, 0.0037215531, 0.037668698, 0.00942959, -0.024731724, -0.0009559953, -0.0007507512, -0.02249693, -0.0063226055, 0.03463931, 0.021156052, -0.0240737, 0.028630197, -0.008554296, 0.014253021, -0.013992295, -0.025501486, 0.0060587754, -0.008274946, 0.0104538705, -0.0026243313, -0.0024179232, -0.021888567, 0.023030797, -0.028183239, 0.008641205, -0.018834349, -0.03558289, 0.007945935, 0.0015604761, -0.022571422, -0.03456482, 0.014526162, -0.0052734935, -0.031634755, 0.0024846566, 0.010329716, -0.015258678, 0.011316749, -0.004714795, -0.028530873, 0.0020004513, 0.015966363, -0.019852422, -0.05616783, 0.00782178, -0.044447575, 0.027239658, -0.049413785, -0.01946754, 0.018263236, -0.032578334, -0.0029595504, -0.0008652068, 0.05095331, 0.027562462, 0.0029579985, -0.0042895633, -0.0060153212, 0.018399807, -0.011720254, -0.004559601, 0.019926915, -0.0030169722, 0.014613071, -0.023068042, -0.016785787, 0.019393047, -0.0014696876, 0.008510841, 0.014079204, 0.0007782981, 0.0063195014, -0.00094978756, -0.004671341, -0.015581482, -0.028307393, -0.047526624, -0.028133577, 0.0069278623, -0.01849913, 0.01270108, 0.012266537, -0.02110639, -0.01564356, 0.018623285, -0.0034949698, 0.026767869, -0.021863736, 0.033795055, 0.007567262, 0.059147555, -0.033521913, -0.007523807, 0.012309992, -0.004115746, -0.018958505, -0.030244214, 0.01586704, -0.020796003, 0.0049879365, -0.004094019, 0.0143150985, -0.015469742, -0.001978724, 0.024657229, -0.007827988, -0.0020128668, -0.021391947, 0.032081712, 0.00057344197, 0.027959758, 0.013495674, -0.009876549, -0.00065996265, 0.0063939947, 0.003858124, -0.000938148, -0.017990094, 0.038016334, 0.025650471, 0.003858124, -0.041194707, 0.021367116, 0.010460079, -0.011552645, 0.025501486, 0.013930217, 0.0088770995, -0.04360332, -0.0164754, 0.020920157, -0.0052269353, 0.0018700883, -0.014911043, -0.0070458096, 0.03503661, -0.006617474, -0.021516101, -0.030492526, -0.028754352, -0.005813569, -0.0034639311, 0.010410417, -0.015209016, 0.035458736, -0.06192863, 0.004246109, 0.0069216546, 0.0022254826, 0.006282255, -0.026196755, 0.03968001, -0.011807163, 0.026668545, 0.024446165, -0.04504352, -0.012912145, 0.01755555, -0.021031898, 0.023837805, 0.014873797, 0.01744381, 0.014799304, 0.013669492, 0.05338675, 0.02041112, 0.018635701, -0.010832544, 0.01955445, 0.016264336, -0.03560772, -0.026171923, 0.0025389746, 0.015792547, -0.011596099, 2.4346065e-05, 0.026445065, -0.0144765, -0.051300943, 0.007120303, 0.016078103, 0.012272745, 0.026271248, 0.07429449, 0.012539679, -0.010565611, -0.04412477, -0.03463931, 0.04933929, 0.0025979483, 0.0068471613, -0.025066942, -0.042560413, 0.0038798512, -0.013371519, 0.017642459, -0.029250974, 0.0028043564, -0.012210667, -0.0011344685, -0.018002508, -0.0028493626, -0.02773628, -0.0038426046, 0.004162304, -0.01220446, 0.01270108, 0.043007374, -0.02360191, -0.027587293, 0.043578487, -0.0033677109, -0.006859577, -0.0016497127, 0.0013199253, -0.03764387, 0.0043733683, -0.008274946, -0.0016248816, -0.006530565, 0.015444911, 0.019964162, 0.03389438, -0.035980187, -0.006636097, -0.0054566227, -0.013408765, -0.015109692, -0.016202258, 0.055224247, 0.0037308647, 0.0059253084, 0.014737226, -0.02704101, 0.024110947, 0.016090518, 0.010273846, -0.007505184, -0.0211933, 0.016785787, -0.0014114898, -0.013458427, 0.017853523, -0.029524116, -0.021031898, -0.062772885, -0.02088291, -0.0009071092, 0.0046247826, -0.0036160212, 0.0075920927, 0.01481172, -0.049686924, 0.016053272, -0.013557752, -0.023043212, -0.019020582, 0.012322407, 0.027562462, 0.005599401, -0.031361613, -0.0026522663, -0.008690867, 0.0145882405, 0.027115503, -0.004637198, 0.01849913, -0.004708587, -0.01586704, 0.016909944, 0.04362815, 0.01084496, 0.0014200255, -0.00812596, 0.008374271, 0.048048075, -0.04025113, -0.0068471613, -0.006096022, -0.0077286633, 0.011558852, -0.048395712, -0.02096982, -0.0037339686, 0.0031426793, 0.02452066, -0.009280603, 0.018052172, 0.0045968476, 0.029300636, -0.0056490633, 0.0073561976, -0.006148788, 0.014737226, -0.03106364, 0.02609743, -0.025141435, -0.0059191007, -0.005959451, 0.0067292135, -0.029822087, -0.0219134, -0.030467695, -0.017158253, 0.042312104, 0.0017986989, -0.0022627292, -0.00022056953, 0.0043609524, -0.028108746, 0.01885918, -0.00156358, -0.0020470095, -0.02157818, 0.0015721157, 0.029697932, 0.002891265, -0.0405491, 0.005081053, -0.012173421, 0.01597878, -0.016090518, -0.021031898, 0.028804015, -0.010466286, 0.007008563, -0.034068197, 0.009951042, -0.0023636052, 0.009497875, -0.031013979, 0.039580688, 0.019616527, 0.038314305, 0.0013393246, 0.031659584, 0.025575979, -0.00023220909, -0.013992295, 0.011701631, 0.009522706, 0.019641358, -0.009690316, -0.0075424304, -0.05810465, 0.018722609, 0.008995047, -0.021329869, -0.00027508143, -0.008163206, -0.02499245, 0.004019526, -0.005164858, -0.0028772976, 0.015345587, 0.018312898, 0.004503731, -0.036352653, -0.039357208, 0.036427148, -0.0118133705, -0.028133577, -0.025340084, 0.010664934, 0.019840006, 0.06252458, 0.002082704, 0.01650023, -0.0023092874, 0.023068042, -0.012458978, 0.013346688, -0.008243907, -0.018623285, 0.007424483, 0.0073561976, 0.01780386, -0.0006359076, -0.006179827, 0.006549189, -0.012936976, 0.029077156, 0.0023061836, -0.036129173, 0.03302529, 0.037817683, -0.013073546, -0.0066236816, 0.0027546943, 0.02728932, -0.021627842, -0.017642459, 0.0071078874, 0.0012493121, -0.0022611772, 0.021391947, -0.008008013, 0.014339929, -0.0014999504, -0.017766614, 0.010758051, -0.015569067, -0.012483809, -0.014079204, -0.0026196754, -0.030765668, 0.02052286, -0.0040443568, -0.006958901, 0.016909944, 0.005164858, 0.011428489, 0.016214674, 0.027860435, 0.05745904, 0.012552095, -0.009951042, 0.020796003, 0.015718052, -0.017481057, 0.009274396, 0.03126229, 0.03247901, -0.020249719, -0.0036377483, 0.0057918415, -0.00015199317, 0.0059035816, 0.045962267, -0.014265437, -0.005810465, 0.013793646, -0.025700133, -0.005884958, 0.0014192496, 0.0071078874, 0.0012050817, -0.015097276, 0.0026507142, -0.0064995266, 0.015581482, 0.018176327, 0.00017517527, -0.020088317, 0.014190943, 0.002661578, -0.013818477, -0.0032218283, -0.029946243, 0.0011430042, -0.035135932, -0.021900984, 0.0133094415, 0.00016217778, -0.007523807, 0.048271555, 0.016760956, -0.04434825, -0.017965263, -0.01292456, -0.001067735, 0.00776591, 0.020622185, -0.0011236048, -0.03747005, -0.01411645, -0.0035539437, -0.009398552, 0.007002355, -0.030939484, 0.033745393, -0.014550993, 0.029499283, 0.0024877605, 0.030269047, -0.0336709, -0.013259779, 0.02560081, -0.014898628], "content_hash": "f14a8f80b705fd4fecd313a1566dcc86"}, {"content": "is split into tokens 2. **Embedding**: How tokens become number vectors 3. **Attention**: How the model finds relationships between tokens 4. **MLP**: How stored knowledge is retrieved 5. **Output**: What the model predicts and how confident it is Click any stage to expand it and see detailed explanations and visualizations.\n\n### 5. Investigation Panel\nAt the bottom, two experiment tabs let you investigate *why* the model made its prediction:\n- **Ablation**: Remove specific attention heads and see what changes\n- **Token Attribution**: Measure which input tokens influenced the prediction most\n\n### 6. AI Assistant (Chatbot)\nThe floating robot icon in the bottom-right corner opens the AI chatbot. It can answer questions about transformers, explain what you're seeing in the dashboard, and guide you through experiments.\n\n## Typical Workflow\n\n1. Select a model (start with GPT-2 if you're new)\n2. Enter a prompt (e.g., \"The cat sat on the\")\n3. Click \"Analyze\" to run the model\n4. Explore the 5 pipeline stages to understand how the model processed your input\n5. Use the Investigation Panel to run ablation or attribution experiments\n6. Ask the chatbot if anything is unclear", "source_file": "dashboard_overview.md", "chunk_index": 1, "embedding": [-0.0012277357, 0.014412472, 0.009448724, -0.01637685, 0.0013958341, -0.044187922, -0.019488873, 0.05841029, -0.003749214, 0.069844514, 0.023825992, -0.0530593, -0.006393026, -0.046046685, -0.027247809, -0.014743389, -0.03548552, 0.021770084, -0.012349524, 0.054185826, 0.025304554, -0.01123004, 0.018700305, 0.053735215, 0.0047454843, -0.0017469929, -0.014560328, 0.012265035, -0.020840703, 0.013461967, 0.018404594, -0.022924772, -0.052214406, -0.03711898, -0.0016581031, -0.01068086, -0.021023761, 0.023051506, -0.004118855, 0.038639788, 0.016785214, 0.0090051545, -0.050017685, 0.052186243, -0.013659108, 0.0063120574, 0.009188215, -0.024769455, -0.035091236, 0.054016847, -0.036443066, 0.040695693, -0.019305812, 0.0004147458, -0.0074843466, 0.029317798, -0.008484137, -0.03250023, -0.02726189, -0.0099204555, 0.022051716, 0.005938896, 0.014011147, 0.027008424, 0.0040414063, -0.012983194, -0.034246344, 0.023572523, -0.037090816, -0.027994132, 0.026402917, -0.0027828673, -0.02219253, 0.022938853, 0.003367252, -0.003911152, -0.03359859, 0.050017685, -0.021333557, -0.037147142, 0.014616654, -0.045765053, -0.010195046, -0.041146304, 0.0007406018, -0.015813587, -0.035091236, -0.023924563, -0.06364863, -0.078068145, -0.018714387, 0.013659108, -0.059086207, 0.021206822, 0.06657759, 0.017165415, 7.937376e-05, -0.004914463, 0.0077800592, 0.033964712, 0.036696535, -0.023657013, -0.03821734, 0.012518503, 0.029937387, -0.021080088, 0.030669628, -0.00081012945, -0.04280793, -0.025304554, -0.097106405, -0.0057347133, -0.01657399, -0.0031296257, 0.0039146724, -0.00787159, 0.0042420686, 0.0043969657, 0.003478144, -0.05159482, -0.046553623, -0.006681698, 0.032218598, 0.008343321, 0.0009751477, -0.0007023176, 0.013180336, -0.02698026, 0.0069950125, 0.013546457, -0.04860953, -0.003303885, 0.040808346, -0.045849543, 0.0087235235, -0.023938645, 0.008737605, 0.024037214, -0.046891578, 0.022319265, 0.0067556263, -0.042216502, -0.014159004, 0.0004173861, -0.06663392, -0.024135785, -0.021389883, 0.058184985, -0.0715343, 0.011131469, 0.0152080795, 0.009293826, -0.04128712, -0.024107622, -0.022981098, -0.015067264, -0.019390302, 0.06421189, 0.0066887387, -0.062691085, 0.016010728, 0.022051716, -0.03207778, -0.017278068, -0.09045991, 0.0477928, -0.029402288, 0.011941159, -0.037963875, -0.053284604, -0.023234567, -0.029261472, -0.012349524, -0.034527972, 0.015883995, -0.031711664, -0.068380035, -0.040808346, -0.024290683, -0.04238548, -0.021784166, -0.01637685, -0.03495042, -0.024910271, -0.018672142, -0.017770924, -0.015982565, 0.022009471, -0.027233727, -0.0004418088, -0.00084753364, 0.034612462, 0.033908386, 0.010709023, 0.0228262, 0.010420351, 0.044018943, 0.0110469805, 0.040047944, 0.012441054, 0.05931151, 0.053481746, 0.039541006, -0.033204306, 0.0432867, 0.03069779, -0.0041610994, 0.01574318, -0.04455404, -0.012081974, 0.02752944, -0.066971876, 0.03537287, 0.008132098, -0.07198491, -0.0022160849, -0.08432035, 0.04424425, 0.014278697, 0.03531654, 0.013954821, 0.017855411, 0.014574409, -0.009293826, -0.02798005, 0.024558233, 0.03658388, -0.044976488, 0.0175597, -0.002099912, -0.002399145, 0.02579741, -0.032979004, -0.009561376, -0.011455345, -0.04354017, 6.4797154e-05, -0.017770924, 0.025276393, -0.03207778, 0.026825363, 0.005861447, 0.040498555, 0.0102020865, 0.020052135, 0.013250744, 0.00035071874, -0.036414903, 0.061958842, 0.029458614, 0.061170276, -0.019207241, -0.033204306, -0.007343531, 0.02453007, -0.03402104, -0.020671723, 0.02789556, -0.022558652, 0.0042667114, 0.007913834, 0.030500649, -0.05080625, -0.022248857, -0.021389883, -0.007188634, -0.030894933, -0.0069879717, -0.046046685, -0.071590625, 0.017052764, -0.0025293995, 0.00033751727, 0.010617493, -0.062071495, -0.015771342, 0.008660156, 0.01575726, -0.018306023, 0.01874255, -0.017278068, -0.033626754, 0.049172793, 0.028008213, -0.011920037, 0.007097104, 0.0037597753, -0.011307489, 0.011307489, -0.0059705796, 0.03410553, 0.000642911, 0.03421818, -0.02181233, 0.01829194, 0.0078082224, 0.038329992, 0.036274087, 0.007385776, 0.013067683, 0.02671271, -0.05514337, 0.022755794, -0.039597332, 0.011103306, 0.026754955, -0.02409354, -0.013546457, 0.023234567, -0.02136172, 0.00037426135, -0.021798247, 0.009209337, -0.013229622, 0.01249034, 0.016320523, 0.04455404, 0.00069087633, 0.036020618, -0.028726373, -0.03528838, -0.051200535, -0.021333557, -0.013785843, -0.007836386, -0.020502744, -0.01204677, 0.031176563, -0.017306231, -0.0190101, 0.039907128, -0.04762382, -0.029148819, 0.014447676, -0.013652068, -0.012469217, -0.014799715, 0.016236033, -0.025318636, 0.048637692, -0.072210215, -0.015222161, -0.06578903, -0.0093219895, -0.06139558, 0.06319802, 0.015729098, 0.010223209, 0.0013271866, -0.0031542685, -0.017066846, -0.007181593, 0.0031349063, -0.029036166, 0.006854197, 0.020756213, -0.016785214, -0.026923934, 0.018629897, -0.009188215, -0.03486593, 0.03204962, 0.06961921, 0.030754117, -0.024192112, -0.0019520556, -0.039794475, 0.011687691, -0.010610452, -0.02334722, 0.015320732, 0.009420561, 0.038780604, -0.0045765056, -0.043934453, -0.028346172, 0.009652906, 0.0035573528, -0.041512426, 0.03883693, 0.039428353, -0.013715435, -0.025684757, -0.03207778, 0.02017887, 0.029176982, 0.0051644105, -0.007128787, -0.042047523, 0.053284604, 0.011835547, -0.059255186, -0.026740873, -0.0006886761, 0.024670884, 0.04100549, 0.0006715142, 0.015306651, 0.051200535, 0.051566657, 0.012814215, -0.03540103, -0.049736053, -0.018263778, 0.011089224, 0.026684547, 0.037766732, -0.013243703, 0.030556975, 0.009786681, 0.051397678, -0.05429848, 0.022248857, -0.0036858472, -0.03883693, 0.01784133, 0.036499392, -0.0019045303, 0.044976488, 0.045229957, -0.014180127, -0.029458614, -0.022798037, -0.028360253, -0.062353127, 0.024009053, 0.051116046, -0.010019027, -0.0010332342, -0.02816311, 0.030106366, 0.04255446, 0.055171534, -0.007171032, 0.021685595, -0.03920305, 0.045793217, 0.033880223, 0.0045377812, 0.020629479, 0.027839234, -0.034387156, -0.023248648, -0.02798005, -0.054073174, 0.0005306986, -0.03188064, -0.072998784, -0.008096894, -0.03849897, 0.007252001, -0.01584175, -0.0028831984, 0.05421399, -0.030303506, -0.08015221, 0.044694856, 0.013828088, 0.080771804, 0.010927287, 0.01910867, 0.027543522, -0.007378735, 0.02779699, -0.006209966, -0.002796949, -0.02869821, -0.015517874, 0.000113422524, 0.010885042, -0.008195465, -0.036161434, -0.013264825, -0.025895981, 0.007033737, -0.030387996, 0.02752944, -0.01376472, 0.007822304, -0.041315284, 0.014715225, -0.0053826747, -0.015588282, 0.02906433, 0.00057558354, 0.07987058, -0.035457358, -0.022108043, 0.02959943, -0.008477096, -0.01050484, -0.016996438, 0.03503491, -0.017587863, 0.045765053, -0.049820542, -0.037090816, -0.022389673, -0.009723314, 0.014687062, 0.0076181213, -0.012117178, -0.0039674784, 0.086798705, -1.7051883e-06, -0.023558442, 0.03666837, 0.034753278, 0.0024501907, 0.022727631, -0.035907965, -0.0064211893, 0.0100260675, 0.020967435, -0.022333346, 0.021756003, 0.02943045, 0.025318636, 0.025107414, 0.004569465, 0.0060515483, 0.014144923, 0.018320104, -0.018629897, 0.016785214, -0.022093961, -0.038076524, -0.04058304, -0.06128293, -0.006512719, 0.009786681, -0.017517455, 0.011032899, -0.010089435, -0.050017685, 0.0013773522, 0.0005298185, 0.00025544822, -0.055565815, -0.0097796405, 0.014320942, 0.0037351325, 0.007533632, 0.004597628, -0.0097444365, 0.0085827075, -0.024684966, -0.012715644, -0.00072652026, 0.027557604, 0.045145467, -0.029374124, 0.018714387, -0.038273666, 0.0090403585, -0.0020471062, 0.017672352, -0.026022714, 0.019981727, -0.019136833, -0.002473073, 0.0036471228, 0.012800134, -0.039428353, -0.007216797, 0.02761393, 0.018207451, -0.02119274, -0.0015604123, -0.029712081, -0.01313105, -0.0071675116, -0.005541092, 0.018348267, 0.0077659776, -0.026585976, -0.0077448552, 0.016771132, -0.023121914, 0.0009082603, -0.014011147, -0.028050458, -0.040723857, 0.0075195506, -0.0034094965, 0.00029769287, -0.012349524, 0.009378316, -0.015827669, 0.014313901, 0.040216923, -0.02172784, -0.0030556975, 0.0020981517, -0.025698839, -0.003970999, 0.026642302, 0.00055446126, -0.024853945, 0.008209547, 0.020418255, -0.006871799, -0.020474581, 0.013025438, -0.03940019, 0.004235028, -0.025895981, -0.0019397342, 0.014067474, -0.017982146, 0.011631365, -0.05303114, -0.0075758765, -0.02072805, -0.024558233, 0.0114131, -0.020404173, 0.017137254, 0.032979004, -0.0027863877, -0.002481874, -0.006963329, 0.01032178, 0.0046328316, 0.0033074054, 0.017728679, 0.0019784586, -0.010173923, -0.009512091, 0.037794895, -0.011427182, 0.038527135, 0.032387577, 0.0037140101, -0.008110976, 0.019671932, 0.0036788064, 0.008322199, -0.007815263, -0.012610033, 0.007667407, -0.018348267, -0.007660366, -0.02381191, 0.004555383, 0.0059776204, -0.006674657, 0.017700516, -0.034556136, -0.023924563, -0.042667113, -0.038893256, 0.040921, -0.0063895057, 0.0076040397, -0.012532584, -0.0016325803, 0.00023982649, -0.0073364903, 0.02334722, -0.02752944, 0.0037104897, 0.016010728, -0.02081254, -0.019333975, 0.040808346, 0.013180336, 0.013328192, 0.017587863, 0.0036083986, -0.06494413, 0.015982565, -0.0154897105, 0.039541006, 0.0013711915, -0.02498068, -0.0038442647, -0.05007401, 0.017531537, 0.061170276, -0.022220694, -0.0077800592, -0.057734378, 0.005586857, 0.03675286, -0.0098148445, 0.01502502, -0.01882704, 0.02761393, -0.008885462, -0.007142869, -0.032556556, -0.013074724, -0.009519131, -0.024417417, 0.009758518, 0.0015586521, 0.048778508, 0.0003106743, -0.07913834, -0.014743389, -0.017095009, -0.027332298, -0.034190018, -0.018587653, -0.016982356, -0.017137254, 0.044328734, 0.025755165, -0.03241574, -0.042441808, -0.0008563346, 0.012032689, 0.01865806, -0.03159901, 0.030613301, -0.0008193705, -0.0027529441, -0.015475629, -0.020333765, -0.009244541, 0.010948409, -0.016278278, 0.017616026, 0.015644608, -0.06533842, -0.040329576, 0.0077448552, -0.027628012, 0.018066635, 0.010687901, 0.018798877, -0.0012629395, -0.067591466, 0.027402706, -0.0018394032, -0.0089699505, -0.006924605, -0.037344284, -0.027599849, -0.0024132265, 0.023431707, 0.0042596706, 0.02037601, 0.022952935, 0.003539751, -0.02182641, 0.039541006, 0.0055727754, -0.04390629, -0.0022706508, 0.013102887, -0.026191693, -0.015222161, -0.011962282, -0.0008092494, 0.014475839, -0.041146304, 0.00036788062, 0.035654496, -0.012384728, 0.015053183, -0.0023287372, 0.0104133105, 0.0031771509, -0.027937805, 0.03731612, 0.012243913, 0.004928544, 0.017616026, -0.0074209794, -0.0092515815, 0.00017645949, -0.027656175, 0.06139558, -0.018587653, 0.03359859, -0.010279535, 0.00579456, -0.022220694, -0.019573363, 0.009026277, -0.007477306, -0.036076944, -0.0055270107, 0.0010261934, -0.015334814, -0.007100624, -0.021023761, -0.026135366, -0.037794895, 0.00019989208, 0.011905955, -0.040216923, -0.008815054, -0.001973178, 0.003946356, -0.10302066, -0.015095428, 0.003251079, -0.008406688, 0.007315368, -0.03233125, 0.049736053, 0.011152592, -0.023966808, -0.030162692, 0.015109509, -0.032359414, 0.0032352374, -0.0023111352, 0.010349943, -0.023051506, 0.01764419, -0.016771132, 0.0042667114, -0.017785005, -0.018418673, -0.0056185406, 0.009469846, 0.01747521, -0.04438506, 0.012187586, -0.017489292, 0.034556136, 0.018841121, 0.0015190478, -0.038414482, 0.013335233, -0.012265035, 0.021319475, 0.007927916, 0.031373706, -0.013426763, 0.01357462, -0.0037844179, -0.002934244, 0.006681698, -0.02017887, 0.043089557, -0.016447257, 0.05097523, -0.017630108, -0.00452722, -0.031655338, 0.0050940025, 0.024487825, -0.028839026, 0.013687272, -0.01023025, -0.001484724, 0.01910867, 0.021164577, -0.014658899, -0.012201668, 0.0052559404, 0.006273333, 0.00452722, 0.025276393, 0.020756213, 0.010173923, 0.0044884956, 0.026205774, 0.013145132, 0.010054231, -0.039315704, -0.024248438, -0.0058825696, -0.008357403, 0.0058825696, -0.019263567, -0.0075758765, 0.005305226, 0.0002462072, 0.032472067, 0.0025100373, -0.010638615, 0.027726583, -0.0023657014, -0.045962196, 0.015405222, 0.007632203, 0.010314739, -0.009758518, -0.03151452, 0.015433385, 0.022769876, 0.029374124, -0.00026666946, 0.00063322997, -0.021488454, -0.019446628, -0.0032035538, 0.0001113873, 0.04500465, 0.052552365, -0.016503584, 0.04472302, 0.020361928, -0.0327537, 0.0016220191, -0.021460291, 0.02988106, -0.016883785, -0.011800343, -0.017404802, -0.0077730184, 0.020544989, 0.0022512886, -0.0040766103, 0.0113638155, -0.019122751, 0.02996555, 0.03393655, 0.008624952, 0.011934118, -0.033880223, 0.004928544, 0.014715225, -0.034077365, 0.008075772, -0.023065588, 0.0038055403, -0.0070583797, 0.021418046, 0.023488034, -0.0014812036, -0.012257994, 0.03033167, -0.0051292065, -0.00072652026, -0.027318217, 0.009498009, 0.016700724, 0.018249696, -0.028275764, 0.0015604123, -0.004203344, 0.018306023, -0.011004736, 0.008040568, -0.05522786, 0.0018358828, -0.007857508, -0.01612338, -0.004030845, 0.016306441, 0.021206822, 0.013109928, -0.0038970704, 0.025853736, -0.008286995, 0.016630316, -0.00095050497, -0.0390904, -0.005657265, 0.029852897, 0.012511462, 0.037259795, 0.04599036, -0.00606563, 0.017672352, -0.004509618, -0.0044216085, -0.00033267675, -0.0024132265, 0.008491178, -0.02988106, 0.033373285, 0.05190461, -0.0008998994, 0.045934033, 0.02943045, 0.021178659, 0.031908803, 0.003946356, 0.034753278, -0.005259461, 0.058072332, -0.0087235235, -0.0030169731, 0.013145132, -0.021784166, -0.00479829, 0.026191693, -0.0020066216, 0.010223209, 0.008385566, -0.009758518, -0.008800972, 0.04255446, -0.030725954, -0.0018886887, 0.018404594, -0.004238548, -0.026867608, 0.004118855, -0.014996856, 0.021629268, 0.030585138, -0.023417626, 0.012786052, 0.011004736, -0.0041223755, 0.0070583797, -0.015010938, -0.011300448, -0.027656175, -0.0019115711, -0.002143917, 0.035513684, -0.012060852, 0.015517874, 0.011124428, 0.012722685, 0.008251792, -0.0060163443, -0.04889116, -0.039259378, -0.003721051, -0.008455974, -0.019348057, 0.019854993, -0.0035819956, 0.016348686, -0.009293826, 0.0040449267, 0.008244751, -0.012821256, -0.0006723943, -0.0012629395, -0.008364444, 0.005981141, 0.004998952, 0.00059538573, 0.039822638, -0.01657399, 0.013982984, 0.019446628, -0.006100834, 0.008026486, -0.011582079, -0.0013711915, 0.033091653, 0.008702401, -0.0012338962, -0.026149448, 0.0064880764, 0.017179497, 0.026093122, -0.01603889, 0.020868866, -0.018629897, 0.052439712, 0.024952516, -0.02136172, -0.011406059, 0.011652487, 0.031035747, 0.013905535, -0.030528812, 0.0046046684, -0.015123591, -0.023741502, -0.016067054, 0.0061606807, 0.0017487531, -0.025304554, 0.036893673, -0.01874255, 0.018869285, -0.004551863, 0.0020400654, -0.010716064, -0.020390091, -0.0007014375, -0.02262906, -0.020052135, -0.03148636, 0.0060726707, -0.0098148445, -0.005143288, -0.0103429025, -0.031430032, -0.030838607, 0.00096546666, 0.0034535015, -0.0032352374, 0.020939272, 0.000371181, -0.011934118, -0.0067415447, 0.017038682, 0.010040149, -0.012701563, 0.000851494, -0.031739827, 0.018967856, -0.011448304, -0.0053122668, -0.04018876, 0.018728469, -0.00534043, 0.0058332845, -0.036443066, 0.017080927, 0.003794979, -0.049820542, 0.008976991, 0.00950505, 0.025473533, 0.017700516, 0.0022231257, 0.0048475754, -0.037372448, 0.015433385, -0.008547504, -0.018784795, 0.012187586, 0.008674238, 0.025839655, -0.011934118, -0.02634659, 0.02781107, -0.006393026, 0.014701144, 0.008441892, 0.014151963, -0.009124848, 0.00090562005, 0.014313901, -0.03917489, -0.015081346, -0.049933195, -0.020953353, 0.00035093876, 0.0015102468, 0.020939272, 0.018235615, -0.021080088, -0.005967059, 0.0133986, -0.0090051545, 0.022896608, -0.025473533, 0.02761393, 0.009110766, 0.070464104, -0.029543104, -0.034443483, -0.003096182, 0.01765827, 0.0072379196, -0.007927916, 0.0031841916, -0.03838632, 0.0039252336, 0.0009373035, 0.016644398, -0.0092163775, 0.009089644, -0.0069210846, -0.011349734, 0.00074104185, 0.009765559, 0.046553623, 0.0017601944, 0.011145551, 0.046046685, -0.014785633, -0.004312476, 0.0044673732, 0.013440845, 0.0071217464, -0.020657642, 0.016052973, 0.017221741, -0.0072942455, -0.056157243, 0.0052911444, 0.029374124, -0.008639034, 0.002951846, -0.02624802, 0.023994971, -0.023417626, 0.011166673, 0.021249067, -0.005868488, 0.012610033, -0.028219437, -0.021347638, 0.015644608, -0.01620787, -0.039372027, -0.015179916, 0.001955576, 0.0061888434, 0.024318846, 0.003133146, -0.0018112401, 0.020432336, -0.03188064, -0.015855832, 0.03207778, 0.0148560405, 0.0063366997, -0.03849897, 0.043709148, 0.001964377, 0.03557001, 0.009209337, -0.040780183, -0.01765827, 0.014644817, 0.005910733, 0.02444558, -0.016278278, 0.0003577595, 0.004682117, -0.0083010765, 0.032810025, 0.028923515, 0.017376639, -0.014912367, 0.024248438, 0.0076392437, -0.022474162, -0.004463853, -0.000498135, -0.002606848, 0.011603202, -0.0075547546, 0.044131596, -0.0041681402, -0.025769247, 0.022037635, 0.0012787812, 0.021220904, 0.018756632, 0.044413224, 0.0064810356, 0.000552261, -0.023417626, -0.0053545116, 0.024065377, -0.01765827, -0.009934537, -0.0017056284, -0.044610366, -0.009181174, -0.012567788, -0.00425263, -0.03773857, 0.011905955, 0.020573152, 0.022417836, 0.014335023, 0.014489921, 0.00033509702, 0.0008382926, 0.00031309458, 0.012194627, 0.014701144, 0.054101337, -0.02634659, -0.036724698, 0.019179078, -0.01539114, 0.004066049, -0.01357462, 0.0033109258, 0.010842797, 0.0151376715, -0.015827669, -0.022474162, 0.0046785967, -0.008815054, 0.031852476, 0.018066635, -0.044441387, -0.0009795482, -0.0013254264, -0.014602573, -0.021023761, -0.03866795, 0.057734378, -0.008413729, 0.032359414, 0.031120237, -0.04111814, 0.043878127, 0.00076612463, 0.016334604, 0.008463015, -0.01285646, 0.010892083, -0.022333346, -0.016672561, 0.04289242, 0.01249034, 0.0112934075, -0.04970789, -0.010800553, -0.023938645, -0.013363396, -0.005748795, 0.030219018, -0.003567914, -0.039118562, -0.0052488996, 0.0006979171, 0.0089347465, -0.012025648, 0.039541006, 0.034358993, 0.0043265577, -0.009026277, -0.0036330412, -0.0090403585, 0.022474162, 0.016827459, -0.01376472, 0.029909223, 0.019854993, -0.0335141, -0.002110473, 0.006502158, 0.0030275343, -0.028036376, -0.013870332, -0.0096740285, 0.040949162, -0.033542264, -0.011898914, -0.021150496, -0.008392607, -0.02634659, -0.039372027, -0.029993713, 0.0032810024, 0.025149658, 0.030387996, -0.024079459, 0.029345961, 0.032922678, 0.003384854, -0.020023972, -0.005125686, -0.017123172, -0.019094588, -0.025135577, 0.06708453, -0.012891664, -0.018320104, 0.0061888434, 0.033570427, -0.023319056, -0.014644817, -0.025262311, -0.01629236, 0.030641465, -0.012469217, -0.013771761, -0.019066425, 0.0038020199, -0.025205985, 0.0022301662, -0.008463015, 0.015771342, -0.022220694, -0.00056106196, 0.011244122, -0.014151963, -0.04044223, 0.042329155, -0.00082773145, 0.019446628, -0.018756632, -0.0036189598, 0.034471646, -0.0100260675, -0.0011089225, -0.00031551486, -0.01430686, 0.01439135, 0.023445789, -0.027754746, 0.01375768, 0.023150077, 0.029683918, -0.014159004, 0.017235823, 0.010511881, 0.0038513055, -0.043878127, 0.012624115, 0.028543312, 0.03033167, -0.021798247, -0.018094799, -0.006871799, -0.01739072, 0.001366791, -0.0005034156, -0.0034728635, -0.016010728, -0.032838188, -0.001828842, 0.0029553664, -0.017799085, 0.011835547, 0.03086677, -0.00019208122, -0.013349314, -0.020756213, 0.016052973, -0.01882704, -0.019700095, 0.004365282, -0.014898285, 0.004544822, 0.03478144, 0.0036858472, -0.006935166, 0.0018570051, -0.028543312, -0.018235615, -0.001167889, -0.0051045637, -0.0058544064, 0.00298705, 0.0056643058, 0.018362349, 0.00068647583, -0.005298185, -0.0060339463, 0.008012405, 0.03213411, -0.007533632, -0.014813797, 0.014644817, 0.0016581031, -0.003467583, -0.022009471, -0.011391978, 0.049764216, 0.008068731, -0.010603411, 0.03703449, -0.0016096978, 0.037513264, 0.036161434, -0.008040568, 0.034612462, -0.0070407777, -0.005414358, 0.0010622774, -0.025952306, -0.006530321, -0.01774276, 0.008153221, -0.020080298, 0.007273123, -0.012715644, 0.00040066426, 0.010814634, 0.02781107, 0.0074561834, 0.0029448052, 0.0079842415, 0.046412807, -0.00029021205, 0.0048581366, 0.0068577174, 0.027106995, -0.026698628, 0.006484556, 0.04770831, 0.02472721, -0.0013747119, 0.004400486, -0.0054706843, -0.001503206, -0.003731612, 0.01729215, -0.013891454, -0.032387577, -0.006417669, -0.008343321, -0.022868445, -0.013384518, 0.038977746, -0.007068941, -0.028923515, -0.0075688357, -0.013116969, 0.01584175, 0.043089557, -0.0027617451, -0.018249696, -0.015729098, 0.014060433, -0.0008668958, 0.014067474, -0.004629311, -0.016855622, -0.049848706, -0.010420351, -0.004509618, -0.0040695695, 0.008125057, 0.052355222, -0.025487615, -0.039597332, -0.01639093, -0.016405012, -0.009723314, 0.0041153347, 0.045934033, 0.004354721, -0.023657013, -0.010195046, -0.032105945, -0.012117178, -0.003423578, -0.01276493, 0.042244665, -0.03385206, 0.020361928, 0.008019445, 0.02996555, -0.01692603, -0.02081254, 0.013539416, 0.016897867], "content_hash": "64dd8abf7a9430ef70ceb42f45496444"}, {"content": "# Embeddings Explained\n\n## What Are Embeddings?\n\nAfter tokenization breaks your text into token IDs, the model needs to convert those IDs into something it can actually compute with. This is where **embeddings** come in. An embedding is a list of numbers (a **vector**) that represents a token's meaning.\n\n## The Lookup Table\n\nThink of embeddings as a giant dictionary. Each token ID maps to a specific vector of numbers. For GPT-2, each token maps to a vector of **768 numbers**. For larger models, this vector might have 2048, 4096, or more numbers.\n\nThis dictionary (called an **embedding table**) was learned during training. The model figured out which numbers best represent each token by seeing how tokens are used across billions of text examples. Once training is done, the table is fixed -- the same token always maps to the same vector.\n\n## Why Vectors?\n\nWhy use lists of numbers instead of just the token ID? Because vectors let the model capture **meaning** and **relationships**:\n\n- Words with similar meanings (like \"happy\" and \"joyful\") end up with similar vectors\n- Related concepts are grouped nearby in the vector space\n- Directions in the space can capture relationships (e.g., \"king\" - \"man\" + \"woman\" \u2248 \"queen\")\n\nThis is what allows the model to generalize -- even if it has never seen a specific sentence before, it can work with the underlying meanings of the tokens.\n\n## Positional Information\n\nThere's one more detail: the model also needs to know the **order** of tokens (since \"dog bites man\" is different from \"man bites dog\"). This is handled by adding **positional encodings** to the embeddings -- extra numbers that tell the model where each token sits in the sequence.\n\n## What You See in the Dashboard\n\nIn **Stage 2 (Embedding)** of the pipeline, you can see:\n\n- The dimension of the embedding vectors (e.g., \"768-dimensional\" for GPT-2)\n- A visual showing the flow: Token ID \u2192 Lookup Table \u2192 Vector\n- An explanation of how the lookup table was created during training", "source_file": "embeddings_explained.md", "chunk_index": 0, "embedding": [-0.03326277, 0.00024296207, 0.021660032, 0.0074122003, 0.009177576, -0.013540491, 0.0023300583, 0.03411871, -0.019603398, 0.0581088, 0.03644877, 0.0015246427, -0.0016360931, -0.08502335, 0.0018902003, -0.043415166, -0.014967058, 0.027366295, -0.010259389, 0.06158011, 0.02056633, -0.025012461, -0.01836704, 0.062340945, -0.013290842, 0.017998511, 0.015466356, 0.049406745, 0.0172139, 0.015906213, 0.0022602158, -0.021505486, -0.029529922, -0.036876738, 0.015739782, -0.023776105, -0.013445387, 0.03166977, -0.029553698, 0.04916898, 0.02563064, 0.032573264, -0.06067662, 0.053353578, -0.029434817, 0.03307256, -0.019853046, -0.03533129, -0.031336904, 0.04436621, -0.00096961926, 0.029101951, -0.03751869, 0.004597704, 0.01129365, 0.009290513, -0.055683635, -0.00986114, -0.028364893, -0.040942453, 0.01903277, 0.006116403, 0.04774242, 0.012423014, 0.013433499, -0.018283824, -0.073087744, 0.06433814, -0.03673408, -0.0055636084, 0.059345156, 0.03326277, -0.033357874, 0.02819846, 0.04916898, -0.024465611, -0.03678163, 0.060153544, -0.05582629, -0.07361082, 0.0065443725, -0.049216535, -0.016678937, -0.0065681487, -0.016441176, -0.035830587, 0.0072398237, -0.024679596, -0.004734417, -0.03797044, -0.052782953, -0.024299178, -0.054780144, -0.007358704, 0.014919505, 0.010075124, 0.0044342433, -0.027294967, -0.023288693, 0.0080244355, 0.008702054, -0.007097167, -0.030647399, -0.022254433, 0.020340458, -0.05730041, -0.014206222, 0.017487325, -0.032264173, -0.055493426, -0.09985964, 0.016393624, -0.0066751414, 0.019627174, 0.033215217, 0.013813917, -0.03742359, 0.046601165, -0.0032365222, -0.0092191845, -0.017297115, -0.043177404, 0.042321466, 0.0127321035, -0.02148171, 0.012000988, -0.025178894, 0.023027157, -0.013885245, -0.020245353, -0.032454383, -0.010895399, 0.032501936, -0.026581684, -0.042036153, -0.026962101, -0.017558653, 0.041037556, -0.0016762153, -0.004603648, -0.017439773, -0.05340113, 0.007852059, 0.00017637039, -0.042868316, 0.0101761725, -0.016179638, 0.061485004, -0.022967717, -0.0077450657, 0.00592025, -0.0049424577, -0.0018604802, -0.031265575, -0.037185825, -0.031218024, -0.012244694, 0.03861239, 0.032858577, -0.02331247, -0.0046422845, 0.018533474, 0.049026325, -0.010853792, -0.069711536, 0.035426397, -0.010669527, -0.036829185, -0.008083875, -0.08064855, 0.028697757, -0.051736802, -0.026391476, 0.002514323, 0.036329888, -0.01996004, -0.048931222, -0.050928414, -0.0048532975, -0.059107397, -0.046648715, -0.0014926936, -0.028935518, 0.022718068, -0.03540262, 0.032430608, -0.009290513, 0.037138276, -0.012708328, 0.008511845, 0.006163955, 0.014075453, 0.045982987, 0.00547742, -0.016453065, 0.010978616, 0.02204045, -0.015299923, 0.06552694, -0.022563523, 0.03678163, 0.01281532, 0.03906414, 0.0013396349, -0.0035040034, -0.0065324847, -0.029434817, -0.009361841, -0.046220746, -0.0030329393, -0.008006603, -0.053496234, 0.007893667, -0.008874431, -0.023383798, 0.04260678, -0.05083331, 0.031218024, -0.01232791, 0.013492939, -0.0084107965, 0.027199863, 0.027080983, -0.0334292, -0.047219343, 0.0021487654, 0.00097184826, -0.063482195, 0.036211006, 0.0023419464, 0.016524393, -0.017653758, 0.012197142, -0.027699161, 0.013207626, -0.009801699, -0.009076527, -0.030599846, 0.00011637288, -0.0038012047, -0.02727119, 0.008505901, -0.0045085438, 0.027508952, 0.016191527, 0.007566745, -0.0117751155, -0.02983901, 0.06395772, 0.05016758, 0.02972013, 0.0055754962, -0.023978202, -0.009629322, -0.015656564, -0.019484518, -0.012898536, 0.028269788, 0.007893667, -0.021778911, 0.042868316, 0.012351686, -0.044746626, 0.0013440929, 0.0058132573, -0.0062828357, -0.023942538, -0.02886419, -0.03571171, -0.07147097, 0.021018077, -0.016334184, -0.008184924, 0.021065628, -0.06115214, -0.023728551, -0.04389069, -0.0051386105, -0.008333525, 0.015918102, -0.00012863244, -0.031146696, 0.05639692, 0.02582085, -0.017356556, -0.008143316, -0.004764137, 0.031764876, -0.013350283, -0.029577473, 0.03259704, -2.7003525e-05, 0.03723338, 0.012280358, -0.014289439, 0.044223554, 0.033643186, 0.019698502, 0.011448194, 0.00906464, 0.010705191, -0.07185139, 0.0021903736, 0.010645751, -0.020899195, 0.019139763, -0.0046779485, 0.016310407, -0.0045293476, -0.03568793, -0.014895729, -0.014325103, 0.007994715, -0.003459423, -0.0035961356, 0.03571171, 0.042273913, 0.026581684, -0.0063006678, -0.0495494, 0.012458678, -0.020174025, 0.011240154, -0.031027816, 0.015490131, -0.01427755, -0.04926409, 0.0147530725, 0.037185825, -0.015347475, 0.052830502, -0.0017445716, -0.023681, 0.02136283, -0.04700536, -0.029125728, -0.0034713112, 0.03925435, -0.0032721863, 0.016607609, -0.034190036, -0.016429288, -0.012957977, -0.03338165, -0.075655565, 0.031717323, -0.007911499, 0.02605861, 0.019674726, -0.0013106577, 0.004835465, -0.015644677, -0.0037269043, -0.04160818, 0.035307515, -0.0024712288, 0.01818872, -0.033928502, 0.008488069, -0.035854366, -0.02257541, -0.004648228, 0.017285228, 0.011156937, -0.02960125, 0.0042321463, -0.01470552, 0.028531324, -0.016453065, 0.01544258, 0.006027242, 0.0087971585, 0.009599602, 0.00224387, -0.018450256, 0.006116403, 0.036401216, 0.02765161, -0.033595636, 0.05577874, 0.024394283, -0.008987367, -0.031360682, -0.033595636, 0.0094034495, 0.049644504, 0.02245653, -0.01324329, -0.039325677, 0.0064849323, -0.03399983, -0.015311811, -0.0007742094, 0.0044699074, 0.026034834, 0.048931222, 0.014634192, 0.018331377, 0.026272595, -0.0022379258, -0.022147441, -0.035735484, -0.018783122, -0.03759002, 0.02093486, -0.025892178, 0.05121373, 0.00026023688, 0.034451574, 0.05316337, 0.010437709, 0.05344868, 0.017534876, -0.049692057, -0.0035485835, 0.019567734, 0.020292904, 0.019377524, 0.014301327, 0.04814661, -0.0023805825, -0.049596954, -0.07080524, 0.004832493, -0.015014609, -0.004324279, 0.050120026, -0.018176831, -0.014229998, -0.017927183, -0.011216377, 0.035473946, 0.05858432, -0.0047849407, -0.012084205, -0.01629852, 0.06543184, 0.03216907, 0.0049186815, 0.023467015, 0.007120943, 0.008440517, -0.015846774, -0.022777507, -0.062150735, -0.02142227, -0.03606835, -0.050215133, 0.017546766, -0.027722938, 0.029339712, 0.03778023, -0.004770081, -0.0026317176, -0.017285228, -0.03509353, 0.006360108, 0.00036109958, 0.06619267, -0.024560716, -0.015668452, 0.02691455, -0.032834798, -0.009801699, 0.002517295, -0.005944026, -0.038588617, -0.045816552, 0.022325762, -0.024941133, -0.0057865093, -0.0003754024, 0.013207626, -0.05045289, -0.011127217, -0.0053615114, 0.03338165, -0.023181701, -0.025487984, 0.018260049, 0.0018515642, -0.013267066, -0.045127045, -0.004327251, -0.0044342433, 0.060201097, -0.006651365, -0.018414592, 0.05563608, -0.02221877, -0.04253545, -0.042559225, -0.006342276, -0.01083596, 0.012648887, 0.008250308, -0.051689252, -0.056492023, 0.024132745, 0.015716005, 0.011715676, 0.0013322048, -0.0045025996, 0.06633533, 0.047694866, 0.028293563, 0.007762898, 0.025868401, -0.007548913, 0.042012375, -0.02734252, -0.0030611735, -0.002252786, 0.026700566, -0.004330223, 0.01879501, 0.019567734, 0.016571945, 0.009415337, -0.019436965, 0.006698917, -0.008624782, 0.024513163, -0.024489388, 0.0016672993, -0.010057292, -0.051308833, -0.036543872, -0.026819445, -0.008095764, 0.0007764384, -0.024073305, 0.011436306, 0.012024765, -0.059582915, 0.033191442, -0.013029305, -0.010437709, 0.005444728, 0.010978616, -0.01281532, 0.04852703, 0.021386605, 0.010948896, -0.00729332, 0.025654417, 0.0026658957, 0.005744901, -0.004817633, 0.009920579, -0.0017059355, -0.019211093, 0.034522902, -0.03471311, -0.03644877, 0.005310987, 0.011364978, 0.017629981, 0.019329973, -0.004505572, -0.0129342005, 0.010574423, 0.047266897, 0.00596483, -0.036115903, 0.06652554, 0.028531324, -0.020637657, 0.01824816, -0.045792777, -0.032668367, -0.016583832, 0.027675385, 0.044461314, -0.0042826706, -0.008173035, 0.00041756782, 0.0012824236, -0.034689337, 0.016464952, -0.022563523, -0.011281761, -0.0053228755, -0.010538758, -0.02788937, 0.014134894, 0.014206222, -0.008672334, -0.009694707, -0.005263435, 0.04619697, -0.012660775, -0.024061417, 0.028769085, 0.005741929, 0.0072636, 0.016191527, -0.027485177, -0.051118623, -0.016322296, 0.0060034664, -0.018509697, -0.034760665, 0.019817382, -0.020102696, -0.0055606365, -0.013659372, 0.02995789, -0.0064789886, 0.004814661, 0.0036882681, -0.05316337, -0.019211093, 0.015109714, -0.03411871, -0.0024935189, 0.0015677369, -0.0141824465, 0.00541798, 0.0060064383, 0.018973332, -0.007875835, 0.011721619, -0.04591166, -0.005319903, 0.047195565, -0.010711135, -0.02282506, -0.0095698815, 0.022123665, -0.005914306, -0.0027000739, 0.041156437, 0.005441756, 0.0037120443, 0.008202756, -0.015918102, 0.017463548, 0.029672578, -0.037756454, 0.00014432838, -0.027841818, -0.00035627006, -0.0048711295, -0.019555846, 0.0035040034, 0.0006817055, 0.03143201, -0.019318085, -0.024917357, -0.010289109, -0.052450087, 0.04167951, -0.023264918, 0.020340458, -0.034736887, -0.021885904, -0.01708313, -0.007715346, -0.010283165, -0.01665516, -0.02264674, -0.009641211, -0.027366295, -0.014907617, 0.039634764, 0.008826879, 0.013623708, 0.002707504, -0.0018738542, -0.07284998, 0.036353663, 0.014943281, 0.014432095, -0.0074775848, -0.036543872, -0.018557249, -0.006669197, -0.002985387, 0.02788937, -0.020281017, 0.008220588, -0.015727893, 0.006158011, 0.0064552124, -0.03478444, 0.029648801, -0.01525237, 0.01684537, -0.007863946, -0.017190123, -0.008838766, 0.008434573, -0.03642499, -0.020174025, 0.027128534, 0.0042380905, 0.022111777, 0.0023389743, -0.033476755, -0.0067167496, -0.008939815, -0.041275315, -0.053971756, -0.034998424, -0.019365637, 0.0054060915, 0.028364893, -0.00869611, -0.01330273, -0.0388026, 0.006651365, 0.008791215, 0.015965654, -0.014194334, 0.0028427304, -0.036353663, -0.009397505, 0.0035337235, 0.006140179, -0.037304707, 0.002995789, -0.009272681, -0.002749112, 0.011519522, -0.03509353, -0.053686444, -0.020768426, -0.014491536, 0.022860724, 0.016179638, 0.02496491, 0.012482455, -0.045816552, 0.017415997, 0.040562034, -0.018890115, 0.00958177, -0.029125728, 0.0015573348, -0.03352431, 0.027532728, 0.004781969, 0.0139803495, 0.002955667, 0.014764961, -0.0071803834, -0.003168166, 0.012672664, -0.039111692, 0.0035515556, 0.00789961, -0.0035069752, -0.037613798, 0.0040092454, 0.006811854, 0.02331247, -0.01189994, -0.014479647, 0.022313874, 0.012375462, 0.02245653, 0.009058695, 0.025345327, 0.027508952, -0.0451746, 0.020518778, 0.011650291, 0.024370506, 0.015680341, 0.011656235, -0.01324329, 0.0029304049, 0.0047671087, 0.052022114, -0.024334842, 0.04591166, 0.01806984, -0.01238735, 0.0066929734, -0.026534133, 0.025939729, 0.017689422, -0.04343894, -0.0014756045, 0.0046898364, -0.010289109, -0.015882438, -0.012292245, -0.05582629, -0.017249564, -0.028032027, 0.015822997, 0.0021784855, -0.019757943, -0.016441176, -0.0009681332, -0.08473804, -0.025464207, 0.044865508, 0.0097898105, 0.025773296, -0.00596483, 0.037304707, -0.006853462, -0.013385947, -0.033785842, -0.0034921153, -0.03808932, 0.026343923, -0.024608267, 0.01795096, -0.01187022, 0.013457275, 0.011632459, 0.008339468, -0.0014614874, -0.03568793, 0.013599931, 0.04051448, -0.010562534, -0.043010972, -0.00077123736, 0.011156937, 0.011067777, 0.03295368, 0.007982827, -0.04852703, 0.013516715, -0.01903277, 0.028769085, -0.0030210512, -0.002769916, 0.009023031, -0.022527859, 0.019246757, 0.001606373, 0.046220746, -0.017653758, -0.0067286375, -0.018283824, 0.058679424, -0.010711135, -0.0074775848, -0.00038301817, -0.0064611565, 0.020399896, 0.010669527, 0.029315937, -0.02453694, -0.015585236, 0.044817958, 0.0033762068, -0.00048666712, -0.021018077, 0.03478444, -0.002549987, -0.00789961, -0.008678278, 0.019389413, -0.00088417385, 0.020185912, 0.010211837, 0.026866999, 0.028935518, -0.029625027, -0.0005691405, -0.0023850405, -0.02031668, 0.0123992385, -0.023645336, -0.03825575, 0.010711135, 0.011864276, 0.013742588, -0.017166346, -0.01916354, 0.03416626, -0.006431436, -0.040966228, 0.042083703, 0.0037239322, 0.010752743, -0.025583088, -0.044461314, -0.008862543, 0.0040033017, 0.014562864, 0.0030849495, -0.028341116, 0.0046601165, 0.01101428, -0.0038130926, 0.006859406, 0.015609012, 0.047932625, -0.00918352, 0.06405283, 0.018105503, -0.022694292, -0.026866999, -0.06295913, 0.01336217, 0.016583832, -0.02551176, -0.007317096, -0.0072160475, -0.01885445, 0.0067048613, -0.024322955, -0.007079335, -0.06847518, 0.029197056, 0.01941319, 0.03350053, 0.023764215, -0.058631875, -0.0051148343, 0.0014978945, 0.00026822416, 0.028792862, -0.03069495, 0.020150248, 0.008143316, 0.039206795, -0.0030151072, 0.0028620486, -0.02117262, 0.05563608, 0.027057206, -0.017059354, -0.032644592, 0.01782019, -0.012613223, 0.009825475, -0.047433328, 0.0066929734, -0.020352345, -0.0036258558, -0.020292904, -0.004585816, -0.0039290013, 0.033928502, -0.037114497, -0.023217365, -0.009385617, -0.004160818, 0.040062737, 0.024085194, -0.036282334, 0.049121432, 0.018557249, -0.0010461486, 0.010889456, -0.045245927, -0.017154459, 0.029458594, 0.010324773, 0.024251627, 0.011311482, 0.0060866824, 0.043581598, -0.010663583, -0.015490131, 0.016821593, -0.028293563, -0.0026272596, -0.01616775, 0.028246012, 0.001821844, 0.03625856, 0.033286545, 0.035545275, -0.018283824, 0.048313044, 0.0018961444, -0.010544702, -0.006205563, 0.029767683, 0.0008878889, -0.0018693962, 0.019472629, -0.03623478, -0.0012608765, -0.025797073, -0.02532155, 0.025701968, 0.00961149, 0.00315925, 0.0024207046, 0.020637657, -0.028626429, -0.0067286375, -0.011911828, -0.005480392, 0.0051504984, 0.023883097, 0.003613968, 0.0018500781, 0.0062650032, -0.0068950704, -0.021992896, -4.411582e-05, 0.010639806, -0.0074775848, 0.015894325, 0.023086596, 0.016964251, 0.021208284, 0.031574667, 0.03704317, -0.025844626, 0.0046452563, 0.013897133, -0.011103441, 0.0025618752, 0.015169155, -0.051594146, -0.015276147, -0.006354164, -0.009498553, -0.0007218277, -0.0012385864, 0.018533474, 0.017784527, -0.039111692, 0.003851729, -0.026391476, -0.021136956, -0.009807643, -0.0047106408, 0.02465582, 0.0026985877, 0.00030481708, -0.03033831, 0.016560057, 0.00590539, 0.0167146, 0.051166177, 0.0032900183, -0.0041667623, -0.0030180793, -0.022016672, 0.03295368, 0.023407575, -0.010437709, -0.011055889, 0.03668653, 0.028103355, 0.009248905, -0.051166177, 0.031693548, 0.01220903, 0.012351686, 0.013992237, 0.026343923, -0.031741098, 0.030219428, 0.04046693, 0.053115815, 0.00053013285, -0.015406916, -0.016310407, 0.013397834, 0.007756954, 0.022515971, 0.018997107, -0.042036153, 0.03568793, -0.012232806, -0.0077272337, -3.3736993e-05, -0.0017460576, 0.0036199118, 0.010509038, 0.018830674, -0.0030433412, -0.025701968, -0.03276347, 0.026248818, 0.00063415326, -0.002765458, -0.011073721, -0.01879501, -0.012280358, 0.028887967, -0.006413604, -0.012173365, 0.014063566, -0.00541798, -0.004939486, -0.006859406, 0.00023831829, -0.0149551695, 0.009724427, 0.0014406833, -0.01629852, 0.018010398, -0.017784527, -0.0004740361, -0.00035181205, 0.006651365, 0.023383798, 0.0323355, -0.024085194, -0.005735985, 0.010818128, -0.0056260205, -0.02465582, 0.039896302, 0.023455126, 0.012042597, -0.010164284, 0.003171138, -0.04883612, 0.020495001, -0.019567734, 0.0024415087, 0.014669856, -0.014883841, 0.04916898, 0.00860695, -0.027247416, 0.053306025, -0.009213241, 0.02136283, 0.014800625, 0.038969036, 0.008707998, 0.017594317, -0.009896804, -0.070567474, -0.02703343, -0.025178894, -0.009225128, -0.02508379, 0.00451746, 0.002032857, 0.0024727148, 0.0048532975, -0.014241886, 0.015145378, -0.008494013, 0.010045404, 0.01192966, 0.030908935, 0.020233465, 0.039325677, -0.021255838, -0.024441835, 0.022920165, -0.017475437, -0.009932468, -0.0018589941, 0.0026302314, -0.061722767, -0.015228595, -0.0025975395, 0.009445057, 0.0014770905, -0.023704777, -0.01684537, 0.000914637, 0.02319359, 0.029577473, 0.049026325, 0.018355152, -0.0060866824, 0.003905225, -0.025440432, 0.002081895, -0.042250138, 0.0073824804, 0.0023731524, -0.0062828357, 0.0149551695, 0.021065628, -0.013350283, -0.042987198, 0.013160073, 0.034380246, -0.0028412445, 0.020720875, -0.008256252, 0.026962101, -0.010087012, -0.0030388832, 0.005504168, -0.026248818, 0.0022334678, -0.030956488, -0.025155118, 0.009510442, 0.026248818, -0.019876823, 0.0027922061, -0.02886419, -0.002028399, 0.024370506, -0.0025767353, -0.01634607, 0.024608267, -0.04826549, 0.0019570705, -0.011685955, -0.015145378, -0.001826302, -0.028983071, 0.060438856, -0.0495494, 0.022908276, 0.0053644837, -0.038588617, -0.02068521, 0.008969535, -0.0069901748, 0.008464293, -0.0005542804, 0.008036323, -0.03618723, -0.0040359935, 0.010235613, 0.0018857423, 0.000901263, -0.017915294, -0.036115903, 0.011275818, -0.024727149, -0.020245353, -0.000919838, 0.017106907, 0.0014622305, -0.010140508, 0.03521241, -0.0062352833, -0.012553783, 0.0440809, 0.016571945, 0.0009933953, 0.00081581756, 0.045602567, 0.008886319, -0.0055814404, -0.019627174, -0.02660546, 0.01861669, -0.00036332858, -0.008731774, -0.007287376, -0.0046809204, 0.023787992, 0.009664986, -0.014574752, -0.043224957, 0.007768842, 0.010051348, -0.008690166, -0.0035218354, 0.016999915, 0.011614627, 0.0021665974, -0.018236272, 0.04027672, 0.009011144, 0.049454298, 0.0023612643, -0.015264259, 0.016607609, -0.03678163, 0.0067821336, 0.0016643273, 0.052687846, -0.004639312, -0.011275818, -0.0011917772, -0.026034834, -0.008464293, 0.00823842, 0.023205478, 0.014622304, -0.023264918, -0.0016227191, 0.0019689586, 0.014075453, -0.004948402, -0.01703558, 0.052640293, 0.0024028725, 0.020055143, -0.00600941, 0.018890115, 0.025583088, -0.0060926266, 0.029387265, -0.0063066115, -0.03278725, -0.0007508048, -0.013207626, -0.022064224, 0.012946089, -0.002955667, 0.018997107, -0.039278124, 0.007786674, -0.0007307437, 0.016322296, -0.030362085, 0.0301481, -0.00029255752, -0.020043256, -0.0010721537, 0.0024459667, 0.0056052166, 0.0051178066, -0.03166977, 0.0063185, 0.027556505, -0.009730371, -0.016857257, -0.0021235032, 0.040062737, -0.0075964653, -0.006853462, 0.028269788, -0.004877073, -0.01464608, 0.0016672993, -0.008761494, 0.0055695525, -0.01666705, -0.021457935, -0.0060569625, 0.02929216, -0.027437624, -0.0151572665, 0.008083875, -0.029006846, 0.0023434323, -0.046363402, -0.02245653, -0.020578219, 0.01012862, 0.008416741, -0.005896474, 0.03276347, 0.02185024, 0.0012527035, -0.012708328, -0.0006114917, -0.025202671, -0.027841818, -0.016013207, 0.056729786, 0.003634772, -0.0060361587, 0.007305208, 0.00592025, 0.00026636667, -0.015704118, 0.0077272337, -0.012048541, 0.013516715, -0.016762154, -0.016227191, -0.020376122, 0.018723682, -0.025583088, 0.025178894, -0.01141253, 0.005236687, -0.033001233, -0.0031384458, 0.015109714, -0.002752084, -0.01055659, 0.015169155, -0.017796414, -0.021065628, -0.039159242, 0.0061461227, 0.03216907, -0.00961149, -0.01861669, 0.004835465, 0.010990504, 0.014562864, 0.011246097, -0.04498439, 0.028626429, 0.028531324, 0.03278725, 0.009777923, 0.028412445, -0.00906464, 0.00860695, -0.020839754, 0.015097826, 0.018771235, 0.042678107, -0.028626429, -0.043843135, -0.018224385, -0.012054484, 0.00640766, 0.005203995, 4.8063022e-05, -0.02563064, -0.023419462, 0.0247747, 0.018783122, -0.010889456, 0.017356556, -0.02062577, -0.008505901, 0.0017698337, -0.0440809, 0.03399983, 0.014860065, -0.012660775, 0.0031116977, -0.004038966, 0.011691899, 0.07128076, 0.017558653, -0.018604802, -0.0049632615, 0.002500949, -0.041655734, -0.013813917, -0.001362668, -0.020352345, 0.020827867, -0.0119593805, -0.011109385, 0.026153715, 0.005269379, -0.018057952, 0.011139105, 0.008119539, -0.006247171, 0.0020298848, -0.014455872, 0.005762733, 0.0015075536, 0.0011984643, -0.011888052, 0.03789911, -0.014170558, 0.02062577, 0.0366152, 0.0005375629, 0.015109714, 0.020352345, 0.012613223, 0.034950875, -0.008315693, -0.0024103026, -0.0037090722, -0.03271592, -0.027794266, -0.015026498, -0.010330717, -0.016060758, -0.0015216706, 0.007061503, 0.010812183, 0.041513078, 0.0074300324, 0.0029452648, 0.018022288, 0.00456204, 0.0571102, -0.030433413, 0.0012787086, 0.0048711295, 0.03882638, -0.008850655, 0.019912487, 0.02850755, 0.0023790963, 0.013706924, 0.018890115, 0.005893502, 0.0141824465, 0.026106162, 0.07689192, -0.018462146, -0.021826463, 0.01201882, 0.00958177, -0.013540491, -0.007513249, 0.055160563, -0.004514488, -0.018640466, 0.0055814404, 0.01575167, 0.002068521, 0.02850755, 0.01238735, -0.023597782, 0.0020016509, 0.008066043, -0.0008225046, -0.017772637, 0.010015684, -0.005236687, -0.04137042, 0.0008343927, -0.017653758, -0.014836289, 0.019436965, 0.028959295, -1.9968213e-05, -0.050120026, -0.0018129279, -0.012280358, 0.007531081, 0.014931394, 0.008077932, 0.027294967, -0.0025975395, -0.02062577, -0.04443754, -0.021232061, -0.008339468, -0.0066929734, 0.015395028, -0.016892921, -0.010859735, 0.010859735, 0.035997022, -0.011721619, -0.005768677, -0.01390902, 0.021600591], "content_hash": "119ceb4bfc9d0c7f00c40ff0b036d7d5"}, {"content": "See in the Dashboard In **Stage 2 (Embedding)** of the pipeline, you can see: - The dimension of the embedding vectors (e.g., \"768-dimensional\" for GPT-2) - A visual showing the flow: Token ID \u2192 Lookup Table \u2192 Vector - An explanation of how the lookup table was created during training\n\nThis is the point where raw token IDs become rich numerical representations that the rest of the model can process.", "source_file": "embeddings_explained.md", "chunk_index": 1, "embedding": [-0.005226687, -0.0048269634, 0.026333297, -0.0054689436, 0.008109541, -0.03323761, 0.015480199, 0.051019248, -0.029482633, 0.040141925, 0.04859668, 0.014486947, -0.014595962, -0.07204712, 0.023280863, -0.023329314, -0.016764158, 0.022856914, -0.012240017, 0.039197125, -0.00060374895, -0.02553385, -0.0059171184, 0.060661063, 0.012900166, 0.018847566, 0.01402666, 0.035563275, 0.030354757, 0.006813468, 0.006462196, -0.03028208, -0.027181195, -0.023414103, 1.0900366e-05, -0.033213384, -0.008999834, 0.03786471, -0.051600665, 0.031251106, 0.016449226, 0.029458407, -0.08217345, 0.0639073, -0.0075341812, 0.046052985, -0.012754812, -0.06269602, -0.02977334, 0.05615509, -0.013215099, 0.0209552, -0.045616925, -0.0061714877, -0.005574931, 0.011755503, -0.020179978, -0.03687146, -0.036992587, -0.03469115, -0.01843573, 0.012451991, 0.02686626, 0.016606692, 0.028586283, -0.01279115, -0.06884933, 0.073161505, -0.015746681, 0.024274115, 0.053441815, 0.019029258, -0.0036156804, 0.003857937, 0.06376195, -0.0049632327, -0.03430354, 0.019998286, -0.038567256, -0.0584323, 0.024371017, -0.043049004, -0.036411174, -0.015468086, -0.002375629, -0.03696836, 0.009708434, -0.034618475, -0.010683518, -0.053926326, -0.049904868, -0.020349557, -0.029506858, 0.0016488592, 0.015310619, 0.054701548, 0.011931139, -0.043097455, -0.02933728, 0.01693374, 0.0030024683, 0.008460813, -0.0047118915, -0.0068376935, 0.017527267, -0.03301958, 0.01174339, 0.029361505, -0.03083927, -0.045738053, -0.10029425, 0.0052509126, -0.009744774, 0.012718474, 0.033528317, 0.0021651688, -0.020470686, 0.026624003, 0.022602543, 0.003014581, -0.042079978, -0.031008849, 0.014704978, -0.0039911782, -0.048984293, 0.03575708, -0.015577102, -0.011004508, -0.02027688, -0.0098053375, -0.039560508, -0.012100719, 0.02771416, -0.016364437, -0.00917547, -0.048257522, -0.013336228, 0.06507013, 0.010029425, 0.0006355451, -0.016279645, -0.04055376, 0.030039823, 0.0102716815, -0.041765045, 0.030185176, -0.054604646, 0.07122345, -0.02921615, 0.004605904, -0.017103318, -0.009538855, 0.0026027446, -0.04636792, -0.032704648, -0.01561344, -0.028222898, 0.020773506, 0.02805332, -0.013287776, -0.025170464, 0.014450609, 0.06880088, 0.0064137443, -0.06817102, 0.018811228, -0.015480199, -0.040190376, -0.030597012, -0.060709514, 0.008890819, -0.01985293, -0.03803429, -0.026914712, 0.0118766315, -0.017854314, -0.05363562, -0.035054535, 0.008648562, -0.062599115, -0.05838385, 0.0018608337, -0.023208186, 0.02536427, -0.023074945, 0.04697356, -0.012173396, 0.014220465, 0.022178596, 0.022493528, -0.008902932, 0.024249889, 0.05407168, 0.009968861, -0.0055870437, 0.007734043, 0.019513773, 0.023087058, 0.045156635, -0.0025406664, 0.029870244, 0.019344192, 0.024431583, -0.0036277932, -0.011059015, -0.007600802, -0.024346791, -0.01360271, -0.023450442, 0.0069709346, -0.014426382, -0.025073562, 0.010465487, -0.0037216675, -0.04508396, 0.06313208, -0.030984623, 0.016364437, -0.0006272176, -0.007921792, 0.0047573145, 0.011101411, 0.04442987, -0.007879397, -0.049807966, 0.009714491, 0.0007600802, -0.04338816, 0.0075220685, 0.0032401825, 0.0021606265, 0.012373257, 0.01637655, -0.022784237, 0.010931831, 0.01882334, -0.02374115, -0.059255973, -0.014450609, -0.03301958, -0.019344192, -0.0032734927, 0.0016094926, 0.03207478, 0.004733089, 0.01578302, -0.0010674433, -0.027738385, 0.051697567, 0.038131196, -0.009254203, -0.011295215, -0.012803263, -0.0036883573, 0.001281689, -0.0144627215, -0.020470686, 0.043775775, 0.018859679, -0.047240045, 0.06700818, 0.008999834, -0.08081681, 0.007382771, 0.0012249101, -0.018738551, -0.011264933, -0.014062998, -0.047627654, -0.067686506, 0.016182743, -0.01856897, 0.0049420353, 0.024237776, -0.056397345, -0.0046816096, -0.02429834, -0.007600802, -0.0029661297, 0.018593198, 0.009454065, -0.014099336, 0.067638054, 0.05911062, -0.020737167, -0.008902932, 0.015795132, 0.024056084, -0.021064214, -0.028489381, 0.046997786, 0.020882523, 0.059498228, -0.0010099073, -0.021936338, 0.021948451, 0.016885288, 0.019416869, 0.02865896, -0.009980974, 0.023074945, -0.07151416, -0.00926026, 0.0021379148, -0.014644413, 0.031299558, 0.012839601, 0.012597345, 0.020519137, -0.04680398, -0.013639049, -0.031469136, 0.01684895, -0.007152627, -0.009302655, 0.025122013, 0.035321016, 0.008460813, 0.02446792, -0.05247279, 0.022069579, -0.02147605, -0.020010399, -0.025994137, 0.027738385, -0.015346958, -0.03255929, 0.030257855, 0.050098673, -0.0118766315, 0.03440044, 0.0037034983, -0.027060065, 0.016800499, -0.04932345, -0.019283628, 0.0005560547, 0.02203324, -0.011137749, 0.034570023, -0.056300443, -0.02228761, -0.009435896, -0.043896902, -0.067638054, 0.028029094, -0.01561344, 0.019259403, 0.040238827, -0.023123397, 0.0033885648, -0.033988606, -0.010065763, -0.032631967, 0.030378982, -0.01262157, 0.010822815, -0.03965741, -0.02360791, -0.03631427, -0.027568806, 0.0005590829, 0.034715377, 0.019404756, -0.040989824, -0.010519994, -0.020337444, 0.018726438, -0.0071707964, -0.0037792036, 0.01176156, 0.01304552, -0.006643888, -0.0047240043, -0.017527267, 0.0014399128, 0.033843253, 0.01882334, -0.04636792, 0.0050449944, 0.029240375, 0.006831637, -0.027883738, -0.02044646, -0.0029509887, 0.049178097, 0.029191924, 0.00740094, -0.035393696, 0.0023196072, -0.02798064, -0.031711392, 0.009768999, 0.0016079784, 0.0079763, 0.010622953, -0.006874032, 0.018399391, 0.02720542, 0.0005412922, -0.04060221, -0.045059733, -0.016606692, -0.04564115, 0.038688384, 0.0017063952, 0.033528317, -0.015770907, 0.021754647, 0.031517588, -0.008273064, 0.022893252, 0.011161975, -0.05625199, -0.0070557245, 0.016715707, -0.011719164, 0.04161969, -0.0062986724, 0.08173739, -0.046246793, 0.007612915, -0.05072854, 0.006322898, -0.024903983, -0.0046634404, 0.049032744, -0.008709126, -0.012015929, 0.010404922, -0.020179978, 0.046295244, 0.040238827, -0.014038772, -0.02129436, -0.034327764, 0.05542832, -6.893904e-05, -0.015407522, 0.020640265, -0.006680227, -0.0029752143, -0.02360791, -0.03066969, -0.049371902, -0.020761393, -0.018411504, -0.040529534, 0.020725055, -0.035224114, 0.038906414, 0.03798584, -0.02865896, 0.0074493918, -0.030790819, -0.082125, 0.019489547, 0.033625223, 0.03624159, 0.008927157, 0.0073222066, 0.016618805, -0.03776781, 0.021912113, 0.009272372, 0.00034332307, -0.012863827, -0.02989447, 0.039778538, -0.0223724, -0.017696848, -0.009229978, 0.009484347, -0.01672782, 0.0010787991, -0.016437113, 0.04338816, -0.029991372, -0.034037057, 0.02382594, 0.011973534, -0.0010212631, -0.061145574, -0.0029479605, 0.020325331, 0.12238805, 0.0007385042, -0.010731969, 0.044672124, -0.0032371543, -0.038470354, -0.048354425, 0.008315459, -0.012276355, 0.010731969, 0.021209568, -0.031129979, -0.055767477, 0.0059776823, 0.001921398, 0.026624003, 0.016315984, 0.0058474694, 0.028198672, 0.047070466, 0.009169414, 0.0077824946, 0.022021128, 0.004687666, 0.033843253, -0.022408739, 0.026430199, -0.011749446, 0.007612915, 0.0007653796, 0.010096045, 0.0144142695, 0.008787859, 0.017139656, 0.0058171875, 0.007734043, 0.0020334416, 0.022772124, -0.019489547, -0.019356305, -0.016182743, -0.028634734, -0.031372234, -0.0049844305, -0.0119856475, 0.021367036, -0.048863165, 0.0041001937, 0.0046452708, -0.0783458, 0.019743916, -0.0036308214, 0.022578318, 0.008503208, 0.039536282, 0.009018003, 0.040820245, 0.02793219, 0.00032610015, -0.009011947, 0.030378982, 0.0018275235, 0.0034188468, 0.0008713668, 0.0419104, 0.025994137, -0.028344026, 0.034376215, -0.0005999637, -0.023208186, -0.004197096, 0.015201604, 0.016994303, 0.0010091503, 0.0033976494, -0.008781803, 0.024177212, 0.061242476, -0.016812611, -0.04658595, 0.054701548, 0.025049336, -0.043872677, 0.027907964, -0.054507744, -0.054943804, 0.016715707, 0.029361505, 0.051455308, -0.014390044, 0.007867284, -0.0033370852, 0.02994292, -0.025097787, 0.01480188, -0.0148503315, -0.010126327, 0.0057657077, -0.02014364, -0.020422235, 0.023462554, 0.019877156, -0.010362527, -0.022820575, -0.007467561, 0.05508916, -0.0049329507, -0.022142256, 0.017188108, -0.011634375, -0.013372567, 0.025097787, -0.01603739, -0.024831304, -0.01830249, 0.0011832722, -0.0045362557, -0.013215099, -0.0024210522, -0.013287776, 0.02441947, -0.022105917, 0.040965598, -0.013069745, 0.012155226, -0.009387445, -0.051649116, -0.011598037, 0.012997068, -0.020737167, -0.006540929, -0.011852406, 0.008357854, 0.0029782425, 0.02553385, 0.015237942, -0.02039801, -0.0045604813, -0.039972346, -0.013663274, 0.042419136, -0.019441094, -0.027108518, -0.029070796, 0.008672788, -0.025049336, 0.01061084, 0.04469635, -0.0070436117, 0.012088606, 0.0119069135, -0.0026315127, -0.0018123825, 0.01265791, -0.041813497, 0.01753938, -0.011537473, 0.0050116843, -0.025558075, -0.01061084, 0.02129436, 0.0034309595, 0.029313052, -0.031735618, -0.030766593, -0.029167699, -0.04457522, 0.03306803, -0.007152627, -0.0030918003, -0.04140166, -0.024443695, -0.02365636, -0.00045385267, 0.003876106, 0.0038488524, -0.005523451, -0.022735786, -0.02352312, -0.011894801, 0.048693582, 0.011083241, -0.0046422426, 0.0040577985, -0.021912113, -0.047336947, 0.053054202, -0.0028661988, 0.00910885, 0.002852572, -0.0354906, -0.033431415, 0.013372567, -0.0016064643, 0.03083927, -0.019223064, 0.025461173, -0.006577268, 0.0043727322, -0.00077597826, -0.01144057, 0.032535065, -0.012815376, 0.022856914, 0.017091205, -0.017333463, 0.0020924916, 0.015674004, -0.013820741, -0.009151245, 0.0068376935, -0.008824198, 0.04518086, -0.008818141, -0.024104536, -0.009569137, -0.02292959, -0.04302478, -0.05256969, -0.0100536505, -0.016425, 0.015019911, 0.011137749, 0.014608075, -0.017914878, -0.031372234, -0.0013566371, 0.012451991, -0.0075038993, -0.014923009, 0.019695465, -0.034061283, -0.0017048811, -0.0068922015, 0.0016367463, -0.028247124, 0.0046937224, -0.02587301, 0.0051509817, 0.009157301, -0.045011282, -0.033213384, -0.010592671, 0.018653762, 0.0070678373, 0.023753263, 0.0017442477, -0.005635495, -0.050534733, 0.026502876, 0.017805863, -0.012633683, 0.02360791, -0.03122688, 0.013421018, -0.040699113, 0.019198839, -0.008648562, -0.008642506, 0.008612223, 0.010556333, 2.5077346e-05, -0.023305088, 0.018677987, -0.05978894, -0.007001217, 0.0080913715, 0.0072010783, -0.035466373, 0.004148645, -0.0018744607, 0.051455308, -0.017781638, 0.008133766, 0.0023044662, 0.01835094, 0.016497677, 0.016836837, 0.04786991, 0.021209568, -0.029797567, 0.012645796, 0.022445077, 0.026575552, 0.0053114765, -0.016352324, -0.0055416203, 0.010235343, -0.0019456237, 0.06589381, -0.040674888, 0.028465156, -0.009605476, -0.0018138966, 0.009799281, -0.03764668, 0.004203153, 0.00042697732, -0.02771416, 0.013699613, 0.008194331, -0.016691482, -0.0108954925, 0.0077946074, -0.024625387, -0.039851215, 0.0020924916, 0.004203153, 0.0035490596, -0.01466864, -0.007394884, 0.012439879, -0.088229865, -0.012924392, 0.027326548, -0.007237417, 0.024576936, -0.004787597, 0.03837345, -0.015734568, -0.002227247, -0.025703428, -0.014450609, -0.011967477, 0.019913495, -0.024952434, 0.015189491, -0.03178407, 0.02887699, 0.008478982, -0.00757052, -0.0075038993, -0.03592666, -0.0055719027, 0.0483302, 0.008248839, -0.06536084, -0.0039124447, -0.00045082445, 0.007019386, 0.041159403, 0.0056142975, -0.02686626, 0.022917477, -0.0041910396, 0.03704104, 0.014232578, -0.0006029919, -0.0070738937, -0.025945686, 0.042758297, 0.031493362, 0.01796333, -0.04627102, -0.007528125, 0.0042939987, 0.050340928, 0.0025648922, 0.004399986, -0.0030251797, 0.012984956, 0.009641814, 0.02715697, 0.027568806, -0.017103318, -0.05402323, 0.02408031, 0.015625553, 0.0028722552, -0.011059015, 0.029991372, -0.0078733405, -0.0055537336, 0.0036277932, 0.023450442, -0.005959513, 0.011361836, 0.006274447, 0.028150221, 0.016824724, -0.010453374, -0.042322233, -0.012984956, -0.008024751, 0.014172013, 0.0024089394, -0.022772124, 0.017478816, -0.0037913164, 0.011713108, -0.035151437, 0.003046377, 0.022965929, 0.0074191093, -0.029627986, 0.028513607, -0.00042319205, 0.01732135, -0.02340199, -0.02502511, -0.012076493, -0.01208255, 0.017272899, -0.0146323005, -0.031711392, 0.02147605, -0.002127316, -0.0008796944, 0.007152627, 0.022445077, 0.045495797, -0.013457356, 0.07054513, -0.0015928374, -0.04169237, -0.015988939, -0.046513274, -0.008394192, 0.012609458, -0.01664303, -0.0082003875, -0.019731803, -0.0068195243, -0.0016412887, -0.024903983, 0.005562818, -0.028368251, 0.022384513, 0.034061283, 0.041813497, 0.017612057, -0.06172699, -0.00038552872, -0.019271515, -0.0071284017, 0.031711392, -0.041474335, 0.0017548465, 0.014620188, 0.022130143, -0.012645796, -0.016861062, -0.044890154, 0.03631427, 0.02587301, 0.009557025, -0.028150221, 0.028973894, 0.003888219, 0.005635495, -0.06608761, 0.02164563, -0.012476217, -0.0039124447, 0.007092063, -0.01552865, 0.007788551, 0.03580553, -0.015867809, -0.013445243, -0.0064137443, 0.0018396364, 0.029240375, 0.017866427, -0.022772124, 0.021233793, 0.020725055, 0.012415652, 0.007297981, -0.038736835, -0.009932522, 0.048354425, 0.018932356, 0.030330531, 0.005853526, 0.00804292, 0.011949308, -0.02613949, 0.011052959, 0.022190709, -0.045956083, 0.01946532, -0.011392118, 0.012288468, -0.0033401134, 0.030330531, 0.027132744, 0.04033573, -0.016001051, 0.050583184, 0.028828539, -0.021318585, -0.02421355, 0.03323761, -0.0107138, -0.0061714877, 0.02887699, -0.025436947, -0.01985293, 0.0062805032, -0.029531084, 0.03306803, -0.0008198873, 0.012803263, 0.0062199393, 0.01796333, -0.00887265, -0.01552865, -0.012512555, 0.017926991, -0.004324281, 0.03134801, -0.0055658463, -0.010871267, 0.006643888, 0.017939104, -0.030766593, -0.011258877, 0.013081858, -0.00567789, 0.025485398, -0.01399032, 0.0159526, 0.012827489, 0.017527267, 0.04217688, -0.027350774, -0.00924209, 0.0026814782, -0.016861062, 0.013299889, 0.0037458932, -0.023547346, -0.012058324, -0.016739933, -0.002548237, -0.022166481, -9.595634e-05, 0.03306803, 0.024952434, -0.02771416, -0.0059534567, -0.020119414, -0.031517588, 3.4865793e-06, 0.00294039, 0.029506858, 0.0065167034, 0.0010583587, -0.013614823, 0.022214934, -0.0010295907, 0.010417036, 0.038615707, 0.0040093474, 0.008170105, -0.003334057, -0.015819358, 0.030378982, 0.026527101, -0.008581941, 0.0035702572, 0.04658595, 0.0050601354, 0.01167677, -0.047554977, 0.030064048, 0.019877156, 0.029724889, 0.0056476076, 0.011065071, -0.019840818, 0.016158517, 0.04564115, 0.05402323, -0.013324115, -0.00591409, -0.01225213, 0.0066923397, 0.0107077435, 0.015855696, 0.03665343, -0.020482799, 0.026672456, -0.021621404, -0.0053114765, 0.012766925, 0.0076553095, 0.014087223, -0.0058474694, 0.023511006, -0.0077219303, -0.028610509, -0.04072334, 0.024576936, -0.012972843, -0.020979425, -0.0028798257, -0.026066814, -6.865515e-05, 0.036362723, -0.019150387, -0.007364602, 0.038252324, 0.0019304826, -0.00055681175, 0.0072919247, 0.004584707, -0.00042394912, 0.0053780973, 0.0054114074, -0.038179647, 0.016861062, -0.020373782, -0.006044303, -0.020882523, -0.005659721, 0.015758794, 0.033431415, -0.004602876, -0.019041372, -0.004618017, -0.00046407286, -0.03490918, 0.014450609, 0.019392643, 0.010810702, 0.004530199, -0.0125610065, -0.038785286, 0.02536427, -0.011791842, -0.009690265, 0.023038605, -0.010326189, 0.027859513, 0.0047785123, -0.02865896, 0.038010065, -0.019428981, 0.016388662, 0.023789601, 0.04811217, 0.0100536505, 0.003482439, 0.0027753527, -0.03374635, -0.032002103, 0.0011090812, -0.013530033, -0.028683186, -0.014244691, 0.02129436, 0.017951217, 0.004518086, -0.008218557, 0.0041092783, -0.00055983994, 0.010205061, 0.019707577, 0.048063718, 0.025170464, 0.042104203, -0.020240542, -0.0019123133, 0.011664657, -0.013360454, -0.004030545, 0.00024888085, -0.008194331, -0.05693031, 0.004487804, 0.01971969, 0.015092588, -0.008006582, -0.0017381913, -0.007885453, -0.0010825844, 0.031154204, 0.0056718336, 0.056445796, 0.011198313, 0.0019198839, -0.0079823565, -0.016352324, 0.008176162, -0.021960564, 0.009781112, 0.009084624, -0.0021394289, 0.013505807, 0.022275498, -0.036677655, -0.05368407, 0.023498893, 0.02647865, -0.009296599, 0.010011256, -0.0001429882, 0.04084447, 0.0035127213, 0.0014611103, 0.020507025, -0.030233629, 0.020034624, -0.028973894, -0.01997406, 0.0070738937, 0.018508406, -0.02019209, 0.0032371543, -0.022820575, 0.001049274, 0.029119248, 0.005856554, 0.012143114, 0.031590264, -0.03156604, 0.0047845687, 0.00294039, -0.011386062, 0.007582633, -0.013336228, 0.029506858, -0.024879757, 0.02977334, 0.0075099557, -0.05591283, -0.011004508, 0.007279812, -0.009593363, 0.022566205, 0.020349557, 0.0068074116, -0.027520353, -0.020422235, 0.015661892, 0.006213883, 0.03876106, -0.014789768, -0.03575708, 0.023765376, -0.0365323, -0.025267366, 0.0052993638, 0.0012612486, -0.002386228, 0.0020940057, 0.032195907, -0.00023184717, -0.0018835453, 0.029749114, 0.008412362, -0.011634375, -0.00675896, 0.037888937, -0.007431222, -0.011567755, -0.010828871, -0.027302323, 0.019489547, 0.0051812637, -0.0043121683, -3.6906285e-06, 0.015274281, 0.013796516, 0.004233435, -0.009714491, -0.023232412, 0.020046737, 0.006347124, 0.00018150322, -0.003624765, 0.012137057, -0.0005288008, 0.022820575, -0.0050268252, 0.020712942, 0.010925774, 0.05848075, -0.0060685286, 0.018108683, 0.020579701, -0.015419635, 0.019101936, -0.010186891, 0.035829756, -0.00045801644, 0.0024286227, -0.016521903, -0.019986173, -0.024613274, 0.012270299, 0.033988606, 0.019961948, -0.023208186, 0.010235343, 0.007933904, -0.0057717646, -0.027011614, -0.0009682695, 0.059013717, 0.007352489, 0.008757577, -0.0025815472, -0.0039578676, 0.03704104, 0.008267008, 0.025291592, -0.00075326674, -0.038082745, 0.0035732854, -0.010441261, -0.021972677, 0.0029752143, -0.0011878145, 0.023074945, -0.044841703, -0.0223724, 0.005420492, 0.011161975, -0.023329314, 0.027229646, 0.0137601765, -0.028780088, 0.0075523504, 0.010986338, -0.008454757, -0.0093692755, -0.030354757, -0.004194068, 0.00326138, -0.00883631, -0.025485398, 0.005995852, 0.032728873, 0.0068800882, -0.019925607, 0.02805332, 0.026042588, -0.028125996, 0.00975083, -0.022069579, -0.0096175885, -0.0061351494, -0.01958645, 0.009968861, 0.029167699, -0.043412387, -0.008612223, -0.00031682625, -0.017575718, 0.009066454, -0.039342478, -0.011458739, -0.016873175, 0.0069709346, 0.0337948, -0.006129093, 0.048039492, 0.011149862, -0.002854086, -0.023438329, -0.003915473, -0.026938938, -0.016291758, -0.018411504, 0.038979094, -0.0019304826, -0.012936505, 0.0071284017, -0.011773673, 0.0055809873, -0.017890653, 0.009133075, -0.0029373616, 0.02301438, -0.019562224, -0.007370658, -0.020240542, 0.019647013, -0.03893064, 0.025146239, -0.010344358, 0.011797898, -0.02156084, -0.0055264793, 0.011222539, -0.0065167034, -0.019259403, 0.041571237, -0.0043273093, -0.023716925, -0.035466373, 0.0042092092, 0.010144496, -0.008024751, -0.0057747927, 0.0136875, 0.018629536, -0.0064743087, -0.0012074979, -0.040989824, 0.024576936, 0.02703584, 0.03648385, 0.008369966, 0.023753263, -0.007297981, 0.020797731, -0.025146239, 0.02933728, 0.009393501, 0.031590264, -0.02164563, -0.037162166, -0.02921615, 0.022990154, -0.0034006776, 0.018132908, -0.014741316, -0.030621238, -0.030621238, 0.025509624, 0.01672782, -0.025267366, 0.01296073, -0.019683352, -0.0034097622, 0.017842202, -0.027956415, 0.03238971, -0.0029343334, 3.60546e-05, 0.012131001, -0.0013240839, 0.016134292, 0.08900509, -0.0065045906, -0.0006033704, 0.00036395274, 0.007649253, -0.009381388, -0.008406306, 0.0069103707, -0.008224613, 0.008115597, -0.0029358475, -0.011458739, 0.011458739, 0.023462554, -0.0056476076, -0.012373257, 0.0007316907, -0.0075947456, -0.024928208, -0.015516537, 0.015443861, -0.0061078956, 0.0045090015, -0.008957439, 0.039124448, -0.0013944898, 0.006831637, 0.036944136, -0.010580558, 0.020228429, 0.025994137, 0.008139823, 0.023716925, -0.0072010783, -0.011289159, -0.034763828, -0.030330531, -0.018241925, -0.011010564, -0.0071284017, -0.023341427, -0.0016640002, 0.008921101, 0.03212323, 0.0336979, 0.012403539, 0.005826272, 0.0054114074, 0.032656193, 0.05232743, -0.026672456, 0.010526051, 0.011579867, 0.039681636, -0.0074251657, 0.014692865, 0.0456896, 0.008485039, 0.0007820347, -0.00024623115, -0.00030301005, 0.011700995, 0.01182818, 0.052375883, -0.024274115, -0.02686626, 0.018496294, -0.017951217, -0.008981665, 0.0051206998, 0.04396958, -0.008878706, -0.010556333, -0.00075970165, 0.014184126, -0.011065071, 0.030064048, -0.0023771431, -0.012597345, 0.01432948, 0.008521377, -0.0036611033, 0.006861919, -0.018532632, -0.02587301, -0.03185675, -0.015395409, 0.003682301, -0.015383297, 0.03742865, 0.015153153, -0.0068255807, -0.057172567, -0.0101626655, -0.0219, -0.0016004079, 0.024092423, 0.016739933, 0.028949669, 0.009018003, -0.014281029, -0.044550996, -0.017394027, -0.003421875, -0.0079763, 0.012706361, 0.006716565, -0.00941167, -0.0006132121, 0.028925443, -0.022941703, -0.020252654, -0.006359237, 0.016521903], "content_hash": "1495cb0f87b7e94b260b27c31df51009"}, {"content": "# Experiment: Exploring Alternative Predictions with Beam Search\n\n## Goal\n\nLearn how beam search reveals multiple possible continuations of a prompt, and see how ablating attention heads can redirect the model's generation from one path to another.\n\n## Prerequisites\n\n- Complete \"Your First Analysis\"\n- Complete \"Your First Ablation\"\n\n## Steps\n\n### Step 1: Generate Multiple Beams\n\n1. Select **GPT-2 (124M)** and enter the prompt: `Once upon a time there was a`\n2. Set **Number of Generation Choices (Beams)** to **3**.\n3. Set **Number of New Tokens** to **8**.\n4. Click **Analyze**.\n\n### Step 2: Compare the Beams\n\nYou should see 3 different generated sequences. Look at how they differ:\n- **Beam 1**: The model's top-ranked overall sequence\n- **Beam 2**: The second-best sequence\n- **Beam 3**: The third-best sequence\n\nNotice how they might start the same but diverge at some point. For example:\n- Beam 1: \"Once upon a time there was a young man who lived\"\n- Beam 2: \"Once upon a time there was a little girl who loved\"\n- Beam 3: \"Once upon a time there was a king who ruled\"\n\nThe beams share a common prefix because the early tokens were confident, but as generation continues, different paths emerge.\n\n### Step 3: Select a Beam for Comparison\n\n1. Click **\"Select for Comparison\"** on Beam 1 (the top-ranked sequence).\n2. This stores it as the baseline for ablation comparison.\n\n### Step 4: Investigate What Drives the Divergence\n\n1. Look at **Stage 5 (Output)** to see the top-5 predictions for the immediate next token.\n2. Note the top prediction and its probability. Are the alternatives close in probability? If so, the model was uncertain, which explains why beams diverge early.\n\n### Step 5: Ablate a Head and Re-Generate", "source_file": "experiment_beam_search.md", "chunk_index": 0, "embedding": [0.04827575, 0.04152533, 0.029295666, -0.020938, -0.025131445, -0.011009619, -0.0128871715, 0.022793636, 0.031326637, 0.07007582, 0.03261243, -0.044184588, -0.029821673, -0.014172966, 0.0050664702, -0.001339674, 0.00023309597, -0.015064255, 0.0061696237, 0.10093489, -0.02730853, -0.02976323, 0.0045477687, 0.012280802, 0.023334254, -0.039771974, 0.004788855, 0.022486798, 0.0046865763, -0.012470749, 0.003638215, -0.022253018, -0.020499662, -0.014728195, -0.048188083, 0.0014976588, -0.024736939, 0.022486798, -0.01203241, -0.0074079316, 0.0028711215, -0.03854462, -0.02966095, 0.025014555, -0.0016035907, 0.05362349, 0.013062507, -0.016539996, -0.009139371, 0.03305077, 0.018614803, 0.0732903, -0.026519518, 0.0035797697, -0.015750987, -0.010213302, -0.031531196, -0.040707096, 0.012083549, -0.040794764, -0.0011871685, 0.017708901, 0.030303847, 0.047954302, 0.0052308473, -0.012441526, -0.048071194, 0.036732823, 0.015108089, 0.030654518, 0.026022734, 0.0007944897, -0.021069502, 0.013975713, -0.030946745, -0.009424292, 0.04491515, 0.0018738998, -0.03635293, -0.03807706, 0.034774907, 0.019579148, 0.033752114, -0.027483866, 0.0074481126, -0.010074494, -0.005409836, -0.021011056, -0.06247794, -0.040473316, -0.022267628, 0.02114256, -0.03459957, 0.044038475, 0.03357678, 0.026417239, -0.009490043, -0.04164222, 0.030946745, 0.034102786, 0.020543495, -0.0042153616, -0.036031477, -0.013654265, 0.07060183, 0.012120077, 0.0033752115, 0.035359357, -0.033313777, -0.033869006, -0.041174658, -0.015239591, -0.005482892, 0.0035578527, -0.0040290672, 0.01725595, -0.012960227, 0.017928071, 0.04959077, -0.019959042, -0.049240097, 0.000374643, -0.01892164, -0.013194009, 0.00043719765, -0.031092858, 0.06428974, -0.06247794, 0.006377835, -0.004134999, 0.017095227, 0.009066314, 0.0066700606, -0.053711157, -0.052834477, 0.031005189, -0.03278777, 0.012025104, -0.05485084, 0.040122643, -0.0044527953, -0.0012730099, -0.013062507, 0.0012173044, -0.061601263, 0.01035211, 0.006933064, 0.045061264, -0.020850332, -0.0009734782, 0.04152533, -0.023202753, -0.028389765, -0.029003441, 0.0053258208, -0.047311407, -0.022150738, 0.024517769, -0.029734006, -0.049999885, 0.027717646, 0.0217124, -0.02776148, -0.0021387297, -0.034541126, 0.051344126, -0.032992326, 0.0443307, -0.02104028, -0.034219675, -0.020090545, -0.006253639, -0.006304778, 0.0011615987, 0.021653954, 0.0048911343, -0.086966485, -0.07498522, -0.040473316, -0.061601263, 0.0067796456, 0.018234909, -0.023246586, -0.00850378, -0.005804341, -0.029164165, -0.0033240719, -0.00079494633, -0.013347427, -0.014662445, -0.0029332195, 0.033518333, 0.04845109, 0.010257136, -0.060841475, -0.020996446, 0.059497233, 0.047340628, 0.033109218, -0.0026756953, 0.04894787, -0.0036546527, 0.022852082, 0.012054327, 0.016949113, 0.037901726, -0.003581596, 0.021244837, -0.05888356, -0.031823423, 0.046873067, -0.05701331, 0.04231434, -0.0018848584, -0.032261763, -0.028404377, 0.006180582, 0.023261197, 0.007890104, 0.045207378, 0.016174715, -0.012514583, 0.018644026, 0.053477377, 0.018234909, 0.028492045, 0.052366916, -0.03471646, 0.03582692, 0.001366157, 0.026431851, -0.007678241, -0.01881936, -0.0018291527, 0.015750987, -0.05184091, -0.009059009, 0.007524822, 0.0128871715, -0.060198575, 0.0009497348, -0.0054354053, 0.048217308, -0.010848894, -0.013727321, 0.032203317, -0.047077626, -0.052337695, 0.00036414113, 0.07498522, 0.024707716, -0.0016839529, -0.019842152, 0.014684361, -0.0107320035, -0.017650457, -0.009110149, -0.01304059, -0.022267628, -0.024590826, -0.009358541, 0.018512523, -0.017986517, 0.002750578, -0.024313211, 0.0070134266, -0.011287233, -0.038456954, -0.043512467, -0.041233104, -0.025613617, -0.011849768, 0.0068527022, -0.005614394, -0.025467504, -0.045937944, -0.0013780287, -0.0010711913, -0.0028802536, -0.01275567, -0.037434164, -0.01035211, 0.03404434, -0.020178212, 0.00027487517, 0.020879555, -0.04164222, 0.016379273, -0.0077951313, -0.011973964, 0.010673558, -0.051227234, 0.026899412, -0.025555173, 0.011316456, 0.024780773, 0.004208056, 0.006096567, -0.04812964, 0.036206815, 0.0166715, -0.029748617, 0.038573846, -0.02194618, 0.016306216, 0.011046147, -0.024342434, 0.016130881, -0.019783707, 0.005135874, -0.0019012961, 0.015371093, -0.045032043, -0.05742243, -0.0036710904, -0.029412556, 0.022326075, -0.02786376, -0.0008200595, -0.032174096, -0.029018052, -0.03658671, -0.0006232635, 0.004018109, 0.008993259, 0.03471646, 0.010191386, -0.018527135, 0.012280802, -0.009139371, 0.004478365, 0.013113646, 0.025482116, 0.017168283, -0.048655644, -0.036031477, 0.009416986, 0.04827575, 0.037317272, 0.026607186, -0.06727045, 0.010008744, 0.0040655956, -0.022501409, -0.035184022, 0.03807706, 0.008175025, -0.026519518, 0.011316456, 0.0029478308, -0.03737572, -0.01947687, -0.01924309, -0.012266191, 0.026271127, 0.028404377, 0.002390775, -0.025613617, 0.029383335, -0.006312084, -0.013033284, 0.06025702, 0.023100473, 0.0038135506, 0.0053477376, -0.028068317, -0.024094041, 0.0007474596, 0.017869625, -0.0017232208, -0.010417861, 0.011367596, -0.0057860767, 0.029251833, -0.0714785, 0.012762975, -0.021084113, 0.0033806907, -0.0008054482, 0.078550376, -0.03404434, -0.02730853, -0.05385727, -0.048509534, 0.012952922, 0.057276316, -0.027001692, 0.008839839, -0.036440596, 0.013150174, 0.0145236375, -0.029748617, 0.007926633, 0.014501721, -0.024210932, 0.05385727, -0.022267628, 0.061192144, 0.048772536, 0.065692425, -0.0041240407, -0.018147241, -0.06417285, -0.026694855, -2.6939593e-05, 0.015312647, 0.039216742, 0.02989473, 0.04061943, 0.022457575, 0.036440596, 0.012346553, 0.048217308, -0.04155455, 0.022048458, 0.033226106, 0.055435292, 0.038603067, 0.020528883, 0.03594381, 0.027250083, -0.08123886, -0.005154138, -0.02554056, -0.041349992, 0.0025788953, 0.015312647, -0.0056691864, 0.050964233, -0.06212727, -0.0120689375, 0.047340628, 0.054354053, -0.022340685, -0.018439466, -0.018965473, 0.0078097424, 0.07545278, -0.034570348, -0.054704726, 0.017781958, -0.004335905, -0.045821052, -0.011857074, -0.031297415, 0.0075832675, -0.056691863, -0.037697166, -0.00929279, -0.016364662, -0.011075369, -0.015707154, -0.0059029674, 0.056604195, 0.00873756, 0.0028857328, 0.05978946, -0.017489731, 0.11057836, 0.0058700917, 0.026563352, -0.042928014, -0.018220298, 0.018790139, -0.008291916, 0.036528263, 0.013639653, 0.010673558, 0.014596694, -0.004376086, 0.044739816, -0.0028528573, -0.0035962074, -0.039362855, -0.030742185, 0.0051577906, 0.035300914, 0.027586143, 0.01700756, -0.037697166, 0.0030263665, 0.014596694, 0.064523526, 0.060958363, -0.020777276, 0.06990048, 0.008861757, 0.008065441, 0.020894166, 0.012646085, -0.00962885, -0.0120689375, 0.007992384, -0.01858558, 0.017781958, -0.059701793, -0.015298036, 0.015298036, 0.01927231, 0.0345119, 0.014509026, -0.020338936, -0.002511318, 0.020338936, 0.028638158, -0.00895673, 0.025905844, 0.024678495, -0.012112772, 0.02855049, -0.00638514, -0.03781406, 0.010995007, 0.013588513, -0.017489731, 0.031589642, 0.012025104, 0.022939749, 0.030099288, 0.03649904, -0.0032437097, 0.011572153, 0.0028272874, 0.0217124, 0.0033350303, -0.01971065, -0.04713607, -0.053243596, -0.020324325, -0.002303107, 0.029251833, -0.029076496, 0.0057678125, 0.03871996, -0.04623017, -0.04038565, -0.007758603, 0.016803, -0.031735756, -0.031618863, -0.0011049799, 0.008372278, -0.002966095, 0.007846271, 0.016627666, -0.011111897, 0.057247095, 0.018746305, 0.0141948825, 0.014808558, 0.03582692, -0.02809754, 0.014034159, -0.03804784, 0.003329551, 0.022442965, -0.025496727, -0.051694795, 0.0039340938, -0.039041407, 0.0037879809, -0.026840968, -0.00021426109, -0.023436533, -0.022150738, 0.019111587, 0.0034099133, 0.0035907282, 0.0039413995, 0.00528564, -0.005168749, -0.0039085243, 0.004631784, 0.012836032, 0.039655082, 0.00023789032, 0.023100473, 0.017592011, -0.0062755556, -0.0039815805, -0.018950863, -0.03123897, -0.027819924, 0.027819924, -0.0077951313, -0.032407876, 0.0023268503, 0.040151868, -0.018833973, 0.01746051, 0.023085862, -0.02117178, 0.00747003, 0.031589642, -0.039129075, -0.0074115843, 0.020455828, -0.0035779434, -0.00024062993, -0.018600192, 0.027337752, -0.028565101, -0.026753299, 0.01870247, -0.018556358, 0.028725825, -0.027030915, -0.009102843, 0.039012186, -0.011491791, -0.012083549, 0.007316611, -0.039713528, -9.5030555e-05, -0.019184643, -0.004357822, -0.014026853, 0.0017926245, 0.030654518, -0.022048458, 0.019783707, 0.009022481, -9.046453e-05, -0.0020254923, 0.011484486, -0.009285484, -0.026490295, -0.0060015935, 0.026402628, 0.036119144, -0.026285738, 0.019330757, 0.027293919, 0.011637905, -0.006118484, 0.029076496, 0.02105489, 0.01277028, -0.007802437, 0.0021533412, -0.0022245713, -0.0020145336, 0.0075175166, -0.019885987, -0.026957858, 0.006253639, 0.010680864, 0.0158971, -0.013449706, -0.045733385, -0.045879498, -0.03693738, 0.08690804, -0.00912476, 0.036002256, -0.018307965, -0.0077951313, -0.024532381, -0.00030181478, -0.037229605, -0.022574466, 0.024897663, -0.04041487, -0.03459957, 0.0040071504, 0.0235242, 0.021639343, 0.030537628, 0.00489844, -0.00828461, -0.04558727, 0.0044491426, -0.002312239, -0.010191386, -0.022194572, 0.006560476, 0.0046537006, -0.0476913, 0.0104324715, 0.017767347, 0.00032921098, 0.017387453, -0.054792393, -0.022223795, 0.0080581345, -0.017387453, -0.015020422, -0.0192577, 0.025277557, 0.0070280377, -0.002648299, 0.0020309715, 0.0071814563, -0.0017232208, -0.002903997, -0.021639343, -0.017986517, 0.022004625, -0.0046756174, -0.07714769, -0.036908157, -0.008525697, 0.0067613814, -0.0029113025, -0.014647833, -0.032846212, -0.013975713, 0.055669073, 0.009957604, -0.047194514, -0.015970156, -0.033518333, -0.008182331, 0.042898793, -0.01791346, 0.013676181, -0.0062755556, -0.00053285604, 0.00056162203, -0.018264132, -0.03305077, 0.025175279, -0.034132008, -0.0075613502, 0.012733752, -0.03950897, -0.0040911655, 0.006312084, -0.022501409, 0.00856953, 0.026972469, 0.0011214176, 0.010819672, -0.014874308, 0.0098041855, 0.021814678, -0.02809754, -0.0021277713, -0.0044710594, -0.016978336, 0.042665012, -0.00794855, 0.019739874, 0.006998815, 0.020134378, 0.009826102, -0.0123684695, 0.03939208, -0.023991764, 0.0125876395, -0.012762975, 0.014012242, -0.009431598, -0.018527135, 0.044739816, 0.005194319, 0.01354468, -0.011703655, 0.00058582204, 0.038661513, 0.012828725, -0.0057020616, -0.016320828, 0.0106224185, -0.01645233, -0.022369908, 0.01393188, 0.0006995162, 0.026183458, 0.025029166, -0.00019462715, 0.013698098, 0.0070097735, 0.0038793015, 0.025175279, -0.010658947, 0.05762699, 0.0012008667, -0.044038475, 0.007627101, 0.018176464, 0.016189326, -0.04208056, -0.010469, -0.0082773045, 0.0072618187, -0.019637594, -0.014326384, -0.053565044, -0.01282142, -0.011382206, -0.0016118096, 0.02799526, -0.03112208, -0.003912177, 0.014151049, 0.0014903531, -0.10549362, -0.0024017333, 0.0028930383, 0.031531196, -0.020938, 0.005530379, 0.026373405, -0.004514893, -0.0036857016, 0.028579712, 0.011871685, -0.033664446, 0.0074955993, -0.010702781, 0.012492666, -0.04021031, 0.014326384, -0.017416677, 0.00046893157, -0.026008124, -0.02742542, 0.015064255, -0.026840968, -0.007802437, -0.027410809, 0.022501409, -0.019316146, 0.011331067, -0.006144054, -0.0045952555, -0.011835157, 0.030800631, -0.012346553, 0.008043524, 0.0037697167, 0.0031158607, -0.005954107, -0.00079996896, 0.0054536695, -0.0029989704, -0.0017524434, -0.016423106, 0.030040843, -0.037667945, 0.045411937, 0.0033715586, -0.015181146, 0.024605438, 0.011426041, 0.0051212623, -0.011886297, -0.007590573, -0.019535314, 0.0017716207, 0.02809754, 0.0030647211, 0.012514583, -0.021098724, 0.018556358, 0.047165293, 0.005252764, 0.039771974, 0.029295666, -0.009891854, -0.010680864, 0.0037879809, 0.0071120528, 0.015780209, -0.027615367, -0.0027085706, -0.038573846, -0.0052089305, 0.005844522, 0.010717392, -0.01700756, -0.015298036, 0.012346553, 0.043717023, -0.026285738, -0.010834282, 0.02172701, -0.0068709664, -0.023305032, 0.022340685, -0.014618611, 0.022413742, -0.0053221676, -0.009380458, 0.014633222, 0.040327203, 0.017562788, -0.039655082, 0.0013533721, 0.018322576, -0.05742243, -0.025058389, 0.005566907, 0.011133814, 0.025788954, 0.008145803, 0.009526571, 0.03492102, -0.03012851, -0.02139095, -0.0035925547, 0.02071883, 0.014808558, 0.0020035752, -0.029047275, 0.012916394, 0.033343, 0.01769429, -0.006352265, 0.018366411, -0.014757418, 0.018118018, 0.037229605, -0.020046711, -0.0045368103, -0.0047596325, 0.015853265, -0.00036048828, -0.0218439, -0.0017588359, 0.004821731, 0.008832534, -0.018381022, -0.012646085, -0.005566907, 0.011900908, 0.024561604, 0.015750987, -0.010768532, -0.011776712, 0.011630598, -0.002524103, -0.01080506, 0.0072179846, -0.02115717, -0.0018510697, -0.013332816, -0.012383081, 0.03401512, 0.019418424, -0.020645775, -0.032203317, -0.020762665, -0.020075934, -0.010878117, 0.029310279, 0.02730853, 0.008467251, -0.022267628, -0.013047895, -0.036557484, 0.0026336878, -0.003762411, -0.0033624265, -0.017299784, 0.026388017, 0.033167664, 0.04108699, 0.037200384, 0.00072828226, 0.014026853, -0.02608118, -0.0072618187, -0.006103873, -0.015108089, -0.0056618806, -0.06002324, 0.013464318, 0.02518989, 0.03492102, 0.055902854, 0.027805313, -0.010366721, 0.010768532, 0.0037551054, 0.052016247, 0.019067753, 0.019184643, -0.01075392, -0.0020181865, -0.0057276315, -0.019228477, -0.019885987, 0.0126972245, 0.020996446, -0.017182894, 0.01277028, 0.04208056, -0.011564848, 0.02283747, -0.032963105, 0.032407876, 0.0015186626, -0.0137054045, -0.034804128, -0.027819924, -0.030742185, 0.042197447, -0.0057714656, -0.0027834536, 0.019579148, 0.025350614, -0.0076636295, 0.014092604, 0.0007912935, -0.0069695925, -0.02955867, -0.029500226, -0.013939185, -0.001370723, -0.04018109, 0.008942119, 0.0017268736, -0.0069038416, -0.01689067, -0.012675307, -0.047369853, -0.04748674, -0.007875494, 0.04275268, -0.009227039, -0.013771155, 0.03369367, -0.008043524, -0.00017487905, 0.0049714968, -0.041583773, 0.00035135623, 0.0004312618, 0.03135586, -0.027834537, 0.042226672, -0.03313844, -0.011294539, -0.009979521, 0.012141994, 0.047399074, 0.021551674, 0.0026592575, -0.024444714, 0.02529217, -0.022998193, 0.019199254, 0.016773779, -0.0073933206, -0.012025104, -0.019447647, 0.022793636, 0.0020218394, 0.005990635, 0.009000564, -0.029500226, 0.022559855, 0.025219113, 0.030391514, 0.0036838753, 0.007210679, 0.011382206, -0.020061322, -0.03763872, 0.007634407, -0.012047021, -0.014151049, 0.009446208, 0.021916958, -0.0133766495, 0.024532381, 0.009263568, -0.024985332, 0.03962586, -0.024473935, 0.001923213, -0.00046596365, -0.01927231, 0.023129696, -0.011630598, 0.020105155, 0.022632912, -0.0061696237, -0.002567937, -0.011462569, -0.003156042, -0.015546429, 0.0005862786, -0.008364972, -0.027790703, -0.0046756174, 0.010264441, -0.015210369, 0.0018994696, -0.023012806, -0.030508405, -0.016379273, 0.038632292, 0.011521014, -0.036440596, -0.022121515, -0.01915542, -0.0016611228, -0.014764723, -0.0099137705, 0.02922261, 0.044973597, -0.048655644, -0.027396196, 0.0068782717, -0.010249831, 0.042694233, 0.0114698745, 0.014764723, -0.0019834847, -0.036878932, -0.0035048868, 0.005187013, 0.0063887932, -0.023801817, -0.040911656, 0.0052710283, -0.02989473, 0.04424303, 0.0070937886, -0.00025820915, 0.008153108, 0.0032564946, -0.01971065, 0.003280238, 0.0032290984, -0.008533002, 0.03831084, -0.008978647, -0.020806499, 0.028345931, -0.025160667, 0.017153673, 0.0071120528, 0.013354733, -0.0024967068, -0.01500581, -0.00056207867, 0.0044929762, -0.0012912741, 0.0008894632, 0.009899159, -0.01779657, 0.018950863, 0.015604874, 0.08655737, -0.031297415, -0.032846212, -0.016773779, -0.0018556358, -0.004408961, -0.013369344, -0.022720579, -0.03144353, -0.0059394953, -0.017665068, 0.028681992, 0.00035911848, 0.018381022, 0.016525386, -0.012295413, 0.0028802536, -0.0012794024, 0.055435292, 0.013274371, 0.020572718, 0.055435292, 0.001198127, -0.0058847032, 0.01393188, 0.019418424, 0.026329571, -0.0014291683, 0.0019871376, 0.032875437, 0.0079777725, -0.044096917, -0.023480367, 0.01169635, -0.023904094, -0.00079494633, 0.014246022, 0.025935067, -0.027060136, 0.021347117, -0.02787837, -0.0012757495, 0.026519518, 0.00094060274, 0.01734362, 0.009855325, 0.0026062916, 0.015838655, 0.010870811, -0.017606623, -0.028521268, 0.0059175785, -0.0029076496, -0.017504344, 0.026358794, -0.05102268, 0.012324636, 0.018161852, 0.012711835, 0.0051431796, -0.032086425, 0.01924309, -0.02730853, 0.02114256, -0.0024875747, -0.060490802, -0.0053112092, 0.0037879809, 0.006958634, 0.03427812, -0.02888655, -0.010797755, 0.01416566, -0.0010775838, 0.021464007, -0.00043902406, -0.017153673, -0.0018026697, 0.038603067, 0.022384519, -0.013654265, -0.024108654, -0.026490295, -0.025701286, 0.009731129, 0.017445898, 0.049269322, 0.008204248, -0.04257734, 0.020338936, -0.027337752, 0.034862574, 0.04827575, 0.04792508, 0.024707716, -0.008518391, -0.0020693261, -0.010257136, 0.0032510154, -0.014238717, 0.028448211, 0.009007869, -0.028258264, 0.009175899, -0.019564537, -0.010980396, -0.036995824, 0.0012565722, 0.00929279, 0.016525386, 1.4204643e-05, 0.02159551, 0.008657198, -0.0061842347, -0.006418016, 0.02128867, -0.003311287, 0.024444714, -0.006732159, -0.022486798, 0.0153418705, 0.012850643, -0.0116598215, -0.011484486, -0.0035907282, -0.024269378, 0.036294483, 0.0029460043, 0.0053733075, 0.0138296, -0.010563973, -0.0010008744, 0.022881303, -0.011732878, 0.0317942, -0.0071120528, -0.020908777, -0.012047021, -0.017679678, 0.037200384, 0.0026903066, 0.011637905, -0.042957235, -0.018278742, -0.0068198266, 0.005636311, 0.008847145, 0.007692852, 0.011586765, 0.021098724, -0.023889484, -0.034336567, 0.021814678, -0.019681428, -0.0038610373, -0.03582692, -0.015049644, 0.011126509, -0.008657198, -0.016174715, 0.036265258, -0.004518546, -0.03194031, 0.016087048, 0.026504908, 0.0073275696, -0.026972469, -0.00012122817, 0.03635293, 0.022676745, -0.02159551, 0.010395943, -0.012551111, 0.0029642684, 0.0032510154, -0.021303283, 0.024663882, -0.011214176, -0.004854606, 0.019301534, -0.010154857, -0.03123897, 0.0013104514, -0.020017488, -0.017898848, 0.023772594, 0.01393188, -0.0046756174, -0.00979688, 0.024590826, 0.006973245, -0.019988265, -0.012200439, 0.005592477, 0.01935998, 0.0072910413, -0.02720625, -0.0058700917, 0.016013991, 0.016218549, 0.006253639, 0.006202499, 0.01668611, 0.022179961, -0.024956109, 0.01248536, 0.0035615056, -0.0031779588, -0.013975713, 0.00979688, -0.032057203, 0.0055048093, -0.032846212, -0.011024229, 0.021405563, -0.018877806, 0.012689918, 0.001030097, -0.0055632545, -0.00095338764, 0.011652516, 0.0076417127, 0.023568034, -0.0020766316, -0.0376095, 0.013172092, 0.01824952, -0.033167664, 0.017168283, 0.0070134266, -0.0075759618, -0.03348911, -0.007933939, -0.0012419609, 0.0041532633, -0.0012739232, -0.0037733696, -0.014903531, -0.009526571, 0.0029058233, -0.0074298484, 0.034804128, 0.017723514, 0.01689067, -0.008810617, 0.064523526, -0.013033284, -0.0018254999, -0.010140246, -0.017898848, 0.025788954, 0.030362291, -0.030274624, -0.015531817, -0.024123264, -0.016116269, -0.011586765, -0.008598753, -0.020850332, 0.03804784, -0.01656922, -0.008240776, 0.0062353746, 0.0023195448, -0.009351235, 0.017314397, 0.019286923, -0.025613617, -0.011586765, 0.013902657, -0.010366721, -0.0077293804, -0.03591459, -0.028506655, 0.017109837, 0.02809754, -0.0060088993, 0.010162163, 0.011236094, -0.011228788, 0.014077992, -0.03214487, -0.017855015, 0.0024784426, -0.029529447, -0.0022446618, 0.007020732, -0.0051212623, -0.020017488, -0.010498223, 0.026592575, 0.024327822, 0.017781958, -0.007071872, 0.026110401, -0.014377524, -0.01298945, -0.014830475, -0.011710961, 0.016773779, -0.041671444, -0.015429538, 0.01169635, -0.009869937, -0.0004958712, 0.0043212934, -0.010783143, 0.05791921, -0.010096412, -0.019096976, 0.031005189, -0.00979688, 0.004664659, 0.0046938816, 0.033343, 0.013698098, 0.013603125, -0.012054327, -0.040531762, -0.02206307, -0.016393885, 0.040297978, 0.016613053, 0.0012994929, 0.0068782717, -0.017036783, -0.0026446462, 0.007090136, 0.028287487, -0.0073421807, 0.021127947, 0.022384519, 0.014151049, 0.013836906, -0.0007237162, -0.014399441, 0.0036984866, -0.02283747, 0.02575973, -0.007006121, 0.013427789, -0.0074809883, -0.02955867, 0.0166715, -0.024488548, 0.047662076, -0.036469817, 0.01824952, -0.015444149, -0.002334156, 0.0038793015, 0.006874619, -0.000499524, -0.05695487, -0.040823985, -0.011849768, 0.0076709352, 0.01914081, 0.03237865, -0.024780773, -0.059701793, 0.0113602895, 0.025803564, -0.0055413377, -0.024488548, 0.01824952, -0.0029496572, -0.03381056, -0.015911711, -0.014267939, 0.0055596014, 0.0061732763, 0.013069812, -0.018307965, -0.03950897, -0.005574213, -0.022048458, -0.0035395885, 0.010016049, -0.021873124, 0.025569784, -0.028024483, 0.055259954, -0.014034159, 0.032963105, -0.01656922, -0.029529447, 0.022004625, -0.0015688889], "content_hash": "43ebc430ddce5bcd25705c23092396a5"}, {"content": "1. Look at **Stage 5 (Output)** to see the top-5 predictions for the immediate next token. 2. Note the top prediction and its probability. Are the alternatives close in probability? If so, the model was uncertain, which explains why beams diverge early. ### Step 5: Ablate a Head and Re-Generate\n\n1. Go to the **Ablation** tab in the Investigation Panel.\n2. From the head categories in Stage 3, pick a **Previous-Token** head (e.g., L0-H3).\n3. Add it and click **\"Run Ablation Experiment.\"**\n4. Look at the **Full Generation Comparison**:\n   - Did the ablated generation diverge from the original?\n   - Did the model take a completely different path, or just change a word or two?\n   - Did the ablated generation match one of the other beams you saw earlier?\n\n### Step 6: Try a Stronger Ablation\n\n1. **Clear** the selected heads.\n2. Add **two or three** Previous-Token heads from different layers.\n3. Run the ablation again.\n4. Compare: does ablating multiple heads cause a bigger divergence than ablating one?\n\n### Step 7: Experiment with Different Beam Settings\n\n1. Change the prompt to: `The scientist discovered that the`\n2. Try with **1 beam** (greedy decoding): note the single output.\n3. Try with **3 beams**: see the alternatives.\n4. Try with **5 beams**: do you get even more diverse options?\n\nNotice how increasing beams reveals the model's uncertainty -- places where multiple continuations are roughly equally likely.\n\n## What You Should Learn", "source_file": "experiment_beam_search.md", "chunk_index": 1, "embedding": [0.039442945, 0.013578846, 0.05983058, -0.0073781237, -0.03699519, 0.008063651, -0.0016712128, 0.02311425, 0.016235746, 0.059706643, 0.027622456, -0.02976037, -0.06562464, -0.02407476, 0.015406917, -0.023362122, 0.014345706, -0.016065333, -0.02765344, 0.08192235, -0.02895478, -0.039195072, 0.022153737, 0.031170152, 0.026476039, -0.027312614, 0.014694279, 0.047870662, 0.023005804, -0.012385953, -0.0018832613, -0.026150705, -0.026538007, -0.011688807, -0.056856092, 0.003336616, 0.006258818, 0.03093777, -0.018962359, 0.009783276, 0.02523667, -0.034175623, -0.042045623, 0.03349397, 0.0131915435, 0.011944427, 0.012819733, -0.05165074, -0.010255786, 0.03742897, 0.014694279, 0.05484212, -0.026816865, -0.003993095, -0.035291057, -0.005871515, -0.02079043, -0.041456923, 0.014903423, -0.05165074, 0.005770816, -0.00087433675, 0.03165041, 0.02224669, 0.0016896097, -0.0062781833, -0.032440506, 0.042045623, 0.0071070115, 0.039257042, -0.0011851474, -0.013695037, 0.010782518, 0.012432429, -0.025794385, -0.0067042164, 0.053757668, -0.0030674406, -0.016808955, -0.008435462, 0.0458257, 0.0068397727, 0.050318416, -0.034392513, 0.022664977, -0.042138577, -0.0005102718, -0.035353027, -0.060667157, -0.06488101, -0.020589033, 0.00031008455, -0.05143385, 0.012354969, 0.013911927, -0.0059877057, 0.031603932, -0.050287433, 0.025159208, 0.0069249794, 0.01579422, 0.014384436, -0.016576571, -0.0010205436, 0.06299098, 0.0064679617, -0.044524364, 0.014345706, -0.03149549, 0.0022405484, -0.04762279, -0.017382164, -0.022819899, 0.008714319, -0.012889447, -0.03860637, -0.051371884, 0.037459955, 0.032440506, -0.01473301, -0.054408338, -0.014384436, -0.0024535651, 0.020914366, 0.01736667, -0.01528298, 0.07826621, -0.03715011, -0.0066151368, -0.0010186072, 0.0023005803, -0.004120905, 0.03575582, -0.024245173, -0.028939286, 0.022850882, -0.018946867, 0.03931901, -0.05332389, 0.041178066, -0.009233305, -0.02057354, -0.00032364015, -0.011712045, -0.028521, 0.029589957, -0.0043377946, 0.044028614, -0.038885232, 0.019644013, 0.032626413, -0.013036622, -0.013625323, -0.026212674, 0.0061968495, -0.033648893, -0.041456923, 0.023548028, -0.030178657, -0.04539192, -9.234758e-05, 0.043966647, -0.017738482, -0.0008351223, -0.03115466, 0.053602748, -0.02560848, 0.03553893, -0.036499444, -0.038854245, -0.021131257, -0.011146583, -0.0059257373, -0.0040860474, 0.0010767026, 0.003144901, -0.0938203, -0.074238256, -0.031464502, -0.057723653, -0.0010466865, 0.033060193, -0.01736667, -0.0041828733, -0.012571858, -0.025701433, 0.003152647, -0.0028040744, -0.017645529, -0.011804998, -0.013222528, 0.021905864, 0.062278338, -0.008458699, -0.030070212, -0.023176217, 0.06413739, 0.028613953, 0.010193817, -0.010178326, 0.037181098, -0.0017331813, 0.03801767, -0.006890122, 0.016142793, 0.008234064, 0.01846661, 0.014252754, -0.036220584, -0.009883975, 0.028722398, -0.040651333, 0.031975742, 0.01852858, -0.044152554, -0.004759955, -0.0022347388, 0.050814167, 0.021069288, 0.049853653, 0.035910744, -0.010681819, 0.027498519, 0.071139835, -0.0033869655, 0.005867642, 0.05326192, -0.024647968, 0.04567078, 0.022974819, 0.03532204, 0.009155845, -0.00691336, -0.01371053, -0.0005611054, -0.032285586, -0.0132302735, -0.028087221, 0.02413673, -0.06091503, -0.015042852, 0.012982399, 0.037367, -0.008636859, -0.026026769, 0.04610456, -0.03882326, -0.027529504, 0.0077499347, 0.034299564, 0.039659835, -0.015856188, -0.022494564, -0.008249556, 0.0066809785, -0.008396731, -0.043439914, -0.0059605944, -0.040434442, -0.015515362, -0.0114486795, 0.020914366, -0.05267322, 0.01640616, -0.04297515, 0.014322468, -0.01579422, -0.011758522, -0.040372476, -0.06500495, -0.024105744, -0.036220584, -0.0042758263, -0.008319271, -0.04436944, -0.029698402, -0.018621532, -0.00081382063, -0.02391984, -0.0009726149, -0.012029634, -0.017041337, 0.035414994, -0.010867725, -0.0012074173, 0.031170152, -0.009907213, -0.01360983, -0.0052092266, -0.029419543, 0.03699519, -0.03876129, 0.038637355, -0.03318413, -0.008288286, 0.001566641, 0.011595855, 0.013695037, -0.06735975, -0.0037064906, 0.018621532, -0.03237854, 0.050225466, 0.002521343, -0.016096316, 0.0012529255, 0.00426808, 0.02378041, 0.00094647193, 0.00038270387, 0.02311425, 0.039349996, -0.02771541, -0.0434709, -0.022757929, -0.043223023, 0.04691015, -0.051774677, 0.024322635, -0.05041137, -0.022742437, -0.026909819, -0.01331548, 0.019148264, 0.0019374837, -0.01276551, -0.01221554, -0.030287102, 0.008466446, -0.028335094, 0.021348147, 0.009605116, 0.041116096, 0.0019248964, -0.058405306, -0.01618927, -0.0048374156, 0.03922606, 0.037026174, 0.02991529, -0.054191448, 0.014802724, 0.02641407, -0.021859387, -0.011386711, 0.024570508, 0.0028795984, -0.007738316, 0.030596944, 0.037274048, -0.03420661, -0.022262182, -0.008358001, -0.012912685, 0.044028614, 0.0479946, 0.024260666, -0.03392775, 0.035291057, 0.00022608817, -0.02472543, 0.06128684, 0.033741847, 0.03268838, 0.019349663, -0.020635508, -0.011611347, -0.00015334782, 0.004217731, -0.0055384343, -0.027823854, 0.0025891212, -0.021673482, 0.028195664, -0.097662345, -0.0059489757, 0.0006322723, 0.023904346, 0.006030309, 0.072627075, 0.0058250385, -0.038916215, -0.053664718, -0.051247943, 0.019752458, 0.035043184, -0.02633661, 0.0070566623, -0.03603468, 0.0013090845, 0.02429165, -0.0123394765, 7.606875e-05, 0.00601869, -0.007753808, 0.03990771, -0.035848774, 0.057227906, 0.030287102, 0.053107, 0.019783441, -0.024709938, -0.060976997, -0.029899798, 0.0096670855, 0.026383087, 0.045113064, 0.011200805, 0.051991567, 0.023656473, 0.041673813, 0.007633744, 0.05564771, -0.031464502, -0.0024264539, 0.038110625, 0.040744286, 0.030333579, 0.02429165, 0.04427649, -0.021162242, -0.0735566, -0.030116688, -0.046507355, -0.029435035, -0.027002772, -0.020620016, -0.011386711, 0.06265015, -0.044121567, 0.0098917205, 0.056949046, 0.04356385, -0.013656307, -0.009945943, -0.008350255, 0.029450526, 0.048397396, -0.045918655, -0.029450526, 0.022943836, 0.00020406031, -0.027095724, -0.032440506, -0.05121696, 0.019008836, -0.06618235, -0.04173578, 0.0030868056, -0.03879228, -0.029589957, -0.0361896, -0.025081748, 0.054377355, -0.0052285916, -0.04325401, 0.09059794, -0.0084044775, 0.08458699, 0.00023322908, 0.022432595, -0.028846335, -0.017939879, -0.0017709434, -0.01846661, 0.041054126, 0.01969049, -0.030395547, 0.018869407, 0.024833875, 0.020976335, -0.028242141, -0.02299031, -0.050752196, 0.007629871, -0.0006588994, 0.043099087, 0.0026220419, 0.030767357, -0.031820823, -0.013493639, 0.00036842207, 0.039133105, 0.04223153, -0.0040783016, 0.08626015, 0.004798685, -0.00798619, 0.003189441, -0.01721175, 0.008853748, -0.04669326, 0.026491532, -0.020960843, 0.062216368, -0.028397063, -0.042572357, 0.0007983285, 0.009155845, 0.006804915, 0.057537746, -0.020449603, -0.01618927, 0.018141277, 0.03312216, -0.017645529, 0.03495023, 0.035848774, -0.010852233, 0.066864006, -0.019876394, -0.037119128, -0.00060467696, 0.005410624, -0.027483027, 0.014105579, -0.0018309754, 0.036437474, 0.022556532, 0.028505508, -0.020186238, -0.011975411, 0.007839015, 0.030906787, 0.014887931, -0.032006726, -0.036685348, -0.036902238, -0.021363638, 0.010364231, 0.035198104, -0.04185972, 0.0029241384, 0.024555016, -0.040961176, -0.047591805, -0.01626673, 0.021177733, -0.024586, -0.01648362, -0.0015869744, 0.029047731, 0.0017854673, 0.0009614799, 0.0072967904, -0.0066654864, 0.03349397, 0.0062084687, 0.007304536, 0.0029919164, 0.012657065, -0.007134123, 0.0026530263, -0.03829653, -0.001510482, 0.0044849697, -0.025159208, -0.025469052, -0.0028641063, -0.028288618, 0.021503069, -0.012308492, -0.005023321, -0.013307734, -0.016746985, 0.013013384, -0.009636101, 0.0050620516, 0.004655383, 0.005151131, -0.024369111, -0.029063225, 0.004798685, 0.023300154, 0.0066345017, 0.008086888, 0.030875802, 0.021719957, 0.014469643, -0.0030345197, -0.014043611, -0.039659835, -0.04093019, 0.01280424, -0.029202653, -0.029698402, 0.0014630374, 0.030720882, -0.013323226, 0.014105579, 0.035879757, -0.012943669, 0.015205519, 0.024926826, -0.016808955, -0.008644605, 0.015143551, 0.0063091675, 0.016592065, -0.01105363, 0.031108184, -0.008342508, -0.021998815, 0.015368187, -0.03721208, 0.011378964, -0.016623048, 0.0022967074, 0.031696886, -0.0037181096, -0.01838915, 0.009791022, -0.053137984, -0.024616985, 0.0024845493, -0.014903423, -0.0132690035, 0.014275992, 0.04777771, -0.013842212, 0.0077228234, 0.0058289114, -0.009651593, 0.015406917, 0.017599052, 0.0029125193, -0.014562597, -0.016111808, 0.013772498, 0.022479072, -0.038885232, 0.01502736, 0.013400687, -0.0023373743, -0.013408433, 0.016468128, 0.0052440837, 0.029791353, -0.018698994, -0.011766268, 0.0016876733, 0.016142793, 0.012819733, -0.008838257, -0.036964208, 0.0054609734, 0.012347222, -0.00119483, -0.025840862, -0.048552316, -0.03259543, -0.033308066, 0.06661613, -0.009031908, 0.020480588, -0.013950657, -0.0029861068, -0.028644936, 0.0025310258, -0.018079309, -0.011518394, 0.025407083, -0.031619426, -0.03296724, -0.0045663035, 0.03457842, 0.01618927, 0.03938098, 0.00700244, 0.0025174702, -0.021053797, 0.024353618, -0.021921355, 0.016592065, -0.040372476, 0.0025310258, 0.007467204, -0.033091176, -0.00995369, 0.012657065, 0.0011715918, 0.01567803, -0.06990046, -0.040713303, -0.0004831606, -0.007924221, -0.0001341037, -0.031092692, 0.0030364562, -0.0059257373, -0.023842378, -0.008846003, 0.009729054, 0.0126725575, -0.003489601, -0.01105363, -0.007242568, 0.02976037, -0.0057165935, -0.06723582, 0.00086465414, -0.00849743, 0.008388985, -0.018079309, -0.016111808, -0.01919474, 0.0025484543, 0.061100937, 0.021859387, -0.017149782, -0.004368779, -0.015631553, -0.021146748, 0.020852398, -0.041023143, 0.012695795, -0.0049226223, -0.008621367, -0.009016416, -0.04142594, -0.018373659, 0.028180173, -0.035848774, -0.0034624897, 0.018652517, -0.031820823, -0.012440176, -0.000913067, -0.021069288, 0.01174303, 0.01127052, 0.016313206, 0.01272678, -0.022432595, -0.0025348987, 0.030240625, -0.01882293, -0.0004165929, -0.008985432, -0.01882293, 0.03959787, -0.020743953, -0.0009953689, 0.022091769, 0.024152221, -0.0032029964, 0.009055146, 0.016901907, -0.028753381, -0.015763236, -0.017242733, 0.0155308535, -0.0042022387, -0.01589492, 0.05326192, 0.006901741, 0.010023404, 0.000856424, 0.01557733, 0.0117043, 0.024028284, 0.0060845315, 0.004512081, 0.022277674, -0.012385953, 0.0085903825, -0.0132690035, 0.021952339, 0.0110149, 0.03420661, -0.010712803, 0.019814426, 0.017753974, 0.010581121, 0.033958737, -0.009395973, 0.037088145, -0.0153062185, -0.051371884, 0.0029512495, -0.02370295, -0.00477932, -0.03820358, 0.021503069, -0.015925903, 0.00585215, -0.017521592, -0.016437143, -0.022262182, -0.028164681, -0.0026685183, 0.005917991, 0.020232713, -0.020883383, -0.010720549, 0.02079043, -0.0059218644, -0.11959919, -0.0068010422, -0.0008007491, 0.008884733, -0.01998484, -0.023548028, 0.03473334, -0.0018542135, -0.017599052, 0.027157692, -0.013849959, -0.032564443, -0.0010892899, -0.00268982, 0.013973895, -0.033215113, 0.0033540449, -0.023021296, -0.018791946, -0.027839346, -0.030070212, 0.010325501, -0.0064369775, -0.009016416, -0.04675523, 0.020108776, -0.022076277, -0.011084614, 0.003580617, -0.004120905, -0.005522942, 0.042789247, -0.017970864, 0.034454484, 0.010488167, -0.015941394, -0.011750775, -0.002641407, 0.0059528486, -0.0017999911, 0.018327182, -0.029311098, 0.025221176, -0.031247614, 0.019210232, -0.01422177, -0.029047731, 0.018420136, 0.034299564, -0.019736966, -0.016375175, -0.015228758, -0.0107437875, 0.0045624305, 0.034454484, 0.012161317, -0.00900867, 0.005623641, -0.005038813, 0.029218145, 0.023377614, 0.046507355, 0.03377283, -0.010217056, 0.00060080393, -0.024338126, 0.029961767, 0.014252754, -0.035848774, -0.03575582, -0.018063815, 0.0017264035, -0.0076531093, 0.01473301, -0.00033840607, 0.01455485, 0.039721806, 0.029342081, -0.006061293, -0.018187754, 0.0277309, 0.009798768, -0.02480289, 0.027235154, 0.0023083263, 0.008141111, -0.006502819, -0.017087812, 0.020263698, 0.033091176, 0.027390074, -0.020511571, -0.0014756247, 0.018729977, -0.063207865, -0.02763795, -0.0009106464, -0.0017080066, 0.021750942, 0.0053254175, 0.0155308535, 0.035074167, -0.017939879, -0.017413147, 0.00060467696, 0.022711454, 0.0018958487, 0.007137996, -0.01961303, -0.0039214436, 0.04040346, 0.010619851, -0.005170496, 0.025716925, -0.033741847, 0.02830411, 0.026832359, 0.011045884, -0.009883975, -0.013346464, 0.0058753877, 0.004167381, -0.027188677, 0.010952932, -0.00537964, 0.002869916, -0.009504418, 0.0023238186, 0.002825376, 0.008807273, 0.0058908802, 0.016716002, -0.012323984, 0.022386119, 0.0054648467, -0.008303778, 0.006254945, 0.004407509, -0.023749426, 0.019783441, -0.021611512, -0.0052440837, 0.02735909, -0.002957059, -0.009798768, -0.011866966, -0.041332986, -0.021936847, 0.0033327432, 0.027018264, 0.04254137, 0.014942153, -0.013137321, 0.0048296694, -0.026197182, 0.019458108, 0.02269596, -0.0063091675, -0.008079142, 0.040062632, 0.042510387, 0.026909819, 0.028877318, -0.018234229, 0.012866208, -0.0060651666, -0.005476466, -0.02378041, -0.0029144557, -0.0013962276, -0.050752196, -0.002318009, 0.017165273, 0.031619426, 0.044338457, 0.021998815, -0.0107050575, 0.016654033, -0.0050349403, 0.04399763, 0.031216629, 0.026150705, -0.021673482, -0.0021863258, 0.018714486, -0.01852858, 0.006495073, 0.025995784, 0.029125193, -0.010736042, -0.0017118796, 0.03764586, -0.009380481, 0.029589957, -0.019287694, -0.0021921354, 0.0076221246, 0.026739405, -0.033215113, -0.014717517, -0.030968755, 0.015104821, -0.0032591554, 0.00029289795, 0.009426957, 0.011518394, -0.017010352, 0.020465095, -0.0012500207, 0.011549378, -0.03129409, -0.02371844, 0.0065996447, -0.005499704, -0.03172787, -0.0038013798, 0.012904939, -0.004368779, -0.0050349403, -0.01502736, -0.034485467, -0.038110625, -0.0031507106, 0.026026769, -0.0012083857, -0.00058143877, 0.036809284, 0.0027208042, -0.0046979864, -0.003996968, -0.01801734, -0.0103719765, -0.0005833753, 0.010333247, -0.029435035, 0.029713893, -0.049048062, -0.0073626316, -0.026646452, 0.013431671, 0.05608149, -0.006510565, -0.02721966, -0.01626673, -0.0013265131, -0.015956888, 0.00040836268, 0.014438659, -0.01360983, -0.024678953, -0.013307734, 0.008125619, 0.023749426, 0.003257219, 0.008195333, -0.020356651, 0.020294681, 0.021022812, 0.03420661, 0.0073316474, 0.013346464, 0.019427123, -0.01506609, -0.026956296, 0.0048722727, -0.020248206, -0.015050598, 0.009605116, 0.036592398, -0.027839346, 0.011386711, 0.029373066, -0.004643764, 0.029357575, -0.022664977, -0.0061658653, 0.009612863, -0.02924913, 0.01247116, 0.00011092603, 0.021936847, 0.0005615895, -0.028582968, -0.005399005, 0.0025736291, -0.0034063307, -0.014779486, 0.007292917, -0.0007547569, -0.022200214, -0.002141786, 0.0096670855, -0.008613621, 0.0011105916, -0.009326259, 0.0008786939, -0.021565037, 0.03275035, 0.033215113, -0.06897093, -0.0050504324, -0.003433442, -0.0003388902, -0.020542556, -0.002205691, 0.012401445, 0.046073575, -0.050225466, -0.030953264, -0.0058444035, -0.0021379131, 0.033834796, 0.010852233, 0.015190028, -0.005755324, -0.043935664, -0.0015337203, 0.0081101265, 0.0054222434, -0.022076277, -0.033277083, -0.0022076275, -0.012966908, 0.030008243, -0.015430155, -0.026104229, 0.004488843, 0.002577502, -0.0045546843, 0.02079043, 0.020743953, -0.001826134, 0.028567476, -0.0033579178, -0.015832951, 0.030085703, -0.034454484, 0.0074246004, -0.01567803, -0.012362715, 0.0081101265, 0.028521, -0.0094811795, 0.004411382, -0.0019694362, 0.00018324278, 0.014245008, -0.011177567, 0.016468128, 0.0010941313, 0.09729054, -0.037831765, -0.050752196, -0.02181291, -0.002957059, -0.009876229, -0.00024266959, -0.023315646, -0.0389472, -0.004752209, -0.005495831, 0.024864858, 0.0024671208, 0.02071297, 0.021611512, -0.00601869, 0.01911728, -0.0073122825, 0.039690822, -0.0059102452, 0.021394623, 0.04734393, -0.013330973, -0.0041828733, 0.007656982, 0.017196257, 0.013098591, 0.0036387125, -0.004639891, 0.048397396, -0.029682908, -0.032874286, -0.032068696, 0.018280705, -0.017134288, -0.0071689803, 0.009395973, 0.018962359, -0.035198104, 0.019055312, -0.028397063, -0.011286012, 0.01911728, 0.004171254, 0.014539358, 0.00554618, -0.0005887007, 0.005879261, 0.00047517248, -0.024183204, -0.0040356982, -0.0042293495, -0.0056933556, -0.011797252, 0.03377283, -0.043130074, 8.744578e-06, 0.025205685, 0.012657065, 0.004748336, -0.026088737, 0.0016344191, -0.012192301, 0.030473007, 0.0060031977, -0.064819045, 0.00081139995, 0.00060612935, -0.0092952745, 0.048180506, -0.03143352, -0.013989388, 0.011510648, -0.00977553, 0.0111155985, -0.017428638, -0.013090844, -0.007064408, 0.014880185, 0.014237261, -0.019628521, -0.03857539, -0.036437474, -0.031820823, -0.011185314, 0.012502144, 0.05887007, 0.022881867, -0.02625915, 0.011619093, -0.035074167, 0.026507024, 0.045639798, 0.043656804, 0.012393699, -0.035136137, -0.003185568, -0.014415421, -0.00563526, 0.0006990821, 0.014887931, 0.0041790004, -0.03786275, 0.00652993, -0.027761886, -0.028288618, -0.0344235, -0.006398247, 0.0057282127, 0.01174303, 0.010193817, 0.012091602, 0.02159602, 0.0053099254, -0.005143385, 0.029094208, 0.015809713, 0.03693322, -0.013888689, -0.034888264, 0.011975411, 0.02611972, -0.010875471, -0.036313538, -0.002254104, -0.025143716, -0.00798619, 0.017784959, -0.0015908475, 1.7368124e-05, -0.0024477555, 0.0052285916, 0.013361957, -0.0057127206, 0.02342409, 0.0011928935, -0.011812745, 0.009419211, -0.021115765, 0.043966647, 0.004883892, 0.008644605, -0.030906787, -0.039411962, 0.01903982, 0.010069881, -0.0024787397, -0.0025407083, 0.0036561412, 0.01948909, -0.019055312, -0.01567803, 0.01917925, -0.015507616, 0.0020488333, -0.043501884, 0.0006971456, 0.00082059845, 0.0027227406, 0.00055142277, 0.034330547, 0.013478148, -0.040961176, 0.016452635, 0.020403126, 0.0043300483, -0.03705716, 0.0069482173, 0.03823456, 0.002893154, -0.02100732, -0.013423925, -0.0039020786, -0.0054183705, -0.012626081, -0.028366078, 0.014051356, 0.015468885, 0.0069288523, 0.0464144, -0.014918915, -0.0073471395, 0.029775862, 0.01787791, -0.015120313, 0.008303778, 0.005766943, 0.00031758854, -0.007746062, -0.004368779, 0.011262774, -0.01579422, -0.012486652, 0.0011890205, 0.021657988, 0.015368187, -0.024973303, 0.010193817, -0.006777804, -0.0022115007, 0.010488167, -0.0077654268, 0.012323984, 0.0092952745, -0.026739405, 0.043966647, -0.001758356, -0.008017174, -0.0059722136, 0.016839938, -0.018420136, 0.0025794385, -0.01897785, -0.014337961, 0.044555347, -0.024539525, 0.003013218, -0.010674073, 0.00078622525, -0.012633827, 0.017304702, 0.014136563, 0.029125193, 0.012192301, -0.034454484, 0.0072115837, -0.01481047, -0.017041337, 0.017397655, -0.00027619553, -0.0026685183, -0.03640649, -0.022928342, 0.009651593, 0.0002885408, -0.0034295688, -0.02472543, -0.013873197, 0.00019232019, 0.015197773, 0.026088737, 0.005739832, 0.0059373565, 0.019860903, -0.007835141, 0.058901053, -0.010859978, 0.001382672, -0.012966908, -0.013633069, 0.015561839, 0.023145232, -0.025283147, -0.0039659836, -0.028970271, -0.028195664, -0.013640814, 0.0067274547, -0.043315977, 0.015035106, -0.0148259625, -0.0042874455, 0.005143385, -0.0051627504, 0.007873872, 0.007924221, 0.016173776, -0.021719957, -0.020743953, 0.014872438, -0.0049226223, 0.013284496, -0.031170152, -0.031820823, 0.018497596, 0.04458633, -0.0041635083, 0.011247282, 0.026305625, -0.02210726, 0.0066499943, -0.028644936, 0.013346464, -0.006464089, -0.00959737, 0.00930302, 0.005898626, 0.0107050575, -0.009326259, -0.0007315187, 0.022959327, 0.029682908, 0.03420661, -0.0008549716, -0.010402961, -0.004283572, 0.011433187, -0.025918324, 0.00849743, 0.016716002, -0.029698402, -0.017815942, 0.006014817, -0.0010912265, 0.018946867, 0.013385195, 0.0013478148, 0.034640387, -0.00029144558, -0.01699486, 0.027668932, -0.013540116, -0.01502736, 0.0048567806, 0.01910179, 0.018218737, 0.014214023, -0.023609996, -0.028242141, -0.008063651, -0.026661946, 0.019210232, 0.011642331, -0.0050852895, -3.7066116e-07, -0.013811228, -0.010170579, 0.0077809193, 0.01831169, -0.015104821, 0.007854506, 0.019504584, 0.016638542, -0.0023799776, 0.0022521676, -0.02210726, 0.010000166, -0.032068696, 0.012138079, -0.020232713, 0.0018454993, -0.0038343007, -0.0119676655, -0.015569584, -0.030953264, 0.027839346, -0.050721213, 0.0062084687, -0.01838915, 0.026677437, 0.0062975483, 0.012099349, -0.0018774518, -0.038885232, -0.012269762, -0.008505176, -0.0019045629, 0.026088737, 0.005646879, -0.040682316, -0.04783968, -0.007176726, 0.03873031, 0.0053137983, -0.015112567, 0.024260666, -0.008458699, -0.06271212, -0.02751401, 0.007544664, -0.0010912265, 0.027111216, 0.026894327, -0.016204761, -0.009605116, -0.007436219, -0.030488499, -0.007970697, 0.007494315, -0.013973895, 0.030287102, -0.014198531, 0.03597271, -0.012262016, 0.031479996, -0.018482104, -0.020015825, 0.008187587, 0.021270685], "content_hash": "623c172e26f32a3a32e5e31420b6b35a"}, {"content": "**1 beam** (greedy decoding): note the single output. 3. Try with **3 beams**: see the alternatives. 4. Try with **5 beams**: do you get even more diverse options? Notice how increasing beams reveals the model's uncertainty -- places where multiple continuations are roughly equally likely. ## What You Should Learn\n\n- **Beam search reveals model uncertainty**: When the model isn't sure, multiple beams show the different paths it's considering.\n- **Ablation can redirect generation**: Removing important heads can push the model from one beam to another, showing that different attention heads support different generation paths.\n- **More beams = more alternatives**: But beyond 3-5 beams, the additional paths are often low-probability and less interesting.\n- **Generation is a chain**: Each token depends on the previous ones, so a small change early (from ablation or beam selection) can cascade into a very different output.\n\n## What's Next?\n\nYou've now completed the core experiments. Try combining techniques:\n- Run attribution to find which input tokens matter, then ablate the heads that seem to process those tokens\n- Compare how GPT-2 and Qwen2.5-0.5B handle the same prompt with the same beam settings", "source_file": "experiment_beam_search.md", "chunk_index": 2, "embedding": [0.03050498, 0.020913439, 0.03920151, -0.014603606, -0.015319616, -0.021167025, -0.021017857, 0.03693415, 0.017124556, 0.07052692, 0.033652436, -0.011150354, -0.046152767, -0.006276269, 0.0024482298, -0.032280087, -0.013469925, -0.024254816, 0.0078015183, 0.073629625, -0.029162463, -0.019973677, 0.008144606, 0.035323124, 0.031922083, 0.0012539485, -0.005772825, 0.028088449, 0.012082657, -0.01525249, 0.012850875, -0.04120037, -0.022822797, -0.0107848905, -0.07094459, 0.018332822, -0.03230992, 0.01743781, -0.012090116, -0.014163559, 0.03320493, -0.02280788, -0.03753082, 0.036904313, 0.012619664, 0.004195367, -0.015095863, -0.032011583, -0.02009301, 0.036725312, 0.025791252, 0.08144606, -0.022613961, 0.01126223, -0.029818805, -0.008495152, -0.051105164, -0.03687448, 0.014506647, -0.033712104, -0.0063657705, 2.8653774e-05, 0.048628967, 0.042960558, 0.002865902, -0.02750669, -0.042274382, 0.025254246, -0.006376958, 0.036784977, -0.0033991795, 0.028655289, -0.023225551, 0.022032203, -0.01254508, 0.011142895, 0.047167115, 0.0028826834, -0.02793928, -0.03720265, 0.03648664, -0.0035035976, 0.034457948, -0.04770412, -0.0008045782, -0.03750099, 0.013335673, -0.0306989, -0.05698241, -0.040693197, -0.028148117, -0.0007104155, -0.055818893, 0.032906596, 0.011993156, 0.010039047, 0.016050542, -0.0412302, 0.009300662, 0.017497478, 0.019198, -0.002823016, -0.043169394, -0.005802659, 0.058921598, -0.012992586, -0.011016102, 0.014297811, -0.042005878, -0.044273242, -0.050389156, -0.011620235, -0.007689642, 0.014051682, -0.0024314483, -0.008830781, -0.01254508, 0.025000658, 0.046838943, -0.022121705, -0.036755145, -0.0061643925, -0.018899662, -0.003701246, 0.00032374248, -0.016766552, 0.051135, -0.04752512, -0.008122231, 0.0150809465, 0.014036766, 0.02304655, 0.039171677, -0.017885316, -0.04737595, 0.01952617, -0.03278726, 0.025269162, -0.057131577, 0.03824683, 0.016766552, -0.051612336, -0.013499759, -0.012052824, -0.044183742, 0.023315053, -0.023300136, 0.0205256, -0.030818233, 0.0010050235, 0.035382792, -0.024254816, -0.048479795, -0.028192867, -0.008681613, -0.042960558, -0.039947353, 0.05558022, -0.022628877, -0.04555609, 0.014141184, 0.050955996, -0.035860132, -0.009203703, -0.023195717, 0.038843505, -0.0358303, 0.043229062, -0.02974422, -0.041946214, -0.0028435267, -0.006376958, 0.00019496803, -0.006742421, 0.028565789, -0.0017331528, -0.09540824, -0.06300882, -0.042035714, -0.068140216, -0.002246852, 0.008927741, -0.016811302, 0.0073726582, -0.015513535, -0.025925504, 0.0045682886, -0.013335673, 0.0019783487, -0.010934059, -0.011515817, 0.036426973, 0.067603216, 0.011232396, -0.016423464, -0.041826878, 0.056863073, 0.049434476, 0.002142434, -0.024508402, 0.016706884, 0.0021256527, 0.043109726, -0.008607029, 0.021301277, -0.009427456, 0.006168122, -0.00094722066, -0.07374896, -0.016826218, 0.019496337, -0.0359198, 0.040693197, 0.017348308, -0.032906596, -0.0023643223, 0.027536524, 0.036218137, -0.004836792, 0.05596806, 0.02380731, -0.012574914, 0.03502479, 0.0410512, -0.0151033215, 0.035860132, 0.022420041, -0.020823937, 0.03663581, -0.009091826, 0.04018602, 0.011068311, -0.0014450708, -0.00026244353, -0.015162989, -0.021733865, -0.002228206, -0.021569781, 0.011336814, -0.08395209, -0.008778572, -0.0104492605, 0.031862415, -0.019556005, -0.030340895, 0.05757908, -0.04722678, -0.038306497, -0.015692538, 0.022464791, 0.017944984, -0.0057952004, -0.044303074, 0.01687097, -0.0032705218, -0.012507788, -0.022017287, 0.004911376, -0.023881894, -0.02617909, -0.016736718, 0.037888825, -0.028476287, -0.0031306762, -0.04689861, 0.027491774, -0.041021366, -0.028968543, -0.035382792, -0.06068179, 0.0016539069, -0.035084456, 0.016274296, -0.0013192098, -0.03559163, -0.033503268, -0.010091256, 0.005205984, -0.0283122, -0.011187646, -0.03359277, -0.014439521, 0.051552672, 0.0029237047, -0.015453868, 0.008465318, -0.016766552, 0.0022785505, -0.004855438, -0.022882463, 0.033473436, -0.018854912, 0.024776906, -0.0460931, 0.004281139, 0.020152679, 0.014573772, -0.0036956521, -0.038604837, 0.0010656232, 0.008465318, -0.033533104, 0.026566928, -0.023508972, 0.008353442, 0.022151537, -0.039410345, 0.03046023, 0.0012781885, -0.0044675996, 0.013410258, 0.009032159, -0.069691576, -0.035173956, -0.027118852, -0.02152503, 0.030609397, -0.044690914, 0.023881894, -0.04424341, -0.008465318, -0.026924934, -0.017422894, 0.00641425, 0.013716053, 0.014290352, -0.0041506165, -0.031892247, -0.022658711, -0.016811302, 0.0073726582, 0.022330541, 0.04006669, 0.032846928, -0.045406923, -0.06181547, -0.030117141, 0.07195894, 0.054625545, 0.020182513, -0.060801122, 0.0029143817, 0.012358619, -0.016885886, -0.026149256, 0.022300707, 0.031295575, -0.0028211512, 0.051910676, -0.00043165666, -0.014506647, -0.01772123, -0.012477954, -0.013566885, 0.04176721, 0.035054624, 0.009464748, -0.020257097, 0.00039086837, -0.009434914, -0.015558286, 0.0566244, 0.034577284, 0.015767122, 0.013141754, -0.028580705, -0.015841706, -2.8296877e-06, 0.013462467, 0.0073913042, -0.013507217, 0.021614531, -0.007037029, 0.021346027, -0.06993024, 0.0035147853, 0.0043557235, 0.029490633, -0.0066268155, 0.060651954, -0.007197385, -0.026447594, -0.031951915, -0.025403414, 0.0022226123, 0.032339755, -0.0037497259, 0.019496337, -0.030997237, -0.0043035145, 0.0039790724, -0.03720265, -0.008771114, 0.005772825, -0.00084466726, 0.041797042, -0.02243496, 0.043557234, 0.021704033, 0.059398938, 0.020510683, -0.018586408, -0.058354758, -0.030639231, -0.01487211, 0.033413768, 0.05620673, 0.029997807, 0.045973763, 0.020719519, 0.028133199, 0.021346027, 0.039649017, -0.016826218, -0.022464791, 0.019511255, 0.03767999, 0.032518756, 0.028237617, 0.041886546, 0.013059711, -0.07261528, -0.023628308, -0.02304655, -0.025194578, -0.01382793, -0.0077045583, -0.026432676, 0.051105164, -0.051582504, -0.01634888, 0.038425833, 0.037321985, -0.010993727, 0.0026906286, -0.016557716, 0.03207125, 0.05492388, -0.028327119, -0.031086737, 0.03046023, 0.00017469042, -0.040036853, -0.021181943, -0.054744877, 0.0106058875, -0.061994474, -0.03374194, 0.0038783837, -0.030669065, -0.030430395, -0.048897468, -0.032578424, 0.03511429, -0.010971351, -0.026029922, 0.07894003, -0.033413768, 0.08902382, 0.0017601895, 0.029087879, -0.04024569, -0.018079234, 0.01767648, 0.02376256, 0.04689861, -0.0004614904, 0.0016352609, 0.022494625, 0.0055005923, 0.017512394, 0.00046871573, -0.014670732, -0.050568156, -0.0070109246, -0.033294432, 0.018586408, 0.03588997, 0.022882463, -0.030102225, -0.004683894, 0.014118808, 0.04024569, 0.050717328, -0.030609397, 0.09027684, 0.019347169, -0.014543939, 0.016557716, 0.006045058, -0.027402272, -0.020898521, 0.016483132, -0.019660423, 0.05012065, -0.03863467, -0.015841706, 0.0012986992, -0.00551178, 0.024418902, 0.024970824, -0.0018058724, -0.018079234, -0.009039617, 0.02741719, -0.023553723, 0.016214628, 0.02570175, -0.023508972, 0.021480279, -0.0050046067, -0.0538797, 0.002280415, 0.034845788, -0.027626026, 0.029580135, 0.011769403, 0.038425833, 0.025269162, 0.051851008, 0.012321327, -0.008674154, 0.013641469, 0.023792394, 0.01335059, -0.042065546, -0.037709825, -0.067722544, -0.008539903, 0.004598122, 0.02802878, -0.05349186, -0.0064030625, 0.0010050235, -0.060323786, -0.041260038, -0.009084368, -0.008271399, -0.01876541, -0.015990874, -0.009628833, 0.03150441, -0.009091826, 0.0015746611, 0.0044452245, -0.002918111, 0.034726452, -0.0041133245, 0.008114772, 0.007928312, 0.024075814, -0.03368227, 0.024060896, -0.023031633, 0.001672553, -0.0012129273, -0.0053775283, -0.02798403, 0.0024482298, -0.030236477, 0.030579565, -0.008345984, 0.017467644, -0.0107998075, -0.023881894, 0.028595623, -0.0053476947, -0.0016622976, 0.01952617, -0.0149690695, -0.036128636, -0.005921994, 0.013932348, 0.00055099156, 0.009651209, 0.014573772, 0.018407406, 0.030072391, -0.007089238, -0.0005626453, -0.015722372, -0.040902033, -0.0068542976, 0.015140614, -0.027372438, -0.03102707, 0.011769403, 0.02295705, -0.011769403, 0.021241609, 0.030639231, -0.0053812573, 0.003222042, 0.00029204416, -0.037142985, -0.010173299, 0.004717457, -0.0075031808, -0.021002939, -0.013992015, 0.021256527, -0.009725793, -0.03383144, 0.011791779, -0.045824595, 0.025865836, -0.026924934, -0.010188215, 0.014081516, -0.008532444, -0.025075242, 0.0149466945, -0.042453386, -0.022837713, -0.013805554, -0.0054148203, -0.016125126, -0.0058287634, 0.052000176, -0.01591629, 0.015841706, 0.018899662, 0.013932348, 0.0017937525, 0.0013238714, 0.01030755, -0.020868689, -0.02452332, 0.020540517, 0.028148117, -0.034577284, 0.00040438678, 0.0150809465, 0.016572632, 0.00070948317, 0.025552582, 0.012194534, 0.020823937, -0.0004283936, -0.013760804, -0.004922564, -0.011724653, 0.003016935, -0.03872417, -0.036755145, 0.0022393938, -0.0141561, 0.009822752, 0.0010590971, -0.042662222, -0.04305006, -0.040424693, 0.07941736, 0.0015830519, 0.016169878, -0.015722372, -0.021167025, -0.027014434, -0.023106217, -0.014693107, -0.011113062, 0.01767648, -0.04913614, -0.043079894, -0.0013704866, 0.042274382, 0.017363226, 0.03159391, 0.0029013294, -0.0057802834, -0.04242355, 0.020838855, 0.006619357, -0.015453868, -0.027685693, -0.007816435, 0.0017051836, -0.047107447, 0.0012474224, 0.022554293, -0.005817576, 0.023911728, -0.05280569, -0.048151627, 0.01691572, -0.013820471, 0.0049486686, -0.021226693, 0.025119994, -0.009770543, -0.008472777, 0.01463344, 0.015364367, -0.0027931822, -0.012321327, -0.027059184, -0.005015794, 0.029281797, -0.013186505, -0.0538797, -0.013365507, 0.019302418, 0.008129689, -0.023389637, -0.008390734, -0.042304218, -0.02641776, 0.063963495, 0.013000044, -0.035770632, -0.014708024, -0.030758567, 0.0039790724, 0.02080902, -0.010397051, -0.011866363, -0.006306103, -0.02361339, 0.008592112, -0.0071190717, -0.010523845, 0.012761374, -0.0230018, 0.0064701885, 0.019198, -0.02746194, -0.01872066, -0.0106058875, -0.027402272, 0.023896812, 0.015334533, 0.012298952, 0.008510069, -0.014185934, -0.0010115496, 0.01980959, -0.032011583, 0.0027148686, -0.006183039, -0.015424034, 0.038187165, -0.020227263, 0.014208309, -0.002489251, 0.005907077, 0.016333962, -0.00803273, 0.024881324, -0.013358049, -0.023792394, -0.0150063615, 0.014432062, -0.014708024, -0.012798666, 0.025060326, 0.009009784, 0.022375291, -0.013425174, 0.0061233714, 0.021793533, 0.012313869, 0.01425306, -0.009278287, 0.015767122, -0.015259949, -0.003563265, -0.02541833, 0.012761374, 0.005992849, 0.011911113, 0.005164963, 0.018302988, 0.0075628483, 0.01520774, 0.053193524, -0.033324268, 0.055222217, -0.0034644408, -0.047853287, -0.001061894, 0.009658667, 0.021241609, -0.027521608, 0.0023344888, -0.006134559, 0.012388453, -0.024314484, -0.026208924, -0.013648927, -0.039499845, -0.0065410435, -0.016378714, 0.022405125, -0.016796386, -0.016975388, 0.020152679, -0.0022748213, -0.12166192, 0.0011318168, 0.007943228, 0.0149094025, -0.0033469705, -0.012738999, 0.04755495, 0.0033898565, -0.019451587, 0.013387882, 0.0034271488, -0.022136621, 0.018123986, -0.013432633, 0.022106787, -0.066827536, -0.0017751064, -0.03254859, 0.020436099, -0.028580705, -0.013164129, 0.021331111, 0.0014571908, -0.006813276, -0.0410512, 6.205647e-05, -0.011814154, -0.0015308429, -0.008584653, -0.025597332, -0.0151704475, 0.036277805, 0.0046540606, 0.021554863, 0.013455008, -0.0105536785, -0.0059369104, 0.00031465254, 0.02171895, 0.00088801933, -0.00883824, -0.027163602, 0.004732374, -0.03711315, 0.021644365, -0.0036080156, -0.0056534903, -0.0027913176, 0.024180232, 0.007637433, -0.009576624, -0.0022785505, -0.023136051, 0.0035949633, 0.022599043, -0.0062799985, 0.008808406, 0.001537369, 0.02850612, 0.03541263, 0.0039044882, 0.035382792, 0.031116571, -0.01382793, 0.0055677183, -0.009546791, 0.035382792, 0.02855087, -0.023300136, -0.023643224, -0.032518756, 0.009084368, 0.0150809465, 0.01534945, -0.023822226, -0.015200281, 0.043557234, 0.042184882, -0.01904883, -0.009315579, 0.0073279077, -0.011239855, -0.018631158, 0.030117141, -0.0047733956, 0.02589567, -0.0046018516, -0.02726802, 0.006410521, 0.04555609, 0.026820516, -0.019078664, -0.0030038827, 0.0067386916, -0.04448208, -0.0268951, -0.011776862, 0.0021815908, 0.02594042, 0.011575484, 0.029237047, 0.042542886, -0.021704033, -0.036755145, -0.009129119, 0.036665644, 0.0017723095, -0.009449831, -0.025492914, 0.0025097618, 0.012269118, 0.0066491906, 0.00821919, 0.021644365, -0.01297021, 0.023732726, 0.049673147, -0.0053327777, -0.017989734, -0.004351994, 0.017348308, -0.005150046, -0.004504892, 0.007298074, -0.0015662704, 0.00779406, -0.012463037, 0.012597289, 0.013231255, 0.0032984908, 0.007868644, 0.012313869, -0.0028584434, -0.0007537676, 0.006485105, -0.006328478, -0.003787018, -0.0019298688, -0.031444743, 0.018780328, -0.005869785, -0.0149317775, 0.014506647, 0.0015112645, -0.020122845, -0.036039136, -0.042751722, -0.00978546, 0.008428026, 0.022270873, 0.042930726, 0.0011392753, -0.011314439, -0.0047099986, -0.034278944, -0.010993727, -0.006496293, -0.010046505, -0.0017601895, 0.030176809, 0.038843505, 0.04209538, 0.045078754, -0.0015187229, 0.019675339, -0.01606546, -0.0015327075, -0.012463037, -0.011202563, 0.015647788, -0.06521651, 0.00422893, 0.012425745, 0.049404643, 0.067006536, 0.015811872, 0.0020939542, 0.018422322, 0.018183652, 0.033085596, 0.038992673, 0.03806783, -0.020257097, 0.0027745361, 0.012664415, -0.025343746, 0.0052432767, 0.004105866, 0.023896812, -0.023165885, -0.016408546, 0.026402844, 0.0015718642, 0.012269118, -0.009569166, 0.017422894, 0.01667705, -0.0069251526, -0.03177291, -0.004191638, -0.02883429, 0.02028693, -0.007913395, -0.0010861339, 0.0036881939, 0.008614487, -0.012627123, 0.014036766, -0.001743408, 0.019943843, -0.024791822, -0.019690257, 0.0034644408, 0.011269689, -0.05507305, 0.008897907, 0.014409687, -0.008010354, -0.025552582, -0.011202563, -0.05668407, -0.028013865, -0.017258808, 0.017810732, -0.0107028475, -0.015274866, 0.03550213, -0.011843988, -0.024210066, 0.010441802, -0.021420611, -8.862713e-05, -0.0026328259, 0.0072906157, -0.014066599, 0.014208309, -0.047763787, -0.0024295836, -0.0051761507, -0.007965604, 0.04800246, 0.023300136, -5.400807e-06, -0.024016146, -3.962291e-05, -0.033562936, 0.0042513055, 0.0047920416, -0.00058595295, -0.015245032, -0.007301803, 0.016319046, 0.009285746, -0.017422894, 0.016751634, -0.03854517, 0.029520467, 0.028565789, 0.024031062, 0.015975958, 0.020749353, 0.04376607, -0.004035011, -0.022151537, 0.0005784945, -0.011829071, -0.0075814943, 0.012112491, 0.030206643, -0.031325407, 0.0150809465, 0.028207783, -0.0051612337, 0.033383936, -0.029938139, -0.001432951, 0.0008549226, -0.019391919, 0.01776598, -1.1428005e-05, -0.0074584302, -0.008368359, -0.009628833, 0.009211161, -0.021614531, -0.025686834, -0.029237047, 0.0066678366, 0.015409117, -0.008122231, 0.002532137, 0.0037609134, -0.00560874, 0.010091256, -0.030236477, -0.0013639604, -0.012209451, 0.023270303, 0.03526346, -0.048927303, -0.014387312, -0.006988549, 0.0014534616, -0.025030492, 0.0016595008, 0.032011583, 0.0411407, -0.04537709, -0.006693941, -0.0002587143, -0.015148072, 0.03421928, -0.0037553196, -0.0012931053, 0.0053066732, -0.006343395, -0.012656956, 0.0073428247, -0.00855482, -0.022792963, -0.030117141, 0.0011103738, -0.021853201, 0.022643795, -0.01396964, -0.019362085, 0.01815382, -0.00054912694, -0.0047398326, 0.02509016, 0.029207213, -0.0192875, 0.050389156, 0.0005258193, -0.026775764, 0.023195717, -0.04075286, -0.0150138205, -0.00021093374, -0.0007849998, 0.015289783, 0.0055826353, 0.004422849, -0.011635152, 0.01164261, 0.011553109, 0.013298381, -0.011955864, 0.023538806, 0.020257097, 0.084608436, -0.022136621, -0.042871058, -0.013716053, -0.01230641, 0.012246743, -0.022986881, -0.004143158, -0.035233624, -0.019466503, -0.016095292, 0.03168341, 0.016990304, 0.007163822, 0.019973677, 0.0061457464, -0.0072085727, -0.0039417804, 0.04722678, 0.00418045, 0.0064776465, 0.0537902, 0.003971614, -0.008592112, 0.0102628, 0.0243294, 0.021346027, 0.008868074, -0.0025097618, 0.03920151, -0.009651209, -0.04000702, -0.021196859, 0.0056534903, -0.035561796, 0.01382793, 0.0014683785, 0.02746194, -0.034607116, 0.022599043, -0.009539332, -0.009845128, 0.017258808, 0.012090116, 0.029997807, 0.0055155093, 0.011038477, -0.00011175992, 0.011903655, -0.016557716, -0.021152109, -0.00912166, -0.0115009, -0.018974246, 0.03896284, -0.04066336, 0.0014599877, 0.029803887, 0.01972009, 0.016498048, -0.036904313, -0.0029348924, -0.03213092, 0.03254859, 0.0024668758, -0.07840302, -0.010934059, 0.0007770752, 0.0105984295, 0.045168255, -0.026253674, -0.018422322, 0.0013210744, 0.003801935, 0.006656649, 0.00037082384, -0.018019568, -0.00888299, 0.006969903, 0.026328258, -0.015901374, -0.026910016, -0.026865266, -0.026432676, 0.000650748, -2.5856863e-05, 0.048062123, 0.028923793, -0.029609969, 0.008823323, -0.032369588, 0.03630764, 0.0411407, 0.077209674, 0.021062607, -0.018138902, -0.0030560917, -0.004493704, 0.0049523977, -0.0008404719, 0.020734437, -0.0010180757, -0.036337472, 0.0021275172, -0.013380424, -0.031146405, -0.026387926, 0.00028015728, 0.018869828, 0.023628308, -0.0067498796, 0.0034532533, 0.0106730135, 0.003382398, -0.020346599, 0.023061467, 0.0015625411, 0.03263809, -0.007055675, -0.038187165, 0.0167218, 0.025836002, -0.0038746546, -0.017094722, -0.009964463, -0.024597904, 0.014611065, 0.008927741, 0.0047398326, -0.012955293, -0.009755627, 0.01088185, 0.01772123, 0.0019084258, 0.03735182, -0.011485983, -0.019183083, -0.008249024, -0.006902777, 0.03320493, -0.0051761507, 0.008621945, -0.038783837, -0.019645505, 0.0061271004, 0.0042923265, 0.0067386916, -0.006119642, 0.009986838, 0.019451587, -0.018869828, -0.035144124, 0.028998377, -0.017646646, 0.0037665074, -0.027208354, 0.0026682534, -0.006775984, -0.0038933007, -0.020734437, 0.03970868, 0.0047920416, -0.03654631, 0.013909972, 0.026044838, 0.010859475, -0.018914578, 0.0032649278, 0.023538806, 0.017094722, -0.0078089763, 0.015573203, -0.001000362, 0.014387312, 0.00389703, -0.01739306, 0.01776598, -0.0006251097, -0.004243847, 0.035084456, 0.0042736805, -0.021808451, 0.004389286, -0.005526697, -0.007965604, 0.01819857, 0.005168692, 0.0037944764, -0.0065410435, -9.521152e-05, -0.0037926119, -0.022390207, -0.004385557, 0.007764226, 0.032429256, 0.020883605, -0.03911201, 0.015073488, 0.01819857, 0.0106282635, -0.007950687, 0.009710876, 0.0043743695, 0.009442373, -0.02770061, 0.021927785, -0.017512394, 0.017079806, 0.0065820646, 0.0044079325, -0.021793533, -0.0040126354, -0.01453648, -0.0063172905, 0.034159612, -0.030609397, 0.0009341684, -0.0052581932, -0.020167595, -0.013857763, 0.02585092, 0.007987979, 0.007928312, 0.013134296, -0.027491774, 0.013298381, 0.0027913176, -0.017661562, 0.039798185, 0.016304128, -0.0041021365, -0.030102225, -0.007823894, 0.017825648, -0.00095094985, -0.0074136797, -0.028088449, -0.01387268, -0.004299785, 0.007950687, -0.008651779, 0.029416049, -0.0030784672, 0.035472296, -0.013648927, 0.05602773, -0.011955864, -0.009158952, -0.011605318, -0.01401439, 0.024075814, 0.03207125, -0.03976835, -0.029923223, -0.007913395, -0.04328873, -0.0042550345, -0.017616812, -0.025119994, 0.0053700698, -0.031325407, -0.009576624, 0.016125126, 0.008502611, 0.008532444, 0.0150884045, 0.02076427, 0.005992849, -0.01112052, 0.011918572, -0.017885316, -0.0038131224, -0.038933005, -0.033891108, 0.015990874, 0.050150484, 0.0039380514, 0.023374721, 0.016095292, -0.009994296, 0.013939806, -0.014767692, -0.012530163, -0.0128285, -0.012194534, 0.031862415, 0.018452156, 0.0032817093, -0.031951915, -0.013216338, 0.015856624, 0.020734437, 0.029908305, -0.010456719, 0.00027689422, 0.00165111, -0.007779143, -0.021510113, -0.0105014695, 0.013059711, -0.028148117, -0.019227833, 0.014267977, -0.009964463, 0.015230115, 0.012642039, -0.008584653, 0.03759049, 0.00874128, -0.009539332, 0.023210635, -0.0067834426, -0.002455688, -0.0025973984, 0.025806168, 0.025746502, 0.03281709, -0.0074285967, -0.035084456, 0.0038933007, -0.010098714, 0.036098804, 0.0074733472, 0.0051985257, 0.015975958, -0.011776862, -0.009233537, 0.0074733472, 0.019108498, -0.009032159, 0.030042557, 0.0064552715, 0.0015177906, -0.01016584, -0.0059555564, -0.034278944, -0.010173299, -0.007387575, 0.021554863, -0.0069065066, 0.008987408, -0.017228974, -0.031713247, 0.006086079, -0.020958189, 0.044183742, -0.036367305, -0.00057616376, -0.013052253, -0.0017378143, 0.022032203, 0.02931163, 0.0012912408, -0.038604837, -0.014722941, 0.005183609, 0.015245032, 0.021480279, 0.015931208, -0.034815952, -0.057937086, -0.006600711, 0.03135524, 0.001525249, -0.010150923, 0.0134923, -0.0067200456, -0.05384987, -0.0055192383, 0.006846839, 0.00095141603, 0.018302988, 0.034159612, -0.007339095, -0.032667924, 0.01534945, -0.030057475, -0.008965033, 0.02532883, -0.020823937, 0.02509016, -0.03174308, 0.04075286, -0.015424034, 0.027894529, -0.025836002, -0.023225551, 0.017870398, 0.020107929], "content_hash": "fedcd60a1e79952b153140d3e9539021"}, {"content": "# Experiment: Comparing Heads Across Categories\n\n## Goal\n\nSystematically ablate heads from each category to discover which types of attention heads matter most for different prompts. Build intuition for how attention head roles vary.\n\n## Prerequisites\n\n- Complete \"Your First Ablation\" (know how to use the ablation panel)\n- Complete \"Exploring Attention Patterns\" (understand head categories)\n\n## Steps\n\n### Step 1: Set Up a Simple Prompt\n\n1. Select **GPT-2 (124M)** and enter: `The cat sat on the`\n2. Set beams to 1 and tokens to 5.\n3. Click **Analyze**.\n4. **Select the generated sequence for comparison** by clicking \"Select for Comparison.\"\n5. Note the original prediction and probability in Stage 5 (e.g., \"mat\" at 45%).\n\n### Step 2: Record the Head Categories\n\n1. Expand **Stage 3 (Attention)** and note one head from each category:\n   - **Previous-Token**: _______ (e.g., L0-H3)\n   - **First/Positional**: _______ (e.g., L0-H1)\n   - **Bag-of-Words**: _______ (e.g., L2-H5)\n   - **Syntactic**: _______ (e.g., L4-H2)\n   - **Other**: _______ (e.g., L1-H8)\n\n### Step 3: Ablate One Head at a Time\n\nFor each head you noted, do the following:\n1. Go to the **Ablation** tab in the Investigation Panel.\n2. **Clear** any previously selected heads.\n3. **Add** just the one head from the current category.\n4. Click **\"Run Ablation Experiment.\"**\n5. Record the results:\n\n| Category | Head | Probability Change | Generation Changed? |\n|----------|------|-------------------|-------------------|\n| Previous-Token | | | |\n| First/Positional | | | |\n| Bag-of-Words | | | |\n| Syntactic | | | |\n| Other | | | |\n\n### Step 4: Analyze Your Results", "source_file": "experiment_comparing_heads.md", "chunk_index": 0, "embedding": [0.013296178, 0.01098963, 0.027249454, 0.0021908856, -0.031433422, -0.030360611, -0.016735885, 0.02944872, 0.026833737, 0.044521745, 0.052031435, -0.036368363, 0.00055694085, 0.0039559985, 0.02197926, -0.0039392356, 0.03610016, -0.008354532, -0.0005695129, 0.06769451, -0.011063385, -0.013436984, 0.01956543, 0.047257416, 0.022368154, -0.017553905, 0.020584602, 0.039667264, 0.025801156, 0.010486748, 0.02789314, -0.005062337, -0.029019594, 0.01968612, -0.038996756, 0.0015011003, -0.005504872, 0.02141603, -0.0098028295, -0.0030625463, 0.016883396, -0.04575548, -0.047096495, 0.012209954, -0.028751392, 0.039640445, 0.055893563, -0.013671662, -0.01882787, 0.017299112, -0.022287693, 0.06270593, -0.025640232, 0.017138189, 0.01625312, 0.013698482, -0.018935151, -0.0030893665, 0.017017499, -0.008723311, -0.0036777374, -0.006601153, 0.05503531, 0.000619801, -0.0008050289, -0.032479417, -0.046935573, 0.015823994, 0.05259466, 0.015059614, 0.034544583, 0.023655528, 0.01104327, 0.0058401264, 0.005746255, -0.031567525, 0.0069196443, -0.0032888427, -0.010768361, -0.03859445, 0.02786632, 0.022274282, 0.053640656, -0.017755058, -0.009400525, -0.032345314, -0.03864809, -0.038809013, -0.02620346, -0.060721222, -0.0063161873, 0.017741647, -0.043073446, 0.011418754, -0.013604611, -0.011163961, 0.037816662, -0.030119227, -0.0009202725, 0.026847148, 0.04527271, -0.0054814043, -0.043609854, -0.0065173395, 0.008857413, -0.0051360927, -0.0076572034, 0.015059614, -0.033310845, -0.02613641, -0.03137978, -0.0039995816, -0.0069129393, 0.023843272, 0.021201469, -0.0010736512, -0.03682431, -0.0040934524, 0.028000422, -0.06903552, -0.019417917, -0.003097748, 0.016065376, -0.0046533267, 0.017500265, 0.0005862756, 0.04830341, -0.056590892, 0.0014340493, -0.01953861, 0.04489723, 0.018666947, -0.0072683087, -0.043985337, -0.020101836, 0.013309588, -0.02450037, 0.048062027, -0.06946465, 0.021469673, -0.0026870617, 0.012028917, -0.012049032, 0.042242017, -0.06334961, -0.011988686, 0.012176429, 0.07498964, -0.031889368, 0.042751603, 0.010412992, 0.01389293, -0.010044212, -0.032345314, -0.008890939, -0.06463699, -0.004308015, -0.0017315875, -0.029207338, -0.07799351, 0.0734877, -0.02293138, -0.0035436358, -0.016507912, -0.03768256, 0.031299323, -0.034088638, 0.010453222, -0.03540283, -0.011532741, -0.048517972, -0.033391308, 0.017138189, -0.026780097, -0.0024959669, 0.03355223, -0.04580912, -0.0040196967, -0.014389106, -0.028134523, 0.0132224215, 0.01271954, -0.016145837, 0.006581038, -0.018465795, -0.044736307, 0.0149255125, 0.002708853, -0.04358303, -0.04006957, 0.028429547, 0.0030139342, 0.057502784, 0.025720695, -0.017701415, -0.026659405, 0.057931907, 0.034893245, 0.035724677, -0.009762599, 0.016829755, -0.0022344685, 0.015019383, 0.001239602, 0.02385668, 0.027464015, -0.014429336, 0.00050665275, -0.049698066, -0.030494712, 0.04586276, -0.07466779, 0.012203249, 0.03685113, -0.073058575, -0.02706171, -0.06624621, 0.034598224, 0.023266634, 0.020758934, 0.05809283, -0.02950236, 0.045540918, 0.048866637, -0.022891149, 0.018479206, 0.060292095, -0.024781983, 0.002733997, -0.04151787, 0.014308644, 0.024111474, -0.014228184, 0.012095967, -0.037870303, -0.04924212, 0.052138716, -0.003414563, 0.041249663, -0.02696784, 0.007704139, 0.0009403877, 0.022167, -0.02781268, 0.02773222, 0.003520168, -0.015233946, -0.06334961, 0.0096217925, 0.04178607, 0.06592437, -0.0010987953, -0.048625253, 0.011572971, -0.018733999, 0.0016335256, -0.00045971718, 0.0017718179, -0.027075121, 0.0055853333, -0.04910802, 0.013027974, -0.016199477, 0.035912417, -0.021778107, -0.038969938, -0.023239814, -0.0074225254, -0.051226825, -0.026659405, -0.06635349, 0.003607334, 0.037199795, 0.010439812, -0.028134523, -0.005018754, -0.0033575697, -0.024486959, -0.029126875, -0.02200608, -0.020852804, -0.0014072291, 0.025827976, -0.0009965427, -0.017728237, 0.008689786, -0.0056423265, 0.00015149295, 0.013564381, -0.02613641, 0.014630489, -0.052004617, 0.06393966, -0.04253704, -0.016615193, 0.015180306, 0.0106610805, 0.021429442, -0.017875748, 0.050180834, 0.03355223, -0.023038661, 0.05482075, 0.0002820325, 0.031406604, 0.020960085, 0.02294479, 0.0019562077, -0.002732321, 0.023762811, -0.01101645, -0.013209011, 0.009722369, -0.021348981, -0.040498696, -0.03202347, -0.0014164485, -0.010922578, -0.008381353, -0.004774018, -0.06281321, -0.04189335, 0.010533684, 0.03119204, 0.0237494, 0.01884128, -0.030682454, 0.005645679, 0.0031497122, -0.025170878, 0.03030697, -0.015140075, 0.0085758, 0.018358514, -0.018506026, 0.009226193, -0.024902673, 0.07134207, -0.007127502, 0.048062027, -0.0426175, 0.018197592, -0.0155289695, 0.012598849, -0.030870197, 0.057824627, 0.011009744, -0.007865061, 0.044602204, -0.0007865899, -0.044387642, 0.017728237, -0.008079624, -0.0044085914, -0.012109378, 0.05077088, -0.02958282, -0.045460455, 0.015180306, 0.012585439, 0.028965954, 0.023655528, 0.019122893, 0.010392877, 0.034490943, -0.019122893, -0.03864809, 0.002752436, 0.01880105, 0.0024775278, -0.0003545312, -0.016869986, -0.0065408074, -0.041222844, -0.0674263, -0.025680464, -0.022730228, -0.020155476, -0.0017005764, 0.047176957, -0.018211002, -0.02950236, -0.041196022, -0.06292049, 0.033793613, 0.029368259, 0.019806812, 0.00811315, -0.07734983, -0.0035302257, 0.011244423, -0.014670719, -0.007838241, 0.026149819, -0.024057833, 0.05428434, -0.024916084, 0.061525833, 0.038835835, 0.06533432, 0.014295234, 0.009628498, -0.041303303, -0.011539446, 0.034330018, 0.0056557367, 0.018975383, 0.007898587, 0.009776009, 0.0030306971, 0.06495883, -0.0046298592, 0.043851234, 0.009963752, -0.0072414884, 0.014858462, 0.019914092, 0.020638242, -0.000861603, -0.012404402, -0.008441698, -0.06823091, -0.010768361, -0.023910321, -0.03690477, -0.014617079, 0.02456742, -0.01969953, 0.045943223, -0.038353067, -0.0035134628, 0.06383238, 0.04151787, -0.009648613, -0.01802326, -0.0072280783, 0.0025261396, 0.063993305, -0.026900789, -0.019243585, 0.014335465, -0.017325932, 0.02385668, 0.010433108, -0.046935573, 0.012793296, -0.013135255, -0.08915077, 0.02610959, -0.029797385, -0.030441072, -0.007529807, -0.0073219496, 0.033284027, 0.013088319, -0.023494607, 0.02952918, 0.008367943, 0.095855854, -0.009226193, 0.018760819, -0.022555895, -0.008019278, 0.019337457, -0.0053338925, 0.040579155, 0.010929284, -0.024339447, 0.021697644, -0.018304873, 0.044575386, -0.0022294398, 0.0053439504, -0.028590469, -0.030709276, -0.012203249, -0.004995286, 0.026002308, 0.0048779473, -0.053882036, 0.030682454, -0.023105713, 0.034222737, 0.049698066, -0.0038520696, 0.04342211, -0.012411106, 0.0039895237, 0.016910216, 0.010734837, -0.0024875854, -0.006866004, 0.010319121, -0.015005973, 0.03419592, -0.042349298, -0.0037615509, 0.019525198, 0.0044722897, -0.004026402, 0.026833737, -0.033874072, 0.010506864, 0.051924154, 0.0067821904, 0.014912102, 0.05551808, 0.013108435, -0.027128762, 0.040847357, -0.030655634, -0.0007023573, -0.008662966, 0.014228184, -0.01640063, 0.026324151, -0.00066967006, 0.01969953, 0.016145837, 0.014147723, -0.025036776, -0.0038822424, -0.03255988, 0.0013963333, 0.0011432165, -0.009360295, -0.01963248, -0.056912735, -0.04015003, -0.019793402, 0.004220849, -0.030950658, 0.002658565, 0.01799644, -0.034920067, 0.0008968047, 0.01644086, -0.005682557, -0.03926496, -0.0047203777, 0.008528864, 0.017661186, 0.012779886, -0.01634699, 0.02294479, -0.008012573, 0.063778736, -0.00617538, 0.016467681, 0.02706171, -0.013537561, -0.03028015, 0.021496493, -0.01394657, 0.0018573077, -0.0065910956, -0.037226614, -0.04186653, -0.006470404, -0.025546363, 0.028939134, -0.046399165, 0.01192834, -0.015931275, -0.00019706655, 0.021912208, 0.002925092, 0.014322055, 0.03186255, -0.012652489, 0.008066214, -0.017553905, 0.020088425, -0.030494712, 0.028885493, 0.02041027, -0.02301184, 0.015703302, 0.0098028295, -0.01476459, -0.022676587, -0.01063426, -0.02209995, 0.049805347, 0.00026003146, -0.042885702, 0.025184287, 0.009615088, -0.030119227, -0.024124885, 0.016802935, -0.031031119, 0.021348981, 0.01880105, -0.022609536, -0.004790781, 0.024661291, 0.009206077, -0.0078047155, -0.0004096386, 0.021871977, 0.016025146, -0.00224285, 0.03030697, -0.028054062, 0.0067788376, -0.028778212, -0.0046935575, 0.01645427, 0.029985126, 0.008723311, 0.0077242544, -0.0490812, -0.013215716, 0.013296178, 0.013732008, -0.0060647465, 0.012531798, 0.007992458, -0.028939134, 0.00041906763, -0.008381353, -0.021939028, 0.012370876, 0.016869986, 0.010533684, 0.022368154, -0.031701628, 0.033176746, 0.003329073, -0.047793824, 0.029073235, 0.018452385, 0.020115245, -0.0085758, 0.008797067, -0.008904349, 0.0070202206, 0.02372258, 0.012813412, -0.014912102, 0.002504348, -0.010929284, -0.016199477, -0.01471095, -0.0051226825, 0.006654794, -0.015676482, -0.03347177, -0.012867052, -0.010949398, -0.027329914, 0.07187848, 0.0079187015, 0.025144057, -0.0024121532, -0.01634699, -0.01228371, -0.025197698, 0.0048779473, -0.03261352, 0.007938817, -0.008321007, -0.020370038, 0.01150592, -0.008059508, -0.0137051875, 0.03202347, -0.016574962, 0.006513987, -0.05398932, 0.01720524, 0.041651968, 0.007040336, 0.010325826, 0.04819613, 0.0029016242, -0.0063295974, -0.007529807, 0.0069196443, -0.02531839, 0.027155582, -0.04900074, -0.014912102, 0.015354638, -0.01889492, 0.008522159, -0.04436082, 0.0002348874, 0.014013621, -0.0081600845, -0.044521745, 0.0020500787, -0.005052279, -0.0057361973, 0.0013334731, -0.024688112, 0.03030697, -0.014858462, -0.09880608, -0.03939906, -0.0098028295, 0.017902568, 0.020007964, -0.00346988, -0.0115796765, 0.01146569, 0.07193212, 0.01102986, -0.02610959, -0.031406604, 0.003129597, 0.0049483506, 0.036985233, -0.017071139, 0.03942588, -0.024956314, -0.0070604514, -0.008911054, -0.033230387, -0.036770668, 0.026726456, -0.001555579, 0.021456262, 0.027195811, -0.018465795, -0.0043884763, -0.022381563, -0.014885282, 0.001623468, 0.041222844, 0.033874072, 0.025801156, -0.054525726, -0.014255004, 0.037092514, -0.027195811, -0.003086014, 0.023494607, -0.016494501, 0.049322583, -0.0048108962, 0.02629733, 0.01142546, 0.020209117, -0.001183447, -0.010875643, 0.014563438, -0.03347177, -0.018157361, -0.022314513, 0.0118210595, -0.0089579895, -0.011184077, 0.023762811, -0.0037079102, 0.0031262445, -0.03344495, -0.007999163, 0.046157785, 0.02141603, -0.0030826614, -0.007637088, -0.007147617, -0.03754846, -0.049858987, 0.036180623, -0.020866215, 0.0013250918, 0.023963964, 0.008213725, 0.008522159, 0.033203565, -0.01387952, 0.04039141, -0.03620744, 0.036395185, -0.015488739, -0.018090311, 0.028751392, 0.012411106, 0.007100682, -0.056108125, -0.022515666, -0.013450394, 0.0012613935, -0.021147829, -0.01634699, -0.014308644, -0.03290854, -0.00812656, -0.0010426402, 0.015998326, -0.01391975, -0.003992876, -0.003414563, -0.019122893, -0.11285994, -0.011090205, 0.015086434, -0.026055949, -0.04495087, 0.0027876378, 0.02947554, 0.015756942, 0.017004088, 0.015971504, 0.020477321, -0.030146047, 0.00022545838, -0.012893872, 0.025640232, -0.038165327, 0.007992458, -0.017527085, 0.013336408, -0.03137978, -0.025787745, 0.026310742, -0.0016787848, -0.0053841807, -0.056322686, 0.007207963, -0.03019969, 0.0027759038, 0.005508225, 0.034705505, -0.032050293, 0.017795287, -0.045460455, 0.009615088, -0.02624369, 0.035080988, -0.0078114206, 0.015743533, 0.011814354, -0.011955161, 0.014469567, -0.0058434787, -0.007838241, -0.014322055, 0.02539885, -0.011780829, -0.0058166585, -0.0048779473, 0.01269272, 0.035510115, 0.0064905193, 0.003848717, 0.009594972, 0.022891149, 0.03494689, -0.024768572, -0.0008310111, -0.02057119, 0.023400735, 0.03274762, 0.018170772, 0.028885493, 0.05221918, -0.016816346, 2.4449977e-05, 0.010037508, -0.009749189, 0.03376679, -0.015193716, -0.0327208, -0.022354743, 0.0026853853, -0.0029301208, 0.00024452596, -0.0060077533, -0.008763542, 0.01394657, 0.024916084, 0.0006654794, 0.0064603463, 0.051119544, -0.02456742, -0.017419802, 0.015381458, 0.0112041915, 0.0344373, 0.009628498, -0.00117758, 0.0058904146, 0.06415422, -0.0013435308, -0.03596606, -0.0022042957, 0.034732323, -0.036180623, -0.0015363019, 0.026498483, 0.012277005, 0.02054437, 0.019310636, 0.0056792046, 0.015368048, -0.00029523313, -0.024312627, -0.03280126, 0.0068056583, 0.012793296, 0.00554175, -0.015984915, -0.013235832, 0.04012321, 0.031674806, -0.017500265, 0.030709276, 0.003216763, 0.007844945, 0.047874283, 0.0021925617, 0.0083411215, -0.0031245681, -0.01559602, 0.02708853, -0.015260766, -0.012176429, -0.015716711, 0.031487066, -0.036502466, -0.012169723, 0.0038017815, 0.006899529, 0.052648302, 0.02041027, -0.020302989, -0.00939382, 0.018036671, -0.010010688, -0.013329702, -0.005769723, -0.014348875, 0.000502043, -0.0212417, 0.0030223157, 0.032157574, -0.0059708753, -0.0162263, -0.018680358, 0.0067218444, -0.014268414, -0.016883396, 0.014228184, 0.008394763, 0.006638031, -0.0031161867, -0.008978104, -0.013839289, 0.01810372, 0.02778586, 0.0032184394, -0.032640338, 0.022167, 0.01813054, 0.0019880568, 0.063671455, 0.021697644, 0.0035972765, 0.00010712416, -0.035671037, 0.0016385544, 0.020973496, 0.0028731276, -0.047096495, 0.028992774, 0.014831641, 0.041598327, 0.02197926, 0.027651757, -0.017366162, -0.02121488, -0.005162913, -0.0033827138, -0.017057728, 0.028778212, -0.01961907, -0.017661186, -0.0035536934, 0.014603669, 0.0026904142, 0.018304873, 0.01799644, -0.018036671, -0.0017081196, 0.023843272, -0.016320169, 0.0277054, -0.026149819, -0.016078787, 0.022864329, 0.0036039816, -0.0377362, -0.03288172, -0.037816662, 0.047257416, 0.0062927194, -0.009152437, 0.034678683, 0.040310953, -0.016735885, 0.015019383, -0.028054062, -0.018211002, -0.03690477, 0.020919856, -0.013537561, -0.0018522788, -0.043636672, 0.012250185, 0.0026350971, -0.007207963, 0.032345314, -0.011076795, -0.058521956, -0.016816346, -0.043019805, 0.022287693, 0.005169618, 0.015180306, 0.040740076, 0.0018103721, 0.019820223, 0.005075747, -0.03030697, -0.016642014, -0.014134313, 0.0050422214, -0.039962288, 0.017956208, -0.016239708, 0.0006759561, -0.00693976, 0.013322998, 0.03376679, 0.0155289695, -0.019337457, -0.0110835, 0.022837508, -0.022622947, 0.037816662, 0.018452385, -0.011251127, -0.010392877, -0.015005973, -0.012531798, 0.034330018, -0.008428288, 0.017486854, -0.01145228, 0.017446622, 0.013329702, 0.056215405, 0.010151494, -0.008381353, 0.018693767, -0.012451337, -0.05168277, 0.009782715, -0.0074024103, -0.016722474, -0.011908226, 0.013396754, -0.0075231018, 0.0114791, 0.015019383, -0.019189944, 0.023347095, -0.008810477, -0.01725888, 0.0081802, -0.010003982, -0.003216763, -0.034034997, 0.028831853, 0.021496493, -0.013564381, 0.007389, 0.0043616556, -0.01881446, -0.017594134, -0.009212783, -0.0013653223, -0.033391308, -0.00011335779, -0.013986801, 0.008012573, -0.021617183, -0.008515454, -0.003557046, -0.0033357781, 0.0027122055, 0.013396754, -0.029985126, 0.010433108, -6.746989e-05, 0.038218968, -0.02610959, -0.008227135, 0.029636461, 0.0026384497, -0.050180834, -0.010560504, 0.015086434, -0.0091993725, 0.011666843, 0.009863175, 0.03022651, -0.015005973, -0.03518827, 0.004096805, -0.018358514, -0.011184077, -0.042885702, -0.019887272, 0.035456475, -0.01872059, 0.0045695133, 0.002403772, -0.00674196, -0.0075566275, -0.008052804, -0.0032821377, -0.03355223, 0.0106610805, 0.020396858, -0.009836355, 0.00634636, -0.020530961, 0.026257101, -0.01709796, 0.00073420646, 0.034061816, 0.0043985336, -0.031433422, 0.003607334, -0.028214986, 0.0024071245, 0.019525198, -0.0025713989, -0.011700368, -0.020195706, 0.0059105298, 0.01969953, 0.08110467, -0.02621687, -0.047874283, 0.009889996, 0.004820954, 0.010459928, 0.0020919857, -0.0134638045, -0.046962395, -0.0102386605, -0.04028413, 0.03365951, 0.0016352019, 0.016145837, 0.004804191, 0.017111368, -0.036502466, -0.00089345215, 0.055839922, -0.009125616, 0.009548036, 0.055679, -0.038353067, 0.018961972, 0.026418023, 0.022703407, 0.023655528, 0.0006378209, 0.0011423783, 0.0009848089, -0.0051394454, -0.051709592, -0.034222737, -0.014617079, -0.023910321, 0.017808698, 0.0035000527, 0.014509797, -0.013618021, 0.0024691464, -0.015877634, 0.027222633, 0.023910321, -0.010801887, 0.0009764276, 0.011264537, -0.012578733, 0.017486854, 0.0009152437, -0.025184287, -0.028724572, 0.018465795, -0.006513987, -0.020450499, 0.036234263, -0.047150135, -0.015274176, 0.017325932, 0.011633317, 0.0034866426, -0.015944684, 0.042215195, -0.017755058, 0.00063823996, -0.0096217925, -0.03274762, -0.024419907, -0.0032184394, -0.006644736, 0.007134207, -0.032157574, -0.026096178, 0.005672499, -0.016789526, 0.011988686, 0.0026820328, -0.008012573, -0.015341227, 0.04146423, 0.008904349, 0.009353589, -0.021027137, -0.02452719, -0.039506342, -0.03019969, 0.0026451547, 0.039720904, -0.026485074, -0.048062027, 0.025975488, -0.029287798, 0.011968571, 0.03347177, 0.042912524, 0.012558619, -0.0073018344, -0.022287693, -0.011331588, 0.020584602, -0.02291797, 0.0015262442, 0.009212783, -0.013255947, 0.005742903, -0.049215302, 0.0009588267, -0.04167879, 0.018452385, 0.027464015, -0.002125511, 0.028510008, -0.009387115, -0.006376533, 0.01471095, -0.004126978, 0.01387952, -0.0060278685, 0.04819613, -0.033176746, -0.012625669, 0.008897644, -0.00984306, 0.015756942, 0.00054269255, -5.610268e-05, -0.0014298587, 0.03674385, -0.003674385, -0.013671662, 0.011727188, -0.01064767, 0.01394657, 0.02786632, -0.02291797, 0.014415926, 0.0036207442, 0.0039694086, -0.012163019, -0.044602204, 0.020142065, 0.0057965433, 0.01641404, -0.014617079, -0.027289683, 0.0015404925, 0.010587324, 0.0032502885, 0.006185438, 0.019766582, -0.016145837, -0.026350971, -0.025600003, 0.037414357, 0.0017517026, 0.0057294923, -0.018613307, -0.005716082, 0.012015507, 0.002281404, -0.010097853, 0.018438974, -0.000418439, -0.005917235, 0.0029703514, 0.017299112, 0.030011946, -0.032398954, -0.0013359876, 0.053774755, 0.013450394, 0.005974228, 0.0063932957, -0.0067319022, -0.003216763, -0.018519437, -0.010982924, -0.0016251443, 0.013571085, -0.004566161, 0.018935151, -0.011311473, -0.0089579895, -0.015837403, -0.025304979, -0.0072616036, 0.030011946, -0.00864285, -0.009098796, 0.0018656891, 0.022260873, -0.006011106, -0.023910321, -0.0008385543, -0.008528864, 0.0542307, 0.038755372, -0.020799164, 0.010640965, 0.028268626, 0.003724673, 0.01810372, -0.009735779, -0.0277054, -0.025103826, -0.019230176, 0.024889264, 0.013571085, -0.016829755, -0.0022026193, 0.009662023, -0.011532741, -0.0033659511, -0.013048089, -0.0029016242, -0.002336721, -0.010037508, 0.021603774, 0.0056993198, -0.01270613, -0.006688319, -0.0277054, 0.005323835, 0.05551808, 0.009527922, -0.021764696, 0.011867995, -0.028241806, -0.026645996, -0.011720483, -0.015958095, 0.011318178, -0.019498378, -0.011117026, 0.0020534312, -0.0019444737, 0.030172868, -0.017714826, 0.0023417498, 0.0025127295, 0.01554238, 0.01387952, 0.038916294, -0.004730435, 0.019310636, -0.022770459, 0.055249874, 0.0075566275, -0.023387326, -0.010406287, -0.028617289, 0.025331799, -0.00043289686, -0.028778212, 0.00043708752, -0.02385668, -0.023910321, -0.005706025, -0.0029921427, -0.018170772, 0.023494607, -0.052836046, -0.016011735, -0.0180769, 0.0072414884, -0.0012320587, 0.034061816, -0.02864411, -0.008830593, -0.034169096, 0.011405344, -0.027624937, 0.0018958619, -0.020222528, -0.025170878, 0.010218545, 0.03784348, 0.0024004192, 0.014617079, -0.019471558, -0.014429336, 0.014053851, -0.03269398, 0.0002564694, -0.0150462035, -0.017741647, 0.019404506, -0.006705082, -0.002244526, 0.02377622, -0.013302882, 0.017433213, 0.022797279, -0.0122703, 0.021402622, -0.0029720275, -0.0058166585, -0.01107009, -0.020396858, -0.008662966, 0.018693767, -0.04108874, -0.022649767, 0.003644212, -0.028724572, 0.01555579, 0.0072481935, -0.0023266634, 0.05820011, -0.014496387, -0.019002203, -0.012753066, -0.0067151394, -0.010822002, 0.0007145103, 0.024983136, 0.027517656, 0.0012924045, -0.015394868, -0.040445056, -0.0025429022, -0.011318178, 0.02778586, 0.021080777, 0.0019109483, -0.007107387, -0.0024624413, 0.0081600845, 0.01964589, 0.025144057, -0.028375907, -0.00081718183, 0.033015825, 0.033632692, 0.027450604, -0.003216763, -0.012632375, -0.013074909, -0.019176533, 0.022354743, -0.008763542, -0.007435936, -0.014965743, -0.014040441, 0.011754009, 0.01630676, 0.044521745, 0.0017986383, -0.00815338, -0.005206496, 0.0079790475, -0.016145837, -0.0003668937, 0.00027574648, -0.01813054, 0.006158618, 0.004914825, 0.0035872187, 0.004579571, 0.022582715, -0.030360611, -0.04087418, -0.006104977, -0.01805008, 0.017419802, -0.012417812, 0.01145228, -0.015421689, -0.041598327, -0.025492722, -0.0043616556, -0.0196593, 0.0054881093, 0.0391845, -0.007382295, -0.024969725, -0.019256996, 0.007818125, -0.047150135, 0.032211214, -0.01967271, 0.015850814, -0.01225689, 0.014858462, -0.002492614, 0.026592355, -0.024191935, -0.016025146, 0.017701415, -0.028268626], "content_hash": "2d562d3bc4c833f7a66e92c8b8cef3e8"}, {"content": "the results: | Category | Head | Probability Change | Generation Changed? | |----------|------|-------------------|-------------------| | Previous-Token | | | | | First/Positional | | | | | Bag-of-Words | | | | | Syntactic | | | | | Other | | | | ### Step 4: Analyze Your Results\n\nLook at the table you've filled in:\n- **Which category caused the biggest probability drop?** Previous-Token heads often have the largest impact on simple prompts because local context matters a lot.\n- **Which category had the least effect?** BoW and Other heads often show smaller effects for short prompts.\n- **Did any ablation change the generated text?** A generation change is a stronger signal than just a probability change.\n\n### Step 5: Try a More Complex Prompt\n\nNow repeat the process with a prompt that requires more sophisticated processing:\n\n1. Enter: `The doctors told the patient that they would need`\n2. Analyze and select the generation for comparison.\n3. Ablate one head from each category again and record results.\n\n**What to expect**: For this more complex prompt:\n- **Syntactic heads** may matter more (there are grammatical dependencies like \"doctors...they\")\n- **First/Positional heads** may show more impact because the sentence structure is more complex\n- The pattern of which categories matter may shift compared to the simple prompt\n\n### Step 6: Compare Results Between Prompts\n\n| Category | Simple Prompt Impact | Complex Prompt Impact |\n|----------|---------------------|----------------------|\n| Previous-Token | | |\n| First/Positional | | |\n| Bag-of-Words | | |\n| Syntactic | | |\n| Other | | |\n\n## What You Should Learn", "source_file": "experiment_comparing_heads.md", "chunk_index": 1, "embedding": [0.0370882, 0.0060815564, 0.02018237, 0.009334002, -0.011282027, -0.0017570088, 0.02289446, 0.01818616, 0.025730455, 0.06255709, 0.028910624, -0.0025468883, -0.010683164, 0.013147451, 0.005744266, 0.005823426, -0.0041266475, 0.0027344632, -0.02085695, 0.017387675, 0.010208203, -0.011371512, 0.023844382, 0.04253992, 0.034472477, -0.023665411, -0.010717581, 0.036675192, 0.036344785, 0.019521555, 0.004253992, -0.020829417, 0.0007244865, -0.009464787, -0.047606163, 3.4094748e-05, -0.017497811, 0.0069041327, 0.026336202, -0.029874312, 0.014097371, -0.018172393, -0.022247413, -0.030782932, -0.014592982, 0.058757402, 0.04036474, -0.06360337, 0.0056306883, 0.019301284, -0.03174662, 0.06641184, 0.0065358663, 0.012555472, -0.0046257, -0.030067049, -0.047688764, -0.010573029, 0.003441741, -0.028112141, 0.029764175, -0.04052994, 0.03587671, 0.029764175, 0.021985842, -0.014606749, -0.010249505, -0.0037549394, 0.012734442, 0.026047096, 0.053636093, 0.03573904, 0.021806872, 0.0091756815, -0.04722069, 0.013154334, 0.021806872, 0.010841484, 0.0009223866, -0.015501602, 0.026157232, -0.020843184, 0.008631886, -0.028002005, -0.014909622, -0.014592982, -0.030672796, -0.024064653, -0.018034723, -0.07692979, -0.035986844, -0.0055790623, 0.006429172, 0.04793657, -0.0061228573, 0.008039907, 0.00021984121, -0.009884681, -0.011275143, 0.025055874, 0.034114536, -0.037198335, -0.03631725, 0.007936655, 0.018736837, -0.010924086, -0.005513669, 0.022137279, -0.0370882, -0.023651645, -0.047853965, 0.0065048905, -0.048074238, 0.047551095, 0.01654789, -0.0051109856, -0.04989148, -0.033756595, 0.050442155, -0.04347607, -0.024243623, -0.02036134, -0.029213497, 0.015171194, 0.019218681, -0.0011882611, 0.023679178, -0.042815257, -0.012734442, 0.0018963993, 0.009492322, 0.019177381, 0.007654432, -0.01990703, -0.05146091, -0.0031285426, -0.002696604, 0.049808875, -0.02639127, 0.030094583, -0.011034221, -0.021159824, -0.011371512, 0.02866282, -0.072579436, -0.033976868, 0.013828916, 0.03328852, -0.038988043, 0.045761388, 0.023266168, -0.004422637, -0.043007996, 0.008432265, 0.0074823448, -0.04149363, 0.012094278, 0.0119841425, -0.0632179, -0.04810177, 0.0093133515, 0.00470486, -0.024092186, -0.024105955, -0.022027142, -0.009038012, -0.010352757, -0.003920143, -0.0354637, -0.030067049, -0.039098177, -0.049313266, 0.003255887, -0.023417605, -0.0031939356, 0.004797787, -0.05333322, -0.049863942, -0.007578714, -0.0153776985, -0.0014739255, 0.031030737, -0.005086893, 0.008714488, 0.03169155, -0.022908228, 0.0091068465, 0.007324025, -0.037693948, -0.033976868, 0.024739234, 0.0001974699, 0.08816364, -0.004315943, 0.005795892, -0.051405843, 0.090035945, 0.0146205155, 0.012906529, -0.01188089, 0.019480255, -0.010917203, 0.012142463, -0.019438954, 0.019438954, -0.009237633, -0.014716884, -0.0055997125, -0.036785327, -0.015997212, 0.017525345, -0.06690744, 0.047275756, 0.06453953, -0.08045414, -0.0033935567, -0.04793657, 0.017084802, -0.022206113, 0.02666661, 0.053195547, 0.0010652188, 0.04554112, -0.018626703, 0.012734442, 0.03105827, 0.051158037, -0.043999217, 0.0011383558, -0.045926593, 0.018612936, 0.030755397, -0.010098068, -0.039015576, -0.022619123, -0.017057268, 0.028332412, -0.010855251, 0.019879496, -0.015529135, 0.037749015, 0.0016133161, 0.03587671, 0.003190494, 0.002605398, 0.028332412, -0.031856753, -0.031498812, 0.027231054, 0.047165617, 0.06575102, -0.042815257, -0.008652537, -0.0045706322, 0.010043, -0.040144466, -0.019920796, 0.037473675, -0.01899841, 0.036923, -0.02289446, -0.0005601433, -0.016947133, 0.04793657, -0.011701919, -0.031003203, -0.019039711, -0.02089825, -0.052755006, -0.047440957, -0.038988043, -0.020567844, 0.022508986, -0.004887272, -0.03887791, -0.02838748, -0.016217483, -0.020733047, -0.039979264, 0.04108062, -0.029901845, -0.011701919, 0.0094854385, -0.006418847, -0.0010617771, 0.0008686094, 0.011151241, 0.022412617, 0.01868177, -0.036923, 0.021875706, -0.043641277, 0.03469275, -0.038519964, -0.03056266, -0.015859542, -0.023018364, 0.0037446143, -0.038299695, 0.0680088, 0.041961707, -0.025248611, 0.0021063455, 0.0029564556, 0.031443745, 0.009836496, 0.019893263, -0.020526543, -0.0246291, 0.0015763174, -0.028442549, 0.03141621, -0.0039855363, -0.030259786, -0.0415487, -0.011681269, 0.04892779, -0.030948136, 0.005128194, -0.032517567, -0.029598972, 0.010290805, 0.027437558, -0.0068903654, -0.0309206, -0.012397151, -0.018131092, 0.017442744, 0.0026105605, -0.007909121, -0.007358442, 0.028635286, 0.033426188, 0.07522269, -0.006098765, -0.027795501, -0.0006255364, 0.0726345, 0.022798093, 0.025799291, -0.07125781, 0.007031477, 0.014675584, -0.008680071, 0.007929771, 0.011185658, -0.0048149955, 0.0200447, 0.033233453, 0.030067049, -0.025193544, 0.025744222, 0.016355153, -0.011433464, 0.0033797896, 0.06734799, -0.021503998, -0.0416313, -0.0033023504, -0.00401307, 0.017263772, 0.01437271, 0.016850764, 0.01265184, 0.01877814, 0.009843379, -0.031911824, -0.0016313852, -0.0029960356, 0.0077439174, -0.027781734, 0.0018034723, -0.0030717538, -0.00640508, -0.073570654, -0.010807067, -0.060464505, 0.00016251471, -0.011481648, 0.05881247, 0.0036723376, -0.020058466, -0.050524756, -0.0038960509, 0.0116399685, 0.036537524, 0.032903045, -0.0013310934, -0.05980369, -0.00019123174, -0.008191343, -0.016933367, -0.019025944, 0.027974471, -0.020595377, 0.0697159, -0.0463396, 0.059197944, 0.05137831, 0.041135687, 0.043999217, 0.018888274, -0.03744614, 0.007317141, 0.012693141, 0.02036134, -0.001860261, -0.006498007, 0.014235041, -0.002287037, 0.09675422, 0.027368724, 0.038244627, -0.031498812, 0.02880049, 0.004866622, 0.023871915, 0.051268175, 0.036950532, 0.027533928, 0.003348814, -0.092073455, -0.01249352, -0.012892762, -0.023926983, 0.007227656, 0.014441545, -0.0009645479, 0.020967087, -0.0371708, -0.029130897, 0.053581025, 0.04240225, -0.019480255, 0.010387175, -0.036895465, 0.0246291, 0.044054285, 0.0092927, -0.042815257, -0.0007055569, -0.021600366, 0.026831813, 0.004918248, -0.02022367, 0.017690549, 0.02063668, -0.06250202, 0.02085695, -0.04870752, -0.041603766, -0.0032782583, -0.017497811, 0.010986038, -0.035794105, -0.021435162, 0.041053087, -0.01340214, 0.06624663, -0.022880694, 0.008666304, -0.001820681, 0.03174662, -0.010283922, 0.012865228, 0.050029147, -0.023706712, -0.013051081, 0.020306272, 0.00031083223, 0.032269765, -0.012961596, -0.017938355, -0.03372906, 0.002858366, 0.005978304, 0.009987933, 0.036757793, -0.0118120555, -0.016451523, 0.040061865, 0.0022818742, 0.06977098, 0.0370882, -0.036565058, 0.06420912, 0.027052084, 0.012748209, -0.0077645676, -0.005327815, 0.019397652, -0.003641362, 0.025689155, -0.015845776, 0.05627935, -0.0012106324, 0.037225872, 0.020333806, 0.021806872, -0.017511578, 0.0027602762, -0.027465094, -0.008569935, 0.08761296, -0.005565295, 0.026969483, 0.06007903, 0.028855557, -0.0015298539, 0.04207184, -0.043090597, -0.024298692, -0.037198335, 0.014730652, -0.0025451675, 0.03450001, -0.013532925, 0.025689155, 0.00039149803, 0.017470278, -0.016671794, 0.02085695, -0.019783128, 0.004395103, -0.033260986, -0.053168014, -0.029433768, -0.017800685, 0.0014980178, -0.02032004, 0.022908228, -0.008549285, 0.0144140115, 0.05046969, -0.03887791, -0.015749406, 0.012789509, -0.012679374, -0.032903045, 0.0050524757, -0.0048803887, 0.028304879, -0.011474765, -0.0047805784, 0.0046325834, 0.033260986, 0.06756826, 0.002245736, 0.012741325, -0.0004001024, 0.022371316, -0.014427778, 0.019962098, -0.026818046, 0.016768163, -0.01591461, -0.014001003, -0.025523951, -0.0026725119, -0.03309578, 0.03160895, -0.014400245, 0.01881944, -0.0045362148, -0.020967087, 0.02729989, 0.041300893, 0.054875117, 0.0031664018, 0.015240029, 0.025826825, -0.01877814, 0.027781734, 0.035078224, 0.024326226, -0.03309578, -0.0069557587, -0.021104757, -0.0028618076, -0.0072620735, 0.0054035336, -0.03728094, 0.0015539461, 0.033068247, -0.023748012, -0.0143451765, 0.032600172, 0.010297689, -0.004636025, -0.0073790927, -0.00802614, -0.04198924, 0.010442242, 0.04030967, -0.0019153289, 0.022522753, -0.012713791, -0.048900258, -0.045486048, 0.009203215, -0.0036861047, 0.01049731, -0.039566256, 0.029406235, 0.003627595, 0.0231285, -0.027286123, 0.0029426885, 0.010118718, 0.014276342, -0.016382687, -0.01203921, -0.037308473, -0.009354652, 0.0038857255, -0.015295097, 0.005059359, 0.008088091, 0.003230074, -0.02666661, 0.0008389244, 0.023238635, -0.007819636, 0.0058544013, 0.016947133, 0.01231455, 0.011233843, -0.039098177, 0.00816381, 0.0042746426, -0.009223866, 0.017814452, -0.008941643, 0.010208203, -0.017360142, 0.020292506, 0.026542706, 0.031278543, -0.0030493825, -0.021779338, -0.0033264428, 0.008645654, -0.019438954, -0.02852515, -0.045816455, 0.00010529577, -0.00830148, 0.021214891, -0.028043306, -0.023183567, -0.013333305, -0.0146205155, 0.057380706, -0.0059473286, 0.023527741, -0.024381293, 0.004663559, -0.0065427497, -0.02285316, -0.009712593, -0.0045121224, -0.010056768, -0.03337112, -0.028332412, 0.016699327, 0.030314855, -0.020526543, 0.04589906, -0.002791252, 0.01836513, -0.08139029, 0.012204413, 0.03995173, 0.020347573, -0.023087198, 0.032545105, -0.009299584, -0.03345372, -0.00610909, 0.003692988, -0.019535322, 0.017607948, -0.04375141, -0.03056266, 0.011929074, -0.010146253, 0.009836496, -0.021352561, 0.0031164964, 0.032682773, -0.021008387, -0.03860257, 0.022343783, -0.0046807677, -0.0042161327, -0.009099963, 0.004728952, 0.04457743, -0.00026307808, -0.06007903, -0.020292506, 0.0072414232, -0.015666805, -0.017897053, -0.0010187554, -0.018282529, -0.021338794, 0.041025553, 0.024601564, -0.012328316, -0.007881586, 0.0010764045, -0.013519159, 0.0039545605, 0.0007395441, 0.009017361, -0.013298887, -0.027506394, 5.4745193e-05, -0.05124064, -0.01296848, -0.0056169215, -0.014007886, -0.0020788116, -0.013147451, -0.018489033, 0.017938355, 0.010352757, -0.023982052, 0.01773185, 0.019494021, 0.010751999, 0.014427778, -0.04022707, -0.019039711, 0.0077163833, -0.033398654, 0.031361144, 0.021627901, -0.009919098, 0.024009585, -0.028332412, -0.015047291, 0.005940445, 0.0035019715, -0.01587331, -0.021022154, 0.043090597, 0.0020788116, -0.04675261, -0.0010239179, 0.012713791, -0.005875052, -0.03364646, 0.06062971, 0.0037927986, 0.008101858, -0.03967639, -0.014909622, 0.045926593, 0.019383885, 0.019920796, -0.015639272, 0.009664409, -0.011991026, -0.028153442, 0.014262575, -0.0094166035, -0.034169603, 0.0041163224, 0.024229856, -0.0053243735, 0.034995623, -0.01140593, 0.05556347, -0.033756595, 0.025799291, -0.00024737514, -0.010944736, 0.004746161, -0.0012708629, 0.013911517, -0.07191862, -0.007296491, 0.0023851264, 0.02085695, 0.009258283, -0.028965693, -0.0018034723, -0.027726665, -0.013326421, -0.030122118, 0.021435162, -0.0036482455, -0.0123627335, -0.014179973, -0.016341386, -0.10182047, 0.013822032, 0.037336007, -0.03885037, -0.029543905, -0.021531532, 0.028139675, 0.007227656, -0.043365937, 0.0042195744, 0.01808979, -0.0056891977, 0.0044467296, -0.006260527, 0.010896552, -0.031911824, 0.011543599, -0.043063063, 0.02040264, -0.056609757, -0.021435162, -0.011440347, -0.026088396, 9.6745185e-05, -0.034114536, -0.0047014183, -0.025647854, -0.00324212, -0.004264317, -0.0057339403, -0.027354958, 0.010738232, -0.0061607165, -0.0076888492, -0.0119841425, -0.007578714, -0.015088593, -0.0025985145, -0.0184615, -0.026267366, 0.025468884, -0.015005991, 0.0015049012, -0.013546692, 0.01899841, -0.02662531, 0.0048218793, -0.005262422, 0.021146057, 0.013340188, 0.017291307, 0.02534498, 0.0042677587, -0.0124659855, 0.006126299, -0.0044157538, 0.0039786524, 0.007014268, 0.0072620735, 0.04521071, 0.02113229, 0.009994816, 0.011350862, -0.025579019, 0.007902238, -0.019989632, -0.01632762, 0.01804849, -0.021627901, -0.008597469, -0.026790513, 0.017456511, -0.012796393, 0.008769556, 0.00470486, 0.008459799, 0.016740628, -0.007874703, -0.022412617, 0.031388678, 0.07081726, -0.030727863, 0.0031337051, 0.049368333, -0.005964537, 0.015515368, 0.020567844, -0.00015606145, -0.0037652648, 0.03573904, 0.01573564, -0.031664018, -0.012913412, 0.018475266, -0.021793105, -0.012562355, 0.023665411, 0.036757793, 0.013911517, 0.0036379201, -0.00073567213, 0.0292961, -0.010456009, -0.021476464, -0.015143661, 0.029764175, -0.0024126605, 0.021146057, -0.013822032, 0.0008544122, 0.028222276, 0.015666805, 0.008865925, 0.046780143, 0.0015238308, 0.017580412, 0.04782643, -0.020086, 0.017718082, -0.018750604, 0.021669202, 0.03560137, -0.005744266, 0.0026208858, -0.0416313, 0.033068247, -0.029268565, 0.011935958, 0.017539112, -0.009843379, 0.031388678, 0.00092840963, -0.027575228, -0.01464805, -0.0048941555, 0.0017053827, 0.0096919425, -0.02026497, -0.013051081, 0.0059989546, -0.022027142, 0.014262575, 0.032490034, -0.014592982, -0.014124905, -0.040502407, 0.00207537, -0.01641022, -0.028139675, -0.0123765, 0.007585597, -0.009327118, -0.01328512, 0.009512972, -0.013994119, 0.0023059666, 0.031719085, 0.0009955235, -0.010820834, 0.0037377307, -0.0034899253, 0.0185028, 0.042429782, -0.00057132903, 0.012059861, -0.03981406, -0.012479753, -0.0035828524, 0.00820511, 0.004240225, -0.04843218, 0.0023661968, 0.003164681, 0.0062949443, 0.016740628, 0.0323799, 0.0051488443, 0.0070899865, -0.0122938985, 0.012231948, -0.030039515, 0.042925395, -0.017153637, -0.013381489, 0.017181171, -0.004914806, 0.006935108, 0.021008387, 0.004918248, -0.028415013, -0.0012089115, 0.033811662, 0.004956107, 0.05473745, -0.024794303, 0.004453613, 0.025606554, -0.012500403, 0.0063224784, -0.025757989, -0.068724684, 0.020843184, -0.0060402555, 0.00073782325, 0.014076721, 0.0371708, -8.2601786e-05, -0.004161065, -7.512675e-05, 0.026831813, -0.028180975, 0.017965889, -0.014978457, 0.0045603067, -0.02866282, -0.014854554, -0.01172257, 0.005644455, 0.00044828677, -0.036124513, -0.050937768, -0.010387175, -0.027781734, 0.030011982, 0.0055205524, 0.017800685, 0.026914414, -0.00895541, 0.016134882, -0.0019170498, -0.02435376, -0.030782932, 0.002417823, 0.0024487986, -0.012734442, 0.020113535, -0.051984057, -0.0088315075, -0.027685365, 0.011557367, 0.027258588, 0.0030321737, 0.0051041017, 0.0025468883, -0.013064849, -0.031719085, 0.021917006, -0.0007447067, -0.00258991, -0.0043778946, -0.017759383, 0.008776439, 0.04149363, -0.04248485, 0.026831813, -0.021462698, 0.017759383, 0.028828023, 0.015074826, 0.01786952, -0.006601259, 0.005424184, 0.0015272725, -0.04064008, -0.0054448345, -0.009678176, -0.0054551596, -0.005723615, 0.027520161, -0.03975899, -0.013415907, 0.033866733, 0.001415416, 0.036675192, -0.007165705, -0.025813058, -0.0030924042, -0.009210099, 0.006680419, -0.007441044, 0.018805673, 0.027533928, -0.006446381, 0.00830148, 0.0052280044, -0.016437756, -0.0056891977, -0.02263289, 0.00054809725, 0.0050834515, 0.018736837, 0.0008931318, -0.009829612, -0.0012820485, -0.012817044, -0.020980854, 0.025234845, 0.024505196, 0.020609146, -0.031774152, -0.0038031237, 0.011646852, 0.054847583, -0.050139282, -0.0019497463, 0.032627705, 0.04322827, -0.039208315, 0.014042303, -0.004181715, -0.02045771, 0.01646529, 0.0007210447, 0.007819636, -0.02063668, -0.017924586, -0.012878994, -0.014049186, 0.0037067551, -0.016892064, -0.00725519, 0.011013571, 0.0153776985, 0.014703117, 0.008404732, -0.02077435, -0.0015255517, 0.01654789, 0.022522753, 0.0012140741, 0.024133489, 0.008019256, -0.021490231, -0.017112337, -0.011522949, 0.027712898, -0.0323799, 0.0073515587, 0.017979655, -0.01605228, -0.025537718, 0.017387675, -0.013553576, -0.017621715, 0.038712703, -0.009464787, -0.0051213107, -0.008088091, 0.015708106, 0.021490231, 0.0832626, -0.039566256, -0.01895711, 0.009444137, 0.012114928, 0.015501602, -0.014537914, -0.0067974385, -0.05853713, -0.012658724, -0.019494021, 0.03681286, 0.004360686, 0.005902586, 0.024229856, 0.0014300434, -0.016107349, 0.008852158, 0.06690744, -0.0063534537, 0.022440152, 0.043999217, -0.023348771, 0.009203215, -0.010896552, 0.03328852, 0.008576819, -0.004494914, 0.0044054287, 0.031443745, 0.015019758, -0.030893067, -0.010669397, -0.027754199, -0.04589906, 0.015584203, -0.001029941, 0.01864047, -0.010958503, 0.013794498, -0.034665216, 0.01326447, 0.021435162, -0.02103592, 0.016878298, 0.015680572, 0.009161915, 0.0050008497, 0.022825627, -0.0184615, -0.03573904, 0.0002026325, -0.031581417, -0.031719085, 0.047083016, -0.06035437, -0.0067354874, 0.007441044, -0.010263272, 0.006632235, -0.022013376, 0.041658834, -0.023142267, -0.017621715, -0.006749254, -0.046229467, -0.025551485, 0.0065737255, -0.029185964, 0.00725519, -0.043558676, -0.0024505197, 0.010765766, -0.005035267, 0.0098640295, -0.0071794717, 0.007124404, -0.023348771, 0.0354637, -0.005685756, 0.012658724, -0.033316053, -0.05236953, -0.024931971, -0.0013775568, -0.0011056593, 0.044962905, -0.0025761432, -0.028497616, 0.0024143814, -0.046532337, 0.018668003, 0.036923, 0.06239188, 0.005076568, 0.0024952623, 0.020595377, 0.010979154, -0.004123206, -0.040887885, 0.0022508986, 0.0032765374, -0.006559958, -5.8281043e-06, -0.027836801, -0.029984448, -0.048349578, 0.0010385454, 0.0092514, 0.0023335004, 0.033068247, -0.00168043, 0.014744419, -0.005623805, -0.01672686, -0.0072758403, -0.022564054, 0.026060862, -0.005978304, -0.053718694, 0.0056306883, 0.003562202, -0.005940445, 0.013746314, 0.0039270264, 0.008514867, 0.012025443, -0.0040956717, -0.0115711335, 0.026597774, 0.0012140741, 0.016492823, 0.017759383, 0.030810466, 0.01881944, -0.026115932, -0.007055569, -0.022385083, 0.0019979307, 0.0069419914, 0.00072147493, 0.027506394, -0.034582615, -0.022977063, -0.0016098743, 0.0063534537, -0.008755789, -0.0032128652, 0.01660296, 0.0016649422, -0.029571438, -0.019232448, 0.016795697, 0.006487682, -0.009395953, -0.0042608753, -0.014152439, -0.020980854, 0.0020512776, -0.02089825, 0.022701724, -0.009974166, 0.0029908728, -0.008920993, -0.0012872111, 0.011756987, -0.012947829, -0.00022801534, 0.019025944, -0.00863877, 0.0003099718, -0.027726665, -0.016768163, 0.010566145, -0.0072896075, -0.00064962864, -0.004054371, 0.041053087, 0.004174832, 0.032490034, -0.014524147, -0.018117325, -0.0088315075, -0.006047139, -0.014166206, 0.027602762, 0.01203921, -0.0048975972, -0.0048322044, -0.0051144273, -0.02498704, -0.034087002, -0.019549089, 0.0010970549, 0.027382491, 0.039208315, -0.03728094, -0.008803974, 0.025592785, 0.006339687, 0.0038960509, -0.021545298, -0.011144358, -0.015997212, -0.018943343, 0.024642866, 0.0013044198, -0.027754199, 0.0036000612, 0.0055756206, 0.007324025, -0.002287037, 0.0049939663, -0.0124659855, 0.026556473, -0.021682968, 0.021517765, -0.0065634, -0.019053478, -0.011343978, -0.034114536, -0.005007733, 0.034582615, 0.005861285, 0.00081956456, 0.0092445165, 0.002870412, -0.015694339, -0.023224868, 0.0047530443, 0.018874507, -0.023706712, -0.015804475, 0.017057268, 0.0058819354, 0.0147581855, -0.04689028, 0.012720675, 0.010683164, 0.015832009, -0.023018364, 0.028125908, 0.003309234, 0.019549089, -0.03372906, 0.060464505, 0.005035267, 0.017952122, -0.0024625657, -0.0046429085, 0.041741434, -0.0073446753, -0.023472674, -0.010366524, -0.026859347, -0.0541317, 0.0066115847, -0.014469079, -0.024133489, 0.026060862, -0.046504803, -0.039180778, -0.008618119, -0.0021562506, 0.0059714206, 0.023665411, -0.009767661, -0.05305788, -0.038464896, 0.032407433, 0.0047599277, 0.015446533, -0.010146253, -0.02294953, -0.0009043174, 0.028332412, 0.027134687, -0.023541508, -0.0032490036, -0.001627083, -0.008377197, -0.0200447, -0.015611737, 0.009843379, -0.012782626, 0.035794105, -0.0141386725, 0.01808979, 0.016740628, -0.037804082, 0.02512471, 0.028717887, 0.0017776593, 0.013326421, 0.0036998715, 0.013526042, 0.023321237, -0.01646529, -0.024105955, 0.0056891977, -0.042044308, 0.007957305, 0.00508001, -0.029048294, -0.0011409371, 0.029846778, -0.023982052, 0.02190324, -0.017814452, 0.0061469493, -0.002524517, -0.014248808, 0.012837694, 0.01142658, 0.004226458, 0.041741434, -0.007441044, -0.007916004, -0.01235585, 0.023596575, -0.02860775, 0.017965889, 0.0020375107, 0.009010478, 0.011715686, -0.0031612392, -0.0046257, 0.0035794107, -0.028139675, -0.030177185, 0.00800549, 0.03799682, 0.024009585, -0.016066046, 0.016630493, -0.022564054, -0.025042107, -0.0140698375, 0.026735444, -0.019948332, -0.0028686912, -0.014730652, -0.003094125, -0.015047291, -0.00034976692, 0.039208315, -0.012603655, -0.0006440358, 0.0025520509, 0.006191692, -0.0057786833, -0.010332107, 0.01016002, -0.0032662123, 0.017841985, -0.011935958, 0.025565252, 0.026405036, -0.0028514825, -0.013608644, -0.03174662, -0.029378701, -0.013057966, 0.04421949, -0.0054035336, 0.027465094, -0.004594724, -0.045871522, -0.017663015, 0.0044398457, -0.01990703, 0.0292961, 0.022646656, -0.016988434, -0.011894657, -0.010001699, -0.006935108, -0.029543905, 0.021380095, -0.057876315, 0.034720283, -0.0014592982, 0.009361535, 0.011997909, 0.035656437, -0.01668556, -0.027533928, -0.00091894483, -0.006219226], "content_hash": "43ac36dd32554e694ad9b0255a3ba4c6"}, {"content": "prompt ### Step 6: Compare Results Between Prompts | Category | Simple Prompt Impact | Complex Prompt Impact | |----------|---------------------|----------------------| | Previous-Token | | | | First/Positional | | | | Bag-of-Words | | | | Syntactic | | | | Other | | | ## What You Should Learn\n\n- No single head category is always the \"most important\" -- it depends on the prompt\n- Simple prompts tend to rely more on Previous-Token heads (local patterns)\n- Complex prompts with grammatical dependencies may rely more on Syntactic heads\n- Some heads are redundant for certain inputs but critical for others\n- Ablation is most informative when you compare across conditions (categories, prompts, or both)\n\n## Advanced Challenge\n\nTry ablating **two heads simultaneously** from the same category. Does removing two Previous-Token heads have a bigger effect than removing one? Or does the model have enough redundancy to compensate?", "source_file": "experiment_comparing_heads.md", "chunk_index": 2, "embedding": [0.032853976, 0.00626801, 0.026533764, 0.012689153, 0.018974563, 0.004600949, 0.03808834, 0.024278533, 0.024529114, 0.050311133, 0.03274261, -0.02425069, -0.021020977, -0.0022743721, -0.012793561, -0.00697799, -0.0046601137, 0.002331797, 0.005961744, 0.06921609, 0.011937409, -0.016719334, 0.024932828, 0.044464238, 0.038338922, -0.015689166, -0.0015234988, 0.0340512, 0.035415474, 0.014798211, 0.0054919044, -0.020714711, -0.01621817, -0.00579469, -0.034608044, 0.029763475, 0.006591677, 0.02401403, 0.018264584, 0.005039466, 0.018194977, -0.014714683, -0.039313402, -0.01195133, -0.019225145, 0.040371414, 0.044659134, -0.03939693, -0.01585622, 0.032798294, -0.04362897, 0.050255448, 0.017652052, 0.011582419, 0.0034976956, -0.010433922, -0.03831108, -0.018737905, 0.013670595, -0.014178718, 0.0012790081, -0.04555009, 0.029791318, 0.017234417, 0.014136955, -0.045494407, -0.002850361, -0.010503528, 0.0064176624, 0.01953141, 0.02934584, 0.05122993, 0.03822755, 0.007475672, -0.020547656, 0.008067322, 0.020840002, 0.01571701, 0.042654485, -0.025893388, -0.008965238, 0.004200715, 0.026241418, -0.03783776, -0.022858573, -0.020408446, -0.03961967, -0.017777342, -0.028677624, -0.060974754, -0.01490958, 0.005704202, 0.01419264, 0.021160189, 0.0019298231, -0.0034976956, 0.008853869, -0.0083457455, -0.02200938, 0.052677732, 0.0134965805, -0.043963075, -0.036724064, 0.01773558, 0.015062713, -0.024306376, 0.017958319, 0.0089304345, -0.009598651, -0.029178787, -0.020310996, 0.012626507, -0.033550035, 0.04507677, 0.019475726, -0.025921231, -0.050311133, -0.009368952, 0.029791318, -0.07071958, 0.010440882, -0.031851653, -0.02745256, -0.008700736, 0.0111160595, -0.010106774, 0.029485052, -0.046301834, -0.021466454, -0.024292454, 0.01644091, 0.018181058, 0.0010327772, -0.0012850986, -0.034942154, -0.007239012, -0.032464184, 0.04841785, -0.040176515, 0.04493756, 0.006494229, -0.020436287, 0.00017042566, 0.03430178, -0.053178895, -0.02352679, 0.005070789, 0.03218576, -0.026157891, 0.035081364, -0.0104826465, -0.010197262, -0.034747258, -0.025475753, -0.024640484, -0.03769855, -0.006281931, -0.0019611458, -0.043239176, -0.049865656, 0.03160107, -0.02638063, -0.0165662, -0.008860829, -0.031211277, -0.0011537175, -0.034969997, -0.00043938713, -0.047749635, -0.03513705, -0.02494675, -0.0356939, -0.026589448, -0.06548522, 0.0011867803, -0.0007634895, -0.04886333, -0.042988595, -0.027703142, -0.009452479, -0.00068561797, 0.027021004, 0.017958319, 0.041457266, 0.02283073, -0.016524436, -0.006581236, 0.022301726, -0.061086122, 6.737632e-05, 0.025239093, 0.029485052, 0.08742499, 0.018946722, 0.006553394, -0.04864059, 0.05084014, 0.030376008, 6.8300775e-05, -0.0089304345, -0.0038666069, -0.005095151, 0.0029199668, -0.020742554, 0.022914257, -0.01313463, -0.021188032, -0.0069257854, -0.053652216, -0.03522058, 0.008255258, -0.058357574, 0.012250636, 0.05086798, -0.06754555, -0.022969943, -0.048919015, 0.0021473416, -0.010670582, 0.010343434, 0.06348056, -0.0014373615, 0.063703306, -0.012863168, -0.016594043, 0.042515274, 0.041930586, -0.042431746, -0.009591691, -0.03939693, 0.03987025, 0.021257637, -0.017164811, 0.0034750737, -0.047638267, -0.016747175, 0.012988458, -0.031155594, 0.014234403, -0.03143402, 0.018765746, -0.0004139375, 0.041568633, -0.008387509, -0.0064281034, 0.016399147, -0.026464157, -0.029986214, -0.0031200838, 0.040983945, 0.07634373, -0.01644091, -0.03808834, 0.012132306, -0.00851976, -0.017206574, 0.007872425, 0.046218306, -0.046942208, 0.01987944, -0.024988512, 0.028065093, 0.009842272, 0.0544318, -0.036000162, -0.043684654, -0.017652052, -0.004520902, -0.037726387, -0.042515274, -0.049837813, -0.033661406, 0.03597232, -0.019322593, -0.023582473, -0.028566254, -0.019183382, -0.026909634, -0.025782019, -0.00721813, -0.022329567, -0.016051117, 0.01915554, -0.0040267003, -0.012153188, -0.005157796, 0.026074365, 0.041429423, 0.02756393, -0.027410798, -0.007997716, -0.058580313, 0.032798294, -0.04627399, -0.0071485243, -0.014199601, -0.0075313565, 0.0077053714, -0.008436233, 0.03087717, 0.038338922, -0.00532833, -0.005665919, -0.0018410756, 0.044380713, -0.01076107, 0.003323681, 0.0017705996, -0.02945721, -0.009807469, -0.028357437, -0.010197262, -0.011860843, -0.010468725, -0.023707764, -0.034468833, 0.0153829, -0.02331797, 0.023666, -0.013872453, -0.034273937, 0.004315565, 0.026686896, 0.007503514, 0.006964069, 0.009793548, -0.033633564, 0.008109085, -0.014798211, -0.0007782808, -0.03535979, -0.005015104, 0.018946722, 0.059359897, -0.032130077, -0.024529114, -0.004994222, 0.04671947, 0.0024518673, 0.050171923, -0.05654782, 0.0032801772, 0.012891009, 0.018529085, -0.0134478565, 0.050338976, 0.0057668476, 0.01857085, 0.032937504, 0.027870195, -0.021870168, 0.0071833273, 0.013733241, -0.00037282653, 0.0041589513, 0.05418122, -0.041986268, -0.05075661, -0.009069647, 0.011937409, 0.03157323, 0.0036473484, 8.34183e-05, 0.016162487, 0.012034858, -0.0066682436, -0.010670582, -0.017958319, -0.015536034, 0.0067726523, -0.014505866, 0.013475699, -0.0147842895, -0.028106855, -0.066877335, 0.010935084, -0.07394929, -0.013942058, 0.011902606, 0.036612693, 0.0045452644, -0.062422555, -0.043963075, -0.0053666136, 0.02046413, 0.027814511, 0.04708142, -0.012215832, -0.071721904, -0.031851653, 0.006946667, -0.016580122, 0.009355031, 0.020394525, -0.029123101, 0.062032763, -0.025962995, 0.06615343, 0.033661406, 0.059694007, 0.024069715, 0.026784344, -0.051870305, 0.008944356, 0.030014057, 0.014352733, 0.03143402, -0.010252947, 0.008798184, 0.012494256, 0.08035303, 0.010489606, 0.02887252, -0.00402322, 0.0049246163, -0.012487296, -0.004764523, 0.023624238, 0.029568579, 0.020324918, 0.024515193, -0.067712605, -0.023457183, -0.04284938, -0.015828378, -0.012424651, 0.03513705, -0.010440882, 0.01822282, -0.02768922, -0.024306376, 0.0307658, 0.06993999, 0.0071485243, 0.011317917, -0.022663675, 0.007468711, 0.06420447, 0.0066195196, -0.03491431, 0.023638159, -0.02459872, 0.018000081, -0.006803975, -0.044185814, 0.023331892, 0.027062768, -0.057410933, 0.01598151, -0.017471077, -0.033410825, -0.004423454, -0.019698465, 0.03243634, -0.012570823, -0.015368979, 0.031378333, -0.00709632, 0.09026491, 0.0029112662, 0.010837636, -0.011373602, 0.03196302, -0.021076662, 0.034802943, 0.077624485, -0.01313463, -0.010949005, 0.032714766, 0.001218103, 0.02507204, -0.023930503, 0.00040306157, -0.026297104, -0.0074478295, -0.014603314, 0.0036160257, 0.02685395, 0.00030648342, -0.038366765, 0.017192652, -0.023596395, 0.053401634, 0.013273842, -0.03728091, 0.077067636, 0.035721738, 0.0011989614, -0.025879467, -0.00010381065, 0.01621817, -0.003925772, 0.027758826, -0.01609288, 0.03174028, 0.002352679, 0.034246095, 0.03574958, 0.046051253, -0.016037196, 0.007823701, -0.004357328, -0.0008783392, 0.07701195, 0.0009214079, 0.019726308, 0.06710007, 0.0076636076, -0.0062401677, 0.037336595, -0.05415338, -0.012194951, -0.01632954, 0.025754178, -0.0125499405, 0.033661406, -0.024640484, 0.033550035, 0.016705412, 0.021675272, 0.002424025, 0.01749892, -0.03207439, -0.011575459, -0.014164797, -0.040399253, -0.011791237, -0.043601125, -0.027577851, 0.0021577824, 0.0054362197, -0.02898389, 0.008770341, 0.036835432, -0.024556955, 0.0024936309, 0.0066055986, -0.0011537175, -0.031628914, -0.004413013, 0.021327242, 0.0072946968, -0.007879386, -0.015522112, 0.009208858, 0.023011705, 0.06910472, -0.008728578, 0.017958319, 0.018250663, -0.00061296683, -0.007628805, 0.017345786, -0.025517518, 0.002801637, -0.019893361, -0.02035276, -0.018723983, 0.0022621911, -0.02567065, 0.009055725, -0.03655701, 0.023401499, -0.017944397, -0.013677556, 0.036389954, 0.04092826, 0.040510625, 0.007503514, -0.004687956, 0.014394497, -0.009076607, 0.02873331, -0.009410716, 0.015786614, -0.014095192, -0.0034211292, -0.008819065, 0.013120709, -0.021842327, -0.0005859945, -0.043127805, 0.0017775602, 0.03499784, -0.022259962, -0.011763395, 0.048222955, 0.003348043, -0.026352787, 0.013858532, -0.014895659, -0.044074446, 0.024932828, 0.03430178, -0.024181085, 0.026199656, 0.0034733335, -0.035443317, -0.029373683, 0.0054362197, 0.0030017537, -0.0044860994, -0.012466414, 0.023039548, -0.0013425235, 0.021814484, -0.022162514, 0.02271936, 0.005220441, 0.018835353, 0.021132346, -0.010684503, -0.022552306, -0.0074965535, -0.0067865737, -0.00063776394, 0.009424637, 0.008686814, 0.0014591133, -0.039118506, -0.0037935206, 0.013378251, -0.019712387, 0.010440882, 0.03452452, 0.017345786, 0.013830689, -0.04944802, 0.022385253, -0.0044164932, -0.020881765, 0.02128548, 0.0038666069, 0.008721617, -0.0033393423, 0.010496567, 0.008568484, 0.034023356, 0.005098631, -0.019962968, -0.016942073, 0.01076107, -0.023749528, -0.05039466, -0.044269342, -0.010378237, 0.0077332137, 0.027410798, -0.033772774, -0.012828364, -0.0033306417, -0.03455236, 0.057076823, -0.0028729828, -0.002377041, -0.015508191, 0.0006769172, 0.02306739, -0.0418749, 0.014032546, -0.0037413163, -0.018014003, -0.03146186, -0.032826133, 0.0012790081, 0.0058364533, -0.005380535, 0.041234527, -0.01241769, -0.0027476922, -0.056102343, 0.013635793, 0.072668545, 0.046385363, 0.00793507, 0.022232119, 0.0035290183, -0.03204655, -0.028092934, -0.007155485, -0.03800481, 0.023011705, -0.046441045, -0.040427096, -0.0007465231, -0.013176394, -0.0035812226, -0.01787479, -0.0018045325, 0.03204655, -0.018793589, -0.04889117, 0.0340512, -0.0133504085, 0.0076218443, -0.014338812, 0.014185679, 0.044798348, -0.020018652, -0.016844625, -0.027675299, 0.0010710605, -0.027021004, -0.007510475, 0.007026714, -0.026631212, 0.013426975, 0.053290267, 0.012939733, -0.026533764, -0.030376008, -0.0025823782, -0.0051055914, 0.012111424, 0.0076844897, 0.0061357585, -0.025601044, -0.033772774, 0.00043242655, -0.03455236, -0.02614397, -0.02093745, 0.0010040649, 0.030821485, 0.026046522, 0.0073155784, -0.003041777, 0.0044791386, -0.0053909756, 0.01432489, 0.03705817, 0.011930449, -0.004980301, -0.018139293, 0.00032344984, 0.0010458285, -0.022023302, 0.029262314, 0.01432489, -0.0023474584, 0.042320378, -0.026575526, -0.005157796, 0.003099202, 0.027772747, -0.011095178, -0.020046495, 0.046552416, 0.0045383037, -0.03251987, -0.022037223, 0.013266882, -0.0043851705, -0.01811145, 0.043378387, 0.009487282, -0.018194977, -0.016190328, -0.013907256, 0.049726445, 0.010148538, 0.0072807753, -0.009668257, -0.0035673014, -0.013259921, -0.0130163, -0.008617208, -0.023220522, -0.02707669, 0.0050429464, 0.03391199, -0.015396821, 0.031378333, -0.036278587, 0.04635752, -0.03243634, 0.028176462, -0.004781924, -0.03274261, 0.021215873, 0.012689153, 0.019002406, -0.05393064, -0.016385226, 0.009104449, 0.010900281, -0.018501244, -0.025545359, 0.0031479262, -0.033772774, -0.020979213, -0.016148565, -0.005359653, 0.0046113897, -0.009215819, -0.003821363, -0.027842354, -0.13965724, 0.0023161357, 0.03800481, -0.033856302, -0.045355193, 0.018083608, 0.027494324, 8.151501e-05, -0.027118452, 0.011672907, 0.0036995527, 0.001348614, -0.02306739, -0.026199656, 0.03391199, -0.033382982, 0.010183341, -0.03505352, 0.0059791454, -0.056102343, -0.017095204, 0.0013442637, -0.0021943254, -0.0063967807, -0.049949184, -0.011937409, -0.038283236, -0.018083608, -0.00971002, 0.018654376, -0.02756393, -0.0077888984, -0.023443261, 0.00038152726, -0.0151740825, 0.02024139, -0.013907256, -0.026297104, -0.00026080455, -0.013559226, 0.023401499, -0.018125372, -0.020380603, -0.019809835, 0.020297077, -0.035443317, 0.0035411993, -0.024863223, 0.031183437, 0.020380603, 0.009487282, 0.028371358, -0.0054501407, 0.005004663, 0.010301671, -0.018236741, 0.004082385, -0.007005832, 0.027842354, 0.044269342, 0.019294752, -0.016983835, 0.0096821785, -5.394456e-05, 0.0022813329, 0.0030226356, -0.018543007, 0.038144026, -0.0072738146, -0.012480335, -0.014450181, 0.014157836, -0.00922974, 0.0011406664, 0.006407222, 0.005568471, -0.012932774, -0.003182729, -0.037949126, 0.024988512, 0.08063146, -0.031044224, -0.005046427, 0.030014057, 0.008464076, 0.033522192, -0.0005346602, -0.01833419, -0.0025997797, 0.03243634, 0.013510502, -0.02792588, -0.015494269, 0.042181164, -0.046775155, 0.012020936, 0.028816836, 0.025503596, -0.003535979, 0.027243743, 0.0378656, 0.015508191, -0.0033410825, -0.025809862, -0.027730985, 0.0015565616, -0.0071485243, 0.019949047, 0.0088817105, 0.013232078, 0.03335514, 0.015355058, 0.0013277322, 0.055879604, 0.005394456, -0.0010754109, 0.041819215, -0.005819052, 0.025141645, -0.0145893935, 0.008317903, 0.029791318, -0.015396821, 0.012000054, -0.025782019, 0.04841785, -0.018014003, 0.015090556, 0.007064997, 0.015285452, 0.04318349, 0.0094664, -0.026631212, -0.022427015, 0.0056241555, 0.0021351604, 0.00077175524, -0.01845948, -0.005752926, 0.0016218171, -0.013635793, 0.027021004, 0.047220632, -0.015577797, 0.0051334337, -0.028538411, 0.01100469, -0.0111160595, -0.031183437, -0.009125331, 0.014575472, 0.0041554715, -0.013614911, 0.009647376, -0.013071985, -0.0015757032, 0.026672974, -0.012891009, 0.00082395965, 0.032241445, -0.028788993, 0.021870168, 0.04627399, -0.014185679, 0.016621886, -0.029011732, -0.007990755, -0.001762769, 0.0031444458, 0.012612586, -0.04329486, -0.025962995, 0.006341096, 0.0070754383, 0.02128548, 0.039118506, -0.026297104, 0.0024588278, 0.0024449066, 0.009334149, -0.030376008, 0.04084473, -0.023345813, -0.011860843, 0.021856248, 0.029512895, 0.009007001, 0.017373629, -0.004165912, -0.014951344, 0.023150917, 0.014352733, 0.0011493672, 0.036974646, -0.02104882, 0.014561551, 0.024041872, 0.0021508217, 0.014714683, -0.019211223, -0.03775423, 0.018069688, 0.003821363, -0.021591745, 0.0027790149, 0.03505352, -0.0013825468, -0.02601868, 0.006222766, 0.0050359857, -0.030570904, 0.03277045, -0.026742581, 0.0032262327, -0.03204655, 0.018793589, -0.0032610358, -0.0119652515, 0.008116046, 0.0022291285, -0.054932963, -0.0016705412, -0.020951372, 0.006720448, 0.029206628, -0.006542953, 0.036863275, -0.020728633, 0.028482728, 0.009320227, -0.02923447, -0.016594043, -0.014728605, 0.011728591, -0.028106855, 0.001620077, -0.062422555, 0.002105578, -0.011812119, 0.015327215, 0.035081364, 0.0069362265, -0.011512813, -0.0012703073, -0.00061818724, -0.038478132, 0.006929266, 0.016120723, -0.011561537, 0.0030957216, -0.010705385, -0.008554563, 0.040816892, -0.03900714, 0.031545386, -0.01587014, 0.02472401, 0.01809753, 0.023958346, 0.0102668675, -0.004127629, 0.013232078, 0.00130076, -0.036807593, -0.021800563, -0.0073573417, -0.006814416, -0.0047958456, 0.0004306864, -0.04788885, -0.00354816, 0.017916555, -0.010468725, 0.039229877, -0.016761096, -0.023457183, 0.0120139755, 9.937871e-06, -0.015591718, -0.036055848, 0.02710453, 0.023011705, 0.0020481532, 0.00957777, -0.0042389985, -0.024807537, -0.0026363228, -0.027884116, 0.0051334337, -0.009222779, 0.021563902, -0.0067239283, 0.0040997867, 0.012160148, 0.0040162597, -0.023373656, 0.018877115, 0.010607936, 0.020617263, -0.05454317, 0.0067343693, 0.006494229, 0.037420124, -0.03705817, -0.015536034, 0.027953723, 0.022065066, -0.03290966, 0.020756474, -0.013809808, -0.016552279, 0.013329526, -0.007071958, 0.031406175, -0.01574485, -0.007976835, -0.0052343626, -0.030849328, 0.016635805, -0.033160243, -0.010085893, 0.0316846, 0.0064176624, -0.003964055, 0.0077332137, -0.02898389, -0.00039240316, 0.0022152073, 0.0036995527, -0.002199546, -0.002091657, 0.0011284854, -0.0041798335, 0.003501176, -0.021549981, 0.03925772, -0.036250744, 0.008777302, -0.001954185, -0.024278533, -0.03321593, 0.022496622, 0.025280857, -0.020533735, 0.030710116, 0.01147105, -0.010990769, -0.0045452644, 0.009396794, 0.018891037, 0.08057577, -0.04070552, -0.027160216, 0.010447843, -0.00922974, 0.011262232, -0.020199627, -0.011436246, -0.035777424, -0.010044129, -0.036863275, 0.061977077, 0.00013638403, 0.00745479, -0.008290061, 0.010559212, -0.0098283505, 0.03421825, 0.06877061, -0.0019576654, 0.031322647, 0.03299319, -0.026923556, 0.010225104, -0.0073991055, 0.02579594, 0.024278533, 0.008213494, -0.0074339085, 0.018946722, 0.0028486208, -0.035916638, -0.01644091, -0.016287778, -0.022872493, 0.0010745408, -0.020840002, 0.022287805, -0.010343434, -0.00025949944, -0.0051334337, 0.032130077, 0.008095164, -0.031322647, 0.0136427535, 0.013218157, 0.0054571014, 0.015452506, 0.024626562, -0.0075383172, -0.030682273, -0.0037413163, -0.03987025, -0.018264584, 0.015549954, -0.030793643, 0.0036717104, 0.005122993, 0.0032488545, 0.0027111492, -0.026338866, 0.063313514, -0.024682246, -0.009299346, -0.0028242588, -0.042376064, -0.043963075, -0.011109099, -0.022176435, 0.023512868, -0.031656757, -0.027856275, -0.0038874885, -0.0044860994, 0.0042633605, 0.0013312126, -0.005039466, -0.008526721, 0.036946803, -0.013141591, 0.025545359, -0.026533764, -0.050088394, -0.026979242, 0.0032505947, -0.0057633673, 0.012786601, -0.012132306, -0.044185814, 0.009897957, -0.04106747, 0.008860829, 0.03157323, 0.048529223, 0.009758745, -0.0028486208, 0.015048792, 0.009612572, -0.0064211427, -0.019085933, 0.010858518, -0.014826053, -0.0108724395, -0.003539459, -0.036473483, -0.010301671, -0.046301834, 0.018612614, 0.007475672, -0.01574485, 0.021953696, -0.033104558, 0.020840002, 0.0148817375, 0.0071415636, -0.0069675487, -0.023123074, 0.063591935, -0.0046287915, -0.02341542, 0.0011163043, -0.019266909, -0.007301657, 0.004910695, 0.008686814, 0.028259989, 0.033327296, 0.008310943, -0.01088636, 0.019851599, 0.0023143957, 0.024696168, 0.027758826, 0.009097489, 0.019364357, -0.018988485, 0.01170771, -0.020770395, 0.0028120778, -0.007047596, 0.020324918, 0.02745256, -0.026784344, -0.0062958524, -0.0001998994, -0.0065707956, 0.0013068505, 0.0042877225, 0.00934111, -0.0028103376, -0.03204655, 0.0029182266, 0.008596326, -0.008561524, -0.009612572, -0.019099854, 0.0076496867, 0.0043956116, 0.006664763, -0.033884145, 0.031322647, -0.004298163, 0.0026467638, 0.011791237, 0.018863194, 0.030598747, 0.00438169, -0.011700749, 0.018236741, -0.00874946, 0.010273828, -0.00095795095, -0.0018236742, 0.027201978, -0.0044443356, 0.013420015, -0.013294724, 0.03705817, -0.007040635, -0.0050359857, -0.029902687, -0.025879467, -0.0050916704, -0.008812105, -0.02696532, 0.013433936, 0.0051647564, -0.016078958, 0.00029538997, 0.007419987, -0.004367769, -0.027410798, -0.0077471347, 0.0017218755, 0.0272159, 0.04532735, -0.03207439, 0.004332966, 0.013176394, -0.009494242, 0.019921204, -0.0134478565, -0.00366823, -0.010587055, -0.02200938, 0.01950357, 0.016357383, -0.010865479, 0.008665932, -0.016120723, 0.003918811, 0.012710035, 0.008150849, 0.0034681133, 0.02153606, -6.063325e-05, 0.038728714, 0.010795873, -0.024849301, -0.019197302, -0.04103963, -0.03062659, 0.050004866, 0.014269206, -0.0001604198, 0.005596313, 0.0022308687, -0.010245986, -0.019851599, 0.01100469, 0.0044652172, -0.014867817, -0.016635805, -0.0046670744, 0.016830703, 0.0133016845, -0.037726387, 0.009153173, 0.007844583, 0.020394525, -0.0050673084, 0.027939802, -0.012243675, 0.013893334, -0.025768097, 0.07344813, -0.002023791, 0.0034594124, -0.0070998003, -0.0030661393, 0.0098770745, 0.012508177, -0.026672974, -0.019781992, -0.016607964, -0.07578689, -0.013907256, -0.008192613, -0.011895645, 0.01135968, -0.058970105, -0.0260326, 0.003967535, 0.010684503, 0.008665932, 0.046385363, -0.013976862, -0.04541088, -0.032714766, 0.024348138, -0.01680286, 0.035081364, -0.025030276, -0.040176515, -0.006838778, 0.026116127, 0.0010806314, 0.012744837, -0.020157864, -0.0041485108, 0.008672893, -0.017401472, -0.034413148, 0.012960616, -0.024556955, 0.034023356, -0.013468739, 0.016761096, 0.001253776, -0.037225228, 0.01798616, 0.022287805, 0.0058120913, 0.017902633, -0.008728578, 0.0009692619, -0.00028059873, -0.023192681, -0.02792588, 0.00065125007, -0.032686923, -0.005213481, -0.0025875987, -0.022844652, 0.008150849, 0.013002379, -0.0153829, 0.043489758, -0.0036369073, -0.011046453, -0.0057076826, -0.0023822614, 0.0016366084, -0.0019054611, 0.0048445696, 0.05420906, -0.014206561, 0.009737863, -0.04031573, 0.035164893, -0.013371291, 0.021744879, 0.004319045, 0.027285507, -0.015508191, 0.003703033, 0.014491945, 0.021981537, -0.011540656, -0.009403755, 0.015355058, 0.02887252, 0.037642863, -0.0051403944, 0.007872425, -0.019252988, 0.0061810026, -0.01407431, 0.026338866, -0.014631157, -0.009487282, -0.02317876, -0.008136928, -0.00075261365, 0.020171786, 0.04543872, -0.0058642956, -0.019684544, -0.021619588, 0.0045696264, 0.00013105483, -0.0063097733, -0.001005805, 0.00014203951, -0.006657803, 0.018974563, 0.028844679, -0.01266131, 0.004165912, -0.028482728, -0.02354071, -0.03182381, -0.027048847, 0.025099881, 0.004305124, 0.011554576, -0.009800509, -0.053568687, -0.0018323748, -0.009201898, -0.017819107, 0.023164839, 0.04165216, -0.012821404, -0.019726308, -3.504765e-05, -0.011540656, -0.021577824, 0.02638063, -0.05039466, 0.020185707, -0.007204209, 0.009793548, -0.007134603, 0.016886387, -0.00709632, -0.016788939, 0.007607923, -0.018584771], "content_hash": "a5efe25b4805f10d2c5aa1847ae8ba66"}, {"content": "# Experiment: Exploring Attention Patterns\n\n## Goal\n\nLearn to read attention visualizations and understand what different attention head categories reveal about how the model processes text.\n\n## Prerequisites\n\nComplete \"Your First Analysis\" first so you're familiar with the basic workflow.\n\n## Steps\n\n### Step 1: Run an Analysis\n\n1. Select **GPT-2 (124M)** and enter the prompt: `The cat sat on the mat because it was`\n2. Click **Analyze**.\n3. This prompt is ideal because it contains a pronoun (\"it\") that needs to be resolved -- the model must figure out that \"it\" refers to \"the cat.\"\n\n### Step 2: Open the Attention Stage\n\n1. Expand **Stage 3 (Attention)** in the pipeline.\n2. Look at the **head categories** section at the top.\n\n### Step 3: Explore Previous-Token Heads\n\n1. Click on **\"Previous-Token\"** to expand the category.\n2. Note which heads are listed (e.g., L0-H3, L1-H7).\n3. In the **BertViz visualization** below, double-click on one of those head squares. This will show only that head's attention pattern.\n4. **What to look for**: You should see a strong diagonal pattern -- each token attends heavily to the token directly before it. Lines should mostly connect each word to the previous word.\n\n### Step 4: Explore First/Positional Heads\n\n1. Click on **\"First/Positional\"** to see which heads focus on the first token.\n2. Double-click one of those heads in BertViz.\n3. **What to look for**: You should see many tokens sending attention lines to \"The\" (the first token). This is a common pattern -- the first token acts as a \"sink\" for attention when there's no better target.\n\n### Step 5: Explore Bag-of-Words Heads\n\n1. Find a **\"Bag-of-Words\"** head in the categories.\n2. View it in BertViz.\n3. **What to look for**: Attention should be spread broadly and evenly across many tokens. Lines will be thin and numerous rather than thick and focused. This head is gathering a general summary of the whole input.\n\n### Step 6: Look for Interesting Patterns", "source_file": "experiment_exploring_attention.md", "chunk_index": 0, "embedding": [-0.002631302, 0.050220467, 0.008488709, -0.018690977, -0.037197415, -0.03429755, -0.0045903614, 0.03498297, 0.018730521, 0.051802214, 0.030554079, -0.0130889565, 0.0045046834, -9.92711e-05, 0.0035984744, 0.02215764, 0.0069465055, 0.022302633, 0.012508982, 0.06316443, 0.012186042, -0.007111271, 0.014617979, 0.051459502, 0.02065498, -0.02559794, -0.008330534, 0.049956843, 0.013721655, 0.002954242, 0.018532801, -0.007579204, -0.053515773, -0.015382489, -0.018730521, 0.030606804, -0.052461274, 0.0320831, -0.030422267, -0.00024673605, 0.006834465, -0.00032664722, -0.025031146, 0.029710481, -0.0130362315, 0.017386036, 0.03885825, 0.013438258, -0.030290455, 0.049587768, -0.027838746, 0.034719344, -0.0043662805, 0.009997959, 0.009997959, -0.0032277524, -0.05362122, 0.021867651, 0.010479074, 0.008205312, 0.03239945, 0.018967783, 0.04318169, 0.01796601, -0.0025571575, -0.02831327, -0.058946438, -0.007935097, 0.01722786, 0.0009869442, 0.065695226, 0.023264863, -0.020364992, 0.0069069616, 0.021920377, -0.02337031, -0.0055756583, 0.003562226, -0.005114315, -0.026428355, 0.023198955, 0.0008905565, -0.0017250928, -0.013537118, -0.013490983, -0.032214914, -0.047531497, -0.04948232, -0.04866508, -0.05467572, -0.00042385876, 0.028893245, -0.03777739, 0.022684887, 0.024780702, -0.010577933, -0.013484393, -0.01366893, -0.012021277, 0.022539895, 0.048585996, 0.009101636, -0.016305175, 0.006060068, 0.04724151, -0.01775511, 0.034508448, 0.00078675436, -0.014407079, -0.046345185, -0.07476391, 0.008317353, 0.016173363, 0.023409855, 0.051274963, 0.022078551, -0.02549249, 0.01797919, 0.03432391, -0.034060284, 0.0003320021, 0.020378174, 0.019007325, 0.016015189, -0.014103911, -0.027469672, 0.020589072, -0.02110314, 0.0019294018, -0.007935097, 0.005552591, 0.028286908, 0.030738616, -0.03316396, 0.0025011373, -0.041178145, -0.0029048123, 0.025307951, -0.066538826, 0.010057274, 0.008330534, 0.0063731223, 0.0019672979, 0.046028838, -0.035984743, -0.035035696, 0.0011871341, 0.07196949, -0.03487752, 0.015619751, 0.0032722391, 0.023554849, -0.028497808, -0.053990297, 0.0029509468, -0.03337486, 0.004165267, -0.0022506942, 0.00046587392, -0.044499815, 0.061688133, -0.015962463, -0.012027867, 0.003605065, -0.042364456, 0.046266098, -0.0391746, 0.010235221, -0.0034106418, -0.04529069, -0.007190358, -0.018374627, 0.04175812, -0.035773844, 0.050009567, 0.0073221703, -0.08267264, 0.008679836, -0.030554079, -0.051169515, -0.0016369433, -0.024582984, -0.017689204, -0.008653474, -0.038515538, -0.045686126, 0.01560657, 0.012528754, -0.026850155, 0.0039214143, -0.008356896, 0.010386805, 0.06242628, 0.0070980894, -0.0025752818, -0.029710481, 0.06253173, 0.019745475, 0.019653207, 0.02056271, 0.023633936, 0.020536348, 0.03656472, 0.0033282593, 0.028181458, 0.030369543, -0.011981733, -0.005582249, -0.062900804, -0.05810284, 0.01616018, -0.059210062, 0.03358576, 0.016002007, -0.06400803, -0.019837743, -0.04692516, 0.012456258, 0.031476766, 0.0685951, 0.051828574, 0.007935097, 0.044209827, 0.04671426, -0.012759426, 0.043998927, 0.077400155, -0.029868655, 0.020299086, -0.034772072, 0.02206537, 0.008297581, 0.011830149, -0.0046002474, -0.0074276202, -0.072444014, 0.055993844, 0.0052889665, 0.01205423, -0.00024405861, 0.01387983, 0.01701696, 0.009048911, 0.018216453, 0.037091967, -0.0026675505, -0.020839516, -0.065695226, 0.03595838, 0.054833896, 0.038726438, 0.001738274, -0.072074935, 0.017741928, -0.024767522, 0.002132063, -0.023633936, 0.019547757, -0.008897327, -0.008528252, -0.013998461, 0.010024321, -0.0015281982, -0.018730521, -0.0016221144, -0.048902344, -0.023541667, -0.023950286, -0.026349269, -0.04028182, -0.041810844, 0.012574889, 0.037276503, 0.020114549, -0.06421892, -0.009299354, -0.018414171, -0.014578435, 0.0027351042, -0.0057667857, -0.03777739, -0.008851192, 0.05541387, 0.041942656, -0.026415175, 0.014117092, -0.0015133694, -0.029183231, -0.000318203, -0.00040099758, 0.044157103, -0.042232644, 0.057206515, -0.012792379, -0.019798199, -0.00775056, 0.0130823655, 0.019218225, -0.010584524, 0.017201498, 0.029578669, -0.03904279, 0.038357362, 0.0043432135, 0.02431936, 0.024662072, 0.021709478, -0.023225319, 0.026573349, 0.013471211, -0.015487939, -0.042891704, 0.009655247, -0.028972331, -0.026863337, -0.006953096, 0.007730788, 0.048717808, -0.0048836437, 0.004297079, -0.03079134, -0.034692984, 0.053542133, 0.03195129, 0.023172593, 0.0035589307, -0.013761199, 0.0148025155, 0.026177913, 0.0036215414, 0.036643803, -0.02077361, -0.009246629, 0.013359171, -0.02775966, -0.021643572, -0.026665617, 0.043735303, 0.011718109, 0.056152016, -0.0806691, -0.009292764, -0.0018008848, 0.013194406, -0.05348941, 0.06906962, -0.0014903023, -0.012845104, -0.005908484, -0.030949516, -0.036960155, 0.0038357363, 0.016529256, -0.03672289, -0.0056217923, 0.057206515, -0.031819478, -0.025123414, 0.011961961, -0.033954836, -0.0043728715, -0.0012546878, 0.04971958, 0.005424074, 0.030448629, 0.0071178614, -0.030923152, 0.0064917533, 0.0066367467, 0.0022688182, 0.018466895, 0.02733786, -0.007697835, -0.03261035, -0.043418955, 0.001234916, 0.005242832, -0.012904419, -0.035009332, 0.07175859, -0.010037502, 0.034139372, -0.020272724, -0.044288915, 0.029394131, 0.025307951, 0.0031223027, 0.0075528417, -0.060844533, -0.008422802, -0.019363219, -0.021722658, -0.044710714, 0.038805526, -0.015369308, 0.031450402, -0.012008096, 0.030633166, 0.052935798, 0.05799739, 0.018730521, -0.019534575, -0.015145227, -0.0037533538, 0.0052131745, 0.0053977114, -0.0037138101, 0.02786511, 0.012614432, 0.0059249606, 0.05457027, -0.010881101, 0.018770063, -0.020707704, -0.023726204, 0.027522398, 0.019244589, 0.022012645, 0.012311264, 0.013510755, -0.016911512, -0.05035228, -0.017386036, 0.0038686895, -0.07144224, 0.010874511, 0.038594626, -0.030527717, 0.014341173, -0.038568262, -0.004488207, 0.044631626, 0.04120451, -0.0022506942, -0.022328995, -0.021801746, 0.033743937, 0.073287606, 0.007783513, -0.054622997, 0.00048811722, -0.014907965, -0.044183467, 0.00017918227, -0.02131404, 0.032636713, -0.035483856, -0.05889371, 0.030527717, -0.03798829, -0.005760195, 0.0045145694, 0.0068805995, 0.02464889, -0.008963233, 0.0027581712, 0.039095514, -0.021221772, 0.076240204, -0.025742931, 0.01668743, 0.011922418, -0.013893011, 0.027944196, -0.0028389064, 0.017346492, -0.020219998, -0.014907965, 0.05072135, -0.026230637, 0.023515305, 0.006920143, 0.012541936, -0.046028838, -0.0075001167, -0.032056738, -0.000750506, 0.00135849, -0.021169048, -0.045607038, 0.016911512, 0.0031750274, 0.03442936, 0.047320597, 0.007730788, 0.03722378, -0.04283898, -0.005941437, 0.028814157, 0.003221162, -0.03445572, -0.012192633, 0.011480846, -0.014433441, 0.0048408047, -0.019060051, -0.0113292625, -0.030185005, 0.007823057, -0.0013980337, 0.01647653, -0.030211367, 0.008376668, 0.03366485, -0.005542705, 6.60091e-05, 0.030000467, 0.013840286, -0.019033689, 0.045396138, -0.035035696, 0.010182496, 0.0076385196, 0.034192096, -0.014433441, 0.020826336, 0.02723241, 0.025057508, 0.0032887156, 0.0027334564, -0.0017547505, 0.015303402, -0.0056876983, -0.008172359, 0.00025209092, -0.038146466, -0.03195129, -0.06300625, -0.04679335, -0.008066909, 0.020615436, -0.016661068, -0.0045046834, -0.007243083, -0.032214914, -0.007783513, -0.0036643804, -0.038515538, -0.06158268, 0.010485664, -0.0051176106, -0.0034172325, 0.02184129, 0.0030613395, 0.010933826, 0.0028389064, 0.023844836, -0.002141949, 0.020312266, 0.039780937, 0.006587317, -0.04283898, 0.02400301, -0.0062248334, -0.020786792, -0.0064686863, -0.028207822, -0.04057181, -0.0046958113, -0.041995384, 0.013378943, -0.028049646, 0.0043761665, -0.03756649, -0.013721655, 0.04845418, 0.01819009, -0.043128967, 0.0026873222, -0.009127998, 0.018203272, -0.012957144, 0.014038004, -0.035668395, 0.028708708, -0.0043333275, -0.05499207, 0.013761199, -0.01237717, 0.020272724, -0.038331002, -0.0017679317, -0.022091733, 0.018203272, 0.00071260997, -0.02087906, 0.04262808, -0.005015456, -0.02227627, -0.027627846, 0.051169515, -0.019218225, -0.006274263, 0.036248367, -0.046450637, -0.022513531, 0.0113029005, -0.014552072, -0.029974105, 0.028866882, 0.02669198, 0.01259466, -0.009411395, 0.01366893, -0.026191093, 0.018638251, -0.041705396, -0.006953096, 0.015817469, 0.023713022, -0.035114784, -0.017781472, -0.027706934, -0.018862333, -0.028603258, 0.03466662, -0.030554079, 0.0035062057, -0.019468669, 0.0024204024, 0.01539567, -0.017465122, 0.01086792, -0.016331537, 0.023185775, 0.020852698, 0.011849921, -0.0049462547, 0.016595162, -0.0010561456, -0.029473219, 0.015791107, 0.03593202, -0.018427351, -0.02194674, -0.029104143, 0.010900873, -0.005252718, 0.011065639, 0.0078428285, 0.0007167291, -0.010960189, -0.025637481, -0.006719129, 0.025479307, 0.0069596865, 0.0130889565, 0.00095151964, -0.029578669, -0.02733786, -0.043577127, -0.023765748, 0.08145997, 0.005443846, 0.029104143, -0.022513531, -0.025044328, -0.035773844, -0.007473754, 0.018651433, -0.032768525, 0.027917834, -0.0060864305, -0.025057508, -0.032873973, -0.011856511, 0.014288448, 0.015896557, 0.01356348, 0.005951323, -0.09015958, 0.0064192563, 0.032847613, 0.004689221, 0.020075005, 0.012027867, 0.009312536, -0.038884614, 0.0021452443, 0.033401225, 1.8034078e-05, 0.020496804, -0.02926232, 0.0010890987, 0.0038126693, 0.0022671707, 0.011513799, -0.02377893, 0.0130955465, -0.013181224, 0.018901877, -0.04455254, 0.014710247, -0.0011442951, -0.0012077297, 0.0006487634, 0.0004992389, 0.038673714, -0.024042554, -0.10903509, -0.015474758, -0.011401759, -0.01065702, 0.012621023, 0.001647653, -0.011856511, 0.011685155, 0.056573816, 0.0130494125, -0.02582202, -0.049904115, 0.005717356, 0.0063335784, 0.036327455, -0.021959921, 0.016713792, -0.019112775, -0.006686176, 0.0058953026, -0.01969275, -0.033928473, 0.027416948, -0.0008477175, 0.007526479, 0.023963466, -0.015870195, -0.006422552, -0.023515305, 0.0056712218, 0.048480544, 0.028365996, 0.022645345, 0.0016970827, -0.07054591, 0.0048506907, 0.019073233, -0.033322137, 0.022025827, 0.00425424, -0.02066816, 0.020997692, -0.0018717339, 0.00980024, 0.007566023, 0.01743876, 0.010129771, -0.01270011, 0.037276503, -0.032346725, -0.017900104, -0.0074078483, 0.01152039, -0.015382489, -0.015461576, 0.020101368, -0.008343715, 0.002061214, -0.019086413, 0.0009844727, 0.032214914, 0.017267404, 0.00022778804, -0.017135592, 0.008917099, -0.030633166, -0.007559432, 0.0318722, 0.020167274, 0.022434445, 0.02248717, 0.022869425, 0.0072694453, 0.03764558, -0.010808604, 0.036459267, -0.015052958, 0.04993048, 0.011230404, 0.012251948, 0.006218243, 0.014156635, 0.0062446054, -0.031608578, -0.027680572, 0.036090195, -0.0032607054, -0.034060284, -0.009543207, -0.019679569, -0.010314308, -0.008014184, -0.036116555, 0.01452571, -0.015408852, 0.01366893, 0.0024319359, -0.022342175, -0.08884145, -0.004089475, 0.008706199, -0.02454344, -0.02873507, -0.018875513, 0.028234184, 0.0059710946, -0.018770063, -0.00527249, 0.014446623, -0.02831327, -0.0113226725, -0.019679569, 0.008838011, -0.035457496, -0.008910508, -0.004082884, -0.006705948, -0.0010396692, -0.031450402, 0.014907965, 0.0014276914, 0.0015430271, -0.035141144, -0.0028520876, -0.018111004, 0.002530795, 0.029763205, 0.03216219, -0.030343179, 0.035879295, -0.015316583, 0.02346258, 0.0020299086, 0.033401225, -0.021933557, 0.04908688, 0.040070925, -0.033216685, 0.0063335784, -0.010861329, 0.0028899836, -0.025110234, 0.04811147, 0.0050121606, -0.013471211, -0.014683885, -0.021010872, 0.032452174, -0.004649677, 0.019943193, 0.01764966, 0.023212137, 0.018796427, 0.025980193, -0.022210363, -0.018308721, 0.02723241, 0.03743468, -0.04552795, 0.0014079196, 0.05509752, 0.005931551, -0.0043761665, 0.030343179, -0.015158408, 0.022302633, -0.041072696, -0.026217457, -0.030158643, -0.009385032, 0.0040202737, -0.012337626, -0.017161954, -0.015659295, 0.0033546216, 0.028708708, -0.012113545, -0.0025291475, 0.043998927, 0.0012686929, -0.009002777, 0.017241042, 0.017992372, 0.020417716, 0.03002683, -0.016436987, -0.014683885, 0.053436685, 0.005552591, -0.0031799704, 0.021656752, 0.019890469, 0.0004839981, -0.021775384, 0.01452571, 0.033005785, 0.058155563, 0.009483892, 0.006854237, 0.034271184, -0.018097822, -0.013009869, -0.055993844, -0.00047287645, 0.020219998, -0.014907965, -0.0025653958, -0.045712486, 0.010070455, 0.022922149, 0.029209593, 0.008225084, 0.009193905, 0.020167274, 0.061529957, 0.00039461293, 0.0021287678, -0.020865878, -0.0015537369, 0.01539567, -0.012548526, -0.0011533572, -0.027496034, -0.012726473, -0.038515538, -0.00871279, 0.0035754072, -0.012363989, 0.020061824, 0.004339918, -0.001277755, -0.017834198, 0.007262855, -0.0039774347, -0.0024863086, 0.027838746, -0.01431481, 0.01616018, -0.006554364, 0.016094275, 0.031371314, 0.022856243, -0.004175153, 0.0046760393, -0.00775056, -0.028603258, 0.0077241976, 0.027074236, 0.019798199, 0.00074597495, 0.027917834, -0.007196949, -0.016661068, -0.009984777, 0.008225084, -0.010512027, -0.005770081, 0.023647116, -0.004613429, 0.018045098, 0.06037001, 0.003115712, 0.030949516, -0.017900104, -0.015712019, -0.013233949, -0.012416714, 0.019020507, -0.02559794, 0.04679335, 0.032346725, 0.032768525, 0.011480846, 0.03543113, 0.0069069616, -0.010314308, 0.02044408, 0.033849385, 0.001106399, 0.021024054, -0.026771067, 0.0055229333, 0.019824563, -0.010142952, -0.012983507, 0.013589843, -0.014683885, -0.0006289916, -0.010281355, 0.03904279, -0.03429755, 0.0016509484, -0.008600749, -0.022698069, 0.025360676, 0.010624067, -0.0030580442, -0.0013288322, -0.028998695, 0.028392358, 0.021182228, -0.035562944, 0.02248717, 0.021551302, 9.067241e-05, -0.0017036733, -0.010241811, -0.025466125, -0.004910006, 0.01797919, -0.042496268, -0.0015471462, -0.019139139, 0.0036413134, -0.005951323, 0.026006557, 0.013629386, -0.014354354, -0.06553705, -0.026942423, -0.031371314, 0.003832441, -0.011856511, 0.023185775, 0.035589308, -0.026256999, 0.017702384, -0.006307216, -0.017043324, -0.011869693, -0.011355625, 0.014723428, -0.017662842, 0.010624067, 0.011988324, 0.020088186, 0.029104143, 0.012647385, 0.037197415, 0.011698337, 0.008950052, -0.015250677, 0.012746245, -0.02496524, 0.033743937, -0.0069069616, 0.0067224246, 0.003993911, -0.009721153, 0.02775966, 0.024609346, -0.0020908718, 0.011790605, -0.008772105, 0.025887925, 0.024464354, 0.008969824, 0.012304673, 0.018572345, 0.036828343, -0.014749791, -0.048612356, -0.010643839, 0.0024335836, -0.02765421, -0.012548526, 0.009398214, 0.013497574, 0.0045475224, 0.022131277, -0.019903649, 0.016924692, 0.0039906157, -0.011210632, 0.013906192, -0.02066816, 0.015777927, -0.008244856, 0.0038719848, 0.012113545, 0.012021277, -0.013748017, 0.012581479, 9.314389e-05, -0.056362916, -0.032689437, 0.0031931517, -0.020325448, 0.0065411828, -0.02346258, 0.007019002, -0.01496069, -0.016489713, 0.0017399216, 0.0034567763, -0.0021386538, 0.008014184, -0.0053318054, 0.015540664, 0.005605316, 0.042285368, -0.039359137, -0.010749289, 0.03358576, 0.012449667, -0.06116088, -0.013286674, -0.002759819, 0.0130164595, 0.0012217348, -0.0073683048, 0.03195129, -0.011803787, -0.017306948, 0.016436987, -0.0078757815, 0.010769061, -0.03485116, -0.01797919, 0.03701288, -0.0159361, 0.008442574, 0.009068683, -0.0046002474, -0.0016567152, -0.0078494195, 0.011830149, -0.012528754, 0.033533037, 0.0021452443, 0.018111004, 0.01356348, -0.027786022, 0.0056745172, -0.025690207, -0.02077361, 0.029341407, 0.015672477, -0.0008374197, 0.007598976, -0.013682111, 0.010182496, 0.034903884, 0.0025242046, 0.011474256, -0.014987052, -0.021709478, 0.043972567, 0.06880599, -0.03013228, -0.038805526, 0.014143454, 0.01981138, 0.0131482715, -0.013919373, -0.007954869, -0.032030378, -0.006748787, -0.023001237, -0.0059875716, -0.02077361, 0.0014655874, -0.0060633635, -0.015751563, -0.019666387, 0.0095497975, 0.0639553, 0.014235723, 0.004132314, 0.05551932, -0.034350272, -0.015553845, 0.027996922, 0.035668395, 0.02443799, 0.010993142, 0.0096420655, 0.016555618, 0.023950286, -0.019956375, -0.027469672, -0.01258807, -0.021129504, 0.016081095, -0.006521411, 0.011006323, -0.021986283, 0.008304171, 0.004972617, 0.035246596, 0.018559165, -0.006188585, 0.019244589, 0.026072463, 0.004382757, 0.027996922, -0.017886922, -0.021933557, -0.027206048, 0.026467899, 0.0043003745, -0.03690743, 0.014591616, -0.029499581, -0.0065708407, 0.01796601, 0.018427351, 0.007664882, -0.010993142, 0.029868655, -0.032478537, 0.009945234, -0.0020579188, -0.04703061, -0.016964236, -0.0006759497, -0.013550299, 0.0130494125, -0.014578435, -0.021155866, -0.018361446, -0.0005614378, 0.032900337, 0.015013415, 0.008027365, -0.00096305326, 0.03788284, 0.0006261082, -0.010558161, -0.0025999965, -0.035009332, 0.0038818708, -0.0013939146, -0.004221287, 0.031503126, -0.028076008, -0.015145227, 0.028102372, -0.02831327, 0.031265866, 0.030264093, 0.050826803, 0.022500351, -0.0056844032, -0.0029838998, -0.027680572, 0.0054405504, -0.025901107, -0.0027136847, -0.01107882, -0.01227172, 0.0047584223, -0.008620521, -0.012851694, -0.03606383, 0.025861563, 0.0049495497, 0.0015240791, 0.006445619, -0.009213676, -0.00061333884, -0.026467899, -0.0041422, 0.01710923, -0.010050684, 0.042074468, -0.038805526, -0.035457496, 0.030422267, -0.012449667, 0.0070849084, -0.0019837744, -0.000104317034, -0.0022391605, 0.020734066, -0.006317102, -0.032768525, -0.006261082, -0.012785788, 0.014591616, 0.0043333275, -0.03295306, 0.00462661, 0.018532801, -0.0096025225, -0.02422709, -0.06348078, 0.009029139, 0.034719344, 0.0060996115, -0.00914777, -0.017741928, 0.014670704, -0.0055031613, 0.00808009, 0.0058096247, -0.0041158376, 0.028207822, -0.015817469, -0.023027599, 0.015303402, 0.018980963, -0.003756649, -0.033954836, -0.018335083, -0.0025999965, -0.024727978, -0.0050418186, 0.026296543, 0.0013189462, -0.010683383, 0.0096091125, 0.020285904, 0.03400756, -0.02905142, 0.016858786, 0.038700078, 0.03680198, 0.0042476496, -0.017359672, -0.006254491, 0.010518617, -0.004478321, 0.0038291458, 0.029631393, -0.002992138, 0.0061292695, 0.025242046, 0.0016155238, 2.0106516e-05, -0.03522023, -0.041151784, -0.0008946756, 0.027100598, -0.0048935297, -0.021775384, -0.014064367, -0.0033513263, 0.0041619716, -0.03522023, -0.003581998, -0.0113819875, 0.027390584, 0.056415644, 0.0072035394, -0.0013453087, 0.026072463, 0.011250176, -0.020681342, -0.02691606, -0.02658653, -0.01712241, -0.02066816, 0.031160416, -0.02108996, -0.002242456, 0.0017217975, 0.008594158, -0.020206818, -0.024569804, -0.037039243, -0.0038027833, 0.0066169747, 0.011671974, 0.009523435, 0.0148552405, 0.015211133, -0.018045098, 0.021485396, -0.01764966, 0.026665617, -0.0065741357, -0.032030378, 0.031424038, -0.022843063, -0.031898566, 0.006294035, -0.010722926, 0.0096157035, -0.037302867, 0.024477534, 0.023409855, 0.008152587, 0.010261583, -0.019547757, 0.017478304, 0.00043580425, -0.0010767413, -0.0048605767, 0.04423619, 0.023923922, 0.02033863, -0.008145997, 0.034271184, -0.008244856, -0.0074671637, -0.008126225, -0.036222007, 0.026731525, -0.002982252, -0.029578669, -0.037830114, -0.013721655, -0.03777739, -0.0113226725, 0.007249674, 0.0025110233, 0.007902144, -0.04658245, -0.005295557, -0.023343949, 0.0015619751, 0.025242046, 0.025017964, 0.008093272, -0.015646113, -0.045923386, 0.006379713, -0.02733786, -0.013695292, -0.027496034, 0.0040960656, 0.0022012645, 0.041942656, 0.01442026, 0.0077439696, -0.016713792, 0.0017975895, 0.016726974, -0.03595838, 0.005219765, -0.0113688065, 0.004007092, -0.0003268532, 0.008343715, -0.011289719, 0.023554849, -0.0032689436, 0.014341173, 0.037487403, -0.010512027, -0.01658198, 0.03216219, -0.005466913, -0.014235723, -0.0025588053, -0.043735303, 0.028998695, -0.026098825, -0.023660298, 0.019943193, -0.012126727, 0.01722786, 0.0049792076, 0.033823024, 0.04120451, -0.027074236, -0.0054306644, -0.0068212836, -0.011685155, -0.009780468, -0.009220267, 0.0076055666, -0.00425424, -0.022039007, -0.026401993, 0.008673246, 0.021973101, 0.0046529723, 0.026573349, 0.018440533, -0.0069135525, 0.032979425, -0.041494496, -0.0029806045, 0.0069267335, 0.037039243, -0.0031882087, 0.013603024, 0.031687666, 0.056099292, 0.023198955, 0.011955371, -0.004995684, -0.024148004, 0.0042904885, 0.014776153, -0.010584524, -0.013893011, -0.02130086, -0.01840099, -0.0049133017, 0.0012003153, 0.03962276, 0.002797715, -0.01637108, -0.016397443, 0.007823057, 0.015224314, -0.0012225586, -0.00446514, -0.034613896, -0.008963233, 0.019363219, 0.015659295, 0.0058063297, -0.011098592, -0.021485396, -0.05509752, 0.017201498, -0.019191863, -0.0060732495, -0.02227627, -0.0019162205, -0.023080325, -0.019218225, 0.009589341, 0.0033183733, 0.009312536, 0.0030695777, 0.027179686, 0.020114549, -0.03551022, -0.031634938, -0.036116555, -0.046635173, 0.033427585, -0.017741928, 0.013174634, -0.028076008, 0.018875513, -0.007071727, 0.009734334, -0.009154361, -0.02862962, 0.037487403, -0.0024896038], "content_hash": "e9bbc428355b61182f95ca4efa7383c8"}, {"content": "categories. 2. View it in BertViz. 3. **What to look for**: Attention should be spread broadly and evenly across many tokens. Lines will be thin and numerous rather than thick and focused. This head is gathering a general summary of the whole input. ### Step 6: Look for Interesting Patterns\n\n1. Now single-click to select multiple heads from different categories.\n2. Look for heads where the token \"it\" attends strongly to \"cat\" -- this would suggest the head is helping resolve the pronoun reference.\n3. Try hovering over the word \"it\" on the left side of BertViz to see which words it attends to most strongly.\n\n### Step 7: Try Different Prompts\n\nRun the analysis again with different prompts and compare:\n- `Alice gave the book to Bob because she` (pronoun resolution: does \"she\" attend to \"Alice\"?)\n- `The dogs in the park were` (subject-verb agreement: does the model connect \"dogs\" to \"were\"?)\n- `1 2 3 4 5` (number sequence: what patterns emerge with non-natural-language input?)\n\n## What You Should Learn\n\n- Different attention heads serve different purposes\n- Previous-Token heads are the easiest to identify visually (strong diagonal pattern)\n- The same prompt can reveal different patterns in different heads\n- BertViz is a powerful tool for understanding attention, but it takes practice to read fluently\n- Not all heads have obvious patterns -- and that's okay. The \"Other\" category captures complex, context-dependent behavior.", "source_file": "experiment_exploring_attention.md", "chunk_index": 1, "embedding": [0.004536667, 0.04139891, 0.027292939, -0.01295357, -0.02277086, -0.024433818, -0.013617295, 0.033725962, 0.008460665, 0.038656488, 0.04778817, 0.009569304, 0.024258768, 0.011254143, -0.030195821, 0.004303269, 0.0015763459, 0.010116329, 0.0031417513, 0.06850221, 0.02277086, 0.027730558, 0.022129016, 0.041077986, 0.025527868, 0.022785446, -0.017825745, 0.02866415, 0.0298749, 0.009605772, -0.0023759152, -0.016760869, -0.024667215, -0.03594324, -0.03025417, 0.026082188, -0.024404643, -0.01077276, 0.007421316, 0.014456067, 0.010298671, 0.014142439, -0.028066067, -0.011793875, -0.01747565, 0.025104836, 0.0349513, -0.016191963, -0.035213873, 0.052456122, -0.042011578, 0.039590076, 0.017606936, 0.02838699, -0.002727835, -0.019284481, -0.0524853, 0.005725536, 0.031975478, -0.0137923425, 0.018453002, -0.023004256, 0.022639573, 0.037664548, -0.0070894537, -0.033755135, -0.06867726, -0.020626517, 0.021210011, -0.031333636, 0.06377591, 0.05298127, 0.017067203, 0.020247247, 0.021705981, 0.031596206, 0.01281499, 0.015097911, 0.0135516515, -0.027905608, 0.038335565, -0.01961999, -0.0014414128, -0.03075014, -0.025163185, -0.04513327, -0.039706778, -0.030896014, -0.03836474, -0.083848104, -0.008176211, 0.026111363, -0.008416903, -0.0060245767, 0.015900215, -0.032617323, -0.002414207, -0.03235475, -0.00793552, 0.024287943, 0.04656283, -0.010415371, -0.027701383, 0.02138506, 0.07276172, -0.004201158, 0.023164717, 0.008978516, -0.002797125, -0.014251844, -0.08308957, 0.016629582, -0.014295606, 0.040757064, 0.03830639, 0.028707912, -0.025425756, -0.0014250021, 0.02894131, -0.039035756, 0.014828045, -0.0096349465, -0.0008825349, 0.024244182, -0.01697968, -0.018788511, 0.028430752, -0.02668027, -0.009532835, -0.020232659, 0.011465659, 0.014353956, 0.03731445, 0.0024087366, -0.02600925, -0.042040754, -0.008942048, 0.016775457, -0.04247837, 0.025994664, 0.00424492, -0.008343966, 0.0044819643, 0.02943728, -0.022143602, -0.0059078783, -0.01091134, 0.05403156, -0.009357788, 0.030166646, -0.0016173729, 0.0056051905, -0.046854578, -0.015812691, -0.03180043, -0.033259165, -0.0018453002, -0.026446871, -0.04247837, -0.039035756, 0.020874502, -0.031712905, -0.024069132, 0.006954521, -0.04711715, 0.021049552, -0.019153195, -0.00081233325, -0.017709047, -0.034484502, -0.028241117, -0.028532863, 0.015068736, -0.04344114, 0.008219973, -0.0035319629, -0.027074128, -0.0118595185, 0.0063090306, -0.026942842, -0.0092994375, 0.016862981, -0.0070165168, -0.0071879183, -0.0018124786, -0.0174173, -0.022858383, 0.015900215, -0.048809286, -0.0048357076, -0.030108297, 0.0065096063, 0.09884391, 0.032996595, 0.023018844, -0.045104098, 0.049363606, 0.011144738, 0.010947809, -0.009678709, 0.0026913667, 0.0070712194, 0.03605994, -0.019488703, -0.020466058, 0.013544357, -0.00396776, 0.020086786, -0.038773187, -0.04723385, 0.0051311017, -0.048488364, 0.030049948, 0.04393711, -0.079822, -0.016658757, -0.041165512, 0.008358553, -0.020291008, 0.059078783, 0.05020967, 0.008081394, 0.072936766, 0.026446871, -0.033142466, 0.056919854, 0.06902736, -0.03570984, 0.002392326, -0.047934044, 0.054381654, 0.041253034, 0.020466058, -0.0019802332, -0.039590076, -0.046942104, 0.027438812, -0.0044746706, 0.0016720754, -0.03862731, 0.00956201, -0.013267198, 0.03191713, 0.01069253, 0.008314791, 0.0034207343, -0.025775854, -0.062433872, 0.06290067, 0.038452264, 0.08425655, -0.029626915, -0.051114086, 0.010758173, -0.007045692, 0.0035501972, -0.039064933, 0.037226927, -0.017242251, 0.0037380094, -0.027220001, 0.03918163, -0.016454535, 0.02336894, -0.038014643, -0.06511795, -0.021866443, -6.854916e-05, -0.031041889, -0.025031898, -0.09079169, 0.0015097911, 0.046358608, 0.016862981, -0.05131831, 0.006217859, -0.012545124, -0.009292144, -0.0058604693, -0.019094845, -0.031712905, 0.0008811673, 0.043003518, 0.017957032, -0.024390055, 0.010284084, 0.0008784322, -0.0034097938, 0.0054483763, 0.015973153, 0.02259581, -0.040465318, 0.03810217, -0.018088318, -0.028182767, -0.0028317699, 0.051347483, 0.01957623, 0.028737087, 0.010860285, 0.028139004, -0.029189294, 0.04145726, 0.015564706, 0.02888296, 0.008752412, 0.023500226, -0.021210011, 0.01956164, 0.005251447, -0.018438416, -0.044258032, -0.022085253, -0.028459927, -0.033142466, -0.059574753, 0.023179304, 0.017315188, 0.015987739, 0.013675644, -0.03273402, 0.0052295662, 0.021064138, 0.05368146, -0.0075416616, -0.0052550943, -0.021355886, 0.024608865, 0.010181973, -0.006053752, 0.011122857, 0.0020677573, -0.006567956, 0.060683392, -0.036906004, -0.051347483, -0.012931689, 0.036526732, -0.0073410855, 0.048430014, -0.06050834, 0.0067502977, 0.008249149, 0.032383926, -0.05776592, 0.07031104, -0.015229197, 0.013909042, -0.0110863885, 0.02166222, -0.020568168, -0.004197511, 0.023427289, -0.028591212, -0.002853651, 0.08098899, -0.05875786, -0.0422158, -0.007490606, 0.0161482, 0.017038029, -0.007348379, 0.0545567, 0.04134056, 0.026490634, -0.0035264927, -0.020349357, -0.021837268, -0.021253774, -0.0065169, -0.016585821, 0.009992337, -0.0034863774, -0.006918052, -0.05881621, -0.004573135, -0.016294073, -0.0075927177, -0.016673345, 0.07071949, -0.008015751, -0.039706778, -0.027263764, -0.04828414, 0.019940913, 0.006768532, 0.013238023, -0.019765863, -0.081222385, -0.022187365, 0.0034954946, -0.012603473, -0.0011788405, 0.01912402, -0.015010387, 0.034076057, -0.039765127, 0.0422158, 0.017621523, 0.078830056, 0.036293335, 0.013194261, -0.024623452, 0.028139004, 0.020538993, -0.00878888, -0.032704845, 0.02805148, -0.00027898312, -0.0074760187, 0.047583945, -0.006403848, 0.02711789, 0.01714014, -0.009343199, 0.020261833, -0.0045549013, -0.00027989486, 0.026928253, 0.032500625, -0.015404245, -0.07381201, -0.022887558, -0.010233029, -0.048605062, -0.011742819, 0.057620045, -0.013303666, 0.026694857, -0.040202744, -0.0120053915, 0.04034862, 0.021705981, 0.0035939591, -0.013777755, -0.062025428, 0.057911795, 0.04772982, 0.03594324, -0.047525596, -0.0058385883, -0.036293335, 0.015404245, 0.0034954946, -0.024492167, 0.020626517, -0.012698291, -0.06710183, 0.038539786, -0.053593937, -0.032588147, 0.0013055681, -0.02562998, 0.047583945, 0.012559711, 0.011509422, 0.05341889, -0.005947993, 0.041982405, -0.024010783, -0.0039568194, 0.025017312, -0.0070274575, 0.012559711, 0.00084561063, 0.033025768, -0.027993131, -0.02331059, 0.04239085, -0.021034963, -0.016119026, 0.0033787957, 0.011093683, -0.061792027, -0.008190799, -0.013245317, 0.0013228906, -0.026446871, -0.0029357048, -0.05633636, 0.040086046, -0.009314025, 0.045862637, 0.027759733, -0.003032346, 0.036293335, 0.01648371, 0.020903677, 0.026257236, 0.006341852, -0.012931689, -0.024302531, 0.014514416, -0.017533999, 0.024039958, 0.0120053915, 0.0119105745, -0.043791234, 0.017665286, -0.009963162, 0.020232659, 0.002705954, -0.0041938643, 0.05292292, 0.008577364, 0.0053973207, 0.0545567, 0.0015170848, -0.020203484, 0.044958223, -0.045220796, 0.010203853, -0.0077896467, 0.04358701, -0.03092519, 0.018117493, -0.0011478424, 0.01725684, -0.011465659, 0.02281462, -0.00013949156, 0.017461061, 0.009226501, -0.009022278, -0.006137629, -0.024871437, -0.010663356, -0.008591951, -0.04414133, -0.015039561, 0.0035811954, -0.036672607, -0.0035337864, 0.0037124814, -0.028357815, 0.010670649, -0.0076364796, 0.0042303326, -0.034134407, 0.0048211203, -0.013595413, -0.007971989, 0.010517482, -0.0023248594, -0.0067502977, 0.012428425, 0.021968555, 0.007534368, 0.022741685, 0.039444204, -0.006881584, -0.019663753, 0.010532069, -0.02354399, -0.0038437676, -0.00701287, -0.021851856, -0.03180043, -0.0045184325, -0.040377796, 0.018642638, -0.040757064, 0.0034790838, -0.019386593, -0.006385614, 0.049101032, 0.025819616, 0.010561244, 0.0010028805, -0.012231496, -0.012844165, -0.023908673, 0.024754738, -0.025075661, 0.020918265, -0.020072198, -0.03428028, 0.0043215035, -0.017038029, 0.004398087, -0.029612327, -0.021297537, -0.02640311, 0.012705585, 0.022406176, -0.027205413, 0.05257282, 0.0047408896, -0.023193892, -0.018715575, 0.040115222, -0.025163185, -0.011874106, 0.030137472, -0.03273402, -0.009656828, -0.0026786027, 0.013427659, -0.03588489, 0.037781246, 0.011006158, 0.01110827, -0.020684866, 0.019780451, -0.039064933, 0.022581223, -0.012384663, -0.02287297, 0.011633415, 0.026855318, -0.034046885, -0.001685751, -0.028066067, -0.0026093128, -0.03708105, 0.005685421, -0.03025417, -0.0037380094, -0.021414235, -0.0062616216, 0.003506435, -0.016162788, 0.015827278, 0.011655295, 0.021589283, 0.025148597, 0.020363946, -0.030691791, 0.018686399, -0.029043421, -0.0349513, 0.009598479, 0.029058008, -0.01323073, -0.018555114, 0.0023649747, -0.022785446, -0.0012289846, 0.0068888776, 0.025250709, -0.013602707, 0.0198388, -0.01719849, -0.021676807, -0.002231865, -0.016717106, 0.011998098, 0.0099194, -0.03390101, -0.0075270743, -0.042565897, -0.051551707, 0.06762697, -0.0068888776, 0.021603871, -0.024842262, -0.031041889, 0.0028390638, -0.019984674, 0.033054944, -0.0057583577, 0.0054192017, 0.016556647, -0.04624191, -0.017213076, 0.0017395419, -0.00016194241, 0.03503882, -0.007556249, 0.003785418, -0.090033144, 0.014893688, 0.06196708, 0.008271029, 0.022902144, 0.017986206, 0.023514813, -0.008263736, -0.027380463, 0.025396582, -0.032996595, 0.014864513, -0.055869564, -0.030633442, 0.021268362, -0.00884723, 0.022843795, -0.022012316, -0.0008702268, 0.01648371, 0.012858752, -0.037110228, 0.034163583, -0.0015398775, -0.0087669995, -0.0018343597, 0.0035483737, 0.030137472, -0.028810022, -0.07241162, -0.022610398, -0.015535532, 0.01267641, -0.04309104, 0.020743217, -0.04145726, 0.0048247674, 0.04029027, -0.009809995, 0.002166222, -0.041690655, -0.0035903123, 0.0029594093, 0.014135146, -0.0149520375, 0.017767396, -0.009824582, -0.015345896, 0.034367803, -0.025819616, -0.02579044, 0.019372005, 0.00743955, 0.00097188243, 0.032646496, 0.0017112789, -0.003196454, -0.02965609, -0.007738591, 0.016687931, 0.023689862, 0.032879893, -0.0007161479, -0.043003518, -0.020582756, 0.015798103, -0.05557782, 0.034076057, -0.021020377, -0.032092176, 0.02832864, -0.031100238, -0.00084925746, 0.004376206, 0.02705954, -0.010984277, -0.023952434, 0.044345554, -0.038160518, -0.015593881, 0.0010821993, 0.015272959, -0.016746283, -0.0161482, 0.03180043, 0.0029120005, 0.0041136337, -0.0053973207, 0.014704052, 0.042740945, 0.017621523, -0.007468725, -0.006476785, 0.004806533, 0.0066919485, 0.0031435746, 0.003920351, -0.0007207064, 0.0029393516, 0.02138506, 0.018175842, -0.003909411, 0.03912328, -0.022479111, 0.050618116, -0.035914063, 0.025221534, 0.009489073, -0.008526308, 0.021705981, -0.005612484, -0.0099194, -0.027220001, -0.028299466, 0.01388716, 0.022289475, -0.030195821, -0.030108297, -0.0056781275, -0.014332075, -0.0070639257, -0.031712905, -0.029058008, -0.022756271, 0.0019364712, 0.009627653, -0.030137472, -0.10736292, -0.01281499, 0.017723635, -0.047000453, -0.034221932, -0.0038291803, 0.043966282, 0.017315188, -0.022129016, 0.021676807, -0.0014514417, 0.011312492, -0.010429958, -0.029568566, 0.016717106, -0.023675274, -0.021297537, -0.014171613, -0.007651067, -0.012647236, -0.009102508, -0.0012791286, 0.0023120956, 0.0035082584, -0.042186625, 0.006254328, -0.043849584, -0.0080449255, 0.01902191, 0.005878703, -0.052222725, 0.009766233, -0.01259618, 0.030108297, 0.016016914, 0.017242251, -0.011152032, 0.011429192, 0.023164717, -0.020101372, 0.023208478, -0.02148717, -0.00015168569, -0.031450335, 0.03719775, 0.007570836, -0.022172777, -0.019970087, 0.010641474, 0.023106368, -0.011567771, 0.01033514, -0.0064621978, 0.019780451, 0.01182305, 0.0053097964, -0.0024251475, -0.007979282, 0.0074359034, 0.029058008, -0.038277216, -0.02948104, 0.043295264, -0.019765863, -0.010721705, -0.009773526, -0.001685751, 0.034426153, -0.025396582, -0.014558178, -0.024127483, 0.016848393, -0.0006350057, -0.016527472, 0.010050686, -0.016629582, 0.00530615, 0.0024415583, -0.006345499, 0.0017568644, 0.048634235, -0.025294472, 0.003488201, 0.03515552, 0.014149733, 0.023514813, 0.011582359, -0.028766261, -0.0050654584, 0.0396776, -0.015127085, -0.027088715, 0.002496261, 0.027322114, -0.034630377, 0.007964695, -0.004595016, 0.03565149, 0.013332841, 0.0117136445, 0.028299466, 0.046329435, -0.016746283, -0.0448707, -0.05400238, 0.0015471712, 0.017315188, -0.0051019266, 0.0068779374, -0.0030305227, 0.017490236, -0.0014569119, 0.02088909, 0.010101742, -0.0019328244, 0.038656488, 0.063075714, -0.0057401233, -0.0023193893, -0.017957032, -0.0030706378, 0.024375467, -0.023223067, -0.018102905, -0.024827676, 0.021414235, -0.033696786, -0.0118595185, 0.0028919429, -0.008934754, 0.006312677, 0.02441923, -0.010189266, 0.0025436697, 0.015550119, -0.016862981, -0.021633046, 0.016804632, -0.01890521, 0.023325179, -0.0031526918, 0.008949341, 0.04711715, 0.019795038, -0.0027497162, -0.021516345, -0.02727835, -0.028270291, -0.0044272617, 0.0041902172, 0.020495232, 0.009226501, 0.038656488, -0.004350678, -0.00722074, 0.02805148, 0.034484502, -0.023471052, -0.005586956, 0.027365874, -0.015272959, 0.037577022, 0.06430105, -0.025936315, 0.014652996, -0.008416903, -0.02485685, -0.009926694, -0.009554717, 0.0397943, -0.05373981, -0.0015572, 0.03544727, 0.020422295, 0.0100725675, 0.026796969, -0.005484845, 0.0030396397, 0.020786978, 0.017183902, 0.012049153, 0.016687931, -0.042915992, 0.002106049, 0.022756271, 0.011181206, -0.011400017, 0.020218072, 0.010634181, -0.0072280336, 0.012027273, 0.03451368, -0.021647632, 0.02701578, -0.008934754, -0.00034371452, 0.01775281, 0.013150499, 0.003902117, -0.001729513, -0.060216594, 0.0072389743, 0.020509819, -0.019911738, -0.0050216964, 0.012231496, -0.011998098, -0.038014643, -0.014113264, -0.0028992365, -0.03285072, 0.020801567, -0.041019637, 0.002571021, -0.02761386, 0.0011214028, -0.0014496183, 0.012917101, 0.03075014, 0.0048101796, -0.028314052, -0.0029539391, -0.033200815, 0.03191713, 0.0048101796, 0.00821268, 0.04778817, -0.014470655, 0.014149733, -0.0022373353, 0.004288682, -0.017344363, -0.0051748636, -0.0023120956, -0.010028806, 0.002018525, 0.00079181977, 0.006341852, 0.010123624, -0.01719849, 0.023354353, 0.017446475, 0.014186202, -0.012494069, -0.0063965544, -0.03428028, 0.014390424, -0.010714411, -0.009642241, -0.005481198, 0.008730531, 0.0023357999, 0.0423325, -0.003923998, 0.009751646, -0.023558576, 0.008446078, 0.021005789, 0.01989715, 0.023354353, 0.009292144, 0.008723238, 0.0010667002, -0.03340504, -0.010284084, 0.020903677, -0.01874475, -0.015141672, 0.029364342, -0.03649756, -0.02755551, 0.01957623, -0.006998283, 0.014820751, 0.0068524093, 0.0110863885, -0.001020203, -0.026373936, 0.021005789, 0.0010712588, 0.0012490421, 0.011808462, 0.000775409, -0.0018580641, 0.0013949156, -0.001359359, -0.035972413, -0.018832274, -0.001117756, -0.03235475, 0.01339119, -0.031391986, 0.0025874318, -0.003732539, 0.02568833, -0.03185878, 0.009576597, -0.0025637273, 0.012858752, -0.027292939, 0.020013848, 0.029028833, 0.03693518, -0.037839595, 0.00010632813, 0.012362782, 0.015477182, -0.047758996, -0.0224937, -0.01918237, 0.018992733, 0.023514813, -0.008854523, 0.036176637, -0.009620359, -0.0058094133, -0.00050508714, -0.03915246, 0.01565223, -0.021968555, -0.0041683363, 0.028532863, -0.0129243955, 0.0029265878, 0.005280622, -0.025221534, -0.011538597, -0.008613832, -0.0032092177, -0.014171613, 0.02188103, 0.0036504853, 0.0075927177, 0.0058859973, -0.041077986, 0.0001701478, -0.040523667, 0.0010967866, 0.024069132, -0.0025673741, 0.008095982, 0.02326683, 0.023981608, -0.015841866, 0.04338279, 0.0024342646, 0.015404245, -0.0040953993, -0.025236122, 0.011575065, 0.072236575, -0.02535282, -0.030575093, 0.029962424, -0.0072936765, 0.019269893, 0.0062689153, 0.0012189557, -0.030458394, -0.00914627, -0.041428085, -0.004963347, -0.008205386, -0.014426893, 0.0039495258, -0.016308662, -0.0121512655, 0.0031435746, 0.05522772, 0.004605957, 0.018088318, 0.041165512, -0.030079123, -0.0008191711, 0.014893688, 0.031129412, 0.028284878, 0.015083323, 0.018861448, 0.021793505, -0.0009126213, -0.016833806, -0.02188103, -0.002065934, -0.03509717, 0.013340135, -0.008453371, 0.024054546, -0.012049153, -0.0035647845, -0.0039604665, 0.03130446, 0.020801567, -0.0005707302, 0.010028806, 0.024039958, 0.0036413681, 0.025381995, -0.005007109, -0.027263764, -0.026096776, 0.034688726, 0.0066700676, -0.019255307, 0.009620359, -0.040261097, -0.017840333, -0.020684866, 0.006272562, 0.0025418464, -0.009642241, 0.04303269, -0.024754738, -0.0063637327, 0.019809626, -0.036234986, -0.013960097, -0.011582359, -0.015695993, 0.011064508, -0.031537857, -0.039473377, 0.020086786, -0.009328612, 0.023660688, 0.0014888218, 0.011881399, -0.0010302318, 0.023952434, -0.022639573, -0.0070967474, 0.0040990463, -0.050501417, 0.0010037923, -0.00077176216, -0.007665654, 0.021341298, -0.023704449, -0.019196957, 0.017738221, -0.029218469, 0.019138608, 0.029991599, 0.051084913, 0.005411908, -0.010546656, 0.00015556045, -0.013135912, -0.009438017, -0.028795436, -0.011691764, -0.016731694, -0.013347428, -0.0009012249, -0.0077240034, -0.041573957, -0.046154387, 0.002118813, 0.021064138, 0.0067575914, 0.024667215, -0.028197354, -0.0037070112, -0.004981581, -0.020407706, -0.0060829264, -0.013084856, 0.027322114, -0.03247145, -0.029904075, 0.0074978997, -0.02188103, -0.012275258, 0.0007403082, -0.0026239003, 0.008898285, 0.010976983, 0.0161482, -0.013733993, 0.003767184, -0.017461061, 0.013821517, 0.012749347, -0.005597897, 0.01077276, 0.0063491454, 0.009168152, -0.029145531, -0.044287205, -0.00083011156, 0.024535928, 0.011348961, -0.0050836927, -0.013923629, 0.022960493, -0.02700119, 0.0026950135, 0.024375467, -0.0074760187, 0.024025371, -0.020772392, -0.024477579, 0.014689465, 0.0001236506, 0.0022956848, -0.039006583, -0.005893291, 0.0065497216, -0.024725564, -0.013048388, 0.018496765, -0.008584658, 0.005565075, 0.012523243, 0.031654555, 0.015973153, -0.012384663, 0.021034963, 0.025338233, 0.011553184, 0.019926324, -0.007454138, -0.02810983, 0.022566635, -0.014186202, 0.0026129596, -0.017461061, 0.030691791, 0.0029211175, 0.018978147, -0.005437436, 0.0032383925, -0.018876035, -0.033084117, -0.004536667, 0.02650522, -0.01083111, -0.024681803, -0.014536298, 0.015725167, 0.0028044188, -0.03915246, -0.006972755, -0.0003494127, 0.029904075, 0.056044612, -0.021370472, -0.003883883, -0.007694829, -0.0077750594, -0.017533999, -0.021560108, -0.0050836927, -0.00658619, -0.035855714, 0.017052617, -0.016002327, 0.013055681, 0.0041610426, 0.008147037, -0.017183902, -0.01339119, -0.016002327, 0.02805148, 0.004303269, 0.0061485693, 0.008504427, -0.0013520654, -0.007337439, -0.005346265, 0.009496367, -0.011093683, 0.018861448, -0.004040697, -0.009168152, 0.02656357, -0.007760472, -0.01515626, -0.0037635372, -0.007264502, 0.018978147, -0.03594324, -0.016191963, 0.03168373, 0.023996197, 0.00701287, -0.024929786, 0.019153195, 0.0037817713, 0.012289845, 0.0019565288, 0.037460323, -0.015054149, 0.019765863, -0.028197354, 0.025454931, -0.0020075846, -0.012690998, 0.0054447297, -0.013639175, 0.029831138, 0.0051858043, -0.043849584, -0.03781042, -0.013135912, -0.033113293, -0.0017668932, 3.333438e-05, -0.008358553, 0.021180836, -0.060274944, -0.0054994323, -0.00862842, 0.017738221, 0.03509717, 0.01884686, -0.014981212, -0.003975054, -0.028372403, 0.02166222, -0.022172777, 0.010225735, -0.027526336, 0.008059513, -0.0114729535, 0.03620581, 0.006637246, 0.001896356, -0.010801936, 0.0106998235, 0.01126873, -0.024652626, -0.016994266, 0.0045038452, 0.001940118, 0.01339119, -0.026432285, -0.0073921415, 0.036234986, -0.014594647, 0.02838699, 0.025381995, -0.019765863, -0.01565223, -0.005262388, 0.017986206, -0.0051821573, -0.021983141, -0.03515552, 0.015302134, -0.027978543, -0.02215819, -0.0022610398, -0.033871833, 0.014317487, 0.02705954, 0.02277086, 0.048400838, -0.035739016, -0.009379668, 0.017183902, -0.0080449255, -0.015535532, -0.009022278, -0.022741685, -0.015185435, -0.008234561, -0.007104041, -0.01408409, 0.010167385, -0.008460665, 0.03191713, 0.012880633, 0.005933406, -0.003953173, -0.005842235, -0.011210381, 0.0023248594, 0.0059005846, -0.0016046088, 0.012267964, 0.034426153, 0.038072992, -0.014448773, 0.021676807, -0.023485638, -0.019940913, 0.02331059, 0.010947809, -0.010138211, 0.004131868, -0.01940118, -0.03322999, -0.016717106, 0.0111374445, 0.052660346, -0.0005762005, -0.02094744, -0.024769327, 0.004015169, 0.0120053915, 0.004441849, -0.009087921, -0.037985466, 0.002640311, 0.0273367, 0.033434212, -0.0012298962, -0.009496367, -0.025498694, -0.043411963, -0.025717504, -0.030195821, -0.0052076853, -0.020407706, 0.0124867745, -0.042682596, -0.022522874, -0.0057474175, 0.01549177, -0.0055614286, 0.015287546, 0.033521738, -0.02751175, -0.025061073, -0.020101372, -0.02188103, -0.028562037, 0.034484502, -0.036234986, 0.024813088, -0.021020377, 0.013464127, -0.03235475, 0.022843795, -0.0012198674, -0.016381597, 0.01813208, -0.005459317], "content_hash": "1c3e110214a54a3579690f8364a21180"}, {"content": "# Experiment: Your First Ablation\n\n## Goal\n\nLearn how ablation works by removing an attention head and observing how it changes the model's prediction. Discover which heads matter and which are redundant.\n\n## Prerequisites\n\n- Complete \"Your First Analysis\"\n- Complete \"Exploring Attention Patterns\" (so you know about head categories)\n\n## Steps\n\n### Step 1: Set Up the Analysis\n\n1. Select **GPT-2 (124M)** and enter the prompt: `The cat sat on the`\n2. Set **Number of Generation Choices** to 1 and **Number of New Tokens** to 5.\n3. Click **Analyze**.\n4. Note the model's prediction in Stage 5 and the generated sequence.\n\n### Step 2: Select a Sequence for Comparison\n\n1. In the generated sequences section, click **\"Select for Comparison\"** on the generated sequence.\n2. This stores the original generation so the ablation experiment can compare against it.\n\n### Step 3: Find a Head to Ablate\n\n1. Expand **Stage 3 (Attention)** and look at the head categories.\n2. Find a head from the **Previous-Token** category. Let's say it's **L0-H3** (yours may differ).\n3. Note this head -- Previous-Token heads often have noticeable effects when removed.\n\n### Step 4: Set Up the Ablation\n\n1. Scroll down to the **Investigation Panel** and make sure the **\"Ablation\"** tab is selected.\n2. In the **Layer** dropdown, select the layer of your chosen head (e.g., 0).\n3. In the **Head** dropdown, select the head number (e.g., 3).\n4. Click the **+** button to add it. You should see a chip appear: \"L0-H3\".\n\n### Step 5: Run the Ablation\n\n1. Click **\"Run Ablation Experiment\"**.\n2. Wait for results to appear.\n\n### Step 6: Analyze the Results\n\nLook at the ablation results:\n\n- **Full Generation Comparison**: Compare the original text to the ablated text. Did the generated sequence change?\n- **Probability Change**: Look at the immediate next-token probability change. For example, \"72.3% \u2192 45.1% (-27.2%)\" would mean removing this head significantly reduced the model's confidence.", "source_file": "experiment_first_ablation.md", "chunk_index": 0, "embedding": [0.020743836, 0.030308522, 0.027893608, -0.0019095455, -0.030606994, -0.01440808, -0.016958663, 0.035436824, 0.04211175, 0.051120196, 0.057252448, -0.058229264, -0.04137914, -0.006352715, 0.025695765, 0.0066986717, 0.03372739, -0.011633643, 0.010717875, 0.07814552, -0.015384899, 0.007068371, 0.0044974373, 0.03513835, 0.020431798, -0.04273583, 0.049302224, 0.02302308, 0.030444192, -0.017040065, 0.0205539, -0.009055926, -0.053507973, -0.01569694, -0.028599089, -0.0044601285, -0.006657971, 0.013641548, -0.030172853, 0.007855252, 0.01043297, -0.05242262, -0.027038893, 0.039968178, -0.027568003, 0.051255863, 0.036793515, 0.0039920695, -0.022046262, 0.024013467, -0.0029101067, 0.06409018, -0.03372739, -0.0034086914, -0.01971275, 0.0012998815, -0.009571469, -0.0019705966, 0.014923624, -0.007543213, -0.010378702, 0.03351032, 0.049030885, 0.022588938, -0.014503049, -0.027269531, -0.039181296, 0.04208462, 0.01767771, 0.004083646, 0.021679955, -0.017040065, -0.0051757838, -0.0053114532, -0.02456971, -0.0428715, 0.002130008, -0.0069428766, -0.017596308, -0.016415985, 0.053860713, 0.027310232, 0.044173926, -0.011308037, 0.0013533013, -0.029982917, -0.057523783, -0.022697475, -0.0324521, -0.04813547, 0.002016385, 0.054213453, -0.032153625, 0.02143575, 0.007855252, 0.014245277, 0.005996583, -0.022995947, 0.019142939, 0.029982917, 0.027296664, 0.00548104, -0.021408616, -0.014964324, 0.022439703, -0.00565741, -0.029738711, 0.031176805, -0.0308512, -0.0125222765, -0.027459467, -0.018301789, 0.0052571855, 0.008628568, 0.012976769, 0.004565272, -0.019875553, 0.004551705, 0.019034404, -0.065881014, -0.036630712, -0.0023165534, 0.019251475, -0.009490068, -0.0024200012, -0.01657879, 0.0583378, -0.03470421, 0.02143575, -0.017433506, 0.04148767, 0.035626758, -0.018071152, -0.05888048, -0.020608168, 0.015846174, -0.017894782, 0.045720555, -0.069842555, 0.031936556, 0.010880678, -0.008838855, -0.023809964, 0.04349558, -0.07353276, -0.0043176757, 0.03535542, 0.068702936, -0.022249766, 0.033781655, 0.009910643, 0.030091451, 0.009429017, -0.020445364, -0.012054218, -0.049112286, -0.013126005, -0.011036698, 0.007997706, -0.07494372, 0.05768659, 0.011111316, -0.0038971007, -0.02006549, -0.033618852, 0.0583378, -0.040565122, 0.017637009, -0.019373577, -0.019739883, -0.028056413, -0.021978427, 0.0020587817, -0.025736468, -0.027866475, 0.022697475, -0.07803699, -0.015330631, -0.023633594, -0.051500067, -0.0005867697, -0.007373627, -0.0427901, 0.019346442, -0.03209936, -0.044228192, 0.014258844, -0.0056404513, -0.04281723, -0.037960272, 0.029223168, -0.002930457, 0.029331703, 0.0025488872, -0.038828555, -0.0028354884, 0.06560968, 0.028490555, 0.040266648, 0.00820121, 0.030389924, 0.020703135, 0.013187056, 0.027554436, 0.036440775, 0.018261088, -0.013722951, 0.019984089, -0.034405734, -0.027473034, 0.03470421, -0.07374983, 0.023063783, -0.0018484943, -0.047972664, -0.02153072, -0.040917862, 0.037010588, 0.047918398, 0.03907276, 0.031176805, -0.017542042, 0.032669168, 0.08085891, -0.0071633393, 0.009605386, 0.050930258, -0.024800349, 0.019821286, -0.03372739, 0.003995461, 0.021272946, -0.021666387, 0.011016347, 0.00069996883, -0.08606861, 0.032343563, 0.015819041, 0.015344198, -0.015724072, 0.008852422, 0.006746156, 0.03323898, -0.013743301, 0.03646791, 0.015968278, -0.02117798, -0.09209232, 0.014326679, 0.055678684, 0.02938597, 0.009951344, -0.04848821, 0.008479332, -0.017270703, -0.009585036, -0.008364012, -0.007760284, -0.017284269, -0.019617781, -0.019210773, -0.006837733, -0.025763601, 0.018953001, -0.021286514, -0.012386608, -0.008018056, 0.0022690692, -0.026808254, -0.02117798, -0.041704744, -0.0118574975, 0.019387143, 0.015452734, -0.04582909, 0.010439753, -0.0016890828, -0.01219667, -0.018505294, -0.0056608017, -0.011403005, -0.018953001, 0.031990822, -0.002599763, -0.031095404, 0.01604968, -0.017745545, -0.0043312428, -0.0037478646, -0.004901054, 0.030796932, -0.042274557, 0.07440104, -0.019102238, -0.019604215, 0.012481576, -0.0069360933, 0.018668097, -0.042274557, 0.035653893, 0.027947877, -0.008872773, 0.054376256, -0.017704844, 0.030661263, 0.029575909, 0.013594064, -0.00058253005, -0.004388902, 0.037770335, 0.012990336, -0.011403005, 0.0037648233, -0.022046262, -0.024651112, -0.0066986717, 0.010778926, -0.009978477, -0.030796932, -0.005046898, -0.028626224, -0.08009916, 0.026021373, 0.0016000499, 0.000405736, 0.018505294, 0.00042460253, 0.005694719, 0.021544285, -0.00097597105, 0.054891802, -0.014774388, -0.008642134, -0.010392269, -0.029196035, 0.0205132, -0.039588302, 0.06099692, 0.018491726, 0.048216872, -0.04618183, 0.01587331, -0.019522812, 0.007387194, -0.046697374, 0.056329895, 0.0066308375, -0.02797501, 0.040348053, -0.0064239413, -0.05025191, 0.013105655, -0.025695765, -0.02364716, 0.037146255, 0.053318035, -0.0025980673, -0.03028139, 0.032967642, -0.022222633, -0.008146942, 0.049003754, 0.017691277, 0.004019203, 0.019373577, 0.00925943, -0.018708797, 0.024094868, 0.008852422, -0.006172953, 0.0062848805, -0.03204509, 0.0018891951, -0.023402955, -0.06788892, -0.016470253, -0.009130544, -0.010168415, -0.015520568, 0.046507437, 0.0004464368, -0.007936655, -0.038692884, -0.0669121, 0.04230169, 0.021964861, -0.025275191, 0.012250938, -0.063221894, 0.024773216, 0.028843295, -0.017026497, -0.027744373, 0.009978477, -0.026102774, 0.06892001, -0.025627932, 0.0530467, 0.037906002, 0.035246886, -0.00017965584, -0.0015356069, -0.03239783, -0.042844366, 0.025695765, 0.0058066463, 0.0141774425, 0.0151135605, 0.023877798, 0.038123075, 0.059803028, 0.008458981, 0.037309058, 0.0011633643, 0.01569694, -0.0006575722, 0.03359172, 0.04184041, 0.007434678, -0.022127664, -0.0291689, -0.061756667, -0.009707139, -0.04265443, -0.035463955, -0.009917426, -0.0038021323, 0.00908306, 0.03478561, -0.06094265, -0.02135435, 0.03204509, 0.030579861, -0.029820113, -0.035898097, 0.009890292, -0.0026319844, 0.084006436, -0.03386306, -0.012590111, 0.023362255, 0.007054804, -0.0016178564, 0.0020706528, -0.051554337, 0.005111341, -0.03717339, -0.08004489, -0.0049383626, -0.03720052, -0.024895318, -0.021652821, 0.0019231124, 0.041976083, 0.027459467, -0.057578053, 0.04699585, -0.01489649, 0.10072089, -0.0055827918, 0.03147528, -0.041134935, -0.018301789, 0.041352004, -0.024908883, 0.037580397, 0.00978854, -0.027337365, 0.021028742, -0.024556143, 0.030824065, -0.009469718, -0.00068089034, -0.026862523, -0.0376618, -0.0145708835, 0.018925868, 0.022331167, 0.001865453, -0.036359373, 0.0101412805, -0.009836025, 0.051418666, 0.028409153, -0.012345906, 0.05812073, -0.01900727, -0.008472548, 0.015208529, 0.00051427144, -0.00017181245, -0.0023826922, 0.018939435, -0.015289931, 0.035653893, -0.032506365, -0.0126715135, 0.0043651597, 0.00822156, 0.0009166157, 0.046833042, -0.017596308, 0.020160459, 0.03416153, -0.005531916, -0.01900727, 0.03513835, 0.030335657, -0.020486064, 0.06501273, -0.035192616, -0.019902688, -0.005124908, -0.0011557329, -0.010059879, 0.020743836, 0.021313647, 0.006973402, 0.017528474, -0.00082037534, -0.016483821, -0.0010124322, -0.017637009, 0.017094333, 0.00978854, -0.0014762516, -0.034649942, -0.028870428, -0.03779747, -0.0011277511, 0.026401246, -0.020676002, -0.006980186, 0.030308522, -0.038611483, -0.012630812, 0.013553363, 0.0052537937, -0.060834117, 0.001902762, 0.003310331, 0.0046195397, 0.034025863, -0.019604215, 0.021327214, -0.0063696736, 0.055787217, -0.012278072, 0.022453269, 0.008872773, -0.003564711, -0.025736468, 0.01507286, -0.017243568, -0.021069443, 0.0036257622, -0.029331703, -0.04322424, 0.003476526, -0.037607532, 0.0072379573, -0.04281723, -0.012542627, -0.027920743, -0.01286145, 0.02161212, -0.006403591, -0.021815624, -0.0023877798, -0.011348737, 0.011409788, -0.014123174, 0.02638768, -0.03682065, 0.032289293, 0.016958663, -0.019332876, 0.010670391, 0.009462934, -0.007156556, -0.016999364, -0.0064578587, -0.04148767, 0.04173188, -0.011592942, -0.04905802, 0.013533013, 0.0020180808, -0.04070079, -0.005457298, 0.036250837, -0.03209936, 0.03247923, 0.04173188, -0.034921277, 0.0032391048, 0.027703673, 0.0070819375, 0.0062713134, 0.0032882849, 0.029575909, 0.012630812, -0.0053351955, 0.049302224, -0.012223804, 0.01337021, -0.036603577, -0.0029084107, 0.021096576, 0.008709969, -0.0006376457, -0.006895392, -0.042898633, -0.008146942, -0.016605923, 0.028002145, -0.00035825177, -0.0031068271, 0.01630745, -0.020486064, 0.021015175, -0.009157678, -0.0025726291, 0.001026847, 0.024610411, 0.018057585, -0.016822994, -0.0054335557, 0.022426136, 0.0063628904, -0.044987943, 0.039452635, 0.034731343, 0.014977891, -0.008194426, 0.015439167, 0.0034578715, -0.0004731467, 0.015629103, 0.002094395, -0.011715044, -0.017460639, 0.011036698, 0.001420288, -0.016117513, 0.015927576, 0.014014639, -0.0010310867, -0.029901514, -0.030824065, -0.016483821, -0.020472499, 0.07450958, 0.013227757, 0.017284269, 0.002801571, -0.0019485503, -0.039968178, 0.00720404, -0.0076788827, -0.037607532, 0.047945533, -0.013302376, -0.007943437, 0.009137328, 0.0030763014, -0.0010828107, 0.007041237, 0.004388902, -0.0039683273, -0.044662334, -0.0070955046, 0.010521155, -0.008526816, -0.0069089592, 0.03853008, -0.0038564, -0.02930457, -0.0070073195, 0.024244105, -0.006003367, 0.017650576, -0.033320382, 0.021055875, -0.014136742, -0.017976183, 0.00317127, -0.033347514, 0.0006397656, -0.00057786645, 0.005216485, -0.03747186, -0.009490068, -0.011626859, -0.007387194, 0.020825239, -0.029738711, 0.031448144, -0.010677175, -0.101155035, -0.02550583, -0.013987505, 0.036305107, 0.022602506, -0.017433506, 0.00624418, 0.014747254, 0.06143106, 0.016388852, -0.024854617, -0.03028139, -0.0029982915, 0.009876725, 0.034622807, -0.035952367, 0.038665753, -0.022711042, -0.0034086914, -0.013512663, -0.047267187, -0.049709234, 0.032723438, -0.018233955, 0.008669268, 0.026618317, -0.030905468, 0.004148089, -0.010066662, -0.008947391, 0.012067785, 0.029548775, 0.010867111, 0.018261088, -0.040727925, 0.014435214, 0.019794151, -0.013845053, -0.014109608, 0.0017789637, -0.025614364, 0.022561805, 0.017297836, 0.033971593, 0.031231074, 0.0032357129, 0.005986408, -0.0050028055, 0.01248836, -0.040293783, -0.015981844, -0.020676002, 0.012122053, -0.008791371, -0.003690205, 0.019861987, 0.031258207, -0.005796471, -0.019984089, -0.012081351, 0.019780584, 0.0037817818, -0.029277436, 0.016321016, 0.014692985, -0.034052994, -0.02002479, 0.03280484, 0.008323312, 0.020458931, 0.0360609, 0.0068614753, 0.012210238, 0.033917326, 0.008438631, 0.010290517, -0.023294419, 0.049085155, -0.011660777, -0.004572056, 0.01622605, 0.0060712015, 0.0044431696, -0.040592257, -0.0019163289, -0.0066037034, -0.017460639, -0.028002145, 0.004890878, -0.0378246, -0.029223168, 0.010507587, 0.010690741, 0.022860277, -0.013451612, 0.0103515675, 0.016063245, -0.01182358, -0.10142637, 0.0031848368, 0.0014601409, -0.0069225263, -0.030932602, -0.011470839, 0.0085743, 0.012983552, 0.008146942, -0.0094358, 0.027215263, -0.03888282, 0.003476526, -0.008872773, 0.030796932, -0.027147427, 0.025967104, -0.0205946, 0.027120294, -0.0205946, -0.018627396, 0.01630745, -0.015507001, -0.012474792, -0.044906538, 0.014191009, -0.023525057, 0.011911765, 0.018193254, 0.041569076, -0.01971275, 0.018640961, -0.035002682, 0.017162167, -0.0132752415, 0.032642033, -0.018301789, 0.026129909, 0.02617061, 0.006434117, 0.027947877, -0.007115855, 0.008438631, -0.005952491, 0.03288624, -0.011403005, -0.016239615, 0.0070887213, 0.0023945633, 0.00642055, 0.010331217, 0.004134522, 0.01957708, 0.01458445, 0.025478695, -0.027120294, -0.0009717314, -0.026930356, 0.018179687, 0.03394446, 0.017352104, 0.053290904, 0.03147528, -0.0102023315, -0.016497387, 0.017202867, -0.010066662, 0.040510856, -0.044445265, -0.05244975, -0.035545357, -0.009089843, -0.014801521, 0.00083394226, -0.0052673607, 0.00393441, 0.014665851, 0.049112286, -0.009110194, -0.012990336, 0.025424428, -0.014557316, -0.017514907, 0.0011107924, -0.0032815014, 0.03725479, 0.0026235052, 0.0035206184, 0.007054804, 0.056764036, 0.026998192, -0.02368786, 0.016334584, 0.021910593, -0.04645317, -0.022751743, 0.024081301, 0.0075906976, 0.032316428, 0.011884631, -0.006980186, 0.0019875553, -0.013112438, -0.013627982, -0.03646791, 0.0063493233, 0.016280316, -0.007984139, -0.016239615, -0.015805474, 0.04132487, 0.024189837, -0.017881215, 0.026713286, -0.016985796, 0.024556143, 0.054023515, 0.0075906976, -0.017406372, -0.017121466, -0.007244741, 0.03142101, -0.010344785, 0.00040001248, 0.010222682, 0.01630745, -0.03823161, -0.007136205, -0.0066172704, 0.017216435, 0.027785074, 0.009646088, -0.025261624, -0.014394513, 0.0012193279, -0.0064510754, -0.013302376, 0.0033205063, -0.01524923, 0.0017060415, -0.0360609, -0.009042359, 0.017772678, 0.0050163725, -0.051852807, -0.0222769, 0.0022724608, -0.017772678, -0.0079095205, 0.022711042, 0.0117082605, 0.029765846, 0.0010319346, 0.011477623, 0.0005973689, -0.008838855, 0.023090916, 0.0063662822, -0.025912838, 0.032235026, 0.015086426, 0.0066952803, 0.034107264, 0.00446352, 0.0020214727, -0.00090559263, -0.029982917, -0.0062204376, -0.01062969, -0.0119185485, -0.026496215, 0.04341418, 0.015018592, 0.041921817, 0.03112254, 0.019685617, -0.019115806, -0.017175734, -0.014923624, 0.020174026, 0.018573128, 0.010317651, -0.0052470104, -0.023674294, -0.018233955, -0.021001609, 0.009781756, 0.0074007604, 0.010995997, 0.001282075, -0.008458981, 0.02617061, -0.02452901, 0.007726367, -0.011904982, -0.009035576, -0.004500829, 0.017786246, -0.049736366, -0.039913908, -0.042573027, 0.054810397, 0.01609038, -0.017311404, 0.052232683, 0.039045624, -0.019346442, 0.023294419, -0.029250301, -0.006013542, -0.01595471, 0.009734273, -0.017569175, 0.0040768627, -0.029847248, 0.0073261424, -0.018545995, 0.00012591807, 0.015289931, -0.019129371, -0.0754864, -0.036874916, -0.030254254, 0.014665851, 0.0016704283, 0.018410325, 0.031041136, -0.009666438, 0.009252647, 0.005135083, -0.03416153, -0.02382353, -0.022792444, -0.0042566243, -0.03353745, 0.038421545, -0.0031560073, -0.0075160796, 0.0054064216, 0.009171245, 0.049736366, -0.00802484, -0.012875017, -0.018220387, 0.009673221, -0.014503049, 0.039913908, 0.018803766, 0.013105655, -0.022154797, -0.017365672, -0.0031984039, 0.0041446975, -0.0011277511, 0.018464593, 0.0011921941, 0.034649942, 0.008764237, 0.05101166, -0.00736006, -0.022928113, 0.032913372, 0.0016848432, -0.05139153, 0.0026726853, -0.017731978, -0.023036648, -0.014733686, 0.024081301, 0.002715082, 0.03801454, 0.015099994, -0.014326679, 0.019563513, 0.0075703473, -0.021829192, 0.013824702, -0.014001072, 0.021205112, -0.015167829, 0.017718412, 0.037363328, -0.015357765, 0.002247023, -0.0056608017, -0.019767018, -0.020608168, -0.005850739, -0.007041237, -0.012284855, -0.013906104, 0.013451612, 0.009062709, -0.017175734, -0.007387194, 0.007977355, 0.0050265477, -0.0029711577, 0.0036359373, -0.028409153, 0.0061966954, -0.012936068, 0.017501341, -0.034568537, -0.015099994, 0.019739883, 0.0151135605, -0.063981645, -0.02607564, 0.01573764, -0.015995411, 0.022304034, 0.008832072, 0.0077195833, -0.007672099, -0.025465129, 0.0042667994, -0.021367915, 0.0038903174, -0.033781655, -0.036630712, 0.023063783, -0.019780584, 0.033130445, -0.011986383, -0.02337582, -0.010439753, -0.01096208, -0.0068580834, -0.0119795995, 0.0033340731, 0.010819627, -0.010093796, 0.0102023315, 0.0013617807, 0.019943388, -0.019658482, 0.0012286551, 0.027690105, 0.0047484254, -0.027378066, -0.00069869694, -0.040429454, 0.011145233, -0.001826448, -0.004327851, -0.005633668, -0.02413557, 0.019644916, 0.03893709, 0.08232413, -0.05502747, -0.038123075, -0.025695765, 0.012264505, -0.00784847, -0.009557903, -0.01926504, -0.045720555, -0.008350446, -0.02811068, 0.015832609, 0.0012015213, 0.021367915, -0.0027846126, 0.012386608, -0.03470421, 0.008296178, 0.0393441, 0.017202867, 0.0077535007, 0.064741395, -0.039968178, 0.001686539, 0.031610947, 0.011179151, 0.023538625, 0.025315892, -0.005921965, 0.018858034, -0.0073329257, -0.041623343, -0.011172367, -0.013159922, -0.026645452, 0.019916253, 0.02881616, 0.0077399337, -0.02687609, 0.0018111853, -0.018939435, 0.034052994, 0.0072718747, -0.010880678, 0.007780635, 0.008689619, -0.008547166, 0.006434117, 0.0077060163, -0.0064646425, -0.012013517, -0.0074753785, -0.0027303447, -0.0132752415, 0.039669704, -0.024108436, -0.0024233928, 0.018790198, 0.017691277, -0.007000536, -0.015547702, 0.038313013, -0.017745545, 0.009754623, -0.015371332, -0.03155668, -0.03302191, 0.011592942, -0.0015627408, 0.016958663, -0.028164947, -0.012976769, 0.021720655, -0.011504757, 0.017257135, 0.008960958, -0.027568003, -0.023090916, 0.039723974, 0.0114369225, -0.0070887213, -0.033211846, -0.028409153, -0.021639254, -0.010012395, -0.0020655652, 0.028843295, -0.0046195397, -0.041948948, 0.008933824, -0.012983552, 0.025139522, 0.049709234, 0.029250301, 0.027079593, -0.010636473, -0.028626224, -0.034324333, 0.012101702, -0.016822994, 0.017949048, 0.004782343, -0.012576545, 0.00822156, -0.029982917, 0.0054844315, -0.031529546, 0.02174779, 0.014692985, 0.0050842073, 0.015086426, -0.00626453, -0.015859742, -0.0038530082, 0.0059830165, 0.021164412, -0.0009819065, 0.05011624, -0.033266112, -0.014204577, 0.01842389, -0.010907812, 0.0012583327, -0.0034019079, 8.081863e-05, -0.012440875, 0.03220789, 0.004578839, -0.003400212, -0.0005168152, 0.003169574, -0.007672099, 0.018396758, -0.03120394, 0.0070073195, 0.0010259991, -0.021028742, 0.0004744186, -0.05909755, 0.04181328, -0.011443706, 0.011009564, -0.01476082, -0.04175901, -0.0061085103, 0.02550583, 0.006027109, -0.0040531205, 0.011335171, -0.023701428, -0.024447609, -0.00064697297, 0.010704308, 0.00788917, 0.0016763639, -0.04004958, -0.010005611, 0.013261675, 0.010758576, 0.0222769, 0.020268993, 0.0009963214, -0.01988912, 0.0068512997, -0.0031424402, 0.015317065, -0.032343563, 0.010480454, 0.053887848, 0.016008979, 0.0018841075, 0.002411522, -0.00061220775, -0.0010370222, -0.00094035786, -0.00927978, 0.01427241, 0.006579961, 0.002715082, 0.019129371, -0.015588403, -0.0052436185, -0.017012931, -0.02153072, -0.012359474, 0.031149672, -0.021856325, -0.0031678784, -0.008581083, 0.0059796246, 0.008709969, -0.031041136, 0.0021401832, 0.0017077373, 0.030308522, 0.01047367, 0.00930013, 0.0073532765, 0.02010619, 0.0011481015, -0.00017287237, -0.014977891, -0.031231074, -0.026835388, -0.028300617, 0.023240153, 0.015615537, -0.02010619, -0.0067665065, 0.009761406, -0.019984089, -0.00401242, -0.026753986, -0.017894782, 0.0037207308, 0.008323312, 0.01595471, 0.002004514, -0.0023402956, -0.011504757, -0.018749498, 0.009347615, 0.053209502, 0.0021893634, -0.02895183, 0.007611048, -0.020703135, -0.034460004, 0.0013940021, 0.0066681462, 0.0010463495, -0.021584986, -0.011681127, 0.0066918884, -0.006966619, 0.017352104, -0.012617245, 8.823804e-05, 0.00017541616, 0.0053487625, 0.025288759, 0.030172853, 0.025166657, 0.007773851, 0.0049553216, 0.0427901, -0.0013516054, -0.034921277, -0.00892704, -0.033971593, 0.03424293, 0.016687324, -0.023972766, -0.004182006, -0.023497924, -0.004093821, 0.0035714945, 0.010405836, -0.012325556, 0.021584986, -0.047321454, -0.0093272645, -0.018844467, -0.0029338486, 0.010941729, 0.032642033, -0.0061831283, 0.0026964275, -0.03690205, 0.02938597, -0.023457224, -0.0222769, -0.039127026, -0.01613108, 0.019631349, 0.04243736, 0.013051387, 0.0072379573, -0.01060934, -0.01182358, 0.02435264, -0.03169235, -0.002891452, -0.011348737, -0.0017060415, 0.009646088, 0.019631349, 0.0057320283, 0.011660777, 0.0058778725, 0.010154847, 0.0308512, -0.0037987405, 0.011348737, 0.014882923, -0.00036948686, -0.009381533, -0.0077399337, -0.006820774, 0.0062747053, -0.036712114, -0.020011222, 0.017542042, -0.002816834, 0.011742178, 0.013044603, 0.006925918, 0.038991358, -0.01918364, -0.01235269, -0.028029278, -0.016334584, -0.017419938, -0.0055081737, 0.03345605, 0.025573663, 0.016497387, -0.027418766, -0.03451427, 0.0019349834, 0.022778876, 0.021951294, -0.0043787267, -0.0010700916, 0.013987505, -0.011579375, -0.003578278, 0.025044553, 0.024868183, -0.014435214, 0.0037783901, 0.029982917, 0.038421545, 0.029250301, -0.010975647, 0.0012473096, -0.008038406, -0.010548288, 0.031610947, -0.007611048, -0.013071737, -0.014123174, 0.0031526154, 0.02010619, 0.007509296, 0.0427087, 0.00052868633, 0.015493435, -0.01354658, 0.016809426, 0.008967741, 0.0069598355, 0.011016347, -0.015167829, -0.0012142403, -0.014855789, -0.012454442, 0.0014737078, 0.017135033, -0.040917862, -0.0427901, 0.014557316, -0.001104009, 0.0040531205, -0.027473034, 0.0024369599, -0.01816612, -0.029575909, -0.03247923, -0.009178028, -0.0060440674, -0.00022809402, 0.031068271, -0.0080316225, -0.030199988, -0.021693522, 0.0002206746, -0.03823161, 0.025071688, -0.016456686, 0.012936068, -0.026360545, 0.020743836, -0.013648332, 0.020716703, -0.012908935, -0.008730319, 0.02691679, 0.0008954174], "content_hash": "d22896e18178cfa41fb2ca3bb0a60bce"}, {"content": "the Results Look at the ablation results: - **Full Generation Comparison**: Compare the original text to the ablated text. Did the generated sequence change? - **Probability Change**: Look at the immediate next-token probability change. For example, \"72.3% \u2192 45.1% (-27.2%)\" would mean removing this head significantly reduced the model's confidence.\n\n### Step 7: Try Ablating a Different Head\n\n1. Click **\"Clear Selected Heads\"** to reset.\n2. Now pick a head from the **\"Other\"** category (these often have less obvious roles).\n3. Add it and run the ablation again.\n4. Compare: was the effect larger or smaller than the Previous-Token head?\n\n### Step 8: Compare Your Results\n\n| Head | Category | Probability Change | Generation Changed? |\n|------|----------|-------------------|-------------------|\n| L0-H3 | Previous-Token | (fill in) | (yes/no) |\n| L?-H? | Other | (fill in) | (yes/no) |\n\n**Typical findings**:\n- Previous-Token heads in early layers often cause noticeable probability drops when ablated\n- Many \"Other\" heads have minimal impact for simple prompts\n- The same head may matter more or less depending on the specific prompt\n\n## What You Should Learn\n\n- Ablation is a tool for measuring the importance of individual model components\n- Not all heads are equally important -- some are redundant\n- The effect of ablation depends on the specific input prompt\n- This technique is used by researchers to understand how models work internally\n\n## What's Next?\n\nMove on to **Experiment: Token Attribution** to learn a different approach -- instead of removing components, measure which input tokens drive the prediction.", "source_file": "experiment_first_ablation.md", "chunk_index": 1, "embedding": [0.022489404, 0.0122117605, 0.0347294, 0.00634588, -0.016376464, -0.0047858804, -0.007334115, 0.035407044, 0.019778816, 0.043115277, 0.04094116, -0.042155277, -0.061609387, -0.0028094107, 0.015529406, -0.006931762, 0.033261165, -0.015698818, 0.0020576464, 0.046672925, -0.024988227, -0.0027811755, 0.010962349, 0.048931748, 0.03921881, -0.031397637, 0.013665877, 0.01609411, 0.0390494, -0.012945877, -0.006730586, -0.018155288, -0.021162346, -0.0025852933, -0.03506822, 0.0030811753, -0.006017645, -0.0065717623, -0.015049406, -0.0168847, 0.03204705, -0.03741175, -0.041562337, 0.04447057, -0.03179293, 0.05615998, 0.03670587, -0.031143518, -0.030663518, 0.036847044, -0.0062894095, 0.07098351, -0.009352937, -0.010221173, -0.037242338, -0.009719997, -0.011971761, -0.018607052, 0.007814115, -0.02807999, -0.02567999, 0.0009494114, 0.042352926, 0.02741646, -0.002567646, -0.021656463, -0.036197633, 0.0087247025, 0.0042952923, 0.0056470567, 0.02855999, -0.016870582, -0.0018388228, 0.0044929394, -0.04077175, -0.048056453, 0.024409402, 0.004076469, -0.026414108, -0.01394823, 0.061439976, 0.013884701, 0.052856453, -0.02519999, -0.0040199985, -0.05375998, -0.036479987, -0.017745877, -0.03543528, -0.08244703, -0.036084693, 0.018762346, -0.018578816, 0.016941171, -0.024649402, 0.026682343, -0.0030264694, 7.63235e-05, 0.022489404, 0.005011763, 0.0176047, -0.002059411, -0.023421168, -0.019863522, 0.013687054, -0.017138816, -0.04867763, 0.0059294095, -0.047802337, 0.012303525, -0.034334105, -0.013319995, -0.042014103, 0.023251757, -0.007496468, -0.028094107, -0.032075282, 0.007955291, 0.039585866, -0.05782586, -0.024282344, -0.020512933, -0.016235288, 0.014654112, 0.0029029401, -0.01215529, 0.04111057, -0.039247043, -0.0016897053, -0.008301173, 0.04478116, 0.031510577, -0.021359991, -0.018776463, -0.03975528, 0.0040835277, 0.005347057, 0.036621165, -0.06640939, 0.04063057, 0.020978816, -0.034701165, -0.027035285, 0.010792937, -0.07764703, -0.00046411747, 0.041505866, 0.040404692, -0.027670577, 0.042155277, 0.014682348, -0.0028799989, -0.019891758, -0.00969882, -0.023378816, -0.060988214, -0.025214108, 0.016432935, -0.0140894065, -0.05963292, 0.02532705, 0.027670577, -0.033910576, -0.019976463, -0.019016463, 0.043143515, -0.014738818, -0.003688234, -0.037552927, -0.026682343, -0.051896453, -0.04017881, -0.005488233, -0.03831528, -0.038992926, 0.022348227, -0.07403292, -0.016009405, -0.01191529, -0.040122338, -0.01017882, 0.02377411, -0.030465871, 0.009261173, -0.020922346, -0.024465874, -0.01939764, -0.00054441154, -0.027769402, -0.024084697, 0.019482346, 0.020498816, 0.051896453, 0.014498818, -0.019807052, -0.013171759, 0.07002351, 0.007941173, 0.013884701, 0.010171761, 0.02551058, 0.014167054, 0.030719988, 0.012847054, 0.026414108, 0.006017645, -0.013411759, 0.012162348, -0.029082343, -0.028447049, 0.02987293, -0.062287036, 0.018310582, 0.0035399988, -0.054465864, -0.014597641, -0.032301165, 0.051444687, 0.016757641, 0.027007049, 0.0356894, -0.011668231, 0.03783528, 0.044922337, -0.010214114, 0.022178816, 0.050625864, -0.010482349, 0.017759994, -0.034221165, 0.019185875, 0.028009402, -0.011661172, -0.0095505845, -0.021543521, -0.08035762, 0.020244699, -0.012889407, 0.032131754, -0.02567999, -0.0035399988, -0.0059470567, 0.052376453, 0.008371762, 0.014682348, 0.049411748, -0.029251754, -0.05663998, 0.02783999, 0.05872939, 0.057515275, -0.013044701, -0.031651754, 0.0019923523, 0.0012070584, -0.008992938, -0.006494115, 0.003818822, -0.018465875, -0.007814115, -0.020484699, -0.017505877, -0.046729393, 0.02719058, -0.040574104, -0.014039995, -0.01729411, 0.015416465, -0.04593881, -0.042945866, -0.047858804, -0.019185875, 0.018861169, 0.004034116, -0.039952926, -0.022757638, -0.01358823, -0.002705293, -0.031482343, 0.014223524, -0.0003611028, -0.008703526, 0.046192925, -0.010870584, -0.024734108, 0.007517644, -0.0049305866, -0.00065823505, 0.009367055, -0.011131761, 0.01083529, -0.030945871, 0.071435265, -0.02041411, -0.035604693, 0.005537645, 0.009910584, 0.010482349, -0.04847998, 0.0137294065, 0.026089402, -0.03981175, 0.040602338, 0.0038117634, 0.02329411, 0.0036741162, 0.02551058, 0.020244699, -0.02035764, 0.009007055, 0.009705879, 0.025397638, -0.008124703, -0.020343522, -0.044865865, -0.011237643, 0.027910577, -0.03314822, 0.008823526, -0.02717646, -0.021924699, -0.049355276, 0.013284701, -0.013496466, -0.0016985288, 0.027303519, -0.010679996, -0.0129176425, -0.00044051453, -0.012430584, 0.03537881, 0.017590582, 0.015585877, 0.020343522, -0.03628234, 0.014781171, -0.057487037, 0.04430116, 0.022757638, 0.04147763, -0.042014103, -0.011223526, 0.008096468, 0.0047152922, -0.004108234, 0.04593881, 0.0064588212, -0.0069776443, 0.027204696, 0.020879993, -0.029251754, 0.0082588205, -0.0044258805, -0.04639057, 0.042804692, 0.080075264, -0.006797645, -0.040517632, 0.051896453, -0.004274116, -0.018395288, 0.036367044, 0.004521175, 0.01334823, 0.015797641, -0.0028729402, -0.035604693, 0.027402343, 0.0012397055, 0.00856235, -0.015938818, -0.011950584, -0.0050011748, 0.006314115, -0.09012703, -0.0075388206, -0.020639993, -0.0051211747, -0.024225874, 0.04514822, 0.026329402, -0.026202343, -0.027444696, -0.045232926, 0.03543528, 0.0056294096, -0.0020629405, -0.0052411747, -0.05615998, 0.022771757, 0.036254104, -0.0150070535, -0.0159247, 0.0029858812, -0.008265879, 0.07476703, -0.042494103, 0.06612703, 0.027515285, 0.044385865, 0.018183522, 0.03035293, -0.056668214, -0.022644697, 0.033374105, 0.019609405, 0.011428231, 0.009910584, 0.03077646, 0.035830576, 0.072225854, 0.009268232, 0.044075277, -0.017251758, 0.030381165, 0.0023629402, 0.039952926, 0.04190116, 0.021529404, -0.005788233, -0.03221646, -0.068385854, -0.008103526, -0.05246116, -0.0065470566, -0.010376466, 0.0050258804, 0.01191529, 0.012112937, -0.060141154, -0.03382587, 0.026583519, 0.0313694, -0.029054107, -0.01418823, -0.041759986, 0.0009970585, 0.072395265, -0.025214108, -0.016037641, 0.01275529, -0.00933882, 0.0065929387, -0.017011758, -0.06861174, 0.01257176, -0.028150577, -0.07476703, -0.004125881, -0.052828215, -0.040743515, -0.020428227, -0.02436705, 0.057317626, 0.0027158814, -0.047407042, 0.07346821, -0.0058164685, 0.09475761, -0.0060035274, 0.035887044, -0.04986351, 0.0059858803, 0.036254104, -0.0055235275, 0.049411748, 0.0150070535, -0.034644693, 0.024508227, -0.007997644, 0.012472937, -0.00018871317, -0.008428232, -0.028207049, -0.01999058, -0.01143529, 0.025821166, 0.0021052933, 0.0062470566, -0.053449392, 0.0043305866, -0.016221171, 0.048056453, 0.02519999, -0.040912926, 0.083858795, -0.0019605875, -0.005890586, 0.001951764, -0.008943526, 0.014597641, -0.024451755, 0.016997641, -0.016602347, 0.053505864, -0.005661174, 0.017675288, 0.014738818, 0.029647049, -0.0062788213, 0.053647038, -0.033684693, -0.0077858795, 0.05423998, -0.005735292, 0.00082544086, 0.0636988, 0.03410822, -0.024592932, 0.065279976, -0.03557646, -0.032611754, -0.011421172, -0.0056470567, -0.026498813, 0.0029699989, -0.005957645, 0.020004699, 0.012959995, 0.013094113, -0.013722348, 0.0190447, -0.007447056, 0.010559996, 0.0020911756, -0.012783525, -0.03681881, -0.007919997, -0.010023526, -0.0039423513, 0.03645175, -0.033035282, 0.00042330867, 0.039190575, -0.029675283, -0.02807999, 0.023265874, 0.0056858803, -0.043256454, 0.0035170575, 0.017082347, 0.022122344, 0.0031341165, -0.0095364675, 0.012924701, 0.0033776457, 0.050682336, -0.030465871, 0.016037641, -0.0030458812, -0.007990586, -0.021345874, 0.02759999, -0.029731754, -0.005717645, -0.0037941162, -0.017745877, -0.009239997, -0.01538823, -0.028475285, 0.027515285, -0.027402343, -0.009239997, -0.0004173528, -0.022404697, 0.030211754, 0.01729411, 0.012825877, 0.017195288, 0.0050329394, 0.007404703, -0.017887052, 0.032724693, 0.005064704, 0.022305874, 0.0024229402, 0.020992933, 0.00010130511, 0.019764699, 0.0051741158, 4.701285e-06, -0.03653646, -0.016898816, 0.03083293, -0.017985877, -0.034051754, 0.017971758, 0.017279994, -0.042352926, -0.0050505865, 0.014011759, -0.02741646, 0.026385874, 0.054409392, -0.024112932, 0.021388227, 0.026315285, -0.008555291, -0.015402347, -0.011484702, 0.033571754, -0.00010637864, -0.017138816, 0.031143518, 0.0035911752, 0.012007055, -0.042663515, 0.0013305878, 0.01023529, 0.0067658797, 0.003395293, -0.015359994, -0.04387763, -0.008167055, -0.0053294096, 0.020188227, 0.006716468, 0.0144211715, 0.040602338, -0.021204699, 0.007461174, 0.0045176456, -0.014004701, 0.010298819, 0.016969405, 0.021811757, -0.010771761, -0.03252705, 0.010743526, -0.00012606612, -0.05183998, 0.044865865, 0.009974114, 0.008195291, -0.012112937, 0.012585877, 0.008957644, 0.020583522, 0.005604704, -0.0049129394, -0.012762348, -0.0067552915, 0.0052235276, -0.0069352915, -0.04291763, 0.0029435284, 0.020795286, 0.021642344, -0.024691755, -0.029110577, -0.0007972056, -0.018818816, 0.075444676, 0.005191763, 0.006790586, -0.01802823, 0.0024105874, -0.010559996, 0.001266176, -0.016715288, -0.030381165, 0.01939764, -0.036564693, -0.019623522, 0.009331761, 0.030550577, 0.0031923517, 0.011957643, 0.0028994107, 0.0045070574, -0.08719055, 0.0065823505, 0.010397643, 0.00820235, -0.003765881, 0.040574104, -0.021204699, -0.033007048, -0.018635288, 0.013404701, -0.015345877, 0.017802346, -0.038287044, -0.020964699, -0.013609407, -0.020202346, 0.011682348, -0.02987293, -0.0011338231, 0.005407057, -0.0122258775, -0.04221175, -0.008555291, -0.015331759, -0.014527054, 0.01657411, -0.0141952885, 0.03998116, -0.0058941157, -0.081035264, -0.008604703, -0.0061588213, 0.011767055, -0.0098823495, -0.01359529, 0.005075292, -0.0025941166, 0.059407037, 0.013207054, -0.027755285, -0.015543523, -0.008901173, -0.0033917634, 0.0069847032, -0.03557646, 0.025454108, -0.0074752914, -0.008209409, -0.0180847, -0.06070586, -0.02209411, 0.014047054, -0.0023664697, -0.002687646, 0.011576466, -0.00999529, 0.022291757, -0.023392932, -0.029195283, 0.007969409, 0.018536463, 0.021769404, 0.02551058, -0.03441881, -0.0013579407, 0.007919997, -0.015967052, 0.0045635276, 0.007976468, -0.019115288, 0.015995288, -0.023971755, 0.01850823, 0.013602348, 0.010905879, -0.0031976458, -0.0014770583, 0.016461171, -0.031341165, -0.017887052, -0.021698816, 0.013397642, 0.001909411, -0.008251762, 0.0380894, 0.013122348, -0.009070585, -0.008823526, -0.025058813, 0.015430583, 0.016503524, -0.0074329386, 0.011082348, 0.026484696, -0.03035293, -0.020371757, 0.006141174, 0.01017882, 0.009303526, 0.054155275, 0.00036992633, 0.019567052, 0.038964693, 0.003988234, 0.030861165, -0.023223521, 0.037044693, -0.009049408, -0.013482348, 0.00087220554, 0.0028588225, -0.00077514676, -0.054861158, 0.015402347, -0.016362347, -0.0011602937, -0.022757638, -0.0070694094, -0.02023058, -0.032894105, 0.009889408, 0.01771764, 0.019835288, -0.021119991, -0.001814117, 0.010637643, 0.0028958812, -0.10605173, 0.0037411752, 0.031228224, -0.022164697, -0.023816463, -0.011167055, 0.024734108, 0.008999997, -0.012338819, -9.816173e-05, 0.010898819, -0.009077644, 0.011534113, -0.01969411, 0.012967054, -0.021599991, 0.005622351, -0.025171755, 0.034983516, -0.03526587, -0.0071752914, 0.0056788214, -0.023237638, -0.013404701, -0.044724688, 0.009832937, -0.030804694, -0.0056541157, 0.010694114, 0.031199988, -0.019256463, 0.012501172, -0.025101168, 0.0073482324, -0.016461171, 0.0145411715, -0.008971761, 0.0007861762, 0.008901173, 0.0077717616, 0.039529398, -0.0051599983, -0.004944704, -0.027684696, 0.009790584, -0.034701165, -0.0126776425, 0.0030758812, 0.017802346, 0.019623522, 0.0016332347, 1.8791354e-05, 0.011322348, 0.017039994, 0.020301169, -0.029195283, 0.014625877, -0.009501173, 0.003167646, 0.04111057, 0.04514822, 0.05167057, 0.01358823, -0.01299529, -0.0050188215, -0.0053188214, -0.0075882324, 0.018776463, -0.0453741, -0.04370822, -0.037242338, 0.016969405, -0.0105882315, 0.020978816, 0.012162348, 0.015190583, 0.025849402, 0.017858816, -0.0015776465, -0.004108234, 0.055708215, -0.018451758, -0.006878821, 0.029759988, -0.0075952914, 0.038343515, 0.010842349, 0.017167052, 0.003670587, 0.05178351, 0.050795276, -0.038767044, -0.009797644, 0.027148224, -0.057487037, -0.026315285, 0.024522344, 0.014851759, 0.030663518, 0.0009952937, 0.006896468, 0.005844704, -0.01826823, -0.023181168, -0.030578813, 0.0077788206, 0.003077646, -0.007828233, -0.023872932, -0.000723088, 0.027741166, 0.035237633, -0.009395291, 0.0366494, -0.0042423513, 0.03681881, 0.03532234, -0.00940235, -0.0039599985, -0.021416463, -0.0002644852, 0.029703518, -0.005131763, -0.0021829405, 0.0030811753, 0.04291763, -0.01753411, -0.0017073523, 0.008089409, 0.0077082324, 0.027684696, -0.0009926467, -0.045063514, -0.019962346, -0.017816463, 0.003275293, -0.009663526, -0.0012670583, -0.005407057, 0.014823524, -0.04071528, -0.003705881, 0.026174108, 0.0007147056, -0.061891742, -0.048564687, -0.01634823, -0.007447056, -0.01418823, -0.009861173, 0.011414113, 0.042635277, 0.004598822, 0.020583522, -0.0012794113, -0.005611763, 0.05006116, 0.0058799977, -0.009967055, 0.058447037, 0.013736466, 0.026894107, 0.015232936, 0.009352937, -0.004468234, 0.0096564675, -0.013009407, -0.0013491171, -0.0052905865, 0.0059399977, -0.040207043, -0.0030282342, -0.0016808817, 0.030861165, 0.02939293, 0.0074823503, -0.00020536757, -0.0019023522, -0.0138494065, 0.02735999, 0.008004703, 0.021825874, -0.006557645, -0.028390577, -0.009317644, -0.010256466, 0.021232933, 0.016390583, 0.023661168, -0.0166447, -0.003280587, 0.02311058, -0.010009408, 0.019185875, -0.03605646, -0.0028270578, 0.014287054, 0.021105874, -0.025312932, -0.03458822, -0.061496448, 0.049327042, 0.022588227, -0.018748228, 0.035011753, 0.023858815, -0.016856464, 0.017011758, -0.0055164685, 0.024762344, -0.0173647, 0.007076468, -0.010644702, 0.0111741135, -0.033797633, -0.0045670574, -0.017618816, 0.008265879, 0.01825411, -0.020625874, -0.059971742, -0.026752932, -0.022108227, 0.01969411, 0.0033776457, 0.021585874, 0.030861165, -0.016277641, 0.0042423513, 0.0035876457, -0.041816454, -0.029901166, -0.023929402, 0.009381173, -0.024056463, 0.03334587, -0.042014103, -0.011477643, -0.004951763, 0.020498816, 0.054974098, 0.005347057, -0.018183522, -0.01814117, -0.00049764686, -0.015585877, 0.025397638, 0.019482346, 0.0034147047, -0.02161411, -0.025595285, -0.002669999, 0.027755285, -0.014527054, 0.019383522, -0.00061323505, 0.04627763, 0.023491757, 0.04895998, 0.0073552914, -0.022828227, 0.032470576, 0.0065152915, -0.03735528, 0.0034976457, -0.0011144114, -0.022884697, -0.011322348, 0.03687528, -0.014879994, 0.025905874, 0.038399987, 0.006741174, 0.03201881, -0.010411761, -0.009705879, 0.014767054, -0.00808235, 0.009303526, -0.016150583, 0.020682346, 0.025962343, -0.018988227, 0.0026152933, -0.0061199977, -0.024522344, 0.0055164685, 0.0041294103, -0.0045176456, 0.001831764, -0.017999994, 0.0029311753, 0.0070976443, 0.0037129398, -0.003513528, 0.002082352, 0.010679996, 0.010143526, 0.013122348, -0.03777881, 0.010199997, -0.0031094106, 0.025891755, -0.03334587, -0.014138819, 0.011237643, 0.026809402, -0.045487043, -0.003945881, 0.014209406, -0.016122347, 0.02891293, 0.005495292, 0.019002346, -0.016475288, -0.030127048, -0.012437643, -0.02987293, 0.010185879, -0.013814112, -0.02573646, 0.018239994, -0.003921175, 0.03537881, -0.018338816, -0.03218822, -0.0029594107, -0.0039670574, -0.008534115, 0.0070094094, -0.00018794111, -0.0065823505, -0.013736466, 0.014075289, -0.015331759, 0.020527052, -0.014061172, -0.0042952923, 0.006794115, -0.008689408, -0.022799991, 0.020597639, -0.019425875, 0.0004914704, 0.0051494096, -0.0041294103, -0.011251761, -0.016249405, 0.013679995, 0.01771764, 0.08990114, -0.053308215, -0.020159993, -0.024889402, 0.004867057, 0.0032717634, -0.015487053, -0.02460705, -0.06358586, -0.01269176, -0.028362343, 0.016602347, 0.0071223504, 0.0042599984, 0.0031394106, 0.015995288, -0.007376468, 0.0057988213, 0.033571754, 0.008068232, 0.01730823, 0.050428215, -0.027698813, 0.0032082342, 0.006938821, 0.012239995, 0.019072935, 0.027303519, 0.007439997, 0.026018813, -0.008872938, -0.03794822, -0.010143526, -0.0016049994, -0.045119982, 0.00088191143, 0.020131757, 0.029364696, -0.018381169, -0.0039176457, -0.027571755, 0.015049406, 0.00873882, 0.0006383821, 0.007616468, 0.010157643, 0.0062647033, -0.0059999977, 0.005435292, -0.021077638, -0.028644696, -0.017816463, -0.024381168, -0.013743524, 0.046898805, -0.036959987, -0.0069423504, 0.023025874, 0.0013844112, -0.0070199976, 0.0009705879, 0.043849397, -0.01843764, 0.0054388214, -0.012974113, -0.05477645, -0.02268705, 0.010214114, -0.0048705866, 0.0150070535, -0.051698804, -0.013432936, 0.021952933, -0.014392936, 0.013369407, -0.00044492632, -0.014336465, -0.02239058, 0.040432926, 0.011110584, 0.0005713233, -0.040404692, -0.034672927, -0.018395288, -0.009564702, 0.00042242632, 0.04159057, -0.0065823505, -0.03998116, -0.0028694107, -0.019976463, 0.013157642, 0.041392926, 0.047491748, 0.012790583, -0.025934108, -0.017491758, -0.017943522, 0.010828231, -0.01987764, 0.013072937, -0.009557644, -0.017096464, 0.001403823, -0.024451755, -0.023901168, -0.051388215, 0.015515288, 0.020512933, -0.00043477924, 0.006737645, -0.0026241166, -0.004944704, 0.015119994, -0.006861174, 0.020498816, -0.004196469, 0.06674821, -0.031171754, -0.03602822, 0.0067658797, -0.0060529388, -0.004228234, -0.0016888229, 0.007757644, -0.0111741135, 0.023816463, 0.007531762, -0.0006397056, 0.0058694095, 0.015331759, 0.0019464699, 0.02401411, -0.0010120585, -0.007651762, -0.010496466, -0.012416466, -0.0046235276, -0.035632927, 0.02939293, -0.008145879, 0.0144211715, -0.027995285, -0.043087043, 7.196688e-05, 0.025552932, 0.014597641, -0.010602349, 0.013524701, -0.016418818, -0.026004696, -0.008294115, 0.004524704, 0.001831764, -0.001874117, -0.03580234, -0.0152470535, -0.001041176, 0.015727052, 0.011364702, 0.03458822, 0.0032311752, -0.024381168, -0.0004021322, 0.011830584, -0.008654115, -0.033797633, 0.0020082346, 0.048112925, 0.01107529, -0.01897411, -0.01274823, -0.016221171, 0.006734115, -0.0029188225, -0.01239529, 0.0024758815, 0.030381165, 0.0043729395, 0.037750576, -0.017378816, -0.005371763, -0.0017514699, -0.019129405, -0.024917638, 0.016842347, -0.021712933, -0.002562352, -0.010192937, 0.018310582, 0.004902351, -0.032159988, -0.005537645, 0.003395293, 0.034334105, 0.034870576, -0.015783524, 0.028630577, 0.024691755, -0.019595288, 0.019425875, -0.0061588213, -0.015374112, -0.031454105, -0.027021166, 0.036084693, -0.005470586, -0.03585881, 0.0057317624, 0.018310582, -0.0040305867, 0.0044258805, -0.0020276464, -0.018959993, 0.021938816, -0.021656463, 0.0031694106, -0.007531762, -0.0075105852, -0.023449402, -0.025905874, 0.01826823, 0.03933175, 0.0037094103, -0.025369402, -0.002592352, -0.022291757, -0.02263058, -0.0028870578, -0.008985879, 0.0045035277, -0.02669646, -0.026301166, 0.023661168, -0.0008077938, 0.019030582, -0.03837175, -0.008830585, 0.010079997, 0.022376463, 0.017491758, 0.02065411, -0.0006383821, 0.018861169, -0.0044999984, 0.037327044, 0.005339998, -0.028263519, -0.004365881, -0.02401411, 0.028941166, 0.008901173, -0.017195288, -0.015430583, -0.014159995, -0.026851755, 0.01406823, 0.0058764685, -0.02532705, 0.0041894102, -0.057628214, -0.020795286, 0.0027564697, 0.011230584, 0.0007208821, 0.014738818, 0.0018388228, -0.010185879, -0.040969398, 0.03794822, -0.0056329393, -0.011131761, -0.012642348, -0.018451758, 0.030804694, 0.03789175, 0.011442348, -0.003393528, -0.00975529, -0.029138813, 0.009515291, -0.022517638, -0.0050364686, -0.014209406, -0.010037643, 0.014258819, 0.0075105852, 0.014470583, 0.020992933, 0.00010935658, -0.016941171, 0.031397637, 0.006501174, 0.0185647, -0.0138494065, 0.0074188206, -0.0013826465, -0.016517641, -0.009232937, 0.0030899988, -0.033374105, -0.0068752915, 0.0035752929, -0.018211758, 0.009917644, 0.030691754, -0.014174112, 0.03526587, -0.015063524, -0.016912935, -0.003555881, -0.024903521, -0.008872938, 0.0032735283, 0.021698816, 0.013602348, 0.017209405, -0.019468227, -0.027585872, 0.0054388214, 0.0025888225, 0.029675283, -0.021162346, -0.011788231, 0.0026947048, -0.0069176443, -0.026922343, 0.036423516, 0.0052941157, 2.7380505e-05, -0.0028870578, 0.023576463, 0.023985874, 0.015303524, -0.0038858808, -0.008103526, -0.019863522, -0.02071058, 0.016418818, -0.012254113, -0.017449405, -0.0073623504, 0.010552937, 0.006787056, 0.017816463, 0.028390577, -0.015303524, 0.016517641, -0.024945874, 0.005541174, -0.0035999988, 0.017322347, 0.009543526, -0.009070585, -0.0023205874, -0.0025835284, -0.0034764693, 0.008364703, 0.012719995, -0.057035275, -0.042776454, 0.0023929402, 0.016983524, 0.022997638, -0.011329408, 0.020159993, -0.020724699, -0.050625864, -0.026414108, 0.002472352, -0.022418816, 0.007447056, 0.028404696, -0.012614113, -0.029929401, -0.006077645, 0.0010279408, -0.017209405, 0.010651761, -0.027967049, 0.021811757, -0.026329402, 0.027571755, -0.012762348, 0.024649402, -0.0144352885, 0.00014944847, 0.017167052, -0.010644702], "content_hash": "9a82f9c41e8422d7a6ef098b8129b12e"}, {"content": "# Experiment: Your First Analysis\n\n## Goal\n\nLearn how to run your first analysis and walk through each pipeline stage to understand how a transformer model processes text.\n\n## Prerequisites\n\nNone -- this is the starting experiment.\n\n## Steps\n\n### Step 1: Select a Model\n\n1. In the **generator section** at the top, find the \"Select Model\" dropdown.\n2. Choose **\"GPT-2 (124M)\"** from the list.\n3. Wait for the model to load. You'll see a status message indicating the model is ready.\n\n### Step 2: Enter a Prompt\n\n1. In the **\"Enter Prompt\"** textarea, type: `The cat sat on the`\n2. Leave the generation settings at their defaults (1 beam, a few tokens).\n\n### Step 3: Run the Analysis\n\n1. Click the **\"Analyze\"** button.\n2. Wait for the analysis to complete. The pipeline stages and generation results will appear.\n\n### Step 4: Explore the Generated Sequences\n\nLook at the **generated sequence(s)** below the generator. You should see how GPT-2 continues your prompt. Common completions might include \"mat,\" \"floor,\" \"bed,\" or similar words.\n\n### Step 5: Walk Through the Pipeline\n\nNow expand each of the **5 pipeline stages** by clicking on them:\n\n**Stage 1 - Tokenization**: Click to expand. You'll see your prompt split into tokens. Notice how each word (and its leading space) becomes a separate token. Count the tokens -- \"The cat sat on the\" should produce about 5 tokens.\n\n**Stage 2 - Embedding**: Click to expand. You'll see that each token was converted into a 768-dimensional vector. This is GPT-2's hidden dimension.\n\n**Stage 3 - Attention**: Click to expand. This is the richest stage:\n- Look at the **head categories**. You should see heads grouped into Previous-Token, First/Positional, Bag-of-Words, Syntactic, and Other.\n- Click on a category (like \"Previous-Token\") to see which specific heads belong to it.\n- Below the categories, you'll see the **BertViz visualization**. Try clicking on individual head squares to see their attention patterns.", "source_file": "experiment_first_analysis.md", "chunk_index": 0, "embedding": [-0.0011164723, 0.029201265, 0.017076392, -0.04438592, -0.02089795, -0.04204982, -0.005459367, 0.03229914, 0.026890555, 0.07973215, 0.05101334, -0.023919646, -0.022091392, -0.038545668, 0.007909734, 0.020453582, -0.0002761424, 0.022776987, 0.0068559493, 0.07912273, 0.037149087, -0.026712809, -0.0059799114, 0.036488887, 0.029099695, -0.029861467, -0.00778912, 0.048753418, 0.00097046595, -0.026585847, 0.018739594, -0.010658462, -0.058961164, -0.011794772, 0.011128222, -0.005326057, -0.03897734, 0.030750202, -0.007643114, 0.010201398, 0.0071479618, -0.0142451385, -0.020986822, 0.0095031075, -0.006014826, 0.034406707, 0.011305968, -0.03095334, -0.020783683, 0.011705899, -0.01430862, 0.03940901, 0.0038342532, -0.0059830854, -0.008963519, -0.024948038, -0.038494885, -0.022776987, -0.03547319, -0.03359415, 0.011915386, 0.0110329995, 0.023361012, 0.04456367, 0.006970215, -0.010867949, -0.037961643, 0.022802379, 0.0030470886, 0.01955215, 0.026636632, 0.004640462, -0.028160177, 0.0055355444, 0.0073130126, -0.012772379, -0.028718809, 0.036488887, -0.025659025, -0.034889165, 0.012251835, -0.020237746, 0.00038941632, -0.030394709, 0.0076177213, -0.0007947029, -0.028693417, -0.02249767, -0.041262656, -0.04235453, 0.004100873, 0.016898645, -0.05271463, 0.029404404, 0.036260355, 0.0070400443, -0.0102394875, -0.0114011895, -0.008379494, 0.03283238, 0.050810203, -0.0054466706, 0.0005832318, 0.012600981, 0.020047303, -0.019767987, 0.023018215, 0.014219747, -0.056421924, -0.034812987, -0.063938074, -0.02419896, 0.013356404, 0.0049388227, 0.038063213, 0.02620496, -0.006722639, -0.0021043955, 0.0053673196, -0.047585364, -0.026154177, -0.015959127, -0.008703247, -0.010309316, -0.032375317, -0.021812076, 0.023716506, -0.00640206, 0.022738898, -0.009566588, -0.020936038, 0.026865164, 0.01835871, -0.025697114, -0.021405797, -0.013242139, -0.023462582, 0.026915949, -0.025468582, 0.029353619, 0.008265228, -0.020415492, -0.01885386, 0.038596455, -0.05418739, -0.061398834, -0.01614957, 0.064750634, -0.043573365, 0.016073393, 0.033035517, 0.024579847, -0.03791086, -0.025189266, 0.01952676, -0.03981529, 0.004916604, 0.01818096, 0.002445606, -0.060078427, 0.03519387, 0.016200354, -0.03773311, -0.008430278, -0.046671238, 0.04596025, -0.032984734, 0.003453367, -0.027957037, -0.057183694, 0.00037295095, -0.02159624, 0.002205965, -0.030369315, 0.02079638, -0.00066893117, -0.10385493, -0.07038774, -0.025367012, -0.0701846, -0.01680977, -0.00065821875, -0.032680023, -0.019183962, -0.03656506, -0.026255745, 0.018295227, 0.0068749934, 0.0047928165, 0.016124176, 0.0045547625, 0.02650967, 0.039662935, 0.0022313576, -0.006728987, -0.0073384047, 0.07262228, 0.01668281, 0.025163872, 0.030089999, 0.04773772, 0.023157872, 0.008976215, -0.015933733, 0.023310227, 0.027119087, -0.03367033, 0.013902341, -0.061652757, -0.036895163, 0.0377839, -0.061449617, 0.031181872, 0.016898645, -0.056167997, -0.028134784, -0.04700134, 0.015324316, 0.043852683, 0.05926587, 0.024884555, 0.025125785, 0.02430053, 0.012042347, 0.0041103954, 0.034559064, 0.026890555, -0.053425618, 0.02477029, -0.010201398, 0.015552848, 0.016835164, -0.015057696, -0.008163658, 0.014346709, -0.034355924, 0.031511974, 0.022015214, -0.022637328, -0.009903038, -0.0052594016, 0.009382494, 0.027119087, 0.009249183, 0.054035038, 0.018866556, -0.03389886, -0.07495838, 0.051698934, 0.06368415, 0.016162265, 0.002707465, -0.026433492, 0.03346719, -0.020148873, -0.004459541, -0.020440886, 0.017165266, 0.0022265965, -0.01143293, 0.008798468, 0.0041611805, -0.035981037, -0.019006215, -0.0076494617, -0.023361012, -0.037199873, -0.03232453, -0.010772727, -0.053171694, -0.0055514146, 0.004205617, 0.020440886, 0.005084829, -0.061449617, -0.008639766, 0.005097525, -0.011109177, -0.0069257785, 0.031613544, -0.044614453, -0.0021567673, 0.04055167, 0.05322248, -0.018561848, 0.019729897, 0.022218354, -0.009541196, 0.0053736675, -0.021964429, 0.0591643, -0.015692506, 0.03189286, -0.02650967, -0.004434149, 0.028083999, -0.004205617, 0.01889195, -0.019336317, 0.039764505, 0.034914557, -0.036031824, 0.054898378, 0.014702202, 0.018904645, 0.02861724, -0.017266834, -0.030089999, 0.012931082, 0.009814165, 0.0063068382, -0.018269835, 0.0041167433, -0.029277442, 0.017216051, 0.018396797, 0.01741919, 0.0026662024, -0.006532196, -0.010410886, -0.03283238, -0.041872077, 0.037428405, 0.0054879333, -0.0059481706, 0.03463524, -0.014994215, 0.019310923, 0.025468582, 0.012366101, 0.023107087, -0.024224354, -0.0056117214, 0.03600643, -0.04192286, -0.035701722, -0.016632024, 0.06617261, 0.028033214, 0.041262656, -0.09816703, -0.005694247, -0.0102394875, -0.01230262, -0.06581711, 0.07338405, 0.01293743, 0.00889369, 0.012810468, -0.013953126, -0.03824096, 0.00047452055, -0.0047356836, -0.018980822, 0.036260355, 0.034127392, -0.011737639, -0.0020298052, 0.00075224997, -0.022002518, -0.019133177, 0.02517657, 0.05428896, -0.018447582, -0.005141962, 0.0223961, -0.006398886, 0.023081696, -0.008652462, -0.011071089, -0.0021535933, 0.026154177, 0.020567847, -0.023018215, -0.009319012, -0.01617496, -0.017089088, -0.013623025, -0.027398404, 0.07749762, -0.021913646, 0.033848073, -0.035955645, -0.02968372, 0.025201961, 0.042126, 0.0024059303, 0.008411234, -0.04339562, 0.0404501, -0.011045696, -0.047483794, -0.058859594, 0.015425886, -0.016720898, 0.03483838, -0.010328361, 0.022421492, 0.059621364, 0.03813939, 0.007985911, -0.026915949, -0.03633653, -0.019298228, -0.008328709, 0.024110088, 0.036895163, 0.03133423, 0.024744898, 0.015324316, 0.04763615, -0.008138265, 0.017914342, -0.04362415, -0.014448278, 0.024821075, 0.03372111, 0.034508277, 0.011020304, 0.024224354, -0.029379012, -0.06830557, -0.024998821, -0.0018742769, -0.06637575, 0.015794076, 0.03697134, -0.013153265, -0.0038723417, -0.025443189, -0.010379145, 0.009147613, 0.03798704, -0.0115345, -0.016657418, -0.031131087, 0.026179569, 0.06881341, 0.02159624, -0.020453582, 0.016936734, -0.019412493, -0.033441797, -0.017152568, -0.043674935, 0.021266138, -0.011394842, -0.06795008, -0.013673809, -0.037022125, -0.0019393449, 0.0027772943, 0.002702704, 0.0033549715, -0.0012005846, -0.019882252, 0.027423797, -0.0015346535, 0.10009686, 0.007858949, 0.0042754463, 0.013089784, -0.0010521978, 0.00013688093, -0.0058719935, 0.010709247, -0.041567367, -0.028820379, 0.020440886, -0.02075829, -0.006659158, -0.023157872, -0.00879212, -0.02560824, -0.007160658, -0.020313924, 0.012772379, 0.02075829, -0.013038999, -0.02788086, 0.009814165, 0.0075415443, 0.038545668, 0.044233568, -0.023653025, 0.07302856, -0.060281567, 0.0005586329, 0.014956126, -0.017368404, -0.0213931, -0.013013607, 0.028896555, -0.0135595435, 0.018777683, -0.0021123306, -0.0059386487, -0.023538759, -0.008062088, 0.018269835, -0.006456019, -0.013699202, 0.0076748542, 0.059621364, 0.0054308004, -0.017863557, 0.029404404, 0.045325443, -0.0076367655, 0.040907163, -0.05007382, 0.010658462, -0.017152568, 0.010309316, 0.0013616676, 0.037225265, 0.020720202, 0.027703114, 0.013978519, 0.0016711376, -0.0007220965, 0.0037898163, 0.011712247, -0.008976215, -0.019729897, -0.026712809, -0.03821557, -0.0785641, -0.031283442, -0.011159962, 0.03105491, 0.0010307729, 0.00051379943, 0.013546848, -0.048067823, -0.0067416835, 0.016200354, -0.024922645, -0.06662967, 0.002755076, -0.01774929, 0.016022608, 0.023335619, -0.002188508, -0.0015283054, 0.010880645, 0.028490277, 0.0014449866, 0.015387797, 0.019945733, 0.036285747, -0.03940901, 0.02497343, -0.012702551, -0.01955215, 0.0056720283, 0.005015, -0.03801243, 0.011617025, -0.041288048, 0.00879212, 0.0048689935, 0.02292934, -0.021773987, -0.009325361, 0.05555858, 0.025659025, -0.026636632, -0.011839208, -0.0083985375, 0.01965372, -0.006532196, 0.024744898, -0.007459019, 0.012892993, -0.032172177, -0.030267745, 0.018739594, 0.0033073607, 0.0042373575, -0.016619328, -0.0058910376, -0.02059324, 0.027169872, -0.009027, -0.0042024427, 0.010614025, 0.0041707023, -0.01881577, -0.026433492, 0.047940858, -0.008341405, -0.007839905, 0.010779075, -0.04309091, -0.01394043, 0.013242139, -0.016657418, -0.03747919, 0.012429582, 0.031867467, -0.009991911, -0.03623496, 0.022586543, -0.02958215, 0.010404537, -0.02524005, 0.002894734, 0.010690202, -0.0037136392, -0.035727113, -0.011477367, -0.03222296, -0.005456193, -0.026331924, 0.025392404, -0.016048, -0.007916082, 0.0027979256, -0.007776424, 0.012213746, -0.02079638, 0.011394842, -0.03613339, 0.012264531, 0.00022317543, -0.006728987, -0.009331709, 0.010493411, 0.03234992, -0.00035231962, 0.04326866, 0.03928205, -0.006716291, -0.013724594, 0.004008826, 0.019514063, -0.015425886, -0.0031930949, -0.027626935, -0.0020329794, -0.023145176, -0.007198747, -0.017596936, 0.004472237, 0.018561848, 0.002829666, 0.027931646, -0.028058607, -0.024427492, -0.064750634, -0.02089795, 0.06525848, -0.0008058121, 0.02704291, -0.009096829, -0.0040469146, -0.047712326, 0.0035549367, -0.0064877593, -0.042278353, 0.026458886, -0.023462582, -0.011648766, -0.010360101, 0.019844163, 0.010906038, 0.022180265, 0.020313924, -0.0049769115, -0.07186051, -0.012175658, 0.011909038, -0.009852253, 0.0067797718, -0.022154873, -0.00032117424, -0.058250174, 0.014676809, 0.026738202, 0.0036215917, -0.0004042947, -0.029734505, 0.022345316, 0.008284272, -0.006602025, 0.0041675284, -0.021608936, 0.015882948, -0.005456193, -0.009223791, -0.018206354, -0.010480715, -0.012378797, -0.0044944556, 0.0052371835, -0.028591847, 0.04890577, -0.02102491, -0.08978754, -0.04014539, 0.0007736748, -0.018003214, 0.0037453796, -0.007160658, -0.019361708, -0.03115648, 0.04593486, 0.028998125, -0.03062324, -0.02694134, -0.010042696, 0.010182355, 0.022548456, -0.024986126, 0.0097697275, -0.0149307335, -0.011693202, 0.0007367765, -0.025836771, -0.020415492, 0.008404885, -0.022408796, 0.024160873, 0.0036660284, -0.052054428, 0.0013965822, -3.394746e-05, -0.022218354, 0.0424561, 0.03697134, -0.004716639, 0.014879949, -0.04730605, 0.023373708, 0.00876038, 0.006881342, 0.01948867, -0.007141614, -0.021939037, 0.00072328676, 0.012550196, 0.0077065947, 0.038317136, -0.0041199178, -0.030699417, -0.022561152, 0.042836986, 0.003958041, -0.033517975, -0.008665158, 0.010207746, -0.008804816, -0.03105491, 0.017063696, 0.007528848, 0.025519367, -0.019945733, -0.016048, 0.037352227, 0.00064512575, 0.0027407927, -0.012061392, 0.024389405, -0.010639417, -0.0061862245, 0.04235453, 0.022319924, 0.011877297, 0.0035422405, 0.020707507, -0.0065702847, 0.03750458, -0.01835871, 0.03392425, -0.018333316, 0.051927466, 0.006862297, -0.014676809, -0.0015417951, -0.009236488, 0.026814379, -0.02777929, -0.033543367, 0.025646329, -0.0032327706, -0.017711202, -0.007268576, -0.03666663, -0.011693202, -0.024554456, -0.02714448, 0.032959342, -0.014321316, -0.014067392, -0.0056244177, -0.02450367, -0.115281515, -0.0012632721, -0.002393234, -0.02915048, 0.0013862666, -0.034432102, -0.0009664984, 0.023081696, -0.0464681, -0.017127177, 0.015819468, -0.019463278, -0.0017314446, 0.0058751674, 0.022484975, -0.046874378, 0.020986822, -0.04034853, 0.012454974, -0.022942038, -0.007827208, -0.0059323004, 0.0009791947, -9.308895e-05, -0.04573172, -0.011648766, -0.007382842, 0.012620025, 0.025900252, 0.015755987, -0.03250228, 0.03631114, -0.014105481, 0.010309316, 0.012042347, 0.017508063, -0.034051213, 0.017876253, 0.04329405, -0.016974822, -0.0018885601, -0.009680854, 0.025557455, -0.009541196, 0.047890075, 0.016657418, -0.0123470565, -0.003634288, 0.008220791, 0.023107087, 0.012543848, 0.014499063, 0.021266138, -0.011077437, 0.033340227, 0.023462582, -0.009642766, -0.029505974, 0.022015214, 0.04298934, -0.028896555, 0.032933947, 0.042557668, 0.008487411, -0.0068115126, 0.020542456, -0.019666417, 0.025722506, -0.026357315, -0.04222757, -0.02237071, -0.017508063, -0.00064869656, -0.015629025, -0.018879253, -6.144764e-05, 0.0002561062, 0.035828684, -0.03336562, -0.01611148, 0.02477029, -0.0026836598, -0.016263835, 0.015832163, 0.008804816, 0.03199443, 0.00070344895, -0.025367012, 0.0021202657, 0.0417959, 0.015057696, -0.02212948, -0.009001607, 0.01578138, -0.02286586, -0.023640329, 0.01243593, 0.04372572, 0.015933733, -0.0028122088, 0.007592329, 0.02036471, -0.05545701, -0.008976215, -0.03651428, 0.02450367, 0.0075796326, 0.0033994082, -0.03199443, -0.015667114, 0.023627631, -0.0008839731, 0.0203901, 0.014156265, -0.015768683, 0.047179088, 0.06596947, -0.0058910376, -0.010487063, -0.03694595, 0.0121883545, -0.008163658, -0.01684786, 0.00939519, -0.0544921, 0.0024424319, -0.046391923, -0.0060021295, -0.0350923, -0.015552848, 0.009680854, 0.018942734, -0.016428886, -0.034482885, 0.012778727, 0.0051260917, -0.009630069, 0.0066210693, -0.018003214, -0.003951693, -0.010633069, 0.026484277, 0.018968126, 0.0046848985, -0.023475278, -0.007928778, -0.010886993, -0.03326405, -0.0077827717, 0.0223961, 0.02567172, 0.0053768414, 0.007211443, 0.008639766, -0.003945345, -0.012023304, -0.010448975, -0.026052607, -0.01143293, 0.014486367, 0.0048499494, 0.028896555, 0.030496277, -0.0060687847, 0.008271576, -0.03534623, 0.00031304074, -0.005088003, -0.03252767, 0.013038999, -0.01922205, 0.06998146, 0.028896555, 0.015806772, 0.022942038, 0.029328227, 0.009230139, 0.02159624, 0.011807468, 0.04489377, 0.0051292656, 0.017888948, -0.0051070475, -0.0031089825, 8.26369e-06, -0.019349013, -0.015451279, 0.013648418, -0.03519387, 0.012156613, -0.01818096, 0.025900252, -0.01832062, 0.037225265, -0.004894386, -0.0014632373, 0.027804682, -0.014778379, -0.020478975, -0.02317057, -0.021837467, 0.007592329, 0.016035303, -0.041034125, 0.01013157, 0.018980822, -0.012531152, 0.018295227, 0.0004741238, -0.013991215, 0.0016441582, -0.007909734, -0.02022505, -0.007909734, -0.018066695, -0.0060433922, 0.0036215917, 0.032806985, 0.0051070475, -0.014511759, -0.07069246, -0.049997643, -0.021380404, -0.0049388227, -0.021202657, 0.0024773465, 0.035117697, -0.01774929, -0.007084481, -0.004002478, -0.024465581, -0.039332833, 0.00219803, -0.0012275641, -0.0057005947, 0.027423797, 0.00491343, -0.00019530329, 0.016086088, -0.009534848, 0.03613339, 0.009623721, 0.023475278, -0.009477715, 0.010099829, -0.01691134, 0.05312091, -0.0034057563, 0.009947474, 0.003513674, -0.0085572405, 0.051902074, 0.013965823, -0.0123470565, 0.026052607, 0.0076304176, 0.018460277, 0.025189266, -0.005652984, -0.0026598542, -0.0043008383, 0.033949643, 0.0065004556, -0.053273264, -0.011210746, -0.032578453, -0.020834468, -0.009376145, 0.027423797, 0.015235443, 0.009579284, 0.029759899, -0.009217443, 0.03105491, -0.00849376, -0.035016127, 0.007935126, -0.028210962, 0.02423705, 0.006195747, -0.005195921, 0.014460974, 0.010734639, -0.01832062, -0.0072050947, -0.018168265, -0.045274656, -0.016238442, -0.014702202, -0.0073955376, 0.008379494, -0.012962823, -0.02788086, -0.018206354, -0.0005312567, 0.003885038, 0.026966734, 0.007097177, -0.00024856784, -0.024186265, -0.014397494, 0.00012487905, 0.018104784, -0.059316657, -0.005779946, 0.021304227, 0.0086080255, -0.07566936, -0.023297532, 0.0010783836, -0.008684202, 0.008931778, 0.012702551, 0.02704291, 0.0049991296, -0.008265228, 0.005564111, 0.0015275119, 0.0017219224, -0.02012348, -0.03930744, 0.013064392, 0.008195398, 0.03367033, -0.0069511705, -0.021545455, 0.018599937, -0.0033771899, 0.02724605, -0.0017092262, 0.034508277, -0.0113504045, 0.010537848, 0.011477367, -0.014626025, -0.0038056867, -0.035549365, -0.018231746, 0.0117185945, -0.009077785, 0.022383405, 0.007274924, -0.026636632, 0.01089969, 0.016390797, -0.011274228, 0.015514759, -0.01013157, 0.02186286, 0.030597847, 0.058808807, -0.022789683, -0.027169872, 0.004075481, 0.0038374271, -0.010042696, -0.0044881073, -0.013648418, -0.028439494, -0.00049316813, -0.015755987, -0.00034359097, -0.017596936, 0.011851905, 0.0067035947, 0.013432582, -0.006906734, -0.016009912, 0.05830096, 0.011496411, 0.02386886, 0.059113517, -0.016530456, -0.024097392, 0.015451279, 0.026865164, -0.00034557475, -0.01069655, 0.0065766326, 0.026433492, 0.018561848, -0.04596025, -0.0053070127, 0.01009348, -0.028769594, 0.029531367, 0.0058529493, 0.016225746, -0.037123695, 0.0014981518, -0.0063544493, 0.018561848, 0.026788987, -0.022637328, 0.017254138, 0.030064607, 0.01494343, -0.008976215, -0.011528152, -0.030547062, -0.021189962, 0.0007518532, 0.0019060173, -0.031004125, 0.041211873, -0.043852683, 0.009757032, 0.0016774858, 0.024046607, -0.00056974206, -0.037199873, 0.02006, -0.01698752, 0.023881556, 0.009814165, -0.06561397, -0.03115648, 0.017533455, 0.006767076, 0.022573847, 0.0020885253, -0.016758988, -0.0024948036, 0.0014053109, 0.024592543, 0.011788424, 0.019209353, -0.008011304, 0.027068304, 0.023741897, -0.021786682, -0.031131087, -0.01631462, -0.0027169872, 0.012931082, -0.006475063, 0.040297747, -0.010944126, -0.023005519, 0.009255531, -0.006452845, 0.01768581, 0.03981529, 0.063379444, 0.010436278, -0.0014489541, -0.013991215, -0.01955215, -0.00057529664, -0.03105491, 0.0098712975, -0.007719291, -0.016974822, 0.004227835, -0.0047007687, -0.029632935, -0.042126, 0.026103392, 0.0074336263, 0.011896341, 0.004691247, -0.00027931645, -0.018079393, -0.03062324, 0.0119788665, 0.0062370095, -0.006513152, 0.03760615, -0.033289444, -0.031308834, 0.033441797, 0.004786468, 0.0053038383, 0.011896341, 0.0091349175, -0.014499063, 0.031537365, -0.0060370443, -0.013521455, -0.0033454492, 0.0019012563, 0.0010958409, 0.023462582, -0.039561365, 0.018269835, 0.00082445965, -0.030470885, -0.020313924, -0.042303745, 0.034178175, 0.009077785, 0.022167569, 0.0049959556, -0.009909386, 0.02577329, 4.9396163e-05, 0.011947126, -0.0060560885, -0.01082986, 0.012734291, -0.020720202, -0.016974822, 0.02581138, -0.008798468, -0.021367708, -0.045223873, -0.016974822, -0.022573847, -0.009280924, -0.0006728987, 0.03095334, 0.007471715, -0.023437189, 0.012429582, -0.008880993, 0.004691247, -0.026535062, 0.01109648, 0.029861467, 0.037047517, -0.022980126, -0.013318316, -0.0068876897, 0.024706809, 0.016593937, -0.00277412, 0.03516848, 0.004338927, -0.008360449, 0.013851557, 0.022535758, -0.017939733, -0.014181658, -0.009261879, -0.029658329, 0.050454706, -0.0015719485, -0.010950474, -0.02075829, -0.008360449, 0.00122439, -0.05860567, -0.01764772, 0.005643462, 0.010594981, 0.0031708765, 0.0006947203, -0.0030439144, 0.012048696, 0.00966181, -0.034711417, -0.010791772, -0.021964429, -0.010144265, -0.029455189, 0.017127177, -0.014435582, -0.009642766, -0.010302968, 0.007401886, -0.024948038, -0.025532063, -0.031689722, -0.016835164, 0.019679114, -0.008366797, 0.002394821, 0.007401886, 0.0043040127, -0.015743291, 0.016073393, -0.020148873, 0.03382268, -0.01206774, -0.01942519, 0.032984734, -0.0031867467, -0.05378111, 0.019171266, 0.0114011895, 0.018079393, -0.025646329, 0.01969181, 0.028083999, -0.010144265, 0.014702202, -0.009433278, 0.0123470565, 0.0051641804, -0.012683506, -0.026585847, 0.032883164, 0.03813939, 0.028464885, -0.015133873, 0.03727605, 0.0017980996, -0.017736595, 0.0014822816, -0.03222296, 0.060433924, 0.009217443, -0.010690202, -0.015006911, -0.03115648, -0.01420705, -0.013648418, -0.012328012, -0.004338927, 0.005662506, -0.03339101, -0.015184658, -0.017977823, -0.022853164, 0.040196177, 0.022053303, 0.0069829114, -0.029607544, -0.05301934, 0.024859164, -0.009668158, -0.050124608, -0.030927949, -0.0017600111, 0.019069696, 0.04210061, 0.015362404, -0.012962823, 0.011077437, -0.00939519, 5.5000346e-05, 0.0032787942, -0.010360101, 0.007979563, 0.02212948, -0.0018092088, 0.020885253, -0.005199095, 0.008062088, 0.002818557, 0.012118525, 0.04860106, 0.0030312182, -0.01627653, 0.019183962, -0.0005939442, -0.0086080255, -0.0038755157, -0.0106648095, 0.04202443, -0.014422886, -0.013026304, 0.02444019, 0.0062243133, 0.024287835, 0.019679114, 0.007731987, 0.0120867845, -0.021240747, -0.0021662896, -0.01848567, -0.0006982911, 0.004192921, -0.009737987, 0.0012053457, -0.0032311834, -0.019361708, -0.011426582, -0.022738898, 0.014232443, 0.0076367655, 0.022383405, 0.008544544, 0.02423705, 0.025519367, -0.012220095, -0.0056275916, 0.0041040475, 0.019869557, -0.0264081, 0.018371405, 0.037428405, 0.04839792, -0.0079224305, 0.0149815185, -0.018612633, -0.006014826, -0.008462018, 0.034686025, -0.006735335, -0.0026011344, -0.0146133285, -0.008747683, -2.2193557e-05, 0.001159322, 0.033441797, -0.005043566, -0.013661114, 0.0025344794, -0.00091253954, 0.025049606, 0.015844861, 0.013800772, -0.029455189, -0.0040469146, 0.020694809, 0.0035549367, 0.012359753, -0.02026314, -0.011578936, -0.053730328, -0.005272098, -0.008011304, -0.002440845, -0.014956126, 0.022459581, -0.008303316, -0.01614957, -0.00082445965, -0.008214443, -0.0031312008, 0.012670809, 0.006881342, 0.01731762, -0.035930254, -0.024033912, -0.04067863, -0.041415013, 0.011102829, -0.033416405, 0.03283238, -0.017800076, 0.027906252, 0.0056148954, 0.035650935, -0.020415492, -0.039891466, 0.02153276, 0.014740291], "content_hash": "8a1d48fce4902742bac6ee5ae8ae076d"}, {"content": "the **head categories**. You should see heads grouped into Previous-Token, First/Positional, Bag-of-Words, Syntactic, and Other. - Click on a category (like \"Previous-Token\") to see which specific heads belong to it. - Below the categories, you'll see the **BertViz visualization**. Try clicking on individual head squares to see their attention patterns.\n\n**Stage 4 - MLP**: Click to expand. You'll see the expand-compress pattern: 768 \u2192 3072 \u2192 768. This shows GPT-2's feed-forward network dimensions.\n\n**Stage 5 - Output**: Click to expand. You'll see:\n- Your prompt with the predicted next token highlighted\n- The confidence percentage\n- A top-5 bar chart showing the model's top predictions\n\n### Step 6: Reflect\n\nThink about what you observed:\n- How many tokens did your prompt become?\n- What was the model's top prediction? How confident was it?\n- Were there any surprising alternative predictions in the top 5?\n\n## What's Next?\n\nTry changing the prompt and running the analysis again. Compare results with different inputs:\n- A factual prompt: \"The capital of France is\"\n- A creative prompt: \"Once upon a time, there was a\"\n- A technical prompt: \"The function takes an input and\"\n\nThen move on to **Experiment: Exploring Attention Patterns** to dive deeper into what the attention heads are doing.", "source_file": "experiment_first_analysis.md", "chunk_index": 1, "embedding": [0.0076011117, 0.013393115, 0.0067180097, -0.023311028, -0.028459476, -0.017890858, 0.014465709, 0.060808893, 0.033808142, 0.04550656, 0.043304168, -0.010475662, -0.014658776, -0.042932335, -0.02884561, -0.008487789, -0.006646503, 0.025427613, 0.0051162704, 0.053029016, -0.019263776, -0.008130258, 0.009689094, 0.055574637, 0.0413878, 0.007493852, -0.026500206, 0.043990627, 0.01791946, 0.012856819, -0.0042796475, -0.019592704, -0.01917797, -0.005495253, -0.009481725, -0.004715835, -0.023039306, 0.022624569, -0.020136151, 0.0275728, -0.0016634135, -0.009782052, -0.029431961, 0.02585665, -0.026013963, 0.02126595, 0.024226308, -0.040958766, -0.02791603, 0.060294047, -0.029017225, 0.057548206, -0.01563126, 0.0321492, 0.00012535935, -0.0044977413, -0.032921463, -0.010904699, -0.002449088, -0.015688466, 0.023639956, -0.003997198, 0.0402151, 0.031977583, 0.002522382, -0.041416403, -0.06492765, 0.01803387, 0.019978838, 0.005884962, 0.054745164, 0.026457302, -0.0031069454, 1.6284424e-05, 0.008723759, -0.004272497, 0.03872777, 0.0060530016, 0.0050519146, -0.03998628, 0.029145936, -0.028216356, -0.010647276, -0.038641963, 0.0012853243, -0.045449354, -0.045334946, -0.05308622, -0.043275565, -0.07270753, -0.022166928, 0.016260516, -0.022581665, 0.034752022, 0.01355043, -0.006296123, 0.020136151, -0.014959102, -0.010318348, 0.0019127915, 0.0413878, 0.0012030922, -0.03160575, -0.008931127, 0.036496777, 0.0064427108, 0.013035584, 0.010068076, -0.03283566, -0.020736804, -0.10171045, 0.013078488, -0.010060926, 0.032235004, 0.034065563, 0.01943539, -0.026128374, -0.026156977, 0.045535162, -0.044791497, -0.007150622, -0.014186835, -0.005720498, 0.016875468, -0.0014935862, 0.0037183235, 0.026342893, -0.032692645, 0.011162122, -0.015173621, -0.01584578, 0.017890858, 0.039700255, -0.017104289, -0.004161662, -0.02840227, -0.011104916, 0.01505921, -0.0482524, 0.01793376, -0.0063497527, -0.007168499, 0.0024115473, 0.009009784, -0.05411591, -0.02345404, -0.027529895, 0.06498486, -0.043475784, 0.01942109, 0.01540244, 0.0055095544, -0.029832397, -0.027186666, -0.03389395, -0.037526466, -0.00142208, 0.025255999, -0.034523204, -0.052228145, 0.0367542, 0.0033089505, -0.0025885252, 0.005895688, -0.038813576, 0.04871004, -0.031233918, 0.0100251725, -0.030747676, -0.05849209, -0.04044392, -0.008316174, 0.025885252, -0.0505692, 0.020250563, 0.006271096, -0.07116299, -0.013722044, -0.012613698, -0.041616622, -0.022553062, -0.0025885252, -0.003664694, 0.00096622785, -0.044105038, -0.008173161, -0.00465148, 0.017576229, -0.024154803, -0.0079514915, -0.0042903735, 0.024226308, 0.085693054, 0.011584008, 0.0007432178, -0.046479046, 0.05723358, 0.002631429, 0.022967799, 0.0051699, 0.0195212, 0.04104457, 0.037755284, -0.021466168, 0.031548545, 0.017561927, -0.018691726, 0.014158232, -0.034666218, -0.029660782, 0.022624569, -0.038212925, 0.057061967, 0.040072087, -0.08746641, 0.0088810725, -0.07819921, 0.023754368, -0.001131586, 0.052571375, 0.048767243, 0.008080203, 0.044848703, 0.019249475, -0.012284769, 0.042560503, 0.056861747, -0.028330766, 0.017576229, -0.013543279, 0.024498032, 0.02424061, 0.0012620848, -0.008580747, -0.031205315, -0.035152458, 0.040615536, -0.026328592, 0.01883474, -0.047422927, 0.031777367, 0.023039306, 0.030347241, 0.0032195677, 0.046250224, 0.02677193, -0.015159319, -0.04318976, 0.050368983, 0.04736572, 0.048452616, -0.029346153, -0.042674914, 0.005720498, -0.0032874986, -0.02814485, -0.032320812, 0.027987536, -0.023811571, -0.0070862668, -0.000114018905, 0.012921174, -0.025885252, 0.0043261265, -0.01631772, -0.030919291, -0.046135817, -0.03469482, -0.05354386, -0.061666965, -0.054659355, -0.025842348, 0.046707865, 0.025127286, -0.052914605, -0.0023114386, 0.002034352, 0.0016669888, -0.018777534, 0.009867859, -0.022023916, -0.014830391, 0.025356106, 0.028802706, -0.014873294, 0.008802416, -0.0006288079, -0.009159947, -0.0075510573, -0.009488876, 0.036382366, -0.05094103, 0.035953328, -0.020064646, -0.010904699, -0.0020951324, 0.031062303, 0.025699336, 0.007915739, 0.018448606, 0.02494137, -0.05654712, 0.013707743, 0.0061745625, 0.010761687, 0.029489167, 0.034094166, 0.011591159, 0.005505979, -0.005541732, -0.024212006, -0.024154803, 0.030375844, -0.019206572, -0.020407876, -0.021880904, 0.027630005, 0.023911681, 0.016274815, -0.012284769, -0.038184322, -0.016904071, 0.00040445707, 0.034666218, 0.010010871, -0.017862255, -0.023482643, 0.020751106, 0.0046193018, -0.028202055, 0.038384542, -0.0026457303, -0.028688297, 0.047651745, -0.021551976, -0.009059838, -0.005330789, 0.03632516, 0.0024794783, 0.061838582, -0.08128827, 0.01586008, -0.015831478, 0.0019378186, -0.03652538, 0.07276473, 0.00362179, -0.0011280107, 0.0027369007, -0.007872835, -0.042474695, 0.011162122, 0.026013963, -0.02402609, 0.007822781, 0.04570678, -0.011905786, -0.009782052, 0.0076797684, 0.0063390266, -0.018176882, 0.009467424, 0.046135817, 0.013629086, 0.017075686, -0.005927866, -0.032435223, -0.01297838, 0.0027154488, -0.022481557, 0.0008330475, -0.006213891, 0.02009325, -0.028631091, -0.056232493, -0.015588356, -0.017604832, -0.0061280834, -0.026328592, 0.08443455, -0.007132746, -0.023396837, -0.020937022, -0.018977752, 0.022710375, 0.027715812, 0.033607926, -0.014715981, -0.05803445, 0.0149448, 0.0004924991, -0.035753112, -0.023854475, 0.003493079, -0.019578403, 0.04676507, -0.017190095, 0.04702249, 0.04230308, 0.071506225, 0.017962363, -0.016603746, -0.047880568, -0.00055149174, 0.021766495, 0.014587269, 0.007479551, 0.016003093, 0.023411138, -0.024412224, 0.049253486, -0.03735485, 0.003083706, -0.01447286, -0.009281508, 0.044333857, 0.051513083, 0.00523068, 0.02378297, 0.039671652, -0.016017394, -0.051084045, -0.020722503, -0.0015373838, -0.056261096, -0.0037183235, 0.05505979, -0.03503805, 0.037383452, -0.020979926, 0.0018135766, 0.056718737, 0.012556492, -0.0097391475, 0.017976664, -0.036468174, 0.035695907, 0.066243365, 0.026114073, -0.033035874, 0.017261602, -0.029174538, -0.019220872, -0.02652881, -0.048424013, 0.011390941, -0.03134833, -0.07791318, 0.027401185, -0.04630743, -0.01563126, 0.029045828, -0.02218123, 0.047994975, -0.026857737, -0.031920377, 0.06584293, -0.014036671, 0.07768436, -0.0050054356, 0.0012996256, 0.017490422, -0.010218239, 0.029832397, 0.0036861459, 0.032320812, -0.007207827, -0.026042566, 0.041816838, -0.0020146878, 0.012306221, 0.0005943955, 0.002091557, -0.03652538, 0.003932842, -0.022567363, -0.01183428, 0.015459646, -0.015116416, -0.055689048, 0.021537675, 0.0065428196, 0.027944632, 0.044991717, -0.0036789952, 0.07739834, -0.03941423, -0.0075081536, 0.027429787, 0.007901438, 0.008423433, -0.015330934, 0.016189009, -0.01746182, 0.033493515, -0.016618047, -0.0076225633, -0.034637615, -0.0033482788, 0.0060637277, 0.02045078, -0.022095423, -0.019206572, 0.054058705, 0.033293296, 0.011176422, 0.05205653, 0.03538128, -0.0032946493, 0.044391062, -0.07139181, -0.008952579, -0.021880904, 0.0144585585, -0.019764319, 0.01654654, 0.016832564, 0.023482643, 0.014279793, 0.020522285, -0.002361493, 0.0070469384, -0.013657688, -0.0014631961, -0.013929412, -0.013736346, -0.021780796, -0.04184544, -0.03415137, -0.034437396, 0.01896345, -0.036496777, 0.0058241817, 0.013857906, -0.02954637, 0.007400894, -0.007894287, -0.009896461, -0.04147361, 0.020393575, 0.0036360915, 0.012892572, 0.01229907, 0.019006355, -0.010196787, 0.009496027, 0.022824787, -0.0073722918, 0.016532239, 0.02734398, 0.007994396, -0.023225222, 0.010547168, -0.014687378, -0.0035878248, -0.010182486, -0.04098737, -0.034980845, 0.021694988, -0.029088732, 0.027415486, -0.01999314, 0.021551976, -0.025255999, -0.0052092285, 0.036153547, 0.029632179, -0.007400894, -0.0074294964, -0.024712551, 0.007629714, -0.0039042397, 0.027658608, -0.001889552, 0.024312116, -0.025227396, -0.01987873, 0.017719243, -0.013922262, 0.015245127, -0.022996401, -0.026700424, -0.01125508, 0.021108637, -0.006253219, -0.0060351253, 0.017032782, -0.004218867, -0.02136606, -0.015874382, 0.02311081, -0.027286775, -0.002331103, 0.034179974, -0.02170929, -0.0097391475, 0.0128139155, -0.01896345, -0.04908187, 0.008780964, 0.017504724, 0.005920715, -0.032235004, 0.0042438945, -0.0333219, 0.02459814, -0.029246045, -0.013149994, 0.02827356, 0.020650998, -0.022881992, -0.024626743, -0.03389395, -0.023153715, -0.029231744, 0.0018770384, -0.031005098, -0.002974659, 0.013200048, -0.0018430729, 0.024397923, 0.0038720618, 0.0030265008, 0.01791946, 0.008294722, 0.011619762, 0.022881992, -0.015960189, -0.0007012079, 0.009374466, -0.02585665, 0.012163209, 0.03297867, 0.007987245, -0.026457302, -0.006821694, 0.0016169344, 0.026013963, 0.016861167, -0.0028888513, 0.021094335, 0.012306221, -0.03492364, -0.019192271, 0.0018824013, -0.009567533, 0.0033232516, 0.008788114, -0.028001837, -0.0195212, -0.059207153, -0.035009447, 0.05045479, -0.006574997, 0.000981423, -0.011104916, -0.0100251725, -0.002901365, -0.016689552, 0.028530983, -0.019120764, 0.0019270927, -0.007143472, -0.05305762, -0.020164754, 0.03046165, 0.010947603, 0.02218123, 0.01907786, -0.0067537627, -0.081345476, 0.011734171, 0.025842348, 0.036239352, 0.013085639, -0.027758716, -0.0070397877, -0.039128207, -0.009960817, 0.03964305, -0.002445513, 0.014129629, -0.07402324, -0.006914652, 0.012842517, -0.0033625802, 0.011905786, -0.024312116, 0.013414567, 0.006600024, -0.0044047832, -0.035695907, 0.0021559126, -0.005995797, -0.007787028, -0.0036450296, -0.0030139873, 0.054316126, -0.0048767244, -0.054401934, -0.01114067, -0.0062925476, -0.01698988, -0.029689383, 0.01080459, -0.031119509, -0.010397005, 0.050511997, 0.022438653, 0.002170214, -0.046107214, -0.011612611, -0.011662665, 0.009274357, -0.022395749, 0.032778453, -0.014208286, -0.011126368, 0.012949777, -0.035896122, -0.024154803, 0.013300157, 0.002652881, 0.010232541, 0.021165842, -0.02253876, -0.016575143, -0.018706027, -0.001489117, 0.02793033, 0.028101945, 0.03492364, 0.0063819303, -0.056518517, -0.0064784638, 0.023196619, -0.026500206, 0.01194869, -0.025327504, -0.039357025, 0.020822613, -0.002937118, -0.006582148, 0.01114067, 0.034780625, -0.0071291705, -0.020307768, 0.035552893, -0.02309651, -0.044190846, -0.021294553, 0.014079575, -0.014858993, -0.016289117, 0.011855732, -0.016031696, 0.014558667, -0.020779708, 0.008115957, 0.033121683, 0.012456384, 0.002080831, -0.02010755, 0.013757797, 6.659017e-05, -0.03572451, 0.02619988, 0.013571881, 0.009445972, 0.008194613, 0.016475033, -0.016875468, 0.034609012, -0.017776446, 0.063955165, -0.028101945, 0.03918541, -0.0022971374, -0.012821066, 0.020236261, -0.0095961355, 0.00862365, -0.031920377, -0.04307535, 0.011362338, -0.009066989, -0.016346322, -0.021394663, -0.013593333, -0.020393575, -0.018577317, -0.040872958, 0.0043976326, -0.04307535, -0.0019682087, -0.010654427, -0.008602198, -0.10811741, -0.011912936, 0.016946975, -0.03735485, -0.024140501, -0.013836454, 0.0321778, -0.010218239, -0.012999832, 0.0057276483, 0.018577317, -0.017447518, -0.00269936, -0.022424351, 0.0028566737, -0.044190846, 0.0060637277, -0.008258969, 0.001224544, -0.040815752, -0.019263776, 0.010497114, -0.0018055321, 0.023025004, -0.03735485, 0.0055381567, -0.046021406, -0.007965793, 0.02804474, 0.015245127, -0.0356387, 0.02378297, -0.0034215727, 0.016231913, 0.0117413215, 0.013285856, -0.025627831, 0.007987245, 0.017418915, -0.012899723, 0.010053775, -0.015216524, 0.0100251725, -0.035238266, 0.04070134, 0.0037612272, -0.019249475, -0.011626911, 0.011090615, 0.033264697, -0.0070612393, 0.0072757583, 0.011498201, 0.0012844305, 0.04113038, 0.014286944, -0.008502089, -0.021509072, 0.026271386, 0.044162244, -0.005309337, 0.01698988, 0.040586933, -0.015316633, -0.009567533, 0.0047051096, 0.007214978, 0.030204229, -0.042131465, -0.01056862, -0.027086558, 0.0029263922, -0.0006180819, -0.007844233, -0.0043976326, -0.009038387, 0.009045538, 0.0033661555, 0.012577944, -0.012785313, 0.058635104, -0.025527721, -0.018620221, 0.021080034, 0.020994227, 0.025441915, 0.02022196, -0.038127117, 0.019792922, 0.033064477, -0.00054881023, -0.018219786, -0.016646648, 0.016946975, -0.020708201, -0.005920715, 0.018934848, 0.03046165, 0.038927987, 0.0014631961, 0.03658258, 0.042102862, -0.0038112816, -0.038184322, -0.053515255, 0.020379273, 0.0004924991, 0.006685832, -0.006149535, -0.011634062, 0.024040392, 0.0058670854, 0.029231744, 0.03363653, -0.00025876315, 0.04953951, 0.05137007, -0.006835995, 0.0006766276, -0.034895036, 0.010032323, 0.022896292, -0.036039136, 0.00591714, -0.025627831, 0.008502089, -0.03011842, 0.006986158, -0.0052342555, -0.017561927, -0.001411354, 0.020851215, -0.022009615, -0.005766977, -0.011941539, -0.0044762897, -0.007286484, 0.01769064, -0.012263318, 0.016060298, -0.0039757458, 0.008030149, 0.031090906, 0.015373838, -0.012914023, 0.005212804, -0.0080945045, -0.017047083, 0.0014033096, 0.025913855, 0.02837367, 0.0043797563, 0.010976206, -0.0057169227, 0.00500186, 0.034551807, 0.025842348, -0.0048767244, -0.011412393, 0.02804474, 0.0019127915, 0.042960938, 0.042503297, -0.020493684, 0.0035270446, -0.018462908, -0.022839088, 0.005352241, 0.0023471918, 0.046707865, -0.0540015, 0.028745502, 0.04113038, 0.01631772, 0.022224134, 0.043933425, 0.010389854, 0.02022196, 0.024054693, 0.02665752, -0.012127455, 0.032006186, -0.020364972, 0.0010377341, 0.020951323, -0.0053343643, -0.009124194, 0.021666385, -0.007218553, -0.021051431, -0.009074139, 0.027901728, -0.006278246, 0.047994975, -0.026714725, -0.007779877, 0.017619133, -0.003108733, -0.018877642, -0.005277159, -0.027057955, 0.030032614, 0.00873091, -0.013514676, 0.0027333254, 0.021008529, -0.01034695, -0.01780505, 0.0062460685, 0.007011185, -0.019263776, 0.010940452, -0.030347241, 0.007400894, -0.033150285, -8.3517036e-05, 0.02045078, 0.008702307, 0.023010703, -0.0063962317, -0.04848122, -0.022839088, -0.030719073, 0.021852301, -0.0053379396, -0.009181399, 0.038756374, -0.014630173, 0.00943167, 0.00442266, -0.01079744, -0.007168499, -0.009438821, -0.0044441116, -0.02712946, -0.0014086725, 6.340591e-05, -0.0066035995, 0.005101969, -0.0014524702, 0.038241528, 0.022038218, 0.012484986, 0.005523856, 0.017776446, -0.020636696, 0.0147445835, -0.0029889601, -0.006203165, -0.018920546, -0.014558667, 0.011205025, 0.019063558, -0.016031696, 0.0020540163, 2.360264e-05, 0.039471436, 0.016975578, -0.00036155334, 0.013807852, -0.006414108, 0.027444089, 0.0011986231, -0.0574338, 0.016975578, -0.0043833316, -0.020121852, -0.013049886, 0.023225222, -0.0076225633, -0.0014766034, 0.037269045, -0.02092272, 0.033836745, 0.0031623626, 0.00746525, 0.007665467, -0.015559754, 0.019592704, -0.006871748, 0.0128854215, 0.0037719533, 0.008216065, -0.011362338, 0.0128854215, 0.023468342, -0.01884904, -0.02010755, 0.006410533, -0.027158063, 0.025413312, -0.013192898, 0.0015579418, 0.011469598, -0.001941394, -0.006861022, 0.0051412974, -0.0028977897, 0.008952579, -0.06006523, 0.036239352, 0.007665467, 0.021795096, -0.02193811, -0.00586351, 0.006986158, 0.03606774, -0.048281003, 0.012141757, -0.004926779, -0.014158232, 0.010840343, -0.005505979, 0.022267038, -0.01171272, -0.010904699, 0.015445344, -0.03572451, 0.0044298107, -0.013922262, -0.02780162, 0.0091670975, -0.02884561, 0.023010703, 0.0033822444, -0.027944632, 0.008344776, -0.011676966, 0.01698988, -0.003092644, 0.010783139, -0.0004889238, -0.010325499, 0.011662665, -0.038584758, 0.0028012563, -0.028859911, -0.006800242, 0.006499916, -0.006378355, 0.0026385796, 0.033836745, 0.014916198, -0.011584008, 0.015802875, -0.0041545117, 0.018234087, -0.007593961, -0.003925692, 0.00884532, 0.069504045, -0.030661868, -0.026156977, -0.009867859, 0.00873091, 0.020522285, -0.005320063, 0.0072686076, -0.0413878, -0.0036611187, -0.018133977, 0.0029442688, -0.019707115, -0.0055202805, 0.023711463, -0.014494311, -0.007829932, -0.012070251, 0.06424119, -0.0026439426, 0.020722503, 0.04032951, -0.019764319, -0.0025348957, 0.006178138, 0.04619302, 0.016475033, 0.011340887, 0.00037965336, 0.0071005677, 0.0002828965, -0.03069047, -0.006921802, 0.014758884, -0.021308854, 0.0051198457, 0.007851384, 0.043504387, -0.0066035995, 0.014758884, -0.012291919, 0.00815886, 0.03403696, -0.021966713, 0.013335911, 0.023411138, 0.0051520234, 0.015116416, -0.017704941, -0.025556324, -0.026257085, 0.025441915, 0.00054121274, -0.012234715, 0.029174538, -0.04344718, -0.019063558, 0.009338712, 0.004065129, 0.0069933087, -0.023768669, 0.02918884, -0.016117502, 0.013772098, -0.0013532552, -0.056918953, -0.019792922, -0.006596449, -0.009989419, 0.026829135, -0.028244957, -0.024140501, -0.0061423844, -0.01666095, 0.02469825, -0.0033089505, 0.026986448, -0.0028298588, 0.030318638, 0.013714894, -0.011548255, -0.010618674, -0.04173103, -0.0161032, -0.006986158, 0.0038899386, 0.039128207, -0.023196619, -0.021537675, 0.017290205, -0.021737892, 0.022910593, 0.013185748, 0.04756594, -0.0029156662, -0.0041509364, 0.003979321, -0.015416741, -0.0041223336, -0.036639787, -0.013521827, -0.017290205, -0.009674792, 0.009345863, -0.01068303, -0.03526687, -0.058349077, 0.0073865927, 0.006835995, 0.001582969, 0.01447286, -0.006507066, -0.002896002, -0.017776446, 0.007722672, -0.0034376616, -0.020851215, 0.05620389, -0.049139075, -0.033951156, 0.029489167, -0.010146733, -0.003925692, -0.0033822444, 0.012284769, 0.009567533, 0.018977752, -0.006768064, -0.03712603, -0.008802416, -0.016332021, 0.029775191, 0.018091075, -0.0039828965, 0.0004947337, 0.0097391475, -0.0010967267, -0.025756542, -0.043990627, 0.041015968, 0.0037433507, 0.025370408, -0.010182486, -0.02608547, 0.016003093, -0.0073579904, 0.019907333, 0.013621936, 0.004218867, 0.014043822, -0.028445175, -0.030604664, 0.02517019, 0.018434305, 0.0041795387, -0.035238266, -0.000581435, -0.0068252687, -0.00471226, -0.018319894, 0.030604664, -0.016932674, -0.021652084, -0.004898176, 0.012020196, 0.023625655, -0.019764319, -0.0013675565, 0.021809397, 0.01229907, -0.00024111004, -0.037612274, -0.027973235, 0.02701505, 0.009496027, -0.011047712, -0.0002609977, 0.045535162, -0.02355415, 0.017490422, 0.0033161012, -0.002407972, -0.026142675, -0.03249243, -0.0039471434, 0.020879816, -0.003544921, -0.016403528, -0.019392489, 0.0029978985, 0.00045652254, -0.039786063, -0.0128854215, 0.008230366, 0.023225222, 0.059950817, -0.018133977, 0.028445175, 0.03134833, -0.017833652, -0.014565818, -0.019792922, -0.027386883, -0.019621307, -0.0275728, 0.037040222, -0.034780625, -0.016045997, 0.009231454, 0.0059922216, -0.028530983, -0.009102742, -0.037240442, -0.010697331, 0.0153881395, -0.007915739, 0.013142844, 0.007980094, -0.004354729, -0.027200967, 0.016460732, -0.018520111, 0.042817928, 0.0027065105, -0.0137864, 0.011197874, -0.0009483513, -0.03929982, 0.008144558, -0.0018019568, 0.030404447, -0.037755284, -0.017304506, 0.023482643, 0.015473947, 0.010654427, -0.02619988, 0.015817177, 0.013293006, 0.015574056, -0.0052807345, 0.045763984, -0.01643213, 0.01884904, -0.01781935, 0.03887078, -0.0108117405, 0.01781935, -0.023039306, -0.011047712, 0.028859911, 0.012935475, -0.03203479, -0.027444089, -0.00918855, -0.040815752, 0.0033125258, 0.0138150025, -0.014773185, 0.0064212587, -0.060179636, -0.014830391, 4.0306048e-05, -0.0003237891, 0.015002006, 0.029632179, 0.0047515887, -0.022710375, -0.04150221, 0.011348038, -0.020608094, 0.0038470349, -0.023339631, -0.010754536, 0.001152144, 0.051799107, -0.001370238, 0.00059082016, 0.004358304, -0.020465082, 0.010268293, -0.031891774, -0.0063497527, 0.002854886, -0.019950235, 0.008630801, 0.0007432178, 0.0107402345, 0.031233918, -0.014673077, 0.029775191, 0.028259259, -0.015931586, -0.021323156, -0.00050054357, 0.0013916899, -0.0034483876, -0.024512334, -0.025685035, 0.019149367, -0.041988455, -0.011469598, 0.012992681, -0.0046479045, 0.035152458, 0.011047712, 0.009510328, 0.020879816, -0.019893032, -0.012713807, 0.020879816, -0.008173161, 0.012463534, -0.008180312, 0.0080516, -0.0141725335, -0.008659404, -0.014201135, -0.016632348, 0.01631772, -0.006410533, 0.016789662, 0.026485905, 0.017533327, 0.022810485, -0.008530692, -0.005266433, 0.0036003382, 0.030433048, -0.024512334, 0.008645102, 0.040529728, 0.034666218, 0.010411306, 0.011455297, -0.0016991666, -0.02010755, 0.017747844, 0.03229221, -0.00057070906, -0.022023916, -0.015273729, -0.010933301, -0.032320812, -0.0084520355, 0.039271217, -0.022424351, -0.033121683, -0.010304047, 0.014844691, 0.0011852157, 0.019278077, -0.028631091, -0.019978838, -0.0020218384, 0.016231913, 0.018148279, 0.033407707, -0.024498032, -0.030747676, -0.060522866, -0.02585665, -0.009639039, 0.0067072837, -0.0206939, 0.019363886, -0.046793673, -0.044162244, -0.028874213, -0.011848581, -0.017676339, 0.0063283006, 0.030261433, 0.0008688006, -0.02894572, -0.028759804, -0.040872958, -0.0402151, 0.021037132, -0.030232832, 0.028645393, -0.021280251, 0.020364972, -0.006306849, 0.013085639, -0.010139583, -0.008716608, 0.016060298, -0.023539849], "content_hash": "c2532e2b268dc04c99cd84078753518c"}, {"content": "# Experiment: Understanding Token Attribution\n\n## Goal\n\nLearn how to use token attribution to identify which parts of your input most influenced the model's prediction. Compare two attribution methods and see how results change with different target tokens.\n\n## Prerequisites\n\n- Complete \"Your First Analysis\"\n\n## Steps\n\n### Step 1: Run an Analysis with a Meaningful Prompt\n\n1. Select **GPT-2 (124M)** and enter the prompt: `The capital of France is`\n2. Click **Analyze**.\n3. Check Stage 5 -- the model should predict something like \"Paris\" or \"the\" with high confidence. Note the top prediction.\n\n### Step 2: Open the Attribution Panel\n\n1. Scroll down to the **Investigation Panel**.\n2. Click the **\"Token Attribution\"** tab.\n\n### Step 3: Run Simple Gradient Attribution\n\n1. Select **\"Simple Gradient (faster, less accurate)\"** as the attribution method.\n2. Leave the **Target Token** dropdown empty (this defaults to the top prediction).\n3. Click **\"Compute Attribution\"**.\n\n### Step 4: Read the Results\n\nLook at the two visualizations:\n\n**Color-coded tokens**: Your input tokens are displayed as colored boxes. Darker blue means higher influence.\n- You should see that **\"France\"** has a very dark color -- it's the most relevant token for predicting \"Paris.\"\n- **\"capital\"** likely also has a notable color -- it sets up the context for a city name.\n- Function words like **\"The\"**, **\"of\"**, and **\"is\"** should be lighter -- they contribute less to this specific prediction.\n\n**Bar chart**: Shows the same information as horizontal bars with scores. Longer bars = more influence.\n\n**Hover over any token chip** to see the exact attribution score.\n\n### Step 5: Compare with Integrated Gradients", "source_file": "experiment_token_attribution.md", "chunk_index": 0, "embedding": [-0.013251405, -0.00042796275, -0.017115343, -0.035826936, 0.017482733, -0.016367894, -0.0051561394, 0.034230687, 0.023867732, 0.029923346, 0.021802742, -0.00716412, -0.025970727, -0.059694666, -0.02660416, -0.00016756266, -0.009780196, 0.014670294, 0.007689869, 0.07003228, -0.04466965, -0.010476972, 0.03972888, 0.041249115, 0.032279715, -0.043808185, -0.021397347, -0.006156963, -0.013846832, 0.025413306, 0.020092474, -0.010717676, -0.046975344, -0.035548225, -0.014188886, -0.030379416, -0.038183305, -0.007189458, 0.015506425, 0.00067579316, -0.0063311565, -0.0056597185, -0.038411338, 0.02847912, -0.026908208, 0.01565845, 0.0064863474, -0.02820041, -0.031266224, 0.05396844, -0.028783167, 0.034458723, -0.015709123, -0.00022170134, -0.045303084, 0.030404754, -0.015493756, 0.005381008, -0.03466142, -0.03785392, -0.03792993, -0.027769675, 0.06623169, 0.009944889, 0.03418001, -0.02979666, -0.039272808, 0.00052772835, 0.0070184306, -0.0032748454, 0.041705187, 0.016139857, 0.022398168, -0.0031497425, -0.04720338, -0.024513833, -0.023639696, 0.05199213, -0.026021402, -0.039576855, 0.011515801, -0.016621266, -0.01072401, 0.0018417047, 0.008836382, -0.02192943, -0.028327096, -0.034813445, -0.03501614, -0.06420471, 0.0033381886, 0.00494394, -0.025248615, 0.014771643, 0.041730527, 0.0005415847, -0.023500342, -0.027237592, 0.013276743, 0.0047475756, 0.03830999, -0.01948438, 0.014568944, -0.0064040013, 0.015519094, -0.021954766, 0.04104642, -0.0032653438, -0.020092474, -0.023829727, -0.029644636, 0.050370544, -0.034306698, 0.028403109, -0.0036992452, -0.026021402, -0.05619812, 0.02885918, 0.052144155, -0.065471575, -0.025413306, -0.037803244, -0.006473679, 0.020168487, 0.015810473, -0.01255463, 0.016633935, -0.028555132, -0.00011956036, -0.027009556, -0.027085569, 0.01807816, -0.0130487075, -0.006575028, 0.002346867, -0.036789753, 0.0053430023, 0.034990806, -0.06466078, 0.019154996, -0.0118388515, -0.04983846, -0.0065560252, 0.009495151, -0.066941135, -0.017774113, 0.00590359, 0.049990483, -0.045911178, 0.03483878, 0.01166149, -0.00946348, -0.005596375, -0.030278068, -0.042389296, -0.04563247, 0.026908208, 0.017875463, 0.006001772, -0.050699927, 0.0055837063, 0.045911178, -0.0002694067, -0.017672764, -0.043225426, 0.025869379, -0.037524533, 0.004263, -0.025552662, -0.053816415, -0.025780698, -0.032862473, 0.025818704, -0.047760803, 0.03278646, -0.021118635, -0.07423828, -0.028453782, 0.0039589526, -0.03955152, -0.021992773, 0.011028058, -2.9246761e-05, -0.014011525, -0.022309488, -0.031443585, -0.007189458, 0.0064610103, 0.02885918, 0.0014188886, 0.046164554, 0.014885661, 0.060302764, 0.0072528007, -0.057769034, -0.024336472, 0.05807308, 0.02042186, 0.02660416, 0.002448216, 0.009970225, 0.0053303335, 0.030430092, -0.026224101, 0.0021410014, 0.0063659954, -0.006701715, -0.027795013, -0.06319121, -0.020928606, 0.01733071, -0.055336654, 0.022702217, -0.01349211, -0.078241564, -0.011401783, -0.0432761, 0.016380562, 0.0678026, 0.049863797, 0.017634757, 0.021219986, 0.022980927, -0.022334825, -0.032558426, 0.02137201, 0.04299739, -0.041654512, 0.027009556, -0.04241463, 0.023766384, 0.040489, -0.011541138, -0.031874318, 0.015430413, -0.06709316, 0.027617652, -0.023462335, 0.027136244, -0.033521242, 0.015139034, 0.022626204, 0.037803244, -0.012459615, 0.012586301, -0.0074681677, -0.019205669, -0.043428123, 0.025375301, 0.022398168, 0.028833842, -0.0068157325, -0.043428123, 0.029137889, 0.004456197, 0.011097736, 0.005944763, 0.010736679, 0.011541138, -0.027566977, 0.014657625, -0.0149870105, -0.033571918, -0.017381385, -0.012491287, -0.01404953, -0.047507428, 0.021308666, -0.046620622, -0.022968259, 0.0016176279, -0.01872426, 0.019750422, 0.00070231815, -0.032735787, -0.010622661, -0.033115845, 0.013479441, -0.025996065, -0.008076263, -0.01349211, -0.0030103873, 0.05640082, 0.028225748, -0.055488676, 0.0048235874, 0.030810151, 0.037245825, -0.0023246969, -0.009425473, 0.0066637085, -0.04955975, 0.034889456, -0.029644636, -0.017254699, 0.023842394, 0.017216692, 0.000498828, -0.025907384, 0.002199594, 0.011984541, -0.032457076, 0.008367642, -0.015417744, 0.028428445, 0.0062488103, 0.0099005485, 0.03501614, 0.033622593, 0.005422181, -0.04355481, -0.02575536, 0.0043136748, 0.021308666, -0.023855064, 0.06455943, 0.040894393, -0.010128584, 0.014214222, 6.987551e-05, 0.0039811227, -0.05731296, 0.022360163, 0.01750807, 0.00038936298, 0.0040729702, -0.02802305, -0.0041996567, 0.00569139, 0.015240383, -0.008792042, 0.013732814, 0.004107809, 0.038740724, -0.062228397, 0.00547919, -0.015164372, 0.033039834, 0.038259316, 0.047532767, -0.06440741, -0.018749598, 0.023411661, -0.022056116, -0.0050421217, -0.0019715582, 0.019357694, 0.017862793, 0.015886484, -0.031342234, -0.01930702, 0.0040983073, -0.010787354, -0.024032425, 0.019915113, 0.032634437, -0.009710519, -0.007056437, 0.028631143, -0.013948181, -0.02754164, 0.032431737, 0.04542977, -0.0009746941, -0.0036929108, -0.01423956, 0.000566922, 0.018344201, 0.0020079806, -0.018204847, -0.025045916, 0.023804389, -0.026781522, -0.023994418, -0.029492611, 0.008697026, 0.0025004742, 0.009919551, -0.037803244, 0.008272627, 0.027313605, -0.011439789, 0.012769997, -0.03017672, 0.04710203, 0.005026286, 0.02192943, -0.011667824, -0.026097415, 0.025616005, -0.024982573, -0.038005944, -0.01724203, 0.061569627, -0.030556777, 0.04373217, -0.0033666932, 0.03792993, 0.055133954, 0.055995423, -0.047076695, -0.015493756, -0.049129017, -0.025185272, 0.04292138, 0.066637084, 0.0040349644, 0.021029955, 0.02632545, 0.04879963, 0.036587056, -0.03744852, 0.043985546, -0.044897687, 0.012966361, 0.041097093, 0.037701894, 0.00735415, 0.035826936, 0.018964965, 0.026072077, -0.03775257, 0.008754035, -0.059339944, -0.02931525, 0.0022090955, 0.027795013, -0.013834164, 0.009913216, -0.04390953, 0.019826433, 0.009406471, 0.067245185, -0.019421037, -0.010426297, -0.0048425905, 0.05181477, 0.036587056, 0.0068410696, -0.016253876, 0.020485204, -0.03296382, -0.030683465, -0.047836814, -0.049230363, -0.014949004, -0.027972374, -0.06496482, 0.023956412, 0.017292704, -0.026300112, 0.014454927, 0.004810919, 0.034332037, -0.016013172, -0.025996065, 0.013137388, 0.033394556, 0.07788684, -0.019598398, 0.04451763, -0.002943877, -0.018597575, 0.020130482, 0.030100707, -0.0021188313, -0.001859124, -0.027693665, 0.015886484, -0.022613537, 0.018876284, -0.0057800706, -0.021600043, -0.029568624, -0.0036644062, -0.023589022, 0.022512186, 0.008836382, -0.006432506, -0.027313605, -0.0042566657, 0.029568624, 0.00993222, 0.007816556, -0.043301437, 0.0626338, 0.00018745639, 0.010217264, -0.015899153, -0.013251405, -0.00065085175, -0.053461693, 0.01657059, -0.013378092, 0.0027364278, -0.029391263, 0.020041801, -0.021511363, 0.014302903, -0.00066074915, 0.030506104, -0.055691376, -0.040260963, 0.075099744, 0.00946348, 0.005833912, 0.05189078, 0.014530939, -0.030835489, 0.0556407, -0.049889136, -0.032609098, -0.018280858, -0.011889526, -0.020510541, 0.021093298, -0.013846832, 0.047456753, 0.032761123, 0.021714061, 0.0057103927, 0.026072077, -0.015379738, -0.013618796, 0.0037594212, -0.009260781, -0.014796981, -0.03007537, -0.038157966, -0.02979666, 0.025185272, -0.022258814, 0.0014861907, 0.017394053, -0.041730527, -0.011826183, 0.016089182, -0.012801669, -0.058985222, 0.012446946, 0.035750926, -0.01825552, 0.0026255772, 0.011826183, 0.00843732, -0.011636153, 0.00018993074, -0.028403109, -0.005168808, 0.019978458, 0.01053398, -0.0402863, 0.02512193, -0.018154172, -0.009286119, -0.023500342, 0.009710519, -0.040412985, 0.003042059, -0.05219483, 0.0215367, 0.0025764862, 0.018242853, -0.020662565, -0.009938554, 0.045404434, 0.054829907, -0.014796981, 0.027896361, -9.10559e-05, 0.025907384, 0.05064925, 0.018610243, 0.006727052, 0.045303084, 0.005444351, -0.019345025, -0.005634381, 0.019623736, 0.009469814, -0.017571414, 0.00035947288, -0.004620889, 0.0059954375, -0.022461513, -0.03810729, 0.010850697, -0.001760942, -0.013390761, -0.008095266, 0.014467596, -0.011186416, 0.003268511, 0.017672764, -0.023183625, 0.021942098, 0.024374478, -0.012947358, -0.017976811, 0.009431808, 0.032634437, -0.027060231, -0.03334388, -0.007024765, -0.004183821, 0.04064102, -0.018344201, 0.025324626, -0.0072274636, 0.0050452887, -0.013099382, -0.047634114, -0.014138211, 0.012630641, -0.01377082, 0.014594282, -0.01873693, 0.01157281, 0.005358838, -0.008133272, 0.017280035, -0.013036039, 0.007335147, -0.022220807, 0.015303726, 0.045835167, -0.00086780236, 0.011585479, 0.0494584, 0.012636976, -0.02931525, 0.03298916, 0.033470567, 0.0067777266, 0.021219986, 0.0006073033, 0.004076137, 0.020434529, -0.011287765, -0.0064483415, 0.019750422, 0.008880722, -0.001039621, -0.021004617, -0.0021584209, 0.0021774238, 0.022980927, 0.04580983, -0.019421037, -0.024615182, -0.04071703, -0.017596751, 0.04740608, 0.0010934627, 0.0048615937, -0.010103246, 0.03643503, 0.0006488723, -0.028783167, -0.010888702, -0.04403622, -0.011129407, -0.046671297, -0.0074048247, -0.030784814, 0.006644706, -0.008912394, 0.032178365, 0.0264268, -0.008120603, -0.10479505, 0.010185593, -0.008887056, 0.022258814, -0.0037024124, -0.00620447, 0.0019493882, -0.04373217, -0.011091401, 0.021435352, 0.010236267, 0.0059796018, -0.04104642, -0.012763662, -0.0057579004, -0.02360169, 0.015797803, 0.0051751425, 0.013618796, -0.04515106, -0.008912394, -0.027693665, -0.0019588897, -0.005108632, -0.02529929, 0.014404252, 0.026629498, 0.04991447, -0.00035828518, -0.08234621, 0.0032415902, -0.017128011, -0.02612275, -0.013745483, -0.006746055, 0.01152847, -0.020193825, 0.024488496, 0.026401462, -0.05553935, -0.038436677, -0.0045892177, 0.030556777, 0.011186416, -0.022714885, 0.022372833, -0.0013808826, 0.016532585, -0.008887056, -0.0588332, -0.0415785, 0.011097736, -0.04525241, 0.029619299, 0.02351301, -0.0149870105, -0.026528148, 0.01873693, -0.008031922, 0.031266224, 0.0007917905, -0.010895037, -0.015721792, -0.03734717, 0.012820671, 0.01166149, -0.02014315, 0.02594539, -0.009064417, -0.005225817, 0.025324626, 0.011325771, -0.006752389, -0.030582115, 0.033951975, 0.02491923, -0.032178365, 0.026629498, -0.0164059, -0.048748955, -0.0043675164, 0.009856208, 0.022081453, -0.025147265, 0.01611452, 0.012231579, 0.014695631, -0.046417926, -0.023804389, 0.01053398, 0.0044118566, 0.034256022, -0.014036861, 0.012104893, -0.024944568, -0.019889778, 0.049787786, 0.022980927, -0.021866085, 0.0479635, -0.013074044, -0.0037214153, 0.03456007, -0.01816684, 0.044390943, -0.014036861, 0.035066817, -0.0015400324, -0.021600043, 0.005168808, 0.02275289, 0.0047222385, -0.03174763, -0.0050009484, 0.020561215, 0.0015384489, 0.009868876, -0.022917584, -0.013226069, -0.011015389, -0.017850125, -0.017634757, -0.00015044019, -0.005833912, -0.008526, 0.028783167, -0.008868054, -0.08457589, -0.018534232, 0.044289593, -0.014226891, -0.009400137, 0.0020000627, 0.028453782, 0.0068284012, -0.043960206, -0.009754859, -0.0070691053, -0.023551015, -0.0053493367, -0.018749598, -0.016621266, 0.0015344899, -0.012858678, -0.0061284583, -0.0028377771, -0.021511363, -0.014036861, 0.0014030527, 0.017837455, -0.016063845, -0.043301437, -0.006689046, -0.030987512, 0.0070057623, 0.021625381, 0.014214222, -0.012953692, 0.022930251, 0.012301257, -0.019066315, 0.014024193, 0.034154676, 0.0030705635, 0.025311958, 0.014999679, -0.026072077, 0.016190533, -0.019370362, 0.022942921, -0.026477473, 0.09243045, -0.042946715, -0.02809906, -0.011408118, -0.018914292, 0.024057763, 0.0023341982, 0.02528662, 0.021486027, -0.0048172534, 0.03326787, 0.024133774, 0.029898008, -0.037524533, -0.007259135, 0.024323804, -0.012098558, 0.034154676, -0.0022011776, 0.012928355, -0.009691515, 0.023272306, -0.007651863, 0.016152525, -0.035902947, -0.03456007, -0.02015582, -0.00037590254, 0.011142076, 0.0014030527, -0.045125723, -0.0040919734, 0.0130487075, -0.0014378915, -0.056704868, 0.014252229, 0.056907564, -0.0065053506, 0.0043706833, 0.048292886, -0.008551337, 0.03448406, 0.01743206, -0.013555453, -0.013606127, 0.03990624, 0.037727233, -0.017964143, 0.0069107474, 0.012801669, -0.025413306, -0.015557099, 0.007702538, 0.0462659, 0.018952297, -0.017444728, -0.012719322, 0.029669972, -0.06192435, -0.01138278, -0.009596501, 0.002221764, -0.0019288016, -0.032685112, -0.01825552, -0.009286119, 0.027617652, 0.031190211, -0.028403109, 0.008158609, -0.027744338, 0.006096787, 0.026958883, 0.013948181, 0.017571414, -0.012117562, 0.018850947, 0.0051783095, 0.0052543217, 6.710424e-05, -0.026629498, 0.023576353, -0.030227393, 0.0092354445, 0.010134918, 0.0055330317, 0.031823643, 0.018496225, -0.01287768, -0.036308344, -0.021676056, 0.012541961, -0.010495975, -0.013124719, -0.03473743, 0.011940201, -0.010014566, 0.01835687, 0.02135934, 0.02585671, -0.052904274, -0.014074868, -0.033571918, -0.042591993, 0.0202445, 0.026477473, 0.013365423, 0.027566977, 0.0073414813, 0.00048813882, -0.024133774, 0.008519665, 0.017229361, -0.008228287, -0.0010902955, 0.035928287, -0.009672512, 0.039779555, 0.007550514, 0.0060176076, 0.05553935, -0.030962175, 0.007430162, -0.0037625884, 0.0066827117, -0.004791916, -0.03542154, 0.011028058, 0.019927783, 0.030987512, 0.012155567, 0.022638874, 0.0016896809, 0.004301006, 0.0039589526, 0.04532842, -0.0032067515, 0.0145056015, -0.005393677, 0.012047884, 0.008215618, -0.04928104, -0.012801669, -0.005770569, -0.026401462, 0.02481788, -0.016849302, 0.032431737, 0.016279213, 0.022448843, -0.032735787, 0.0035535556, 0.024235124, 0.00044181908, -0.0019129658, -0.00311332, -0.046797983, -0.00019834352, 0.024767207, -0.010603658, 0.03744852, 0.030202055, -0.018027486, 0.008906059, 0.01724203, -0.009957557, 0.013707477, 0.005561536, -0.040894393, 0.02969531, -0.012592636, -0.0071514514, -0.03889275, 0.013327417, -0.016735284, -0.0019335523, -0.070387006, -0.025375301, -0.0093557965, 0.0013832579, -0.008982072, -0.023538347, 0.041527826, -0.03734717, 0.02313295, -0.007094443, -0.027034894, -0.020079806, -0.022778228, -0.0077342093, -0.0324064, 0.015797803, -0.0088743875, -0.0108696995, 0.023576353, 0.008354973, 0.032558426, 0.0108126905, 0.004224994, -0.004576549, -0.0013017035, -0.04074237, 0.02146069, 0.026452137, -0.011705831, 0.040869057, 0.02210679, 0.07591054, 0.02247418, -0.041857213, 0.022030778, -0.009735855, 0.048850305, 0.02512193, 0.021600043, -0.00095252396, 0.00622664, 0.035269514, 0.011110404, -0.026958883, 0.0012431111, -0.029213902, -0.01976309, 0.009210107, 0.009875211, 0.029948683, 0.00088442996, 0.024374478, -0.0016611764, 0.030810151, -0.0005613795, -0.021954766, 0.0049946145, -0.03164628, -0.0027427622, -0.007550514, -0.023158288, 0.026148088, -0.022486849, 0.016139857, 0.005225817, 0.009849873, -0.00039391578, -0.0067903954, -0.023652365, 0.004066636, 0.0018923793, -0.007233798, 0.012985364, 0.01602584, 0.014366247, 0.0138595, 0.020117812, 0.0047855815, -0.010755681, -0.010394625, 0.030506104, -0.01863558, 0.0010514978, -0.014682963, 0.005621712, 0.049965147, 0.022195471, -0.015683785, 0.010103246, -0.010673336, -0.028403109, -0.008177612, -0.019509718, 0.030379416, 0.0005736522, -0.016823964, -0.021029955, -0.038411338, -0.0035092153, -0.025134597, 0.0043928535, -0.0051941457, 0.0025147265, 0.04066636, 0.011826183, -0.0065623596, 0.0055900407, 0.013188062, -0.010850697, -0.006236142, -0.0069677564, -0.0068727415, 0.0096598435, 0.00016597907, -0.027769675, 0.01855957, -0.0132134, -0.026198763, 0.0020174822, -0.050269194, 0.021676056, 0.008443654, 0.0057452316, -0.00805726, 0.033293206, 0.010698673, 0.003813263, 0.0011663074, 0.03129156, 0.06116423, 0.065167524, -0.024969904, 0.012953692, 0.0068727415, 0.004063469, 0.038563363, -0.019839102, 0.014024193, -0.029061878, 0.023804389, 0.0076138573, 0.023855064, 0.0014093871, -0.014543608, -0.012117562, 0.024222454, 0.0037309167, 0.009444477, 0.044441614, 0.035928287, -0.008468991, 0.048419572, -0.007981248, -0.0015384489, 0.021777404, 0.015506425, 0.009704184, 0.016165195, 0.018521562, 0.022448843, 0.012674982, -0.020687902, 0.01657059, 0.0019256345, -0.025768029, 0.025083922, -0.005577372, 0.026933545, -0.03491479, 0.0008606762, 0.005415847, 0.008076263, 0.02276556, 0.0006734178, -0.0030452262, 0.016507247, 0.001583581, -0.027972374, -0.010229933, -0.027338943, -0.018420214, -0.005441184, -0.032735787, -0.012915687, 0.0144422585, -0.001304079, 0.0022550193, -0.020016463, 0.0003188936, -0.022930251, 0.005203647, 0.02078925, -0.034104, 0.013884838, 0.022499518, -0.05039588, -0.014657625, -0.0026065742, 0.0033571916, 0.007550514, -0.020409191, -0.0068980786, -0.042110585, 2.7935357e-05, 0.016836632, 0.017698102, -0.00024505914, -0.0053144977, 0.033470567, -0.00018013234, -0.013264074, -0.010660667, -0.029898008, -0.010039903, -0.019813765, -0.027186917, 0.04140114, -0.0061727986, -0.013441435, 0.035649575, -0.008792042, 0.013314749, 0.0049376055, 0.023297643, 0.001485399, -0.015367069, 0.0030024694, -0.0006559984, -0.009254447, -0.002445049, 0.01232026, -0.009533158, -0.026072077, 0.014720969, 0.012377269, 0.0066763773, -0.02483055, 0.001946221, 0.014847655, -0.0011607648, 0.052296177, 0.0062899836, 0.010172924, -0.016367894, -0.0011742252, 0.03795527, -0.014936335, 0.036587056, -0.036384355, -0.032609098, 0.013479441, -0.009178435, 0.0021631715, 0.010008232, 0.022600867, 0.008741367, 0.045379095, 0.0049946145, -0.037549872, 0.0029264577, 0.027060231, 0.0071324487, 0.016380562, -0.003011971, -0.018901622, 0.010850697, -0.017596751, -0.0009754859, -0.00782289, 0.035649575, 0.0062488103, 0.0128080025, -0.0018638748, -0.02698422, 0.006084118, -0.006261479, 0.033217195, -0.0105656525, -0.01095838, -0.0040983073, -0.0067777266, -0.029543286, 0.0120225465, -0.012415275, -0.007316144, -0.037423186, -0.013074044, -0.007107111, -0.025578, 0.01825552, 0.046139214, -0.0052131484, -0.01835687, -0.0006369954, 0.009051749, 0.0075631826, -0.0294166, 0.026756184, 0.01938303, 0.0065053506, -0.01442959, -0.011693162, 0.001050706, 0.019231007, -0.00843732, -0.024513833, 0.02436181, -0.0053335004, -0.005571038, 0.027389616, -0.023107613, -0.034129336, -0.014024193, -0.014188886, -0.044086892, 0.013264074, -0.0013373342, -0.006575028, -0.011085067, -0.01573446, -0.00936213, -0.05508328, 0.010052572, 0.008741367, 0.034990806, 0.034788106, -0.0046493937, 0.020586552, -2.6920876e-05, 0.012199908, -0.01545575, -0.008576674, 0.008722364, -0.028327096, 0.0011876858, 0.030962175, -0.026072077, -0.051561397, -0.004491036, 0.018597575, -0.0066573746, 0.0011457208, -0.0059827687, 0.0009984478, -0.00070825656, -0.009051749, 0.001564578, -0.0044276924, 0.016760621, -0.028732494, -0.007240132, -0.009621838, 0.013821495, -0.011522135, -0.032609098, 0.0013412931, -0.009400137, -0.021891423, 0.043225426, -0.012358266, 0.008513331, -0.023791721, 0.005419014, 0.02014315, 0.0018322031, 0.024045093, -0.04560713, 0.0005740481, 0.0010261604, 0.02903654, -0.027338943, 0.02885918, 0.01489833, 0.03869005, 0.01048964, 0.03146892, -0.0063058194, -0.009450811, 0.0001239152, -0.030886162, 0.052448202, -0.016684609, -0.045885842, -0.014834986, -0.007189458, -0.0046683964, 0.014112874, 0.02078925, -0.00355989, 0.011116738, -0.057110265, 0.01545575, 0.0032146694, 0.0019335523, 0.0132134, 0.017812118, 0.016557923, -0.0076772003, -0.035472214, 0.0009770695, -0.011142076, -0.026376124, 0.004779247, -0.01947171, 0.041071754, 0.04806485, 0.040919732, -0.0028409443, -0.0077532125, -0.022537524, 0.011655156, 0.0023246969, -0.010255271, -0.014100205, 0.0086906925, 0.015810473, 0.01157281, 0.017482733, -0.0153544005, -0.00059503055, 0.004614555, 0.042009234, 0.00025515447, -0.015075691, -0.0048172534, 0.0083296355, 0.010736679, -0.016912645, -0.019129658, 0.038436677, 0.003499714, -0.018040154, 0.056350145, -0.023170957, 0.030810151, 0.021815412, -0.008139607, 0.045049712, -0.03775257, -0.0067587234, 0.016811296, 0.00012916082, -0.024653189, -0.010394625, 0.020485204, -0.013441435, -0.01221891, -0.020776583, -0.0001080794, -0.0016152526, 0.02144802, 0.009970225, -0.0026002398, -0.019243676, 0.018052824, -0.0025147265, 0.015025016, 0.048571594, 0.031342234, -0.0033761945, 0.018888954, 0.029644636, 0.040412985, 0.045404434, 0.0012541962, -0.010755681, -0.033039834, 0.012073222, 0.013973518, -0.0037657556, -0.006109455, 0.017394053, 0.019446375, 0.0068980786, -0.004456197, 0.0436815, -0.029467275, -0.023335649, -0.008766704, 0.0054855244, 0.01695065, 0.041375805, -0.00590359, -0.0016485078, -0.0049977815, 0.011940201, 0.019877108, 0.0029913844, 0.01743206, -0.021346672, -0.013200731, -0.016697278, -0.004015961, 0.027313605, 0.0010063658, 0.019535055, -0.023373654, -0.04271868, -0.0138595, 0.008373976, 0.015316395, 0.007892568, 0.021384677, 0.009685181, -0.024589846, 0.018762266, -0.017900798, -0.014340909, 0.015392407, -0.018787604, 0.025071254, -0.016633935, 0.01432824, -0.0049249367, 0.009773862, -0.023779051, -0.033622593, 0.02220814, 0.002584404], "content_hash": "844343e7d8b8fdc70c74179aff4cb513"}, {"content": "like **\"The\"**, **\"of\"**, and **\"is\"** should be lighter -- they contribute less to this specific prediction. **Bar chart**: Shows the same information as horizontal bars with scores. Longer bars = more influence. **Hover over any token chip** to see the exact attribution score. ### Step 5: Compare with Integrated Gradients\n\n1. Now switch to **\"Integrated Gradients (more accurate, slower)\"**.\n2. Click **\"Compute Attribution\"** again.\n3. Compare the results. Integrated Gradients should give a more refined picture:\n   - The relative ordering of token importance may shift slightly\n   - Integrated Gradients tends to produce more reliable scores, especially for distinguishing tokens of moderate importance\n\n### Step 6: Change the Target Token\n\n1. In the **Target Token** dropdown, select a different token from the top-5 predictions (e.g., if the model also considered \"a\" or \"the\" as alternatives to \"Paris\").\n2. Run attribution again.\n3. **What to look for**: Different target tokens are driven by different input tokens. For example:\n   - \"Paris\" might be strongly driven by \"France\" and \"capital\"\n   - A generic token like \"the\" might be driven more by \"is\" (as a common grammatical continuation)\n\n### Step 7: Try a Different Prompt\n\nRun attribution on: `Alice gave the book to Bob because she`\n- Which tokens drive the prediction of the next word?\n- Does \"Alice\" have high attribution (suggesting the model connects \"she\" to \"Alice\")?\n- Does \"Bob\" have lower attribution than \"Alice\" for this prediction?\n\n## What You Should Learn\n\n- Token attribution reveals which input tokens \"caused\" a particular prediction\n- Content words (nouns, verbs) typically have higher attribution than function words (the, of, is)\n- Different target tokens can be driven by completely different input tokens\n- Integrated Gradients is more accurate but slower; Simple Gradient gives a quick approximation\n- Attribution helps you understand the \"why\" behind a model's prediction\n\n## What's Next?", "source_file": "experiment_token_attribution.md", "chunk_index": 1, "embedding": [0.006119887, 0.007317256, 0.0013586814, -0.010031293, 0.022723407, -0.018998258, 0.0062529277, 0.033260256, -0.0060500405, 0.0068250042, 0.014940506, 0.009033485, -0.041508798, -0.062901795, -0.025876477, -0.015752057, -0.0032112775, -0.0033642747, -0.02160586, 0.061518166, -0.05273746, -0.017268725, 0.043850318, 0.0369854, 0.027326625, -0.004157532, -0.019264339, -0.008554538, -0.0019074756, 0.02354826, 0.022204546, -0.016989337, -0.025064928, -0.05614331, -0.003781691, -0.0078095077, -0.026262296, 0.002998412, 0.012871718, 0.024599284, 0.00059452705, 0.003377579, -0.04470178, -0.011115577, -0.03421815, 0.017721064, 0.00733056, -0.04946465, -0.040098563, 0.096481346, -0.039193884, 0.03775704, -0.029109374, 0.018173402, -0.035309087, 0.020235538, -0.020248843, 0.0074968613, -0.023694605, -0.04251991, -0.031317856, -0.032222535, 0.037836865, 0.015911706, 0.007995765, -0.02416025, -0.047921374, 0.007343864, -0.0158984, -0.022257762, 0.027699139, 0.024798846, 0.04084359, -0.0019490509, -0.06040062, -0.03200967, 0.011148837, 0.034776922, 0.0016497086, -0.014075739, 0.020421796, -0.017375156, 0.009891599, -0.028790075, 0.00053964765, -0.047362603, -0.031184815, -0.04890588, -0.05231173, -0.06354039, -0.0150070265, -0.019716678, -0.020794312, 0.020461708, 0.014022524, -0.016696647, -0.022271067, -0.01933086, 0.02088744, -0.005078841, 0.016244309, -0.027752357, 0.008827272, -0.02902955, 0.023282178, -0.024439635, 0.018093579, -0.0074103847, -0.010370547, -0.00094542274, -0.031078381, 0.038768154, -0.06981993, 0.04635149, -0.025676915, -0.033872243, -0.044355877, 0.034244757, 0.023947382, -0.032994173, -0.029641539, -0.034138326, -0.0036785842, 0.040524293, 0.037544176, -0.009226395, 0.01018429, -0.048027806, -0.005750698, -0.026288904, -0.008468061, 0.0031963105, -0.0026425272, 0.000784942, -0.0019174537, -0.022138026, -0.0025194644, 0.035202652, -0.057526935, 0.039619613, -0.024439635, -0.056409393, -0.0034357843, 0.0024679108, -0.071469635, 0.009033485, -0.0017661195, 0.036852363, -0.05420091, 0.022630276, 0.030599434, -0.012020256, -0.036692712, -0.014567992, -0.04265295, -0.045792717, -0.0040012086, 0.029269025, -0.026036127, -0.05422752, 0.0044768304, 0.037224878, 0.009299567, -0.028949725, -0.024772238, 0.031211423, -0.044488918, -0.011840651, -0.027459666, -0.03775704, -0.04188131, -0.026541684, 0.013131148, -0.044994473, 0.023654694, -0.01829314, -0.05763337, -0.030918732, 0.04145558, -0.021313172, -0.030120486, 0.0171889, -0.0019141276, -0.012126689, -0.0111222295, -0.0073904283, -0.036639497, 0.02095396, -0.007669815, -0.0038615155, 0.044914648, 0.010729758, 0.075141564, 0.022204546, -0.05082167, -0.014980419, 0.051486872, 0.04339798, 0.025823262, 0.0049324958, 0.025357617, 0.03123803, 0.05954916, -0.031770196, -0.013284146, 0.017454982, 0.02030206, -0.01871887, -0.068649165, -0.00051802845, -0.0029834448, -0.06683981, 0.031903237, 0.020222235, -0.087753855, -0.012146645, -0.041136283, 0.025703525, 0.041854706, 0.033978675, 0.01329745, 0.033100605, 0.0191313, -0.025730133, -0.055291846, 0.028497387, 0.058857348, -0.02636873, 0.024865367, -0.049384825, 0.038422246, 0.04310529, -0.006525662, -0.048081025, -0.010343939, -0.05795267, 0.015219892, -0.032355577, 0.039007626, -0.055345062, -0.0027822203, 0.013084584, 0.029162591, -0.008873836, -0.011714262, 0.021685686, -0.021366388, -0.030998558, 0.023721213, 0.028683644, 0.047149736, -0.02948189, -0.033286862, 0.003397535, 0.019610247, -0.02286975, -0.0058271964, 0.026408643, 0.012372815, -0.03004066, 0.0026841026, 0.0034125021, -0.047256168, -0.011441528, -0.013144452, -0.017055858, -0.032408793, 0.019038169, -0.027645923, -0.043052074, -0.003395872, -0.0029019571, 0.01696273, 0.00018303534, -0.034005284, -0.0113217905, -0.044302657, 0.014395038, -0.025091536, -0.0151932845, -0.0050189723, 0.00499569, 0.06934098, 0.015326325, -0.039353535, 0.0042573125, 0.006006802, 0.03051961, -0.01576536, -0.0025926367, 0.01585849, -0.04991699, 0.026288904, -0.03227575, -0.023295483, 0.01232625, 0.026222384, -0.005604353, -0.03153072, -0.01333071, 0.011754174, -0.022084808, -0.0035422172, -0.017574718, 0.029348848, 0.0026441903, 0.030918732, 0.041056458, 0.02873686, -0.022138026, -0.012771938, 0.009159874, -7.566708e-05, 0.017175596, -0.02471902, 0.02393408, 0.06370004, -0.04621845, 0.022031592, 0.024093729, -0.02993423, -0.050316114, 0.014940506, 0.014461558, -0.0003666943, -0.017202204, -0.0010177637, 0.0074303406, -0.0010817897, -0.011607829, -0.0016838004, 0.03496318, 0.01076967, 0.06311466, -0.05954916, -0.022031592, -0.036240373, 0.0040178387, 0.009425956, 0.06071992, -0.050741844, -0.058431614, 0.02286975, -0.016177788, 0.025863174, 0.015552495, 0.0043837014, 0.028550602, 0.03509622, -0.007317256, 0.00019800245, 0.005630961, 0.018705567, -0.008162066, 0.006089953, 0.042280436, -0.011142185, -0.013324058, 0.010969232, 0.00019550794, -0.014182173, 0.031051774, 0.039885696, 0.02490528, -0.007343864, -0.013576835, -0.016137876, 0.015499279, 0.00053216406, -0.0064558154, -0.030599434, 0.0034756965, -0.041269325, -0.022390803, -0.05566436, 0.0014634512, -0.0019839741, 0.011148837, -0.02501171, 0.0140358275, 0.045420203, -0.046245057, 0.002629223, -0.027778964, 0.048799444, -0.011248618, 0.015086851, -0.03200967, -0.026342122, 0.03035996, -0.014607904, -0.01865235, -0.027592707, 0.03419154, -0.008800663, 0.03020031, 0.010716454, 0.053509098, 0.04706991, 0.06087957, -0.039087452, -0.010084509, -0.05747372, -0.01583188, 0.04220061, 0.041375756, 0.00017752661, 0.009465869, 0.0118073905, 0.03256844, 0.06327431, -0.035282478, 0.04102985, -0.030599434, 0.008241891, 0.028790075, 0.031477503, 0.008986921, 0.0408702, 0.025517266, 0.008095546, -0.060187757, -0.008162066, -0.07067139, -0.035309087, -0.00755673, 0.026262296, -0.017854104, 0.018439485, -0.047335993, 0.006542292, 0.035202652, 0.05364214, -0.039805874, 0.0047395863, -0.02118013, 0.062742144, 0.036373414, 0.01858583, -0.018918432, 0.031903237, -0.05135383, -0.031637155, -0.056196526, -0.055185415, -0.021885248, -0.033872243, -0.05614331, -0.008880488, 0.0062429495, -0.033978675, 0.015512583, -0.0078095077, 0.05566436, -0.014155564, -0.04121611, 0.03094534, 0.016976034, 0.059016995, -0.023694605, 0.034031894, 0.0059635635, -0.01680308, 0.01933086, 0.007856072, 0.0074103847, 0.0027057217, -0.02393408, 0.03775704, -0.004303877, 0.013031367, -0.010483632, -0.006166451, -0.05185939, 0.021738902, -0.026501771, 0.018466093, -0.015259805, 0.008328368, -0.025477355, -0.016656736, 0.008461408, -0.012951543, 0.011780782, -0.018998258, 0.064977236, 0.03315382, -0.014208781, -0.0022716753, -0.031184815, -0.00054047914, -0.057260852, 0.020089194, -0.014820769, 0.017375156, -0.023694605, 0.014128956, -0.018133491, 0.022656886, -0.0126788085, 0.042307045, -0.03123803, -0.052205294, 0.07115033, -0.016390653, -0.0025760068, 0.050156463, 0.029109374, -0.03182341, 0.047575466, -0.047442425, -0.03969944, -0.031903237, 0.00943926, -0.025091536, 0.00651901, -0.018106882, 0.055717576, 0.040364645, 0.015525887, -1.6279335e-05, 0.027433058, -0.008720838, -0.0060334103, 0.0013753115, -0.01770776, -0.02451946, -0.015978226, -0.03216932, -0.042839207, 0.022164633, -0.026927503, 0.0054247477, 0.0048859315, -0.04664418, 0.002486204, 0.003788343, -0.030173702, -0.044409093, 0.014767553, 0.016856296, 0.0040377947, 0.0050588846, 0.03717166, -0.005657569, 0.0072440836, -0.008115502, -0.034697097, -0.0042273784, 0.010510241, -0.009026833, -0.026009519, 0.022537148, -0.03065265, 0.0005687504, -0.03613394, 0.022470627, -0.011767478, -0.022563757, -0.028710252, 0.024173552, 0.006013454, 0.013430491, -0.0066021606, -0.014514775, 0.045366988, 0.042253826, 0.007184215, 0.023069313, 0.0046963478, 0.010955928, 0.010869451, 0.031025166, 0.011867259, 0.032754697, 0.01748159, -0.009851688, 0.004849345, 0.008428148, -0.004397006, -0.013450447, -0.009652126, -0.009272959, 0.0097785145, -0.02474563, -0.015313021, 0.01651039, 0.008215283, 0.0039712745, 0.02844417, 0.0062462757, -0.021060392, 0.004177488, 0.013111192, -0.025038319, 0.0265949, 0.0017212181, -0.019157907, -0.020275451, 0.014288605, 0.039167274, -0.032382183, -0.031291246, -0.0015299717, -0.002210144, 0.021392995, -0.00862771, 0.014900594, -0.011261922, 0.0048626494, -0.013390578, -0.054387167, -0.020035977, -0.00013293709, -0.008441453, -0.004180814, -0.031025166, 0.005996824, 0.016696647, -0.039593007, 0.0064425115, -0.009971424, -0.006013454, -0.0009778515, -0.010869451, 0.040364645, 0.0029019571, -0.00032491112, 0.022177938, -0.0008061454, -0.027246801, 0.0010385514, 0.019942848, 0.007689771, 0.01284511, 0.012053516, -0.019543726, 0.021911856, 0.0070179137, 0.0007105222, -0.0040078606, 0.02377443, -0.015778664, -0.023268875, -0.011674349, -0.0055511366, 0.010210898, 0.023388611, -0.015432758, -0.0291892, -0.03419154, -0.033366688, 0.048240673, -0.016497087, -0.0014476526, -0.023441827, 0.027246801, 0.016377348, -0.03395207, -0.013729833, -0.027353233, -0.015459366, -0.03820938, -0.042040963, -0.015366238, 0.013822962, 0.0071309987, 0.03259505, 0.022417411, -0.008315064, -0.10244159, 0.024918582, 0.0002673293, 0.0363202, -0.0071509546, -0.011082317, 0.008441453, -0.054280736, -0.02993423, 0.03781026, -0.012233121, -0.0064724456, -0.040896807, -0.033419903, 0.00033551283, -0.021951769, 0.02636873, 0.00061905646, 0.006286188, -0.030971948, -0.020488316, -0.029907621, 0.0069713495, -0.007856072, -0.024200162, -0.00733056, 0.020634662, 0.053934827, -0.016616823, -0.08754099, -0.004214074, -0.003794995, -0.027619315, -0.02156595, -0.0044036577, 0.0026574943, -0.013290797, 0.03509622, 0.0090933535, -0.03392546, -0.034564056, -0.0031214748, 0.016111268, -0.010077857, -0.021246651, 0.017228812, 0.0048925835, 0.017534805, -0.010696498, -0.060613487, -0.026009519, -0.00071800576, -0.04015178, 0.0015133416, 0.024253378, -0.013377274, -0.020541534, 5.3320346e-05, -0.02137969, 0.019729983, -0.007224127, 0.003415828, -0.012419379, -0.045074295, -0.006415903, 0.010490284, -0.026648115, 0.03363277, -0.025291098, -0.016856296, 0.0045533287, 0.0013171061, 0.0022550453, -0.017135683, 0.025663612, 0.03701201, -0.036825754, 0.028710252, -0.022936272, -0.041375756, -0.019317556, 0.011647741, 0.0150070265, -0.011528004, 0.015605711, 0.0037683868, 0.0047429125, -0.04488804, -0.00862771, 0.014408343, 0.004836041, 0.03123803, -0.012073472, 0.0039047538, -0.01998276, -0.018625742, 0.027672531, 0.026541684, -0.031876627, 0.058591265, -0.0012073473, -0.008601102, 0.03866172, -0.028364345, 0.03451084, -0.003645324, 0.022244459, 0.011840651, -0.019024866, 0.0040976633, 0.0029119353, -0.0010610021, -0.02286975, 0.009831731, 0.025996216, 0.0074968613, -0.0008930378, -0.036932185, -0.008747447, -0.00755673, -0.020807615, 0.0035621733, 0.011182098, -0.02251054, -0.0043338113, 0.025996216, -0.010836191, -0.1151603, -0.012924935, 0.059123427, -0.020754399, -0.001566558, 0.0008780707, 0.046750613, 0.005890391, -0.036187157, 0.023308786, -0.02695411, -0.019889632, -0.022590365, -0.016670039, -0.011853955, -0.008813967, -0.012266382, -0.01511346, -0.013623401, -0.014687728, -0.017002642, 0.0036386717, 0.00817537, -0.03203628, -0.051619913, -0.0071243467, -0.02740645, -0.01897165, 0.013064628, 0.01329745, -0.010383852, 0.007689771, 0.02416025, 0.0005055559, 0.014408343, 0.021233346, -0.0022517191, 0.016976034, 0.0033609488, -0.02977458, 0.01608466, -0.02857721, 0.038608503, -0.033659376, 0.0765252, -0.05954916, -0.003139768, -0.012592332, -0.008155414, 0.015752057, -0.025823262, 0.021007176, -0.0012971499, 0.008095546, 0.024865367, 0.019876327, 0.031876627, -0.0360009, -0.021326475, 0.022949576, -0.008102198, 0.013876178, -0.008441453, 0.00023282178, 0.0053249667, 0.010530196, -0.00546466, 0.01089606, -0.0428126, -0.015419454, -0.005870435, 0.0246525, -0.0037650608, -0.019357469, -0.024705717, 0.014581296, 0.0038781457, 0.003026683, -0.04459535, 0.02007589, 0.059336293, 0.00084314746, 0.0025560504, 0.040550902, 0.0048626494, 0.035921074, 0.0027339929, -0.011315138, -0.002107037, 0.034564056, 0.057793017, -0.008667623, 0.00836828, -0.0056642215, -0.0431319, -0.008155414, 0.028204696, 0.043318156, 0.025038319, -0.012798546, 0.008813967, 0.03983248, -0.04595237, -0.009612214, -0.014408343, 0.0010618336, -0.0057773064, -0.020634662, -0.007869376, 0.0015673895, 0.018772088, 0.03480353, -0.02030206, 0.008354976, -0.017441677, 0.02977458, 0.0070977383, 0.023561563, 0.023441827, -0.029348848, 0.017827496, 0.012924935, -0.008022374, -0.006685311, -0.017215507, 0.027326625, -0.018732175, -4.516119e-05, 0.008488017, -0.0014967114, 0.03123803, 0.002620908, -0.009319523, -0.015219892, -0.039140668, 0.00276559, 0.015738752, -0.024572676, -0.05672869, 0.027100455, -0.011195402, 0.02296288, 0.024559371, 0.006392621, -0.043451197, -0.020741094, -0.0431319, -0.04874623, 0.025317706, -0.00093877065, 0.032754697, 0.020315364, 0.000783279, 0.011248618, -0.024213465, 0.028311128, 0.025956303, -0.005930303, 0.017268725, 0.050316114, -1.0218434e-05, 0.03272809, 0.011315138, -0.013543576, 0.055930443, -0.038635112, -0.0018143469, 0.0025244532, -0.007450297, -0.011887215, -0.027193584, 0.0043936796, 0.013337362, 0.039885696, 0.022044897, 0.030838909, 0.009040137, 0.0046597617, 0.0072773434, 0.02715367, -0.0022949576, 0.013969307, -0.0071443026, 0.018027058, 0.03347312, -0.027326625, 0.0027838834, 0.0052551203, -0.0015939977, 0.000117242394, 0.009166526, 0.0230294, 0.012838458, 0.050449155, -0.03674593, 0.0018659002, 0.031158207, 0.010576761, 0.009299567, 0.0136500085, -0.045233946, -0.0031031817, 0.035149436, 0.0041375756, 0.022204546, 0.01770776, -0.02682107, -0.0008780707, 0.0084348, -0.0007932571, -0.015259805, 0.0096787345, -0.032062884, 0.042040963, -0.0068782205, -0.001834303, -0.032834522, -0.00021234594, -0.0017627935, -0.02007589, -0.05332284, -0.02828452, -0.005780632, 0.0036952142, -0.0018742153, -0.00274896, 0.056835122, -0.01284511, 0.008388236, -0.0017677825, -0.014248693, -0.007929245, -0.013516967, -0.015619015, -0.01917121, 0.019902937, -0.007463601, -0.00042614702, 0.018612439, 0.0066753333, 0.039619613, 0.026248993, -0.009425956, -0.01066989, -0.0048759533, -0.025118144, 0.017242115, 0.023495045, -0.0051819477, 0.003505631, 0.0123927705, 0.059815243, 0.048081025, -0.037411135, -0.0011375007, -0.018692262, 0.030466393, 0.026022824, 0.014554687, 0.020860832, 0.022457324, 0.025344314, 0.010323983, -0.02449285, 0.0042706165, -0.02127326, -0.007716379, 0.010011337, 0.01998276, 0.010117769, -0.026036127, 0.03586786, 0.0022982836, 0.036666103, 0.007303952, -0.0071243467, 0.005667547, -0.03879476, 0.008873836, -0.004609871, -0.014993723, 0.01533963, -0.041109674, 7.254893e-05, 0.009272959, 0.0035023047, 0.00057249214, -0.01284511, -0.017960537, -0.0031198119, 0.0037018664, 0.011833998, 0.012173253, 0.0046298276, 0.0059569115, 0.0074236887, 0.017614631, 0.005334945, 0.01261894, -0.028790075, 0.027353233, 0.0026591574, -0.0070378697, -0.012366163, 0.009618865, 0.035335694, 0.016483782, -0.027353233, 0.008255195, -0.003918058, -0.039167274, -0.00050389284, -0.020328667, 0.01576536, 0.0059569115, -0.023721213, -0.017268725, -0.03983248, -0.02257706, -0.016403958, 0.0028088284, -0.0067252233, -0.0017378483, 0.035601776, 0.006419229, -0.004955778, -0.0011042404, 0.0148074655, -0.015073547, 0.005477964, -0.025264489, -0.008155414, 0.008900444, -0.0041309237, -0.050422546, 0.014860681, -0.028417561, -0.01709577, -0.008022374, -0.0431319, 0.012924935, 0.020009369, 0.0094924765, -0.032089494, 0.034271367, 0.008940357, 0.0071576065, 0.01044372, 0.01858583, 0.017335245, 0.080942154, -0.018825304, 0.005318315, 0.010403807, -0.01378305, 0.025703525, -0.012352859, 0.009292915, -0.04121611, 0.013397231, -0.0030682585, 0.02474563, 0.036187157, -0.0066986154, 0.00016432644, 0.0062595797, 0.011873911, 0.009745254, 0.029348848, 0.021033784, 0.0056941556, 0.04028482, -0.027233496, 0.012725374, 0.005734068, 0.023428524, 0.012046864, 0.008680927, 0.016590215, 0.013956003, 0.011581221, -0.020341972, -0.01284511, 0.016909514, -0.03349973, 0.016430566, -0.02630221, 0.026515074, -0.024080424, 0.0059868456, 0.011614481, 0.021353083, -0.0060500405, 0.0063360785, -0.009825079, 0.016829688, 0.030306743, -0.02620908, -0.007962505, -0.019956153, -0.006655377, 0.0052784025, -0.023521652, -0.019117994, 0.005627635, -0.0012406075, -0.005341597, -0.013576835, 0.0063959467, -0.020395188, 0.004140902, 0.016550303, -0.019769896, -0.0017627935, 0.020501621, -0.041668445, 0.0043837014, -0.013390578, 0.0015657265, 0.019543726, -0.035761427, -0.013729833, -0.012179906, -0.0012830143, 0.015592407, 0.0071043903, -0.0038548636, -0.0051786215, 0.024373114, -0.008986921, -0.013131148, -0.021419603, -0.04031143, -0.00039808368, -0.028923117, -0.016616823, 0.05050237, -0.0012189883, -0.021845335, 0.030812299, -0.027965222, 0.001239776, 0.027672531, 0.012199861, 0.008973616, -0.024891974, -0.015565799, -0.0014094033, -0.01333071, -0.0037983211, 0.017082466, -0.01229299, -0.029428674, -0.011860606, 0.008341672, 0.0023199026, -0.05183278, -0.007895985, 0.017268725, -0.0019307578, 0.04017839, -0.010403807, 0.023175744, -0.0024512808, -0.0091066575, 0.0090933535, 0.0027256778, 0.051619913, -0.03363277, -0.041934527, 0.0030699214, 0.0060633444, -0.0005126237, -0.01517998, 0.0065090316, 0.013995915, 0.014567992, 0.012539116, -0.03703862, -0.001172424, 0.016616823, 0.021007176, 0.020501621, 0.005225186, -0.004872627, 0.002474563, 0.0078028557, 0.0043737236, -0.008867184, 0.014634512, 0.010543501, 0.011501396, -0.014448254, -0.036346804, 0.019211123, -0.011973691, 0.0291892, 0.002614256, 0.004693022, 0.007822812, 0.0007013756, -0.021685686, 0.014900594, -0.012459291, 0.007197519, -0.025384227, -0.016324133, -0.011641089, 0.00020933173, 0.014514775, 0.03395207, -0.01261894, -0.009139918, -0.010057901, 0.017175596, -0.0014202129, -0.03509622, 0.026967414, 0.020155715, -0.013862874, -0.011780782, -0.015778664, -0.022550453, 0.026914198, -0.010310679, -0.032967564, 0.007583338, 0.031637155, -0.005075515, 0.031610545, -0.014501471, -0.029694755, 0.008048981, -0.010317331, -0.03453745, 0.015273109, 0.010463676, -0.009745254, 0.0034823485, -0.008115502, -0.0071043903, -0.07189537, -0.0079492, 0.0046497835, 0.029614931, 0.034324583, -0.019197818, 0.014940506, 0.0035921074, -0.0018659002, -0.01585849, 0.006129865, 0.036666103, -0.022244459, -0.00033655218, 0.037730433, -0.013430491, -0.036719322, 0.0032395488, 0.027087152, 0.0046497835, 0.0043870276, 0.009692038, 0.023801038, 0.028550602, -0.011241966, -0.00651901, -0.016310828, 0.01579197, -0.026807765, -0.0010327309, -0.016816385, 0.016670039, -0.0020837549, -0.030120486, -0.0041741617, 0.0023199026, -0.018891824, 0.033659376, 0.0067218975, 0.023388611, -0.035309087, -0.022364195, 0.025211273, 0.026049431, 0.014780857, -0.043903537, -0.01897165, 0.0023947384, 0.039779264, -0.016137876, 0.015619015, -0.016151179, 0.016377348, 0.004533373, 0.015060243, 0.004842693, -0.008095546, 0.004067729, -0.010111118, 0.035734817, -0.0062429495, -0.04076377, -0.021126913, -0.0118073905, -0.028896509, 0.020501621, 0.022829838, 0.0012289664, 0.012386119, -0.06449828, 0.008840576, 0.012605636, -0.0050522326, 0.013876178, 0.03126464, -0.003026683, -0.0030599434, -0.024146944, -0.0039014278, -0.009166526, 0.0013279157, -0.010643281, -0.030280136, 0.032515224, 0.042147394, 0.016403958, -0.019224428, 0.0041109673, -0.023122529, 1.7201788e-05, -0.00061115716, -0.005231838, -0.03035996, 0.01488729, 0.01336397, 0.0029152613, 0.020421796, -0.021725599, 0.0026275602, 0.012771938, 0.030546218, 0.0046364795, -0.026634812, -0.017734367, 0.00840154, 0.015219892, -0.02393408, -0.0062296456, 0.014554687, 0.012851763, -0.004829389, 0.035229262, -0.018013753, 0.023987295, 0.03554856, -0.0021851987, 0.04039125, -0.03153072, -0.009638822, 0.02902955, -0.0012580691, -0.020089194, -0.001962355, -0.0028005135, -0.0068715685, -0.015738752, -0.011647741, -0.00036108165, -0.0059036952, -0.005630961, -0.005637613, 0.019450597, -0.0017228812, 0.0064491634, 0.009266307, 0.0088073155, 0.049943596, 0.005887065, 0.0039879046, 0.010270767, 0.0389278, 0.037730433, 0.013902786, 0.013510316, -0.005630961, -0.023016095, -0.0026890915, 0.0126788085, -0.011900519, -0.009519084, 0.011880563, 0.005388161, -0.013091236, 0.0035654993, 0.039459966, -0.027858788, -0.015446062, -0.0291892, 0.0073638204, 0.014953811, 0.03331347, 0.011501396, -0.015073547, 0.0030848887, 0.017947232, 0.016497087, -0.0013062964, 0.0075500775, -0.043770496, 0.0015449388, -0.022750014, -0.011661045, 0.016071355, -0.004895909, 0.014860681, -0.021712294, -0.050395936, -0.008527929, 0.011261922, 0.007283996, 0.0063261003, 0.0340585, -0.0014326854, -0.015406149, 0.017814193, -0.013257537, -0.014940506, 0.013430491, -0.009186482, 0.03166376, -0.008135458, -0.008441453, -0.008328368, 0.01770776, -0.031477503, -0.025863174, 0.014913898, 0.005584397], "content_hash": "9eb7f48406d75b5edf6310b1bcf2c8b2"}, {"content": "typically have higher attribution than function words (the, of, is) - Different target tokens can be driven by completely different input tokens - Integrated Gradients is more accurate but slower; Simple Gradient gives a quick approximation - Attribution helps you understand the \"why\" behind a model's prediction ## What's Next?\n\nMove on to **Experiment: Comparing Heads** to combine ablation with your understanding of head categories, or try **Experiment: Beam Search** to explore how the model generates longer sequences.", "source_file": "experiment_token_attribution.md", "chunk_index": 2, "embedding": [-0.01083987, 0.036451172, 0.041786857, -0.0038999664, 0.019419095, -0.009070669, 0.007975449, 0.011914029, -0.0021307648, 0.022803044, 0.02375785, -0.0024326523, -0.034457307, -0.027113717, -0.041983437, -0.02069685, 0.014322109, -0.049060244, 0.007182116, 0.05287947, -0.033025097, -0.030413419, 0.028251061, 0.064983055, 0.03369908, -0.014476563, -0.033446338, 0.015389246, 0.012742464, -0.00186047, 0.018955732, -0.025667466, -0.032014124, -0.04858284, 0.0023396288, 0.008565183, 0.014869719, 0.017411191, 0.013325178, 0.0011680592, 0.017144406, -0.005314626, -0.079698324, -0.012272081, -0.02024753, -0.006311557, -0.013767478, -0.038782023, -0.01971396, 0.08261891, -0.035636775, 0.047740363, -0.03195796, 0.053441122, -0.026846934, 0.0310874, -0.009295329, -0.02219927, -0.00689778, -0.014230841, -0.031620968, -0.027787698, 0.040888216, 0.027408585, -0.0017753447, -0.03128398, -0.051391095, 0.021904401, -0.022999622, -0.013591962, 0.008354563, -0.0017990394, 0.047543786, -0.015276916, -0.047431454, -0.026776727, 0.017355027, -0.0054234457, 0.0011136492, -0.020921512, 0.0136551475, -0.0072523225, -0.004212385, -0.03426073, -0.02026157, -0.043584142, -0.017003994, -0.05908572, -0.055434987, -0.047768444, -0.0011584059, 0.025246227, -0.024530122, 0.0061746542, -0.001187366, 0.018731073, -0.0136551475, -0.019826291, 0.017116325, 0.0051566614, -0.027928112, -0.011661286, 0.019208476, -0.02878463, 0.04378072, -0.056530207, -0.028054483, -0.030413419, -0.017607769, 0.012833733, -0.009330433, -0.008796864, -0.04355606, 0.05417127, 0.010011435, -0.038669694, -0.018085172, 0.04847051, 0.017635852, -0.027661327, -0.0014199248, -0.018464288, 0.0031698197, 0.004117606, 0.018422164, -0.023140034, 0.04170261, -0.051475342, -0.00448619, 0.0017902636, -0.0052619707, 0.020079035, -0.0024975932, -0.010720519, -0.029711355, 0.008881112, -0.012117627, 0.038978603, -0.077901036, 0.055631563, -0.010130785, -0.052008912, 0.0059605245, 0.006062324, -0.06779131, 0.0040649516, -0.010769664, 0.039540254, -0.047684196, 0.027871948, -0.012777568, -0.017888594, -0.039203264, -0.012932022, -0.053469207, -0.05144726, -0.031115483, 0.035608694, 0.00831946, -0.034962796, 0.022620507, 0.018127296, 0.03852928, -0.022592425, -0.030862741, 0.029599024, -0.077620216, 0.000859151, -0.016400218, -0.01981225, -0.036928575, -0.026495902, -0.038445033, -0.019573549, 0.025330475, -0.012770547, -0.052598648, -0.017509481, -0.019868415, -0.032014124, -0.0064625004, 0.034822382, 0.008691554, 0.009119813, -0.004531824, -0.044510867, -0.01565603, 0.015417329, -0.0030101, 0.010376508, 0.014532728, 0.045269098, 0.091493, 0.0013181254, -0.0061535924, -0.030357255, 0.057513095, 0.040747803, -0.0009039076, 0.008052676, 0.032435365, -0.007568252, 0.063635096, -0.028208937, -0.032210704, 0.033783328, 0.030441502, 0.021904401, -0.05349729, -0.02165166, 0.01903998, -0.056867197, 0.01499609, 0.017888594, -0.085427165, -0.031592887, -0.01816942, 0.021202337, 0.042376593, 0.017481398, 0.0106994575, 0.009913146, 0.031340145, -0.008340522, -0.066106364, 0.019840334, 0.05987203, -0.056979526, 0.033193592, -0.047234878, 0.039764915, 0.018478328, -0.010629252, -0.046251986, -0.028756548, -0.061163828, -0.011218986, -0.03381141, 0.05756926, -0.047740363, -0.034822382, -0.007287426, 0.0066064238, -0.013388364, -0.018029008, 0.026229117, -0.026369529, -0.02549897, 0.036928575, 0.009807836, 0.030666163, -0.009337453, -0.04981847, 0.025133897, 0.0033751735, -0.012194854, 0.0038262496, 0.013739396, 0.017158449, 0.00015324744, 0.014083407, 0.011935091, -0.014434439, 0.025990415, -0.032182623, -0.023294488, -0.011478749, 0.012356329, -0.03569294, -0.035412118, -0.022255434, -0.03400799, 0.028517846, -0.0067257746, -0.015052255, -0.002309791, -0.010889015, 0.033951823, -0.023280447, -0.028068526, -0.00843179, 0.005588431, 0.06307344, -0.016105352, -0.057204187, 0.008895153, -0.016681043, 0.015810484, 0.006258902, -0.008417749, 0.0071610543, -0.04015807, 0.018534495, -0.023687644, 0.01432913, 0.038641613, 0.010713499, 0.004314184, 0.003053979, -0.012328247, 0.02715584, -0.016077269, 0.020949595, -0.002144806, 0.022522219, 0.012187834, 0.00043176947, 0.05897339, 0.02943053, 0.00037845646, 0.012201875, 0.018014966, 0.02023349, 0.028602093, -0.023589356, 0.008066718, 0.035945684, -0.03793955, 0.020064993, 0.017509481, -0.02121638, -0.075092785, 0.018365998, -0.012489721, 0.028180854, 0.0024975932, -0.036731996, 0.02167974, -0.011562997, 0.0028942595, -0.010411612, 0.004605541, -0.021061925, 0.03282852, -0.014925884, 0.01324093, -0.071835205, 0.043640308, 0.007203178, 0.045044437, -0.07234069, -0.029346282, -0.007792912, -0.0048407326, 0.031705216, 0.05144726, 0.031817548, 0.05262673, 0.03939984, -0.008516038, -0.0030609998, -0.019405054, 0.0065151555, 0.018969774, 0.016554672, 0.02287325, -0.0051917643, -0.03324976, 0.01488376, 0.009239164, -0.015347122, 0.022311598, 0.029570943, 0.04060739, -0.021595493, -0.023617439, 0.011675327, 0.01269332, -0.0086564515, -0.007820995, -0.024052719, -0.0049776346, -0.040775888, -0.0031470025, -0.043836884, 0.023673603, -0.009021524, 1.0071604e-05, -0.035861436, 0.0149118425, 0.008066718, -0.0105520245, -0.045212932, -0.058467902, 0.030385336, -0.026243158, 0.0031750852, -0.005735864, -0.03875394, 0.038894355, 0.008937277, -0.035805274, -0.031649053, 0.03063808, -0.015459453, 0.028588053, 0.032744274, 0.056726784, 0.04133754, 0.05484525, -0.032210704, -0.00045765808, -0.040944383, -0.04102863, 0.040747803, 0.050492454, 0.024263337, 0.050801363, 0.028152773, 0.020472191, 0.053834278, 0.00908471, 0.042208098, 0.0032909256, -0.0027240089, 0.0038122083, 0.007561231, -0.010334385, 0.02506369, 0.05341304, 0.0025993926, -0.04642048, -0.027197964, -0.052430153, -0.023069829, 0.017692016, 0.023477025, -0.017355027, 0.015164586, -0.018506411, -0.008031614, 0.02976752, 0.021792071, 0.007406777, -0.005570879, -0.011134737, 0.06341043, 0.060546014, -0.00448619, -0.023715727, 0.04465128, -0.024530122, -0.0396245, -0.049397234, -0.06442141, 0.012321226, -0.06093917, -0.045943078, -0.022480095, -0.014715265, -0.009393618, 0.014687182, -0.021398915, 0.022564342, -0.0009267247, -0.034288812, 0.040663555, 0.00492849, 0.07599142, -0.004222916, 0.023420861, -0.009913146, -0.005525245, -0.011780636, 0.028644217, 0.04080397, 0.007080317, -0.052233573, 0.040551227, -0.009878043, 0.023041746, 0.02878463, 0.021469122, -0.055828143, 0.0017753447, -3.7955342e-05, 0.028335309, -0.02976752, -0.010222054, -0.03515937, -0.030834658, 0.019011898, 0.02419313, 0.018674906, -0.022746878, 0.07419414, 0.0085581625, -0.021370834, 0.00246249, -0.009470846, -0.0115981, -0.060433682, 0.022058856, -0.016877621, 0.016610838, -0.022044815, -0.008586245, -0.012279102, 0.0069434145, -0.0026257199, 0.053104132, -0.02287325, -0.05714802, 0.060096692, 0.006774919, 0.0055603483, 0.044033464, 0.0161896, -0.023491066, 0.03942792, -0.03763064, -0.016400218, -0.018941691, 0.023799974, -0.010292261, 0.005111027, -0.026186993, 0.054789085, 0.03990533, 0.0161896, -0.011254088, 0.016596796, -0.026439736, -0.028419558, 0.0043808804, -0.020921512, -0.017060159, -0.047628034, -0.049930803, -0.016582755, 0.007498045, -0.04246084, 0.024165047, -0.016582755, -0.064983055, -0.007157544, 0.0072242403, 0.0049811453, -0.04465128, 0.015290957, 0.016835498, 0.0047705257, 0.03105932, 0.032238785, -0.010692437, 0.008600286, -0.0013856991, -0.02035986, 0.0122510195, 0.013921932, -0.00093286776, -0.044679362, 0.017312903, -0.044286206, -0.009253206, 0.0033277841, 0.012707361, -0.0042194054, -0.012651196, -0.025190063, 0.011864884, -0.0047670156, -0.007757809, -0.021749947, -0.02702947, 0.043752637, 0.02406676, -0.008551141, 0.040073823, -0.020710893, -0.0155437, -0.0024712658, 0.03872586, -0.029233951, -0.00145064, 0.033558667, -0.0043738596, 0.017481398, 0.016147476, -0.03240728, 0.0023326082, -0.030862741, -0.0115138525, -0.0023993042, -0.003020631, -0.03763064, -0.0005932442, 0.022831127, -0.0027204987, 0.02811065, 0.02461437, -0.015698155, 0.014616976, -0.007694623, -0.0151084205, 0.002888994, 0.007842056, 0.0055322656, -0.024965402, -0.0050373105, 0.04616774, -0.027183924, -0.011022408, -0.011296213, -0.011787658, 0.007982469, -0.05768159, 0.00084028306, 0.010102703, 0.005507693, -0.010678396, -0.015894733, -0.021637617, -0.0052268677, -0.028868878, -0.0033663977, -0.023392778, -0.014364232, 0.035608694, -0.049060244, -0.006413356, 0.00027358564, 0.018183462, 0.013142641, -0.019826291, 0.033979904, -0.01969992, -0.02812469, 0.0133111365, -0.009337453, -0.02670652, 0.024558203, 0.024221214, 0.011155799, 0.021426998, 0.0007196158, -0.030357255, 0.013170724, 0.01871703, 0.013065414, -0.029964099, 0.023477025, -0.011015386, -0.025091773, -0.01913827, 0.007055745, 0.0027169883, 0.005918401, -0.0032909256, -0.01882936, -0.026495902, -0.035018962, 0.043134823, 0.010348425, -0.011534914, -0.012700341, 0.016105352, 0.014574852, -0.017242696, -0.032744274, -0.030946989, -0.027324338, -0.040579308, -0.0475157, -0.029514777, 0.018450245, 0.006150082, 0.030609997, -0.0077086645, -0.02725413, -0.08458469, 0.032856602, -0.0028907491, 0.0323792, 0.0065397276, -0.023687644, 0.003533138, -0.04650473, -0.016175559, 0.050211627, 0.01378152, -0.0038578426, -0.03777105, -0.047094464, -0.010643292, 0.0036717956, 0.0042650397, -0.0026853953, 0.012321226, -0.019854374, -0.005483121, -0.014251903, -1.0750357e-05, -0.000221589, -0.007210199, -0.007926304, 0.01247568, 0.05768159, -0.0052022953, -0.016456384, 0.002255381, 0.00013350189, -0.012791609, -0.030048346, -0.023294488, -0.008838988, 0.0027626224, 0.06762282, 0.019615673, -0.038922437, -0.030918906, 0.0024923277, 0.010776685, -0.0009653382, -0.012377391, 0.030666163, 0.005044331, -0.0011724471, -0.01017291, -0.044510867, -0.025948292, -0.001641075, -0.025428765, 0.008628368, 0.013233909, 0.018239627, -0.013669189, -0.0023887733, -0.031817548, 0.009154917, 0.014743348, 0.006760878, -0.011176862, -0.030441502, -0.0017735895, 0.023041746, -0.034457307, 0.015220751, -0.017972842, -0.039203264, 0.03406415, -0.01313562, 0.007364653, -0.01203338, 0.0021886851, 0.015164586, 0.02121638, 0.017565645, 0.0033400701, -0.055266492, -0.013332198, 0.013261992, 0.0068767183, -0.003547179, 0.025948292, 0.0013979853, 0.026453778, -0.04552184, -0.018099215, -0.0017990394, 0.008487956, 0.017790306, -0.0020482722, 0.0010855667, -0.0029521799, -0.037237484, 0.011696389, 0.01565603, -0.022803044, 0.057513095, -0.0063326187, -0.021300627, 0.04321907, -0.034176484, 0.028166814, 0.0022378296, 0.038051877, -0.01302329, -0.027450709, 0.013360281, -0.00048968976, 0.0037139193, -0.024361625, 0.006378253, 0.0017077711, 0.0057604364, -0.003162799, -0.02277496, 0.0037139193, -0.06020902, -0.006869698, -0.0006432663, 0.016105352, -0.01641426, -0.002945159, 0.025625343, -0.010495859, -0.120867364, 0.0016779334, 0.021721864, -0.03347442, -0.015922815, -0.00033326106, 0.06807214, -0.0013137376, -0.006006159, 0.029711355, -0.0004218967, -0.006301026, -0.0032698638, -0.023154076, 0.026453778, -0.038613528, -0.023701686, -0.022311598, -0.024768824, -0.01630193, 0.0036928575, -0.014841637, 0.005430466, -0.03653542, -0.058467902, -0.0057639466, -0.028503805, -0.022957498, 0.01913827, 0.02659419, -0.017888594, -0.025920209, 0.013044352, 0.0028100119, 0.0002795093, 0.033895656, 0.0006476542, 0.0067257746, 0.024263337, -0.012728423, 0.012356329, -0.054704838, 0.033895656, -0.0023115461, 0.052739058, -0.04279783, 0.030160677, -0.0012163261, 0.004528314, 0.009555093, -0.0056445957, -0.010811788, -0.002429142, 0.019938622, 0.022058856, -0.008480935, 0.009330433, -0.012721403, -0.017958801, 0.045999244, 0.008024593, 0.014434439, -0.004289612, -0.012925001, -0.005876277, -0.02419313, 0.007891201, 0.038950518, -0.022817085, -0.04246084, 0.00021160653, 0.023224283, -0.018000925, -0.0069469246, 0.0075261276, -0.0061535924, 0.021735907, 0.008249254, -0.033305924, 0.03777105, 0.047094464, 0.0054480177, -0.03347442, 0.02857401, 0.001827122, 0.04872325, -0.020444108, -0.014673141, -0.026313365, 0.0378553, 0.04967806, -0.046897884, -0.0028854837, 0.022423929, -0.03743406, -0.023392778, 0.002464245, 0.016498508, 0.03743406, -0.00941468, 0.0022150124, 0.02604658, -0.029936016, -0.023041746, -0.018071132, -0.019292723, -0.013711313, -0.011773616, -0.009604238, -0.0034629314, 0.023715727, 0.031649053, -0.028714424, 0.0120544415, -0.016512549, 0.021497205, 0.0076876027, 0.0030030794, -0.0025835962, -0.0120544415, -0.018688947, 0.012616093, 0.005374301, 0.014673141, -0.008193089, 0.013465591, -0.0067222645, 0.0019043491, 0.006880229, -0.0034436246, -0.008284357, -0.0058411737, 0.0039456007, 0.0010513411, -0.017383108, -0.0049390215, 0.013444529, 0.0012110607, -0.03709707, 0.03959642, 0.0011250578, -0.012981166, 0.041646447, -0.021286584, -0.033530585, -0.01280565, -0.029065456, -0.015950898, 0.029570943, -0.0038227392, 0.02264859, 0.011071552, -0.012847774, 0.009534031, -0.010229074, 0.041169044, 0.029711355, -0.02023349, 0.016681043, 0.06874612, -0.0022132574, 0.023561273, 0.014420398, -0.0047986084, 0.032014124, -0.011850843, 0.008115862, 0.005609493, 0.017706059, -0.016358094, -0.046336234, -0.011226006, 0.004215895, 0.032547694, 0.04330332, 0.02385614, 0.009470846, -0.008024593, 0.0024045696, 0.03176138, 0.01717249, 0.010966242, -0.043499894, 0.04046698, 0.035917602, 0.0055498173, -0.0010732806, -0.012770547, -0.00039951838, 0.021932485, -0.02746475, 0.032968935, 0.0155998655, 0.019306764, -0.041646447, 0.00061869406, 0.012882877, 0.01181574, 0.0068381047, -0.007940345, -0.04552184, -0.0049706143, 0.017214613, -0.0065151555, 0.00069767627, 0.038332704, -0.017355027, 0.0053883423, 0.026074663, -0.0076173963, -0.020935554, 0.0076244166, -0.025091773, 0.03510321, -0.010762643, 0.013086475, 0.008024593, 0.0062237987, 0.025260268, -0.0045739478, -0.054704838, -0.025330475, -0.0011847332, 0.004889877, 0.0020868857, 0.007568252, 0.038613528, -0.006490583, 0.015178627, 0.012447597, -0.030413419, -0.034513474, -0.019742044, -0.0013154927, -0.010004414, 0.028251061, -0.027843865, -0.008494976, 0.008740699, 0.013613024, 0.04378072, 0.022971539, -0.027380502, -0.015094379, -0.016442342, -0.019952664, 0.0011899987, 0.04891983, -0.005044331, 0.017663933, -0.0060272207, 0.020275613, 0.04423004, -0.023701686, 0.004917959, -0.036395006, 0.02854593, 0.045774583, 0.02549897, 0.009175979, -0.0022922396, 0.045128684, 0.01279863, -0.038585447, 0.011731492, -0.00787716, -0.0071259513, 0.0001284558, 0.027057553, -0.0044897003, -0.016512549, 0.032660026, -0.0033190083, 0.021090006, -0.006862677, -0.010818808, 0.004738933, -0.03619843, 0.0105028795, -0.012482701, -0.014925884, 0.0011724471, -0.032856602, -0.011127717, 0.004236957, 0.0013742906, -0.011914029, 0.006227309, -0.010011435, -0.0034857485, -0.0119561525, 0.025091773, 0.01958759, 0.0060202, -0.0042966325, 0.006750347, 0.014841637, -0.0038683736, 0.015192668, -0.0347943, -0.0041316478, 0.008551141, 0.024136966, -0.017663933, 0.042741667, 0.021104049, -0.0011794678, -0.05175617, 0.0025748203, -0.01784647, -0.0247267, 0.0053462186, -0.02627124, 0.030020263, -0.02363148, 0.004619582, -0.037518308, -0.03423265, -0.021581452, -0.017018035, 0.0029188318, -0.0028100119, -0.011085593, 0.0177201, 0.020486232, -0.030694244, -0.008621348, 0.0028328288, -0.012138689, 0.0032136987, -0.0081650065, -0.010776685, 0.019826291, -0.013858747, -0.024122924, 0.028840795, -0.028840795, -0.01729886, -0.02287325, -0.011914029, 0.0061781644, 0.031705216, 0.005504183, 0.0023133014, 0.033418253, 0.01083987, -0.0062975157, 0.011654265, 0.031031236, 0.025400681, 0.07531744, -0.029233951, -0.019728003, -0.018520452, -0.010629252, 0.037602555, -0.004103565, -0.020191366, -0.032126456, 0.030918906, -0.0053637703, 0.016063228, 0.012595031, -0.012496742, 0.01083285, 0.022437971, -0.0058271326, -0.0026186993, 0.011099635, 0.004738933, 0.00025362067, 0.06874612, -0.010594148, 0.010636272, 0.003921028, 0.01979821, 0.0067433263, 0.026678437, 0.0026976815, -0.0009925432, -0.0118578635, -0.027450709, -0.012117627, 0.004430025, -0.020191366, 0.028040443, -0.011408542, -0.007940345, -0.03777105, 0.012960104, 0.004991676, 0.027857905, -0.00284687, 0.0006230819, 0.004114096, -0.006996069, 0.027197964, -0.010362467, 0.0047108503, -0.007066276, -0.01871703, 0.011352377, -0.00678896, -0.01641426, 0.022437971, -0.03381141, -0.0066590784, -0.0012057952, 0.0037068988, -0.021806113, -0.011801698, 0.014869719, -0.019531425, -0.0021500716, 0.01937697, -0.041871108, 0.0029083008, 0.0075471895, 0.010348425, 0.00037538493, -0.02330853, -0.029964099, 0.0019025939, -0.0062764534, 0.0015164586, 0.015824527, -0.013879809, -0.003963152, 0.0039982554, 0.022887291, -0.011113675, -0.030441502, -0.03271619, 0.0008898663, -0.009814857, -0.01871703, 0.03830462, -0.015684113, -0.036451172, 0.036366925, -0.022030773, 0.017930718, 0.032351118, 0.024010595, 0.010025476, -0.02780174, -0.018408122, 0.011548955, 0.0027661328, -0.0032259848, 0.003283905, 0.0008446709, -0.036254592, 0.014869719, -0.00024528368, -0.0050408207, -0.038894355, -0.0005050474, 0.012573969, -0.0090987515, 0.01728482, -0.002362446, 0.0058797873, 0.0053532394, -0.0052373987, 0.025147937, -0.002602903, 0.032772355, -0.018983815, -0.040129986, 0.00459501, 0.00415973, 0.008572203, -0.020851305, 0.028616136, 0.017748183, 0.025442805, 0.0016507284, -0.0044897003, -0.009049607, -0.002057048, 0.034653887, 0.023069829, -0.0013032065, 0.00295569, -0.0015840322, 0.012398453, -0.005026779, -0.020710893, 0.02680481, 0.018983815, 0.005511204, -0.0051917643, -0.023729768, 0.015698155, -0.014181696, 0.04192727, 0.0067538572, -0.014687182, -0.0004686278, -0.0071961577, 0.0062343297, 0.008635389, 0.003643713, 0.015080338, -0.017958801, -0.023505108, -0.009920167, -0.0028082565, 0.0077156853, 0.03339017, 0.0050338, -0.032519612, -0.005514714, 0.02999218, 0.0069644763, -0.026650356, 0.017144406, 0.0076103755, -0.011176862, -0.003152268, -0.003963152, -0.020458149, 0.016540632, -0.026734604, -0.046111573, -0.0002913566, 0.03774297, 0.0012075504, 0.023996552, -0.038922437, -0.0082632955, 0.0006059691, -0.014083407, -0.043190986, 0.0032154538, -0.0033628873, 0.00039096197, -0.0037384916, -0.0055638584, -0.016119393, -0.06908311, 0.011296213, -0.007582293, 0.026313365, 0.020373901, -0.021047883, 0.043752637, 0.013992138, -0.018478328, -0.004521293, 0.001793774, 0.013641107, -0.008347543, -0.017228654, 0.031620968, 0.0009758692, -0.014258923, -0.009428722, 0.008572203, 0.0012391432, -0.009583176, 0.003238271, 0.010846891, 0.010215034, -0.0074137975, 0.006630996, 0.01138046, 0.014813554, -0.02066877, -0.015670072, -0.015908774, 0.01628789, -0.0042509986, -0.035749108, 0.020753017, 0.0011066287, -0.002767888, 0.03793955, 0.0033997456, -0.013198806, -0.04715063, -0.00711191, 0.01805709, -0.010874974, 0.03010451, -0.05231782, -0.023336612, -0.0001772712, 0.028293185, -0.0030346722, 0.012180813, -0.019952664, 0.02527431, 0.0034436246, 0.025260268, 0.0005550695, -0.0413937, 0.0023290978, 0.017340984, 0.030722328, 0.029065456, -0.03875394, -0.017677976, -0.0072523225, -0.038360786, 0.012644175, 0.025541093, -0.0042545088, -0.0013637596, -0.060096692, 0.0037033884, 0.023603396, -0.02702947, 0.005507693, 0.023125993, -0.007947367, 0.028026402, -0.008600286, -0.0010855667, -0.0015252344, -0.00010476114, -0.03403607, -0.029964099, 0.04257317, 0.017986884, -0.00042277426, 0.007399756, -0.022381805, -0.0310874, 0.014392315, -0.018211544, 0.0020008828, -0.016456384, 0.010671375, -0.01390087, 0.009821878, 0.025344515, -0.022943456, 0.00057832536, -0.0020605584, 0.03184563, 0.016344054, 0.0002696365, -0.022901332, 0.009070669, 0.005974566, 0.0039771935, 0.0061114687, 0.015262875, 0.016512549, -0.025133897, 0.027015429, -0.004795098, 0.020458149, 0.035018962, -0.0067678983, 0.04257317, -0.02353319, -0.0060482826, 0.022030773, -0.012917981, -0.020107117, 0.01488376, -0.001979821, -0.0030487136, -0.017467355, -0.01082583, -0.009161937, -0.0003808698, -0.012454618, -0.0018762664, 0.004408963, 0.020724934, -0.0069012907, 0.003380439, -0.003338315, 0.04869517, -0.00678194, 0.012861815, 0.00017496756, 0.045465674, 0.026369529, 0.0035541998, 0.007069786, -0.014125531, -0.01432913, -0.014588893, 0.01784647, -0.0068240636, -0.013261992, 0.00919002, 0.004236957, -0.020458149, -0.0046581957, 0.02954286, -0.032014124, -0.0068521462, -0.0071259513, 0.0038613528, -0.0075471895, -0.00078675064, 0.02111809, -0.019433135, 0.012770547, 0.030301088, 0.026186993, 0.00831946, 0.018323874, -0.05318838, -0.02210098, -0.018745113, 0.016456384, 0.021918442, 0.019475259, -0.00087801897, -0.0085090175, -0.054761004, -0.016456384, 0.0060693445, -0.007280405, 0.027506873, 0.044904023, 0.009351495, -0.012707361, 0.0023940387, -0.019025939, -0.028194897, 0.025527053, -0.036984738, 0.035861436, -0.016891664, -0.009934207, -0.02176399, -0.013767478, -0.024684576, -0.015571783, 0.006097427, 0.0054620593], "content_hash": "89478f3781f96fb856690c0490fe95fc"}, {"content": "# GPT-2 Overview\n\n## What Is GPT-2?\n\nGPT-2 (Generative Pre-trained Transformer 2) is a language model created by OpenAI in 2019. It was one of the first models to demonstrate that scaling up transformers could produce impressively fluent text. It remains one of the most well-studied and accessible models for learning about transformer internals.\n\n## Architecture Details\n\n| Property | Value |\n|----------|-------|\n| Parameters | ~124 million (small variant) |\n| Layers | 12 |\n| Attention Heads | 12 per layer (144 total) |\n| Hidden Dimension | 768 |\n| MLP Dimension | 3072 (4x hidden) |\n| Vocabulary Size | 50,257 tokens |\n| Positional Encoding | Learned absolute positions |\n| Max Sequence Length | 1024 tokens |\n| Normalization | LayerNorm |\n| Activation Function | GELU |\n\n## Why Start with GPT-2?\n\nGPT-2 small is the **recommended starting model** for learning with this dashboard:\n\n- **Fast**: Small enough to load quickly and run interactively\n- **Well-studied**: More research papers have analyzed GPT-2's internals than almost any other model. Many examples and references use GPT-2.\n- **Clear patterns**: With 12 heads per layer, the attention patterns are easy to visualize and categorize\n- **Manageable size**: 144 total attention heads is small enough to explore systematically\n\n## What to Expect in the Dashboard\n\nWhen analyzing GPT-2, you'll typically see:", "source_file": "gpt2_overview.md", "chunk_index": 0, "embedding": [-0.05763732, 0.011029581, 0.042829685, -0.01631886, 0.021965442, -0.01574483, -0.03505099, 0.03502756, 0.0048851143, 0.061011214, 0.01254666, -0.028490642, -0.04006497, -0.033036023, 0.004709391, -0.042431377, -0.022398893, -0.022469183, 0.019575601, 0.038143724, 0.014561623, -0.034324665, -0.023582099, 0.03345776, 0.013800155, -0.01165047, -0.00070326024, 0.028256346, -0.009295776, -0.01937645, 0.0043433, -0.02647568, -0.057309303, -0.049764905, 0.0063670496, -0.013612716, 0.027366012, 0.048218537, -0.0019358876, 0.054263428, 0.027319152, -0.005957028, -0.026873987, 0.052904498, -0.033809207, -0.0005128931, 0.014245321, -0.011351741, -0.054450866, 0.04817168, 0.00039647624, 0.048640274, -0.033293754, -0.0015112225, -0.021766288, -0.012710669, -0.033410903, -0.016482867, -0.025140181, -0.025866505, -0.01319098, 0.02757688, 0.022082591, 0.037065953, 0.01629543, -0.013940734, -0.06115179, 0.05515376, -0.03390293, -0.005450358, 0.021016534, 0.0014453261, -0.03704252, -0.021379696, 0.008879896, -0.0053097797, -0.04069757, 0.051170696, -0.025796216, -0.025023032, -0.0056290105, -0.027998617, -0.03760484, -0.029849572, -0.009114194, -0.0043110843, -0.014186746, -0.014948215, -0.0066774944, -0.048312258, -0.0053097797, -0.0124646565, -0.026616259, 0.0037926997, 0.025843075, -0.0016913391, 0.023113502, -0.017853511, -0.01808781, 0.04327485, 0.023617243, 0.037979715, -0.01966932, 0.0032479567, 0.0069235074, -0.0023298012, 0.0056495117, -0.006841503, -0.05435715, -0.049015153, -0.09498443, 0.026639689, 0.029638704, 0.015463672, 0.018743845, -0.0073686736, -0.0129449675, 0.02417956, 0.017818367, -0.051030114, 0.0168226, -0.02635853, 0.0021906868, -0.00020025161, -0.010590272, 0.020301927, -0.0323097, 0.010361832, -0.00091010146, 0.016096275, -0.031677097, 0.047468785, 0.007485823, -0.043907456, -0.018696984, 0.0054181423, -0.049952343, 0.022410609, -0.026171092, 0.0040328554, -0.002530419, -0.036480207, 0.02219974, 0.056559548, -0.095640466, -0.009348492, -0.019821616, 0.047937382, -0.08219176, 0.05543492, 0.0016649805, 0.050795816, -0.019915335, -0.04109588, -0.025960224, -0.040721, 0.014456189, 0.03830773, 0.024671584, -0.04224394, 0.0045629544, 0.0026329244, -0.003681408, 0.007233952, -0.04554754, 0.05871509, -0.025632206, -0.009565217, -0.062932454, -0.06237014, 0.034207515, 0.018966427, -0.020126203, -0.023172077, 0.023535239, 0.0064314813, -0.08720573, -0.009196199, -0.032098833, -0.05482574, -0.025819644, -0.022024017, -0.03376235, -0.009541788, -0.046344154, 0.0019988553, -0.016400864, 0.0276706, 0.017115472, 0.0075443974, 0.023406375, 0.013436994, 0.07684976, 0.022504328, -0.010209537, -0.033129744, 0.06743098, 0.0046888897, 0.0069235074, 0.03713624, 0.016166566, 0.0065369154, 0.019774755, -0.060776915, -0.0045834556, -0.01663516, 0.010894859, 0.011369313, -0.043673158, -0.028303204, 0.013671291, -0.01631886, -0.035824172, -0.010976864, -0.045219522, -0.017806651, -0.04910887, 0.029755853, 0.037768845, 0.02040736, 0.0074038184, 0.053654253, 0.022668336, 0.004961261, -0.024999602, 0.015592535, 0.01749035, -0.025468199, 0.012780959, -0.0015492958, 0.02640539, -0.039924387, -0.013284699, -0.0135189975, -0.0012300648, -0.022363748, 0.04437605, 0.031794246, -0.011140872, -0.017771507, 0.004120717, 0.01436247, -0.022996353, 0.010361832, 0.06213584, -0.016002556, -0.051030114, -0.07057057, 0.06780586, 0.037839133, 0.0035320432, -0.015920553, -0.04093187, 0.025725925, -0.02337123, -0.0012988899, 0.0041119307, 0.016131422, -0.017244337, -0.012441226, 0.008557736, 0.020887671, -0.026991135, -0.000904244, -0.03378578, -0.016682021, -0.030622754, -0.029334115, -0.04812482, -0.051217552, 0.04224394, -0.0077025485, -0.0049378313, 0.036597356, -0.06237014, -0.024695015, 0.014713918, 0.012136639, -0.012101495, 0.03950265, 0.00060807663, -0.0138118705, 0.0480311, 0.039877526, 0.010414548, -0.004454592, 0.0029711921, -0.010613701, 0.006138609, -0.024484146, 0.023008069, -0.0034412525, 0.027764319, -0.0198099, -0.020583084, 0.020126203, 0.013612716, 0.030763334, 0.01013339, -0.011357598, -0.007749408, -0.05037408, 0.017654357, -0.002753002, 0.015393382, 0.042595387, 0.0069586523, 0.050327223, -0.010490695, 0.027178574, -0.02769403, -0.028514072, 0.0026182807, 0.016623447, 0.022832345, 0.036855083, -0.002017892, -0.014526479, 0.006191326, -0.04090844, -0.03711281, -0.035683595, 0.014913071, 0.00023869114, 0.025444768, 0.038659178, -0.00420565, 0.010279827, 0.026709978, -0.0074741077, 0.033223465, -0.015147369, -0.04097873, 0.014795922, -0.022422323, 0.0052482765, 0.0050432654, 0.019013287, -0.02870151, 0.028350065, -0.030646184, -0.019294444, -0.03376235, -0.0010828963, -0.057262443, 0.057262443, 0.007848985, 0.035683595, 0.04224394, -0.021543706, 0.0105727, -0.025116751, -0.015662825, 0.018696984, 0.020020768, 0.026264811, 0.01087143, -0.0055967947, 0.02525733, 0.0069235074, -0.033996645, -0.007848985, 0.020559654, 0.009547645, -0.05145185, 0.020653373, -0.018896138, 0.03823744, -0.011919913, -0.004929045, 0.01560425, -0.001041162, -0.0025626349, 0.0036228336, -0.03950265, -0.002753002, -0.003915706, 0.008704172, -0.02886552, 0.023710962, 0.023664104, 0.034957267, 0.0031161641, -0.0036404058, -0.0015463671, 0.043509148, 0.017279482, -0.007151948, -0.026171092, 0.037839133, 0.008792034, -0.007409676, -0.029334115, 0.018099524, -0.015498816, 0.034207515, -0.011750047, 0.00034632182, 0.04322799, 0.047398496, -0.011492319, -0.010713278, -0.03003701, -0.023535239, 0.010865572, -0.011299023, 0.0006110054, 0.057590462, 0.022375464, -0.0055323625, 0.054966323, -0.0076674037, -0.00721638, -0.0051721293, 0.011568466, 0.008153572, 0.01443276, 0.042267367, -0.01199606, 0.034207515, -0.01261695, -0.001707447, -0.047421925, -0.00055243087, -0.052201606, 0.01615485, 0.02999015, 0.00093865657, -0.038729466, -0.025796216, 0.0067653563, 0.008012993, 0.026686547, -0.042689104, 0.008891611, 0.00059123646, 0.03711281, 0.027881468, 0.0033680345, 0.023218937, 0.022070875, -0.0385186, -0.004577598, -0.023710962, -0.035004128, -0.005066695, -0.028420353, -0.046062995, -0.007620544, -0.025140181, 0.030083869, 0.06743098, 0.001723555, -0.025304189, -0.003895205, -0.03711281, 0.03856546, 0.033364043, 0.074928515, 0.0015815118, 0.035683595, -0.008633883, -0.022656621, -0.037065953, -0.01562768, -0.0012966932, 0.021895153, 0.0023839828, -0.007866557, -0.025889935, 0.03127879, -0.020501079, 0.014444475, -0.002086717, -0.013647862, 0.02406241, 0.03104449, 0.001259352, -0.01914215, -0.041986212, 0.02229346, 0.032333132, 0.01794723, 0.024577865, -0.01268724, 0.09123566, -0.036691073, 0.011515749, 0.032145694, -0.017103758, -0.023874972, -0.025819644, 0.044001173, -0.010379404, -0.008874038, 0.00011147462, -0.020372216, -0.0721638, -0.027272293, 0.021203974, 0.026663117, -0.023535239, -0.008926756, 0.060448896, 0.024812164, 0.0030341598, 0.015323092, -0.008012993, -0.006823931, 0.009237201, -0.05140499, 0.0005015443, -0.0072808117, -0.0019842116, -0.008674885, 0.035871033, 0.04053356, 0.028092336, 0.024390427, 0.011966773, 0.00082004315, 0.017548924, -0.03943236, -0.04423547, -0.0038688465, -0.014081312, -0.027764319, -0.061714105, -0.0287718, -0.016881175, 0.025140181, -0.013577572, 0.0011356133, -0.0014416652, -0.087908626, 0.021403126, 0.011632898, -0.028350065, -0.049999204, 0.009746799, 0.009377779, 0.02509332, 0.005040337, -0.014526479, 0.006402194, 0.018860994, 0.011439602, -0.00983466, 0.009518358, 0.059886582, 0.06026146, -0.0314428, 0.03966666, -0.010514125, -0.031841107, -0.031723958, 0.015979126, -0.005283421, 0.0023708034, -0.009887378, 0.010748423, 0.006946937, 0.03624591, -0.017115472, -0.0018099524, 0.020594798, 0.009237201, -0.014667057, 0.011574323, -0.020723663, -0.0008793499, 0.00017984206, 0.009729226, -0.013390133, 0.0005582883, -0.019868474, 0.008428873, 0.021836579, 0.0046947473, 0.030786764, -0.03219255, -0.023722678, 0.0005227775, 0.035589874, -0.028584363, 0.029498124, 0.006097607, 0.008270721, -0.009869806, 0.037347108, 0.043720014, -0.01691632, -0.00074279803, 0.03116164, -0.023664104, -0.02392183, 0.035425864, -0.019505313, -0.055997234, 0.014116458, -0.003403179, -0.0057900907, -0.037745416, -0.013741581, -0.006613062, 0.0077259783, -0.010209537, 0.01806438, -0.020501079, -0.006777071, -0.0072280946, 0.0002441825, -0.004463378, -0.0138353, -0.053326234, 0.01620171, -0.016307143, -0.0010272505, 0.042782824, 0.002118933, 0.038659178, -0.026311671, -0.015803404, -0.029873, -0.0062499004, 0.0057490882, -0.01615485, -0.0143859, -0.020313641, 0.016834315, -0.01686946, 0.020114487, 0.094562694, 0.018931283, -0.0066247773, 0.009506644, 0.020114487, -0.008926756, 0.018146385, -0.020969676, -0.012675525, -0.018966427, -0.014585054, -0.015194229, 0.0005473056, -0.010918289, -0.020747092, 0.022117736, -0.021192258, -0.0044223755, -0.03612876, -0.025960224, 0.054685164, 0.0056085098, 0.012417797, 0.00393035, 0.00014222624, -0.03479326, -0.025749356, 0.040205546, -0.058105916, 0.008071568, -0.010467266, -0.0312085, -0.014081312, 0.02741287, 0.029544985, 0.011433745, -0.00077684445, -0.024296708, -0.03610533, 0.0063787643, -0.0078607, -0.0115040345, -0.0043696584, -0.014889641, -0.008270721, -0.051311273, 0.00930749, 0.031700525, 0.006244043, 0.010783568, -0.041306745, 0.031841107, 0.020208206, 0.010068959, -0.008669028, -0.0033270323, 0.03125536, -0.0012534945, -0.01995048, -0.020747092, 0.022469183, -0.003915706, -0.024390427, -0.015733114, -0.0034236803, 0.06602519, -0.011281451, -0.057215583, -0.03950265, 0.0034236803, -0.011732475, 0.021332838, -0.0052219178, 0.03327032, -0.019657606, 0.035683595, 0.0055762934, -0.03603504, -0.025491627, 0.008844751, 0.0059746, 0.007737693, -0.017712932, 0.005037408, -0.011263879, -0.0016195853, 0.0052658487, 0.0041470756, -0.026311671, 0.028045477, 0.01562768, 0.019200725, 0.005582151, -0.02989643, 0.009781944, -0.010555127, -0.04943689, 0.0075502545, 0.021215688, 0.009664794, -0.008587023, -0.014948215, 0.0033592484, 0.020817382, -0.011632898, 0.008704172, -0.035496153, -0.007907559, 0.0002465621, 0.027084855, -0.013753296, 0.013272985, -0.008838894, -0.010625416, 0.0020940388, 0.022082591, 0.008546022, -0.057121865, -0.002477702, 0.0077084056, -0.004738678, -0.003212812, 0.004741607, 0.03373892, 0.009125909, 0.008341011, -0.015264518, 0.023511808, 0.01507708, -0.019610746, -0.007140233, 0.018216673, -0.019868474, -0.03015416, 0.004952475, 0.020899385, -0.0072105224, 0.0108245695, 0.003491041, -0.007848985, 0.016389148, -0.017080327, 0.0074565355, -0.01050241, 0.024577865, -0.02284406, 0.031091351, 0.018896138, -0.018837564, 0.013952449, 0.014749062, 0.008264864, -0.014807637, -0.012874678, -0.013729866, 0.008979472, -0.014339041, 0.006525201, -0.022281744, 0.014327326, 0.030997632, 0.014128173, -0.008920899, -0.023277512, -0.011550894, -0.10159163, -0.02872494, -0.016049417, -0.02521047, 0.00344711, -0.008680743, 0.017701218, -0.014737347, 0.0026065658, -0.051873587, 0.0030195161, -0.046414442, 0.0055352915, 0.0071343756, 0.028068906, -0.042618815, 0.003704838, 0.0038600604, 0.046227004, -0.030224448, -0.014866211, 0.003040017, 0.013729866, -0.0059072394, -0.03509785, -0.008282436, 0.024413858, 0.007661546, 0.0023254082, 0.008727603, -0.038050003, 0.049483746, 0.0067477836, 0.02038393, -0.017677788, 0.01620171, -0.020126203, -0.023804681, 0.03355148, 0.016236855, -0.003022445, -0.025140181, -0.01326127, -0.0022390108, 0.055762935, -0.008786177, 0.036152188, -0.0072222375, -0.014350755, 0.01985676, 0.034957267, 0.02640539, 0.004026998, 0.026194522, 0.013472138, 0.00415879, 0.013495567, -0.048218537, 0.038096864, 0.006413909, 0.0063026175, 0.03966666, 0.015779974, 0.0032303843, -0.002555313, 0.009787801, 0.023230651, 0.021801433, -0.024366997, -0.018029235, -0.028795231, 0.010560985, 0.004855827, -0.030575896, -0.019622462, 0.010625416, 0.009881521, 0.044563487, -0.009448069, 0.0017499136, 0.008428873, -0.0011407386, -0.016658591, 0.0073511014, 0.044212043, 0.028232915, -0.00602146, -0.007679119, 0.012488086, 0.008118427, 0.02640539, 0.0051926305, -0.008007136, 0.0030722332, -0.0040211403, -0.0024235204, 0.019645892, 0.03983067, 0.034722973, -0.025796216, 0.059933443, 0.010578557, -0.05899625, -0.01691632, -0.0139055895, 0.006349477, 0.006777071, 0.0036872656, -0.029662132, -0.031677097, -0.007825555, 0.014268751, 0.0057490882, 0.024929313, -0.027623741, 0.035871033, 0.08069225, 0.014327326, 0.017642643, -0.0068005007, -0.004832397, -0.005564579, 0.012324077, 0.02160228, -0.04721106, 0.0018202029, 0.016494583, 0.07150777, 0.0030575895, -0.00037194815, 0.008762747, 0.015569106, 0.006308475, -0.0276706, -0.001787987, 0.0062381853, 0.021684285, -0.007679119, -0.005869166, 0.012007775, -0.033246893, 0.003180596, -0.013319844, -0.008405442, -0.0052482765, -0.027740888, -0.0004612743, -0.05145185, 0.0027544664, 0.004577598, 0.05407599, 0.042314228, -0.017759793, 0.051873587, 0.007737693, -0.0038454167, -0.026709978, -0.019997338, -0.03622248, 0.0024264492, 0.031419367, 0.03125536, 0.03394979, -0.00086543843, 0.021039965, 0.011568466, -0.009155196, 0.026920846, -0.015147369, -0.024320137, -0.030318167, 0.050092924, 0.040299263, 0.032098833, 0.011275593, 0.004823611, -0.0066774944, 0.017818367, -0.00031428886, 0.015873693, 0.004437019, 0.03132565, 0.02171943, 0.012628665, 0.012300648, -0.025819644, 0.012488086, -0.017092044, -0.039104346, 0.03493384, -0.008276579, 0.011509892, -0.012581806, 0.014467904, 0.00027310365, -0.02654597, 0.029474694, -0.006027317, -0.019458452, -0.006220613, 3.9537797e-05, 0.013530713, 0.0038454167, -0.026147662, -0.0105434125, 0.04444634, 0.011773477, -0.00075780775, -0.00926063, -0.024531005, 0.005339067, 0.023910116, 0.0017015896, -0.012851248, 0.0013296414, -0.0014951144, 0.022082591, 0.010789425, -0.026077373, -0.02215288, -0.07103917, -0.023792967, -0.011738332, -0.028443784, -0.0076556886, 0.032614287, -0.01849783, 0.028115766, -0.00077391573, 0.013155836, 0.0038922764, -0.011685615, 0.008774462, 0.010338401, 0.018427541, 0.03357491, 0.017185763, 0.008786177, 0.0050022635, 0.0029199393, 0.029708993, 0.008440588, 0.0021848292, -0.0054708594, -0.021977156, -0.0116739, 0.027342582, 0.0054152138, -0.009711654, -0.017279482, 0.0116739, 0.022410609, 0.0110237235, -0.016881175, 0.01562768, 0.030435316, 0.024366997, 0.019763041, 0.011410315, 0.0013970021, 0.009477356, 0.054966323, 0.03573045, -0.016693735, 0.021778004, -0.004961261, -0.015943982, -0.00843473, 0.040439844, 0.029404406, -0.030388458, 0.023101788, -0.010139248, -0.011439602, 0.007913416, -0.02870151, 0.028209485, 0.0021980086, 0.026709978, 0.003063447, -0.025585348, -0.009547645, 0.010836285, -0.020137917, -0.006525201, 0.008147715, -0.03484012, -0.0048821857, -0.009178626, -0.016928034, -0.009951809, 0.010531697, 0.0067594987, -0.023828112, -0.013331559, 0.008329296, 0.0056319395, -0.009711654, -0.0012783888, -0.032731436, 0.0054591447, -0.009061477, 0.02649911, -0.048687134, -0.004234937, 0.02509332, 0.016904604, -0.044774357, 0.013741581, 0.026709978, -0.026827127, -0.028279774, -0.009776087, 0.03605847, 0.038823187, 0.030365027, -0.011855481, -0.029544985, 0.010719135, -0.0111174425, 0.0078431275, 0.00037890387, 0.005295136, -0.012370937, 0.0019871404, -0.02171943, 0.021356268, 0.02399212, 0.019388163, 0.00049349025, 0.028514072, 0.0119784875, 0.0009855161, 0.02157885, -0.04683618, -0.030271308, -0.008200432, -0.03130222, 0.019470168, -0.04458692, 0.016810885, -0.0014702203, -0.036363058, -0.019517027, 0.009992812, 0.0041031446, 0.018427541, -0.025140181, 0.021473417, 0.023218937, 0.047468785, -0.03123193, -0.034348093, 0.0051838444, -0.013952449, 0.035285287, -0.022410609, -0.0039537796, -0.013026971, 0.0038073433, 0.020665089, 0.010162678, -0.024109269, -0.007889987, 0.032005113, 0.0039039913, -0.02406241, -0.021883437, 0.035168137, 0.00276911, 0.014678773, 0.024976172, -0.03622248, 0.0025040605, -0.0029375118, 0.021215688, 0.028584363, -0.024132699, 0.04444634, 0.02109854, -0.012406082, -0.07539711, -0.0033826781, 0.039924387, -0.006946937, 0.019165581, 0.002590458, 0.006179611, -0.05304508, 0.0062499004, 0.0064959135, -0.021871723, 0.007503395, -0.017010039, -0.021496845, 0.009155196, 0.0099635245, -0.0044135894, 0.017302912, -0.03017759, -0.023125218, 0.0020471793, -0.01620171, -0.014854496, 0.033082884, -0.03954951, 0.016283715, 0.02098139, -0.0052541336, -0.0025816716, -0.04566469, 0.017701218, -0.024390427, 0.00063150644, -0.010285684, -0.022972925, -0.020231636, 0.013015256, -0.0029316545, -0.0011070583, -0.009500786, -0.010186108, -0.0040855724, -0.018275248, 0.034113795, 0.0028584362, -0.0043608723, -0.01312069, -0.0045717405, 0.015885407, -0.031770814, -0.006507628, -0.007005512, 0.014901356, -0.008704172, -0.020946246, 0.011779334, -0.051076975, -0.050936397, 0.030739903, 0.0138118705, 0.022609761, 0.024812164, 0.03994782, 0.009325062, 0.008774462, -0.02649911, -0.0074331057, 0.008235577, 0.013647862, 0.012921537, 0.0076674037, -0.0173732, -0.005614367, -0.009471498, -0.0023180863, -0.0023927689, -0.0013098725, 0.007128518, -0.0068473606, -0.0042407946, 0.005476717, 0.027740888, 0.008762747, 0.020208206, 0.010971006, 0.006349477, 0.06977396, 0.0063201897, -0.036386486, 0.042665675, -0.0054708594, 0.029170107, -0.00039940496, -0.0031952397, -0.006308475, 0.02640539, -0.011662185, -0.051592432, -0.0047035334, -0.008218004, 0.008024708, 0.01266381, -0.03505099, 0.0101568205, 0.042829685, 0.015088795, -0.011006151, -0.028959239, 0.0029873, 0.014011024, -0.010191965, 0.013729866, -0.0035144708, 0.028467214, 0.0056553693, 0.011837909, -0.024788734, -0.037885994, -0.018146385, -0.022434039, -0.019165581, 0.0028906523, -0.0015287948, -0.017279482, -0.06358849, -0.015920553, -0.008967758, -0.004735749, 0.0033768206, -0.0070113693, 0.032005113, -0.012511516, 0.021590564, -0.005037408, 8.168399e-05, 0.014760777, 0.023886686, -0.0116739, 0.028350065, 0.014467904, -0.0032743153, 0.002821827, 0.02222317, 0.03357491, 0.0013530712, 0.029568413, -0.0016576587, -0.0347464, 0.02348838, -0.001331838, 0.018708698, -0.04076786, -0.029662132, 3.592265e-05, 0.03458239, -0.0048411833, -0.0027632527, 0.0025582418, 0.0054620733, -0.019306159, -0.058152776, -0.0066657793, 0.021215688, -0.026733408, 0.007040656, 0.0051486995, 0.01500679, 0.0131324055, -0.025772786, -0.017724648, -0.019927049, -0.009711654, -0.0037956284, -0.036644217, 0.0358476, -0.013659576, 0.00060807663, 0.0061971834, 0.00012556912, -0.016810885, -0.028420353, -0.03994782, 0.010291542, 0.042853113, -0.02098139, -0.016541442, -0.023359515, 0.007837269, -0.0064841984, 0.013647862, -0.008868181, 0.022410609, -0.007907559, -0.0056290105, 0.008153572, -0.030435316, -0.021731144, 0.002001784, 0.012441226, 0.004507309, -0.029380975, 0.0469299, 0.017748076, -0.021543706, -0.0041353605, -0.0071870927, -0.0046068854, 0.0075092525, 0.009073192, -0.02420299, 0.029427836, -0.00983466, 0.01937645, -0.0062088985, 0.041447327, -2.193112e-05, 0.00018597407, -0.062276423, 0.00753854, 0.010385261, 0.00043601403, -0.001799702, -0.022445753, -0.043579437, -0.008628026, -0.041822203, -0.008967758, 0.022562902, 0.00452781, -0.037698556, -0.016025987, 0.008610453, -0.016014272, 0.009852233, 0.029380975, -0.00093499565, -0.040228974, -0.01749035, 0.019610746, 0.0066657793, -0.03840145, 0.0034207515, -0.0012688704, -0.0035701166, 0.031630237, -0.01087143, 0.009202056, 0.003652121, -0.010039671, -0.006413909, 0.0004993477, -0.0017616284, -0.020337071, 0.032544, -0.0005370551, 0.0069059352, -0.007966134, 0.032614287, 0.009489071, -0.051967308, 0.03017759, -0.0068005007, -0.007040656, 0.015791688, 0.020032484, -0.017478634, -0.009940094, -0.020829096, 0.06668123, -0.000464203, 0.0047679655, 0.023230651, -0.010115818, 0.023617243, 0.0030605183, -0.013296414, 0.019036716, -0.027811179, -0.02052451, -0.009254773, -0.032661147, -0.01326127, -0.018907852, -0.009887378, -0.0012344578, -0.0029155463, 0.00962965, 0.00095110363, 0.048265398, -0.008920899, 0.010660561, 0.017244337, 0.0010880216, 0.08294151, 0.01087143, 0.013026971, 0.04573498, 0.018860994, -0.0022961209, 0.02346495, 0.023582099, 0.05651269, 0.028326634, -0.0027793606, -0.016436009, -0.015955698, 0.03722996, 0.05201417, 0.0029536197, -0.014924786, -0.022375464, 0.006718497, -0.01500679, -0.012839533, 0.019107006, 0.010279827, -0.037347108, -0.0034734686, -0.0036872656, 0.027459731, 0.01734977, -0.005066695, 0.005344924, 0.014104743, 0.036456775, 0.0022258314, -0.020079343, -0.031911395, -0.029708993, -0.058152776, 0.00040782505, 0.015569106, -0.007731836, -0.011925771, 0.02874837, 0.009412924, -0.040111825, -0.006138609, -0.03837802, -0.0013494103, 0.015979126, 0.037464257, -0.0074448204, -0.012066349, 0.021590564, -0.025679067, -0.018240103, 0.027436301, 0.016471153, 0.019763041, -0.023933547, -0.030927343, 0.0017381987, 0.012827818, -0.013987593, -0.013272985, 0.021344552, 0.016951464], "content_hash": "684ce36d285d76b89e0dcb331f6dca24"}, {"content": "model. Many examples and references use GPT-2. - **Clear patterns**: With 12 heads per layer, the attention patterns are easy to visualize and categorize - **Manageable size**: 144 total attention heads is small enough to explore systematically ## What to Expect in the Dashboard When analyzing GPT-2, you'll typically see:\n\n- **Tokenization**: GPT-2 uses BPE (Byte-Pair Encoding). Common words are single tokens; rare words get split. Spaces are typically attached to the beginning of the following token.\n- **Embeddings**: 768-dimensional vectors, which capture rich semantic information despite being relatively compact.\n- **Attention patterns**: You'll see a good mix of head categories. Expect several Previous-Token heads (especially in early layers), some First/Positional heads, and a variety of other patterns.\n- **Output**: GPT-2 can produce reasonably coherent text for simple prompts. For factual prompts, it sometimes produces outdated or incorrect facts (it was trained on data from before 2019).\n\n## GPT-2 Variants\n\nThe dashboard includes GPT-2 Small and Medium in the dropdown. Larger variants can be loaded by typing the model ID:\n\n- **GPT-2 Small** (124M params, 12 layers) -- in dropdown as \"GPT-2 (124M)\"\n- **GPT-2 Medium** (355M params, 24 layers) -- in dropdown as \"GPT-2 Medium (355M)\"\n- **GPT-2 Large** (774M params, 36 layers) -- enter `gpt2-large`\n- **GPT-2 XL** (1.5B params, 48 layers) -- enter `gpt2-xl`\n\nComparing GPT-2 Small and Medium is a great way to see how attention heads specialize as models scale up within the same architecture.\n\n## HuggingFace Model IDs\n\n- `gpt2` or `openai-community/gpt2`\n- `gpt2-medium` or `openai-community/gpt2-medium`\n- `gpt2-large` or `openai-community/gpt2-large`\n- `gpt2-xl` or `openai-community/gpt2-xl`", "source_file": "gpt2_overview.md", "chunk_index": 1, "embedding": [-0.014857079, 0.006151759, 0.066289395, -0.024374895, -0.0002379857, -0.0058680302, -0.016740007, 0.050374784, 0.031364944, 0.033763744, 0.0017990994, -0.030358996, -0.02562588, -0.05328945, -0.046118848, -0.027779642, -0.015669575, -0.0011744121, -0.015669575, 0.02850186, -0.024581242, 0.024207236, -0.0016669075, 0.042249817, 0.0104979705, 0.01165868, -0.019229084, 0.015256879, 0.03162288, 0.022298515, 0.0199642, -0.027882816, -0.030926453, -0.041604977, -0.00546823, -0.0061453106, 0.00073471287, 0.059634663, 0.0058357883, 0.06128545, -0.0032935126, 0.0056487853, -0.015231085, 0.044648618, -0.033351045, 0.0075510587, 0.037735946, -0.014869975, -0.06809495, 0.05524976, -0.03025582, 0.044184335, -0.047124796, -0.012393796, -0.008292623, 0.0013049919, -0.041295458, -0.011239535, 0.024581242, -0.02718639, -0.011194396, 0.008511868, 0.04555139, 0.017733058, 0.03317049, -0.006738562, -0.06319417, 0.05073589, -0.02244038, -0.0029549724, 0.02922408, 0.016120963, -0.044339094, -0.041940294, 0.021021735, 0.014341208, -0.023613986, 0.03265462, 0.017655678, -0.048001777, 0.0066224914, -0.039025623, -0.043307353, -0.032396685, -0.013709266, -0.012471177, -0.010452832, -0.034666516, -0.037632775, -0.025548501, -0.008376452, -0.030668518, -0.028940352, -0.0046170433, 0.02044138, -0.010162654, 0.039670464, -0.015721163, -0.024168545, 0.04686686, 0.01752671, 0.03644627, -0.02623203, 0.017423537, 0.0121487565, -0.030668518, 0.0024552224, 0.003620768, -0.0427141, -0.026618933, -0.097602755, 0.020467173, -0.0019683694, 0.02667052, 0.0073318137, 0.019461226, -0.034099057, 0.053960085, 0.004758908, -0.043694254, -0.0023746176, -0.003633665, 0.05086486, -0.016456278, 0.015966201, 0.01668842, -0.007847684, 0.0024487742, -0.00974351, 0.0055198176, -0.034950245, 0.03897404, 0.03510501, -0.020067373, -0.00024060535, -0.021627882, -0.029353047, 0.018068375, -0.03776174, -0.021189392, 0.02482628, -0.028450273, 0.019293567, 0.062214017, -0.08161076, 0.008382901, -0.029559396, 0.042224023, -0.10492812, 0.058035463, -0.0034595584, 0.020131858, 0.0096725775, -0.031132802, -0.052360885, -0.047176383, 0.00734471, 0.015940407, 0.024503862, -0.05114859, 0.033041522, 0.0040141195, -0.008073377, 0.02826972, -0.034718104, 0.07124176, -0.01225838, -0.010684974, -0.041553393, -0.06340052, 0.026567345, 0.009911167, -0.0063581076, -0.035156593, 0.012709767, -0.00041309965, -0.08119806, 0.009601645, -0.046505753, -0.04168236, -0.00710612, -0.03265462, -0.006087275, 0.0060389126, -0.052799374, -0.017165601, -0.019254876, 0.004623492, 0.0015008616, 0.014805491, 0.016482072, 0.010942909, 0.07738062, 0.028605035, -0.015824337, -0.048259713, 0.072170325, -0.011194396, 0.047924396, 0.035156593, 0.010504419, 0.038380787, 0.041966088, -0.03680738, 0.02770226, -0.012077825, 0.0038238922, 0.0018394018, -0.026644725, -0.012825837, 0.0074607814, -0.027728055, 0.009427538, 0.008872977, -0.070313185, 0.026051475, -0.07732903, -0.005110345, 0.03964467, 0.058912445, 0.0047331145, 0.036472064, 0.005026516, 0.0037400632, -0.021408638, 0.012845183, 0.0419145, -0.0283471, 0.0076929233, -0.008969704, 0.024761798, -0.006803046, -0.022362998, -0.0042269165, -0.024129856, -0.009891822, 0.042739894, -0.024852075, -0.0065612313, -0.06345211, -0.0017958751, 0.007125465, -0.006854633, 0.008286174, 0.04619623, -0.021769747, -0.031442326, -0.06933303, 0.03350581, 0.032603033, 0.036420476, -0.028243925, -0.04962677, 0.030797485, -0.008879426, 0.016894769, -0.029095111, 0.055198174, -0.039618876, -0.02442648, 0.0050200676, 0.03345422, -0.025200289, 0.010813941, -0.02323998, -0.022272721, -0.035311356, 0.008840736, -0.06092434, -0.05844816, 0.038716104, -0.0033499359, 0.0030968369, 0.03141653, -0.04647996, -0.017694367, -0.00499105, 0.0032983487, -0.03492445, 0.006516093, -0.005413419, -0.0027582967, 0.046118848, 0.027160596, 0.03162288, -0.005026516, 0.01960309, -0.009756406, -0.01964178, -0.019151703, 0.025522707, -0.035981987, 0.015424537, -0.040031575, -0.0119359605, 0.01808127, 0.017874923, 0.02127967, -0.012980599, -0.020273723, 0.009627438, -0.031003835, -0.00686753, -0.029017732, -0.0030436376, 0.027573293, 0.029172493, 0.05875768, -0.015772749, 0.008266829, -0.037787534, -0.04021213, -0.006096948, -0.017797543, 0.006019567, -0.0021199065, -0.008582801, 0.021460224, -0.0010825226, -0.023962198, -0.008615042, -0.023575295, -0.0070803263, 0.025097113, 0.026618933, 0.03760698, -0.019396741, -0.014560453, 0.01512791, -0.018184446, 0.042352993, 0.013083773, -0.044725996, 0.04080538, -0.00622914, -0.007950858, -0.00022186474, 0.022014786, -0.019861026, 0.009311467, -0.023175495, 0.0012332536, -0.036936346, -0.0030549222, -0.051122796, 0.026876869, 0.024104062, 0.034950245, 0.032757796, -0.013502918, -0.0029275666, 0.0024681194, -0.027934402, -0.0042462614, 0.024400689, 0.035698257, 0.009769303, -0.005813219, -0.009878926, -0.0071190167, -0.035182387, 0.014392795, 0.014818388, 0.011368503, -0.006119517, 0.022969147, -0.021202289, 0.005632664, 0.00046186557, -0.008911667, 0.004775029, 0.01640469, 0.009717716, -0.024852075, -0.04163077, 0.0004928984, 0.00086085935, 0.04444227, -0.021782644, 0.022491965, 0.02623203, 0.033763744, 0.016314413, 0.00087295007, 0.0010583411, 0.0431268, 0.012825837, 0.0061904495, -0.070003666, -0.0030194563, -0.006084051, -0.0064999717, -0.008402245, 0.007912168, -0.035956193, 0.013657679, 0.012406693, 0.002060259, 0.025703263, 0.048130743, 0.022092167, 0.005232864, -0.022853076, -0.008640836, 0.023652675, -0.0029178942, 0.014508866, 0.030436376, 0.03038479, -0.009517816, 0.05297993, 0.014882872, -0.0010269053, -0.019267773, -0.0017362277, 0.0037787536, 0.01369637, 0.043307353, 0.01712691, 0.04782122, -0.022994941, -0.044777583, -0.05218033, -0.0050297403, -0.058912445, -0.0074930233, 0.032345098, 0.0029049974, -0.0048362887, -0.026954249, -0.010556006, 0.043487906, 0.04663472, -0.043384735, -0.009679026, -0.0075704036, 0.061233863, 0.035672463, 0.0032499859, 0.0013759241, 0.021653676, 0.0021585969, 0.014986047, 0.0007677608, -0.014792595, -0.010033687, -0.034073267, -0.059789423, 0.0020167322, -0.02747012, 0.016108066, 0.054682303, -0.031597085, 0.009105119, -0.01868742, -0.021976095, 0.0650513, 0.009079326, 0.08016632, -0.034047473, 0.07516237, 0.0015177886, -0.022324309, -0.0037594084, -0.01808127, 0.03309311, 0.015824337, 0.0133868465, 0.0077380617, -0.01880349, 0.03696214, -0.05999577, -0.0017136582, -0.024465172, -0.000787912, -0.009833787, 0.023678469, 0.008821391, -0.025496913, -0.033118904, 0.03270621, 0.011871477, 0.0045848014, 0.029894711, -0.010459281, 0.08898771, -0.035414528, -0.010330313, 0.035801433, 0.01209717, -0.053237867, -0.017733058, 0.028476067, -0.014044582, 0.026206236, 0.008060481, -0.022853076, -0.072686195, -0.041140694, 0.004104397, 0.011265328, -0.004781477, -0.008163655, 0.05094224, -0.012793596, 0.0015008616, 0.037839122, -0.001910334, -0.0045364387, -0.001726555, -0.057674352, -0.016159652, 0.013709266, 0.02826972, 0.0011308854, 0.026489966, 0.036987934, 0.013554505, 0.013361054, 0.0077316132, -0.0023101338, 0.0061098444, -0.055198174, -0.017023737, -0.002131191, -0.023962198, -0.018635832, -0.042739894, -0.022324309, -0.010323864, 0.019551503, -0.038870864, 0.011955305, -0.02958519, -0.070829056, 0.029275667, 0.011175051, -0.019667573, -0.041579187, 0.0045557837, -0.00080886926, 0.019822335, 0.009001945, 0.0016846405, 5.390447e-06, 0.023665572, 0.012251931, -0.011471677, -0.0013380399, 0.06716637, 0.0375296, -0.02044138, 0.042069264, -0.01341264, -0.018635832, -0.01908722, -0.00810562, -0.0076993713, 0.009924064, -0.013992995, 0.026051475, -0.00830552, 0.019370949, -0.023085218, -0.012419589, 0.04021213, 0.008969704, -0.02499394, 0.0299463, -0.03505342, -0.006184001, 0.00365301, 0.009891822, -0.048285507, 0.008582801, 0.0030678192, -0.0014670076, 0.019925509, 0.0004957196, 0.024607036, -0.030281615, -0.00914381, 0.0006267024, 0.019693367, -0.016675523, 0.009285674, 0.0029227305, -0.020505864, -0.027753849, 0.04294624, 0.046531543, -0.01177475, 0.016959252, 0.03017844, -0.027057422, -0.015153704, 0.036188334, -0.036188334, -0.054269608, -0.0032983487, -0.020338206, -0.017513813, -0.01712691, 0.002579354, -0.026464172, 0.028295513, -0.00365301, 0.021382844, -0.014354104, 0.009936961, -0.016507866, -0.0017990994, -0.016740007, -0.026567345, -0.056281503, 0.032732002, -0.014328311, -0.009169603, 0.01573406, 0.013980098, 0.019615987, -0.010143309, -0.0009495247, 0.0016910889, 0.01752671, 0.019899715, -0.0057713045, -0.020467173, -0.0019248428, -0.005010395, -0.02491656, 0.0013017677, 0.08934882, 0.008640836, -0.014586247, 0.006725665, -0.0013033798, 0.00646773, 0.032319304, -0.042249817, -0.017952302, -0.013657679, -0.009047084, -0.018390793, -0.01592751, 0.007983101, -0.0052360883, 0.00592929, -0.011929512, -0.0060389126, -0.018816387, -0.035698257, 0.028218132, 0.02303363, 0.009633887, -0.011007393, 0.010807493, -0.0247489, -0.035362944, 0.054321192, -0.06696003, 0.0145991435, -0.02850186, -0.04539663, -0.017733058, 0.024787592, 0.013760854, 0.0041463114, -0.012671076, -0.011452331, -0.051612873, 0.033995885, 0.04395219, 0.004372005, 0.016894769, -0.021099115, -0.0066740783, -0.053753737, -0.011645783, 0.02048007, -0.004023792, 0.032835174, -0.06319417, 0.02323998, 0.029688364, -0.010252932, 0.0069126687, -0.00750592, 0.025612984, 0.0051329145, -0.0027792538, -0.04204347, 0.014031685, -0.006171104, -0.04067641, 0.0017072099, -0.014715214, 0.05370215, -0.0045848014, -0.042972036, -0.03213875, 0.0010978375, -0.023291565, -0.0019844903, -0.013006392, 0.029404635, -0.013116015, 0.046325196, -0.0021553726, -0.021743953, -0.03270621, 0.012780699, -0.009801545, 0.012742008, -0.009176051, -0.0007605064, -0.018893767, 0.010001445, 0.006519317, -0.011561954, -0.036110956, 0.026257822, 5.914378e-05, 0.03337684, 0.01428962, -0.018171549, -0.0113362605, -0.02223403, -0.031210182, 0.011091222, 0.013103118, 0.03200978, -0.021253876, -0.009930513, -0.006848185, 0.0043010726, -0.0089826, -0.0031387513, -0.012780699, 0.0025648451, 0.010884874, 0.004874979, -0.03476969, -0.0041140695, 0.017346155, -0.0051651564, 0.018197343, 0.006548335, 0.005523042, -0.05721007, -0.0056552333, 0.01033676, 0.0039786533, 0.0013767302, 0.003032353, 0.0012969314, 0.017191393, 0.006783701, -0.0050007226, 0.05233509, 0.033763744, -0.005432764, -0.022259824, 0.018094167, -0.01548902, -0.040237922, -0.00197643, 0.00632909, -0.011916615, 0.012593696, -0.018339206, -0.006538662, 0.016894769, 0.01461204, 0.044545442, -0.0614918, 0.038716104, -0.005597198, 0.014495969, 6.6398214e-05, 0.005800322, 0.0027518482, 0.020235032, -0.010091722, -0.0018958251, 0.009350158, -0.014392795, -0.010091722, -0.021950303, -0.011510367, -0.011987547, 0.0075510587, 0.0017749179, 0.00070932234, -0.0055617318, 0.0040786033, 0.004642837, -0.11163444, -0.029610982, 0.01548902, -0.029688364, -0.009846684, 0.0024407136, 0.052567232, 0.012193896, 0.0038077713, -0.049343042, 0.03082328, -0.06541242, -0.004359108, -7.052921e-05, 0.004775029, -0.042172436, -0.01345133, -0.010375451, 0.03265462, -0.015656678, -0.03577564, 0.020273723, 0.007763855, 0.022878869, -0.052077156, 0.0038013228, 0.0010591472, -0.0065805768, 0.015953304, -0.017114013, -0.042920448, 0.047537494, 0.014070376, 0.018300517, -0.006100172, 0.035801433, -0.0113362605, -0.025535604, 0.039696258, -0.014728111, 0.011046084, -0.027831228, -0.0025261547, -0.026167545, 0.034331203, -0.006648285, 0.014792595, -0.00038307437, -0.009201845, 0.022504862, 0.025548501, 0.032989938, -0.0014299293, 0.0419145, 0.021060424, 0.00081209344, 0.009711267, -0.037864916, 0.02499394, 0.03510501, -0.0002867516, 0.018855076, -0.010807493, 0.0069836006, 0.012825837, 0.0044106953, 0.03242248, 0.026412584, -0.01808127, -0.014367001, -0.028192338, 0.019254876, 0.009588748, -0.019035632, -0.016830284, 0.009130913, 0.0150376335, 0.05524976, 0.01181989, -0.0033596086, 0.015501917, -0.03242248, -0.036033574, 0.0109106675, 0.015347156, 0.020183444, 0.025071321, -0.027134804, 0.01501184, 0.020415585, 0.032319304, 0.0061098444, -0.0031935626, 0.021460224, -0.025277669, 0.0052522095, -0.0041688806, 0.03644627, 0.042120848, -0.0025906386, 0.057932287, 0.011748957, -0.04846606, -0.0068997717, -0.03373795, -0.00207638, 0.014392795, -0.0069191167, -0.00405281, -0.021795541, 0.011310467, 0.04488076, 0.03017844, 0.012941908, -0.02187292, 0.023859024, 0.072686195, -0.00045985045, 0.007621991, 0.0067321137, 0.0036272164, 0.004994274, -0.003198399, 0.017668575, -0.022208238, 0.02323998, 0.0017039857, 0.055095, -0.003788426, 0.011175051, -0.011832786, 0.029997885, -0.0010978375, -0.009595197, 0.0066418364, 0.0032096836, 0.0072286394, -0.0175654, -0.0187648, 0.0056197676, -0.026876869, -0.016056478, -0.007409194, -0.01964178, 0.030333202, 0.00562944, -0.010684974, -0.038200233, 0.0016185446, 0.0017394518, 0.026515758, -0.006548335, -0.017049529, 0.031932402, 0.00024423256, 0.006238812, -0.018493967, -0.022375895, -0.040650617, 0.012845183, 0.021627882, 0.035878815, 0.018855076, 0.00037440934, 0.036833175, -0.0034821278, -0.003633665, 0.005897048, -0.006751459, 0.018700317, -0.017616987, 0.050271608, 0.030230029, 0.025819333, 0.0307459, 0.03840658, -0.014186447, -0.01788782, 0.025858022, 0.025522707, 0.010169103, 0.03747801, 0.00031375422, 0.0067321137, 0.02682528, -0.04727956, -0.011239535, -0.0017104341, -0.024490966, 0.036755793, -0.005948635, 0.009369503, -7.204055e-05, -0.001305798, 0.0117360605, -0.017217187, 0.03417644, -0.023201289, -0.014895769, 0.0057938737, 0.010968703, 0.013361054, -0.011794096, -0.00040584523, -0.021924509, 0.03685897, 0.0014863526, -0.0035659568, 0.0005618155, -0.011510367, -0.009214742, 0.018210238, -0.0072931233, 0.008112068, -0.028966144, 0.012006893, 0.0076026455, 0.007931514, -0.019358052, -0.007873477, -0.035801433, -0.006635388, -0.0399284, -0.021705262, 0.0070609814, 0.02571616, -0.00650642, 0.016572349, 0.015218188, 0.009859581, -0.004894324, 0.001256629, 0.0011607093, -0.003788426, 0.011645783, -0.0022150201, 0.010285174, 0.02770226, 0.011181499, 0.008292623, 0.039618876, 0.023381844, 0.00064282335, 0.0027792538, -0.0037787536, -0.01720429, 0.016520761, -0.0039593084, 0.011942409, -0.012432486, 0.010646284, 0.0129741505, 0.008395797, -0.03724587, 0.019345155, 0.018648729, 0.029507808, 0.03221613, 0.007725165, -0.011968202, 0.0028437376, 0.04663472, 0.019744955, -0.013528711, 0.030307408, -7.838818e-05, -0.011162154, -0.019409638, 0.03564667, 0.006751459, -0.03097804, 0.034511756, -0.022272721, -0.0053618317, 0.008395797, 0.004559008, 0.012664628, 0.013606092, 0.022788592, -0.009079326, -0.03206137, 0.004175329, -0.001256629, -0.0013090221, 0.015243982, -0.009924064, -0.011252431, -0.021808438, 0.028192338, -0.019848129, 0.011826337, 0.014444382, 0.016765801, -0.007325365, -0.0060582575, 0.019100117, 0.0015693756, -0.011865028, 0.015269775, -0.02906932, 0.0029678692, -0.022685418, 0.02587092, -0.051896602, 0.0010921952, 0.036343098, 0.03549191, -0.05896403, 0.0038174437, 0.022414586, -0.013825337, -0.014882872, 0.021189392, 0.028140752, 0.026103063, 0.0064096944, 4.92445e-05, -0.022337206, -0.0014162265, -0.004143087, 0.0035595084, 0.011626438, 0.017513813, -0.009827338, 0.017152704, -0.02710901, 0.014147757, -0.007641336, -0.013348157, -0.010839735, 0.04364267, 0.005145811, 0.0133997435, 0.03280938, -0.061749734, -0.03097804, -0.0291467, -0.018390793, 0.0076284395, -0.026644725, 0.016314413, 0.010543109, -0.020260826, -0.008440936, 0.045216076, -0.0013388459, 0.017062426, -0.025174495, 0.01313536, 0.007235088, 0.05034899, -0.024439378, -0.026399687, -0.0023020734, 0.0075188167, 0.049523596, 0.005687475, -0.0066934233, -0.026954249, -0.01541164, 0.019899715, 0.015914613, -0.015746955, 0.0125356605, 0.016972149, 0.0034982488, -0.015824337, -0.007563955, 0.041063316, -0.0042365887, 0.0027389515, 0.02539374, -0.013618989, -0.010459281, 0.01948702, 0.03017844, 0.038354993, -0.0111234635, 0.016237034, 0.010059481, -0.022711212, -0.05114859, 0.002497137, 0.024645727, -0.0076606814, 0.015153704, -1.0176357e-05, 0.021447327, -0.046841066, 0.00043929621, 0.0034821278, -0.011510367, 0.015540607, -0.016546555, -0.010730112, -0.008608594, 0.0018845404, -0.011690922, 0.007828339, -0.0090599805, -0.0010422202, 0.013464227, -0.016017787, -0.008718216, 0.028476067, -0.033118904, -0.001658847, 0.030513758, 0.009956306, 0.0125614535, -0.04315259, 0.023975095, -0.013006392, 0.017281672, -0.0011316915, -0.048672408, -0.03017844, -0.0059067206, -0.0041688806, 0.026722107, -0.025251875, -0.0032548222, -0.022530656, -0.029997885, 0.033712156, -0.00810562, -0.018352102, 0.0042914003, 0.0034627826, 0.0055552837, 0.008208794, -0.0016781922, -0.00949847, 0.010646284, -0.014857079, -0.023472121, 0.0431268, -0.016146755, -0.02471021, 0.026399687, 0.009924064, 0.0074801263, 0.011155706, 0.042017676, 0.009105119, 0.010813941, 0.0022569348, -0.01908722, 0.004107621, 0.012387347, 0.012129412, -0.010175551, -0.025612984, 0.01189727, -0.009375951, -0.0041624326, -0.016262826, 0.0058228914, 0.022982044, 0.0041914503, 0.0050361888, -0.008047584, 0.023046527, 0.009691922, 0.017320361, 0.018958252, 0.01413486, 0.054579128, -0.013045083, -0.039593082, 0.036214128, -0.0199642, 0.010149757, -0.006783701, 0.0003135527, 0.007222191, 0.008718216, -0.0070480844, -0.047331143, -0.008963255, 0.005764856, 0.03301573, 0.023265772, -0.018855076, 0.010304519, 0.036188334, 0.006583801, -0.027779642, -0.025587192, 0.018571349, 0.010323864, 0.030874867, -0.008447384, -0.018248929, 0.010478625, -0.006151759, 0.0028598586, -0.008208794, -0.019332258, -0.016469175, -0.022827283, -0.035749845, 0.0064548333, 0.0051329145, -0.0019893267, -0.050374784, -0.0017313913, 0.0015943631, -0.0065773525, -0.012355105, -0.010684974, -0.00019818707, -0.014147757, -0.009627438, 0.0008584412, 0.02204058, 0.005145811, 0.015231085, -0.0073382617, 0.02008027, 0.03038479, -0.0074156425, -0.0074285395, 0.030591138, -0.003385402, -0.01381244, 0.008086274, 0.025974095, -0.011800544, 0.037349045, -0.0020908888, 0.01800389, -0.028991938, -0.03053955, 0.008808494, 0.012548557, 0.009911167, 0.0042914003, -0.008737561, 0.010730112, -0.005887375, -0.02667052, -0.014895769, -0.004239813, 0.009994997, 0.048646614, 0.0067450106, 0.023575295, 0.036601033, -0.03562088, -0.00794441, -0.019396741, -0.019757852, -0.01345133, -0.05561087, 0.01142009, 0.0035304907, -0.002163433, 0.022801489, 0.010446384, 0.011039635, -0.0025600088, -0.04227561, 0.027366946, 0.03329946, -0.027083216, -9.445876e-07, -0.01760409, 0.008524764, -0.024478069, 0.02543243, -0.0061259656, 0.048543442, 0.0011808604, -0.0045976983, -0.0074285395, 0.007286675, -0.013528711, 0.028553449, 0.011955305, 0.009414642, -0.04307521, 0.02482628, -0.0043300902, -0.01512791, -0.00854411, -0.01628862, 0.008892323, -0.0010446383, 0.010265829, -0.0179652, 0.033480015, -0.013709266, 0.018816387, -0.004123742, 0.037916504, 0.002948524, 0.0133997435, -0.04795019, 0.011955305, 0.017823335, 0.019848129, -0.023278669, -0.04044427, -0.037116904, -0.011316915, -0.033144698, 0.0063935737, 0.011491022, 0.011316915, -0.043591082, 0.001073656, 0.009517816, -0.0021182944, 0.0070996718, 0.030565344, 0.0042688306, -0.033686362, -0.020802489, 0.0017281672, -0.012445383, -0.0047524595, -0.0076993713, -0.0013219189, 0.00076131243, 0.03719428, 0.0058583575, 0.021524709, -0.0071835006, -0.0011728, 0.009305019, -0.023123909, -0.010265829, -0.02015765, 0.019757852, 0.01800389, 0.014637833, 0.006096948, 0.010691422, -0.001073656, -0.028398687, 0.0068417364, 0.009266329, -0.015785646, 0.017900717, 0.034589138, -0.010839735, -0.04374584, -0.0411149, 0.057364833, -0.020905664, 0.0069513586, 0.02067352, -0.009691922, 0.012748457, 0.0014919951, -0.002864695, 0.03892245, -0.009956306, -0.005181277, 0.016133858, -0.045190282, -0.022543553, 0.0036691308, -0.0021876146, -0.0025825782, -0.002126355, 0.020815386, 0.019022735, 0.021769747, 0.0065547833, 0.023420533, 0.022698315, -0.029714158, 0.054630715, -0.010485074, -0.0038819276, 0.036601033, 0.028888764, -0.033196285, 0.029043525, 0.008099171, 0.04446806, 0.009279226, 0.012103618, -0.0077316132, -0.016340207, 0.011181499, 0.05058113, 0.004023792, -0.03280938, -0.028527655, -0.019177496, -0.026077269, -0.029817332, 0.042095058, -0.0059905495, -0.06200767, -0.008885874, -0.0070158425, 0.009388848, 0.009608094, -0.0014976374, -0.007950858, 0.015991995, 0.020118961, -0.0115361605, -0.0009785424, -0.015514814, -0.024478069, -0.024568345, -0.010375451, 0.017062426, 0.00666763, 0.0051071206, 0.031081215, -0.008840736, -0.04758908, -0.004423592, -0.006938462, -0.020467173, 0.017191393, 0.03229351, -0.0087827, 0.007937961, 0.010826838, -0.031287562, -0.017062426, 0.03046217, 0.015966201, 0.022543553, -0.038793482, -0.024736004, -0.014753904, 0.00013602062, -0.004962032, -0.003030741, 0.024439378, 0.0035079212], "content_hash": "5b807519962c118054ef095d0fcac91b"}, {"content": "# GPT-Neo Overview\n\n## What Is GPT-Neo?\n\nGPT-Neo is a family of open-source language models created by EleutherAI in 2021. It was one of the first serious open-source alternatives to OpenAI's GPT-3, designed as an open replication of the GPT architecture. GPT-Neo is the predecessor to EleutherAI's later Pythia suite.\n\n## Architecture Details\n\n| Property | GPT-Neo 125M |\n|----------|-------------|\n| Parameters | 85M (actual) |\n| Layers | 12 |\n| Attention Heads | 12 per layer |\n| Hidden Dimension | 768 |\n| Vocabulary Size | 50,257 |\n| Positional Encoding | Learned absolute |\n| Normalization | LayerNorm |\n| Activation Function | GELU |\n\n## Key Architectural Feature: Local Attention\n\nGPT-Neo's most distinctive feature is its **alternating local and global attention** pattern:\n- **Even layers**: Use local attention with a window of 256 tokens (each token only attends to nearby tokens)\n- **Odd layers**: Use standard global attention (each token can attend to all previous tokens)\n\nThis alternating pattern is a significant architectural difference from GPT-2 (which uses global attention in every layer) and creates interesting attention visualization patterns in the dashboard.\n\n## Why GPT-Neo Is Useful for Learning\n\n- **Same size as GPT-2**: At 125M parameters with 12 layers and 12 heads, it's directly comparable to GPT-2\n- **Same positional encoding**: Uses learned absolute positions like GPT-2, so attention pattern differences are due to architecture, not PE\n- **Different attention pattern**: The local attention in even layers creates visually distinct attention maps \u2014 great for understanding how attention scope affects behavior\n- **Bridge to Pythia**: GPT-Neo \u2192 GPT-NeoX \u2192 Pythia is an evolutionary chain. Comparing GPT-Neo (absolute PE) with Pythia (rotary PE) shows the effect of positional encoding\n\n## Comparing GPT-Neo and GPT-2", "source_file": "gpt_neo_overview.md", "chunk_index": 0, "embedding": [-0.010226652, 0.04108446, 0.05716762, 0.0013934607, 0.02411203, -0.0068664663, -0.0014538043, 0.03559637, -0.023641987, 0.023794433, -0.015841782, -0.06372284, -0.030591028, -0.013402631, -0.032623656, -0.06113124, -0.010728456, -0.042558115, 0.017569514, 0.011446227, 0.027110156, -0.004611521, -0.0012553056, -0.0026281225, 0.036384013, -0.02177451, -0.0032585543, 0.05853964, -0.005564315, -0.017798185, 0.027338825, -0.01680728, -0.021177426, 0.013682117, -0.009318322, -0.014228385, -0.007596941, 0.026805261, -0.018153895, 0.062249184, 0.020440599, -0.040449265, -0.02547135, 0.048173245, -0.058285564, 0.018687459, 0.036282383, -0.0031982106, -0.008194026, 0.013046921, 0.011516099, 0.044133402, -0.040804975, 0.0013069153, -0.017607626, -0.043117087, -0.027059339, -0.0059136725, -0.015079548, -0.0006324168, -0.000592717, -0.03831501, 0.050536174, 0.02987961, 0.00023303744, 0.0011354124, -0.082067296, 0.01709947, -0.010525194, 0.00078843674, 0.026525775, 0.02265108, -0.04603899, -0.020656565, 0.015422553, -0.016045045, 0.0018166598, 0.020783605, -0.011033351, -0.042812195, 0.023400612, 0.011560563, 0.008562439, -0.017188398, -0.0023121126, -0.018115783, -0.015295514, -0.041490987, -0.043066274, -0.015524184, -0.013351815, -0.041618027, -0.028609216, 0.008302009, 0.017594922, -0.0035062805, 0.008892741, -0.027796166, 0.01534633, 0.049469046, 0.0036904875, 0.010499786, -0.023375204, -0.0017483763, 0.039763253, 0.0017531402, 0.0113573, 0.0035666241, 0.006180455, -0.040347636, -0.10589984, 0.006288438, 0.0240104, -0.0052022533, 0.009318322, -0.015841782, -0.018585827, 0.011624082, 0.013504262, -0.0028282092, -2.1053265e-05, 0.03254743, 0.04332035, 0.011223909, 0.03884857, -0.014101346, 0.039864887, -0.003303018, -0.016845392, 0.0060311835, 0.002089794, 0.04743642, 0.005792985, -0.025788948, -0.012602284, 0.016172085, -0.03267447, -0.015104955, -0.03897561, -0.041516393, -0.005243541, -0.013898083, 0.025687316, 0.038238786, -0.04634388, 0.030133687, -0.023146534, 0.025623797, -0.03806093, 0.047004487, 0.032394983, 0.00817497, -0.033868637, -0.03478332, -0.027313417, -0.027846983, 0.014723837, 0.06783891, 0.012538765, -0.034757912, 0.018153895, 0.023934176, 0.00875935, 0.021622064, -0.0031950346, 0.07368271, -0.023121126, -0.017785482, -0.051730342, -0.069261745, 0.00054468034, -0.003004476, -0.040017333, 0.0366635, 0.019475102, -0.009203986, -0.030540213, -0.013593189, -0.019830812, -0.00687917, -0.020567639, -0.004262164, -0.03313181, 0.03468169, -0.06412937, -0.012684859, 0.016997838, 0.047588866, -0.015613112, 0.009013427, -0.0063106697, 0.04370147, 0.08882578, 0.018179303, -0.008645014, -0.05528744, 0.08689478, 0.0060280077, 0.006285262, 0.0040684287, 0.011014295, 0.0353677, 0.022727305, -0.0629606, 0.054423574, -0.030743474, 0.0010687169, -0.047741313, -0.0749531, -0.033665374, -0.0023105247, -0.046648778, -0.020936051, -0.013555078, -0.016184788, -0.01164949, -0.05376297, 0.039178874, 0.029422268, 0.06255408, 0.0049799345, 0.0033855934, -0.005659594, 0.035494737, -0.032877732, 0.012170351, 0.030972145, -0.03521525, 0.0030108278, -0.03968703, -0.0042494596, -0.005837449, -0.014736542, -0.014825469, -0.01469843, -0.041846696, 0.018115783, -0.026195474, -0.0048751277, -0.006739427, -0.008524328, 0.013491558, -0.039788663, 0.0120242555, 0.03384323, -0.016692944, -0.021444209, -0.054880913, 0.060927976, 0.04075416, -0.0053007086, 0.024150142, 0.00330937, 0.0047385604, -0.020999571, 0.001352967, -0.036002897, 0.028787073, -0.050993517, -0.007184064, -0.008689478, 0.01168125, 0.018573124, 0.009267506, -0.010791976, -0.02540783, -0.04779213, -0.016489683, 0.005735818, -0.057472512, 0.08506542, -0.025623797, -0.02670363, 0.028837888, -0.025420533, -0.02190155, -0.035647187, 0.006040712, -0.02501401, 0.015613112, -0.008333769, -0.023311684, 0.054271124, 0.02037708, -0.0060375356, -0.008683126, 0.009877294, -0.013085033, -0.0029536602, -0.033894047, 0.008390936, -0.040881198, 0.041846696, -0.020110298, 0.022435114, 0.0039826776, 0.07292047, -0.030362356, -0.0055833706, -0.00016892237, -0.009940813, -0.05152708, -0.011948032, -0.032750692, 0.011433524, 0.020593047, 0.009343729, 0.016248308, 0.011827345, 0.027364234, -0.025369719, 0.006304318, -0.0081495615, -0.024162848, -0.0011457343, 0.024061216, -0.0109571265, -0.004046197, 0.015181178, -0.01920832, -0.046064395, -0.05340726, 0.0005419014, -0.018738275, -0.016489683, 0.030997554, -0.008657718, -0.037984706, 0.011484339, -0.012303742, 0.03325885, -0.02225726, -0.020554936, -0.0127610825, -0.027008524, 0.017607626, -0.000895626, 0.06697504, -0.009369138, 0.0028615568, -0.02975257, 0.0017467884, -0.025191864, -0.0053515243, -0.04240567, 0.011020646, -0.011528803, 0.008943556, 0.05574478, -0.011306484, -0.05020587, -0.0026027146, -0.00898802, 0.017188398, 0.025001304, -0.028177284, 0.0012870654, -0.002855205, -0.025306199, 0.004757616, -0.033563744, -0.0018357157, 0.006008952, -0.0051546134, 0.009496177, -0.015956117, -0.009883646, 0.053000733, -0.004646457, 0.026906893, -0.006860114, 0.024632892, -0.02764372, -0.0024883794, -0.046445515, 0.009299266, -0.023375204, 0.024556668, -0.031683564, 0.0072285277, 0.013046921, 0.039051834, 0.04144017, -0.03546933, 0.0004204202, 0.041999143, 0.00067727745, 0.012602284, -0.06641607, 0.018344454, 0.048986297, -0.00566277, -0.022943271, 0.02797402, -0.01236091, 0.035875857, -0.016388051, -0.020402487, 0.030286133, 0.03585045, -0.024099328, 0.014736542, -0.0044082585, -0.066568516, -0.017607626, -0.023870656, 0.0067330752, 0.07286966, 0.011115925, -0.0026392385, 0.058641274, 0.037247878, 0.011560563, -5.7465368e-05, 0.02329898, -0.019678365, 0.019513214, 0.0046305773, -0.0036428478, 0.051806565, 0.027618311, -0.07205661, -0.044743188, -0.018801795, -0.02553487, 0.004646457, 0.032039274, -0.014596798, -0.0016785047, -0.044133402, 0.0259541, -0.0011330304, 0.010506138, -0.043726876, -0.0020723261, 0.038619902, 0.024353405, 0.051120553, -0.032064684, 0.013567781, 0.04309168, -0.03595208, -0.018750979, 0.009693087, -0.0041287723, -0.010245708, -0.03877235, -0.031048369, -0.005243541, -0.027211787, 0.026271697, 0.086488254, 0.017874409, -0.014571391, -0.003303018, 0.0006610006, 0.08714886, 0.005389636, 0.0782053, -0.0470553, 0.04039845, -0.05432194, -0.051933605, 0.0012267218, -0.024124734, -0.0027440456, 0.0010329871, 0.01291353, -0.000603039, -0.041160684, -0.0041224207, -0.027669128, -0.011535155, -0.028304324, -0.0059486083, 0.01885261, 0.0039350376, 0.041516393, -0.0038588142, -0.0023772202, 0.015575, 0.020986868, 0.02534431, 0.0071713603, -0.030311542, 0.04880844, -0.008422696, 0.005586547, 0.014482464, 0.01722651, -0.0060978793, -0.017023247, 0.048884667, -0.013364519, 0.037756037, -0.020288153, -0.029803386, -0.10280008, -0.039153468, -0.0060819993, 0.0019802228, -0.00038627844, -0.012341853, 0.05045995, -0.00083051843, -0.014660318, -0.0035634483, 0.020491416, -0.0210885, 0.0056119547, -0.019170208, 0.016172085, -0.023819841, 0.02135528, 0.027719943, 0.020745493, 0.007959003, 0.012144943, 0.0060597677, -0.008041578, 0.02810106, 0.0050148703, -0.055084176, -0.021050388, 0.031556524, -0.028431362, -0.027262602, -0.05208605, -0.023629282, -0.028964927, 0.021215538, -0.010531546, 0.0036968393, -0.012716619, -0.06697504, 0.016388051, -0.013046921, -0.053508893, -0.059301876, 0.021228243, 0.003912806, 0.017175693, 0.01411405, 0.008009818, 0.0005097446, 0.02751668, 0.042507302, 0.0028869647, -0.0083528245, 0.035748817, 0.0470553, -0.039991926, 0.011738418, -0.03160734, -0.041186094, -0.008676774, -0.019119393, -0.011757473, 0.035418514, 0.0395854, -0.0038175266, 0.004973583, 0.015752856, -0.018319046, 0.00056453026, 0.02323546, 0.017874409, -0.016959727, 0.0156004075, -0.03069266, -0.037273288, -0.011827345, 0.017467882, -0.025026713, 0.00062606484, 0.00096708554, 0.016032342, 0.008784757, 0.010563306, -0.016057748, -0.012907178, 0.009134115, 0.016870799, 0.04075416, -0.04484482, -1.4217468e-05, -0.018662052, 0.013478854, -0.009489825, 0.05787904, 0.01891613, -0.019259136, 0.0020516822, 0.015244698, -0.023121126, -0.00804793, 0.014126753, -0.019551326, -0.03745114, 0.024861561, -0.006275734, 0.022600265, -0.008676774, 0.0037857667, 0.0071205446, 0.021291763, 0.020288153, 0.0069490415, -0.021952365, -0.0069871536, -0.01803956, 0.021685584, -0.022244556, -0.015549593, -0.046547145, 0.013605894, -0.010595066, 0.0054563317, 0.042888418, -0.0007610439, 0.032979365, -0.017823594, 2.7293569e-05, 0.018306341, 0.05129841, 0.010785624, -0.0240104, -0.04240567, -0.027084747, 0.0059549604, 0.008136857, 0.0041446523, 0.058387194, 0.03831501, 0.0023565763, 0.011484339, 0.013872676, -0.015181178, 0.027694535, -0.009591456, -0.034351386, 0.035672594, -0.026729038, 0.008981668, -0.013351815, 0.008168617, -0.023032198, 0.033639967, 0.008765701, -0.013809156, 0.010588713, -0.03173438, 0.046877448, -0.0165532, -0.00060423, -0.035291478, -0.012754731, -0.02787239, 0.0062471502, 0.00924845, -0.061893474, 0.0032490264, -0.03033695, -0.017467882, -0.009096003, 0.0015236759, 0.011693954, 0.020351673, 0.0165532, -0.03445302, -0.03432598, 0.044997267, 0.04827488, -0.013250183, 0.020872533, -0.018712867, 0.009527937, -0.073479444, -0.009102356, -0.0027916853, 0.007673165, 0.043066274, -0.038924795, 0.005199077, 0.026729038, -0.021126611, -0.025509462, -0.004783024, 0.022663785, -0.009680384, 0.014431648, -0.0493166, 0.017887112, -0.023743618, 0.0014768301, -0.03534229, 0.0020024546, 0.048376508, 0.013593189, -0.05889535, -0.039661624, 0.0048370156, -0.02371821, 0.015028732, -0.031785198, -0.012449837, -0.034173533, 0.0015117659, -0.011039702, -0.012608636, -0.0691093, -0.011319188, 0.013809156, 0.012437133, -0.0014156925, -0.066568516, -0.011414467, -0.0037889427, -0.01258958, -0.0135296695, -0.03115, 0.011344597, 0.025560277, -0.0013696408, 0.008384584, -0.025484053, 0.004214524, 0.0017197925, -0.03498658, 0.026424143, 0.014457055, 0.0023057605, 0.0034935768, -0.01739166, -0.007050673, 0.0023549884, -0.027491273, -0.0027710416, -0.032420393, -0.004147828, 0.03186142, -0.015422553, -0.011954384, -0.001659449, 0.02089794, 0.021139314, 0.017709257, 0.02310842, 0.016045045, -0.030489396, -0.008810165, 0.009909054, 0.0036301438, -0.007793852, -0.029295228, -0.01476195, 0.006189983, -0.014291905, -0.0119607365, 0.03209009, -0.0068347063, -0.02394688, -0.021888847, 0.016261011, -0.00092262187, -0.02797402, 0.016718352, -0.004916415, -0.007762092, 0.00329349, 0.008854629, 0.019030465, 0.004519418, 0.027618311, 0.02135528, -0.029498491, 0.06758483, -0.02764372, 0.00032236186, 0.029422268, -0.03056562, -0.014304608, 0.021799918, -0.03346211, 0.008282953, 0.00879111, -0.03292855, -0.022244556, -0.042507302, -0.03150571, -0.030972145, -0.001945287, 0.025128344, 0.024505852, -0.0045734094, 0.027186379, -0.011166741, -0.08816517, -0.013694821, -0.014431648, -0.043523613, -0.014711134, -0.0070824325, 0.05975922, -0.024582077, 0.023476835, -0.05752333, 0.025877876, -0.054982547, 0.026449552, 0.0010877728, 0.0008384584, -0.011865457, -0.02413744, 0.0048211357, 0.039483767, -0.023451427, -0.0470553, 0.010023389, 0.01054425, -0.002086618, -0.057574145, 0.032242537, 0.004513066, 0.018280935, -0.0034872247, 0.013161256, -0.05305155, 0.016870799, -0.0035602723, 0.031175409, -0.0011743181, 0.014355424, 0.012189407, -0.0055992506, 0.03793389, 0.009102356, 0.020580342, -0.0314803, -0.003144219, 4.252834e-05, 0.011141334, -0.019652957, 0.018344454, 0.0054531554, -0.004049373, 0.020745493, 0.026856076, 0.0015054139, -0.0050434545, 0.011935328, 0.005234013, 0.032979365, -0.003989029, -0.031556524, 0.029015742, 0.01927184, 0.0074127344, 0.023438724, 0.015879894, -0.013313703, -0.01291353, 0.00658698, 0.031988457, 0.04260893, 0.00087418815, 0.0028742608, -0.011370004, 0.030489396, -0.01574015, -0.018166598, -0.04636929, -0.011477987, 0.025496759, 0.04204996, 0.0016578609, -0.00048314576, 0.003471345, -0.008390936, -0.044336665, 0.00049545267, -0.0074444944, 0.025738133, 0.017988743, -0.0076096454, 0.017594922, 0.017810889, 0.0053674043, -0.012386317, -0.026932301, 0.014888989, -0.032013867, 0.016642129, 0.011846401, 0.0331064, 0.035774224, -0.0059073204, 0.053610522, 0.02559839, -0.030006647, -0.0005113326, 0.007457198, 0.022320779, -0.011312837, 0.018166598, -0.003306194, -0.030540213, -0.008105098, 0.045124307, -0.0015443197, -0.004179588, -0.03219172, 0.007330159, 0.05879372, 0.001951639, 0.018115783, 0.0059962478, -0.008930853, -0.016489683, 0.003287138, 0.020034075, -0.017188398, 0.0030013, -0.0072221756, 0.044641558, -0.0019992786, -0.030032055, -0.0059454325, -0.0053229406, -0.024061216, -0.038721535, -0.019462397, -0.025382422, 0.011560563, -0.0036873114, -0.023146534, 0.008632311, -0.0017023246, -0.023273572, 0.0057866336, 0.023603875, -0.004020789, -0.011465283, 0.0019230552, -0.026627406, -0.025903283, 0.009115059, 0.037374917, 0.008860981, -0.037501957, 0.01845879, -0.0006586186, 0.007076081, 0.014215681, -0.028787073, -0.008670422, 0.02622088, 0.0081178015, 0.03485954, 0.017467882, 8.753792e-05, 0.011547859, -0.039915703, -0.043345757, -0.00083528244, -0.012678508, 0.009038836, -0.007857371, 0.004386027, 0.048097022, 0.059200246, 0.03315722, 0.017277325, 0.008600551, 0.019652957, 0.010785624, 0.013555078, -0.0036619036, 0.033741597, 0.0142029775, 0.0076922206, 0.065755464, -0.019754589, -0.0012870654, -0.021990478, -0.01074116, 0.011503396, 0.009680384, -0.0026249464, -0.008994372, -0.02810106, 0.0007590589, -0.0033887695, 0.034961175, -0.028863296, -0.020046778, -0.009146819, -0.020593047, 0.0040112613, -0.00571041, -0.018306341, 0.0078510195, 0.05996248, 0.00034717418, -0.0013656709, 0.0023645163, 0.034376796, -0.0039350376, 0.01567663, -0.017950632, 0.021761807, -0.027415048, 0.02037708, -0.013097737, 0.009782014, -0.02810106, 0.0047099767, -0.028380547, -0.01852231, 0.0024296239, -0.03254743, -0.0088863885, 0.024264477, -0.0012394257, 0.028787073, 0.014495167, 0.030413173, -0.031454895, 0.005535731, -0.009178579, 0.024734523, -0.0155368885, 0.0113573, 0.011998848, -0.007457198, 0.021367986, 0.03745114, 0.02482345, 0.02751668, 0.015219291, -0.005081566, -0.0019675188, -0.01476195, 0.0039604455, 0.008155914, 0.00093929574, -0.014660318, 0.0006193159, 0.024810746, -0.0013656709, -0.0081178015, 0.025433239, 0.04669959, 0.011350948, 0.04108446, 0.012856362, 0.009750255, 0.001499856, 0.039991926, 0.0015276459, 0.0070951367, 0.01469843, -0.016464274, -0.019589437, 0.023553059, 0.027059339, 0.019424286, -0.014342721, 0.02115202, 0.020046778, -0.0019278191, 0.0072729914, -0.007673165, 0.031683564, 0.010004333, 0.028380547, 0.008867333, 0.0068347063, -0.0070951367, 0.008778405, 0.018598532, 0.006015304, -0.021253651, -0.0068220026, -0.022130221, 0.022447819, -0.0112556685, 0.0055992506, 0.0038365824, -0.0051260297, -0.016756464, -0.01285001, 0.01469843, 0.00010133357, 0.015778262, 0.0074381423, -0.022752712, 0.024493149, -0.027796166, 0.0015435257, -0.0005498413, -0.025903283, 0.006580628, 0.014558687, -0.07368271, 0.0012838894, 0.02904115, -0.011236613, -0.0062185666, 0.0139997145, 0.020872533, 0.013720228, 0.030540213, -0.0029139605, -0.033182625, 0.015498777, 0.0019929267, 0.012189407, 0.020199224, -0.014355424, 0.00969944, 0.02323546, 0.011103222, -0.004601993, -0.0016943847, -0.01405053, 0.011293781, 0.022612968, 0.023337092, 0.024925081, -7.9746846e-05, -0.024645595, 0.005812041, -0.00014490403, -0.013936195, -0.03668891, -0.008225785, -0.0035539202, -0.02047871, -0.023984991, -0.012163999, 0.001969107, 0.0062280945, -0.0063614855, 0.0042176996, 0.037070025, 0.046775814, 0.05200983, -0.011789233, -0.019132096, 0.017569514, -0.012545116, 0.06545057, -0.027262602, -0.017201101, -0.025839763, -0.014355424, -0.009566048, 0.0019992786, -0.0112556685, 0.0042399317, 0.039432954, 0.012659452, -0.046089806, 0.020453304, 0.022498634, -0.0072539356, -0.008625958, 0.04555624, -0.040627122, 0.0066314437, -0.02777076, 0.010321931, 1.4049984e-06, 0.0030775236, 0.007158656, 0.003588856, -0.002859969, -0.026932301, 0.007870075, 0.03442761, 0.007812908, 0.0295239, -0.0032252064, -0.0059549604, -0.01255782, 0.011706658, 0.002551899, -0.021063091, 0.009807423, 0.0015268518, 0.007069729, -0.0024518557, 0.0060121277, -0.0132628875, -0.0024264478, -0.028177284, -0.01739166, 0.011281077, -0.038365822, -0.007508014, 0.04695367, -0.0051768455, 0.052390948, 0.019055873, -0.024505852, 0.008708534, -0.042990047, 0.030768882, -0.018064968, 0.015956117, -0.025001304, -0.05023128, -0.0020215106, -0.014126753, 0.017378956, 0.022892455, -0.008740294, 2.3608936e-05, -0.00626303, -0.030387765, 0.019818109, 0.021914253, -0.01551148, 0.018382566, 0.01138906, 0.024416925, -0.012811898, 0.017480588, -0.031683564, 0.003414177, 0.007908187, 0.016121268, -0.0014776242, -0.02160936, -0.03135326, 0.05981003, -0.014787357, 0.022015885, 0.008651366, 0.013123144, 0.010912663, 0.012119535, -0.035291478, 0.012322797, -0.008613255, 0.001788076, 0.026957707, -0.0042780433, -0.013936195, 0.020910645, -0.02135528, -0.01826823, 0.0029298405, 0.011801938, -0.0015093839, 0.011744769, 0.0056468905, 0.012208463, 0.00869583, -0.0005764401, -0.015231994, 0.03033695, 0.017874409, 0.035494737, -0.008022523, -0.009159523, 0.020986868, -0.04652174, 0.014368128, 0.006285262, 0.006405949, 0.0057739294, -0.0064027733, -0.023934176, -0.053864602, -0.036968395, 0.0053229406, 0.024416925, 0.00072293216, -0.020910645, 0.0030409996, 0.028202692, 0.030489396, -0.01891613, -0.031658158, 0.022219148, 0.025674613, -0.007342863, -0.010182188, 0.02881248, -0.0021120259, -0.00020296489, 0.037425734, -0.013644005, -0.009064243, -0.040271413, -0.0088863885, -0.015727447, -0.020135706, 0.023883361, 0.003131515, -0.047741313, -0.0051641413, -0.01226563, -0.022346187, -0.0042939233, -0.011611379, -0.014164865, 0.0036396717, 0.009210339, 0.022219148, 0.0018627116, 0.009394545, 0.026017617, -0.00083528244, 0.011903568, 0.032166313, -0.05752333, -0.0006395627, 0.0031346912, -0.017086767, -0.019297248, 0.021291763, -0.024734523, -0.0028869647, 0.03150571, -0.0031267512, 0.009502528, 5.3098396e-05, -0.036714315, -0.022435114, -0.014622207, 0.017302733, 0.008867333, 0.008162266, -0.013046921, 0.0066314437, -0.021126611, -0.0148000615, 0.006199511, -0.012081424, 0.022816231, -0.006802947, -0.0068982258, 0.04827488, -0.010601417, -0.034478426, 0.010048797, -0.024543963, -0.018420678, -0.010487082, 0.020707382, -0.0054340996, 0.021939661, -0.02160936, 0.033182625, 0.004319331, 7.091365e-05, -0.0068220026, -0.021939661, 0.031886827, -0.023819841, 0.024493149, -0.008613255, 0.009889998, -0.0068664663, 0.0134661505, -0.009585104, 0.019564029, 0.014977916, -0.009261155, -0.0053165886, -0.013428038, 0.010728456, -0.023438724, 0.0012989753, 0.01048073, -0.038365822, 0.04006815, -0.005116502, -0.019513214, -0.006853762, 0.01446976, -0.027389642, -0.023083014, 0.005097446, 0.013364519, 0.069058485, 0.0011314425, 0.020631159, -0.010582361, 0.009381841, 0.013644005, 0.021291763, -0.041135278, 0.005100622, 0.0130596245, 0.0026535303, -0.008511623, -0.03968703, -0.030286133, -0.009591456, -0.036968395, -0.0041668843, 0.008003467, -0.008378232, -0.041414764, 0.007241232, -4.538176e-05, 0.00074159104, 0.0055928985, 0.045607056, -0.02411203, -0.015702039, -0.01398701, 0.0046813926, -0.008066986, -0.005227661, -0.010620473, -0.016362643, -0.008282953, -0.001036957, 0.0032394985, 0.0298542, 0.004023965, -0.01469843, -0.011795585, -0.019030465, 0.008473512, -0.029219005, 0.044260442, -0.011071462, 0.024620188, 0.004792552, 0.010950775, -0.014876285, -0.038950205, 0.030387765, -0.0037063672, -0.010614121, 0.004198644, 0.0076985727, -0.0017150285, -0.032369576, -0.0065488685, 0.04860518, -0.008060634, -0.029320637, 0.016083157, 0.0028361492, 0.034757912, 0.017378956, -0.020326264, 0.03280151, -0.02096146, -0.007050673, 0.009909054, -0.02553487, -0.043345757, 0.006726723, -0.013250183, 0.014126753, 0.0066568516, 9.567636e-05, -0.0019373471, 0.027719943, -0.019767292, 0.022930566, -0.00094961765, -0.019907035, 0.050968107, -0.010404507, 0.012888122, 0.036155343, 0.030032055, 0.012659452, 0.012011552, 0.020288153, 0.061080426, -0.0009662915, -0.003750831, -0.027542088, -0.03442761, 0.009432657, 0.048376508, -0.008778405, -0.032293353, -0.028787073, -0.02670363, -0.018992353, 0.011522451, 0.038569085, 0.0010679229, -0.04065253, 0.00058715907, 0.020783605, -0.01311044, -0.014685726, -0.03290314, -0.009915406, -0.014952508, 0.007971707, -0.0041128923, 0.0051450855, -0.014431648, -0.047842946, -0.013516965, 0.009445361, 0.0065488685, 0.004640105, -0.026043026, 0.0030711715, 0.02459478, -0.0165532, -0.0026090667, -0.015714742, 0.002404216, 0.031810604, 0.015981525, -0.01745518, -0.010950775, 0.011611379, -0.010201244, -0.0156004075, 0.020224633, -0.00018073304, 0.0071268966, -0.023413315, -0.025306199, 0.0026217706, 0.00020723262, -0.0036968393, 0.0020024546, 0.00067092554, 0.01574015], "content_hash": "bd7f60727e8ee3797e8d578a328487f7"}, {"content": "attention in even layers creates visually distinct attention maps \u2014 great for understanding how attention scope affects behavior - **Bridge to Pythia**: GPT-Neo \u2192 GPT-NeoX \u2192 Pythia is an evolutionary chain. Comparing GPT-Neo (absolute PE) with Pythia (rotary PE) shows the effect of positional encoding ## Comparing GPT-Neo and GPT-2\n\n| Feature | GPT-2 (124M) | GPT-Neo 125M |\n|---------|-------------|--------------|\n| Layers \u00d7 Heads | 12 \u00d7 12 | 12 \u00d7 12 |\n| Hidden Dim | 768 | 768 |\n| Attention | All global | Alternating local/global |\n| PE Type | Learned absolute | Learned absolute |\n| Training Data | WebText | The Pile |\n| Creator | OpenAI | EleutherAI |\n\n## HuggingFace Model IDs\n\n- `EleutherAI/gpt-neo-125M` (in dropdown)\n- `EleutherAI/gpt-neo-1.3B`, `EleutherAI/gpt-neo-2.7B` (larger, enter manually)", "source_file": "gpt_neo_overview.md", "chunk_index": 1, "embedding": [0.023152947, 0.0045151743, 0.06888527, 0.009324132, 0.017682977, -0.023642587, 0.00022755115, 0.046501752, -0.0055119405, 0.012975443, 0.020872626, -0.065191984, -0.02165605, -0.028930692, -0.02439803, -0.018704224, 0.0022505925, -0.035645746, 0.026650371, 0.022033771, -0.0057008015, -0.0075474414, 0.02116641, 0.029322404, 0.037660263, 0.0138497995, -0.0015537307, 0.042220905, 0.021879885, 0.0045081796, 0.014339439, -0.026986124, -0.00403253, -0.011828288, -0.0052951, -0.01664774, -0.05758159, 0.06552774, -0.027321877, 0.07492881, 0.030301683, -0.040821932, -0.020998534, 0.066982664, -0.03139288, -0.010380355, 0.037660263, 0.008037081, 0.006144974, 0.057805426, 0.0010938194, 0.030105827, -0.06407281, 0.028790794, 0.009799782, -0.008729571, -0.019599564, 0.008044075, -0.018494379, 0.030273704, 0.022915123, -0.008414802, 0.036625024, 0.01383581, 0.017361213, 0.006903915, -0.10083773, 0.03178459, 0.001515259, -0.005637848, 0.048516266, 0.029490279, -0.026412547, -0.009345117, 0.061946377, -0.012150051, -0.0042423755, 0.032428116, 0.0071452376, -0.033379417, 0.035589788, 0.0037212593, -0.011912227, -0.03307164, -0.00758941, -0.03732451, -0.02306901, -0.04415148, -0.063457265, -0.038779438, -0.01895604, -0.02578301, -0.041605357, -0.006022564, 0.003969576, 0.028762816, 4.5275247e-05, -0.040821932, -0.030721374, 0.0066765826, 0.03083329, 0.024803732, -0.032036405, -0.018354481, 0.02574104, -0.0019165884, 0.02990997, 0.006729044, -0.026496485, -0.01839645, -0.12042331, -0.007883194, 0.009345117, 0.031448837, 0.034246776, -0.0034764395, -0.019123916, 0.013045392, 0.003158174, 0.0030812307, -0.013772856, 0.010786056, 0.044850968, 0.020690762, 0.023110978, -0.00015629113, 0.020998534, -0.034302734, 0.013465083, -0.021236358, -0.016284006, 0.038779438, 0.04272453, -0.019669512, -0.0018274041, 0.00024219662, -0.014437366, 0.018592307, -0.031169044, -0.031113084, -0.019781431, -0.024048287, 0.021152422, 0.05170592, -0.03970276, 0.03735249, -0.044039562, 0.044291377, -0.050362907, 0.040290326, 0.009149262, -0.000131372, -0.04096183, -0.032232262, -0.065975405, -0.037464406, 0.017403182, 0.044095524, -0.00095916854, -0.041605357, 0.008533714, 0.00691091, -0.017641008, 0.027000114, 0.017710956, 0.10872792, -0.02574104, -0.011128804, -0.04292039, -0.067038625, -0.021726, -0.013695912, -0.012408861, 0.004378775, 0.006228912, 0.0050922493, -0.034834344, 0.0034519576, -0.0415494, -0.01979542, -0.015416645, -0.006431763, -0.03452657, 0.032987703, -0.07313813, -0.0038261819, 0.012213005, 0.024272123, 0.01610214, -0.006582152, -0.00074670004, 0.052685197, 0.10011027, 0.005284608, -0.014003686, -0.06726246, 0.06603137, 0.014010681, 0.023670565, 0.008918432, 0.016325975, 0.041717276, 0.03307164, -0.023138957, 0.050027154, -0.04233282, 0.007421534, -0.032400135, -0.049971197, -0.0057637547, 0.008009101, -0.07123554, -0.005106239, 0.006127487, -0.03186853, -0.009240194, -0.027098041, 0.024342071, 0.04423542, 0.0643526, -0.019543605, 0.025908917, 0.02764364, 0.03740845, -0.036680985, 0.013031403, 0.051649958, -0.020564854, -0.03046956, -0.0057847393, 0.014479335, 0.034022942, 0.00022164926, -0.019445678, -0.016228048, -0.03270791, 0.031728633, -0.037576325, 0.013898764, -0.024258133, -0.010212478, -0.0031669175, -0.019837389, 0.0099117, 0.023712534, -0.012856531, -0.010352375, -0.046809524, 0.023376783, 0.036708962, 0.012919485, 0.0010684631, -0.022761237, 0.009589937, -0.013828815, -0.0010160017, -0.023880411, 0.0405981, -0.03819187, 0.0034729422, -0.0056693247, 0.050111093, -0.00807905, 0.0033085633, -0.03167267, -0.054056186, -0.029574217, -0.006078523, -0.03220428, -0.054168105, 0.047788803, 0.017850852, 0.021781957, 0.023628598, -0.05058674, -0.0023222896, 0.00093730964, 0.0129124895, -0.023376783, -0.010737092, -0.0112687005, 0.0085407095, 0.049467567, 0.009051333, 0.0025863452, -0.005253131, 0.020550864, -0.015850326, -0.010044602, -0.01940371, 0.028846754, -0.047536988, 0.07515265, -0.034358695, 0.009806777, 0.009939679, 0.03307164, -0.031924486, -0.023348803, -0.016913543, 0.0095129935, -0.05581889, -0.0012826803, -0.025545185, 0.012744613, 0.03819187, -0.0070717917, 0.03740845, 0.0057287808, 0.029406343, -0.03178459, -0.0034834344, -0.0010579708, -0.011576474, -0.019221842, 0.013800835, -0.011317665, 0.00877154, -0.014591253, -0.0061065024, -0.032372158, -0.06664691, -0.018676246, -0.0024761762, 0.0069388896, 0.035058178, -0.023432741, -0.0031913994, 0.017291265, -0.015514573, 0.037100676, 0.009394081, -0.02585296, -0.019935317, -0.022845173, 0.0023135461, -0.032344177, 0.07979723, -0.029993908, 0.005714791, -0.039562862, -0.020103194, -0.025699072, -0.012331917, -0.054587796, 0.00078823196, 0.03578564, 0.003145933, 0.051873796, -0.0035988493, -0.035309993, 0.011702381, 0.0023345307, 0.011653417, 0.008337859, 0.01523478, 0.006061036, 0.005235644, -0.02624467, 0.01608815, -0.022775225, -0.00041772358, 0.01266767, -0.011212742, 0.017613027, 0.003917115, -0.016172089, 0.046613667, 0.0148570575, 0.019235833, -0.012786582, 0.023222895, -0.011128804, -0.023726525, -0.025041556, -0.019445678, -0.012485804, 0.02534933, -0.017850852, 0.0044487235, 0.030553497, 0.04090587, 0.03914317, -0.053244784, 0.015500584, 0.012408861, 0.016983492, 0.019669512, -0.08718379, -0.00832387, 0.05682615, -0.009785793, -0.012080103, 0.028427063, -0.013940732, 0.00611, -0.0058197137, -0.019193863, 0.033939004, 0.07157128, -0.037940055, 0.00495585, 0.007526457, -0.046166, 0.015444625, -0.011947201, -0.024775753, 0.03265195, 0.012471815, 0.007841225, 0.04384371, 0.01885811, 0.017081419, -0.0063233427, -0.025671093, -0.014535295, 0.017193336, 0.0021788953, -0.039562862, 0.036177356, -0.0084287925, -0.07095574, -0.043675832, -0.01709541, -0.03307164, 0.009380091, 0.052237526, -0.037436426, 0.020131173, -0.0022908128, -0.009505998, 0.030609455, 0.023222895, -0.025824979, 0.014367418, -0.006480727, 0.037436426, 0.042584635, -0.029546238, -0.01222, 0.049075853, -0.008939416, -0.036065437, -0.0006583901, -0.0075334515, -0.022299577, -0.01653582, -0.047732845, -0.017794894, -0.03592554, 0.017417172, 0.06261788, -0.01083502, -0.0112687005, 0.008107029, 0.0017041198, 0.08494543, 0.011100824, 0.06491219, -0.048572227, 0.04280847, -0.05170592, -0.051929753, 0.019333761, -0.021516154, 0.0184664, 3.415453e-05, -0.0015353692, 0.0076034, -0.031169044, 0.007785266, -0.057973303, 0.0018326503, -0.037940055, 0.008037081, 0.005434997, -0.009247189, 0.03953488, 0.0009145764, -0.039982554, -0.0028556467, -0.013639954, 0.0038716486, 0.0065751574, -0.003710767, 0.049215753, -0.026902186, -0.006022564, 0.032428116, 0.012045128, -0.03125298, -0.02301305, 0.041633338, -0.01699748, 0.014395397, -0.00017869651, -0.02354466, -0.09563357, -0.024020309, -0.021977814, 0.018676246, -0.019921327, 0.004207401, 0.009212215, 0.00022602103, -0.025755031, 0.020592833, 0.025307361, -0.041185666, 0.008855478, 0.008729571, 0.017403182, -0.0114575615, 0.03687684, -3.6832247e-05, 0.03038562, 0.032819826, 0.021879885, 0.0010693374, -0.0075824154, -0.02066278, -0.01800474, -0.039814677, -0.0035236548, 0.01383581, -0.031840548, -0.017822873, -0.058309056, -0.027098041, -0.025880938, 0.025363319, -0.021935845, 2.991937e-05, -0.031560756, -0.06328939, 0.0073585804, -0.01428348, -0.038471665, -0.049635444, 0.028217217, 0.012576737, 0.019501638, 0.011135798, 0.0073445905, -0.004683051, 0.01979542, 0.040402245, -0.0061659585, -0.004917378, 0.03315558, 0.033407394, -0.035841603, 0.039003275, -0.0008267036, -0.025866948, 0.013500057, -0.012052123, -0.009261179, 0.02028506, 0.022425484, 0.024132226, 0.0031669175, 0.013814826, -0.009415066, -0.027126022, 0.05310489, 0.016829604, -0.009687865, 0.041297585, -0.06743034, -0.025153475, -0.005071265, 0.028273176, -0.042780492, 0.010408334, 0.002441202, -0.0025356326, 0.032148324, 0.0023904894, -0.03181257, -0.028399084, 0.007281637, -0.016186079, 0.023446731, -0.03863954, -0.00018241252, -0.0071382425, -0.030553497, -0.034162838, 0.0506427, 0.026272649, -0.023180926, 0.00087129575, 0.022355534, -0.0075824154, -0.024146216, 0.0010981911, -0.014968975, -0.04378775, 0.0048614196, -0.0029780567, 0.026118763, -0.016018203, 0.015066903, -0.004864917, 0.02207574, -0.008030086, 0.01608815, 0.0072606523, 0.0048019635, -0.034750406, 3.5985217e-05, -0.004791471, -0.011261705, -0.0340789, 0.01660577, 0.0014295721, -0.009058328, 0.040178407, 0.0025671092, 0.011842278, -0.015850326, -0.0086526275, 0.01937573, 0.027867476, 0.0188721, -0.023726525, -0.043479975, -0.017529089, 0.0026335604, -0.030329661, -0.0006522696, 0.05251732, 0.029854013, -0.025559176, 0.0048474297, 0.00026121386, -0.022145689, 0.045410555, -0.023908392, -0.010974917, 0.0013106597, -0.022355534, -0.0056938063, -0.014248506, 0.002367756, 0.0014925257, 0.025824979, 0.005235644, -0.012310932, 0.0001530123, -0.025013577, 0.023292845, 0.007918168, -0.0059666056, -0.028818775, -0.018354481, -0.032540035, -0.0035096651, 0.016284006, -0.061106995, 0.013583995, -0.024747772, -0.03553383, 0.018060697, 0.007232673, 0.016242037, 0.01845241, -0.0028276674, -0.015836336, -0.036177356, 0.035645746, 0.04149344, 0.007323606, 0.022663308, -0.02490166, -0.0102614425, -0.0460261, -0.015766388, 0.00059543643, 0.023460722, 0.019389719, -0.045886204, -0.009918694, 0.023572639, -0.027070062, -0.021753978, -0.01975345, 0.033043664, 0.0006395914, 0.0042598625, -0.030161785, 0.025307361, -0.012835546, -0.005326577, -0.012527773, -0.0092891585, 0.037968036, 0.005088752, -0.04890798, -0.05791734, 0.011835283, -0.012548758, 0.035170097, -0.040737994, -0.016843595, -0.037520364, 0.034134857, 0.007470498, -0.002495412, -0.06390493, 0.009310143, 0.011933211, 0.027070062, 0.012303938, -0.026482495, -0.005071265, -0.00031717264, -0.0061169947, -0.021712009, -0.0161581, 0.01789282, 0.027993383, -0.014619233, 0.004742507, -0.022761237, 0.012884511, -0.030497538, -0.013157309, 0.021837916, -0.0019707985, 0.01569644, 0.011205747, -0.014010681, -0.00077249354, -0.00040417106, -0.01105186, -0.006141477, -0.035729684, -0.018704224, 0.007897183, -0.023194917, 0.004340303, 0.018088678, 0.022257607, 0.02025708, 0.00852672, 0.013087361, 0.011380618, -0.03824783, -0.018340493, 0.012884511, 0.00832387, -0.013828815, -0.008442782, -0.017473131, 0.022187658, -0.02116641, 0.017850852, 0.028734837, 0.028189238, -0.009457035, -0.04132556, 0.02860893, 0.0014112106, -0.040262345, 0.028846754, -0.020173142, 0.01664774, 0.010394344, -0.019781431, -0.0017941785, 0.013024407, 0.027042083, 0.04051416, -0.045102783, 0.054867588, -0.03419082, 0.019431688, 0.016339965, -0.029210486, -0.012863526, -0.008330864, -0.02583897, -0.0025181454, 0.011625438, -0.028343124, -0.04194111, -0.015766388, -0.033519313, -0.014940996, -0.006805987, 0.00900237, -0.022957092, 0.01336016, 0.026468506, 0.0059176413, -0.10565019, -0.015710428, 0.0013972209, -0.03922711, -0.004676056, -0.022145689, 0.058197137, 0.0022243618, 0.027223948, -0.045746308, 0.06642308, -0.046809524, 0.023250876, -0.0007536949, 0.008477756, -0.0104852775, -0.019669512, 0.012296943, 0.02856696, -0.029630177, -0.048963938, 0.027265918, 0.027853485, 0.01705344, -0.058365013, 0.025167463, -0.015528563, 0.0076103946, 0.009303148, -0.0025513708, -0.034946263, 0.026398556, 0.0044836975, 0.044403296, 0.004578128, 0.03368719, -0.014437366, 0.026412547, 0.021236358, 0.01565447, 0.008442782, -0.02630063, 0.005697304, -0.01568245, 0.0010789554, 0.00758941, 0.0018274041, 0.0074285287, 0.0066486034, 0.024300102, 0.019067956, 0.0040640067, 0.021642061, 0.016745666, -0.0035271521, 0.02350269, 0.0018081683, -0.019095935, 0.05296499, 0.008351848, 0.017794894, 0.02767162, -0.0019305781, -0.0021002032, 0.01744515, 0.008904442, 0.02578301, 0.039758716, 0.014339439, -0.0021439209, -0.021334287, 0.008337859, 0.0015764639, -0.016927533, -0.021781957, -0.010226468, 0.014815088, 0.06093912, 0.010946938, 0.008638637, 0.01610214, -0.02583897, -0.030777331, 0.009995638, 0.022005793, 0.012310932, 0.04057012, -0.051034413, 0.012982438, 0.037548345, 0.027629651, -0.015122862, -0.021837916, 0.026664361, -0.027154, 0.011226731, 0.0062708813, 0.03780016, 0.050446846, -0.0060470463, 0.056378476, 0.0152068, -0.019207854, -0.027881464, -0.0323162, 0.0062079276, 0.008155993, 0.010883984, 0.007981122, -0.007470498, -0.0034799369, 0.03780016, 0.0037702231, -0.008792524, 0.0026895192, 0.0029308414, 0.04244474, 0.023754504, 0.0051307213, 0.0015003949, 0.012891505, -0.020117184, 0.012094093, 0.014829078, 0.0024307098, 0.007862209, -0.0023869919, 0.032036405, 0.004871912, -0.008086044, -0.0061065024, 0.001232842, -0.013353165, -0.010177504, -0.014451357, -0.0062638866, 0.0109329475, -0.017752925, -0.018298524, 0.008379828, -0.008463766, -0.029294424, -0.0013639954, -0.010058592, -0.022929111, -0.0032438608, -0.023124969, -0.035645746, -0.0072116884, 0.016186079, 0.03908721, -0.0009652891, -0.031448837, 0.0060085747, 0.011863262, 0.01174435, -0.0052811103, -0.014066639, -0.005319582, 0.027475763, 0.016479863, 0.020355009, 0.023418752, 0.009471024, 0.014815088, -0.020368997, -0.030133806, -0.013339176, -0.017137378, 0.025503216, -0.022341546, 0.017165357, 0.04613802, 0.05721786, 0.01709541, 0.024090257, 0.005434997, 0.024789741, 0.016172089, 0.0129124895, -0.008463766, 0.023236886, 0.015920274, 0.0038716486, 0.0562106, -0.010946938, -0.0054664738, 0.007897183, 0.0131363245, 0.013290212, -0.0054384945, 0.00897439, -0.023236886, -0.0021369262, 0.009387086, -0.020243092, 0.023838442, -0.028874733, -0.036261294, -0.003784213, -0.010513257, 0.025643114, 0.0023048024, 0.0030672408, -0.02029905, 0.05663029, -0.0044347337, -0.020243092, 0.010086571, 0.02396435, -0.010569216, 0.010114551, -0.0017670735, 0.031420857, -0.022775225, 0.018200595, -0.0045956154, 0.0047145276, -0.0066835773, 0.0017355967, -0.027937423, -0.024957618, -0.017878832, -0.01794878, -0.0016385432, 0.013542026, 0.004455718, 0.006704562, 0.025517207, 0.010352375, -0.0030672408, 0.0031109587, -0.016311986, 0.027559701, -0.0019393216, -0.004822948, 0.0198234, -0.0030602461, 0.021739988, 0.024719793, 0.026006846, 0.015990224, 0.015248769, 0.0003678853, 0.013388139, -0.018774172, -0.0012660675, 0.016465873, 0.0075194617, -0.011947201, -0.027098041, 0.0367929, -0.011072844, -0.014290475, 0.007645369, 0.015766388, 0.005861683, 0.018984018, 0.0211804, 0.01245083, 0.007288632, 0.03690482, 0.017375203, -0.024621865, 0.008883458, -0.01198917, -0.008603663, 0.005994585, 0.047704864, 0.023306834, -0.027937423, 0.029182507, 0.014150578, -0.007659359, 0.0011934959, 0.0057217856, 0.028846754, 0.0013666184, 0.021949833, -0.021907864, -0.014913016, -0.00920522, 0.01980941, 0.02122237, 0.018340493, -0.018340493, -0.022803206, -0.01930578, 0.02990997, -0.015318718, -0.011527509, 0.008694597, 0.013702908, -0.03553383, -0.00247093, 0.027685609, -0.007159227, -0.002016265, 0.018942049, -0.019949308, 0.010918958, -0.0099606635, 0.015024934, -0.016745666, -0.025167463, 0.017333234, 0.03307164, -0.069332935, -0.0006763144, 0.024118237, -0.0063233427, -0.009177241, 0.010471288, 0.017585048, 0.008575683, 0.025279382, 0.0150808925, -0.03309962, 0.0011357885, -0.031113084, 0.0043927645, 0.027853485, -0.02021511, 0.034806363, 0.022299577, 0.0033487836, -0.0031791585, -0.019207854, 0.0021509158, -0.01750111, 0.015556542, 0.021306308, 0.0031389382, 0.008365839, -0.049915235, -0.010695123, -0.0020302546, -0.011359634, -0.01699748, -0.021054493, 0.013730887, -0.0036583056, -0.018228574, -0.010324396, 0.019039977, 0.0035568804, -0.013052387, -0.011128804, 0.02811929, 0.013430108, 0.06731842, -0.0161581, -0.01038735, -0.0037562335, 0.00020175765, 0.05903652, -0.003777218, 8.0550046e-05, -0.026678352, -0.036597047, 0.0088764625, 0.0012433343, 0.00894641, -0.00761739, 0.008253921, 0.02029905, -0.03228822, 0.0068409615, 0.02028506, 0.0012809315, -0.026874207, 0.02816126, -0.024915649, 0.002124685, 0.0018553835, 0.03046956, -0.007981122, -0.0126536805, 0.004280847, -0.0071976986, -0.016885564, -0.039283067, -0.0007274642, 0.035226054, 0.0023607612, 0.045298636, -0.013003423, -0.0020250084, -0.017403182, 0.009659885, -0.001608815, -0.009282163, 0.0037492386, 0.014381408, -0.011506525, -0.009862736, 0.003735249, 0.0042843446, -0.009380091, -0.01266767, -0.020564854, 0.006151969, 0.001694502, 0.0031354406, 0.027755558, -0.020550864, 0.015990224, 0.041661315, -0.0020809674, 0.02028506, -0.02438404, 0.028902713, -0.014940996, 0.017766915, -0.010534242, -0.04096183, -0.011709376, -0.022943102, 0.004543154, 0.01657779, -0.015766388, -0.0107091125, -0.019193863, -0.026496485, 0.011814299, 0.019907339, -0.00715573, 0.031504795, 0.0058372007, 0.032623973, -0.0134371035, 0.0022908128, -0.011877252, 0.0029500772, 0.015094883, -0.021236358, 0.02122237, -0.015864316, -0.033015683, 0.044403296, -0.0049278704, 0.009834756, 0.013961717, 0.020117184, 0.006935392, 0.016325975, -0.018536348, 0.00874356, -0.001782812, 0.0007222181, 0.027349856, 0.0008524971, -0.009156256, 0.02760167, -0.020019256, -0.004791471, 0.006669588, 0.0035673727, 0.009492009, 0.01657779, 0.0040115453, 0.01243684, 0.017193336, -0.003917115, -0.0067080595, 0.021893876, 0.025307361, 0.046193976, -0.019123916, -0.025978867, 0.014150578, -0.0388354, 0.027251929, -0.004871912, 0.0006312851, 0.0047320146, 0.0012302189, -0.026538454, -0.05246136, -0.016521832, 0.00037313142, 0.0395069, 0.002782201, -0.014171562, -0.004060509, 0.024412021, 0.010562221, -0.021795947, -0.04143748, 0.015864316, 0.037184615, -0.0029710617, -0.017710956, 0.012730624, 0.0030200258, 0.007967132, 0.021977814, -0.019333761, -0.027000114, -0.046725586, -0.011590463, -0.048180513, 1.49050375e-05, 0.01568245, 0.006966869, -0.03500222, -0.00968087, -0.005945621, -0.012975443, -0.0019130909, 0.0011742602, 0.0002367319, -0.011534505, -0.010737092, 0.027979393, 0.003710767, -0.008603663, 0.016018203, 0.01129668, 0.021879885, 0.041745253, -0.04132556, -0.016060172, -0.0043647853, -0.0070962734, -0.022817194, 0.0066835773, -0.0004612228, 0.0060470463, 0.041577376, 0.015360687, 0.009750819, -0.010625174, -0.021572111, -0.011751345, 0.007124253, -0.002028506, 0.009009364, -0.00033859437, -0.026874207, -0.0013622467, -0.016409913, -0.020984545, -0.0014418131, -0.0024062279, 0.048208494, -0.029182507, 0.024495957, 0.062114254, -0.015948255, -0.015276749, -0.016311986, -0.035729684, -0.010800046, -0.027014103, 0.008393818, -0.016284006, 0.01266767, -0.01705344, 0.014437366, 0.011842278, 0.0053230794, -0.03363123, -0.027895454, 0.0281053, -0.018158626, 0.038975295, -0.005949118, 0.01129668, -0.0123529015, 0.014010681, -0.006952879, 0.028133279, 0.0025723553, -0.007470498, -0.005473469, -0.004462713, -0.0011795063, -0.0070193303, 0.017738935, -0.0028976158, -0.046613667, 0.032456096, -0.009240194, -0.0001331207, -0.009540973, -0.005490956, -0.00015793055, -0.0023415254, 0.011653417, 0.0009399327, 0.054979507, -0.023418752, 0.029434321, -0.022537401, 0.010856004, 6.803583e-05, 0.020173142, -0.047173258, 0.016242037, 0.011548494, 0.025209432, -0.030805312, -0.02669234, -0.043675832, -0.0033365425, -0.024635855, -0.0071976986, 0.0020862136, 0.00852672, -0.0350302, 0.026832238, 0.0013491313, -0.01888609, -0.00784822, 0.015514573, -0.01242285, 0.004029033, 0.0029605695, -0.0065471777, -0.016801625, 0.00017213884, -0.011373623, -0.019879358, -0.007855214, 0.029378362, -0.011821293, 0.036205333, 0.0056028734, -0.023334814, -0.023110978, -0.0304136, 0.018536348, -0.02951826, 0.040849913, 0.0062324097, 0.027951414, -0.00084637664, 0.026860217, -0.0008782906, -0.028636908, 0.00851273, 0.010457298, -0.022033771, 0.0144093875, 0.024286114, 0.0046970407, -0.029378362, -0.0084287925, 0.04879606, 0.0077712764, -0.016032193, 0.015794367, 0.0018973525, 0.023138957, 0.01891407, -0.014871047, 0.0433121, -0.007232673, 0.0014286977, 0.008037081, -0.020047234, -0.022817194, 0.010960927, 0.0030654923, 0.014262496, -0.0069738636, 0.0017452146, 0.01654981, 0.012185026, -0.013730887, 0.020984545, 0.026468506, -0.040290326, -0.00085337146, -0.011926216, 0.0021579107, 0.0433121, 0.035337973, -0.0145213045, 0.012947464, 0.015094883, 0.04101779, 0.0073585804, 0.0025898425, -0.022159679, -0.032092363, 0.0019235832, 0.03307164, -0.013241247, -0.03600948, -0.034386672, -0.045382574, -0.009072318, 0.000401548, 0.029098568, -0.008554699, -0.036205333, 0.0061204922, 0.02021511, -0.012094093, -0.00068505795, -0.027279908, -0.012632696, -0.010506262, -0.009023354, -0.009841751, -0.0013132828, 0.0022418487, -0.0396468, -0.02993795, 0.0048439326, 0.01744515, -0.0054384945, -0.015976233, 0.015346697, -0.010009628, -0.016787635, -0.007813245, -0.0028678877, 0.0012678163, 0.04043022, 0.022327555, 0.005046783, 0.00655767, 0.0065681622, -0.023334814, -0.041997068, 0.03399496, -0.00392411, 0.0006500837, -0.036680985, -0.026160732, -0.0017924298, 0.0064387578, -0.02347471, 0.004987327, 0.031560756, 0.001864127], "content_hash": "296ad426bb851a204edf35b61b90fa41"}, {"content": "# Attention Head Categories\n\nThis document explains the different types of attention heads found in transformer models. These categories are determined through **offline analysis** using TransformerLens and **verified at runtime** against your actual input.\n\n## Categories\n\n### Previous Token\n**Symbol:** \u25cf (active on most inputs)\n\nAttends to the immediately preceding token \u2014 like reading left to right. This head helps the model track local word-by-word patterns. It's one of the most common and reliable head types.\n\n**What to look for in the visualization:** Strong diagonal line one position below the main diagonal.\n\n### Induction\n**Symbol:** \u25cf when repeated tokens exist, \u25cb otherwise\n\nCompletes repeated patterns: if the model saw [A][B] before and now sees [A], it predicts [B] will follow. This is one of the most important mechanisms in transformer language models.\n\n**Requires:** Repeated tokens in your input. If no tokens repeat, this category appears grayed out.\n\n**Try this prompt:** \"The cat sat on the mat. The cat\" \u2014 the repeated \"The cat\" activates induction heads.\n\n### Duplicate Token\n**Symbol:** \u25cf when duplicate tokens exist, \u25cb otherwise\n\nNotices when the same word appears more than once, acting like a highlighter for repeated words. Helps the model track which words have already been said.\n\n**Requires:** Repeated tokens in your input.\n\n**Try this prompt:** \"The cat sat. The cat slept.\" \u2014 the repeated words activate duplicate-token heads.\n\n### Positional / First-Token\n**Symbol:** \u25cf (active on most inputs)\n\nAlways pays attention to the very first word, using it as a fixed anchor point. The first token often serves as a \"default\" position when no specific token is relevant.\n\n**What to look for:** Strong vertical line at column 0 (all tokens attending to position 0).\n\n### Diffuse / Spread\n**Symbol:** \u25cf (active on most inputs)", "source_file": "head_categories_explained.md", "chunk_index": 0, "embedding": [-0.0022736886, 0.038427796, 0.0031454058, 0.0048954072, -0.0046064765, -0.029313345, 0.011386501, 0.013658548, -0.025228914, 0.060412806, 0.015457799, -0.024191389, 0.014249542, 0.008904322, 0.012424025, 0.014748605, 0.008556292, 0.030495334, 0.034093834, 0.07533214, 0.002536353, 0.0024017373, 0.038033802, 0.046570394, 0.0418687, -0.005676834, -0.02839402, 0.045388404, 0.014814271, 0.010972804, 0.024795517, -0.0071510375, -0.044652943, -0.01336305, 0.0068161404, 0.025636043, -0.0069474727, 0.050773025, -0.010027212, 0.011596632, 0.0027070846, 0.0036017853, -0.02734336, -0.0016728437, -0.030784264, -0.014814271, 0.027238296, -0.017007519, -0.06051787, 0.045756135, -0.0018764086, 0.01375048, 0.0024050206, 0.05573738, 0.010985937, -0.023166997, -0.045309603, 0.034540366, 0.036379017, -0.025688576, 0.019529097, -0.0041927802, 0.03611635, 0.03215012, -0.002350846, -0.016889319, -0.068975665, -0.01179363, -0.011846163, 0.0058639823, 0.08095316, 0.027238296, 0.02440152, 0.003946532, 0.013894945, 0.007768299, -0.02933961, 0.023823658, -0.021275815, 0.00687524, 0.021761743, 0.00012404735, 0.0016449357, -0.04454788, 0.001085953, -0.004317546, -0.045257073, -0.059624813, -0.028709216, -0.032596648, 0.0134418495, 0.030469067, -0.023482194, -0.020120092, -0.043602288, -0.008162295, 0.009416518, -0.04357602, -0.024729852, 0.025675442, 0.035485957, -0.005624301, 0.01367168, -0.006172613, 0.03322704, -0.021485945, 0.03251785, -0.036037553, -0.022589136, -0.01652159, -0.061148267, -0.018924968, 0.012312393, 0.06303945, 0.06808261, -0.012496257, -0.019227032, 0.023810524, -0.0023360713, -0.04003005, 0.002695593, -0.042630427, 0.036983144, 0.031440925, -0.039347123, 0.00096200826, 0.02554411, -0.013592881, -0.034829296, 0.012075994, 0.0143677415, 0.008346161, -0.004048315, -0.02521578, -0.010697006, -0.03892686, -0.024243921, 0.034067567, -0.054529123, 0.011609766, -0.0064615435, -0.02807882, -0.012726089, 0.023600394, 0.008214829, -0.012791755, -0.02933961, 0.044836808, -0.010362109, 0.010033779, -0.0038611663, 0.0006533776, -0.03362104, -0.041054443, -0.040135115, -0.03385744, 0.015063802, 0.038532864, 0.01382928, -0.031887453, 0.008004697, -0.010112578, -0.010434343, 0.012883687, -0.06193626, 0.0026118688, -0.016455922, -0.0054929685, -0.02023829, -0.024742983, 0.00035028756, -0.023810524, -0.020041293, -0.03138839, 0.003588652, 0.0062973783, -0.051587284, -0.0034015037, -0.020448422, 0.0006997543, 0.036825545, 0.039347123, -0.02219514, -0.010020646, -0.008319894, -0.028577885, -0.028998148, 0.02035649, -0.023600394, 0.026923098, -0.043234557, 0.01962103, 0.05668297, 0.029759873, 0.0056374343, -0.028052555, 0.024480319, 0.0033095712, 0.025412777, -0.029182011, 0.02852535, -0.014013144, 0.032911845, -0.014617273, 0.013271118, 0.019410897, -0.030311469, 0.014065677, -0.028341485, -0.06251412, 0.02155161, -0.056893103, 0.018530972, 0.048855573, -0.061988793, -0.0028614, -0.034172636, -0.004701692, 0.013284251, 0.02465105, 0.04467921, 0.018701704, 0.033148244, 0.04076551, -0.054581657, 0.040450312, 0.04018765, -0.03545969, -0.03191372, -0.017362116, 0.01803191, 0.019923093, 0.030679198, -0.03440903, -0.041106973, -0.08111076, 0.020553486, 0.0058377157, 0.046202663, -0.046018798, -0.025478445, 0.008510326, 0.0006344986, 0.019883692, 0.00042518796, -0.007847098, -0.01489307, -0.036405284, 0.048330244, 0.045073207, 0.089673616, 0.0076697995, -0.06246159, 0.034855563, -0.03186119, 0.042656694, -0.016915586, 0.052401543, -0.03769234, 0.042709228, 0.0028909498, 0.042998157, 0.008621958, -0.028131355, -0.034093834, -0.043917485, -0.0034671698, -0.006737341, -0.03015387, -0.014446541, -0.048619177, 0.0095084505, 0.029024413, 0.005640717, -0.06540343, -0.009521583, -0.007893064, -0.015641663, -0.020540355, -0.009396818, -0.01489307, -0.014617273, 0.055422183, 0.040266447, -0.032255184, -0.00020756642, -0.002866325, -0.039399657, -0.009278619, -0.024099456, 0.03672048, -0.06524583, 0.040135115, -0.043260824, -0.014840537, 0.029681074, 0.020185757, 0.013632281, 0.009127587, -0.016166992, 0.041106973, -0.044731744, -0.016771121, 0.017651046, 0.030994397, 0.0010982654, -0.01448594, -0.022287073, 0.018228907, -0.035485957, -0.02970734, -0.051928747, 0.0019437163, -0.010355543, -0.031808656, -0.007971863, 0.041238308, 0.037324607, 0.010270177, 0.011452166, -0.04622893, -0.041606035, 0.040818043, 0.023902457, 0.03848033, -0.023285197, -0.03766607, 0.004980773, -0.016915586, -0.020527221, 0.040135115, -0.0023590543, -0.0027448428, 0.0062448457, -0.050510358, -0.02252347, -0.002318013, 0.029970005, -0.05484432, 0.023469063, -0.023941858, -0.0034835862, -0.033515975, 0.009757982, -0.05442406, 0.0342777, -0.016639788, 0.022759868, -0.011714831, -0.013461549, -0.046544127, -0.01485367, 0.031230794, -0.035118226, -0.021210149, 0.042709228, -0.03863793, -0.025110714, -0.010854606, 0.021472812, 0.0013395883, -0.0021653394, 0.013619148, 0.016416524, 0.0049512233, -0.017191384, -0.02554411, 0.0020783318, 0.015405266, -0.017335849, -0.00027107785, 0.024007523, 0.0013100386, -0.02978614, -0.042709228, -0.016534721, -0.02314073, -0.025557244, -0.012378058, 0.04310322, -0.00089223805, 0.0132185845, -0.06834527, -0.0022917467, 0.038243935, 0.037219543, -0.03215012, -0.023560993, -0.02897188, -0.0075581674, -0.03687808, -0.0071379044, -0.00038763517, 0.042998157, 0.0017483598, 0.04415388, 0.019358365, 0.029549742, 0.03569609, 0.03913699, 0.034592897, -0.020474687, -0.03532836, 0.01987056, 0.05350473, 0.038427796, 0.011629465, 0.008523459, 0.007459668, 0.01990996, 0.07485934, 0.04126457, 0.022667935, 0.01824204, -0.0029319911, 0.041474704, -0.00799813, 0.0030058655, 0.023757992, 0.031283326, -0.02444092, -0.07296816, -0.04859291, 0.0067045083, -0.07496441, -0.013540348, 0.029024413, -0.039084457, 0.013028153, -0.052979402, -0.0059559145, 0.053268336, 0.04688559, -0.004934807, 0.016692322, -0.029470943, 0.024046924, 0.028341485, 0.005279554, -0.026318971, 0.0037561005, -0.051482216, -0.011018771, -0.0228124, -0.03545969, 0.031020662, -0.020330222, -0.070394054, 0.045677334, -0.016744854, -0.016784253, 0.028866814, 0.016166992, 0.02122328, -0.012253293, -0.0073152026, 0.032491583, 0.014748605, 0.043759886, -0.0095938165, -0.010316143, -0.008766424, 0.011530966, -0.011386501, -0.00041923695, -0.005949348, -0.0003509032, -0.01843904, 0.040345248, 0.0038348997, -0.021472812, 0.017362116, 0.01415761, -0.046202663, 0.043943748, 0.0015234534, 0.032123853, -0.042998157, 0.022786135, -0.064825565, 0.0076894993, -0.013054419, -0.012272993, 0.0050924057, 0.021525346, 0.05019516, -0.00492824, 0.013579748, 0.015694195, -0.010821772, -0.0010391659, -0.017559113, 0.024454053, -0.014420274, 0.025071314, -0.0046130433, -0.025150113, -0.032123853, 0.006323645, -0.018412773, 0.050746758, -0.034093834, 0.030258935, 0.055632316, -0.011754231, 0.0020454987, 0.05957228, 0.025189513, -0.025202647, 0.03619515, -0.027474694, 0.02174861, 0.0024756116, 0.020514088, -0.023022532, 0.024414653, 0.009613516, 0.021197015, -0.017821778, 0.0023163715, 0.008530025, 0.0021472813, -0.010559108, -0.031729855, 0.017033784, 0.0027005181, -0.01681052, -0.07480682, -0.054581657, -0.023429662, 0.00948875, -0.01183303, -0.0077551655, 0.036431547, -0.011859297, 0.011176369, -0.017454047, -0.030206403, -0.04386495, 0.01938463, 0.007105071, -0.0003100671, 0.015142601, 0.004225613, 0.01415761, -0.00092671276, 0.021958742, -0.023403395, 0.007518768, 0.072075106, -0.03598502, -0.049275838, 0.010677307, -0.02933961, -0.028341485, -0.016258925, -0.011872429, -0.017716711, 0.0041665137, -0.035801154, 0.020960618, -0.058731753, -0.00011768595, -0.012660422, -0.014288942, 0.029103212, -0.00047854165, -0.0059000985, 0.022878068, -0.04402255, -0.016140725, -0.030416535, 0.016968118, -0.04307696, 0.011957795, 0.004481711, -0.01692872, 0.024335854, 0.0015497198, 0.014000011, -0.028000021, -0.0055586346, -0.020803018, 0.00496764, 0.0063991607, 0.0040450315, 0.0074662347, 0.008516892, -0.043943748, -0.011498133, 0.03023267, -0.0076107, -0.005561918, 0.032833047, -0.022667935, -0.015825529, -0.00032853568, 0.0021751893, -0.012719522, 0.020829285, 0.01179363, 0.022024408, -0.0022030973, 0.02174861, -0.019975625, 0.009127587, -0.024953116, -0.019371498, 0.0013707797, 0.039688587, -0.028131355, 0.004343812, -0.02583304, -0.009363985, 0.0027760342, 0.03995125, -0.032176387, 0.027395895, -0.005548785, 0.0071379044, -0.014906203, -0.012969053, -0.0055980342, -0.003460603, 0.007794565, 0.045073207, 0.038086332, -0.05484432, 0.021761743, -0.010092879, -0.0456248, -0.013251417, 0.046544127, 0.010329276, -0.017598514, -0.0026217187, -0.025018781, -0.010257044, 0.039636053, -0.0041369637, -0.049853697, 0.008720458, -0.022786135, -0.0067077912, 0.008247661, 0.025399646, 0.015037536, -0.0047345255, -0.018570371, -0.010513142, -0.01103847, -0.048409045, 0.0536098, 0.0055356515, -0.0013576464, -0.02027769, -0.010263611, 0.0029352743, -0.010040346, 0.023495328, -0.02068482, -0.011590065, 0.009134154, -0.04541467, -0.029444676, 0.002909008, 0.019555362, 0.026515968, 0.009672616, -0.027842423, -0.076435335, 0.02329833, 0.051482216, 0.014433407, 0.019410897, -0.0037889336, 0.0017844761, -0.010007513, -0.035223294, -0.009830214, -0.027553493, 0.018504705, -0.040634178, -0.04696439, 0.013619148, 0.0052434374, 0.035879955, -0.021026284, 0.022339605, 0.019647295, -0.0003486459, -0.01983116, 0.034172636, -0.02897188, 0.001204152, -0.0025905273, 0.016889319, 0.029470943, -0.023088198, -0.047358386, -0.026621034, -0.009429651, -0.01636399, 0.012167927, 0.015904328, -0.01876737, 0.029891206, 0.047594786, 0.009442784, -0.037876204, -0.04231523, 0.021814276, 0.016232658, 0.03742967, 0.005453569, 0.026699834, -0.031099461, -0.021499079, -0.014748605, -0.0045408104, -0.009757982, 0.005699817, -0.0060839634, 0.01216136, 0.008254228, 0.008411827, -0.018898701, -0.014170743, -0.032491583, -0.004616326, 0.0019913244, 0.010250477, -0.008444659, -0.036589146, -0.007124771, 0.031572256, -0.04454788, -0.0060609803, -0.02329833, -0.028472818, 0.035906218, -0.0050267396, 0.028131355, 0.003404787, 0.04165857, 0.01946343, -0.009994379, 0.012877121, -0.032097585, -0.020041293, 0.0014224917, 0.010920271, 0.018806769, -0.013198885, 0.011373367, 0.012818022, 0.0069540394, -0.036746744, 0.0019076, 0.024821782, -0.005847566, 0.0010802072, -0.0122073265, 0.0090881875, -0.008549726, -0.053058203, -0.038217667, -0.015720462, -0.005548785, -0.010316143, -0.010500008, 0.010197944, 0.03257038, -0.0228518, 0.03296438, -0.020921217, 0.025806775, -0.02521578, -0.03774487, -0.009941846, 0.020172624, -0.011314268, -0.027737359, -0.008589125, 0.0073940023, 0.011353668, -0.017086318, -0.026844298, 0.0061693294, -0.023455929, -0.012095694, -0.026318971, 0.029891206, -0.03133586, -0.020632286, 0.0011532608, -0.02329833, -0.09739595, -0.023101332, 0.010782372, -0.036799278, -0.013632281, -0.011826463, 0.03782367, 0.043707352, 0.042105097, -0.015116335, 0.010151978, 0.0025954524, 0.0018107425, -0.032360252, 0.033883706, -0.0421839, -0.0019716246, -0.022234539, 0.012693256, -0.004127114, -0.031099461, -0.02248407, 0.020461556, 0.009626649, -0.039110724, 0.0143414745, -0.007380869, -0.0068161404, 0.016272059, 0.03942592, -0.014433407, -0.0018008926, -0.028814282, 0.027474694, 0.011642599, 0.021696078, -0.0059624813, 0.021052549, 0.010664173, -0.01076924, 0.014919336, -0.031545993, 0.025649177, -0.0068883733, 0.023902457, 0.012240159, -0.010040346, -0.010316143, 0.02978614, 0.010500008, -0.01595686, 0.014013144, 0.014105077, 0.025018781, 0.029891206, 0.0016909019, -0.014919336, 0.011918396, 0.015405266, 0.0038874326, -0.0050956886, 0.009331152, 0.032228917, -0.0017582097, 0.0002630748, -0.0060215807, -0.009068487, 0.030547867, -0.021354614, -0.03740341, 0.011209202, -0.010979371, -0.015287067, -0.012542224, -0.0030058655, -0.014906203, 0.0021932474, -0.011360235, 0.017861178, 0.002002816, 0.08063796, 0.010388376, -0.00873359, 0.021998141, 0.031519726, 0.0341201, -0.010828339, -0.019450298, 0.013198885, 0.04165857, -0.0042715794, -0.030547867, 0.00803753, 0.009672616, -0.0229306, -0.011800197, -0.0037987835, 0.016574122, 0.05862669, -0.007860231, 0.04060791, 0.027448427, -0.00728237, -0.048566643, -0.040345248, -0.0056210174, -0.00087089656, -0.00031335038, -0.0020389322, -0.014840537, 0.03228145, 0.019765494, 0.036983144, 0.042499095, 0.0055159517, 0.00093327934, 0.03651035, 0.0073940023, 0.031519726, 0.008845223, -0.017296448, 0.03530209, 0.01307412, 0.006166046, -0.040082585, -0.013001886, -0.023166997, -0.004343812, -0.0065830257, -0.009495317, 0.018478438, 0.009206386, -3.2416222e-06, -0.020264557, 0.029917473, -0.016508456, 0.003621485, 0.012614457, -0.03215012, -0.0016449357, -0.01946343, -0.0082739275, 0.031283326, 0.004616326, -0.01599626, -0.0044488776, -0.01072984, -0.04157977, -0.03687808, 0.014932469, 0.026437169, 0.028893081, -0.011754231, 0.032491583, 0.007696066, 0.03619515, 0.009042221, -0.042210165, -0.015799262, 0.04638653, -0.0027514093, -0.0017368682, 0.066401556, -0.007518768, 0.023219531, 0.015339599, -0.025806775, 0.029996272, -0.0040286146, 0.0027842424, -0.007814265, -0.0020143073, 0.03664168, 0.039977517, -0.008530025, 0.023455929, -0.015457799, 0.01692872, 0.020816151, 0.020816151, -0.006550193, 0.046465326, -0.005269704, -0.0063400613, 0.014091943, 0.050273962, 0.0025609776, 0.035275824, 0.0039793653, -0.024756117, 0.029024413, 0.04733212, -0.015654797, -0.0046852757, -0.014170743, -0.037219543, 0.04806758, 0.007814265, -0.0054043196, 0.001497187, -0.018819902, 0.045729868, 0.02134148, -0.005450286, 0.0026381353, 0.047305852, -0.0024542701, -0.00086925493, -0.009554417, 0.014472807, -0.03272798, 0.027001897, -0.022024408, 0.00651736, -0.013894945, -0.0024477036, 0.028577885, -0.024296455, 0.05079929, 0.02831522, -0.052243944, -0.024611652, -0.01648219, 0.0014758455, 0.00958725, 0.03716701, 0.026621034, 0.025767375, -0.0017697013, -0.02207694, -0.018478438, -0.019581629, 0.007157604, 0.02562291, -0.008700757, -0.0028318502, 0.026844298, -0.0050694225, 0.018570371, 0.042735495, 0.016771121, 0.01930583, -0.004140247, 0.003962949, 0.011544099, -0.019266432, 0.013113519, 0.02970734, 0.011261735, 0.006543626, -0.016797386, 0.017624779, 0.03501316, -0.024572251, 0.0065698926, -0.0010219286, 0.031808656, 0.0051383716, 0.031362128, 0.0075581674, 0.040213916, 0.028577885, 0.0029960156, -0.03456663, -0.020212023, 0.00080810336, -0.031729855, -0.013632281, 0.007078805, -0.0041763633, -0.02329833, 0.03015387, 0.0018567088, 7.464388e-05, 0.0017106017, -0.010880872, -0.0014717414, -0.0151294675, 0.011846163, -0.029812407, -0.0059756143, -0.040450312, 0.012410891, -0.010861171, 0.013139785, 0.0064254273, -0.040397782, -0.014932469, 0.0006681525, -0.021722343, -0.011695132, -0.022103207, -0.0056932503, -0.009626649, -0.0036871512, -0.0074005686, 0.008845223, 0.021000016, 0.0027333512, -0.021893075, 0.010907138, 0.0065239263, 0.025924973, -0.015300199, 0.01189213, 0.005614451, 0.007682933, -0.07685559, -0.016941853, 0.0073020696, -0.0032373385, 0.027711092, 0.0114259, 0.06582369, -0.0113668, -0.015864927, -0.0027661843, -0.033962503, 0.01661352, -0.05521205, -0.0017713428, 0.036851812, -0.01448594, 0.015562864, 0.021985007, -0.006921206, 0.02314073, -0.01628519, -0.016626654, 0.00017145007, -0.013461549, -0.0044226116, 0.014814271, 0.0073349024, -0.02166981, -0.010631341, -0.031440925, 0.007702633, 0.0109137045, -0.024834916, 0.0072364034, 0.0122335935, -0.029943738, 0.016692322, 0.029864939, -0.005033306, -0.00687524, -0.012719522, -0.025058182, -0.005561918, 0.07538468, -0.01706005, -0.028682949, 0.021157615, 0.011839597, -0.0068818065, -0.013310517, 0.00044611903, -0.025596643, -0.016101327, -0.029076947, 0.025281446, -0.012844288, 0.015195134, 0.009646349, -0.014801137, -0.023876192, -0.0076566665, 0.044863075, 0.0016991101, 0.025255179, 0.025793642, -0.03088933, -0.0027727508, 0.020435289, 0.012240159, 0.017624779, 0.018819902, 0.007893064, 0.0004206734, -0.018609772, -0.023954991, -0.026305838, -0.008910889, -0.029812407, 0.03023267, -0.028288953, -0.010250477, -0.024926849, -0.0025199363, -0.011360235, -0.016941853, -0.0052598543, -0.015825529, -0.02407319, 0.029392144, -0.008687624, 0.00097103737, -0.010841472, -0.031362128, -0.00613978, 0.012351792, -0.0018879002, -0.0191745, 0.030101337, -0.031887453, 8.131309e-05, 0.02089495, -0.017046917, 0.010545975, -0.02378426, 0.04068671, 0.01684992, -0.02399439, 0.015484065, -0.033069443, -0.007078805, -0.019568495, 0.001837009, 0.021472812, -0.0017237349, -0.04312949, 0.011537532, 0.003923549, 0.01754598, -0.016061926, -0.011412767, 0.014433407, 0.012115394, 0.00380535, -0.00076049543, -0.01681052, -0.023022532, 0.009068487, -0.027080696, -0.010046912, 0.01640339, -0.053636067, -0.050142627, 0.032833047, 0.0012451933, 0.023797393, 0.035774887, 0.03884806, 0.017283315, -0.003729834, -0.013317084, -0.019489696, 0.021197015, -0.015562864, 0.008011264, -0.026056306, -0.017335849, 0.02603004, -0.004557227, 0.010434343, -0.036536615, 0.011760797, 0.0010284951, -0.03162479, 0.0044258945, -0.026056306, -0.012378058, 0.01034241, -0.008352727, 0.005840999, -0.013152919, 0.05757603, -0.016009394, -0.019450298, 0.040450312, 0.001638369, 0.0026972347, 0.029602274, 0.015103201, -0.020816151, 0.015536598, 0.002091465, -0.035407156, -0.0074925013, -0.010703573, 0.014472807, 0.015904328, -0.028630417, 0.008201695, 0.025452178, 0.021630412, 0.008648224, -0.03876926, 0.01382928, 0.03015387, 0.0026151522, -0.0105197085, 0.021971874, 0.047069456, 0.029234545, 0.0072889365, 0.022392137, -0.016652921, 0.02550471, -0.004324112, -0.021958742, 0.026844298, 0.021617278, -0.002748126, -0.04901317, 0.015838662, -0.00229503, -0.010368676, -0.015562864, 0.007748599, 0.0033883704, -0.010053479, 0.010434343, 0.009370551, 0.019489696, -0.010618207, 0.025360245, 0.032045055, 0.015523464, 0.014538473, -0.0066651083, -0.021249548, 0.019594762, 0.003933399, -0.0055553513, 0.006520643, 0.0044029118, -0.012391192, 0.02138088, -0.029182011, -0.004074581, -0.013868679, -0.0060609803, 0.00092671276, 0.010887438, -0.026791766, -0.013179185, -0.016245792, 0.014065677, -0.01403941, -0.03622142, -0.020303955, -0.0018517838, 0.016153859, 0.036037553, -0.011471867, 0.015759863, 0.033673573, -0.021499079, 0.012496257, -0.003460603, -0.0044849943, -0.003578802, -0.03304318, 0.008648224, -0.0021653394, 0.0022736886, -0.0073152026, 0.011931529, 0.00083642185, -0.022759868, -0.037088208, -0.026949365, 0.014774871, 0.0073349024, 0.035433423, -0.00040097357, -0.0024247204, -0.018675437, -0.0025741109, 0.007190437, 0.029943738, 0.0014750246, -0.02558351, 0.01303472, -0.040949374, -0.025872441, -0.025885575, -0.012056295, 0.035223294, -0.054161392, -0.011005637, 0.022759868, 0.014538473, -0.0003987163, -0.0090028215, 0.01971296, 0.018202642, 0.019489696, 0.018741103, 0.037639804, -0.016679188, 0.014354608, -0.005604601, 0.006940906, 0.01071014, 0.0014889786, -0.022050675, -0.0016687396, 0.010368676, 0.018189508, -0.035380892, -0.03175612, -0.018478438, -0.025557244, -0.0068883733, 0.0042190463, 0.017230783, 0.005860699, -0.04875051, 0.0050694225, -0.010224211, -0.01725705, -0.0051449384, 0.026975632, -0.023902457, 0.009521583, -0.024414653, 0.0042420295, -0.014761738, -0.01713885, -0.016298324, -0.005847566, 0.010841472, 0.03569609, 0.0057228, -0.013172618, -0.021236414, 0.025150113, 0.0017664179, -0.009771114, 0.01382928, 0.016180126, 0.008444659, -0.009567549, 0.0057031, -0.0031634641, 0.034671698, -0.015523464, 0.006789874, 0.029864939, -0.017887443, -0.025347112, -0.015838662, 0.008247661, -0.009153853, -0.02240527, -0.0496961, 0.050904356, -0.02064542, -0.032911845, -0.0097054485, 0.0025888858, 0.01448594, 0.01566793, -0.020474687, 0.041474704, -0.0014939036, -0.0027727508, -0.02068482, 0.01574673, -0.01994936, -0.019240165, -0.020015026, 0.002214589, 0.007098505, -0.035879955, -0.0038644497, 0.018058175, -0.014788005, 0.03162479, 0.041921236, -0.0008241095, 0.018636037, -0.011931529, 0.021945609, 0.020001892, 0.013376183, 0.01072984, 0.009501884, 0.033936236, 0.0454672, -0.009620083, 0.018399639, 0.00830676, -0.0022638387, -0.012837721, 0.02957601, -0.024165122, -0.019016901, -0.021105083, -0.030101337, -0.019686695, 0.0088911895, 0.029392144, 0.005476552, -0.020908084, -0.02750096, 0.00653706, 0.021446547, 0.024270188, -0.010106011, -0.023849925, 0.020803018, 0.05663044, -0.0016121026, -0.024007523, -0.0052729873, -0.03863793, -0.01587806, 0.003719984, -0.0065337764, 0.0008930589, -0.014669805, 0.0033653872, -0.0042912792, -0.04491561, -0.02362666, 0.001147515, -0.015011269, 0.02048782, 0.03832273, 0.013080685, -0.003347329, -0.015943727, -0.006757041, -0.04599253, 0.030731732, -0.016508456, 0.048986904, -0.025662309, -0.013310517, -0.010434343, 0.012752355, -0.033673573, -0.010388376, 0.02831522, -0.016009394], "content_hash": "50037ecb99b1af4e2346edf2c08e4a4a"}, {"content": "first word, using it as a fixed anchor point. The first token often serves as a \"default\" position when no specific token is relevant. **What to look for:** Strong vertical line at column 0 (all tokens attending to position 0). ### Diffuse / Spread **Symbol:** \u25cf (active on most inputs)\n\nSpreads attention evenly across many words, gathering general context rather than focusing on one spot. Provides a \"big picture\" summary of the input.\n\n**What to look for:** No strong patterns \u2014 attention is spread roughly evenly across all tokens.\n\n### Other / Unclassified\n\nHeads whose dominant pattern doesn't fit the categories above. These may perform more complex or context-dependent operations.\n\n## How It Works\n\n1. **Offline Analysis:** A TransformerLens script analyzes each head across many test inputs and assigns categories based on dominant behavior patterns.\n2. **Runtime Verification:** When you enter a prompt, the app checks whether each head's known role is actually active on your specific input.\n3. **Active vs Inactive:** A filled circle (\u25cf) means the head's role is triggered. An open circle (\u25cb) means the role exists but isn't triggered on your current input (e.g., no repeated tokens for induction).\n\n## Important Note\n\nThese categories are simplified labels based on each head's dominant behavior pattern. In reality, attention heads can serve multiple roles and may behave differently depending on the input.", "source_file": "head_categories_explained.md", "chunk_index": 1, "embedding": [-0.0027694267, 0.041920617, 0.025434293, 0.011016419, 0.04072551, -0.037691783, 0.00012676447, -0.0028977473, 0.0013703491, 0.0898474, 0.017083878, 0.011399466, -0.0033574032, 0.005048554, 0.012249829, -0.001600177, -0.00013227077, 0.001760099, -0.008687496, 0.06471955, 0.016624222, -0.013031244, 0.0023538212, 0.06422925, 0.035883803, -0.004703812, -0.03281943, 0.059264965, 0.026997123, 0.0031946085, 0.018876536, -0.019795846, -0.037293416, -0.020286147, 0.011751869, 0.008204858, 0.004960453, 0.043452803, 0.012510302, -0.0038132286, -0.03940783, 0.014241671, -0.023243265, -0.0017936156, -0.034106467, -0.0231973, 0.011116012, -0.020929664, -0.033554878, 0.043391515, -0.029310724, 0.022737645, 0.0143948905, 0.053749096, -0.0236876, -0.0034684867, -0.06144067, 0.0161186, 0.02073048, -0.016762119, 0.02007164, 0.011912748, 0.04020457, 0.030980807, -0.013804998, -0.0046272026, -0.09242148, 0.014096114, 0.020470008, 0.029724414, 0.098427646, 0.0073966295, 0.033003293, -0.009162474, -0.013942895, 0.008917324, -0.011200282, 0.014402552, -0.020056318, -0.017283062, 0.015076713, -0.011560346, -0.008794749, -0.06533242, 0.0036466033, -0.018125763, -0.034841917, -0.05142017, -0.047160693, -0.05083794, -0.011882105, -0.00048575093, -0.009760027, -0.008541939, -0.027027767, -0.016072635, 0.001118496, -0.008641531, -0.0135062225, 0.06508727, 0.038672384, -0.008043978, -0.01291633, -0.0199031, 0.043146364, -0.027380168, 0.021358676, -0.02895832, -0.019841813, -0.044709194, -0.08261549, -0.009698739, -0.019259581, 0.039836843, 0.0630035, 0.00094277336, -0.01644036, 0.012717146, 0.0046003894, -0.06422925, -0.019106364, -0.012073628, 0.022186058, 0.008074622, -0.05515871, -0.011483736, 0.017620143, -0.008924985, -0.02335052, -0.003682993, -0.01464004, -0.008595565, 0.031440463, -0.013759033, -0.038703024, -0.020439366, -0.026353605, 0.04201255, -0.050929874, 0.03717084, 0.011177299, -0.032543637, 0.020163571, 0.034811273, -0.013092532, -0.037538566, -0.045996234, 0.059173036, -0.031777542, 0.011575667, 0.0054507526, -0.00097581116, -0.047007475, -0.0029417977, -0.0049489615, -0.02492867, 0.021833654, 0.031624325, 0.0039453795, -0.024131935, -0.0136134755, -0.021251423, -0.017374992, 0.019765204, -0.042227056, 0.005443092, -0.008174214, -0.013123176, -0.039591692, 0.006159389, 0.007231919, -0.040327143, -0.04893803, -0.03404518, 0.0041254116, -0.024392406, -0.041399673, -0.019045075, -0.009920906, -0.013513884, 0.011085368, 0.035301574, 0.0020416381, -0.017145164, -0.01554403, -0.023059404, -0.022262666, 0.012403048, -0.028483342, 0.01751289, -0.021435287, 0.041338388, 0.082676776, 0.018799925, 0.0018979958, -0.034811273, 0.033524238, 0.0062704724, -0.0013406631, -0.030061495, 0.014034826, -0.013199785, 0.027686607, -0.012150237, -0.007952047, 0.024821417, -0.028345447, 0.009507216, -0.058376297, -0.031563036, 0.04253349, -0.05377974, 0.030122783, 0.0322372, -0.08586372, -0.017727396, -0.021726402, 0.0059602046, -0.0030911858, 0.031563036, 0.048478376, 0.0373547, 0.049367044, 0.0076685925, -0.051941115, 0.057334412, 0.06361637, -0.030046172, 0.008243162, -0.02779386, 0.04902996, 0.017727396, -0.0057303766, -0.054239396, -0.03447419, -0.06845808, 0.021588504, -0.015000104, 0.02730356, -0.050531503, -0.043544736, -0.011108351, 0.02114417, 0.0020492992, -0.0050294017, -0.0011098775, -0.014762615, -0.05240077, 0.045107566, 0.036251526, 0.07256434, -0.019811168, -0.04991863, 0.006852703, -0.03306458, 0.020408722, 0.00051088835, 0.05871338, -0.014218689, 0.045199495, -0.018845892, 0.038059507, -0.0032214217, -0.008312111, -0.037936933, -0.047313914, 0.014287638, -0.01339897, -0.01529888, -0.024239188, -0.028437378, 0.026782617, 0.008587904, -0.021419965, -0.05503613, -0.008764106, -0.0026008862, -0.023059404, -0.04722198, -0.00275602, -0.019091042, 0.008947968, 0.061379384, 0.022155413, -0.024315797, 0.022155413, -0.0003770615, -0.022124771, -0.013559849, -0.0130618885, 0.04072551, -0.03140982, 0.05362652, -0.0018845891, -0.014831563, -0.005443092, 0.022829575, 0.027977722, -0.023059404, -0.03001553, 0.043943103, -0.016869372, 0.00460422, 0.012793755, 0.008656853, 0.009744705, 0.010127751, -0.028942998, 0.0068067377, -0.024959315, -0.02295215, 0.0056690895, 0.022262666, -0.016685508, -0.049367044, -0.026843904, 0.05099116, 0.005849121, 0.014846886, 0.010066464, -0.04516885, -0.043054435, 0.05923432, 0.012449014, 0.028835746, -0.012410709, -0.0028441208, 0.0035680789, 0.007833302, -0.004152225, 0.04038843, -0.009660435, 0.01389693, 0.020546619, -0.033983894, -0.030061495, 0.008771767, 0.028897034, -0.027364848, 0.030413898, -0.06563886, -0.039009463, -0.022814253, 0.033554878, -0.036466032, 0.06888709, -0.011246247, 0.015030747, 0.02911154, 0.03288072, -0.036067665, -0.016624222, 0.0069063297, -0.03447419, 0.007837133, 0.009315693, -0.032543637, -0.05206369, 0.004424188, 0.0322372, -0.0005946798, -0.04317701, 0.024315797, 0.021925585, -0.00017632113, -0.02385614, -0.010104769, -0.011690581, 0.001800319, -0.03609831, 0.0034359277, 0.033585522, 0.02022486, -0.039377186, -0.038519163, -0.0029092387, -0.005163468, 0.004209682, -0.023994038, 0.07219662, -0.009583825, 0.0056805806, -0.061961614, 0.0010773184, 0.019611984, 0.046180096, -0.003642773, -0.0118438, -0.013705406, -0.01702259, -0.020623228, -0.009246744, 0.0143948905, 0.027778538, 0.012433692, 0.023978716, 0.012487318, 0.043943103, 0.040265854, 0.020607905, 0.061134234, -0.012142576, -0.05518935, 0.020132927, 0.023120692, 0.015375489, 0.019198295, -0.005749529, -0.0070863618, 0.046363957, 0.100572705, 0.044372115, 0.033677455, 0.008174214, -0.020868378, 0.040020704, 0.011223265, -0.0074464255, 0.03330973, 0.04991863, -0.03257428, -0.08898938, -0.047681637, -0.022140091, -0.046854258, 1.1147254e-06, 0.03677247, -0.027027767, 0.020761125, -0.04201255, -0.017742718, 0.06052136, 0.041644823, -0.010242665, -0.0146170575, -0.02764064, 0.020745803, 0.05616995, 0.012050645, 0.005898917, -0.0337081, -0.012318778, -0.043697953, -0.02713502, -0.048171937, 0.030444542, -0.03922397, -0.085986294, 0.033677455, -0.029892955, -0.013261072, 0.004615711, 0.024867384, 0.03989813, -0.004366731, -0.049121894, 0.029647805, 0.013483239, 0.048171937, 0.025158498, -0.033095226, 0.008051639, -0.0007924276, -0.012755451, 0.022140091, 0.013215107, -0.014931155, -0.010303953, 0.040756155, 0.03717084, -0.013130836, 0.03346295, 0.021619149, -0.041889973, 0.03214527, 0.00021833654, 0.0076187965, -0.048325155, -0.0016490155, -0.054484546, 0.005887426, -0.022875542, -0.0023978716, -0.011644616, 0.042686712, 0.087947495, -0.022032838, 0.027073732, -0.008664514, 0.0022159244, 0.010288631, -0.019995032, 0.025648799, -0.019841813, 0.046915546, -0.012410709, -0.0049068267, -0.029234115, 0.0064045386, -0.023626313, 0.04663975, 0.006059797, -0.009905584, 0.0534733, 0.016471002, -0.015444438, 0.07734477, -0.005190281, -0.04440276, 0.022094127, -0.038212728, -0.017804004, -0.013452596, 0.023258587, -0.02688987, 0.01973456, 0.008296789, 0.03447419, -0.016838728, -0.007009752, 0.0009863449, 0.006477318, -0.0019860964, -0.020086963, 0.0014957968, -0.008733462, -0.025771374, -0.06119552, -0.058897242, -0.03775307, 0.002953289, -0.0025108703, 0.008534278, 0.02361099, -0.030276, 0.00016866019, -0.02532704, -0.01916765, -0.019811168, 0.007143819, 0.00085946074, 0.011522041, 0.012242168, -0.018049154, 0.014877529, 0.015076713, 0.010955132, -0.011897427, 0.01316148, 0.037232127, -0.009422946, -0.046762325, -0.0054316004, -0.041430317, 0.005033232, -0.0010619967, 0.011054724, -0.025510902, 0.0070020915, -0.03726277, 0.007297037, -0.026997123, 0.019596662, -0.023090048, -0.0070518875, -0.0043475786, 0.0068565337, 0.008151231, 0.016762119, -0.030613082, -0.038427234, -0.05427004, 0.021021595, -0.005979357, 0.014341264, -0.02187962, -0.0063126073, 0.037845, 0.006488809, -0.005420109, -0.011989358, -0.036312815, -0.041399673, -0.0031984388, -0.0036082987, 0.016562933, 0.017451601, 0.028023688, -0.030659048, -0.004753608, 0.026276994, 0.009514877, -0.0021278237, 0.0373547, -0.028575275, -0.0046655075, 0.014915833, -0.028330125, -0.0074157817, 0.006094271, 0.01208895, 0.003610214, -0.01875396, 0.00723958, -0.017068556, 0.0061325757, -0.03422904, -0.02310537, 0.028605917, 0.0134909, -0.028988965, -0.00411009, -0.03001553, 0.0043207654, -0.00773371, 0.03266621, -0.022385241, 0.02532704, 0.0036887384, -0.018524133, -0.05856016, -0.019596662, -0.0135062225, -0.014846886, 0.0102809705, 0.061256807, 0.009943889, -0.035883803, 0.014241671, 0.005033232, -0.04342216, -0.0036983148, 0.048141293, -0.012035323, -0.017865293, -0.009530199, -0.026782617, 0.00060712884, 0.033616167, 0.0012841637, -0.014387229, 0.0052170944, -0.022768289, -0.0033459119, 0.026215708, 0.016838728, 0.0043475786, -0.002236992, -0.0050294017, -0.012517962, -0.013904591, -0.02764064, 0.05485227, 0.0008676005, 0.0020339773, -0.022002196, -0.017329028, 0.0052860426, -0.012387726, -0.007377477, -0.0061708805, -0.018616064, -0.010618051, -0.04136903, -0.045996234, 0.00017811665, 0.022707, 0.013268733, 0.0046693375, -0.028238194, -0.08635402, 0.033248443, 0.039836843, -0.003642773, 0.0044471705, -0.013176803, 0.0009418158, -0.03140982, -0.030490506, 0.012104272, -0.015735554, -0.0013952472, -0.037967578, -0.018891858, -0.0016394394, 0.021419965, 0.053841025, 0.0021584674, 0.03134853, -0.01168292, 0.0010619967, -0.008450007, 0.01743628, -0.0051366547, -0.012694163, -0.011774852, 0.02204816, 0.037569206, -0.032604925, -0.011215604, -0.0041675465, 0.02327391, -0.012050645, -0.0017735056, 0.009308032, -0.038519163, 0.02920347, 0.04044972, 0.007672423, -0.011721225, -0.03266621, 0.002072282, 0.017895935, 0.032267842, 0.0031026772, 0.021910263, 0.0020914343, -0.05558772, -0.019703915, -0.038457874, -0.006274303, -0.005500549, -0.009591486, -0.00070911495, 0.011897427, 0.008258484, -0.0023231774, -0.018861214, -0.04645589, 0.0016346512, -0.008151231, 0.0073506637, 0.0049872664, -0.04121581, -0.019933743, 0.042656068, -0.033769388, 0.0023921258, -0.03140982, -0.027594674, 0.01826366, -0.01866203, 0.027487421, 0.011330518, -0.01562064, -0.00279624, -0.0032558958, 0.03496449, -0.021833654, -0.00046181053, -0.0035355198, 0.016348427, -0.0007804574, 0.0024936332, 0.01867735, -0.00797503, -0.016072635, -0.003667671, 3.6598904e-05, 0.043238297, -0.001870225, 0.0023078555, -0.003610214, 0.008089944, 0.008656853, -0.0007799786, -0.03248235, -0.022385241, -0.0041943598, 0.014463838, -0.010671678, 0.0053626522, 0.02393275, -0.016808083, 0.036128953, -0.01587345, 0.023289232, -0.0059410525, -0.034014534, -0.009491894, 0.0013588577, -0.018723316, -0.035424147, -0.012157898, -0.0003433055, 0.01841688, 0.0056116325, -0.021619149, 0.0058606127, -0.015635962, -0.016578255, -0.019765204, -0.0015187797, -0.040817443, -0.020270824, 0.0037557716, -0.022967473, -0.11589457, -0.028130941, 0.0055235317, -0.02146593, -0.02220138, 0.00092362106, 0.046762325, 0.033003293, 0.031302568, 0.0069178212, -0.008174214, -0.018891858, -0.01719113, -0.02138932, 0.009805992, -0.05742634, 0.010518459, -0.018324949, -0.008978612, -0.0132840555, -0.048233226, -0.019213617, 0.010533781, 0.017620143, -0.04121581, 0.01274779, -0.0055235317, 0.00452378, 0.0077183885, 0.031440463, -0.021695757, -0.0019765203, 0.002384465, 0.011123672, 0.022844898, -0.0030854403, -0.013827981, 0.01619521, 0.028130941, -0.012104272, 0.021404643, -0.028820423, 0.023335198, -0.0063700643, 0.043789886, -0.00600234, -0.017298384, 0.008710479, 0.013291717, 0.027747894, -0.0067148064, 0.010893845, 0.013621137, -0.0035221132, 0.007990352, 0.024560947, -0.004661677, 0.013659441, 0.01850881, 0.019749882, -0.014195706, 0.015176306, 0.010602729, 0.016026668, 0.0034723172, -0.008518956, -0.0039530406, 0.032298487, -0.034841917, -0.02295215, 0.021772368, -0.0046501854, 0.011797834, -0.008863698, 0.026552789, 0.0056461063, 0.001805107, 1.5164754e-05, -0.0151686445, 0.014800919, 0.059418183, 0.0015589995, -0.00087286736, 0.014693666, 0.038457874, 0.02105224, -0.007607305, -0.030490506, -0.004393544, 0.0036485186, 0.005205603, -0.030750979, 0.003060542, 0.0049796053, -0.0383353, -0.0072663934, -0.00022312463, 0.017068556, 0.0279624, -0.014486821, 0.042747997, 0.016348427, -0.010028159, -0.03971427, -0.04201255, -0.0038860075, 0.0052439077, 0.005822308, -0.0017610566, -0.0076034744, 0.03478063, -0.0034531648, 0.010166056, 0.04020457, -0.023871463, -0.00053435, 0.029218793, 0.012104272, 0.021036917, -0.014800919, -0.031685613, 0.044801127, -0.01702259, -0.011453093, -0.040265854, -0.01012009, -0.017451601, -0.010303953, -0.0036255359, 0.001887462, 0.013912251, -0.0033171833, -0.021680435, -0.005339669, 0.024331119, -0.002022486, 0.0049144877, 0.02336584, -0.034903202, 0.017972546, -0.00074454676, -0.011223265, 0.03159368, -0.008250823, -0.026751973, -0.0040104976, -0.013690084, -0.033493593, -0.03824337, 0.006400708, 0.01735967, 0.018355591, -0.0010150734, 0.010641034, -0.000373231, 0.022998117, -0.0048493696, -0.044249542, -0.033738744, 0.043943103, -0.0062781335, 0.005232416, 0.053412013, 0.00028872135, 0.04044972, -0.015321863, -0.02517382, 0.031685613, -0.00020660575, -0.0068067377, -0.013238089, -0.013850964, 0.03579187, 0.015597656, 0.006710976, 0.027196307, -0.0021891112, 0.0051366547, -0.008028656, 0.02080709, 0.0017562686, 0.045812372, -0.040020704, 0.02031679, 0.018784603, 0.032053337, 0.0026545126, 0.0005669089, -0.033677455, -0.03165497, 0.014448517, 0.053350728, -0.00027196307, 0.0059831874, -0.0203934, -0.04170611, 0.04308508, 0.0048800134, 0.009055221, -0.004209682, -0.01924426, 0.017160486, 0.029724414, -0.006063627, 0.007346833, 0.036956333, -0.0030567118, 0.0054584136, -0.007967369, 0.0038687703, -0.048141293, 0.036557965, -0.012885687, 0.010840218, 0.010487815, 0.006044475, 0.0150613915, -0.0013713067, 0.032053337, -0.0034895544, -0.035240285, -0.033983894, -0.010717643, -0.0061057624, 0.0100588035, 0.042165767, 0.034259684, 0.013605814, -0.00073592825, -0.017650787, -0.01784997, 0.00026454154, 0.031026773, 0.016087957, 0.008120587, -0.011744208, 0.031072738, -0.0005338712, 0.01948941, 0.030321967, 0.020623228, 0.023503738, -0.025112534, 0.00097341713, -0.015643623, -0.0074464255, -0.014609396, 0.026568111, 0.016425038, 0.004228834, -0.002574073, 0.013536866, 0.03340166, -0.03701762, 0.002541514, 0.0063968776, 0.026828581, 0.020500652, 0.025020603, 0.01628714, 0.025020603, 0.02836077, -0.004960453, -0.015720231, -0.008365737, -0.0058184774, -0.029096218, 0.0055388534, 0.017129842, -0.023120692, -0.028130941, 0.037140194, 0.00025376835, 0.002039723, -0.024285153, -0.0036753318, -0.025970558, -0.016164565, 0.00084749056, -0.010533781, -0.008978612, -0.029801024, 0.016133921, -0.0010217767, 0.0031160838, -0.01110069, -0.03091952, -0.02911154, -0.00979067, -0.009430607, -0.007569, -0.0017581838, 0.00060712884, 0.005155807, -0.009775349, -0.0038840922, 0.016915336, 0.02952523, 0.030858232, 0.002204433, 0.000793864, 0.021848977, 0.024622234, -0.018692672, 0.003734704, -0.009246744, 0.0060176617, -0.083841234, -0.010962793, -0.017712073, 0.024622234, 0.008480651, 0.020132927, 0.055403855, -0.016639544, -0.035301574, 0.008495973, -0.03084291, 0.021848977, -0.029601838, -0.0032137607, 0.03380003, 0.009430607, 0.019857135, 0.02114417, 0.003644688, 0.0166089, -0.0034569954, 0.006818229, -0.007519204, 0.0028230532, 0.00259131, 0.0025032093, -0.0027847486, -0.03010746, 0.01003582, -0.019719237, 0.004481645, -0.007817981, -0.015436777, 0.015206949, 0.04228834, -0.012196203, 0.015107357, 0.035485435, -0.0066956542, 0.010794252, -0.028039008, -0.01061039, -0.008902002, 0.08145102, -0.016486324, -0.007883099, 0.02402468, 0.0062206765, 0.0006286752, -0.0025932253, -0.028897034, -0.026598753, 0.007492391, -0.03858045, 0.019673273, -0.005485227, 0.004711473, -0.002376804, -0.012441353, -0.018324949, 0.004159886, 0.03652732, 0.0063930475, 0.026185064, 0.047896143, 0.00057840033, 0.0077183885, 0.0044356794, -0.000101567195, 0.02862124, -0.005956374, 0.012311117, 0.0040028365, -0.013000601, -0.032421064, -0.021282068, -0.010932149, -0.029387332, 0.019029753, -0.023534382, 0.015482742, -0.027931755, 0.0009375065, -0.022645714, -0.011997019, 0.005699733, -0.009001594, -0.013598153, 0.0010926403, 0.00822018, 0.014471499, -0.004703812, -0.045383357, -0.027747894, 0.013345343, -0.0059333914, -0.020470008, 0.03891753, -0.039377186, 0.022507817, -0.0035106218, 0.009905584, -0.0068718554, -0.026522145, 0.015965382, 0.008404042, -0.013123176, 0.011047063, -0.050316997, -0.01579684, -0.017987868, -0.014188045, 0.0018137256, -0.0014201452, -0.03447419, 0.011154316, 0.004451001, 0.01332236, -0.015475081, 0.005010249, -0.010296292, 0.0020665363, -0.011698242, -0.01743628, -0.0049489615, -0.037845, 0.001390459, -0.0065309443, -0.006439013, 0.0074540866, -0.032604925, -0.04204319, 0.029663127, -0.022707, 0.021925585, 0.042625424, 0.050562147, 0.003725128, 0.0023404146, -0.0018185136, -0.008656853, 0.007017413, -0.038794957, 0.014938816, -0.008074622, -0.012985279, 0.0060904403, 0.004316935, -0.022094127, -0.042410918, 0.016915336, 0.008902002, -0.009798331, 0.0101967, -0.02187962, 0.020286147, 0.011568006, 0.0059219, 0.005075367, -0.019627307, 0.04458662, -0.01570491, -0.027089054, 0.02187962, 0.010097108, -0.0028268837, 0.028835746, 0.0012611808, -0.024698842, 0.024085969, -0.0077681844, -0.03257428, 0.0032463197, -0.00017356798, 0.01570491, 0.03438226, -0.028636562, -0.008626209, -0.0026296147, 0.02393275, -0.016394394, -0.01702259, 0.009905584, 0.024147255, 0.028146261, -0.0023078555, -0.010541442, 0.06373895, -0.011453093, 0.026920512, 0.012556267, -0.021496573, 0.037477277, -0.010357579, -0.02024018, 0.023136012, 0.03364681, -0.0052285856, -0.02928008, 0.001270757, -0.009606808, -0.0081665525, -0.009851958, 0.008089944, -0.009024577, -0.011797834, 0.012295795, 0.002129739, 0.0083887195, -0.023304554, 0.02385614, 0.032451704, 0.010970454, 0.032604925, -0.014425534, -0.015153322, 0.022676358, 0.015015426, -0.0045505934, -0.01357517, 0.007791167, -0.022645714, 0.0068680253, -0.014425534, -0.0076034744, 0.0014345094, -0.011016419, 0.0046923207, 0.008250823, -0.025189143, -0.020776447, 0.014364246, 0.018723316, 0.01069466, -0.031471107, -0.036067665, -0.005048554, 0.010143073, 0.0327275, -0.031333208, -0.0024285153, 0.013437274, -0.0058606127, 0.017466923, -0.0030873553, -0.017420959, -0.0032712177, -0.04244156, 0.027870469, -0.024867384, -0.008434686, 0.0026181233, -0.007327681, 0.00075938983, -0.016394394, -0.0086108865, -0.015000104, 0.02327391, 0.013015922, 0.017972546, 0.008863698, 0.010051142, -0.021557862, -0.0028422056, -0.012510302, 0.024239188, -0.0199031, -0.010579746, 0.0072932066, -0.021573182, -0.0231973, -0.028636562, -0.020944986, 0.016379071, -0.039836843, -0.010962793, 0.015099696, 0.0138356425, 0.015858129, -0.010464832, 0.02623103, 0.009231422, 0.018003188, 0.0076187965, 0.023396485, -0.0027828333, 0.015475081, -0.024193222, 0.015482742, 0.021864299, 0.0057303766, -0.015030747, 0.0017744632, 0.029157504, 0.016991947, -0.03257428, -0.0148392245, -0.015567013, -0.0393159, 0.016256496, -0.002279127, 0.011583328, 0.0010141158, -0.044372115, 0.016256496, -0.01316148, -0.021986874, 0.03183883, 0.020255502, -0.009438268, 0.0034953, -0.02014825, 0.027747894, 0.0006430394, -0.021251423, -0.022140091, 0.010832557, 0.012571588, 0.032298487, -0.0010370986, -0.022078805, -0.014027165, -0.004496967, -0.0048570307, -0.0035336046, 0.0127248075, 0.02993892, 0.038457874, -0.007703067, -0.00082355016, 0.003734704, 0.032206558, -0.023733566, -0.0055733277, 0.03478063, -0.0052745515, 0.000746462, -0.023090048, 0.005856782, -0.010097108, -0.020362755, -0.037446633, 0.025939913, -0.004393544, -0.02178769, 0.00071390305, 0.010870862, 0.014379568, 0.018355591, -0.0005908493, 0.017298384, -0.010556764, 0.0009940058, -0.003585316, 0.013130836, -0.026185064, -0.021665115, 0.00048790558, -0.024085969, -0.0008747826, -0.021021595, -0.021833654, -0.01077127, -0.01997971, 0.026261672, 0.017497567, 0.025005281, 0.02122078, -0.024438372, 0.011920409, 0.002359567, 0.005799325, 0.014318281, -0.0031850322, 0.027180985, 0.061900325, -0.01085554, 0.028268836, -0.008848376, 0.0010993436, 0.00080152496, 0.024346441, -0.020684514, -0.012563928, -0.010166056, -0.045843013, -0.0029513738, 0.006481148, 0.016670186, 0.0014632379, -0.0057265465, -0.041154522, 0.008243162, 0.01850881, 0.04087873, 0.0077107274, 0.002154637, -0.0009193118, 0.030873554, 0.017650787, -0.0246682, -0.03527093, -0.042135123, -0.0043207654, -0.001460365, -0.019504732, 0.030276, -0.029142184, -0.008679835, -0.00058701885, -0.044617265, -0.0013598154, -0.0028690188, -0.00015273983, 0.007821811, 0.033432305, -0.012073628, 0.014134418, -0.0018137256, -0.04351409, -0.040327143, 0.033187155, -0.026506823, 0.050531503, -0.0072932066, 0.007067209, -0.0102809705, 0.010893845, -0.0046999813, -0.010985776, 0.0060904403, 0.012809077], "content_hash": "e02bac998434341f85591475b9473c34"}, {"content": "# Interpreting Ablation Results\n\n## Quick Reference\n\nWhen you ablate an attention head and see the results, here's how to interpret what happened.\n\n## Probability Changes\n\nThe dashboard shows the immediate next-token probability before and after ablation (e.g., \"72.3% \u2192 45.1% (-27.2%)\").\n\n### Large Probability Drop (>10%)\n\nThe ablated head was **important** for this prediction. It was actively contributing to the model's confidence in the top token. This head likely plays a significant role in processing this specific input.\n\n**Example**: Ablating a Previous-Token head when the model is predicting a word that commonly follows the previous word (like predicting \"the\" after \"on\").\n\n### Small Probability Drop (1-10%)\n\nThe head has **some contribution** but isn't critical. Other heads or MLP layers may provide overlapping information. The model has some redundancy that compensates for the missing head.\n\n### Negligible Change (<1%)\n\nThe head was likely **redundant for this input**. It may serve a function that isn't relevant to this particular prompt, or other heads provide the same information.\n\n**Important**: This doesn't mean the head is useless -- it might be critical for other prompts. Try the same head with different inputs.\n\n### Probability Increase\n\nOccasionally, ablating a head can **increase** the probability of the top prediction. This means the head was actually pulling the model away from this prediction -- it was a \"competing signal.\" This is an interesting finding that suggests the head was promoting a different output.\n\n## Generation Changes\n\nThe full generation comparison shows whether the ablated model produces different text.\n\n### Generation Changed\n\nThe head was important enough that removing it altered the model's entire output sequence. This is a strong signal of importance. Look at where the texts diverge -- the point of divergence tells you where the head's contribution was most critical.\n\n### Generation Stayed the Same", "source_file": "interpreting_ablation_results.md", "chunk_index": 0, "embedding": [0.02914487, 0.013198973, 0.04447271, 0.03246865, -0.028032366, 0.009813388, -0.003222486, 0.01476472, 0.017182013, 0.046093397, 0.009167861, -0.032303832, -0.06202556, 0.035270512, 0.025367849, 0.0036259405, 0.033704765, 0.008639078, -0.007835602, 0.04329153, -0.027180819, -0.029254748, 0.021563359, 0.053537562, 0.024255345, -0.02955691, 0.022085274, -0.0021065478, 0.02567001, 0.014668577, -0.00450839, -0.01731936, -0.019667981, -0.02171444, -0.05425176, 0.00027361946, 0.0054423446, 0.009284605, -0.01928341, -0.0131509015, 0.012793802, -0.029282216, -0.031891797, 0.04444524, -0.025079422, 0.049417175, 0.045516543, -0.059278633, -0.041341215, 0.054993432, -0.016797444, 0.048620567, -0.034418967, 0.008934372, -0.020890363, -0.0010532739, 0.0068364087, -0.0058921534, 0.010616864, -0.052273978, -0.020176161, -0.0050337394, 0.060981728, 0.020533262, -0.0029168907, -0.032715872, -0.027139615, 0.016701302, 0.010582527, -0.026686372, 0.04345635, -0.017209483, 0.0033666994, 0.009339544, -0.02620566, -0.023362594, -0.0036980473, -0.011523349, -0.05801505, -0.02109638, 0.033677295, 0.023802102, 0.032770813, -0.03571002, 0.0071214023, -0.048730444, -0.01714081, -0.014229069, -0.010733608, -0.05977308, -0.016975995, -0.0012721695, 0.014902066, 0.00056097354, -0.034748595, 0.01622059, 0.004896393, -0.015588798, -0.021206258, 0.029584378, 0.0063144933, -0.0055831242, -0.004927296, -0.022552252, 0.027661532, -0.026480354, -0.04623074, -0.00047641975, -0.0026250298, -0.005703302, -0.019613042, -0.04452765, -0.038456947, 0.032221425, -0.02388451, -0.038072377, -0.044939686, 0.018528007, 0.026068315, -0.031699512, -0.024873402, -0.012725129, 0.0011022035, 0.0014816225, -0.0069703213, -0.0006326511, 0.04051714, -0.016206855, -0.008007285, -0.021590827, 0.031946734, 0.017374298, -0.0106512, -0.045022096, -0.067958914, -0.008151499, 0.024076795, 0.032880686, -0.057685416, 0.02477726, 0.011022035, -0.013954377, -0.024639914, -0.0068467096, -0.075925, 0.021316135, 0.026356742, 0.030353518, -0.023925714, 0.016083244, 0.007464768, 0.011866715, -0.007972949, -0.025793623, -0.006376299, -0.03389705, -0.0061428105, 0.014490028, -0.0023245849, -0.06488236, 0.027276961, 0.047137227, -0.027702736, -0.01708587, -0.025738684, 0.0358199, -0.01589096, 0.014462559, -0.02881524, -0.064717546, -0.04373104, -0.020698078, -0.018253315, -0.034089334, -0.033375133, 0.00030409315, -0.061311357, -0.010870954, 0.009882062, -0.03941837, -0.0031933, 0.013198973, -0.03535292, 0.030490864, -0.0072381464, -0.017511645, -0.03573749, 0.033375133, -0.039885346, -0.017072137, 0.023046698, 0.0049616327, 0.054498985, 0.006613221, -0.023417532, -0.028059835, 0.07587006, 0.008096561, -0.007320554, 0.0047659143, 0.036314342, 0.023939447, 0.018967515, -0.0019297146, 0.023953183, 0.010850352, -0.0070595965, 0.004237131, -0.03098531, -0.0358199, 0.017841276, -0.06900275, 0.01565747, 0.0049925353, -0.030023886, -0.013679685, -0.052686013, 0.058894064, 0.042522393, 0.025848562, 0.024584975, 0.013604145, 0.018225845, 0.048263468, -0.016193122, 0.028732833, 0.064607665, 0.0022919653, 0.015396513, -0.031150125, 0.010994566, 0.029282216, -0.0028173146, -0.019338349, -0.0020979636, -0.08152872, 0.04370357, -0.01639914, 0.032221425, -0.015066882, 0.0049719336, 0.0030765557, 0.048181057, 0.045791235, 0.033320196, 0.023994386, -0.022387436, -0.04263227, 0.024997015, 0.061366297, 0.028513078, 0.009202197, -0.044994626, 0.02207154, -0.02620566, -0.020286039, -0.02914487, 0.002492834, -0.021700704, 0.025093157, -0.02572495, 0.0044465843, -0.045434132, -0.0016807745, -0.0262606, 0.0002882125, 0.00053049985, 0.013253911, -0.07361758, -0.03395199, -0.029776663, -0.02370596, 0.005160785, 0.011736236, -0.033842113, -0.028513078, -0.024763526, 0.009827123, -0.03579243, 0.008384988, -0.013981846, -0.02334886, 0.03104025, -0.011626358, -0.024708588, 0.0043435744, -0.01696226, -0.024873402, 0.0031177595, -0.008934372, -0.0006609787, -0.01580855, 0.056366894, -0.022593455, -0.020533262, 0.006716231, 0.02299176, 0.020450855, -0.03428162, 0.018431865, 0.031672042, -0.026700107, 0.0077119907, 0.0037941898, 0.04090171, -0.0013768959, 0.028513078, 0.020849159, -0.024736056, 0.031562164, -0.018321987, 0.011798041, -0.024845934, -0.036259405, -0.06740953, -0.0180885, 0.048235998, -0.033704765, -0.0055144513, -0.0075265737, -0.005528186, -0.062190372, 0.002204407, 0.004607966, -0.01747044, 0.021810582, -0.0061084735, -0.0065102116, 0.0031538128, -0.02712588, 0.07658426, 0.016261794, 0.013714021, 0.037468053, -0.052054223, 0.015369044, -0.0422477, 0.04837334, 0.0052363253, 0.02720829, -0.043511286, -0.008330049, -0.018967515, 0.004072316, -0.015437717, 0.0017700496, -0.008082826, 0.013933775, 0.027661532, 0.02186552, -0.032496117, 0.014119193, 0.015272901, -0.027043473, 0.01246417, 0.07416697, -0.0016138182, -0.009442554, 0.013459931, -0.003351248, -0.02735937, 0.0023692225, 0.0023537711, 0.009250268, 0.015918428, -0.023664756, -0.050186314, 0.05068076, 0.013006688, 0.0041856263, -0.018692823, -0.009408217, -0.012203213, -0.019146064, -0.10581154, -0.023609817, -0.032578528, -0.013659083, -0.035078228, 0.037989967, 0.018582946, -0.016110713, -0.024008121, -0.05318046, 0.028458139, 0.011042637, -0.02881524, -0.011592022, -0.07938612, 0.038868982, 0.030380987, -0.013164637, -0.007073331, 0.026713843, -0.020698078, 0.06312433, -0.045434132, 0.020835424, 0.037440583, 0.05801505, 0.00773946, -0.0019417324, -0.04743939, -0.02955691, 0.01663263, 0.017635256, 0.0016447211, -0.00081978535, 0.02439269, 0.026425416, 0.067189775, -0.00592649, 0.033072975, -0.0393909, 0.0390338, -0.003649976, 0.060322464, 0.052740954, 0.009181595, 0.011832378, -0.020890363, -0.080759585, -0.015877225, -0.03213902, -0.04444524, -0.026864924, 0.006135943, 0.0077669295, 0.05834468, -0.068068795, -0.025381584, 0.032715872, 0.051916875, -0.041615907, 0.017484175, -0.035297982, -0.0049719336, 0.06378359, -0.03683626, -0.012065867, 0.018706556, -0.021137586, 0.00468694, -0.034116805, -0.05279589, 0.023692226, -0.022359967, -0.071859546, -0.0039658723, -0.048345875, 0.0048242863, 0.037385646, -0.032358773, 0.04636809, 0.0013322585, -0.056311954, 0.0780676, -0.008563537, 0.067574345, -0.012951749, 0.025024483, -0.040654484, -0.006935985, 0.03213902, -0.037001073, 0.0031194764, 0.026645169, -0.02138481, 0.019091126, -0.0051951213, 0.03205661, -0.013466798, 0.024639914, -0.03175445, 0.0023263018, -0.0019623344, 0.033759706, -0.01990147, 0.02753792, -0.048263468, 0.015781082, -0.013137167, 0.01547892, 0.010733608, -0.00073480234, 0.05911382, 0.0073411562, 0.0038147916, 0.007313687, -0.00848113, 0.001022371, -0.023596082, 0.0404622, -0.016234325, 0.05427923, -0.003591604, 0.0040654484, -0.013391257, 0.03502329, -0.030078825, 0.039308492, -0.017456707, 0.04916995, 0.050735697, 0.002364072, -0.0005347919, 0.030820495, 0.03650663, -0.038456947, 0.06488236, -0.010850352, -0.021755643, -0.011070106, 0.023101637, -0.021700704, 0.019970141, 0.022703333, 0.008542935, 0.02569748, 0.018651618, 7.2053124e-05, 0.0024121432, 0.00018649043, 0.014187866, 0.017484175, -0.0077600623, -0.044719934, -0.031205064, -0.017717663, -0.037193358, 0.042412516, -0.030326048, 0.0027503583, 0.03672638, -0.03315538, -0.024667384, 0.02192046, -0.014709782, -0.03672638, 0.0148333935, 0.019928938, 0.015286636, 0.012690792, 0.008604742, 0.028540548, 0.014517496, 0.03279828, -0.034885943, 0.01121432, 0.004261167, -0.0028877046, -0.021590827, 0.028458139, -0.012979219, -0.019324616, -0.0071694735, -0.029254748, 0.00030688298, -0.0020292907, -0.018431865, 0.006242386, -0.017030932, -0.0284032, -0.012155142, -0.02712588, 0.01969545, 0.0064278035, 0.023225248, 0.0029941478, -0.013041025, 0.0184456, -0.020450855, 0.021233728, -0.015011943, 0.027469246, -0.018582946, -0.009868327, 0.0111319125, 0.025903499, -0.015877225, -0.015025678, -0.03205661, -0.033732235, 0.01948943, -0.014819658, -0.018582946, 0.011049504, 0.008694016, -0.014393886, 0.009840857, 0.018994983, -0.03620447, 0.032523587, 0.05172459, -0.016028306, 0.029199809, 0.01606951, -0.012436701, -0.022236355, 0.002631897, 0.0059436583, 0.001731421, -0.015066882, 0.021192525, 0.0057754093, 0.021288667, -0.05867431, 0.01734683, 0.01075421, -0.0041753254, 0.00014303322, -0.028513078, -0.053427685, 0.0025803924, -0.0016181103, 0.026301803, -0.0012661606, 0.01642661, 0.015726143, -0.028650424, 0.01323331, -0.002206124, -0.00756091, -0.0037152155, 0.0009562732, 0.018047296, -0.00986146, -0.02646662, 0.0043435744, -0.0015288353, -0.03494088, 0.037303235, 0.009470022, 0.014668577, -0.016330468, 0.030051356, 0.008316315, 0.00720381, 0.009284605, -0.00955243, 9.598141e-05, 0.005538487, -0.010067479, 0.0072244117, -0.03713842, -0.0032259196, -0.0019692017, 0.016124448, -0.022002866, -0.026631435, 0.006812373, -0.027661532, 0.063399024, 0.032990564, 0.0027022872, -0.029117402, 0.0038731638, -0.025889765, 0.007911143, 0.013803297, -0.016866118, 0.032386243, -0.028059835, -0.014654843, 0.010362773, 0.02278574, -0.006383166, 0.011379136, -0.0028173146, 6.2556916e-05, -0.04340141, 0.009689777, 0.027881285, 0.015781082, -0.013830766, 0.042412516, -0.014064254, -0.0213024, -0.02848561, 0.013295115, -0.0039143674, 0.031479757, -0.053427685, -0.0070870654, -0.013404992, -0.016742505, 0.013075361, -0.031946734, -0.0020893796, 0.007938612, -0.010246029, -0.018788965, -0.0033357965, -0.015053147, -0.032743342, 6.1001043e-05, 0.00042191046, 0.03941837, -0.013006688, -0.05845456, 0.012319957, -0.008412457, 0.019118596, -0.01054819, -0.023101637, 0.0078493375, 0.007869939, 0.037742745, 0.014792189, -0.053812254, -0.026919862, 0.0146823125, -0.0020807954, 0.018679088, -0.02950197, 0.036149528, -0.02418667, 0.019970141, -0.017223218, -0.048730444, -0.0139749795, 0.022936821, -0.038127314, -0.027592858, 0.0073686256, -0.009332676, -0.012924281, -0.0180885, -0.026425416, -0.0015957915, -0.008213305, 0.010630598, -0.0020842291, -0.05579004, -0.0072793504, 0.029804133, -0.0035400991, 0.010197958, -0.017236952, -0.007272483, -0.002065344, -0.00656515, -0.006455273, 0.018170906, 0.0048242863, 0.0136728175, -0.008014153, 0.018019825, -0.048290934, -0.034089334, 0.0032739907, 0.011942255, -0.0018816434, -0.01660516, 0.02150842, 0.027139615, -0.0075059715, -0.013494267, 0.005857817, -0.0030490863, 0.019956408, -0.017401768, 0.023101637, 0.0316171, -0.019008718, -0.009421951, 0.018555475, 0.006870745, 0.003311761, 0.023156574, 0.0023606385, 0.004896393, 0.033979457, 0.0101224175, 0.011379136, -0.0205882, 0.036341812, -0.009840857, -0.029804133, 0.011200585, -0.010685537, -0.023156574, -0.02439269, 0.0042336974, -0.023266451, 0.009854592, -0.026809985, -0.036946137, -0.023266451, -0.020945301, -0.0064106355, 0.017388033, 0.02810104, -0.025271706, 0.0018061029, 0.011557685, -0.006935985, -0.12943509, -0.00018112535, 0.013089096, -0.011104443, -0.008233907, -0.013075361, 0.01356294, 0.0063316613, -0.0015399946, -0.013528604, 0.027551655, -0.016440343, 0.016509017, -0.007574645, 0.017195748, -0.012889944, 0.007856204, -0.02109638, 0.02240117, -0.02950197, -0.042110354, 0.023376329, -0.025930969, 0.010438314, -0.045379195, 0.026068315, -0.03315538, 0.006702496, 0.016124448, 0.036122058, -0.004240565, 0.0019468828, -0.0052328915, 0.015781082, -0.026494088, -0.003917801, -0.004498089, 0.0014361265, 0.0003573148, 0.0063797324, 0.058124926, -0.029419564, 0.019942673, -0.03972053, 0.019036189, -0.020931566, -0.026301803, -0.008748955, 0.014517496, 0.008886301, 0.008783292, 0.0036087723, 0.0073686256, 0.012052132, 0.018129703, -0.029749194, 0.00031847158, -0.019983876, 0.013061627, 0.032276366, 0.026809985, 0.04329153, 0.017786337, -0.015327839, 0.013315718, -0.0066475575, 0.012471038, 0.01747044, -0.02183805, -0.03458378, -0.025862295, 0.025477726, -0.024873402, -0.010623731, 0.0142016, 0.028732833, 0.046175804, 0.016412875, 0.0019382987, -0.0045736297, 0.06202556, -0.023870775, 0.0001167443, 0.01856921, -0.00015505102, 0.04532426, 0.004738445, 0.0042817686, 0.026562762, 0.029859072, 0.039528247, -0.029694255, 0.003303177, 0.01714081, -0.048620567, -0.0029254747, 0.027098412, 0.015602532, 0.03749552, -0.0003764145, 0.028595485, -0.0063866, -0.005696435, -0.004450018, -0.03364983, -0.011502747, -0.007492237, -0.02848561, -0.022181416, 0.017690195, 0.03246865, 0.041039053, 0.0047830828, 0.03312791, -0.0043367073, 0.011482145, 0.032111548, 0.002575242, -0.0102734985, -0.027661532, 0.00010590682, 0.025203034, -0.02064314, 0.0059367907, 0.022346232, 0.049664397, 0.0018352891, -0.009009913, 0.008790159, 0.02373343, -0.0012661606, -0.011090708, -0.038484413, -0.0006657, -0.0154239815, -0.0051264483, -0.009627971, -0.010108682, 0.008790159, 0.027084677, -0.040022694, 0.0053427685, 0.02242864, -0.014242805, -0.058124926, -0.026219396, -0.015259166, -0.029282216, -0.022978025, 0.027441777, 0.023609817, 0.026370477, -0.0072656157, 0.01246417, -0.0066784606, 0.03793503, 0.03104025, -0.030875433, -0.027963692, 0.042742148, 0.023568613, 0.030326048, 0.025065687, -0.008865699, 0.008288845, 0.01565747, -0.020533262, 0.004412248, -0.020656873, -0.0059608263, -0.02848561, -0.0052122897, 0.014696047, 0.018802699, 0.01877523, 0.020849159, -0.003220769, 0.020849159, -0.016151916, 0.027373103, -0.009435686, 0.014888331, -0.0026198793, -0.011798041, 0.01021856, -0.018431865, 0.018582946, 0.020505793, 0.021274932, -0.032990564, 0.012835005, 0.037770215, -0.003093724, 0.025601339, -0.024543772, -0.0034044697, 0.016838647, 0.025367849, -0.034446437, -0.022140212, -0.050598353, 0.049115013, 0.000785878, -0.019667981, 0.043483816, 0.025051953, -0.011791174, -0.0023503373, 0.0075471755, 0.023582349, -0.015602532, -0.0018387227, -0.00028563724, -0.005408008, -0.031205064, -0.0030576705, -0.015163024, -0.007451033, 0.013109698, -0.032578528, -0.038923923, -0.034006927, 0.0021803714, 0.027194554, 0.023211513, 0.019709185, 0.008089693, 0.004401947, 0.020093754, -0.006877613, -0.04477487, -0.010149887, -0.019819062, 0.013995581, -0.025271706, 0.04669772, -0.00883823, -0.0043710438, -0.011179984, 0.025354115, 0.046093397, -0.0138238985, -0.0422477, -0.031672042, -0.016412875, -0.015039412, 0.024832198, 0.03568255, 0.009648573, -0.030463394, -0.035078228, 0.010314702, 0.0060913055, -0.016302997, 0.023156574, 0.005366804, 0.04447271, 0.004669772, 0.05026872, 0.024571242, -0.013109698, 0.019049922, -0.009765317, -0.04345635, 0.013535472, 0.011543951, -0.021879256, -0.012711394, 0.0379625, 0.0033426639, 0.00848113, 0.033594888, -0.0012455586, 0.021769378, -0.0070801983, -0.029584378, 0.009792786, -0.00033070397, 0.022936821, -0.023719694, 0.018486803, 0.0005738497, 0.0012841873, 0.0023022662, -0.00020129807, 0.0015279768, -0.005909322, -0.0009888929, -0.002312567, -0.0068673114, -0.019709185, 0.033704765, 0.013013556, -0.017030932, -0.025326645, 0.013343186, -0.018129703, 0.023225248, 0.0026799683, -0.038841516, -0.0017700496, -0.017813805, 0.025573868, -0.016687566, -0.020546997, 0.013459931, 0.028595485, -0.055542815, -0.031507228, 0.025958437, -0.020134958, 0.02388451, 0.006554849, 0.03282575, -0.0078081335, -0.02477726, 0.008419324, -0.035435326, 0.01580855, -0.030738087, -0.009923265, 0.04584617, 0.011386002, 0.031287473, -0.016509017, -0.017456707, 0.0109121585, 0.009895796, -0.0049684998, 0.011207453, 0.009916398, 0.015616267, -0.016495282, 0.04480234, -0.017800072, 0.031287473, -0.007313687, -0.012958617, 0.019461961, -0.023829572, -0.017923683, 0.006465574, -0.025299177, -0.009964469, 0.020148693, -0.020656873, 0.0017803506, -0.0087420875, -0.0016035172, 0.018706556, 0.083231814, -0.057355788, -0.023939447, -0.028952586, -0.0014180999, -0.0049204286, -0.016728772, -0.017305626, -0.05724591, -0.021426013, -0.014270274, 0.04233011, 0.0064003346, 0.008075958, 0.0068604443, 1.4365021e-05, -0.018788965, 0.009662308, 0.028238386, 0.009209065, 0.018390661, 0.019544369, -0.03749552, -0.0027692434, 0.006723098, 0.023390064, 0.018720292, 0.007897409, 0.01856921, 0.025326645, -0.022634659, -0.054883555, 0.011296728, -0.0129654845, -0.0129998205, 0.00030302012, 0.00904425, 0.0065926192, -0.015053147, -0.004559895, -0.016138183, -0.008233907, -0.002312567, -0.0070870654, -0.009724113, 0.0048105516, 0.00468694, 0.004515257, 0.005497283, -0.032633465, 0.01111131, -0.005638063, -0.013178371, -0.018061029, 0.03818225, -0.032880686, -0.004092918, 0.04518691, -0.020244835, 0.0037220828, -0.01731936, 0.04669772, -0.016412875, -0.0054560793, -0.010781679, -0.0422477, -0.02334886, 0.019750388, -0.0050165714, 0.027606593, -0.04730204, -0.0042714677, 0.027963692, 0.003821659, 0.012539711, -0.008831363, -0.027579123, -0.021851785, 0.042522393, -0.027235758, -0.005802878, -0.054471515, -0.02914487, -0.002465365, -0.020698078, 0.004089484, 0.030655678, 0.012786934, -0.050790638, -0.0065308134, -0.009181595, 0.013988714, 0.017717663, 0.040379792, 0.008687149, -0.008694016, -0.020656873, -0.021206258, 0.0051813866, 0.0013837633, 0.0014799057, -0.007643318, -0.02064314, 0.007890541, -0.006352263, -0.003701481, -0.034446437, 0.003629374, 0.003660277, 0.0027984295, -0.0043641767, 0.010438314, -0.008343783, 0.011173116, -0.008130897, 0.031177595, 0.01714081, 0.055707633, -0.025848562, -0.032935627, 0.013006688, -0.01494327, -0.011530216, -0.021426013, -0.0023675056, -7.602329e-05, 0.010369641, 0.01586349, -0.005737639, 0.0077119907, -0.0013777544, -0.011056372, 0.02641168, -0.016275529, -0.020739282, 0.0020224233, 0.0057342052, 0.010561925, -0.044005733, 0.05103786, -0.013679685, 0.005226024, -0.039912816, -0.05296071, -0.0004613975, 0.038868982, 0.011351666, 0.014029917, -0.003983041, -0.022565985, -0.03285322, -0.018582946, 0.026329273, -0.007828735, 0.021398542, -0.040269915, -0.0051916875, 0.010342171, 0.024681117, 0.007155739, 0.020093754, 0.010809149, -0.02171444, -0.008728353, -0.0030576705, -0.02260719, -0.03095784, 0.019667981, 0.04559895, -0.00022297303, -0.023472471, -0.01762152, -0.0013614445, -0.009215932, -0.010342171, 0.0073480234, 0.0003146087, 0.014641108, -0.014036785, 0.024763526, -0.0072312793, -0.014270274, 0.014476293, -0.017964887, -0.012498507, 0.023060432, -0.0069016484, -0.008508599, -0.0061428105, 0.006723098, 0.011866715, -0.024997015, -0.002743491, 0.009421951, 0.028760301, 0.04233011, -0.002444763, 0.023101637, 0.023664756, -0.009305207, 0.01221008, -0.0128144035, -0.0036431088, -0.02370596, -0.026727578, 0.03104025, -0.0006373724, -0.022126477, -0.006503344, 0.02005255, -0.026397945, 0.00055024336, -0.0033460977, -0.025958437, 0.004302371, 0.00401051, 0.011351666, -0.015094351, 0.003519497, 0.0009185029, -0.025985908, 0.020862892, 0.030573271, -0.0044225487, -0.01749791, 0.016454078, -0.030106295, -0.025972173, -0.017511645, -0.019915204, 0.02038218, -0.036369283, -0.03216649, 0.012532843, 0.0102734985, 0.008639078, -0.025354115, 0.0022971157, 0.0077463277, 0.03062821, 0.019750388, 0.014077988, -0.010472651, 0.021673236, -0.014174131, 0.027689, 0.01494327, -0.019173535, -0.026645169, -0.009518094, 0.020945301, -0.005806312, -0.014888331, -0.005208856, -0.025436522, -0.041423623, 0.010671802, -0.0027778274, -0.021467216, -0.00053736713, -0.07938612, -0.011509614, -0.0033855846, 0.012203213, -0.0058269138, 0.04732951, -0.0008202145, -0.012457304, -0.023431268, 0.016124448, -0.0053977068, 0.0061805802, -0.010994566, -0.021467216, 0.028925117, 0.06367371, 0.005905888, -0.0052397586, -0.016536487, 0.0015348442, -0.010486385, -0.0009949018, 0.004796817, -0.017401768, -0.017278155, 0.019420758, 0.028004896, 0.0082545085, 0.029639317, -0.010987699, 0.009140392, 0.04005016, -0.0063213604, -0.006074137, 0.0080690915, 0.0163442, 0.012910546, -0.031205064, 0.0026559327, 0.010493252, -0.038429476, -0.015080616, -0.00029465058, 0.0010292383, -0.0028756869, 0.05147737, -0.014462559, 0.02010749, -0.01203153, -0.021398542, 0.0002957236, -0.005754807, -0.0109121585, 0.0070870654, 0.0180885, 0.011042637, 0.025917234, -0.016481549, -0.0057822764, 0.0036156396, -0.00055753987, 0.016385406, 0.011612624, -0.019832795, 0.0059814285, -0.005435477, -0.0063282275, 0.044719934, 0.0039967755, -0.023980653, -0.014064254, 0.019242207, 0.027043473, 0.0087077515, -0.0033272125, 0.0034164875, -0.0034645586, 0.00045109654, 0.016357936, -0.028513078, -0.0047727814, 0.0071694735, 0.0036877464, -0.013137167, 0.025216768, 0.015547593, -0.0036156396, 0.024845934, -0.016715037, 0.016509017, -0.0022421773, 0.0032945927, -0.029996417, 0.0021288665, 0.014064254, -0.032743342, -0.012285621, -0.005469814, 0.019599307, -0.04661531, -0.044939686, -0.005995163, 0.00026288928, 0.027070943, -0.0058200466, 0.013425594, -0.011550819, -0.062520005, -0.037825152, -0.0012781784, -0.0052534933, 0.002764093, 0.02988654, -0.010108682, -0.015355309, -0.022717066, 0.013961244, -0.017566584, 0.01874776, -0.014215335, 0.049087543, -0.02109638, 0.0138238985, 0.000725789, 0.024008121, -0.01790995, -0.005198555, 0.029666787, -0.0019932373], "content_hash": "f2efecf3a4321698359e809d348f9fcd"}, {"content": "text. ### Generation Changed The head was important enough that removing it altered the model's entire output sequence. This is a strong signal of importance. Look at where the texts diverge -- the point of divergence tells you where the head's contribution was most critical. ### Generation Stayed the Same\n\nEven if the probability shifted, the model still chose the same tokens. This means the head's contribution wasn't large enough to cross the decision boundary. The model is robust to losing this head for this particular input.\n\n## Multi-Head Ablation\n\nWhen you ablate multiple heads simultaneously:\n\n- **Additive effects**: If ablating heads A and B together has a bigger effect than either alone, the heads contributed independently to the prediction.\n- **Redundant heads**: If ablating both has about the same effect as ablating just one, the heads may have been providing the same information.\n- **Synergistic effects**: Rarely, ablating two heads together can have a much larger effect than the sum of their individual effects. This suggests the heads work together as a circuit.\n\n## Tips for Interpretation\n\n- Always compare ablation effects across different head categories on the same prompt\n- Try the same head on multiple prompts to see if its importance is consistent or input-dependent\n- A head's category (Previous-Token, Syntactic, etc.) gives you a hypothesis about why it matters -- ablation lets you test that hypothesis\n- Remember that ablation is a blunt tool: removing a head removes all of its functions, not just the one you're interested in", "source_file": "interpreting_ablation_results.md", "chunk_index": 1, "embedding": [0.055395074, 0.031762846, 0.065832645, 0.017442837, -0.0125194555, -0.0041215736, -0.007821144, 0.0053348355, 0.00547902, 0.03719263, 0.05111877, -0.033113256, -0.059193112, 0.030721901, 0.002945237, 0.0059221247, 0.014531009, -0.013771402, -0.003812104, 0.05303185, 0.0016431786, 0.0053629694, 0.024096435, 0.02954029, 0.04124387, -0.020270266, 0.029287087, 0.028977618, 0.052834917, -0.016261226, 0.009783463, -0.017259968, -0.01676763, -0.011766882, -0.061950207, 0.03032803, 0.021184606, 0.015164015, 0.007378039, 0.005486054, 0.046504855, -0.044704303, -0.059305646, 0.047067527, -0.02606579, 0.036376756, 0.047123794, -0.03477314, -0.020987673, 0.074160196, 0.0006461938, 0.06256915, -0.013722167, 0.027092665, -0.040512398, -0.0009486301, -0.014369241, -0.01862445, 0.013672934, -0.029512156, 0.0024704826, 0.009692028, 0.026361192, 0.020495335, -0.0007420239, -0.033225793, -0.014742011, 0.0060557593, -0.019918595, -0.0067590997, 0.025545316, -0.017302169, 0.013025861, 0.002783469, -0.031228306, -0.028977618, 0.020875137, 0.008489316, -0.027444335, -0.015304683, 0.05885551, 0.031228306, 0.057195626, -0.044563636, -0.025700051, -0.03632049, -0.039161984, -0.017274035, -0.026220523, -0.05342572, -0.034519937, 0.01664103, 0.0131946625, -0.0036503358, -0.026881663, 0.020495335, -0.0049233814, -0.022971092, -0.01593769, 0.013651834, 0.0027869856, 0.0016546078, -0.03288819, -0.0021451877, 0.015220283, -0.031622175, -0.036376756, 0.005014816, -0.03508261, 0.0028661115, -0.019355923, -0.03336646, -0.031678442, 0.022169285, -0.008383815, -0.04692686, -0.04189094, 0.035982884, 0.003977389, -0.02245062, -0.00095830107, -0.013926136, -0.00478623, 0.0004048602, -0.018272778, -0.011091676, 0.033478994, -0.01050087, -0.013447865, -0.016303426, 0.028808815, 0.028161742, -0.027050465, -0.03508261, -0.0653825, 0.0010550104, 0.0016774664, 0.031200172, -0.060937397, 0.029427756, 0.008025112, -0.013581499, -0.03648929, 0.0072162705, -0.059924588, 0.020213997, 0.009952264, 0.059361916, -0.009481026, 0.046983127, -0.010177333, 0.004234108, -0.0049515152, -0.020087397, -0.0070967027, -0.04366336, -0.024659108, 0.010655604, -0.0027711606, -0.062794216, 0.025390582, 0.022295885, -0.04234108, 0.00087214186, -0.03547648, 0.038599312, -0.021578478, 0.00044969816, -0.02931522, -0.03522328, -0.04085, -0.0442823, -0.030524965, -0.028597813, -0.02280229, 0.0059748753, -0.070334025, -0.021817613, 0.0101703, -0.043213222, 0.016964566, 0.044535503, -0.024926377, 0.035814084, -0.025278047, -0.041694008, -0.007905544, 0.021902014, -0.04523884, 0.00477568, 0.029343354, 0.0033672412, 0.04793967, 0.002162771, -0.0045154444, -0.035842218, 0.06594518, 0.012955527, -0.002076612, 0.005658372, 0.032325517, 0.013475998, 0.010634504, -0.0028063275, -0.016036157, -0.0049128314, -0.011844249, 0.021733213, -0.040484264, -0.034266736, 0.016387828, -0.064144626, 0.003921122, 0.039865322, -0.030440565, -0.027697537, -0.040878132, 0.03271939, 0.0127234245, 0.010704838, 0.020987673, -0.010613404, 0.03857118, 0.045013774, -0.040371727, 0.01329313, 0.059361916, -0.008362715, 0.00880582, -0.037080098, 0.04045613, 0.023491563, 0.008243147, -0.0057709063, -0.034913808, -0.077536225, 0.029877892, -0.0020432032, 0.032860056, -0.030299896, -0.019960795, -0.007518707, 0.03632049, 0.0238151, -0.006541064, 0.04824914, -0.027458401, -0.05269425, 0.006636015, 0.04701126, 0.058180302, 0.0006259728, -0.050415426, 0.025601584, -0.031200172, 0.003089422, 0.0048811813, 0.025700051, -0.009945231, 0.03032803, -0.045632713, -0.0008739002, -0.03260685, 0.028724415, -0.033000723, -0.0072514378, 0.0041321237, 0.005556388, -0.048614874, -0.029849758, -0.055845212, -0.019749794, 0.014432541, 0.00781411, -0.047883403, -0.010634504, -0.02429337, 0.0060663093, -0.05109063, -0.0036081353, -0.0019922112, -0.022816356, 0.05286305, -0.018849518, -0.019904528, 0.0134197315, -0.023632232, 0.016415961, -0.0011561154, -0.0027113766, 0.015079615, -0.039696522, 0.080574654, -0.04062493, -0.011429278, 0.030215496, -0.0049831653, 0.04287562, -0.028794749, 0.0034621921, 0.031312708, -0.020270266, 0.015248416, -0.0056970557, 0.024082368, -0.0021240874, 0.00882692, 0.010381302, -0.022253685, 0.012132619, -0.018216511, 0.0021152957, -0.033563394, -0.034801275, -0.06476357, -0.032184847, 0.031734712, -0.03539208, 0.0066922824, -0.014573209, -0.008418983, -0.034351137, 0.004399393, -0.0025355415, -0.0014260223, 0.044394836, -0.0114363115, 0.0015957031, -0.008440083, 0.004198941, 0.034829408, 0.00072927587, 0.013532266, 0.029062018, -0.03755837, -0.0077015753, -0.053003717, 0.05145637, 0.013222796, -0.0023438812, -0.05789897, 0.007001752, -0.018933918, 0.028147675, -0.0061331266, 0.020410933, -0.0026164255, 0.026614394, 0.03437927, 0.0069138343, -0.012484289, -0.00375232, -0.007117803, -0.029821625, 0.02755687, 0.08158746, -0.02015773, -0.016415961, 0.03446367, 0.0015455901, 0.010747039, 0.017147435, -0.0046244618, 0.02086107, 0.012041184, -0.01395427, -0.047911536, 0.038177308, -0.0057779397, 0.016922366, -0.01936999, -0.0034903258, -0.01333533, -0.0009846763, -0.10173113, -0.008665152, -0.051484503, -0.0017768132, -0.032184847, 0.033028856, -0.011443345, -0.024068302, -0.045717113, -0.05286305, 0.033394594, 0.013483032, -0.003836721, 0.012716391, -0.06408836, -4.060691e-05, 0.030693768, -0.004343126, 0.002141671, 0.030440565, -0.0064672134, 0.057336293, -0.038683712, 0.040118527, 0.035645284, 0.050556093, 0.02600952, 0.014559142, -0.054691736, -0.010690771, 0.043353893, 0.04802407, 0.003590552, -0.012041184, 0.030862568, 0.026445592, 0.070502825, 0.035335813, 0.032466184, 0.014144172, 0.030131094, -0.0011060025, 0.038599312, 0.041525207, 0.041750275, 0.005953775, -0.014826411, -0.08316295, -0.012385821, -0.026417458, -0.03001856, -0.028147675, 0.010606371, 0.008629984, 0.036207955, -0.044310432, -0.05260985, 0.016697297, 0.0442823, -0.028555613, -0.0038965049, -0.022084882, 0.0027359934, 0.062794216, -0.044169765, -0.034576204, 0.03421047, -0.011816116, -0.003977389, -0.037670903, -0.060149655, 0.013257963, 0.012202953, -0.05846164, 0.001881435, -0.059586983, -0.01672543, 0.005282085, -0.0074905735, 0.04917755, 0.0146857435, -0.031987913, 0.07573567, -0.0006989443, 0.055704545, -0.021254942, 0.02522178, -0.030750034, -0.0039246385, 0.013651834, -0.017639773, 0.022366218, 0.026909797, -0.01223812, 0.039077584, -0.004009039, 0.012153719, 0.011112776, -0.005612655, -0.029343354, 0.019932661, -0.027697537, 0.012097452, -0.008629984, 0.039274517, -0.04326949, 0.011795015, -0.02579852, 0.029849758, 0.0052082343, -0.028836949, 0.068027064, 0.015262483, 0.0036538525, 0.026164256, 0.010465702, 0.016387828, -0.019862328, -0.0010576479, -0.0168661, 0.057673898, 0.0062491777, 0.024405906, -0.005813107, 0.018765116, -0.03010296, 0.07219084, 0.0055352873, 0.004177841, 0.017105235, -0.007202204, 0.00033320743, 0.04476057, 0.032550585, -0.028766615, 0.039640255, -0.01918712, -0.033169523, -0.0038859546, 0.013321264, -0.02443404, 0.022703823, 0.011373011, 0.013075095, 0.02698013, 0.017822642, -0.0034498838, 0.0114363115, -0.01085254, 0.012076352, 0.0010752314, -0.02596732, -0.041581474, -0.006505897, -0.027134866, -0.021058006, 0.026375258, -0.02786634, 0.0035501097, 0.029906025, -0.030046694, -0.024701308, 0.015304683, -0.0052856016, -0.029877892, 0.017105235, 0.0050640497, 0.002222555, 0.0036433022, -0.017527238, 0.015149948, 0.002823911, 0.045098174, -0.03345086, 0.01906052, 0.0067590997, -0.0063968794, -0.0062491777, 0.02346343, -0.023941701, -0.028499346, -0.013553366, -0.020804804, -0.00089368166, 0.0035378013, -0.022703823, 0.023308694, -0.02681133, -0.009762363, -0.01047977, -0.037136365, 0.017020833, -0.00078730145, 0.050949965, 0.023829166, -0.019215254, -0.002542575, -0.026951997, 0.026797263, -0.018202445, 0.0025619168, 0.0069314176, 0.0011921617, 0.005563421, 0.022928892, -0.0077859764, -0.012202953, -0.023294628, -0.026037656, 0.015445352, -0.025292113, -0.024645042, 0.02346343, 0.020523468, -0.050499827, 0.018230578, 0.025812587, -0.030918837, 0.04543578, 0.039949723, -0.01773824, 0.016176825, 0.027022332, -0.0018427514, -0.010669671, -0.006214011, 0.02143781, 0.0030612883, -0.009445859, 0.018779185, -0.0055106706, 0.0076171746, -0.0373333, 0.018469714, 0.012146685, 0.00644963, 0.018469714, -0.009706095, -0.040343594, 0.008665152, -0.019482523, 0.018511914, 0.019609125, 0.00023935548, 0.03615169, -0.031059504, 0.003024363, -0.009150457, 0.011154976, -0.0030226046, 0.0069490015, 0.029793492, -0.0005758598, -0.028021075, 0.028274277, 0.012786726, -0.045379512, 0.035167012, 0.027444335, 0.013279064, -0.013806568, 0.009980397, 0.008039178, -0.010550103, 0.0074061723, -0.02319616, 0.0014312974, -0.0015719654, 0.0052856016, -0.0010523729, -0.018371247, 0.016401894, 0.0061120265, 0.017288102, -0.021606611, -0.02654406, 0.0140527375, -0.039190117, 0.06403209, 0.03032803, 0.019173054, -0.0203828, -0.011316744, -0.020987673, 0.010247667, 0.007117803, -0.017977376, 0.009115289, -0.025911054, -0.044788703, 0.00608741, 0.0105571365, -0.008411949, 0.021676946, -0.0050675664, -0.009649828, -0.045857783, 0.016486295, 0.051175036, 0.018905785, -0.004156741, 0.04374776, 0.007117803, -0.009663895, -0.028147675, 0.0060065254, -0.046110984, 0.03997786, -0.036995694, -0.017639773, -0.022942958, -0.023702566, 0.0065375473, -0.03353526, -0.007378039, 0.013862835, -0.0019078103, -0.037952237, 0.019158987, -0.0067837164, 0.008313482, -0.006650082, -0.013658867, 0.048643008, -0.015600086, -0.041018803, -0.0027957773, -0.025573451, 0.030215496, 0.007518707, -0.018652583, 0.0055317706, 0.010275801, 0.05902431, 0.023336828, -0.034519937, -0.023829166, 0.00712132, 0.01976386, 0.015811088, -0.008707352, 0.031397108, -0.022127083, -0.007574974, -0.004174324, -0.04374776, -0.025854787, 0.010972108, -0.007849277, 0.0034534005, 0.0038191373, -0.01756944, 0.0101491995, -0.0442823, -0.020847004, -0.0044662105, -0.0033619662, 0.015951756, 0.008102479, -0.03587035, 0.0018656099, 0.025657851, -0.014756078, -0.0038929882, -0.008693285, -0.0059115747, 0.01575482, -0.03305699, 0.011555879, 0.022942958, 0.0123154875, 0.012357688, -0.001244033, 0.031087637, -0.02003113, -0.0045365444, -0.004800297, 0.013222796, -0.0018779184, -0.0003191406, 0.008756585, 0.03992159, -0.014003504, 0.009396626, 0.013595566, 0.03271939, 0.016711364, -0.023210227, 0.008960554, 0.008123579, -0.017091168, -0.020748537, -0.0046033617, 0.00025430144, -0.005201201, 0.021226808, 0.002171563, 0.02553125, 0.034829408, 0.0042305915, 0.02094547, -0.03719263, 0.049290083, -0.025292113, -0.03226925, 0.01804771, 0.029062018, -0.0072162705, -0.047911536, 0.014087904, -0.015698554, 0.025742251, -0.024419973, -0.024841977, -0.015515685, -0.04296002, 0.016261226, 0.018033642, 0.022506887, -0.014868612, -0.003270532, 0.021254942, -0.0144606745, -0.13549146, -0.0038754046, 0.018497847, -0.036686227, -0.029793492, -0.0021240874, 0.028738482, 0.027050465, -0.007279571, -0.0077930097, -0.000348593, 0.006425013, 0.0015693279, -0.011872383, 0.026051722, -0.056632955, -0.009945231, -0.03446367, 0.031059504, -0.045210708, -0.033957265, 0.0045576445, -0.026572194, 0.00041760824, -0.05111877, 0.0021645294, -0.019482523, -0.0017108751, -0.019524725, 0.027697537, -0.027317734, -0.0062315944, -0.028133608, 0.027064532, -0.021803547, 0.016936433, -0.0025320249, -0.013518199, 0.003611652, -0.00050684455, 0.020298399, -0.03103137, -0.002823911, -0.022971092, -0.009150457, -0.00475458, 0.0007143299, -0.00237729, 0.009094189, 0.00038200166, -0.006164777, -0.006836467, -0.006027626, 0.028921349, 0.014516942, -0.03966839, -0.012055252, 0.013082128, 0.017935175, 0.02284449, 0.02271789, 0.045126308, 0.0062386277, -0.0046666623, 0.0074202395, -0.00479678, -0.010233601, 0.037164498, -0.035420213, -0.03376033, -0.01932779, 0.023843233, -0.016387828, 0.010810339, 0.02579852, 0.020185864, 0.028541546, 0.034238603, -0.009431792, 0.009382558, 0.059305646, -0.033788465, -0.007188137, 0.0238151, -0.024110503, 0.05083743, -0.0042763087, 0.012786726, 0.01936999, 0.04529511, 0.025629718, -0.048389807, -0.0034358168, 0.03705196, -0.05159704, -0.017878909, 0.012006017, 0.0063019283, 0.023843233, -0.005633755, 0.017133368, -0.008362715, -0.003801554, -0.047602065, -0.03511074, -0.0129695935, -0.002111779, -0.002823911, -0.01432704, -0.0011613906, 0.022324018, 0.020453133, 0.015262483, 0.03150964, 0.0073850723, 0.025109246, 0.027570937, -0.0024792743, -0.028119542, -0.028640013, -0.0051343837, 0.037023827, -0.016922366, -0.012139652, 0.0064390795, 0.029906025, -0.0140527375, -0.0008615918, 0.01761164, 0.02152221, -0.0051238337, -0.023224294, -0.03679876, 0.009804563, -0.0092911245, -0.016345626, -0.011773915, -6.2146704e-05, -0.0033461412, 0.014130105, -0.025714118, -0.021226808, 0.030721901, -0.029962294, -0.053903993, -0.05497307, -0.006579748, -0.011942717, -0.04718006, 0.0045119277, 0.016894232, 0.034323003, -0.00019495712, 0.004670179, 0.0011728198, 0.0078070764, 0.02848528, -0.021142406, -0.015965823, 0.04552018, 0.013159496, 0.02288669, 0.027247401, 0.003360208, 0.018919852, 0.008756585, -0.0030331546, 0.013370497, -0.01901832, -0.0028080859, -0.02249282, -0.01734437, -0.008531516, 0.0124983555, 0.015445352, 0.020790737, -0.009748296, -0.003216023, -0.011302677, 0.026529994, 0.011872383, 0.0046736957, -0.019918595, -0.023857301, 0.008580751, 0.0040864064, 0.02923082, 0.024602842, 0.03843051, -0.019496592, 0.0081657795, 0.03547648, -0.0025513666, 0.016753564, -0.020565668, -0.011956784, 0.024448106, 0.020059263, -0.031200172, -0.036545556, -0.06735186, 0.05168144, 0.010085898, -0.027064532, 0.0056513385, 0.036939427, -0.02782414, 0.0017047209, -0.00094687176, 0.0077789426, -0.009537294, 0.020804804, 0.011696548, -0.0012950252, -0.03494194, -0.009762363, -0.012301421, 0.013630734, 0.030356163, -0.021972349, -0.037839703, -0.026600327, 0.0062210443, 0.026093923, 0.00050640496, 0.008644051, 0.017485037, 0.00648128, 0.008411949, 0.0024265237, -0.03857118, -0.037755303, -0.019116787, 0.017499104, -0.015515685, 0.02615019, -0.030975103, -0.003910572, -0.008700319, 0.027374001, 0.04523884, -0.007891477, -0.03159404, -0.028893216, -0.00542627, -0.03508261, 0.017921109, 0.017259968, 0.004729963, -0.01941219, -0.047208194, 0.0011552363, 0.029793492, -0.022549087, 0.029652823, 0.0003804631, 0.03305699, 0.022338085, 0.04296002, 0.01567042, 0.0016396618, 0.020087397, 0.00019572639, -0.03648929, 0.013799535, 0.0050534992, -0.036433022, 0.0020432032, 0.02526398, -0.018427514, 0.003847271, 0.04476057, -0.010233601, 0.0306375, -0.010536036, -0.016626963, 0.014671677, 0.0015482276, -0.004483794, 0.0021803547, 0.036123555, 0.017414704, -0.0065480974, 0.0052539515, -0.00955136, -0.017752307, -0.0058201402, 0.009445859, 0.00957246, -0.021339342, -0.018779185, 0.0041426737, 0.009720162, -0.00044200537, -0.004096957, 0.0028608362, -0.0012800792, 0.029849758, 0.027162999, -0.04132827, -0.010507903, -0.013250929, 0.029258953, -0.04644859, -0.017456904, 0.010979141, 0.020213997, -0.07089669, -0.0036186855, 0.016415961, -0.0021258458, 0.03499821, 0.0042903754, 0.027430268, -0.013996471, -0.027416201, 0.0101703, -0.027767872, -0.004128607, -0.038965046, -0.004905798, 0.035335813, 0.015965823, 0.019665392, 0.0022454136, -0.050640497, 0.0077930097, -0.018244645, -0.009037922, -0.0020572701, 0.009656861, 0.006425013, -0.00712132, 0.031903513, -0.005851791, 0.031312708, -0.048474208, -0.012639023, 0.013187629, -0.012209986, -0.021409675, 0.023252428, -0.016458161, 0.009910064, 0.0018744017, 0.002894245, -0.008918354, -0.014376274, -0.0035254927, 0.017625706, 0.076523416, -0.059699517, -0.04310069, -0.020270266, 0.0004655233, 0.010943973, -0.012821892, -0.03305699, -0.04675806, -0.014214505, -0.034097932, 0.034660608, 0.018638516, 0.016008023, 0.019257456, 0.016795764, -0.025601584, -0.0035694516, 0.03837424, 0.0050499826, 0.033169523, 0.044535503, -0.03288819, -0.007659375, -0.0042622415, -0.00649183, 0.024785709, 0.031228306, 0.023519697, 0.026206456, -0.022000482, -0.06268168, -0.008658118, -0.02253502, -0.039780922, -0.0007938953, 0.017724173, 0.012027117, -0.019777928, -0.01742877, -0.02191608, 0.014826411, 0.0065234806, -0.004870631, 0.008299415, 0.000498932, 0.0052856016, 0.010437569, 0.010929907, -0.011569947, -0.016444094, 0.002628734, -0.019960795, -0.022014549, 0.034126066, -0.044366702, -0.008468216, 0.021058006, -0.0095161935, -0.00034309816, 0.003019088, 0.050387293, -0.017963309, 0.003064805, 0.0012554623, -0.04217228, -0.02007333, 0.023168027, -0.016190892, 0.029906025, -0.037839703, -0.016570695, 0.048839945, -0.0057603563, 0.007434306, 0.0064390795, -0.01910272, -0.029737225, 0.033394594, -0.006509414, 0.00042200414, -0.040287327, -0.032634985, -0.00646018, -0.017161502, -0.0044486267, 0.015459418, -0.0027711606, -0.029090151, 0.008510416, -0.0009266507, 0.014854546, 0.03904945, 0.055873346, 0.023055492, -0.022928892, -0.01593769, -0.00849635, 0.007975878, -0.007047469, 0.00221728, -0.012020084, -0.01918712, -0.008137646, -0.03195978, -0.016852032, -0.03873998, 0.01831498, 0.017161502, 0.0075609074, -0.0036819861, -0.012526489, -0.021508144, 0.02495451, -0.013679967, 0.018891718, 0.011380045, 0.042425483, -0.023407163, -0.03820544, -0.0035290096, -0.017063033, -0.02439184, -0.013757335, 0.0013178837, -0.01852598, 0.016247159, 0.01742877, 0.00038244124, -0.001464706, -0.020143664, 0.009706095, 0.021564411, -0.005268018, -0.01052197, -0.015965823, 0.009037922, -0.008088413, -0.032381784, 0.02170508, 0.007469473, 0.015768887, -0.03195978, -0.03671436, -0.0028625948, 0.027655337, 0.020523468, 0.010859573, -0.0022981642, -0.0043396093, -0.025671918, 0.013250929, 0.014073838, -0.006115543, -0.0057709063, -0.031087637, -0.0069701015, 0.01602209, 0.021184606, 0.0030964552, 0.027669404, 0.01093694, -0.013588533, 0.00018616536, 0.009607627, -0.006315995, -0.03226925, 0.00845415, 0.028625946, -0.018835451, -0.003836721, 0.0059854253, -0.013771402, -0.0014814103, -0.01593769, 0.00040266226, 0.011112776, 0.018582249, 0.011316744, 0.0063230284, 0.0039949724, -0.015993956, 0.01095804, -0.014432541, -0.015909556, 0.013581499, -0.013876903, -0.0035342847, -0.013215763, 0.009537294, 0.0030876636, -0.034041665, 0.011935684, -0.0046139117, 0.007989945, 0.041609608, -0.017175568, 0.020002997, 0.010690771, -0.025545316, 0.02923082, -0.02584072, 0.008222047, -0.009952264, -0.024068302, 0.0086159175, 0.020636002, -0.0078070764, 0.01852598, 0.009769396, 0.012751558, 0.008011045, -0.0037945204, -0.035673417, 0.01434814, 0.00646018, 0.018132111, -0.013229829, -0.009713128, -0.0155860195, -0.026881663, 0.00959356, 0.022197418, 0.010817372, -0.02187388, 0.01672543, -0.013743267, -0.02433557, 0.00041826765, 0.004599845, 0.013553366, -0.035504613, -0.025629718, 0.002393115, 0.010550103, 0.017020833, -0.04543578, -0.0019130853, 0.015445352, 0.015206216, 0.02848528, 0.028808815, -0.020453133, 0.018371247, -0.01335643, 0.03845864, 0.010289867, -0.034323003, -0.0021838713, 0.015909556, 0.027289601, 0.02685353, -0.020495335, -0.0058377236, -0.02852748, -0.034407403, 0.015656354, -0.01335643, -0.024152704, 0.0020414449, -0.06887107, -0.0041110236, -0.009938197, 0.0064074295, -0.0014436058, 0.02954029, -0.026951997, -0.013103228, -0.039190117, 0.022309951, 0.0028520445, 0.008608884, -0.014214505, -0.018568182, 0.024532506, 0.044029098, -0.0020484785, 0.016542561, -0.008362715, -0.00886912, 0.0076664085, -0.01058527, 0.0018585766, -0.0024687243, -0.013996471, 0.015951756, 0.014924879, -0.00014264618, 0.023955768, -0.014643543, 0.0004725567, 0.024152704, 0.014397374, 0.014130105, 0.0028327028, 0.0029135868, -0.0031140388, -0.01333533, 6.874052e-05, -0.008573717, -0.04529511, -0.029765358, -0.016612897, -0.014882679, 0.0041356403, 0.040737465, -0.009839729, 0.029652823, -0.0057111224, -0.019398123, -0.014924879, -0.002212005, -0.01751317, 0.013778435, 0.0105571365, 0.017836709, 0.024307437, -0.010725939, -0.025587518, -0.01050087, -0.0022594805, 0.016289359, -0.008447116, -0.009206723, 0.0026146672, -0.0101069985, -0.01505148, 0.03134084, -0.0052891183, -0.0013873386, -0.0037945204, -0.0025003746, 0.031678442, -0.017921109, -0.011738748, -0.02117054, 0.0010207224, -0.011281577, 0.010325034, -0.006903284, -0.020002997, -0.018413447, -0.008791753, 0.010824406, 0.01870885, 0.037389565, 0.010050732, 0.017189635, -0.04718006, 0.0073358384, -0.00614016, -0.0060452092, -0.0033725163, -0.011605114, 0.007821144, -0.0134197315, 0.00480733, -0.020706335, 0.03446367, -0.058011502, -0.029793492, -0.006080376, -0.0017170293, 0.019341856, 0.0015570194, 0.024138637, -0.0026006005, -0.04476057, -0.010022598, 0.013862835, -0.014981147, 0.02284449, 0.029455889, -0.02094547, -0.008735485, -0.005387586, 0.008172813, -0.024884177, 0.03260685, -0.030299896, 0.029596556, -0.022324018, 0.011140909, -0.029062018, 0.03015923, 0.01091584, -0.003717153, 0.02007333, 0.005872891], "content_hash": "2658a78bbeb4acf810866e700735391e"}, {"content": "# Interpreting Attention Maps\n\n## Quick Reference\n\nThe BertViz attention visualization in Stage 3 shows how tokens attend to each other. Here's how to read the patterns.\n\n## Reading the Visualization\n\nThe BertViz display shows:\n- **Left column**: Each token in your input (the \"query\" -- the token doing the looking)\n- **Right column**: The same tokens (the \"keys\" -- what's being looked at)\n- **Lines between them**: Attention connections. Each line shows how much one token attends to another.\n\n### Line Properties\n- **Thicker, more opaque lines** = stronger attention (the model focuses more on this connection)\n- **Thin, faint lines** = weak attention (some attention, but not much)\n- **No line** = very little or no attention between those tokens\n\n### Interacting with BertViz\n- **Single-click** a head square at the top to select/deselect it\n- **Double-click** a head square to view only that head (deselects all others)\n- **Hover** over a token or line to see exact attention weights\n\n## Common Attention Patterns\n\n### Diagonal Pattern (Previous-Token)\n\nYou see each token strongly attending to the token directly before it, creating a diagonal line of strong connections.\n\n**What it means**: This head tracks local word order. It's useful for bigram patterns -- sequences of two words that commonly appear together.\n\n**Looks like**: A staircase pattern of thick lines, each shifted one position to the left.\n\n### Vertical Stripe (First-Token / Positional)\n\nYou see many or all tokens attending to the same position (usually the first token), creating a vertical column of lines.\n\n**What it means**: The first token often serves as a \"default sink\" for excess attention. This is an artifact of the softmax function -- attention weights must sum to 1.0, so when a head has nothing specific to attend to, it sends attention to the first token.\n\n**Looks like**: Many thick lines all pointing to the same token on the right side.\n\n### Uniform / Diffuse (Bag-of-Words)", "source_file": "interpreting_attention_maps.md", "chunk_index": 0, "embedding": [-0.016242096, 0.03925803, 0.026767982, 0.0037803797, -0.042632356, -0.032005746, 0.018055169, 0.02918541, -0.032962643, 0.024891958, -0.016015463, 0.006729768, 0.016884226, -0.0083791595, -0.014643068, 0.019767513, 0.014303118, 0.03371809, 0.004501201, 0.02199608, -0.019163156, 0.00740967, 0.028631417, 0.05247834, 0.020006739, 0.005398294, -0.0023025377, 0.021744264, 0.035002347, 0.0016777205, -0.02038446, -0.0116464645, -0.014932657, -0.033290002, -0.0033711793, 0.013963167, -0.032232378, 0.0596299, -0.032333106, 0.003745755, -0.019780105, 0.029739406, -0.022927796, 0.028203331, 0.0061411494, 0.0109162005, 0.04580523, 0.01776558, -0.045931138, 0.069954336, -0.025332635, 0.046031866, 0.0024158545, -0.018483255, 0.012729271, 0.00798255, -0.03286192, 0.013887622, 0.053233787, -0.012565591, 0.057262834, 0.002204959, 0.035556342, 0.027120525, -0.023154432, -0.006175774, -0.06940034, -0.0019012067, 0.04640959, -0.018168485, 0.058924817, 0.02181981, 0.02151763, 0.0024473316, 0.04124737, -0.016909407, -0.016896818, 0.007623713, -0.015776237, -0.016720546, 0.043866254, -0.018118123, 0.0014314135, -0.027422704, -0.022059035, -0.06219842, -0.020535551, -0.05917663, 3.5682053e-05, -0.02068664, 0.0022175498, 0.003437281, -0.017400447, 0.011369468, 0.016884226, -0.027473066, 0.009965597, -0.024703097, -0.03638733, -0.014152029, 0.059428446, -0.0043532597, -0.0024190021, -0.035380073, 0.055903032, -0.031023664, 0.028102605, 0.0020223928, 0.0059271064, -0.012823702, -0.069299616, -0.023104068, 0.017790763, 0.03585852, 0.045125328, 0.01776558, -0.027724883, 0.0018665821, 0.048978105, -0.03195538, 0.007799984, -0.03031858, 0.033441093, 0.03860331, -0.016959772, -0.022940388, 0.015247426, -0.02068664, -0.005351078, -0.042733084, -0.00055399403, 0.0022222714, 0.013157357, -0.0121438, 0.0035002348, -0.03470017, -0.003991275, 0.014693432, -0.052981973, 0.00021483006, -0.0019562913, 0.00305641, 0.003981832, 0.0012874065, 0.0036198471, 0.020623686, -0.050212003, 0.040819287, -0.033214457, 0.0011481211, 0.0007684306, 0.022071624, -0.022537483, -0.05207544, -0.022235304, -0.020245962, 0.008712815, -0.0018303836, -0.016984953, -0.010041141, 0.04353889, -0.014189801, -0.009764144, -0.017501174, -0.057212472, 0.040466744, -0.035757795, 0.01971715, -0.01904984, -0.061644424, -0.0074852146, -0.031905018, 0.035657067, -0.028681781, 0.028656598, -0.014063893, -0.064615846, -0.009638237, -0.001499089, -0.037873045, 0.010821769, -0.019200929, 0.00010008681, 0.0018209405, -0.022726346, 0.0020066544, -0.03273601, 0.032559738, -0.028228512, 0.009720077, -0.025508905, 0.007611122, 0.047719028, 0.036865782, -0.01989342, -0.073731564, 0.029034322, 0.010928791, 0.013598035, 0.020623686, 0.007976254, 0.011929757, 0.03323964, -0.028127786, 0.00019466515, 0.008165116, -0.0016368005, -0.015524423, -0.017992215, -0.028908415, -0.001169368, -0.04301008, 0.053082697, 0.011249855, -0.05570158, 0.004164398, -0.032056108, 0.03663915, 0.0060561616, 0.047769394, 0.024350554, -0.004740426, 0.04384107, 0.022184942, -0.03963575, 0.03842704, 0.091056466, -0.031527296, 0.018105531, -0.021580584, 0.033315185, 0.029512772, 0.04429434, 0.017841125, -0.031250298, -0.051042993, 0.02574813, -0.019150566, 0.008347683, -0.012836293, -0.0066982913, 0.0071704453, 0.02476605, 0.015020792, 0.011847917, 0.023557335, -0.028782507, -0.020094873, 0.033164095, 0.037873045, 0.03583334, 0.008630975, -0.067083634, 0.02639026, -0.028429965, 0.021832401, -0.035606705, 0.051521443, -0.0030721487, 0.037495323, -0.005766574, 0.024010604, -0.012439684, -0.034448355, -0.015725875, -0.03976166, -0.012748158, 0.0075544636, -0.07947296, -0.025609631, -0.021089545, 0.019805286, 0.038301133, 0.021630948, -0.038855124, -0.009411602, 0.009978187, 0.0012850458, 0.003251567, -0.024501644, -0.032358285, 0.014315709, 0.057313196, 0.042028, -0.025370408, -0.0029430932, -0.005656405, -0.028732143, -0.0008837149, 0.005045752, 0.032156833, -0.03228274, 0.042103544, 0.016355414, -0.02916023, 0.013333629, 0.044697244, -0.009115719, -0.004236795, -0.031905018, 0.046208136, -0.06270205, 0.01301886, 0.038527764, 0.026339896, -0.0033082257, 0.010689566, -0.017337494, -0.007113787, 0.00508982, -0.038074497, -0.016493913, -0.021102136, -0.033441093, -0.0859698, -0.00863727, 0.03422172, 0.07342939, 0.0035537456, -0.016279869, -0.0044162134, -0.020094873, 0.02200867, 0.049179558, 0.017085679, -0.02149245, -0.026692439, 0.013333629, 0.0069689928, -0.006773836, 0.046334043, 0.0060183895, -0.010481819, 0.0052409093, -0.05182362, -0.022222714, -0.0039220257, 0.048801836, -0.016594639, 0.08184002, -0.047341306, -0.018911341, 0.0038181518, 0.009468261, -0.048197478, 0.027347159, 0.0029430932, -0.015927328, -0.018118123, -0.016821273, -0.03046967, 0.020585913, 0.014529752, -0.058874454, -0.005731949, 0.03064594, -0.030419307, -0.020913273, -0.02184499, 0.0050709336, 0.011092471, -0.017123451, 0.026264353, 0.017022725, 0.0094053075, -0.042430904, -0.008819836, -0.0037205734, 0.010884723, -0.020183008, 0.017652264, 0.018344756, -0.02363288, -0.024174284, -0.04872629, -0.014680841, -0.0039440594, -0.0022946685, -0.033466276, 0.0387544, -0.03898103, 0.001702902, -0.030520033, -0.021895355, 0.01531038, 0.0059869126, -0.016544275, -0.0020711822, -0.05983135, -0.026188808, -0.017375266, 8.749603e-05, -0.013799487, 0.056003757, 0.024325373, 0.021933127, -0.0029399456, 0.015184472, 0.016065827, 0.05278052, 0.03225756, 0.00067911483, -0.03487644, 0.0003051295, 0.003487644, 0.009682304, -0.019339427, 0.015990281, 0.0035380071, 0.020510368, 0.04076892, -0.015990281, 0.0035065303, -0.010047437, -0.026541349, 0.023695834, 0.012848884, -0.008599498, 0.020573324, 0.044797968, -0.01565033, -0.023015933, -0.035782978, 0.00019663246, -0.06436403, 0.0023104069, 0.052881245, -0.021051772, 0.017501174, -0.014970429, 0.006635337, 0.05832046, 0.02739752, 0.011306514, -0.00053117325, -0.034246903, 0.024841595, 0.03046967, -0.016859045, -0.036034793, -0.0046837674, -0.03764641, -0.046006683, -0.004482315, -0.0150711555, 0.0077810977, -0.016065827, -0.045553416, 0.050161637, -0.03187984, 0.006503134, 0.044546153, -0.0075985314, 0.04900329, -0.026944254, 8.9660076e-05, 0.051042993, 0.0064181467, 0.012848884, -0.042909354, 0.024816412, 0.008977221, -0.03046967, 0.03162802, -0.012320071, -0.00071964134, 0.0036544718, -0.021404313, 0.07685408, 0.015914736, 0.0017359528, -0.012653727, 0.029714223, -0.015234835, 0.009965597, -0.036286607, 0.016632412, -0.036236245, 0.0035443027, -0.0361607, 0.049229924, 0.022839662, -0.02427501, -0.0016950328, 0.020170419, 0.05197471, -0.028278876, -0.035228983, 0.029563135, 0.0030910347, -0.02704498, -0.040164564, -0.0028659746, -0.012861474, 0.008032913, -0.011627579, -0.02067405, -0.04416843, 0.013044041, -0.04839893, 0.031552475, -0.025861448, 0.01661982, 0.0167835, 0.027649337, 0.0023434577, 0.033264823, 0.018029986, -0.020623686, 0.04578005, -0.01824403, -0.023192203, -0.008121048, 0.018281803, -0.033592183, 0.03258492, 0.0033774748, 0.03240865, 0.00023922468, -0.0012299612, -0.007019356, -0.0021010851, 0.01531038, -0.007755916, 0.007674076, -0.011457603, -0.038905486, -0.011331695, -0.04802121, -0.029739406, 0.008819836, -0.031980563, 0.0003065066, -0.00042847975, -0.00092306104, -0.0008577464, -0.013723942, -0.027422704, -0.0145045705, 0.037016872, -0.007264876, 0.007674076, 0.02996604, 0.043589257, -0.005980617, 0.029009141, 0.008857609, -0.006210399, 0.028127786, 0.055500127, -0.031426568, -0.040416382, 0.0044067707, -0.0115898065, -0.020422233, -0.023658061, -0.030520033, -0.021404313, 0.0026897038, -0.04190209, 0.028077424, -0.035707433, -0.026919072, -0.020913273, -0.0125089325, 0.047844935, 0.03487644, -0.017576719, 0.022537483, -0.031703565, 0.0031146423, -0.01155833, 0.034926806, -0.03145175, -0.0026283239, -0.016745728, -0.036034793, 0.01661982, -0.018105531, 0.015335562, -0.045125328, 0.006937516, -0.02574813, -0.014605297, 0.022701163, -0.009801917, 0.031502113, -0.025534088, -0.017198997, -0.011772373, 0.060838614, -0.0057508354, -0.010318139, 0.038049314, -0.017627083, -0.011098767, -0.015411106, -0.014454207, -0.030394126, 0.032635283, 0.0043123397, -0.000718461, -0.0121563915, -0.002492973, -0.016670182, 0.057615377, -0.051445898, 0.0019767513, 0.0022380098, 0.036286607, -0.037873045, -0.029537953, -0.034448355, -0.021567995, -0.028203331, 0.028656598, -0.057615377, 0.0040982966, -0.016418368, 0.010311843, 0.01181644, 0.0050205705, 0.004875777, 0.0028455146, 0.0054643955, 0.04321153, 0.024526825, -0.030117128, -0.005011128, -0.036513243, -0.055852666, 0.0014219704, 0.020183008, -0.004614518, -0.0026456362, -0.0075481683, 0.0072963526, 0.014454207, 0.026163626, -0.0038559241, -0.0201956, 0.012647431, -0.019968966, -0.01645614, 0.01971715, -0.027145706, -0.001251208, -0.028278876, -0.02330552, -0.02052296, -0.023431428, -0.040189747, 0.043287076, -0.0045924843, -0.00080817024, -0.033038188, -0.037570868, -0.019767513, -0.006389817, 0.030368945, -0.008007731, 0.030444488, -0.0027321978, -0.043916617, -0.023758788, 0.005225171, 0.012231936, 0.014429025, 0.012553001, 0.0135350805, -0.052176163, 0.018911341, 0.029714223, 0.020787366, 0.0095941685, 0.010544772, 0.016657593, -0.03956021, -0.04774421, 0.018949114, 0.0060152416, 0.016833864, -0.039862387, -0.034951985, 0.020535551, 0.024564598, 0.030167492, -0.012483751, 0.019855648, -0.027825609, 0.006906039, -0.015260017, 0.044747606, -0.006336306, -0.017010134, 0.013094404, 0.025685176, 0.048449293, -0.030368945, -0.070155784, 0.010702156, 0.004151807, -0.013270674, -0.012559296, -0.014152029, -0.0042588287, 0.03963575, 0.03517862, 0.008876495, -0.042430904, -0.04545269, 0.0036859487, -0.020233372, 0.020900683, -0.024463871, 0.042103544, -0.037042055, 0.011614988, 0.0008726979, -0.0041895797, -0.027347159, 0.030998483, -0.031426568, -0.013635807, 0.028152969, 0.011306514, -0.039937932, -0.035304528, 0.03784786, 0.03273601, 0.0064905435, 0.054190688, -0.0029289285, -0.06582456, -0.019012067, 0.023972832, -0.026818346, -0.0010828065, -0.023746198, -0.018596571, 0.0011127095, -0.013774306, 0.0065723835, -0.0019625868, 0.0037174257, 0.018659525, -0.024149103, 0.0024898252, -0.05917663, -0.009059061, 0.0016320789, 0.009499738, 0.007195627, -0.010072619, -0.0016305051, -0.001743822, 0.011829031, -0.017236767, 0.035405252, 0.040517107, 0.016934589, -0.0075041004, -0.030343762, 0.0011394649, -0.0062355804, -0.002113676, 0.020258553, 0.006566088, 0.037721954, 0.02022078, 0.014441617, -0.005640666, 0.035077892, 0.020371871, 0.07428556, -0.0019326836, 0.05343524, -0.004186432, -0.010771406, -0.00813364, 0.0048285616, -0.019805286, -0.019855648, -0.04235536, 0.026289534, 0.012093437, -0.03817522, -0.05094227, -0.0006248171, -0.013069222, -0.03094812, -0.03631179, 0.015662922, -0.044546153, 0.0012142226, 0.0129936775, -0.004328078, -0.10576249, -0.025332635, 0.010563659, -0.046006683, -0.033944722, -0.002894304, 0.03729387, -0.024186874, 0.0032578625, -0.014920066, 0.00044854629, 0.007478919, -0.022550074, -0.013749124, 0.023972832, -0.033642545, -0.022184942, -0.00031162164, -0.0227893, 0.011369468, -0.044218794, 0.0071326727, 0.0012559296, 0.028253695, -0.01548665, 0.036437698, -0.026490986, 0.004712097, 0.03306337, 0.025445951, -0.027775245, 0.025043048, -0.00033109798, 0.04822266, 0.017085679, 0.017123451, 7.84956e-05, 0.060385346, 0.025508905, -0.008561726, 0.01335881, -0.036085155, 0.008889086, -0.032182015, 0.049431372, 0.009235332, -0.032156833, -0.021769445, -0.012364139, 0.016720546, -0.035002347, 0.0028895824, 0.0057130633, 0.021731675, 0.015134109, 0.0345239, -0.015537013, -0.024816412, 0.0125341145, 0.018168485, -0.07121341, 0.005388851, 0.05998244, 0.003877958, 0.016229507, -0.012320071, 0.009317172, 0.02671762, -0.007856642, -0.004164398, -0.0019783252, 0.004359555, 0.0068367897, -0.02509341, 0.009789326, -0.007932187, 0.03454908, 0.002760527, -0.0062544663, -0.010104096, 0.050715633, -0.014580115, 0.005517906, 0.023670653, 0.018621754, 0.014378662, 0.018395118, -0.02231085, -5.523218e-05, 0.049582463, -0.007976254, 0.0030501147, 0.00075977447, -0.0047781984, 0.02134136, -0.022738935, 0.025521496, 0.032307923, 0.09014993, -0.008114753, 0.03585852, 0.012842588, 0.0007294779, -0.033843998, -0.042607177, 0.011419831, 0.02003192, -0.019931193, 0.002740067, -0.03031858, 0.00258583, 0.013245493, 0.047819756, 0.012074552, -0.009839689, 0.028807689, 0.06239987, 0.023167022, 0.014731204, -0.025722949, -0.0056752907, 0.0070004696, -0.008221774, -0.020132646, -0.006292239, -0.011256151, 0.009644532, -0.010053732, 0.0003472299, -0.013711352, 0.0023733606, 0.018621754, -0.009745258, -0.0066605187, -0.017450811, 0.012842588, -0.014189801, 0.0345239, -0.011369468, 0.023519564, -0.03177911, 0.00781887, 0.035581525, 0.025269682, -0.01483193, 0.014718614, -0.009367535, -0.035279345, 0.016028054, 0.026566532, 0.05003573, 0.027750064, 0.03225756, 0.0015518127, 0.0007641025, 0.031804293, 0.0014227573, -0.04872629, -0.008246956, 0.04849966, -0.021139907, 0.014403844, 0.056154847, -0.0047435737, 0.036362153, -0.003950355, -0.026591713, 0.0035443027, -0.0047561643, 0.034901623, 0.0061474447, -0.018508436, 0.033692908, 0.04562896, 0.014920066, 0.01970456, -0.0033837701, -0.013598035, 0.03177911, 0.02231085, 0.0051118536, 0.027598973, -0.02981495, 0.005766574, 0.02378397, -0.007995141, -0.009336058, 0.0064401804, -0.008612089, -0.04787012, 0.028555872, 0.02165613, -0.034397993, 0.016632412, 0.015234835, -0.0074222605, 0.01694718, 0.0025071376, 0.0071767406, 0.034095813, 0.0023938208, 0.005993208, 0.02199608, -0.02280189, -0.011199492, 0.012376729, 0.004759312, -0.018042577, 0.00035116452, 0.0033082257, -0.02166872, 0.027447885, -0.039862387, -0.00026971797, -0.0050331615, 0.007850347, -0.0050237183, -0.017249359, 0.0057571307, 0.022348622, -0.042405725, -0.03399509, -0.01139465, 0.009638237, 0.0046900627, 0.032811552, -0.0075041004, -0.015335562, -0.018583981, -0.008542839, -0.018193666, 0.019125383, -0.0161036, 0.011300218, 0.0032295333, 0.0055525308, 0.03809968, 0.0038496288, 0.029361682, 0.0114827845, 0.04142364, 0.0066038603, 0.0031020516, 0.010230003, -0.029537953, -0.041751, 0.0069941743, -0.011961234, 0.0046554385, 0.022449348, -0.025017865, 0.01565033, 0.019163156, 0.00014813831, -0.02086291, -0.0054738387, 0.030822212, 0.023028523, 0.008372864, 0.037747137, 0.039006215, 0.036588784, 0.0018697297, -0.039837204, -0.01742563, 0.009953006, -0.02948759, -0.019238701, 0.00749151, 0.025269682, -0.017639672, 0.03341591, 0.010828065, -0.007460033, 0.013459536, 0.024665324, -0.00398498, -0.0046396996, 0.046661403, -0.02623917, 0.010576249, -0.0485752, 0.030268218, -0.034120996, 0.026944254, 0.018004805, -0.026566532, -0.031577658, 0.017853716, -0.028329238, 0.0098900525, -0.0055556786, 0.015411106, -0.021102136, -0.016368005, -0.004350112, -0.032685645, -0.0013354089, -0.0022096806, -0.032383468, 0.010362206, 0.033315185, 0.038477402, 0.0034341333, 0.00266767, 0.016091008, 0.03485126, -0.025445951, -0.018029986, 0.018181076, 0.023431428, 0.034322448, -0.019490516, 0.030520033, -0.008196593, -0.011545739, 0.025005275, -0.03162802, 0.016796092, -0.02445128, -0.00521258, 0.058874454, -0.014630478, 0.01075252, 0.0007275106, -0.006553497, 0.0042210566, -0.0031791702, 0.013106994, 0.0072145127, 0.011526853, 0.013799487, 0.00749151, 0.020170419, -0.04170064, -0.0019657344, -0.006021537, -0.011665351, 0.027095344, -0.031351026, 0.030066766, 0.010041141, -0.007932187, -0.006122263, 0.04447061, -0.0076866667, 0.025156364, 0.00034290183, -0.026667258, 0.019251293, 0.06204733, -0.040844467, -0.017967032, 0.018445482, 0.003478201, 0.012181573, -0.0074852146, 0.01180385, -0.03930839, -0.0260629, -0.0074222605, -0.0052157277, 0.0011394649, -0.03988757, 0.009984483, -0.030041585, -0.020925865, 0.023519564, 0.03845222, 0.03112439, 0.018659525, 0.029462408, -0.0401142, 0.0046522906, 0.009077948, 0.028908415, 0.0045484165, 0.0029682748, 0.010148163, 0.026667258, -0.017967032, -0.011344287, -0.0067423587, -0.021064363, -0.022386394, 0.005684734, -0.018949114, 0.019528288, -0.010122981, -0.002563796, 0.011961234, 0.014000939, -0.0038212996, 0.012445979, -0.0091346055, -0.003723721, 0.017664853, 0.014743795, -0.03399509, -0.0150711555, -0.0056658476, 0.043412983, 0.0064464756, -0.026365079, 0.013396583, -0.03306337, 0.0015950935, 0.006629042, -0.006975288, -0.015537013, 0.007006765, 0.029512772, -0.006081343, -0.016116189, 0.022109397, -0.033818815, -0.0045074965, -0.015348152, -0.026591713, 0.013006268, -0.028203331, -0.007541873, 0.0025386147, 0.0017406744, 0.03747014, -0.0026424883, 0.014403844, -0.0014282658, 0.010053732, -0.03747014, -0.009556397, -0.0014093796, -0.054996494, 0.028807689, -0.020460006, 0.0035348595, 0.02817815, -0.041977637, -0.015184472, 0.033642545, -0.024854185, 0.01775299, -0.004318635, 0.022613028, 0.027422704, -0.019465335, 0.017337494, -0.025181545, 0.003764641, -0.006330011, 0.0067360634, -0.008083276, -0.015562195, 0.005857857, 0.018697297, -0.010991745, -0.032937463, 0.016179143, 0.026994618, -0.009883757, -0.0009151918, -0.016884226, 0.023393655, -0.015197063, -0.008983516, 0.030091947, 0.0030217855, 0.0319302, -0.034322448, -0.011602397, 0.02396024, -0.009814507, -0.0035820748, -0.028404783, 0.0030926086, 0.029210594, -0.009084242, 0.018193666, -0.039283212, -0.008750587, -0.026944254, 0.0016871636, 0.036236245, 0.007541873, -0.009921529, 0.021379132, -6.374079e-05, 0.0084609995, -0.047668666, 0.026944254, 0.02441351, 0.0047750506, -0.013547672, -0.01776558, 0.03243383, -0.001918519, 0.016909407, 0.019666787, -0.0002201418, 0.0025401884, 0.00093958643, -0.04414325, 0.0033711793, 0.017450811, 0.01954088, -0.03794859, -0.013761715, -0.004110887, -0.016607229, -0.0046963585, 0.013711352, 0.009474557, -0.022348622, -0.0052849767, 0.034826078, 0.015335562, -0.016468732, -0.0075229867, 0.02885805, 0.033189278, -0.00035706646, -0.010884723, -0.009896347, 0.016242096, 0.0022364361, 0.013081813, 0.01270409, 0.007183036, 0.013081813, 0.007321534, -0.0021687606, 0.00027601334, -0.01499561, -0.050564542, 0.007233399, -0.0079195965, -0.028606236, -0.031174753, -0.0048191184, -0.00749151, 0.018067759, -0.0050016847, 0.010299252, 0.0046617338, 0.015851783, 0.068695255, -0.010286662, 0.027070161, 0.016745728, 0.025534088, -0.007579645, -0.029537953, -0.013396583, -0.00041313472, -0.018357348, 0.021253224, -0.036840603, 0.015411106, 0.0012480604, 0.0021797775, -0.0032893394, -0.011696828, -0.016582048, -0.0047152443, -0.00058940554, 0.025647404, 0.026088081, 0.018332165, 0.015020792, -0.008335091, 0.02200867, -0.011451308, 0.021316178, -0.00035411547, -0.009713781, 0.047316123, -0.018583981, -0.021014, 0.007415965, -0.040945195, 0.00079400564, -0.045402326, -0.0044791675, 0.01824403, 0.025068229, 0.006204103, -0.0061033773, 0.02135395, 0.002574813, 0.02068664, -0.016191734, 0.028958777, -0.007346716, 0.0051748077, -0.010122981, 0.014114256, -0.021794628, 0.018986885, -0.011684237, 0.003078444, -0.013774306, 0.00863727, -0.028782507, -0.037117597, -0.011363173, -0.028732143, 0.007378193, 0.0018870421, 0.012357843, -0.0020381315, -0.03925803, 0.0022757822, 0.007875528, 0.018974295, -0.0017579866, 0.024891958, 0.014466798, -0.00480338, -0.02966386, 0.0007373471, -0.029462408, -0.0046806196, -0.017639672, 0.03162802, -0.007963664, 0.04545269, 0.012326366, 0.008819836, -0.012074552, 0.021895355, 0.029563135, -0.026591713, 0.0008490902, -0.013044041, 0.0017485436, 0.0052157277, 0.0016667036, 0.0014117404, 0.01042516, -0.012496342, 0.026365079, 0.014366072, -0.004749869, -0.031552475, 0.0015407958, 0.0013204573, -0.014353481, -0.022096807, -0.028807689, 0.018432891, -0.028631417, 0.0037300165, 0.023985423, 0.0028171854, 0.0038936965, 0.011086175, 0.010702156, 0.035455618, -0.016368005, 0.01626728, 0.017992215, -0.001456595, -0.014945247, -0.0022332883, -0.0046932106, -0.017211586, -0.002656653, -0.028455148, 0.017866306, 0.018521028, -0.0016588343, 0.019666787, 0.027573792, -0.01581401, 0.034599442, -0.03817522, 0.0018886159, 0.017060498, 0.034020267, 0.011092471, 0.013497309, 0.023431428, 0.05625557, 0.013472127, 0.0020066544, 0.0061946604, -0.0026802607, 0.0058956295, 0.01124356, -0.031527296, -0.010595135, -0.022575255, -0.039988294, -0.028883234, 0.0042966013, 0.01661982, -0.012364139, -0.018055169, -0.016418368, 0.007787393, 0.01548665, 0.016393187, -0.033038188, -0.03940912, 0.0017878897, 0.02344402, 0.028354421, 0.026616894, -0.023494381, -0.027095344, -0.026667258, 0.01319513, -0.029512772, -0.021970898, -0.045049787, 0.01058884, -0.02885805, -0.03177911, -0.0095941685, 0.012175277, 0.00590822, 0.010154459, 0.018470664, 0.012263413, -0.014957838, -0.011048404, -0.0063961125, -0.025382997, 0.01972974, -0.005838971, 0.031048845, -0.02035928, 0.003017064, -0.009619351, 0.004192727, -0.016758319, -0.024375737, 0.03930839, -0.027271613], "content_hash": "240077b750347b62acd28ee208fffecd"}, {"content": "an artifact of the softmax function -- attention weights must sum to 1.0, so when a head has nothing specific to attend to, it sends attention to the first token. **Looks like**: Many thick lines all pointing to the same token on the right side. ### Uniform / Diffuse (Bag-of-Words)\n\nYou see many thin lines spreading from each token to many other tokens, with no strong focus on any particular one.\n\n**What it means**: This head is gathering a broad summary of the entire input, rather than focusing on specific relationships. It helps the model maintain an overall sense of context.\n\n**Looks like**: A dense web of thin, similarly-weighted lines.\n\n### Structured Connections (Syntactic)\n\nYou see specific, purposeful-looking connections that skip across tokens -- like a token attending to a word several positions away in a consistent pattern.\n\n**What it means**: This head may be tracking grammatical relationships. For example, a verb attending to its subject, or a pronoun attending to its antecedent.\n\n**Looks like**: A few thick lines making specific connections, often spanning several token positions.\n\n### Mixed / Context-Dependent\n\nSome heads show patterns that change based on the input. They might show one pattern for factual prompts and another for creative prompts.\n\n**What it means**: These heads are flexible and context-sensitive. They don't have a single fixed pattern but adapt to the input.\n\n## Tips for Reading Attention Maps", "source_file": "interpreting_attention_maps.md", "chunk_index": 1, "embedding": [0.02615375, 0.03889316, 0.04049613, 0.021190163, -0.016999938, -0.046767406, 0.017576445, -0.005765074, -0.008549183, 0.030850178, -0.0025925257, 0.016887449, 0.00089288346, 0.01739365, -0.004418718, -0.0072274343, -0.0124933375, 0.0060814503, 0.008731978, -0.003624263, -0.024269557, 0.008035951, 0.018110769, 0.060969174, 0.015045437, 0.013822117, -0.020529289, 0.03574346, 0.03259376, 0.023510253, -0.01642343, -0.029697163, 0.0077406666, -0.04409579, -0.020037148, 0.03051271, -0.014651725, 0.05666646, -0.0061341794, 0.010159185, 0.0054662745, 0.027306765, -0.024972614, 0.007195797, 0.023397764, -0.025717854, 0.017759241, -0.0015853954, -0.037993245, 0.0695746, -0.047273606, 0.07564902, -0.0035662607, 0.02297593, -0.017182734, 0.015762556, 0.0022409968, 0.031834457, 0.012711286, -0.009231149, 0.061812844, -0.006285337, 0.017098365, 0.039764952, 0.00090606575, -0.012901111, -0.09066634, -0.0010080092, -0.0027612597, 0.015425089, 0.0768864, 0.033353064, 0.0104615, 0.0010660114, -0.0030530286, 0.00030692868, -0.012359756, -0.0005145504, -0.04547378, -0.030850178, 0.040018052, -0.010363072, -0.003733237, -0.06237529, -0.0013779934, -0.03833071, -0.038949404, -0.043055262, -0.025703793, -0.018110769, -0.011354384, 0.013449496, 0.015003254, 0.022413483, 0.015509455, -0.004647212, -0.004021491, -0.0018209198, -0.034590445, 0.0023376672, 0.03866818, -0.0059478693, -0.008816345, -0.03295935, 0.040468007, -0.03526538, 0.005237781, -0.062769, -0.020683961, -0.023018112, -0.046486184, -0.0036910535, 0.0011978347, 0.07283679, 0.048004787, 0.011009885, -0.04924217, 0.022076016, 0.03965246, -0.0061728475, 0.019193478, -0.015551639, 0.047807932, 0.0350404, -0.020655839, -0.009814687, 0.033831142, -0.021302652, -0.0071184603, -0.01520011, 0.020824572, -0.0013850239, 0.0016100024, -0.04572688, -0.027658295, -0.022483788, -0.021611998, 0.031159522, -0.055597812, 0.0108763045, -0.0016609741, -0.014595481, 0.015186049, 0.028473841, 0.023721172, 0.020880817, -0.040186785, 0.04938278, -0.031243889, 0.030456465, -0.0034432258, -0.00209687, -0.04426452, -0.044601988, -0.018082647, -0.06237529, 0.0020283218, 0.018588848, -0.01070757, -0.025366327, 0.03833071, -0.0058670174, 0.005782651, 0.0078601865, -0.0072555565, 0.035771582, -0.030315854, -0.004714003, -0.009920145, -0.051604442, -0.02103549, -0.014321288, -0.0057896813, -0.011157528, 0.030962666, -0.029528428, -0.06215031, -0.024578901, -0.024522657, -0.031300135, 0.011705913, -0.011748096, -0.0039300933, 0.024888247, -0.045501903, -0.037177697, -0.011002854, 0.048932824, 0.0020810512, 1.1465884e-05, -0.030147118, 0.0065454682, 0.061475374, 0.035968438, 0.0073891375, -0.07379295, 0.03965246, 0.02613969, 0.0031760638, -0.003831665, 0.009013201, 0.015636006, 0.03281874, -0.02017776, 0.004046098, 0.036080927, 0.0043589585, -0.023369642, -0.020093393, -0.019896537, 0.012662072, -0.075930245, 0.037880756, 0.04718924, -0.05677895, 7.618291e-05, -0.018167013, 0.030315854, 0.0024905824, 0.05326366, 0.026477158, 0.017913913, 0.06828097, 0.04682365, -0.068730935, 0.046514306, 0.064737566, -0.016873388, 0.0012734134, -0.020262126, 0.034140486, 0.020374615, 0.00972329, -0.035940316, -0.048257887, -0.07238683, 0.018420115, -0.013224518, 0.06040673, -0.011038008, -0.008956957, -0.019868415, 0.023397764, 0.01677496, -0.0060111443, 0.031159522, -0.027742662, -0.012542551, 0.028909737, 0.021949464, 0.09505342, 0.011108314, -0.0763802, 0.018560726, -0.051041998, 0.028783187, -0.03987744, 0.046879895, -0.004415203, 0.020149637, -0.0067880233, 0.028220741, -0.01958719, -0.005286995, -0.015396967, -0.036193416, 0.014117401, 0.020641778, -0.048567235, -0.023791477, -0.004137495, 0.014356441, 0.01483452, 0.0069637876, -0.045530025, -0.018012341, -0.033324942, -0.01874352, -0.025717854, 0.0011486207, -0.037965123, -0.023496192, 0.068055995, 0.04502382, -0.01617033, 0.009174905, -0.05967555, -0.0204168, -0.008050012, 0.0036172324, 0.030597076, -0.07266805, 0.04597998, 0.0055998554, -0.0042359233, -0.005448698, 0.00662632, 0.007635208, -0.006995425, -0.035237256, 0.04561439, -0.04049613, -0.008900712, 0.0119590135, 0.036193416, 0.0057439823, 0.010032635, -0.025774099, 0.014145523, -0.020852694, -0.039118137, 0.011607485, -0.006594682, -0.018321687, -0.067156084, -0.012387879, 0.0347873, 0.036221538, -0.0072274343, 0.01971374, -0.0591131, -0.017281162, 0.015509455, 0.008155471, 0.028052008, 0.00036031712, -0.01872946, 0.027630173, -0.022708768, -0.0048300074, 0.024958553, 0.0038949403, 0.0097443815, 0.03672774, -0.0108763045, -0.015157927, 0.010531806, 0.03734643, -0.00985687, 0.035715338, -0.078123786, -0.049804617, -0.038021367, 0.0117551265, -0.03987744, 0.04257718, 0.0013560228, 0.0061341794, -0.0019791077, -0.00206699, -0.0016196695, -0.003539896, 0.014412685, -0.038021367, 0.0144408075, 0.049045313, -0.03160948, -0.009498311, -0.011023947, 0.0103419805, -0.02019182, -0.028516024, 0.00668608, 0.0060005984, -0.015059498, -0.018560726, -0.016100025, 0.00937176, 0.0058810785, -0.022652524, 0.00252222, 0.035574727, -0.026941177, -0.031946946, -0.04502382, 0.006394311, 0.00011995924, -0.010524776, -0.0352935, 0.015776617, -0.019910598, 0.0062818215, -0.04544566, -0.013927575, 0.029022226, 0.017281162, -0.0028403536, 0.017351467, -0.0381901, -0.054416675, -0.027939517, 0.0021847521, -0.0071817357, 0.037712023, 0.0126972245, 0.028276986, 0.018138891, 0.022441605, 0.041396044, 0.033606164, 0.038105734, 0.012127748, -0.027897334, -0.009716259, 0.012472246, 0.007515688, -0.00094473397, -0.001705794, 0.021246407, 0.034871668, 0.08487314, 0.013273732, 0.02723646, 0.014250982, 0.007438352, 0.016859327, -0.00034999097, 0.021780731, 0.020430861, 0.03782451, 0.006809115, -0.036530886, -0.032621883, 0.015593822, -0.042352203, -0.0024712482, 0.066931106, -0.04412391, 0.03040022, -0.023552436, -0.00011973953, 0.050901387, 0.0152422935, -0.012064472, 0.009224119, -0.043223996, 0.018307626, 0.03858381, 0.0063486123, -0.08009235, 0.012676133, -0.015875045, -0.06979958, -0.0068161455, -0.043111507, 0.021105796, -0.012373817, -0.03968058, 0.017421773, -0.031047033, 0.006355643, 0.023102479, -0.005620947, 0.041621022, -0.031159522, 0.011115344, 0.04451762, 0.0067880233, 0.039314993, -0.05604777, 0.013477619, -0.029697163, -0.023664925, 0.002304272, -0.02553506, 0.021752609, 0.01847636, -0.05447292, 0.053432394, 0.048454747, 0.007733636, -0.0029581157, 0.0238196, 0.0015643036, 0.035434116, -0.032537516, 0.031215766, -0.02833323, 0.027686417, -0.056582093, 0.039596215, -0.0017286434, -0.0016530647, -0.011895739, 0.014384563, 0.07598649, -0.006225577, 0.0034098304, 0.051520076, 0.025563182, -0.020346493, -0.036024682, 0.012071502, -0.017013999, 0.054416675, -0.02114798, -0.03172197, -0.044405133, -0.004341382, -0.045389414, 0.03892128, -0.009941238, -0.0071114297, 0.0035768067, 0.025225714, 0.022750951, 0.008992109, 0.038386956, -0.017942036, 0.010869274, 0.0050374093, 0.00016730578, -0.005195597, 0.047864176, -0.036896475, 0.0060111443, 0.013365129, 0.027827028, -0.0025591305, 0.000561128, -0.02907847, 0.00061341794, -0.009083507, 0.009259271, 0.013322946, -0.011052069, -0.025619427, -0.007234465, -0.045051944, -0.013632291, 0.00936473, -0.04949527, 0.010124032, 0.024494534, -0.030147118, 0.009899054, 0.008956957, -0.057819474, -0.004583937, 0.031047033, 0.003297341, -0.003638324, 0.021977587, 0.011832463, 0.0051850514, 0.017702997, 0.032284416, -0.013772903, 0.016367186, 0.032425027, 0.010081849, -0.030484587, 0.0076211467, -0.014026004, -0.012570674, -0.021738548, -0.019249722, -0.006904028, 0.010306828, -0.024592962, 0.041114822, -0.040664863, -0.013702597, 0.0012584735, -0.036783986, 0.030231487, -0.0104615, -0.007649269, 0.033381186, -0.049326535, -0.02189322, -0.027812967, 0.012465215, -0.025141347, -0.0011837736, -0.018138891, -0.020346493, 0.010117002, 0.0046542427, 0.01567819, -0.02238536, -0.0033746776, -0.037740145, -0.019193478, -0.00754381, -0.012169931, 0.03748704, -0.016957754, -0.0260272, 0.02019182, 0.039708704, -0.00985687, -0.0016108812, 0.04949527, -0.032256294, 0.002225178, -0.0045663607, -0.013217487, -0.014426746, 0.013266701, 0.009329577, 0.0032358237, -0.0009860386, 0.001963289, -0.023411825, 0.044826966, -0.07930492, 0.03200319, 0.039455604, 0.02456484, -0.019854352, -0.010770846, -0.015776617, -0.010032635, -0.021921342, 0.03574346, -0.037262063, -0.0013630533, -0.0055611874, 0.00074831716, 0.0014026003, -0.011565302, 0.037740145, 0.019910598, 0.001726007, 0.049185924, -0.004682365, -0.023566497, -0.0038105734, -0.011902769, -0.06833722, 0.003757844, 0.022807196, 0.009582678, -0.021668242, -0.0014447838, -0.0023359095, -0.007705514, 0.025281958, -0.018167013, 0.0013902968, -0.014553296, -0.015481333, -0.009554556, 0.044798844, -0.002893083, -0.02348213, -0.013829147, -0.033493675, -0.024508595, 0.006225577, -0.046401817, 0.03245315, 0.016240636, 0.0012505641, -0.036080927, -0.02114798, 0.00019806458, 0.004218347, 0.008120318, -0.019263783, 0.009624861, -0.020163698, -0.03830259, -0.018560726, 0.017056182, 0.009842809, 0.0042745913, 0.008289051, 0.004714003, -0.08768537, 0.03878067, 0.017857669, 0.03175009, 0.025942834, 0.00055849156, -0.0005286116, -0.025563182, -0.03613717, 0.0035785644, 0.010363072, 0.039933685, -0.030343976, -0.03284686, 0.019080989, 0.02710991, 0.050648287, 0.009139752, 0.036812108, -0.026730258, 0.00063758553, -0.008542152, 0.029725285, -0.013554955, 0.012971417, -0.019474702, 0.0130136, 0.05067641, -0.02128859, -0.011572332, 0.002863203, 0.011741065, -0.018771643, -0.0014948767, 0.0051463833, -0.014539235, 0.022933746, 0.041564777, 0.006981364, -0.02737707, -0.031665724, -0.0056033707, -0.0010405256, 0.035209134, 0.0052975407, 0.02784109, -0.028403535, -0.021597937, -0.004647212, 0.001355144, -0.029500306, 0.0052026277, -0.014426746, -0.0030829085, 0.022483788, -0.013702597, -0.015171988, -0.030540831, -0.006288852, 0.02224475, -0.017773302, 0.02491637, -0.0029581157, -0.04533317, -0.023004051, 0.035715338, -0.01969968, -0.008767131, -0.033887386, -0.02284938, 0.03669962, -0.007916431, 0.003659416, -0.00681263, 0.013442465, 0.042999018, -0.007213373, 0.0003750374, -0.062769, -0.016690593, 0.001790161, 0.013175303, 0.019952781, 0.003975792, -0.00010990771, -0.0072977403, 0.0016557012, -0.0021882674, 0.030850178, 0.023622742, -0.011895739, 0.0065208613, -0.021372957, 0.014222859, 0.016395308, -0.02868476, -0.0031672756, -0.011354384, 0.018040463, 0.0012918687, -0.0007259072, 0.015228232, 0.018377932, 0.015875045, 0.03720582, -0.018870072, 0.037993245, -0.041986614, -0.03864006, -0.015453211, 0.02992214, -0.023383703, -0.0028350807, -0.0036805077, 0.005265903, -0.0050795926, -0.029134717, -0.03222817, 0.0009948268, -0.015846923, -0.016268758, -0.027138032, -0.003659416, -0.03658713, 0.0033201906, 0.021049552, 0.006489224, -0.09471595, -0.043505218, 0.019235661, -0.040552374, -0.032621883, -0.020641778, 0.032537516, 0.023636803, 0.026603708, -0.021626059, -0.0028016854, 0.019024745, -0.040158663, -0.006724748, 0.021626059, -0.05227938, -0.043702073, 0.014609542, -0.020248065, 0.002690954, -0.044911332, -0.0029247205, 0.005993568, 0.014665786, -0.023580559, 0.025450693, -0.019545007, -0.00028144283, 0.03307184, 0.019629374, -0.025000736, -0.003771905, 0.0052799643, 0.024536718, 0.01532666, -0.00026957874, -0.0007412866, 0.026195934, 0.028305108, 0.0038949403, -0.0042253775, -0.0481454, 0.006724748, -0.024747634, 0.004513631, -0.0020458982, 0.002566161, -0.028487902, -0.0147079695, 0.011586393, -0.04536129, -0.011136436, 0.017618628, 0.010159185, 0.0006459344, 0.042380325, -0.024846064, 0.0025046435, 0.034224853, 0.03354992, -0.052616846, 0.0040531284, 0.067943506, -0.011593424, -0.007874248, -0.016662471, 0.003010845, 0.038499445, -0.028909737, -0.009934207, 0.0002366229, -0.016128147, 0.008837436, -0.0023763354, 0.016662471, -0.0012391394, 0.013428404, 0.005554157, 0.010869274, 0.008471847, 0.07435539, -0.02540851, 0.016915571, 0.027503623, 0.008211715, 0.006517346, 0.013034692, -0.004415203, -0.0075578718, 0.04983274, 0.016029717, -0.005483851, 0.007614116, 0.00081950176, -0.01763269, -0.040692985, 0.025239775, 0.03878067, 0.08419821, -0.007818003, 0.03464669, 0.009413945, 0.011656699, -0.0350404, -0.04693614, 0.004970619, -0.0006261609, -7.255776e-05, 0.018406054, 0.008106257, 0.015186049, 0.012212114, 0.025717854, 0.032903105, -0.01324561, 0.0070692464, 0.062206555, -0.0063521275, 0.022315055, -0.015157927, -0.013913514, 0.024410168, 0.011923861, -0.01033495, -0.0034151033, 0.0069005126, -0.0031479413, 0.008816345, 0.0073750764, -0.0066087437, 0.003545169, -0.014180676, -0.011649668, -0.0015142108, -0.028628515, 0.009568617, 0.021682303, 0.036671497, -0.036052804, 0.03855569, -0.007585994, -0.023004051, 0.069237135, 0.000108314845, 0.0014852097, -0.0074453824, -0.004446841, -0.021471385, -0.003367647, 0.018940378, 0.02760205, 0.020234004, 0.013737749, 0.012704255, 0.041902248, 0.022525972, 0.0022656037, -0.0381901, -0.024747634, 0.03790888, -0.045305047, 0.008232807, 0.04851099, 0.026688075, 0.030006507, 0.0044116876, -0.03658713, -0.0025679187, -0.016873388, 0.0038843944, -0.020683961, -0.019615313, 0.0104825925, 0.020360554, 0.008282021, 0.028909737, -0.018391993, -0.016240636, 0.0011538937, 0.052701212, 0.002061717, 0.033128083, -0.04283028, 0.026125629, 0.056497727, 0.016732777, 0.008978048, -0.0037472981, -0.019066928, -0.020866755, 0.017013999, 0.022666585, -0.011614515, -0.021583876, -0.0125566125, -0.015368843, 0.010306828, 0.00899914, -0.015607883, 0.020023087, -0.0080640735, 0.014679847, 0.008443724, -0.024986675, -0.0078039416, 0.036334027, 0.004422234, -0.002200571, -0.00013380068, -0.0054557286, -0.013119059, 0.026055323, -0.011073161, -0.0055647027, -0.041142944, -0.020712083, -0.0037824512, -0.010053727, 0.028234802, -0.021907281, -0.027911395, -0.037852634, -0.0145814195, 0.014026004, -0.005367846, 0.026336545, -0.0025521, 0.0026188905, -0.03172197, -0.009638922, -0.032115683, -0.003937124, 0.008570274, 0.021738548, -0.0024835519, 0.0016697623, 0.01884195, -0.008106257, 0.030062752, 0.01313312, 0.02443829, 0.019010684, -0.02370711, -0.008021889, -0.01567819, -0.052701212, -0.017618628, 0.02394615, 0.0067669316, 0.0064821932, -0.013822117, 0.034421712, 0.035237256, -0.041002333, -0.01070757, -0.01640937, 0.016142208, 0.019362213, -0.0041621025, 0.03391551, 0.025478816, 0.019263783, 0.01009591, -0.051745057, -0.014750153, 0.010960671, -0.022624401, -0.000988675, -0.016043779, -0.0060603586, -0.020782389, 0.018138891, 0.008239837, -0.01946064, 0.032031313, 0.023172786, -0.00656656, -0.004046098, 0.01750614, 0.007346954, -1.8661241e-05, -0.032762494, 0.012683163, -0.009463158, 0.0308783, 0.00426053, -0.030315854, -0.025085103, 0.015228232, -0.0058775633, 0.003912517, -0.0018894679, 0.00814844, -0.009069446, -0.01594535, -0.005666646, -0.0060427818, 0.011663729, 0.0078601865, -0.029725285, -0.0031461837, 0.0075649023, 0.02930345, -0.024269557, 0.009624861, -0.0037472981, 0.022582216, -0.055879034, -0.011684821, 0.0006389038, 0.015340721, 0.014054126, 0.014426746, 0.041564777, 0.007818003, -0.013632291, 0.011712943, -0.021485446, 0.0011486207, -0.030540831, 0.014623603, 0.03405612, -0.005216689, 0.0048300074, 0.029247206, -0.0026575585, -0.004179679, -0.027812967, -0.0022708767, -0.017970158, 0.020838633, -0.00473861, 0.011691852, 0.009828748, -0.016128147, 0.0019474701, -0.037543286, -0.014068187, -0.008816345, -0.014932948, 0.025563182, 0.0149751315, -0.018532604, 0.005751013, 0.020487105, 0.008071104, -0.012219145, -0.0075649023, -0.00838748, 0.0032410966, 0.069349624, -0.04052425, -0.006679049, 0.009582678, 0.0086616725, 0.00814141, -0.025675671, 0.008000798, -0.025985017, -0.009652983, -0.029753407, 0.024002394, 0.014363471, -0.044405133, 0.0065454682, -0.009174905, -0.01884195, 0.009786565, 0.03200319, 0.02882537, 0.007016517, 0.030006507, -0.022877501, -0.021682303, 0.0075227185, 0.012113687, 0.0007689695, 0.0010844667, 0.0022234204, 0.028122313, -0.014595481, -0.043252118, -0.011691852, -0.03419673, -0.0064962544, 0.014736092, -0.008591367, 0.0036699618, -0.018687276, -0.0040531284, -0.0014201768, 0.010763816, 0.0013612957, -0.011178619, 0.0029299934, 0.011403598, 0.0056139166, 0.015298538, -0.010032635, -0.018462298, -0.014736092, 0.029050348, -0.003638324, -0.016620288, 0.012542551, -0.022146322, 0.007677391, 0.027447376, -0.0068302066, 0.00741726, 0.018926317, 0.014609542, -0.008900712, -0.014047096, 0.019671557, -0.028417597, -0.002893083, 0.0040074294, -0.03405612, 0.011466873, -0.031693846, -0.022005709, 0.011494995, 0.010623204, 0.015537578, -0.011333292, 0.017084304, 0.005033894, 0.0025538576, -0.0019791077, -0.014004911, -0.0001812571, -0.049410902, 0.010735692, -0.018223258, -0.005265903, 0.012662072, -0.018504482, -0.050310817, 0.040889844, -0.01520011, -0.007125491, 0.0013068087, 0.015157927, 0.009526433, -0.007832064, -0.00016829447, -0.045108188, 0.019193478, -0.025760038, 0.012008227, -0.0054170606, 0.00047104873, 0.010960671, -0.0051709902, -0.012071502, -0.04193037, 0.033521798, 0.03464669, -0.006169332, 0.026364667, -0.0036137172, 0.013934606, -0.010018574, -0.0016618529, 0.01094661, -0.006773962, 0.020726144, -0.028234802, -0.048004787, 0.02955655, -0.008619489, 0.004365989, -0.023102479, 0.0046120593, 0.0041867094, -0.020937061, 0.004777278, -0.04631745, -0.0038246345, -0.020248065, 0.020937061, 0.024635145, -0.029809652, 0.009624861, 0.020430861, 0.0066579576, -0.010496654, -0.042886525, 0.033746775, 0.026336545, 0.024621084, -0.022483788, 0.00523075, 0.018335748, 0.011600454, 0.014201768, 0.005315117, -0.03537787, 0.01058102, 0.0039336085, -0.036502764, 0.007881278, 0.02359462, -0.003485409, -0.025985017, 0.010292767, -0.0016548224, -0.039343115, -0.01215587, 0.005705314, 0.012050411, 0.0018877103, 0.006359158, 0.0316376, -0.0005453092, -0.0096037695, 0.007065731, 0.019545007, 0.010377133, 0.0478923, -0.008802284, -0.009702198, 0.012416001, -0.027039604, -0.011579363, 0.008204685, -0.0015546366, -0.008640581, 0.0072063426, -0.021176102, 0.0024607023, -0.011762157, -0.02272283, 0.01971374, -7.903908e-05, -0.023411825, -0.023875844, 0.0039090016, 0.01155827, 0.000709649, -0.034477957, -0.020374615, -0.0350404, 0.012176962, 0.07716762, -0.021724487, 0.024860125, 0.034252975, 0.017435834, 0.009294424, -0.033634286, 0.0061728475, -0.0053362087, -0.010763816, 0.004815946, -0.019432519, 0.0015484849, -0.0017760998, 0.018574787, 0.008921804, -0.0079094, -0.02588659, -0.021063613, 0.0062712757, 0.024114883, 0.016845265, 0.007157129, 0.009083507, -0.0073258625, 0.008310144, 0.0037789359, 0.024367984, -0.014293166, -0.047976665, 0.039230626, -0.032650005, -0.017013999, 0.0048018848, -0.013639322, 0.010433378, -0.036024682, -0.009800626, 0.005729921, 0.012971417, 0.0040144604, -0.024325801, 0.016043779, 0.0061728475, 0.027939517, 0.012985478, 0.02491637, -0.016760899, 0.013301854, -0.03684023, 0.01946064, 0.014553296, 0.01737959, -0.043195873, 0.020248065, -0.009146783, 0.017351467, -0.03858381, -0.040805474, -0.011389537, -0.013407312, 0.013344037, -0.010735692, 0.025281958, 0.013941636, -0.054697897, 0.008992109, 0.008021889, 0.017182734, -0.017070243, 0.028909737, 0.0032727341, -0.01398382, -0.027166154, -0.009034293, -0.031946946, 0.01070054, -0.014806397, 0.020683961, 0.011459842, 0.014736092, 0.005874048, 0.015636006, -0.02758799, 0.016971815, -0.009188966, -0.026378728, 0.015354782, -0.013048753, 0.010834121, 0.0020054723, 0.016240636, 0.030765811, 0.03135638, -0.02516947, 0.006636866, 0.0049565574, 0.009463158, -0.01861697, 0.009463158, 0.01167779, -0.0012039865, -0.021597937, -0.030484587, 0.011389537, -0.011523117, -0.025225714, 0.0084577855, 0.008851498, 0.005181536, 0.03563097, 0.0026522856, 0.041367922, 0.0021249922, 0.00923818, 0.017365528, 0.014145523, -0.01957313, -0.011691852, 0.0041726483, -0.013358098, -0.00045215405, -0.04027115, 0.013182334, 0.0049178894, 0.014096309, 0.01617033, 0.027939517, -0.012514429, 0.035462238, -0.032509394, 0.0044925394, 0.022540033, 0.015748495, 0.012324603, 0.004657758, 0.033240575, 0.041958492, 0.016212514, 0.012500368, -0.0050936537, -0.0223713, 0.019291906, 0.020234004, -0.037177697, -0.015846923, -0.011853555, -0.05107012, 0.004671819, 0.009526433, 0.011656699, 0.006077935, -0.01362526, -0.03928687, -0.009772504, -0.00838748, 0.0052553574, -0.013779934, -0.026927114, 0.008485908, 0.023763355, 0.008317174, 0.015720373, -0.014201768, -0.033578042, -0.028023886, -0.0045030853, 0.013779934, 0.0068548135, 0.0024940977, -0.0047245487, -0.015762556, -0.038246345, -0.004450356, 0.021443263, 0.012148839, 0.019010684, 0.033043716, 0.0006481314, -0.03900565, -0.025394449, -0.012176962, -0.014412685, 0.026125629, -0.021654181, 0.02360868, -0.03298747, -0.010173246, -0.006893482, 0.036783986, -0.009013201, -0.01874352, 0.0036207477, -0.025703793], "content_hash": "f6e3fe41379649aec77dee82cd39e9ac"}, {"content": "Some heads show patterns that change based on the input. They might show one pattern for factual prompts and another for creative prompts. **What it means**: These heads are flexible and context-sensitive. They don't have a single fixed pattern but adapt to the input. ## Tips for Reading Attention Maps\n\n- **Start with one head at a time**: Double-click individual heads to isolate their patterns. Looking at all heads simultaneously is confusing.\n- **Compare heads across layers**: The same type of pattern (like Previous-Token) may appear in early layers but not late layers, or vice versa.\n- **Match patterns to categories**: Use the head categories as a guide. If a head is categorized as \"Previous-Token,\" look for the diagonal pattern to confirm.\n- **Hover for details**: The exact attention weight tells you more than just the visual thickness. Two lines might look similar but have meaningfully different weights.\n- **Context matters**: The same head can show different patterns for different prompts. Try multiple inputs to understand a head's full behavior.", "source_file": "interpreting_attention_maps.md", "chunk_index": 2, "embedding": [0.029645778, 0.042226423, 0.04595848, -0.00037057223, 0.0073775854, -0.014017789, 0.03747106, 0.0080886325, 0.012964385, 0.039939035, 0.031210836, -0.014950803, -0.019788934, -0.0032749556, -0.030970057, -0.0006762473, -0.0146874515, -0.019834079, -0.0043866723, 0.030413259, -0.0015161483, 0.024965659, -0.0033596042, 0.07560426, 0.052068222, -0.0070352294, -0.016929695, 0.043851674, 0.02844189, 0.03440114, -0.02594382, -0.013197639, -0.010391071, -0.014536966, -0.011722874, 0.009405387, -0.012031371, 0.051255595, 0.007129283, 0.0029965562, -0.004499537, 0.034130268, -0.025206437, 0.024213228, -0.011813166, -0.016327752, 0.032595307, -0.016147168, -0.06470906, 0.107326746, -0.018238926, 0.078734376, -0.012580645, 0.026440423, -0.02114331, 0.010285731, -0.0435808, 0.003237334, 0.030578794, 0.008758296, 0.05543911, -0.0028009242, 0.005101482, 0.013679194, 0.0077726115, -0.020827288, -0.076747954, -0.02802053, 0.02973607, 0.023235068, 0.092759684, 0.05480707, 0.02353604, 0.0036925548, -0.0056056106, 0.0031733776, 0.014138177, -0.015266824, -0.00743778, -0.0039690733, 0.021820497, 0.0066740625, 0.007516785, -0.045145858, -0.008690577, -0.017471446, -0.036447756, -0.055800278, -0.025417117, -0.054656584, -0.0070690885, 0.010699568, -0.008359508, 0.023129728, 0.00018704963, -0.012422635, -0.016357848, -0.0022723414, -0.036658436, 0.040811855, 0.038825437, -0.00097627915, -0.021805448, 0.015334543, 0.044844884, -0.0132202115, -0.019247184, -0.029013736, -0.0446643, -0.0110757835, -0.07692854, -0.017110279, -0.011008064, 0.042557493, 0.053542987, 0.012986958, -0.05841874, 0.0056206593, 0.058057573, -0.031632196, -0.010887676, -0.010150294, 0.018946212, 0.02376177, -0.00914956, -0.02540207, -0.0013261596, -0.018389411, 0.006933651, -0.0062827985, 0.014168275, -0.00031907775, 0.016147168, -0.052579872, -0.012369964, -0.034551628, -0.0015885698, 0.030488502, -0.050262388, 0.007840331, 0.031842876, -0.0026128164, -0.010977968, 0.037832227, -0.028803056, 0.015921438, -0.022843804, 0.07090909, -0.024258373, 0.043099243, -0.0067304946, 0.011459523, -0.026169548, -0.033919588, -0.020180197, -0.023806915, 0.00525573, -0.031872973, -0.04075166, -0.030037044, 0.013363173, -0.029043835, -0.0063655656, -0.009510727, -0.03518367, 0.020511268, -0.014010265, -0.040029325, -0.020857386, -0.0036342414, -0.02189574, -0.034912795, -0.018825822, -0.024107886, 0.031090446, -0.022091372, -0.052008025, -0.012264624, -0.028757911, -0.010172866, 0.0010327115, 0.011685252, -0.0007655985, 0.02397245, -0.024920512, -0.017606884, -0.012934288, 0.020902531, -0.027117612, -0.004401721, -0.010819957, 0.0022253145, 0.073798425, 0.0036775062, 0.0017870235, -0.07325668, 0.08397129, 0.012625791, 0.01449182, -0.0021538336, 0.022587977, 0.011895933, 0.03289628, 0.018344266, -0.009119463, 0.02550741, 0.019849129, -0.010187915, -0.028923444, -0.024228277, 0.024484104, -0.06657509, 0.00996971, 0.039367188, -0.06079642, 0.0066928733, -0.026004013, -0.016613675, -0.020420976, 0.07879457, 0.034160364, 0.015206629, 0.03780213, 0.044604108, -0.058328446, 0.031180738, 0.06747801, -0.027644312, -0.01831417, -0.034039974, 0.04622936, 0.03530406, 0.016523384, -0.0145821115, -0.032836087, -0.046018675, 0.017847663, -0.024047693, 0.022828754, -0.03397978, -0.00179831, 0.01927728, 0.010421168, 0.01765203, -0.011421902, 0.020481171, -0.029615682, -0.029073931, 0.03930699, -0.0029927942, 0.08300818, -0.0053196866, -0.053452693, 0.013385747, -0.009029171, 0.00017317668, -0.0076936064, 0.038193393, -0.01182069, -0.0010402357, 0.020797191, 0.04011962, 0.002187693, 0.01774232, -0.015454931, -0.038975924, 0.005146628, 0.0069900835, -0.063986726, -0.05318182, -0.041774966, 0.005470173, 0.05294104, 0.02046612, -0.059562434, -0.019954469, -0.031421516, -0.02647052, -0.010887676, 0.014469247, -0.028607424, -0.033137057, 0.04282837, 0.015199105, -0.026320035, 3.8062426e-05, -0.018870968, 0.007151856, -0.012994482, -0.012821423, -0.00024477518, -0.09119463, 0.061458558, 0.003068037, -0.0078252815, -0.010255634, -0.019292329, 0.0115949605, -0.015304445, -0.01777242, 0.012603219, -0.039818645, 0.006790689, 0.032444824, 0.028456938, 0.011655156, 0.027298195, -0.0041684676, 0.013927497, -0.023957402, -0.028351597, -0.002627865, 0.03397978, -0.030368112, -0.038945824, -0.044303134, 0.052910943, 0.02648557, 0.0024811411, -0.009864369, -0.053843956, -0.0020597796, 0.032444824, 0.038644854, 0.005473935, 0.0010806789, -0.030006945, 0.026711298, -0.004059365, -0.026139451, 0.008427227, -0.0027087515, -0.0077124173, 0.03563513, -0.042226423, -0.020240393, -0.015289397, 0.010752238, -0.005075147, 0.021098163, -0.05646242, -0.032083657, -0.026380228, 0.018359315, -0.033708908, 0.038163297, -0.06416731, -0.011361707, -0.0022949143, 0.0021764066, -0.019593302, 0.017832613, 0.003577809, -0.025311777, -0.0023099629, 0.052038122, -0.013701768, -0.02145933, -0.010609277, 0.027945286, -0.013212687, -0.033076864, 0.009510727, 0.038434174, -0.0015471862, -0.012859045, -0.026395278, -0.020436024, -0.01045879, -0.026876833, 0.010345926, -0.010240586, -0.027870042, -0.020752044, -0.036327366, 0.014935754, -0.014822889, -0.010541557, -0.0017550451, 0.035454545, -0.007900525, -0.0040029325, -0.0026617243, -0.0042211376, 0.03343803, 0.03500309, 0.022196712, -0.0031677342, -0.07349745, -0.034220558, 0.00015812807, -0.01274618, -0.021444282, 0.019668546, -0.02005981, 0.030473454, 0.0030266535, 0.04761383, 0.03527396, 0.026891882, 0.037019603, 0.014664879, -0.044965275, 0.019758837, 0.026440423, 0.029450146, -0.0133782225, -0.02004476, 0.036808923, 0.011828214, 0.077650875, 0.005078909, 0.020210296, 0.010067526, -0.016327752, 0.026109353, 0.023942353, 0.0010327115, -0.0013111109, 0.026921978, -0.008841064, -0.07897515, -0.038945824, 0.009495678, -0.03214385, -0.014153226, 0.06380615, -0.031963266, 0.037892424, -0.052068222, -0.039186604, 0.07650718, 0.02690693, 0.015966585, -0.00754312, -0.02440886, 0.032053556, 0.015454931, 0.017245717, -0.053422596, 0.02114331, -0.0030755615, -0.03801281, 0.004115797, -0.018901065, 0.02701227, -0.02418313, -0.07289551, 0.0076860823, -0.06771878, -0.019548155, 0.028893348, -0.024484104, 0.077169314, -0.008999074, 0.019262232, 0.03930699, -0.0063053714, 0.054536194, -0.036989506, 0.0009217279, -0.021384086, 0.007072851, 0.01971369, 0.014589636, 0.028171014, 0.019969517, -0.02526663, 0.07090909, 0.015831146, 0.004431818, -0.004860704, 0.03214385, -0.008961452, 0.027207904, -0.015018522, 0.0018199424, 0.0029570535, 0.00806606, -0.060405158, 0.02025544, -0.031632196, 0.00058219343, 0.01546998, 0.0132728815, 0.053452693, -0.042557493, -0.0067756404, 0.03587591, -0.0001695321, -0.00909689, -0.033287544, 0.0046650716, -0.018931162, 0.04616916, -0.014288664, 0.0067229704, -0.027870042, -0.0031338749, -0.038584657, 0.034160364, -0.016478237, -0.008585237, 0.04622936, -0.013603952, 0.025687993, 0.04011962, 0.014627257, -0.022708366, 0.015891341, -0.015951537, 0.0040292675, -0.024438957, 0.034340948, -0.03987884, 0.003622955, 0.026184596, 0.036959406, 0.010586703, -0.0039765975, -0.03629727, 0.0056733293, -0.051255595, 0.008540091, 0.010481363, -0.03473221, -0.010985492, -0.03527396, -0.049931318, -0.012460256, 0.011632582, -0.03575552, 0.013287931, 0.006008161, -0.039036117, -0.0132202115, -0.007607077, -0.038975924, -0.01950301, 0.010466314, 0.01981903, 0.0072910558, 0.024333617, 0.021173406, -0.0095032025, 0.01766708, 0.022603026, -0.027253048, 0.015236727, 0.044483718, -0.0028479511, -0.030789474, -0.0021274984, -0.01323526, 0.0028761672, -0.0073249154, -0.023355456, -0.016508335, 0.0073813475, -0.02299429, 0.029389953, -0.037561353, 0.005477697, -0.019864177, 0.0010853816, 0.037440963, 0.0077650873, -0.0032241666, 0.014363906, -0.042858467, 0.0011079545, -0.021504477, 0.020962726, -0.0588401, 0.019969517, 0.0022742224, -0.028396744, 0.011948603, 0.000103047765, 0.003455539, -0.012753705, -0.0042211376, -0.029239466, 0.002894978, -0.0070013697, -0.011391805, 0.027388487, -0.014281139, -0.035815712, 0.015846195, 0.015861245, 0.0022008605, 0.0012010679, 0.040149715, 0.0030322967, 0.004014219, -0.0245142, -0.023882158, -0.019728739, 0.013363173, -0.0043151914, 0.018825822, 0.0006122907, -0.0015471862, -0.0015415429, 0.011151027, -0.057034265, 0.009420435, 0.023806915, 0.023460796, -0.014604685, -0.015161484, -0.026756445, -0.007990817, -0.02756907, 0.03181278, -0.01569571, 0.013084774, -0.0004361748, 0.01623746, -0.0042211376, 0.008209022, 0.01291924, 0.01084253, 0.015846195, 0.012053943, 0.027493827, -0.007095424, -0.008525043, -0.028833153, -0.042527396, 0.00773499, 0.038313784, 0.010722141, -0.009593494, 0.020676803, -0.018359315, -0.010391071, 0.017019987, -0.014807841, 0.0046763583, -0.0030943723, -0.03235453, -0.020180197, 0.015184056, -0.0017061372, 0.023280213, -0.016959794, -0.03169239, -0.018600093, -0.011655156, -0.025326826, 0.054445904, -0.0003581101, -0.00026711298, -0.057244945, -0.02036078, -0.005515319, -0.016794259, 0.003796014, -0.022753512, -0.009856845, -0.031632196, -0.062361475, -0.022211762, -0.0049246605, 0.008201498, 0.01367167, -0.0064031873, 0.010518985, -0.094264545, 0.054415807, 0.03443124, 0.039698258, 0.002917551, 0.015116338, -0.008005865, -0.017576788, -0.029766168, 0.014679927, -0.0030699181, 0.026891882, -0.03530406, -0.03473221, 0.023280213, 0.022693317, 0.024363713, -0.015199105, 0.024469053, -0.012986958, -0.0022949143, -0.017757371, 0.033919588, -0.011098356, 0.006418236, 0.0021707632, 0.009337668, 0.048877914, -0.01291924, -0.027824895, -0.022738462, -0.0017616289, -0.006136074, -0.008984026, 0.0053535462, -0.023415651, 0.024754979, 0.04698179, 0.0062602255, -0.004209851, -0.048306067, 0.0015086241, 0.0029006214, 0.056342028, -0.004774174, 0.057816792, -0.022256907, -0.003624836, -0.0056244214, 0.0030021993, -0.011760496, 0.010308304, 0.0028253782, 0.0013289811, 0.030518599, -0.0007637174, -0.0066853487, -0.03780213, -0.010917773, 0.017847663, 0.003455539, 0.035574935, -0.0014503107, -0.029013736, -0.0066703004, 0.02973607, -0.037079796, 0.0150561435, -0.007938147, -0.029374903, 0.043279827, -0.02680159, 0.010902724, -0.010782335, 0.028893348, 0.013784535, -0.013754438, 0.036778823, -0.035785615, -0.034070075, 0.003478112, 0.015951537, 0.016222412, -0.019849129, 0.02594382, 0.006271512, -0.00050506927, 0.0037019602, 0.01927728, 0.03343803, 0.054536194, 0.015357115, -0.046289552, 0.002699346, 0.011880885, -0.0121743325, -0.004646261, 0.0020597796, 0.0035044472, 0.0020315635, -0.005477697, -0.0037113656, 0.060314864, -0.0045145857, 0.045025468, -0.035785615, 0.050262388, -0.006760592, -0.011369231, -0.009932089, 0.02039088, -0.010165342, -0.040149715, 0.0010308304, 0.0059479666, 0.012460256, -0.029510342, -0.039367188, 0.0135362325, -0.009939613, -0.0066289166, -0.02636518, -0.0070427535, -0.05044297, -0.012219478, 0.013995215, -0.01459716, -0.119726814, -0.0095032025, 0.021053018, -0.05655271, -0.03662834, 0.0023626331, 0.03759145, 0.0038505653, 0.011933555, -0.03202346, -0.009029171, 0.009006598, -0.02299429, -0.010940346, 0.033739004, -0.03443124, -0.03202346, -0.0053685945, -0.012302246, -0.0119636515, -0.019773886, -0.018585045, 0.009947137, 0.0144767715, -0.030759377, 0.011617534, -0.02973607, -0.01590639, 0.022347199, 0.016282605, -0.037531257, -0.0005140044, -0.002208385, 0.022467587, 0.012588169, 0.00893888, -0.0062940847, 0.015311969, 0.01721562, -0.009051744, 0.0012923002, -0.017260766, 0.000364929, -0.01302458, 0.01753164, 0.01094787, 0.0028629997, 0.0053159245, 0.00948063, 0.029510342, -0.027749654, 0.00010698628, 0.010481363, 0.02756907, -0.0002976805, 0.02659091, -0.02157972, 0.01557532, 0.0065800087, 0.026289936, -0.029811313, 0.01351366, 0.05023229, 0.0046349745, 0.009006598, -0.01829912, 0.0012132949, 0.012497878, -0.026094304, -0.0076033147, -0.00607588, 0.006162409, 0.012211954, 0.0190666, 0.02016515, -0.019518059, 0.020646704, 0.021534573, 0.0010007331, 0.014032837, 0.07415959, -0.029796265, 0.00637309, 0.031572003, 0.027854994, 0.0058501507, 0.016899599, -0.012467781, -0.0018321694, 0.035815712, -0.0283215, 1.9560266e-05, -0.0054325513, 0.018901065, -0.02212147, -0.01438648, 0.038644854, 0.036869116, 0.078493595, 0.0056959023, 0.013310503, 0.006764354, 0.0027764703, -0.049600247, -0.053121623, -0.010850054, 0.010684519, 0.0073625366, -0.0023231306, -0.01611707, 0.02484527, -0.002530049, 0.030970057, 0.0066439654, 0.010631849, 0.0016826236, 0.05306143, -0.017787468, 0.022738462, -0.021338942, 0.007975768, 0.016478237, -0.017260766, -0.022151567, -0.018178731, 0.013656622, 0.0030228912, 0.0021237363, 0.016944746, 0.0013694243, 0.023656428, -0.008788394, -0.0063354685, 0.0061661717, -0.0020033473, -0.017411252, 0.016764162, 0.016147168, -0.0067079216, 0.0067944513, -0.012640839, -0.0009903872, 0.057997376, -0.010646897, 0.016478237, -0.039517675, -0.020195246, -0.0019252826, -0.0008864577, 0.024454005, 0.020857386, -0.00081309566, 0.015786001, 0.004349051, 0.02756907, 0.029013736, 0.013385747, -0.019578254, -0.0072383857, 0.026982173, -0.020315636, -0.0031997126, 0.067839175, 0.023295261, 0.004379148, -0.00036681007, 0.0009330144, 0.027584119, -0.012610743, 0.009864369, -0.034250658, -0.003406631, 0.013122396, 0.006817024, 0.01590639, 0.03443124, -0.0097138835, -0.026831686, 0.0106017515, 0.03825359, -0.036146782, 0.053783763, -0.02952539, 0.022587977, 0.033377837, 0.017968051, -0.0037395817, 0.0017070777, -0.01993942, -0.025251582, 0.0073587745, 0.048667233, -0.010872627, 0.00032542637, -0.0031790207, -0.0115949605, 0.036658436, 0.00710671, -0.014732597, -0.004721504, -0.009247377, 0.033588517, -0.00032801286, -0.0038073005, 0.00093818735, 0.05652261, -0.011459523, -0.018976308, 0.0028216161, 0.02963073, -0.033618614, 0.036237072, -0.008931356, 0.008321886, -0.043881774, 0.015229202, -0.0019506773, -0.0123323435, 0.022136519, -0.011707826, -0.018600093, -0.027915187, -0.038313784, 0.00019586718, 0.0024905463, 0.0069675106, 0.033618614, 0.00963864, 0.0036756252, -0.0029307185, -0.014446674, -0.0028197349, 0.012753705, 0.03256521, -0.015334543, 0.015357115, -0.0038110628, 0.0045597316, 0.022061275, 0.013634048, 0.027463729, 0.021820497, -0.027659362, -0.0063016093, -0.02156467, -0.042015743, 0.00056902587, -0.0049171364, 0.008389605, -0.020691851, -0.026891882, 0.020932628, 0.04282837, -0.045928385, -0.020962726, -0.01351366, 0.010579179, 0.030909862, 0.0077500385, 0.039728355, 0.022347199, 0.018043295, -0.00012826597, -0.04400216, 0.0048531797, 0.0032674314, -0.03226424, -0.009029171, -0.016478237, -0.03497299, -0.017817564, 0.03093996, -0.00352702, 0.0069599864, 0.015981633, 0.02669625, 0.00017576317, -0.0015377807, 0.022587977, -0.005594324, -0.0028310213, -0.0033031718, 0.024378762, -0.019096697, 0.024258373, -0.005278303, -0.015545223, -0.012091565, 0.01580105, -0.021624865, 0.013152493, -0.020556413, 0.01831417, 0.0059968745, -0.011850787, -0.013340601, 0.005455124, 0.008622859, 0.027870042, -0.022633122, 0.015424835, 0.020767095, 0.018600093, -0.006297847, -0.011459523, 0.01611707, 0.028788008, -0.045115758, 0.0021820497, -0.005086433, 0.0056770914, 0.017606884, 0.011128454, 0.043099243, 0.016071925, 0.0058764857, -0.0023889681, -0.012490354, 0.00018928341, -0.028968591, 0.0037264142, 0.033528324, -0.006561198, 0.019156892, 0.03509338, -0.018946212, -0.0005629124, -0.031752586, 0.0054137404, -0.010646897, -0.0072835316, 0.0009438306, 0.002526287, 0.008494945, -0.017305912, -0.0024754978, -0.026575861, -0.0045409207, -0.0040518404, -0.01012772, 0.0031677342, 0.027057417, 0.0028799295, 0.01829912, 0.031361323, 0.011685252, 0.015770953, -0.022843804, -0.02540207, 0.010360974, 0.07506251, -0.028803056, -0.023581186, 0.018058343, 0.023927303, 0.010398596, -0.001131468, -0.0018763747, -0.030909862, -0.04475459, -0.039939035, 0.01030078, -0.0010524627, -0.00509772, 0.021504477, 0.0011333491, -0.03608659, -0.015981633, 0.034130268, 0.0076973685, 0.028833153, 0.050954625, -0.01950301, -0.015786001, 0.014303712, 0.034491435, -0.0016845048, 0.00054410164, -0.004525872, 0.0042850943, -0.024754979, -0.03530406, -0.00719324, -0.035936102, -0.016402995, -0.004153419, -0.0023005577, 0.015590369, -0.030548695, 0.012196906, -0.020752044, 0.02081224, -0.0005864259, -0.017080182, -0.012836472, 0.00710671, 0.014025313, 0.036237072, -0.012430159, -0.0174263, -0.026169548, 0.022843804, -0.013144968, 0.006444571, 0.018269023, -0.029073931, 0.002336298, 0.018359315, -0.0082993135, 0.008111206, 0.008284264, 0.02495061, -0.020405928, -0.012896666, -0.009510727, -0.029721022, -0.0035684037, 0.0038599707, -0.037410866, 0.019442815, -0.017351057, -0.023475844, 0.024423908, -0.013957594, 0.033076864, -0.0217904, 0.009337668, 0.0026165785, 0.0077199414, -0.0316021, -0.0013289811, -0.017110279, -0.057034265, -0.02472488, -0.020887483, 0.0022930333, 0.025672944, -0.029389953, -0.028366646, 0.037982713, -0.017080182, 0.02102292, 0.026109353, 0.04093224, 0.018269023, -0.011685252, 0.0033539608, -0.041714773, 0.0039841216, -0.011263891, 0.009992283, 0.00089069013, 0.003086848, 0.0108651025, -0.024800124, -0.014251042, -0.042226423, 0.029555487, 0.028607424, -0.018253975, 0.016056877, -0.013408319, 0.01513891, 0.011752971, -0.00020680093, 0.0027294431, 0.003380296, 0.013257833, -0.01720057, -0.04129341, 0.02865257, -0.009172133, -0.012979434, -0.012430159, -0.00050224765, 0.01733601, 0.0018453369, -0.0017117804, -0.034009878, -0.004228662, -0.0135362325, 0.0381332, 0.02799043, -0.003867495, -0.019096697, 0.0063354685, 0.018449606, -0.01405541, -0.023505943, 0.00032236963, 0.04595848, 0.04024001, -0.009269949, -0.018253975, 0.025281679, 0.025341874, 0.0067154462, 0.015755903, -0.011564864, 0.015635515, 0.0005615016, -0.022482637, 0.011098356, -0.008509994, -0.004913374, -0.025537506, -0.0008394308, 0.011188648, -0.0121743325, -0.016026778, 0.002772708, -0.00093818735, -0.011113405, 0.018028246, 0.02254283, 0.023520991, 0.00011750855, 0.019051552, 0.026530715, 0.0019318664, 0.039818645, -0.0059554907, -0.020330684, 0.010202964, -0.014416576, -0.0074791634, -0.002526287, 0.037290476, -0.0035853335, 0.040601175, -0.01105321, -0.0044619157, -0.007614601, -0.019021453, 0.011135978, 0.007520547, -0.02114331, -0.016929695, 0.017140377, 0.016719015, -0.0015782239, -0.034702115, -0.028998688, -0.015116338, 0.007633412, 0.06386634, -0.03521377, 0.025311777, 0.016372897, 0.003867495, 0.022512734, -0.027659362, -0.020285537, -0.01613212, -0.032595307, 0.026350131, -0.0021011634, -0.0036737442, 0.0017691533, -0.0016139643, -0.00550027, 0.0019807743, -0.03364871, -0.006162409, 0.020541364, 0.018449606, 0.028757911, 0.003621074, 0.011775544, -0.012701035, 0.004420532, 0.011888409, 0.027087513, -0.0022008605, -0.020405928, 0.014935754, -0.011256367, -0.026756445, -0.012053943, -0.008953928, 0.018359315, -0.048396356, -0.025296729, 0.014800317, 0.010503936, 0.009217279, -0.033919588, 0.023566136, 0.010391071, 0.01018039, 0.015259299, 0.028998688, -0.001787964, 0.02440886, -0.026861785, 0.023129728, 0.002432233, 0.015500077, -0.00021820498, 0.025116146, 0.010240586, 0.0035834524, -0.022136519, -0.019126795, -0.007592028, -0.02472488, 0.01155734, -0.007294818, 0.004770412, 0.029540438, -0.03975845, -0.010225536, 0.015041094, 0.012279673, 0.0021782876, 0.011956127, -0.008201498, -0.019232135, -0.029239466, 0.01012772, -0.008931356, 0.021805448, -0.008652956, 0.018539898, 0.005733524, -0.011181124, -0.005135341, 0.034762308, -0.030533647, 0.010609277, 0.00080274977, -0.042978857, 0.0007519607, -0.0056620426, -0.0060232095, 0.0123323435, 0.018765628, 0.0014540728, 0.039487574, -0.04039049, 0.014220945, 0.018675337, -0.0020785904, 0.008103681, 0.0032279287, -0.0062827985, 0.0059705395, -0.027283145, -0.036989506, 0.016929695, -0.034039974, -0.0033501987, 0.0014051648, -0.015424835, 0.014484296, 0.0068734568, -0.0049735685, 0.049841024, -0.0134835625, -0.018780677, 0.016944746, 0.0048983255, -0.003794133, -0.017591836, 0.021925837, 0.0030812046, -0.03479241, -0.01187336, 0.007953195, 0.003790371, 0.008005865, 0.018539898, 0.012475305, 0.007945671, 0.023400603, -0.035394352, 0.008449799, 0.018178731, 0.00012215246, -0.004303905, 0.024785075, 0.028502084, 0.041805062, -0.006147361, 0.013340601, -0.0017240074, -0.01960835, -0.0060608312, 0.011625058, -0.014235993, -0.011188648, -0.023581186, -0.049871124, -0.00893888, 0.01993942, 0.0067229704, -0.004499537, -0.021293795, -0.036778823, 0.011188648, -0.009382814, -0.0068471218, -0.027523924, -0.017802516, 0.013784535, 0.005789956, -0.001302646, 0.007441542, -0.026244791, -0.031993363, -0.021760304, -0.011459523, -0.015786001, 0.028848203, -0.009300047, -0.011451999, -0.023806915, -0.05733524, -0.0024736167, 0.0074264933, -0.018329218, 0.03584581, 0.049208984, -0.028381696, -0.017110279, -0.025025854, -0.025281679, -0.032956474, 0.021399137, -0.043610897, 0.016628724, -0.041714773, -0.016643772, -0.010669471, 0.027102562, -0.017381154, -0.022061275, -0.008449799, -0.011339135], "content_hash": "dfe19cc84fe7f544b03e606f288273e1"}, {"content": "# Interpreting Attribution Scores\n\n## Quick Reference\n\nToken attribution scores tell you how much each input token influenced a specific prediction. Here's how to read and interpret them.\n\n## Understanding the Scores\n\nAttribution scores are **normalized** so that the most influential token gets a score of 1.0, and all other scores are relative to it.\n\n### High Score (0.7 - 1.0)\n\nThis token was **highly influential**. The model relied heavily on this token when making its prediction. For factual predictions, these are usually the content words that carry the key information.\n\n**Example**: In \"The capital of France is\" \u2192 \"Paris\", the token \"France\" typically gets the highest score because it directly determines which capital the model predicts.\n\n### Medium Score (0.3 - 0.7)\n\nThis token had **moderate influence**. It contributed context that helped the prediction but wasn't the primary driver.\n\n**Example**: In the same prompt, \"capital\" might get a medium score -- it tells the model to predict a city name, but \"France\" specifies which one.\n\n### Low Score (0.0 - 0.3)\n\nThis token had **minimal influence** on this specific prediction. It may be a function word (the, of, is) or a word that doesn't directly relate to what's being predicted.\n\n## Comparing Attribution Methods\n\n### Integrated Gradients vs. Simple Gradient\n\n- **Integrated Gradients** averages gradients over many intermediate steps between a \"blank\" baseline and the actual input. This produces more reliable, less noisy scores. Use it when you want trustworthy results.\n- **Simple Gradient** takes a single gradient measurement. It's faster but can be noisy -- scores may overemphasize some tokens or miss subtle contributions. Good for quick exploration.\n\n**When they disagree**: If the two methods give very different rankings, the attribution is likely noisy. Trust Integrated Gradients for the more accurate picture.\n\n## Why Results Vary by Target Token", "source_file": "interpreting_attribution_scores.md", "chunk_index": 0, "embedding": [-0.0010772948, 0.007208415, 0.012402624, -0.02312064, 0.025464443, -0.018262131, -0.005282712, 0.0033783722, -0.025440028, 0.009936748, -0.030225292, 0.013367001, -0.0531262, -0.028613929, -0.015039402, -0.0029343315, -0.00014057478, -0.001954695, -0.015796255, 0.038770407, -0.04746201, -0.023962945, 0.05376098, 0.017529693, 0.04116304, -0.016345585, -0.030347366, -0.025635345, -0.009808571, 0.02185108, -0.0010849244, -0.014331379, -0.013061819, -0.056104783, 0.0024933424, -0.026538685, -0.025317956, 0.009637669, 0.03549885, -0.03237378, -0.012536905, 0.017432034, -0.04433694, 0.00023479993, -0.025415614, 0.0052918675, 0.049805813, -0.07900569, -0.025830662, 0.07910335, -0.05117303, 0.021741213, 0.0021439083, 0.013806464, -0.018323168, 0.028858075, -0.01099268, 0.004425149, -0.027979149, -0.053272687, -0.020752423, -0.02719788, 0.051124204, 0.010186998, 0.038453016, -0.015564317, -0.03217846, 0.0068055736, -0.0067689517, -0.037842654, 0.056788392, 0.031128634, 0.06772614, -0.0019058659, -0.038770407, -0.027735002, -0.009216517, 0.043726575, -0.029297538, -0.031299535, 0.023767628, -0.005862559, -0.008752639, -0.019348582, -0.0045014443, -0.019043399, -0.008398627, -0.053370345, -0.0042756093, -0.046900474, -0.041236285, -0.02443903, -0.004391579, 0.010852296, 0.01627234, -0.03576741, -0.046802815, -0.02130175, -0.001602209, -0.0074952864, 0.008740432, -0.026685173, 0.022546897, -0.017737217, 0.021509275, -0.041871063, 0.002853458, -0.022424823, 0.003668296, -0.015039402, -0.022607934, 0.036499847, -0.052930884, 0.02243703, -0.0017807409, -0.03540119, -0.077003695, 0.03962492, 0.004684554, -0.029004563, -0.013818672, -0.0489513, 0.003259351, 0.030762414, 0.014709805, -0.028369782, 0.011853295, -0.018457448, -0.025952736, -0.044898476, -0.003787317, -0.016760632, -0.025562102, -0.01627234, -0.009741431, -0.012213411, 0.032349363, 0.019043399, -0.04270116, 0.02357231, -0.031153047, -0.024085017, -0.01373322, -0.010205309, -0.063526824, -0.0018753476, -0.011950954, 0.031177463, -0.06089005, 0.03313063, 0.011206308, -0.017895913, -0.04318945, -0.016650766, -0.054298103, -0.019507278, 0.011407728, 0.03068917, -0.0015594835, -0.055567663, -0.0065309093, 0.039917894, 0.019165473, -0.050928887, -0.041846648, 0.01962935, -0.047291107, -0.01563756, -0.009381316, -0.032569095, -0.037256703, -0.043140624, -0.012854295, -0.036426604, 0.00489513, -0.032569095, -0.05571415, -0.028638342, 0.022998566, -0.015320171, -0.02683166, 0.008416939, -0.02058152, 0.022888701, 0.020825665, 0.000917074, -0.041553672, 0.015307963, -0.003100656, -0.0015747426, 0.035254702, 0.017346583, 0.07910335, 0.014941744, -0.038745992, -0.043995135, 0.0486095, 0.03969816, 0.0024429874, 0.016711803, -0.001076532, 0.0011291759, 0.037232287, -0.02452448, -0.011102546, 0.0023880545, 0.0017670078, -0.04189548, -0.07426926, -0.009411834, -0.0018738216, -0.043213867, 0.02751527, 0.004583844, -0.09536348, 0.010956058, -0.025415614, 0.0486095, 0.035645336, 0.0088136755, 0.007403732, 0.011596941, -0.009631566, -0.031616926, -0.050196446, 0.010193101, 0.050538253, -0.019372996, 0.059815805, -0.055421174, 0.045484427, 0.03386307, -0.019202095, -0.027100222, -0.016064817, -0.044825234, -0.0011383314, -0.01004051, 0.025952736, -0.046461012, -0.00073014956, -0.00022030374, 0.04714462, 0.049708154, 0.000374421, -0.007245037, -0.02384087, -0.038745992, 0.03535236, 0.0075135976, 0.030664755, -0.0062806597, -0.030152049, 0.03640219, 0.0059113884, -0.014856293, 0.013635562, 0.024597723, 0.014502281, 0.0026337265, -0.009662083, -0.010333486, -0.035742994, -0.02592832, -0.02570859, -0.014319171, -0.0390878, 0.029102221, -0.058741562, -0.034473434, -0.009295864, -0.016333377, 0.009185999, -0.01264677, -0.0027405405, -0.016394414, -0.04384865, 0.01359894, -0.033887483, -0.023059603, -0.0035645338, -0.008563426, 0.04633894, 0.0137820495, -0.028247708, -0.015369, 0.008252139, 0.052735567, 0.021387203, 0.009424041, -0.009759742, -0.026685173, 0.04238377, -0.034473434, -0.025586516, 0.0071168602, 0.01890912, -0.0020157315, -0.042994134, -0.027075807, 0.030200878, -0.038477432, 0.007946957, 0.010650875, 0.029297538, -0.012768843, -0.0004333594, 0.026441026, 0.016394414, -0.0140384035, -0.030005561, 0.009027303, 0.007220622, 0.033106215, -0.039478432, 0.053419176, 0.06513819, -0.03242261, 0.017712804, 0.022925323, -0.008117859, -0.05825327, 0.032837655, 0.0063111777, -0.007599049, 0.010498284, -0.0122988615, 0.0020477758, -0.0108767105, 0.016467657, 0.021362787, 0.009399626, -0.009051718, 0.060011122, -0.06460107, -0.0010933169, -0.021228507, 0.022949737, -0.0067689517, 0.048389766, -0.03540119, -0.04265233, 0.004455667, -0.014502281, 0.029346367, -0.014795256, 0.009497285, 0.03762292, 0.035718583, -0.0024048395, 0.017212303, 0.002851932, -0.0077943658, -0.007855402, 0.023230506, 0.041553672, -0.012335484, 0.012219515, -0.02624571, -0.016345585, -0.018371997, -0.009253139, 0.0243902, 0.02053269, 0.008435249, -0.061085366, 0.013501282, 0.016943742, -0.005743538, -0.024561102, -0.008075134, 0.043506842, -0.055128198, -0.031299535, -0.02793032, 0.009234828, -0.004150484, 0.018750424, -0.036109213, -0.0012932116, 0.03164134, -0.021741213, -0.008117859, -0.033325948, 0.048487425, 0.007916438, 0.017810462, -0.008123963, -0.018103436, 0.036060385, -0.03371658, -0.008593944, -0.009259243, 0.05004996, 0.013379209, 0.058643904, -0.009308072, 0.03994231, 0.053321518, 0.042359356, -0.026123637, 0.0077943658, -0.071925454, -0.010003888, 0.03440019, 0.075880624, -0.01767618, 0.023596724, 0.018396411, 0.0563001, 0.05263791, -0.034204874, 0.029224293, -0.0563001, 0.0074403537, 0.010199205, 0.050196446, 0.028223295, 0.045850646, 0.03747643, -0.020300752, -0.051710155, -0.0030258864, -0.050928887, -0.013586733, -0.012805466, 0.005359008, -0.012463661, 0.025317956, -0.015832877, 0.016931536, 0.015417829, 0.04121187, -0.03630453, -0.008679396, -0.007452561, 0.06567531, 0.0069520613, 0.0015488021, 0.006604153, 0.0155887315, -0.024182675, -0.041724578, -0.07715018, -0.038501848, -0.011926539, -0.03127512, -0.06460107, -1.353056e-05, 0.007336592, -0.0011665608, 0.039966725, -0.0013923959, 0.04079682, -0.03142161, -0.01940962, 0.044214867, 0.020544898, 0.033472437, -0.036158044, 0.057276685, 0.003436357, -0.00729997, 0.006091446, 0.008777054, -0.022266129, 0.0005157587, -0.02619688, -0.0013786628, 0.011725118, 0.008850298, -0.012939746, -0.015442244, -0.039234284, 0.0064210435, -0.025269127, 0.060450584, -0.03440019, -0.006219623, -0.010211413, -0.013721013, -0.0122988615, -0.037012555, -0.012817672, -0.013757635, 0.025781833, 0.023694383, 0.013171684, -0.01767618, -0.024402408, -0.012573526, -0.067481995, 0.014807464, -0.011761741, 0.012018094, -0.02507381, 0.019202095, -0.013147269, 0.0064576655, -0.033325948, 0.04316504, -0.034033973, -0.045337938, 0.08437691, -0.01949507, 0.01092554, 0.07280438, 0.005667242, -0.04316504, 0.034619924, -0.029712586, -0.01067529, -0.022754421, 0.0005947247, -0.043458015, 0.011065924, -0.018384205, 0.047974717, 0.010705808, 0.004269506, 0.011389418, -0.003958219, -0.02058152, -0.008392524, 0.0170414, -0.016113646, -0.027905904, -0.019641558, -0.032129634, -0.05000113, 0.007855402, -0.020776836, 0.016528694, 0.035034973, -0.040284112, 0.0029740052, 0.010577632, -0.02729554, -0.049610496, 0.010449455, 0.021631349, -0.014783049, 0.023535687, 0.029444024, 0.012524697, 0.006021254, -0.00228887, -0.03269117, -0.00096437725, 0.011035406, -0.033667754, -0.041431602, 0.038330946, -0.025586516, -0.01745645, -0.029028976, 0.022864286, -0.005316282, -0.0148196705, -0.025049394, 0.0139773665, -0.013367001, 0.010504388, 0.0093935225, -0.024683176, 0.011932643, 0.028223295, 0.026441026, 0.024902906, 0.014514488, 0.021741213, 0.0319099, 0.008764846, 0.03295973, 0.014380208, 0.0058015226, 0.0021179677, -0.0070008906, 0.03662192, -0.0126833925, -0.003436357, -0.0098451935, -0.007849298, -0.0068116775, 0.00054513256, -0.012500282, 0.019116644, 0.0027756365, 0.009185999, 0.032837655, 0.011676289, 0.012866502, 0.014404623, 0.018530693, -0.019824667, 0.054884054, 0.016308961, -0.023193883, -0.044507843, 0.013989574, 0.017102437, -0.03916104, -0.009570529, -0.0012764266, 0.01405061, 0.030176463, -0.013891916, 0.018640557, -0.0058411965, 0.0016296755, -0.026099222, -0.054200444, -0.0039368565, 0.010797363, -0.0095888395, 0.011889917, -0.02076463, 0.027686173, -0.009228724, -0.031226292, 0.0040924996, -0.016919328, -0.01309844, 0.014929537, 0.0040802923, 0.038208872, 0.007953061, -0.0074281464, 0.018555107, -0.0010284656, -0.0329109, 0.029004563, 0.009558322, 0.0047120205, 0.017065816, 7.55804e-05, -0.0058045746, -0.0030945523, 0.004071137, 0.0044526155, 0.02393853, 0.03630453, -0.011194101, -0.007104653, -0.0010505914, -0.02081346, 0.005545169, 0.008593944, -0.023230506, -0.03010322, -0.03386307, -0.031714585, 0.018542899, -0.007525805, -0.004138277, -0.028662758, 0.02434137, 0.008282658, -0.04206638, -0.0137820495, -0.0023193883, -0.015344585, -0.057911467, -0.020935532, -0.020544898, 0.011413832, -0.004354957, 0.026758417, 0.0053620595, 8.7024746e-05, -0.08598827, 0.022754421, 0.00092928123, 0.039746992, -0.0009956585, 0.0072816587, 0.010870607, -0.035059385, -0.024280334, 0.030298537, -0.0012382787, 0.013708806, -0.027026977, -0.0477794, -0.03264234, -0.026392197, 0.01117579, -0.0028595617, 0.02008102, -0.0571302, -0.022888701, -0.0022415668, -0.014929537, -0.006781159, -0.03762292, -0.006555324, 0.023438029, 0.055958297, -0.0037598505, -0.060987707, 0.016162476, -0.024011774, -0.03295973, -0.010962162, -0.029346367, 0.01867718, 0.009045615, 0.016247926, 0.00549634, -0.0681656, -0.041773405, 0.000987266, 0.01047387, -0.01054101, -0.010974369, 0.03396073, -0.0025864232, -0.0020355685, -0.0115236975, -0.0583021, -0.01727334, 0.0103640035, -0.04565533, 0.009564425, 0.010919436, -0.016199097, -0.022644555, -0.009124962, -0.023315957, 0.020715801, -0.0013130485, 0.0013290705, -0.018140059, -0.03364334, -0.0023178624, 0.007104653, -0.02008102, 0.03413163, -0.01428255, -0.009832986, 0.0024368837, -0.016540902, -0.008551219, -0.01899457, 0.003952116, 0.016516486, -0.025537686, 0.025562102, -0.017261133, -0.044092793, 0.0035736891, 0.0088747125, 0.025464443, -0.02012985, 0.031763412, 0.0022415668, 0.00472728, -0.03381424, 0.0024704537, 0.030908901, -0.002642882, 0.029785829, -0.023022981, 0.0032288327, -0.008484079, -0.0039917897, 0.019482862, 0.029248709, -0.024207091, 0.044922892, -0.017151266, -0.007470872, 0.012732222, -0.011664082, 0.028394196, -0.013183892, 0.032935314, -0.0030228344, -0.030200878, 0.0043976824, 0.02153369, -0.008050719, -0.01110865, 0.016150268, 0.022827664, 0.020630348, -0.0075441157, -0.044898476, -0.0030182567, -0.016601937, -0.026465442, 0.023047395, 0.019971155, -0.037281115, -0.024011774, 0.032031976, -0.01976363, -0.095998265, -0.0028259917, 0.052247275, -0.040186454, 0.003414994, -0.0028748207, 0.017151266, 0.0051331725, -0.043262698, 0.017163474, -0.023242712, -0.008844194, -0.009424041, -0.024121638, -0.019434033, -0.028418612, -0.023474652, -0.014514488, -0.0038147834, -0.0024368837, -0.012671185, 0.002026413, 0.009192103, -0.02135058, -0.030493854, 0.0026123638, -0.023792041, 0.00040665592, 0.004348853, 0.031128634, -0.011377211, 0.008624462, 0.029517269, -0.0036072594, 0.0021027087, 0.03671958, 0.003329543, 0.04084565, -0.0010528802, -0.023547895, 0.028247708, -0.032764412, 0.028613929, -0.011773948, 0.064161606, -0.058106784, 0.0026825557, 9.691934e-06, 0.0056153615, 0.017822668, -0.0063844216, 0.020984361, 0.0010895021, -0.0076906034, 0.03078683, 0.013244928, 0.02793032, -0.030493854, -0.013049611, 0.034888484, -0.0045868955, -0.0024124691, 0.006054824, 0.007312177, 0.019958947, 0.018872498, 0.0014290179, -0.0043213866, -0.027490856, -0.031177463, -0.009710913, 0.023108432, 0.0037049176, -0.018005777, -0.017554108, 0.002116442, 0.026441026, -0.0059571657, -0.058350928, 0.02474421, 0.069483995, -0.009729224, 0.012781051, 0.033032972, 0.00972312, 0.027026977, -0.011145271, 0.0067750555, -0.0069703725, 0.05141718, 0.041871063, -0.010638668, 0.020373996, -0.0042145727, -0.045264695, 0.0035004453, 0.04038177, 0.044751987, 0.020886702, -0.013220513, 0.017224511, 0.013574526, -0.039600503, -0.005081292, -0.02375542, -0.008172792, -0.0148196705, -0.02788149, -0.011822777, 0.007946957, 0.017810462, 0.047632914, 0.0036957623, 0.014343586, -0.023657762, 0.0017609041, 0.021948738, 0.01605261, 0.02108202, -0.0022659814, 0.0044068377, -0.0032562993, -0.0107485335, 0.0070863417, -0.018213302, 0.06323385, 0.0035828447, -0.00613112, 0.011633564, 0.004296972, 0.016211305, -0.0010200731, -0.002253774, -0.018799253, -0.02624571, 0.020251922, 0.016894912, -0.031397194, -0.037525263, 0.017773839, -0.02058152, 0.031543683, 0.040723577, -0.0057008127, -0.073585644, -0.021509275, -0.025244711, -0.053419176, 0.016577523, 0.014783049, 0.017883705, 0.013623355, -0.017602937, 0.011480972, -0.0065064947, 0.033570096, 0.0175419, -0.025220297, -0.0032990247, 0.034619924, -0.0052094683, 0.029517269, 0.022046397, 0.011297863, 0.06723785, -0.021692384, 0.0037049176, 0.004190158, 0.029639341, -0.016870499, -0.025244711, -0.03689048, 0.017651767, 0.031934317, 0.018213302, 0.015686389, -0.0002273611, 0.007220622, -0.0056153615, 0.029028976, -0.0022293595, 0.012347691, -0.009185999, 0.026270125, 0.025415614, -0.031616926, 0.012097442, -0.013830879, -0.038965724, -0.00067254633, 0.011126961, 0.03371658, 0.011596941, 0.04248143, -0.04538677, 0.0016067869, 0.020373996, -0.006836092, -0.000107576896, 0.029639341, -0.031348366, -0.011145271, 0.031348366, -0.005020255, 0.02026413, 0.028540684, -0.031153047, -0.006170794, 0.05263791, -0.0009811623, -0.0046601393, 0.005728279, -0.040772405, -0.00061913935, -0.028125636, 0.00954001, -0.03132395, -0.015979365, -0.016919328, 0.002676452, -0.039502844, -0.029785829, 0.015784048, 0.004684554, 0.009295864, -0.0035706374, 0.047217865, -0.026099222, 0.026123637, -0.00031491037, -0.03630453, 0.0047852644, -0.029395195, -0.00970481, -0.004617414, 0.0140384035, 0.012042508, 0.0056886054, 0.037744995, 0.008777054, 0.034717582, 0.016247926, -0.031104218, -0.022461446, -0.03169017, -0.031372778, 0.0037903688, 0.022266129, -0.0052979714, 0.0034302534, -7.295774e-05, 0.05117303, 0.0456065, -0.04968374, 0.020239715, -0.024097225, 0.025513273, 0.029566098, 0.02919988, 0.029639341, 0.00324104, 0.01092554, 0.013769843, -0.017493071, 0.0074647684, -0.029444024, -0.0019272286, 0.0053284895, 0.003399735, 0.023132848, -0.04411721, 0.01645545, 0.032764412, 0.034522265, 0.004336646, -0.024866285, 0.0045014443, -0.026367784, -0.0026856076, -0.0036896586, -0.025244711, 0.013537903, -0.03322829, -0.016492072, 0.020410618, -0.003964323, 0.021692384, -0.008197207, 0.0034851863, 0.015515488, -0.016931536, 0.021594726, 0.013879708, 0.007379317, -0.009912333, 0.013830879, 0.017505279, 0.0008171266, 0.003137278, -0.023022981, 0.018054606, -0.0017715854, -0.0038666646, -4.3154738e-05, 0.008184999, 0.023865286, -0.004300024, -0.010058821, -0.0024216245, -0.0018829771, -0.014392415, 0.010956058, -0.030860072, 0.05141718, 0.014795256, -0.025147052, -0.039380774, -0.04941518, -0.012036405, -0.009753639, 0.01212796, 0.009594943, -0.0109438505, 0.04487406, 0.012378209, -0.00032692694, 0.020251922, 0.007904232, -0.015369, 0.00507824, -0.0137820495, 0.007397628, 0.025366785, 0.0029633238, -0.05434693, 0.013855293, -0.008496285, -0.017761633, -0.009967267, -0.055518832, 0.0049805813, 0.0070863417, 0.02484187, -0.03269117, 0.046436597, -0.00977195, 0.008331487, 0.020740215, 0.0077089146, 0.029395195, 0.0706559, -0.013061819, 0.011981472, 0.004498393, -0.008252139, 0.023535687, -0.052784394, -0.004348853, -0.032886487, 0.0025665862, 0.005514651, 0.037647337, 0.027857075, -0.020007776, 0.0019485914, 0.008716017, 0.0023727953, 0.022998566, 0.033155046, 0.028174466, 0.00087816315, 0.029273123, -0.0064576655, 0.009662083, -0.013696599, 0.009936748, 0.0014870026, 0.019934533, 0.038770407, 0.006100602, -0.004846301, -0.028418612, 0.013696599, -0.010962162, -0.03603597, 0.014746427, 0.0016876602, -0.0013687443, -0.021094227, -0.00042420393, 0.014575525, 0.0048340936, 0.00044632968, 0.009167688, -0.015832877, -0.0001702347, 0.026489856, -0.015283548, -0.011474868, 0.0040131523, 0.002819888, 0.001744119, -0.017236717, -0.03747643, 0.01849407, -0.017932534, 0.001513706, -0.011023198, -0.0073426953, -0.02111864, 0.013830879, 0.023877492, -0.02044724, -0.011993679, 0.02103319, -0.027319953, 0.016846083, 0.004269506, -0.0051789503, 0.01201199, -0.024719797, -0.0085207, -0.022681177, 0.020862289, 0.018799253, -0.0126223555, -0.0025955788, 0.0021225456, 0.023376994, -0.027539685, -0.016870499, -0.016235719, -0.043238282, -0.001832622, -0.029395195, -0.017285546, 0.026074808, -0.0011177316, -0.029907903, 0.01759073, -0.023718799, 0.0034119424, 0.0059632696, 0.023633346, 0.0023315956, -0.01867718, 0.010785156, 0.013342586, 0.003050301, 0.019556107, 0.017151266, -0.02397515, -0.046973716, -0.0155887315, 0.010132065, 0.01099268, -0.03237378, 0.0131350625, 0.03762292, 0.0069886832, 0.04851184, -0.0038697163, 0.017016986, 0.007220622, -0.018005777, 0.022546897, -0.0105532175, 0.05620244, -0.009930644, -0.024646552, -0.0036347257, -0.012091338, -0.0037049176, -0.019665971, 0.011639668, 0.019067815, 0.024207091, 0.008612256, -0.036231287, -0.006659086, 0.02108202, 0.0017136007, 0.039283115, 0.011279552, -0.019421825, 0.005038566, -0.0034027868, 0.020837873, 0.012042508, 0.011548112, 0.009039511, 0.0063539036, -0.025635345, -0.038453016, 0.0060883947, -0.012481972, 0.027173465, 0.02729554, -0.0063111777, -0.014929537, -0.0011474869, -0.0130740255, 0.00743425, 0.00384225, -0.0031525372, -0.030225292, -0.027564099, 3.7027243e-05, -0.02425592, 0.0133303795, 0.026148051, -0.0088747125, -0.026587514, -0.005774056, 0.014941744, 0.00096590316, -0.014648769, 0.022058604, 0.02031296, -0.0019852133, -0.01984908, -0.011816673, 0.006872714, 0.030908901, -0.027954733, -0.012561319, 0.006591946, 0.0039338046, -0.00686661, 0.026270125, -0.027783832, -0.02968817, 0.0023743212, -0.009363005, -0.033936314, -0.0077028107, 0.009143273, -0.00908834, 0.016882706, -0.0059968396, -0.011444351, -0.0626479, -0.0036377776, -0.005090447, 0.026782831, 0.04250584, -0.015881706, 0.025098223, 0.006323385, 0.008960163, -0.013757635, 0.009381316, 0.030640341, -0.021008777, -0.012231722, 0.032666754, -0.027075807, -0.03535236, 0.008514597, 0.026270125, 0.009594943, -0.0042206766, 0.009942852, 0.01809123, 0.006958165, -0.010736327, -0.0077638472, -0.0097170165, 0.018970156, -0.013708806, 0.0008827409, -0.003451616, -1.252918e-05, -0.020776836, -0.022095226, -0.008435249, 0.008172792, -0.000748842, 0.01912885, -0.015857292, 0.003469927, -0.024158262, -0.014294757, 0.009460663, 0.019263132, 0.016846083, -0.04455667, -0.020117642, 0.004538066, 0.033521265, -0.008795365, 0.009643773, -0.03413163, 0.013696599, 0.0077821584, 0.024780834, 0.025903907, -0.01024193, 0.01062646, -0.01668739, 0.03549885, -0.017200096, -0.022888701, -0.011926539, -0.005389526, -0.045044962, -0.0020859237, 0.0051728464, 0.010864504, 0.0031281225, -0.080568224, -0.0009300442, 0.031519268, 0.0054536145, -0.0005443696, 0.014673184, -0.0014549585, -0.013830879, -0.03010322, -0.012066923, -0.0055207545, -0.0078431945, -0.0033020766, -0.017712804, 0.044214867, 0.037159044, 0.04221287, -0.019372996, -0.019739216, -0.0011474869, -0.00289008, 0.025317956, -0.01790812, -0.004153536, 0.015710805, 0.008654981, 0.011096442, 0.030664755, -0.013367001, -0.016553108, 0.00055962877, 0.038965724, 0.01428255, -0.032617923, -0.028711587, 0.012524697, 0.011346692, -0.027417611, -0.009051718, 0.016711803, 0.016235719, -0.0051820017, 0.026025979, -0.02166797, 0.020605935, 0.035059385, -0.018213302, 0.022656763, -0.035376776, 0.026343368, 0.021704592, -0.004669295, -0.025366785, -0.011474868, 0.014856293, -0.015784048, -0.0133303795, -0.008496285, -0.012817672, 0.0022003672, -0.0016113645, -0.006094498, 0.022925323, -0.01981246, -0.004498393, 0.018469656, 0.013171684, 0.042994134, -0.0039551677, 0.0024979203, 0.02312064, 0.024451237, 0.025171468, 0.01759073, 0.0033539576, -0.0035187562, 0.005322386, -0.006903232, 0.0047486424, -0.022339372, 1.42696745e-05, 0.0068910248, 0.008752639, -0.0015457503, 0.020044398, 0.028809246, -0.008587841, -0.008300968, -0.031934317, -0.023108432, 0.008374212, 0.024036188, 0.013354794, 0.0025864232, 0.0074769757, 0.024646552, 0.025220297, -0.015430036, -0.008539012, -0.029126635, 0.0053620595, -0.006622464, 0.00055581395, 0.026318954, 0.012793258, 0.024780834, -0.0067018117, -0.055958297, -0.0027832661, 0.013550111, 0.0041352254, 0.0073549026, 0.016931536, 1.1724499e-05, -0.009320279, 0.022192884, 0.0031952627, -0.0074220425, 0.008416939, -0.018555107, 0.04106538, -0.009350797, -0.019507278, 0.0089418525, 0.028174466, -0.020227509, -0.027735002, 0.0034973936, 0.0060029435], "content_hash": "dce30421164d4389bf4067fa798e236b"}, {"content": "faster but can be noisy -- scores may overemphasize some tokens or miss subtle contributions. Good for quick exploration. **When they disagree**: If the two methods give very different rankings, the attribution is likely noisy. Trust Integrated Gradients for the more accurate picture. ## Why Results Vary by Target Token\n\nAttribution is computed **with respect to a specific target token**. Different targets can be driven by entirely different input tokens.\n\n**Example** with prompt \"Alice gave Bob a gift because she\":\n- Target \"liked\": High attribution for \"Alice\" (she liked something) and \"Bob\" (she liked Bob)\n- Target \"was\": High attribution for \"she\" and \"Alice\" (describing Alice's state)\n- Target \"wanted\": High attribution for \"gift\" (she wanted to give a gift)\n\nThis is one of the most powerful uses of attribution -- it reveals which input tokens support different possible continuations.\n\n## Common Patterns\n\n- **Content words dominate**: Nouns, verbs, and adjectives typically have higher attribution than function words\n- **Recent tokens often matter more**: Tokens closer to the prediction point tend to have higher attribution, especially for local patterns\n- **Distant tokens can matter too**: For long-range dependencies (like pronoun resolution), distant tokens can have surprisingly high attribution\n- **Punctuation varies**: Commas and periods sometimes have notable attribution because they signal sentence structure\n\n## Tips\n\n- Try the same prompt with multiple target tokens to see how attribution shifts\n- Short prompts (5-10 tokens) give the clearest attribution results\n- If all scores are roughly equal, the model may be uncertain or the prediction may not depend on any single token\n- Use attribution alongside ablation for a fuller picture: attribution tells you which input tokens matter; ablation tells you which internal components matter", "source_file": "interpreting_attribution_scores.md", "chunk_index": 1, "embedding": [-0.00022011965, 0.012414076, 0.028954975, -0.010653119, 0.032396235, -0.017058432, -0.0025893464, 0.043042634, -0.015324361, 0.024424879, 0.020217402, 0.0072253016, -0.029116284, -0.039332524, -0.044897687, -0.018227924, -0.0005784442, -0.026696648, -0.020674445, 0.043822292, -0.036321424, -0.0219918, 0.05352772, 0.018832833, 0.015216821, -0.025110444, -0.022045571, -0.02609174, 0.009450022, 0.013166851, 0.009523955, -0.0153378025, -0.0074672652, -0.062910534, -0.018644638, 0.025742237, 0.0031824932, 0.002959013, 0.020069536, -0.027382212, 0.006983338, 0.012528337, -0.060598437, -0.03836467, -0.02825597, 0.023241946, 0.025231425, -0.0439836, -0.02531208, 0.09753821, -0.07500871, 0.059738122, -0.026562225, 0.022569826, -0.030299218, -0.0074202167, -0.00011783122, 0.0021003783, -0.016238445, -0.040649883, -0.023618335, -0.0439836, 0.026884843, 0.02137345, 0.027315, -0.020526577, -0.05629686, 0.00092416647, 0.002086936, -0.010310337, 0.05156513, 0.040891845, 0.036348306, -0.004835911, -0.037746318, -0.038337786, -0.00011342043, 0.032450005, 0.0065296562, -0.028954975, 0.037127968, -0.035568647, 0.0034177357, -0.030164793, -0.028847435, -0.01576796, -0.050812352, -0.037826974, -0.037988283, -0.05207594, -0.030352987, 0.0055752443, 6.348394e-05, -0.0026632797, 0.02593043, -0.021696068, -0.032611314, -0.013187015, 0.023282275, 0.0069698957, 0.0077092284, -0.020566905, -0.0051148413, -0.030998224, 0.02814843, -0.027798928, -0.0031287235, -0.022300977, -0.0018248087, 0.0033152371, -0.023443583, 0.024344224, -0.069900595, 0.032907046, -0.009113962, -0.036428962, -0.060759746, 0.02470717, 0.015808288, -0.03331032, -0.028175315, -0.015109282, -0.0031959356, 0.038203362, 0.032181155, -0.015902385, 0.018080058, -0.027556963, -0.007561362, -0.010081816, -0.0124342395, 0.01088164, -0.02176328, 0.0035051114, -0.016493851, 0.0066943257, 0.007077435, 0.01699122, -0.06570656, 0.02359145, -0.022717692, -0.028390395, 0.007500871, 0.013200458, -0.061888907, -0.0037470749, -0.021938032, 0.033713594, -0.05223725, 0.06162006, -0.006889241, -0.019625936, -0.03874106, -0.0011098399, -0.05629686, -0.03443948, -0.016372869, 0.0065531805, -0.017260069, -0.053635262, 0.03314901, 0.024290456, -0.00015227743, -0.025110444, -0.02653534, 0.02914317, -0.036267653, 0.008912326, -0.033498514, -0.03747747, -0.041429542, -0.050812352, -0.026602551, -0.025056673, 0.027772043, -0.04503211, -0.032772623, -0.029788407, 0.019921668, -0.0116814645, -0.004456163, 0.0080788955, -0.0012761899, -0.012945052, -0.015149609, -0.03142838, -0.052049056, 0.032019846, -0.00072169, 0.0020382071, 0.032342467, 0.015929269, 0.077750966, 0.008730853, -0.028820552, -0.057210945, 0.07576149, 0.05446869, 0.005118202, -0.022637038, 0.014410276, 0.02082231, 0.04513965, -0.02348391, -0.015848614, 0.018846275, 0.009510513, -0.026051413, -0.08328924, -0.014921088, 0.017623015, -0.057641104, 0.028659243, 0.013818809, -0.08221385, -0.03129396, -0.02254294, 0.015781403, 0.029788407, 0.036240768, 0.009913785, 0.016964335, 0.007957914, -0.036186997, -0.042666245, 0.03704731, 0.05091989, -0.03930564, 0.044171795, -0.08732197, 0.034278173, 0.020956734, -0.028336624, -0.03191231, 0.009329041, -0.045758, 0.006089417, -0.018805947, 0.03597192, -0.038660403, 0.006334741, -0.004570423, 0.04796256, -0.012965215, -0.014181755, 0.014880761, -0.046752743, -0.021790165, 0.026602551, 0.022220323, 0.039493833, -0.022973098, -0.05925419, -0.0034227765, -1.3508058e-05, -0.036025688, 0.01598304, 0.014235524, 0.014181755, 0.0064153955, -0.00015427278, 0.015888942, -0.025338965, 0.018174153, -0.027556963, -0.02559437, -0.025459945, 0.029196938, -0.025836334, -0.026804188, -0.019209221, -0.014773221, 0.022408517, 0.001532436, -0.025164213, -0.0037201901, -0.032019846, -0.0068052257, -0.01959905, -0.02814843, -0.024572747, 0.008811507, 0.06156629, 0.009127405, -0.04118758, -0.014329622, 0.0042410837, 0.040811192, 0.006895962, 0.016507294, -0.00031988762, -0.04218232, 0.045274075, -0.022999983, 0.004348623, -0.0007926828, 0.04551604, -0.015351245, -0.022126226, -0.024653401, 0.031455267, -0.0120309675, 0.012924888, 0.0008023445, 0.0365365, -0.004691405, 0.027449425, 0.038606633, 0.0282022, -0.0027792207, -0.019195778, 0.02353768, -0.007346283, 0.031751, -0.027852697, 0.0024078737, 0.0498445, -0.042343628, 0.026817631, 0.023470467, -0.0014912686, -0.064631164, 0.041886587, 0.012750137, 0.009295435, 0.008845113, 0.0009308877, -0.0022751298, -0.019894784, -0.0013870898, -0.0137583185, 0.020257728, -0.01110344, 0.0462957, -0.026548782, -0.00038625958, -0.036240768, 0.0031673706, -0.0013795285, 0.04315017, -0.040622998, -0.06387839, 0.016977778, -0.022368189, 0.012756858, 0.009355925, 0.009443301, 0.040945616, 0.049037952, -0.0071580894, 0.0073798895, -0.012608991, -0.0020701329, 0.008589707, 0.0019877981, 0.025809448, -0.02121214, -0.012481288, 0.024774382, 0.0031455266, 0.0023389813, 0.013926349, 0.018012844, 0.033579167, -0.010901804, -0.012420798, 0.010350664, 0.0067480956, -0.0051921355, -0.004674602, -0.032530658, 0.021776723, -0.03637519, 0.005450902, -0.049441226, 0.027610734, -0.0048291897, -0.000516273, -0.0074672652, 0.029976599, 0.023403255, -0.056028012, -0.020513136, -0.009644938, 0.039440066, 0.0058877803, 0.012367028, -0.023241946, -0.043284595, 0.015310918, -0.006170071, -0.0134088155, -0.050570387, 0.046752743, 0.0043822294, 0.03196608, 0.0072521865, 0.062157758, 0.049145494, 0.06753472, -0.021857377, 0.015956154, -0.06162006, 0.008952653, 0.03293393, 0.06156629, -0.006227202, 0.032530658, 0.031347726, 0.030487413, 0.042101663, -0.020311499, 0.03814959, -0.011741955, 0.024344224, 0.00019291975, 0.02265048, 0.007897423, 0.054656886, 0.040058415, -0.016534178, -0.05468377, -0.027879583, -0.082052544, -0.01793219, -0.0056424565, 0.018120384, -0.014168312, 0.030541182, -0.011930149, -0.00029405297, 0.028444164, 0.054656886, -0.039440066, -0.01876562, -0.032342467, 0.06178137, 0.045569807, 0.0038310902, -0.012259488, 0.02887432, -0.042451166, -0.050839238, -0.05336641, -0.06791111, -0.026333703, -0.041698392, -0.05463, 0.009725592, 0.00699678, -0.019357087, 0.026387474, -0.00067296124, 0.039278757, -0.011426059, -0.008011683, 0.057641104, 0.014611912, 0.041725274, -0.0032278614, 0.051108085, -0.031025108, -0.02619928, 0.004009202, 0.012353585, 0.013570124, -0.0020382071, -0.016803026, 0.027772043, 0.012541779, 0.000102340935, -0.018469887, -0.004862796, -0.053608377, 0.006916126, -0.016372869, 0.049226146, -0.030460527, 0.0039386293, -0.019558724, -0.015606651, 0.009907065, -0.008677083, 0.021064274, -0.031562805, 0.0611899, 0.011748676, 0.01088164, -0.006983338, -0.03226181, 0.0033958917, -0.07258908, 0.016251888, -0.015364688, 0.0041839536, -0.010579186, 0.031939194, 0.0046342746, 0.004990499, -0.034896526, 0.04707536, -0.03376736, -0.060383357, 0.045435384, -0.025338965, 0.010720331, 0.071621224, 0.0021944754, -0.032718852, 0.04263936, -0.050785467, -0.034466367, -0.022905886, 0.011284913, -0.035353567, 0.007816768, -0.027032709, 0.04608062, 0.036160115, 0.03591815, 0.0027674586, 0.0022465647, -0.0264278, -0.031562805, -0.0019323481, -0.023712432, -0.02809466, -0.0033992522, -0.041725274, -0.029196938, 0.0036832234, -0.041268233, 0.013455864, 0.0025238146, -0.05441492, -0.002691845, 0.011298355, -0.023712432, -0.052210364, 0.0019592328, 0.026589109, -0.010478367, -0.0012232603, 0.03158969, -0.0038512538, 0.0063717077, 0.0014089338, -0.024666844, 0.0046712416, 0.008502332, -0.005118202, -0.046914052, 0.015391572, -0.023457026, 0.0070707137, -0.0141011, 0.018389232, -0.0012367028, -0.017502032, -0.03898302, 0.030326102, -0.016225003, 0.010021325, -0.008260368, -0.027167134, 0.034762103, 0.024989462, 0.0047149295, 0.032342467, 0.011311797, 0.026320262, 0.0025439782, 0.046215046, 0.022327863, 0.035810612, 0.01348947, 0.00810578, 0.01282407, 0.018469887, -0.016628275, -0.011466386, -0.0033958917, -0.0070303865, 0.012864397, 0.011627695, -0.019464627, 0.022556383, 0.011284913, -0.009644938, 0.038875483, 0.018859718, -0.0017307118, 0.020526577, 0.025513716, -0.026293376, 0.037450586, 0.004980417, -0.014988299, -0.015566324, -0.0020499693, 0.024760941, -0.032450005, -0.0290894, -0.0040596114, 0.009329041, 0.027933352, -0.023134407, 0.0048863203, -0.002481807, 0.009812968, -0.018953813, -0.0374237, -0.0071580894, 0.002470045, -0.020082977, 0.004224281, 0.00023860298, 0.01309964, 0.001920586, -0.03465456, -0.002159189, -0.007998241, -0.016453523, 0.0120847365, -0.0036765023, 0.03586438, -0.009322319, -0.019410856, 0.028713012, -0.016668603, -0.03814959, 0.03075626, 0.013939791, 0.008616592, 0.025849776, 0.010424598, -0.018469887, 0.013072755, 0.015539439, 0.009873458, 0.0022616873, 0.048769105, -0.023120964, -0.0018802587, -0.022852117, -0.010753937, 0.008683804, 0.0050106626, -0.0029506115, -0.017448263, -0.0400853, -0.032450005, 0.037880745, -0.018295135, 0.0075143133, -0.020445922, 0.010895083, 0.013859136, -0.029842176, -0.026911728, -0.016144348, -0.012777021, -0.03981645, -0.021131486, -0.032073617, 0.016265329, -0.001604689, 0.032718852, -0.004597308, -0.003713469, -0.07931028, 0.029627096, -0.004614111, 0.011573925, -0.0006561582, -0.0073866104, 0.021037389, -0.041268233, -0.026279934, 0.014343063, -0.010780822, -0.016601391, -0.034627676, -0.049898267, -0.009739034, -0.015176494, 0.022959655, -0.005729832, 0.0053265593, -0.030675605, -0.009355925, -0.015082397, 0.016561063, 0.007843653, -0.01871185, -0.007561362, 0.021144928, 0.043714754, -0.019007584, -0.05785618, -0.012178834, -0.015929269, -0.014020446, -0.038284015, -0.016238445, 0.0023910706, -0.0010191036, 0.035945036, 0.003036307, -0.038633518, -0.036348306, 0.017918747, 0.0065935077, 0.006556541, -0.037665665, 0.03664404, 0.0003568543, 0.0045334566, -0.014732894, -0.06834127, -0.026132068, 0.0046477173, -0.0308638, 0.019948553, 0.001749195, -0.017811209, -0.008737574, -0.007998241, -0.029331364, 0.006556541, 0.0076420167, 0.0136575, -0.024720613, -0.045274075, -0.034493253, 0.0073798895, -0.012958494, 0.043634098, -0.02914317, -0.008421677, 0.026105182, -0.0025826253, 0.011002622, -0.009893622, 0.0038445324, 0.019720033, -0.020916408, 0.02609174, -0.009772641, -0.034762103, -0.012333422, 0.011889822, 0.025110444, 0.0062608076, 0.031132648, 0.006012123, 0.01610402, -0.045220304, -0.0057533565, 0.031509034, 0.009315598, 0.0021138208, -0.016601391, -0.0047586174, 0.0034950296, -0.015512554, -0.001265268, 0.021064274, -0.02387374, 0.052344788, -0.0007095078, -0.0032950735, 0.04785502, -0.02898186, 0.04395672, -0.00688252, 0.01215867, 0.006949732, -0.03852598, -0.0014349784, 0.016184675, 0.0006578385, -0.033229664, 0.00025960675, 0.013845694, 0.017031549, -0.016829912, -0.0374237, 0.0011963755, -0.024572747, -0.018577427, 0.0049602534, -0.013523076, -0.018523656, -0.011089997, 0.038176477, -0.036052573, -0.10183978, -0.013307997, 0.05769487, -0.04785502, -0.012010803, 0.0045065717, 0.05946927, -0.001454302, -0.042827554, 0.0016088898, -0.0077361134, -0.0052929535, -0.024223244, -0.03269197, -0.006916126, -0.013704549, -0.016695488, -0.012797185, 0.0029690948, -0.008865277, -0.010572464, -0.020069536, 0.0017811208, -0.034681447, -0.031858537, 8.186224e-05, -0.020661002, -0.004375508, -0.0011165611, 0.02082231, -0.018133827, -0.0061364654, 0.01937053, -0.008898883, 0.016924009, 0.023053752, 0.016614832, 0.016332543, -0.0030127827, -0.019128565, 0.029170055, -0.032019846, 0.039440066, -0.013872579, 0.06710457, -0.05024777, 0.016345985, -0.0029287676, -0.005400493, 0.00682875, -0.024774382, 0.008246926, -0.01777088, 0.002510372, 0.0033706871, 0.018456444, 0.03696666, -0.016682046, -0.022193437, 0.02353768, -0.015929269, 0.020069536, 0.008018404, -0.0046208324, 0.0038714174, 0.001804645, -0.01132524, 0.0053198384, -0.016130906, -0.010021325, 0.0026380753, 0.025554042, -0.011620973, -0.010841313, -0.014679125, 0.010142307, 0.031401496, 0.013428979, -0.05984566, 0.041671507, 0.06452362, -0.007265629, 0.017905306, 0.05790995, 0.0036395355, 0.04280067, -0.0016735814, -0.0023457026, -0.015888942, 0.04968319, 0.04446753, -0.039923992, 0.009019865, 0.0071043195, -0.04446753, -0.013744876, 0.024156032, 0.05075858, 0.02275802, -0.0113319615, 0.024626516, 0.019693147, -0.05178021, -0.014141427, -0.006758177, -0.007951193, -0.01338193, -0.022018686, 0.0044326386, 0.014168312, 0.014020446, 0.03704731, -0.02109116, 0.016534178, -0.026777303, 0.03091757, 0.0101557495, 0.0024364388, 0.011903265, -0.009974277, 0.0023557844, 0.00783021, 0.003197616, -0.015310918, -0.02380653, 0.04479015, -0.015095839, 0.0048560747, 0.00299766, 0.017313838, 0.037074197, 0.005585326, 0.009799525, -0.025554042, -0.025097001, -0.0007086677, 0.029627096, -0.019101681, -0.03825713, 0.0172063, -0.0045166537, 0.014289294, 0.038929254, -0.0089795375, -0.05229102, -0.035945036, -0.011553762, -0.04874222, 0.029707752, -0.01104967, 0.033874903, 0.02015019, 0.0022549662, -0.007749556, -0.016964335, 0.012145228, 0.01443716, -0.0062641683, 0.020190516, 0.042451166, 0.0113319615, 0.027852697, 0.021601971, -0.008865277, 0.060383357, -0.037208624, -0.0010031407, -0.0034261371, -0.0024935692, -0.014222082, -0.029707752, -0.014625355, 0.007373168, 0.03419752, 0.02537929, 0.028121546, -0.0035958476, 0.0011451263, 0.014894203, 0.028766781, 2.9615334e-05, 0.0016366148, -0.031267073, 0.009853295, 0.034842756, -0.026212722, 0.001304755, -0.009228222, -0.017582687, 0.01093541, 0.013139967, 0.022179995, 0.023860298, 0.038176477, -0.03368671, 0.0021121404, 0.011836052, -0.010559022, 0.018604312, 0.004583866, -0.069470435, -0.00040936374, 0.034493253, -0.002404513, 0.01443716, 0.03546111, -0.012548501, -0.0054441807, 0.020902965, 0.0007809207, -0.027046151, 0.02459963, -0.04067677, 0.0063212984, -0.006250726, -0.0012215801, -0.028686127, -0.012367028, 0.0024229966, -0.012998821, -0.042558707, -0.018227924, 0.0029472508, 0.02009642, -0.0067245713, 0.011318519, 0.051081203, -0.01104967, 0.01010198, 0.0022230404, -0.027745157, -0.021696068, -0.025473388, -0.006250726, -0.017461706, 0.037101083, -0.018967256, 0.010874919, 0.020849196, 0.005739914, 0.04290821, 0.01582173, -0.014087657, -0.0077629983, -0.03570307, -0.035891265, 0.007023665, 0.021118045, -0.012669482, 0.012165391, -0.0030077419, 0.04202101, 0.06145875, -0.040058415, -0.0059079444, -0.016561063, 0.017179415, 0.030218564, 0.022166552, 0.029842176, 0.01309964, 0.019330202, -0.011513434, -0.017179415, -0.008488889, -0.017663341, -0.012447682, -0.0045267353, 0.024855036, 0.004778781, -0.051699553, 0.023013426, 0.020997062, 0.025513716, 0.009618052, -0.012306537, -0.0035051114, -0.025890104, 0.0024683645, -0.0074941497, -0.014988299, 0.01959905, -0.05463, -0.015969597, 0.0029892584, -0.011163931, -0.0031337645, -0.0136575, -0.018832833, -0.006089417, 0.003458063, 0.01399356, 0.024008164, 0.0065363776, 0.0062171197, 0.0022381633, 0.023080638, -0.0013677664, 0.017192857, -0.015257148, 0.0005213139, 0.022462286, 0.018832833, -0.018684966, 0.016077137, 0.027906466, 0.004402393, -0.0152840335, 0.0070438287, -0.0067312927, -0.036482733, 0.0032463449, -0.032154273, 0.023228504, 0.011190816, -0.0014467406, -0.02431734, -0.05301691, -0.027059594, 0.0011174013, 0.013496191, 0.007904144, -0.0012787103, 0.027126806, 0.018214481, 0.0086300345, -0.0054609836, 0.0035723236, -0.015633537, -0.017058432, -0.029465787, -6.0228347e-05, 0.024881922, -0.0051114806, -0.032315582, 0.020781983, -0.021480989, -0.010848034, -0.005639096, -0.039332524, 0.031885423, -0.0007981438, 0.01332144, -0.024801267, 0.054361153, 0.009974277, 8.706068e-05, 0.0033555645, 0.020580348, 0.0066640805, 0.07355693, -0.024169473, -0.02121214, -0.0024582827, -0.018779064, 0.050194003, -0.0042814114, -0.001865136, -0.032234926, 0.02914317, -0.010841313, 0.026024528, 0.019639378, -0.009134125, -0.003069913, 0.0019474708, -0.004691405, 0.030460527, 0.031939194, 0.021601971, 0.013664221, 0.053258874, -0.0114395, 0.0072589074, -0.0012476248, 0.015835172, 0.0028296297, 0.010552301, 0.025957316, 0.0063448227, -0.010774101, -0.016278772, -0.010680004, -0.0044830474, -0.041563965, 0.015270591, -0.012918167, 0.01715253, -0.013724713, 0.0043553445, 0.01215867, 0.030621836, 0.0024011526, -0.00047384534, -0.003169051, 0.0032144191, 0.023376372, -0.03325655, -0.009060192, 0.0048157475, 0.00047552565, -0.0014064134, -0.027315, -0.02675042, 0.012736694, -0.014974858, 0.0021356647, -0.017273512, -0.0015198338, -0.012622434, 0.00460739, 0.030164793, -0.022408517, -0.013549961, 0.021413777, -0.026279934, -0.008582986, 0.003992399, -0.0025338964, 0.010115422, -0.029304478, -0.012346864, 0.0071715317, -0.015095839, 0.029869061, 0.010921967, -0.0063582654, 0.006277611, 0.023067195, -0.0037672387, -0.015915828, -0.008152829, -0.033579167, 0.004304935, -0.029680867, -0.008959373, 0.034493253, -0.009577725, -0.018698407, 0.036213882, -0.025433062, 0.010928689, 0.02614551, 0.01665516, 0.002154148, -0.021561643, -0.02387374, 0.012125064, 0.0010577505, 0.001038427, 0.011708349, -0.00394199, -0.02548683, 0.012360306, -0.0020163632, -0.011190816, -0.054038536, 0.013052591, 0.032718852, 0.013886021, 0.044064257, -0.011506713, 0.03142838, 0.012629155, -0.007863817, 0.027933352, 0.004079775, 0.03309524, -0.035810612, -0.011264749, -0.015122724, -0.011594089, -0.0015525997, -0.010747216, 0.022717692, 0.01798596, 0.021171814, 0.008838392, -0.029170055, -0.0016601391, 0.0057533565, 0.009577725, 0.02531208, 0.01743482, -0.011412616, 0.011352125, -0.00082460855, -0.003508472, 0.0060826954, -0.0030077419, 0.026790746, 0.015122724, -0.03070249, -0.017824652, -0.004180593, -0.030084139, 0.035514876, 0.0047149295, 0.023672104, -0.006506132, 0.0071312045, -0.015929269, 0.018980699, 0.002431398, 0.0012635876, -0.007346283, -0.033337206, -0.0036294537, 0.005084596, 0.014665682, 0.040649883, -0.0071043195, -0.013314718, 0.0032715492, 0.020056093, -0.009349204, -0.029008746, 0.012299815, 0.007332841, -0.0005406374, -0.010263288, -0.004345263, -0.008945932, 0.013859136, -0.020701328, -0.0172063, 0.01149327, 0.0335254, -0.005756717, 0.029788407, -0.027395654, -0.033041473, -0.0011409256, -0.017192857, -0.031885423, -0.0020970178, 0.0056659807, -0.0036899447, 0.0023053752, 0.003898302, -0.025997642, -0.049817614, 0.00084981305, 0.0047149295, 0.019088238, 0.035004064, -0.029304478, 0.029869061, -0.003780681, 0.011473106, -0.011580646, -0.010028047, 0.037396815, -0.0026968857, 0.0011686505, 0.031509034, 0.0055819657, -0.02992283, 0.0024246767, 0.028444164, 0.026118625, 0.0014022125, 0.0054441807, 0.019612493, 0.023013426, -0.01404733, 0.015620094, 0.007594968, 0.015189936, -0.001709708, 0.0035017508, -0.009685265, 0.0132609485, -0.022059014, -0.021252468, -0.002431398, 0.012346864, 0.010814428, 0.01566042, -0.0003980217, 0.013247507, -0.045811772, -0.009120683, 0.029223824, 0.010834591, 0.023376372, -0.03747747, -0.019921668, 0.0050812354, 0.027556963, -0.01848333, 0.01337521, -0.014356506, 0.022045571, -0.014598469, 0.033041473, 0.024787825, -0.016372869, 0.0027388933, -0.013462585, 0.029573327, 0.0028464326, -0.023416698, -0.018953813, -0.015754517, -0.05358149, 0.00538369, 0.0048762383, 0.00793775, -0.003169051, -0.051699553, -0.00199788, -0.0018348906, 0.0037874023, 0.009134125, 0.02903563, 0.002453242, -0.0033001145, 0.0003453022, -0.001954192, 0.00015080716, 0.013926349, -0.026723534, -0.0152840335, 0.025070116, 0.01348947, 0.018671524, -0.0019609132, -0.02898186, -0.013254227, 0.0015542799, 0.015418457, -0.020499693, -0.008892162, 0.012608991, 0.0045603416, 0.001571083, 0.021749837, -0.026253048, -0.015862057, -0.009665101, 0.03637519, 0.017838093, -0.008697246, -0.016803026, 0.017300397, -0.010263288, -0.021938032, -0.01970659, -0.002520454, 0.010263288, -0.0120309675, 0.043392137, -0.029223824, 0.029842176, 0.026857957, -0.026172394, 0.050489735, -0.022005243, -0.0022549662, 0.010028047, -0.0037605173, -0.027718272, 0.0038680567, 0.019410856, -0.00019880082, -0.018241366, -0.0014215361, -0.017945632, -0.0021978358, -0.015956154, 0.00074983505, -0.006039008, -0.027772043, -0.022959655, 0.017663341, -0.0020785343, 0.037719436, 0.0025490192, 0.0042780507, 0.022999983, 0.036482733, 0.035837494, 0.008421677, -0.00015899864, -0.011291634, -0.03242312, 0.00019859077, 0.0015114322, -0.011762119, -0.0005179533, 0.010774101, 0.018348904, 0.002349063, 0.013334882, 0.04331148, -0.024734056, 0.00019050432, -0.016023366, 0.008240204, 0.016466966, 0.026132068, 0.01748859, -0.012266209, -0.00027178894, 0.013502913, 0.042209204, -0.007151368, 0.010061652, -0.033848017, -0.00460739, -0.027099922, 0.0046040295, 0.028175315, 0.0046712416, 0.023403255, -0.02504323, -0.037343048, -0.019800687, 0.0045939474, -0.005323199, 0.0066136713, 0.037907626, -0.005578605, -0.012010803, 0.025634697, -0.0054643443, -0.024828153, 0.019464627, -0.02320162, 0.03613323, -0.0065094926, -0.018241366, -0.004946811, 0.021064274, -0.008320859, -0.034788985, 0.0047653383, -0.0044763265], "content_hash": "c7aa929eff6fa5ed76ea4f783e051e52"}, {"content": "# Key Terminology\n\nAn extended glossary of terms you may encounter while using the Transformer Explanation Dashboard.\n\n## Core Concepts\n\n**Token**: A small piece of text that the model processes. Can be a word, part of a word, or a punctuation mark. The model's fundamental unit of input and output.\n\n**Embedding**: A vector (list of numbers) that represents a token's meaning. Similar tokens have similar embeddings.\n\n**Attention**: The mechanism that lets each token look at all other tokens to gather relevant context. Uses Queries, Keys, and Values.\n\n**Attention Head**: One instance of the attention mechanism. Each layer has multiple heads that look for different patterns simultaneously.\n\n**Layer**: One complete processing step in the Transformer, containing both attention and MLP components. GPT-2 has 12 layers; larger models have more.\n\n**MLP / Feed-Forward Network (FFN)**: The component in each layer that processes tokens individually, storing and retrieving factual knowledge. Uses an expand-then-compress pattern.\n\n## Architecture Terms\n\n**Residual Stream**: The \"conveyor belt\" of information running through all layers. Each layer reads from it and adds back its contribution. This preserves information from earlier layers.\n\n**Layer Normalization (LayerNorm)**: A technique applied before or after each sublayer that stabilizes the numbers, keeping them in a reasonable range. This helps training and makes the model more robust.\n\n**Parameters / Weights**: The learnable numbers in the model. These are adjusted during training to improve predictions. GPT-2 has ~124 million parameters.\n\n**Hidden Dimension**: The size of the internal vector representations. For GPT-2, this is 768 -- meaning each token is represented by 768 numbers at each layer.\n\n**Vocabulary**: The complete set of tokens the model knows. GPT-2 has a vocabulary of about 50,257 tokens.\n\n## Training and Inference", "source_file": "key_terminology.md", "chunk_index": 0, "embedding": [0.0044527943, 0.03728695, 0.034718715, -0.01653895, 0.05407559, -0.009054213, 0.018179767, 0.047916584, -0.0072528827, 0.060638852, 0.010308605, -0.014755455, -0.059449855, -0.0251354, -0.0028075194, -0.030961486, -0.0058974256, 0.039664943, 0.015801772, 0.0663936, -0.006783229, -0.0250165, 0.003679946, 0.06415828, 0.011539217, -0.005855811, 0.011604612, 0.043445956, 0.016027682, -0.0047976035, 0.023292454, -0.017513927, -0.056168225, -0.009357408, -0.00035725647, -0.030200528, -0.018584026, 0.056215785, -0.012912509, 0.03381508, -0.0064265295, -0.019440103, -0.0425661, 0.07657141, -0.013887486, 0.0136140175, -0.021806207, -0.015445073, -0.056929182, 0.059640095, -0.023470804, 0.006563264, -0.028131671, 0.011604612, 0.016051462, 0.015730431, -0.036858913, -0.010486955, -0.020070272, -0.03341082, 0.015373733, -0.011390593, 0.03234072, 0.014541435, -9.902117e-05, -0.02853593, -0.038642406, 0.056168225, -0.03819059, 0.017192898, 0.04156734, 0.020462641, -0.055882864, 0.017359357, 0.01598012, 0.0004607364, -0.027798753, 0.04261366, -0.02863105, -0.033529717, 0.0012387864, -0.04154356, 0.0241842, -0.039498486, -0.0013049244, -0.013602127, -0.031698663, -0.01690754, -0.032483403, -0.058926698, -0.014696005, -0.023768052, -0.0247787, 0.01665785, 0.01689565, 0.018988283, -0.01062369, -0.035788815, -0.027251814, 0.0125677, 0.05402803, -0.0056863786, 0.0075322967, -0.025753679, 0.022115346, -0.058260858, -0.014303636, -0.03709671, -0.043255717, -0.034932736, -0.10700974, -0.012098046, 0.004764906, 0.018358115, 0.053029273, -0.011497602, -0.039569825, 0.011069563, -0.015373733, 0.007330167, -0.005579369, -0.00815652, 0.01253203, 0.008715349, -0.025848798, -0.0069318535, -0.016051462, 0.033648618, 0.0013160713, -0.0066940538, 0.0109744435, 0.0028372444, 0.024136642, -0.036430873, -0.0076690316, -0.02033185, -0.05859378, 0.021770537, -0.04147222, -0.038880207, 0.0010864461, -0.052315872, 0.023399463, -0.009755721, -0.038785085, 0.022531496, -0.017204788, 0.07376538, -0.03750097, 0.0047678784, 0.004577639, -0.009280123, -0.017823067, -0.031127945, -0.016943209, -0.043113034, 0.02085501, 0.09240886, 0.056834064, -0.02901153, -0.0032310998, 0.039284464, 0.015147824, 0.04116308, -0.019451993, 0.01242502, -0.019963263, -0.0016304123, -0.027299374, -0.100351356, -0.008435935, -0.024065303, -0.013197868, -0.012650929, 0.0039712503, -0.010124311, -0.050793957, -0.031555984, -0.021544628, -0.030604785, -0.044944093, -0.0069734682, -0.04477763, 0.023744272, -0.034718715, -0.018405676, -0.001471384, 0.01702644, -0.0018236244, 0.04190026, -0.014505765, 0.046418447, 0.06301685, -0.01246069, -0.00834676, -0.03360106, 0.03305412, -0.051459797, 0.026467076, 0.022650395, 0.029891388, 0.00621251, 0.04855864, 0.002279902, 0.020367522, 0.0006480034, -0.027394494, 0.0045092716, -0.04499165, -0.0497952, 0.017704166, -0.035646133, 0.00010756708, -0.03196024, -0.075810455, 0.021199819, -0.06168517, 0.02079556, 0.031936463, 0.0027406383, 0.0013316768, 0.023898842, 0.013102748, 0.01632493, -0.04187648, 0.014577105, 0.01665785, -0.040925283, 0.0071934327, -0.0021550574, 0.01613469, 0.008013841, 0.0033589168, -0.030866366, -0.03360106, -0.060258374, 0.017834958, -0.017502038, 0.005609094, -0.026776215, -0.01640816, 0.038452167, -0.023423243, 0.04532457, 0.008055456, 0.008388375, -0.007455012, -0.044397153, 0.04984276, 0.04085394, 0.06919964, -0.01262715, -0.035598572, 0.030366987, -0.029582249, -0.0064205844, -0.005918233, 0.031223064, -0.012781719, 0.010772314, 0.011884026, 0.0059866006, -0.035812594, -0.03645465, -0.017644718, -0.019380653, -0.030723685, 0.0020421026, -0.027180474, -0.055787746, 0.02427932, 0.007823601, 0.0017626883, 0.020343741, -0.035788815, -0.008186245, -0.0055407267, 0.015254833, -3.0026822e-05, 0.0015456963, 0.013935046, -0.019820582, 0.06739236, 0.025967697, -0.031651102, 0.0085310545, 0.008007896, 0.015445073, 0.021330608, -0.042994138, 0.02014161, 0.013839927, 0.04879644, 0.0011191436, 0.0064503094, -0.00086722476, 0.025610998, 0.010546405, 0.031009044, -0.02095013, -0.0031954297, -0.026871335, -0.031009044, 0.011129013, 0.0053772395, 0.027346933, -0.0071577625, 0.015124043, -0.015135934, -0.022388816, 0.016241701, -0.059735216, -0.0071815425, 0.011913751, -0.011134958, 0.039617386, 0.029629808, 0.00074572413, 0.01262715, -0.004226885, -0.024326881, -0.05897426, -0.006289795, -0.014696005, 0.048939124, 0.012983848, -0.044516053, 0.0027361796, -0.0019559003, -0.018346226, 0.07814089, -0.023351904, -0.01267471, -0.0028090058, -0.08475172, -0.0123774605, -0.034385797, 0.03215048, -0.03766743, -0.0076155267, -0.031199284, -0.039213125, -0.04606175, -0.011432207, -0.05902182, 0.03609795, 0.014184736, 0.010843654, 0.060686413, 0.0060519953, -0.026514636, -0.0016274399, -0.0051661925, -0.03721561, -0.008840194, 0.004286335, 0.031555984, 0.0012618232, -0.00032660266, 0.005029458, -0.043850213, 0.006206565, 0.023149773, 0.019142853, -0.039831404, -0.019059625, 0.019190414, -0.00066249433, 0.01032644, -0.023910733, 0.019582782, 0.044872753, 0.0047143735, -0.003528349, -0.04575261, 0.004987843, -0.012044541, 0.0059984904, -0.01653895, 0.013459448, 0.05031836, 0.009720052, -0.008025731, 0.027489614, 0.021485178, 0.025753679, -0.011491657, 0.005362377, -0.012127771, 0.05754746, 0.009190948, -0.017347468, -0.02042697, 0.012948179, 0.02103336, 0.034314457, 0.0102551, 0.038689967, 0.059497416, 0.03317302, 0.007948446, -0.007151818, -0.018084647, -0.0061352253, 0.055407267, 0.010439395, 0.043184377, 0.012829279, 0.01678864, 0.034647375, 0.055549946, -0.007092368, 0.004348757, -0.0101897055, 0.003917746, 0.0051156604, 0.035384554, 0.02896397, 0.011747291, 0.04601419, 0.017775508, -0.006147115, -0.059973016, 0.0076630865, -0.034885176, 0.046870265, 0.04201916, -0.03253096, -0.03410044, -0.044753853, 0.0018592943, 0.02884507, 0.04615687, 0.00401881, 0.006194675, -0.040568583, 0.026229277, 0.051745154, -0.02033185, 0.002129791, 0.048629984, -0.02844081, 0.014077726, -0.029962728, -0.055169467, -0.0072885524, -0.03678757, -0.06268393, -0.003540239, -0.02872617, 0.010344275, 0.018631585, 0.004595474, 0.0149575835, -0.013233538, -0.031698663, 0.046347108, 0.06420585, 0.05735722, -0.007728481, -6.929624e-05, -0.025087839, -0.028250571, -0.026039038, -0.036740012, -0.014053946, -0.008911533, 0.013090858, 0.014921914, -0.0075620217, 0.0044022617, -0.0070864228, -0.013114639, -0.044135574, 0.0016437885, 0.0057458286, 0.02970115, -0.006007408, 0.019773023, -0.026728656, 0.0034153943, -0.014149066, -0.009292013, -0.006699999, 0.007098313, 0.10177816, -0.010617744, -0.026966454, 0.031627323, -0.025396978, -0.024564682, -0.027537173, 0.0036888637, -0.011604612, 0.017597158, -0.0048956955, -0.061970532, -0.06677408, 0.028345691, -0.0060876654, 0.018821824, 0.0033589168, -0.00010886755, 0.07990061, -0.0018013307, 0.020319961, 0.043017916, 0.047536105, -0.027204255, 0.02494516, -0.034647375, 0.01250825, 0.00521078, 0.018358115, -0.013447558, 0.047726344, 0.04211428, -0.0025325639, 0.0164795, 0.0059271506, 0.016182251, -0.009262288, -0.02106903, -0.0015092832, 0.009999466, 0.014303636, -0.034266897, -0.048392184, -0.04572883, -0.011010113, 0.0027792808, -0.03657355, -0.0028387306, -0.007865216, -0.026086597, 0.01684809, 0.0062422347, -0.009969741, -0.04508677, -0.015813662, 0.012650929, 0.027418274, 0.04168624, -0.0035788813, 0.02896397, 0.013055189, -0.00067847146, -0.02951091, -0.014660334, 0.01661029, 0.0010871893, -0.021639748, 0.027180474, -0.042661216, -0.030676126, -0.0057963612, 0.00630763, -0.0012075752, 0.0044349595, -0.016943209, 0.0065573193, -0.0013926129, -0.005736911, -0.023518363, -0.042637438, 0.03640709, -0.004524134, 0.0016958072, 0.019642232, -0.040235665, -0.0040931227, -0.025539659, -0.0065335394, 0.03276876, 0.017252348, -0.019737352, 0.019035844, 0.024540901, 0.0007646738, 0.013578347, -0.034029096, -0.0038731582, -0.034219336, -0.020105941, -0.024053412, 0.020308072, -0.0011444098, 0.0073717823, -0.04511055, -0.00502054, 0.047892805, 0.005546672, 0.0015561, 0.019297423, -0.034552258, 0.0058290586, 0.04128198, -0.01674108, -0.00810896, 0.030795027, -2.278137e-05, -0.004776796, -0.036549773, 0.004348757, -0.020070272, 0.003516459, -0.013982606, 0.021556517, 0.0065038144, 0.019071514, -0.009155278, -0.02078367, -0.017121559, 0.0022873331, -0.05778526, 0.03448092, -0.03231694, 0.019309314, 0.023934511, -0.01255581, 0.021604078, -0.014755455, -0.018893164, -0.034813836, 0.0060638855, 0.0334346, -0.035503455, -0.028107893, -0.009416858, 0.007829546, -0.05483655, 0.019285534, 0.064729005, 0.01678864, -0.040069204, 0.02877373, 0.0038166807, -0.0052940096, 0.022472046, -0.022483936, -0.009743831, -0.029320668, -0.019428214, 0.0037304785, 0.0077641513, 0.021948887, -0.02511162, 0.042399637, -0.0040693427, -0.012650929, -0.014743565, -0.017395029, 0.04194782, 0.0021030386, 0.0050889077, -0.0026261974, -0.009416858, 0.000588182, -0.008400265, 0.02506406, -0.040069204, -0.0143987555, -0.003917746, -0.03795279, -0.018453235, 0.039379586, 0.009250398, 0.014684115, 0.0052969824, -0.019701682, -0.072148345, 0.03314924, -0.020748, 0.038285706, 0.013007629, -0.02439822, -0.0009415371, -0.04163868, 0.0014446315, 0.03664489, -0.014684115, 0.0337913, -0.03735829, -0.009280123, -0.008269475, -0.0045687216, 0.023256784, -0.022519605, 0.04941472, 0.019570893, -0.017573377, -0.010284825, -0.0029323641, -0.05807062, -0.054931667, 0.019214194, 0.015219163, 0.050841518, -0.014660334, -0.09155278, -0.010706919, 0.01231801, -0.056691382, 0.0048719156, -0.03381508, -0.01672919, -0.018584026, 0.040259443, 0.0076571414, -0.064253405, -0.0495574, 0.029320668, 8.8849665e-05, 0.022174796, -0.0034064767, 0.008834248, -0.013292988, 0.014969474, -0.006289795, 0.01250825, -0.030200528, 0.013911267, -0.021211708, 0.02064099, 0.018524576, 0.0028357583, -0.02130683, -0.014826794, -0.018893164, 0.017395029, 0.0034510642, 0.025420759, -0.025610998, -0.039403364, -0.0008040593, 0.0018147068, -0.0132216485, 0.0009214728, -0.03754853, -0.010748534, -0.02427932, -0.008834248, -0.017716058, 0.010391835, 0.022020226, 0.023007095, 0.00084716047, -0.015528303, -0.011687842, -0.04917692, 0.004021783, 0.0077879312, -0.013816147, -0.021889437, 0.015278613, 0.00826353, 0.027299374, -0.006705944, 0.027465833, 0.0030378876, -0.0036264414, -0.005368322, -0.027941432, 0.020212952, 0.002351242, -0.051126875, -0.0026247113, 0.012746049, 0.02427932, 0.027656073, 0.002762932, -0.002496894, 0.0023215169, 0.0037364236, 0.02002271, -0.052363433, 0.011455988, -0.007728481, 0.0064205844, -0.008644009, -0.024243651, -0.004764906, 0.0068307887, -0.031341966, -0.019154744, -0.0040842053, -0.019190414, -0.009226617, -0.028464591, -0.01036211, -0.027370714, 0.013078968, 0.023304343, -0.03769121, -0.006361135, -0.014220405, 0.022258027, -0.07381294, -0.009077993, 0.023137884, 0.0047797686, 0.021080919, -0.03386264, 0.004595474, -9.1218375e-05, 0.03262608, -0.07485926, 0.00083081174, -0.005017568, 0.01663407, 0.007026973, 0.023102215, -0.024148531, 0.01662218, 0.026823776, 0.030200528, -0.0072826073, -0.022555275, 0.0023423242, 0.026799995, 0.008542945, -0.02437444, -0.009054213, 0.010980388, -0.014410646, 0.0054069646, 0.0009876108, -0.010825819, 0.03274498, -0.0076095816, 0.025563437, -0.01218722, 0.011325197, -4.6398756e-05, -0.014446315, 0.010790149, -0.026039038, 0.0328401, -0.035812594, 0.023233004, -0.010171871, 0.042899016, 0.0007691325, 0.006991303, -0.0031924574, 0.011479767, 0.034362018, -0.006729724, 0.012109936, -0.011194408, -0.0021193875, 0.0115927225, 0.02886885, 0.0029085842, -0.009648711, 0.021211708, 0.02929689, -0.0021312775, 0.018548355, 0.00803762, 0.0095417015, -0.0073004426, -0.0046222266, 0.055454828, 0.007728481, -0.0137210265, -0.000118713935, 7.914262e-05, -0.029891388, -0.007900886, -0.012757939, -0.0044230693, 0.019226084, -0.011670006, 0.007508517, -0.0055407267, -0.030557226, 0.034623597, 0.003950443, -0.031484645, 0.030105408, 0.025373198, 0.015433183, -0.02874995, -0.02473114, 0.0017582295, 0.025420759, 0.05440851, 0.011717566, -0.01690754, 0.001507797, 0.0045449417, 0.014850575, 0.02437444, 0.03362484, 0.061066892, -0.024422001, 0.06082909, 0.025825018, -0.043231934, -0.040806383, -0.02910665, 0.01218722, -0.018964505, -0.01267471, -0.02905909, 0.008804523, 0.0057874434, 0.009559537, 0.036193073, 0.016003901, -0.01603957, 0.007847381, 0.038903985, 0.047250748, -0.008043566, -0.016087132, -0.0011213729, 0.00512755, -0.014303636, 0.01690754, -0.02903531, -0.017763617, 0.023934511, 0.028274352, 0.014826794, -0.015801772, -0.019190414, 0.026371956, -0.026799995, -0.01619414, -0.008471604, 0.034742497, -0.00096606015, 0.021615967, -0.022269916, 0.014755455, -0.031389523, -0.015623422, -0.0057696085, -0.009161223, -0.026966454, 0.017537707, -0.0018176794, -0.021936998, 0.012014816, -0.0039801677, 0.013709137, 0.015992012, -0.030866366, 0.035313215, 0.011527327, 0.019202303, 0.0026470048, -0.047631226, -0.032364503, 0.039926525, 0.0015211732, 0.042756338, 0.030176748, -0.005778526, 0.024422001, -0.0042803898, -0.013399998, 0.009161223, -0.007021028, -0.003021539, 0.0012276395, 0.011895916, 0.018655365, 0.03664489, 0.035360776, 0.02124738, -0.036977813, 0.034861397, 0.03381508, 0.0032102922, 0.008965039, 0.03269742, 0.025087839, -0.012757939, -0.02520674, -0.00828731, -4.205613e-05, -0.014220405, -0.013923156, 0.026966454, 0.01038589, 0.023209224, 0.0059806556, 0.0065335394, -0.0136140175, -0.031936463, 0.033719957, 0.00033682058, -0.0057963612, -0.008566724, -0.01224667, 0.015028924, -0.008126795, -0.028274352, -0.008602394, 0.022460155, -0.01255581, 0.012888729, 0.028226793, 0.008632119, 0.0055110017, -0.0055585615, -0.0061114454, 0.017549599, -0.02936823, -0.0064740893, 0.01674108, -0.013292988, -0.0028550792, 0.019808693, -0.05069884, -0.022115346, 0.00077582063, -0.019321203, 0.01603957, 0.026776215, -0.007930611, 0.011854301, -0.0012432451, -0.0038434332, -0.011996981, 0.0052405046, -0.0045806114, 0.010784204, 0.028250571, 0.030057847, 0.00821597, 0.009339573, 0.06458633, 0.006991303, 0.008941258, 0.021485178, -0.0029026393, -0.005368322, -0.004286335, 0.0040633976, 0.009684382, 0.052078076, -0.00078845373, -0.035955273, 0.007906831, 0.014470095, 0.015801772, -0.03766743, 0.018762374, 0.019475773, 0.050033, 0.023387574, -0.0011979146, -0.00847755, 0.03735829, 0.035360776, 0.048011705, -0.020224841, 0.015706653, -0.0064384197, -0.0075620217, 0.0019529278, 0.027798753, 0.018262995, -0.03659733, 0.05274391, -0.026538417, -0.003552129, -0.0036829186, -0.027465833, 0.00032771734, -0.014612774, 0.0012216945, -0.027703634, -0.02877373, -0.029558469, 0.017430698, -0.0037304785, 0.0039118007, 0.011693787, -0.018488906, 0.010736644, -0.010183761, -0.03417178, -0.015754212, -0.005644764, 0.016669739, -0.0064859795, 0.011360868, 0.009642767, -0.009101773, -0.005133495, -0.008061401, -0.028702391, -0.0019900838, -0.0147911245, -0.005445607, -0.01622981, 0.016669739, 0.014065836, 0.017181009, -0.047987923, -0.000648375, 0.009690327, 0.0007832519, -0.00839432, -0.014327415, 0.039664943, 0.029938947, -0.015147824, 0.0056299013, -0.042899016, 0.03398154, -0.018429456, 0.020189172, 0.0251354, -0.010837709, -0.0025563438, 0.017763617, -0.03405288, 0.03800035, -0.009714107, 0.004155545, 0.023958292, -0.0034243118, 0.017620938, 0.007799821, -0.006539484, -0.04563371, -0.015932562, -0.014386865, -0.021199819, 0.017288018, -0.024172312, -0.018227326, 0.004964063, 0.0010619231, -0.023328124, 0.030913925, -0.01651517, 0.0011250885, -0.00307653, 0.008911533, -0.006043078, 0.052839033, -0.005942013, -0.0069496883, 0.0002641803, -0.030890146, 0.02473114, -0.030723685, -0.0033381095, -0.033529717, -0.02070044, 0.01705022, 0.028464591, -0.017989527, -0.020224841, 0.007954391, 0.019202303, -0.027323155, 0.009862731, 0.019249864, 0.020165391, 0.014648445, 0.00021123278, -0.014410646, -0.004940283, 0.000491576, 0.009714107, 0.006634604, 0.0008367567, 0.035717472, -0.0048273285, -0.0125677, -0.03764365, 0.0057190764, 0.036169294, -0.0071577625, 0.021378169, -0.015885001, 0.004747071, -0.03750097, 0.0059509305, 0.027846312, -0.026419517, -0.0031032825, -0.0004354702, -0.030890146, 0.0076274164, 0.012912509, -0.011699731, -0.013269208, 0.009696271, -0.0050651277, -0.0072944975, -0.014410646, -0.005888508, 0.045134332, -0.05379023, 0.014196626, 0.034742497, -0.0050681, 0.012841169, -0.026847554, 0.03348216, -0.0020970937, 0.009838951, -0.01051668, -0.036502212, -0.0143987555, 0.02083123, -0.0063670794, -0.008715349, -0.002991814, -0.013233538, -2.027333e-05, -0.01690754, 0.016396271, 0.025848798, -0.026633536, 0.010379945, -0.0018890192, 0.02070044, -0.022341257, -0.009838951, -0.017834958, 0.018643474, 0.0021134424, -0.02470736, 0.039831404, -0.03250718, -0.014767344, 0.025920138, -0.0050621554, 0.009363352, 0.0016854034, 0.044088013, -0.003269742, -0.011224133, -0.018203545, -0.019428214, 0.023910733, 0.0032489346, -0.005442634, 0.0040574525, -0.022590946, 0.0007490682, -0.0045627765, -0.010171871, -0.017775508, -0.0059063435, -0.0017537707, 0.014600885, -0.017371248, -0.0034718716, 0.0067891735, -0.0036056337, 0.030295648, 0.03286388, 0.016241701, 0.060068134, -0.019570893, -0.02014161, 0.026443297, -0.02886885, 0.01674108, -0.010766369, 0.0032132647, -0.014529545, 0.02078367, -0.0023423242, -0.027560953, -0.026015257, 0.0021996447, 0.01628926, -0.0018414593, -0.028036552, 0.0024998665, 0.020153502, 0.004342812, 0.021520847, -0.021592189, 0.01631304, -0.00010366569, 0.010540459, 0.00035075416, 0.03329192, 0.060305934, 0.02029618, -0.0005090394, -0.0035937438, -0.029891388, -0.01235368, -0.010813929, -0.02970115, -0.011378703, 0.018179767, 0.018298665, -0.06306441, 0.0034837616, 0.003153815, 0.0030913926, -0.014303636, 0.018536465, 0.028345691, -0.029867608, 0.0047738235, 0.013578347, -0.010463174, -0.017216679, 0.014374975, 0.032959, 0.0019365791, -0.008756964, -0.0022754432, -0.009416858, 0.02544454, -0.003668056, -0.0095417015, 0.007859271, -0.01238935, -0.026443297, 0.0013257317, 0.013150308, 0.012829279, -0.018560246, -0.0097973365, -0.020058382, 0.026942676, -0.01675297, -0.0049046134, 0.007621472, -0.015266723, 0.00025990733, -0.048273284, -0.010998224, 0.0018236244, 0.021378169, 0.04168624, 0.0036710287, 0.038903985, 0.03381508, -0.0035788813, 0.0016943209, 0.0009980145, -0.0065038144, -0.010308605, -0.038737528, 0.038737528, -0.017644718, 0.003798846, 0.0006182785, -0.0020004876, -0.015385623, -0.027109135, -0.009274177, -0.020094052, 0.013792367, 0.013625907, 0.00523456, -0.019570893, -0.0035967163, -0.010885268, -0.021294938, -0.0076274164, 0.042946577, -0.036502212, -0.0008003437, -0.009743831, -0.025872577, -0.02482626, 0.009482252, 0.0046341163, 0.009613042, -0.0409015, 0.013257318, 0.00815652, -0.015944451, -0.014089616, -0.012888729, 0.0017062109, -0.005192945, 0.01619414, -0.008031676, 0.026276836, -0.0035075415, 0.035574794, -0.010813929, 0.022900084, 0.01656273, 0.007151818, -0.060686413, 0.0064086947, 0.033957757, 0.009892456, 0.00831109, -0.027965212, -0.02045075, -0.025729898, -0.0025236465, 0.00845377, 0.005029458, -0.014731674, -0.025087839, -0.0051126876, -0.004565749, -0.014065836, -0.0009868676, 0.02451712, -0.0038345158, -0.019309314, -0.02951091, 0.027156694, -0.0005993289, -0.02898775, -0.009214728, -0.0140063865, 0.017573377, 0.040877722, 0.009464418, 0.01266282, 0.01724046, -0.0022278833, -0.01661029, 0.014279855, 0.0016809447, -0.00399503, 0.02492138, 0.010118365, 0.008661844, 0.01663407, 0.009624932, -0.012008871, -0.015968231, 0.018465126, -0.011414372, -0.03238828, -0.0013651174, -0.0027376658, -0.015956342, -0.03652599, -0.0055823415, 0.047250748, -0.011027948, 0.005620984, 0.02128305, 0.0058974256, 0.015064593, 0.051697597, -0.029439569, 0.04206672, 0.0036502213, -0.007080478, -0.015670983, -0.022757405, -0.01653895, -0.009083938, -0.01643194, -0.02105714, -0.009137443, -0.008709404, 0.011390593, 0.042589877, 0.010605854, 0.013851817, 0.008328925, 0.0028075194, 0.074336104, 0.003727506, 0.009000708, 0.012056431, 0.037596088, -0.012757939, 0.019475773, 0.02439822, 0.014018276, 0.019582782, 0.008305145, -0.0014327415, 0.030676126, 0.035289437, 0.02033185, -0.015338063, -0.03312546, 0.0032073196, 0.0009222159, -0.035027854, 0.005499112, 0.010689084, -0.022198576, -0.01219911, -0.015647203, -0.0011815659, 0.010790149, 0.046727587, -0.02097391, -0.00823975, 0.0067118886, 0.0137210265, 0.012757939, -0.00808518, 0.003275687, -0.055597506, -0.013887486, -0.002262067, 0.015421293, -0.01638438, 0.0064146393, 0.03700159, 0.009280123, -0.067107, -0.02072422, 0.0136140175, 0.00067847146, 0.018500796, 0.025848798, 0.011093344, -0.009012599, -0.010463174, -0.027822534, -0.022781186, 0.0032429898, 0.027109135, 0.027679853, -0.022959534, -0.0043398393, -0.009559537, 0.020498311, -0.0137210265, 0.007437177, 0.022876306, 0.025278078], "content_hash": "d39c81a382c4f7945e6cc285198af819"}, {"content": "~124 million parameters. **Hidden Dimension**: The size of the internal vector representations. For GPT-2, this is 768 -- meaning each token is represented by 768 numbers at each layer. **Vocabulary**: The complete set of tokens the model knows. GPT-2 has a vocabulary of about 50,257 tokens. ## Training and Inference\n\n**Training**: The process of adjusting the model's parameters by showing it billions of text examples. The model learns to predict the next token and its parameters are updated to reduce prediction errors.\n\n**Inference**: Using the trained model to make predictions on new text. This is what happens when you click \"Analyze\" in the dashboard -- no learning occurs, the model just processes your input.\n\n**Forward Pass**: One complete trip of data through the model, from input tokens to output predictions. The dashboard visualizes this forward pass.\n\n**Gradient**: A measure of how much each parameter contributed to the model's prediction error. Used during training to update parameters, and in attribution experiments to measure token importance.\n\n**Loss**: A number measuring how wrong the model's predictions are. During training, the goal is to minimize this. Lower loss means better predictions.\n\n**Fine-tuning**: Taking a pre-trained model and training it further on a specific dataset to specialize its behavior.\n\n## Prediction Terms\n\n**Logits**: The raw, unnormalized scores the model assigns to every possible next token before converting to probabilities.\n\n**Softmax**: The function that converts logits into probabilities (positive numbers that sum to 1.0).\n\n**Probability Distribution**: The complete set of probabilities over all possible next tokens. The dashboard shows the top 5.\n\n**Temperature**: A setting that controls prediction confidence. Low temperature = more focused; high temperature = more spread out.", "source_file": "key_terminology.md", "chunk_index": 1, "embedding": [-0.01888086, 0.0065460494, 0.053035837, -0.013798045, 0.019882021, -0.012861062, 0.001687854, 0.040611178, 0.014709358, 0.06638464, 0.008156891, -0.009241481, -0.040226117, -0.031061646, 0.009177305, -0.019176073, -0.015582165, 0.03747934, -0.020857511, 0.064741716, -0.03052256, -0.030240182, -0.00554168, 0.038942575, 0.01478637, -0.008278827, 0.0038281553, 0.03139537, 0.05182931, -0.00076290354, 0.03614446, -0.01821342, -0.044513136, -0.052394066, 0.0147607, -0.042844534, -0.0064979168, 0.0590428, -0.005859356, 0.02690298, -0.043306608, -0.01673735, -0.08476492, 0.05421669, -0.036580864, 0.01645497, 0.02297535, -0.021653304, -0.045539968, 0.035836414, -0.032601893, 0.044179417, -0.053959984, 0.018457292, 0.014170271, 0.009908922, -0.017160919, -0.012039597, -0.008991191, -0.04692619, -0.00017768994, -0.027955482, 0.038993917, 0.044436123, 0.020703485, -0.025811972, -0.043820024, 0.070132576, -0.03968703, 0.00893985, 0.018316103, 0.00217239, -0.014349967, -0.03018884, 0.0013204409, 0.0065685115, -0.0021001908, 0.016531983, -0.019342935, -0.036375497, 0.026197033, -0.021435102, -0.015517987, -0.03968703, -0.021871507, -0.0453346, -0.00991534, -0.048235398, -0.03278159, -0.04797869, -0.03709428, -0.02345026, -0.022359252, 0.022423428, 0.008400763, -0.022064038, 0.03411647, -0.013464325, -0.020292753, -0.014426979, 0.01094217, 0.014619511, 0.016647503, -0.013066428, 0.015479482, -0.031035975, -0.06535781, -0.010621286, -0.019227415, -0.029778108, -0.0879481, 0.025439745, -0.049775645, 0.006597391, 0.0112181315, -0.043537647, -0.035425678, 0.034655556, 0.016852869, -0.02378398, -0.01835461, -0.02280849, 0.029316032, 0.010197719, -0.023013856, 0.0039789714, -0.009119545, -0.02428456, 0.0050667706, 0.01626244, 0.0051983334, 0.03486092, 0.011539017, -0.045026552, -0.0044153747, -0.026517918, 0.008952686, 0.052034676, -0.035271656, -0.026197033, -0.0081697265, 0.0045565637, 0.017802687, 0.03350037, -0.06325281, -0.00818898, -0.02211538, 0.06499842, -0.049595952, 0.030086156, -0.0004001032, 0.016172593, 0.008914179, -0.034835253, 0.007078718, -0.04043148, 0.017507473, 0.0760882, 0.015851708, -0.040508494, -0.00013176331, 0.027288042, 0.021820165, 0.017828358, -0.016429301, 0.032396525, -0.04161234, -0.0034463024, -0.05704048, -0.08430285, 0.018559976, -0.0073418436, -0.025375567, -0.0137852095, -0.0035489856, -0.025914654, -0.079682104, -0.021242572, 0.006013381, -0.0717755, -0.063458174, -0.02152495, -0.034373175, 0.022872666, -0.031600732, -0.021383762, -0.005057144, 0.009536696, 0.019676654, 0.020421106, 0.021576293, 0.026055845, 0.08959103, 0.007316173, -0.03611879, -0.028289203, 0.052984495, -0.0018033724, -0.0112694735, 0.048466437, 0.018675493, 0.024181876, 0.030368535, -0.0538573, 0.0005896258, 0.005852938, 0.008503446, -0.0033532458, -0.050212048, -0.013079263, 0.0011287122, -0.0056732427, 0.007836007, -0.007078718, -0.06874635, 0.021229737, -0.0638689, 0.029932132, 0.01690421, 0.03588775, -0.005333105, 0.02656926, -0.008914179, 0.0105314385, -0.008548371, 0.018059395, 0.029136337, -0.0015787531, 0.051957663, 0.0010781728, 0.0026328596, -0.012828973, -0.019753667, -0.027647432, 0.007945107, -0.03116433, 0.038146783, -0.013515667, 0.006395234, -0.0430499, 0.014272955, 0.033988114, -0.007251996, 0.021281078, 0.041329958, 0.010383831, -0.01854714, -0.037941415, 0.0505201, 0.016044239, 0.051418576, -0.011384992, -0.022449099, 0.012925238, 0.0036580865, -0.019843515, -0.013348807, 0.031883113, 0.00089045527, -0.016929882, 0.029701095, -0.008355839, -0.039507333, -0.023103705, -0.016660338, -0.0030179212, -0.016852869, -0.023899497, -0.03704294, -0.0879481, 0.021768823, 0.004832525, -0.015312621, 0.014195942, -0.04027746, -0.021062877, -0.0031222089, 0.015543658, -0.024990506, 0.021422267, 0.0033211575, -0.007483033, 0.06279074, 0.00156672, 0.0170454, -0.0052657193, 0.022859832, -0.0088564195, 0.0003451517, -0.029110666, 0.01971516, -0.004951252, 0.02823786, 0.010967841, 0.0071557304, 0.05775926, 0.029932132, 0.022731477, 0.0019573972, -0.009658632, 0.02526005, -0.058221333, -0.016275275, -0.013772375, 0.028417556, 0.04143264, 0.012289887, 0.042895876, 0.0033500372, -0.018316103, -0.008541953, 0.0066358973, -0.00048734376, 0.0012458351, -0.0076819817, 0.03486092, 0.030394206, -0.0029729975, 0.042844534, -0.024425749, -0.0130215045, -0.023552943, 0.013047175, 0.022628793, 0.03668355, 0.015440975, 0.00015572939, -0.006725745, 0.017648663, -0.006995288, 0.048132718, -0.002369734, -0.030060485, 0.049441926, -0.047850337, -0.0064176954, -0.023219222, 0.044282097, -0.05121321, 0.026748955, -0.04987833, -0.008721648, -0.037864402, -0.010595615, -0.013387313, 0.010691881, 0.010781729, 0.04004642, 0.06602525, 0.010865158, 0.0118278125, -0.005692496, 0.011917661, -0.0029457223, 0.03193445, 0.014401309, 0.019291593, 0.017777016, 0.0029890416, -0.015492317, -0.06525513, 0.003937256, 0.041894715, 0.007842424, -0.0137081975, -0.020318424, -0.01306001, -0.020202905, -0.00061288994, -0.02049812, -0.027775787, 0.022744313, -0.020036045, -0.023180716, -0.035579704, -0.020664979, -0.0015867752, 0.018906532, -0.029264692, 0.0019156821, 0.043666, 0.021165559, 0.009530278, 0.032293845, 0.025272885, 0.03550269, 0.023860991, -0.02311654, -0.0010356556, 0.036375497, -0.0039918064, -0.03049689, -0.02595316, 0.006462619, -0.012476, 0.016172593, -0.043563318, 0.058632065, 0.041560996, 0.031677745, -0.041201606, -0.026492247, -0.04299856, -0.021563457, 0.0057855523, 0.03314098, 0.0511362, 0.045077894, 0.058324017, 0.013990576, 0.0638689, -0.010082199, 0.0067578335, -0.023155045, 0.040482823, -0.0069182757, 0.03450153, 0.026748955, 0.038198125, 0.035040617, 0.005378029, 0.0064754547, -0.044872526, -0.013964905, -0.02988079, -0.0044538807, 0.02790414, -0.012264216, -0.015607836, -0.03966136, 0.0024162624, 0.029239021, 0.016724514, -0.014773535, 0.0025558472, -0.05188065, 0.038891234, 0.056629743, -0.02918768, 0.0032425406, 0.037582025, -0.016673174, 0.001608435, 0.0020985864, -0.041150264, -0.01873967, -0.027827127, -0.05416535, 0.009478936, -0.05647572, 0.012379735, 0.05380596, -0.041329958, 0.031857442, -0.029136337, -0.038608856, 0.028340543, 0.019638149, 0.07372648, -0.03450153, 0.014542498, 0.007970778, -0.050391745, -0.0351433, -0.0047715567, 0.0038859146, 0.01707107, 0.005021847, -0.001926913, -0.005589813, -0.007611387, -0.016236769, 0.010518603, -0.04733692, 0.017802687, 0.004450672, 0.018033724, 0.03190878, -0.0049865493, -0.014439815, 0.029983474, 0.0130985165, -0.022333581, 0.01776418, -0.0017873282, 0.06925977, -0.007585716, -0.009324912, 0.036272816, -0.026979992, -0.00902328, -0.048671804, 0.020767663, -0.013515667, 0.0030612408, 0.020254247, -0.022911172, -0.04618174, 0.007829589, 0.016057074, 0.038172454, -0.00952386, -0.017995218, 0.06427964, -0.013271795, 0.056629743, 0.027467737, 0.018264761, -0.02721103, 0.028956642, -0.07562613, -0.0032842557, -0.023758309, -0.0019750458, -0.0136311855, 0.032037135, 0.026389563, -0.0066679856, 0.007861677, 0.002286304, 0.0098190745, 0.020613639, -0.031061646, 0.007354679, 0.018636988, -0.03116433, -0.027878469, -0.05146992, -0.019882021, -0.027288042, 0.032319516, -0.036965925, 0.011282309, 0.017995218, -0.03768471, 0.02278282, -0.0015554889, -0.036401168, -0.024746634, 0.031626403, 0.009729226, 0.010987095, 0.023193551, 0.035554033, 0.006241209, 0.020575132, 0.022949679, -0.020870347, -0.028648593, 0.036632206, 0.039147943, -0.013348807, 0.026723284, -0.036888912, -0.030394206, -0.03193445, 0.005368402, -0.007585716, 0.004505222, -0.017109577, 0.014195942, 0.0069246935, 0.018264761, -0.020934522, -0.025349896, 0.027057005, 0.011404245, -0.02309087, 0.018393116, -0.028674264, -0.03509196, 0.012033179, 0.0019237043, 0.0007107598, 0.007887348, -0.035194643, 0.013964905, -0.0008872464, -0.018444456, 0.019920526, -0.015672013, -0.0019590016, 0.01155827, 0.013900729, -0.041869044, 0.019484123, 0.019381441, 0.0093890885, -0.036401168, 0.006337474, 0.041766364, -0.032447867, -0.0071172244, 0.021833, -0.04420509, 0.005759882, 0.014041917, -0.015890215, -0.018251926, 0.027288042, 0.02069065, -0.014799206, -0.029367374, -0.024900658, -0.009511025, 0.016852869, -0.013567008, 0.04556564, -0.017481802, -0.020819005, -0.022769984, -0.052984495, -0.033628725, -0.01673735, -0.035271656, -0.006719327, -0.025362732, -0.018868025, 0.04530893, -0.015890215, 0.03314098, -0.018251926, 0.0037864402, -0.025401238, 0.019959033, 0.018572811, -0.023655625, -0.017468967, 0.0020456405, -0.0060807667, -0.027416395, 0.018405952, 0.056681085, -0.004607905, -0.009414759, -0.0063823983, 0.012739126, -0.007502286, 0.012013926, -0.032319516, 0.025503922, -0.0015121695, -0.0025751004, -0.02013873, 0.018277597, 0.0023536899, -0.034912262, 0.029393045, -0.025722124, -0.017892536, -0.020228576, -0.020575132, 0.012604354, 0.01673735, 0.00389875, -0.016634667, -0.01690421, -0.011179626, -0.0076563107, -0.0055545154, -0.02954707, -0.0069631995, -0.013964905, -0.015800366, -0.006642315, 0.04166368, 0.017802687, 0.008529117, 0.030779269, -0.021062877, -0.055038158, 0.0471829, -0.0013830133, 0.008984773, -0.009010444, -0.022898337, -0.016724514, -0.075574785, -0.0069182757, 0.05254809, -0.0209987, 0.027519079, -0.0340908, -0.0015049496, -0.00829808, -0.01255943, 0.049364913, 0.0047747656, 0.036170132, 0.0031141867, 0.0021691811, -0.0010396667, -0.0018900114, -0.015607836, -0.065152444, -0.0026023756, -0.0015779509, 0.060891096, 0.016891375, -0.04376868, 0.0069631995, 0.025029013, -0.03771038, -0.029444387, -0.044179417, 0.017340614, -0.019869186, 0.021845836, 0.004412166, -0.04045715, -0.026415234, 0.026440905, 0.0010661397, 0.008394346, -0.011962584, 0.0137081975, -0.010383831, 0.0015185872, -0.014465486, -0.024810812, -0.03152372, 0.014478321, -0.029957803, 0.011866319, 0.0077204877, -0.046412773, -0.029136337, -0.024117699, -0.023437424, 0.005053935, 0.018405952, 0.015440975, -0.019779338, -0.023373246, -0.009812657, 0.0075921337, -0.00088483974, 0.0009971494, -0.011462004, -0.023347577, -0.0099217575, 0.017918207, -0.01578753, -0.0008655867, 0.019342935, 0.01183423, -0.00966505, 0.009350582, 0.0073097553, -0.0690544, -0.003520106, 0.010345325, -0.021447938, -0.022924008, 0.037350986, 0.0025414075, 0.007033794, -0.0030644496, 0.0013084076, 0.026517918, 0.014696523, 0.015312621, -0.007579298, 0.044898197, -0.015107255, -0.053035837, 0.023886662, 0.018200584, -0.024079194, 0.02380965, 0.006401651, -0.009652214, -0.0023873828, 0.026415234, 0.030111827, -0.032730248, 0.020703485, 0.008464941, 0.0060165897, 0.0074894507, -0.015800366, -0.009645796, 0.020677814, -0.015646342, -0.0035361503, -0.03473257, -0.008009284, -0.007194237, -0.040919226, -0.031600732, -0.015363963, 0.013746704, 0.0024579773, -0.02564511, 0.02576063, -0.0010966237, 0.01690421, -0.09220945, -0.009838327, 0.0035361503, -0.019394277, 0.01776418, -0.035554033, 0.04895418, -0.010897247, 0.012257799, -0.04438478, 0.010993512, -0.027493408, 0.019099062, -0.0092093935, -0.0011471631, -0.034296166, -0.0032858602, -0.000916126, 0.010852323, -0.027262371, -0.025965996, 0.011044854, -0.0030917248, -0.0014752678, -0.06356086, -0.008452105, 0.015171432, 0.0021435104, 0.023514437, 0.0024178668, -0.017982382, 0.014362803, 0.008663889, 0.016339453, -0.012777632, 0.021268243, -0.010396667, -0.027955482, 0.019086227, 0.012046015, 0.029932132, -0.037273977, 0.03745367, 0.006616644, 0.054627426, -0.02506752, -0.013682527, -0.009491771, 0.004132996, -0.008862837, -0.0020809378, 0.052933153, 0.0106084505, -0.023193551, 0.011410663, 0.02188434, 0.01083307, -0.023694132, 0.030419877, 0.03830081, -0.022757148, 0.022423428, -0.0012362086, -0.017558815, -0.023270564, 0.006116064, 0.030009145, -0.0036356244, -0.039584346, -0.013541337, -0.015992897, 0.006456202, -0.0059042796, -0.010492932, -0.014144601, 0.03085628, -0.0013412983, 0.0019445617, -0.044256426, 0.0072840843, 0.04240813, 0.0045212666, -0.038891234, 0.02219239, 0.04045715, 0.0721349, -0.016352288, -0.02414337, -0.015376799, 0.009536696, 0.059504874, -0.012103774, -0.013913563, -0.005188707, -0.028366214, 0.0032056388, 0.0071493126, 0.04887717, 0.022230897, -0.0063823983, 0.023527272, 0.0058080144, -0.057964627, -0.012418241, 0.0017536354, 0.009042533, -0.0066936566, 0.005496756, -0.011840648, -0.01921458, -0.014298625, 0.016249605, 0.010114288, 0.013258959, -0.04107325, 0.015569329, 0.0544734, 0.047439605, -0.023848157, -0.05313852, 0.0014592236, 0.0038474086, 0.011314397, 0.03573373, -0.041329958, 0.012090938, 0.0026938277, 0.0386602, 0.0021066086, -0.009883251, -0.04266484, -0.0011632073, -0.008760154, 0.008984773, -0.0019124732, 0.026825966, 0.007264831, -0.010287566, -0.033859763, 0.0099217575, -0.013759539, 0.018328939, -0.013246124, -0.008599712, -0.013515667, -0.007001706, -0.0040848632, -0.046874847, 0.015505153, 0.014581004, 0.031010306, 0.0054037, -0.025555264, 0.022141049, -0.021820165, 0.0216148, 0.017982382, -0.021011535, -0.03445019, 0.041766364, 0.010987095, 0.05226571, 0.021845836, 0.014067588, 0.030933293, -0.03275592, -0.011051272, 0.0066294796, -0.018226255, -0.0002038621, -0.008028537, 0.015235609, 0.00991534, 0.02409203, 0.027390724, 0.043306608, -0.021370927, 0.014670852, 0.013913563, 0.026312552, 0.024964835, 0.017443296, 0.007097971, -0.009549531, 0.010576362, -0.034193482, -0.015607836, -0.0009056973, -0.034681227, 0.041715022, -0.004935208, 0.01971516, 0.016467806, 0.038506173, -0.008497029, -0.033269335, 0.023065198, -0.0038056935, -0.0069567817, 0.013181946, 0.0055320533, -0.014067588, -0.017815523, 0.0015891818, -0.0070209587, 0.032268174, -0.004992967, 0.0051566185, 0.03478391, -0.0029986682, 0.013156276, 0.00091532385, 0.011487675, -0.0098960865, -0.031857442, -0.025157366, 0.0050892327, 0.03283293, -0.026466576, 0.007502286, -0.0426135, -0.001290759, 0.010345325, -0.007001706, 0.010210553, 0.0071364776, 0.017456131, 0.012848226, -0.018393116, 0.015197103, -0.012476, 0.01835461, -0.010409502, 0.0071364776, 0.037505012, 0.029957803, 0.011372156, 0.0046207407, 0.02464395, 0.006616644, 0.027467737, 0.036632206, 0.0023280191, -0.008933432, 0.0012747148, -0.017584486, 0.0049223723, 0.041560996, -0.020292753, -0.017828358, 0.013008669, 0.04523192, 0.02364279, -0.011179626, 0.008073461, 0.0137338685, 0.035554033, 0.039481662, 0.006112855, -0.012828973, -0.010223389, 0.03547702, 0.02013873, -0.015235609, 0.031215671, -0.021653304, -0.0073867673, 0.012347646, 0.041715022, 0.020677814, -0.023732638, 0.021409431, -0.029290363, 0.0070145414, 0.0064465753, -0.027057005, 0.007393185, -0.013438654, 0.012334811, 0.0138108805, -0.035631046, 0.0016557655, -0.014658016, -0.0034110053, -0.012001091, 0.012469582, 0.013682527, -0.0025526383, -0.0021691811, 0.0035682388, 0.00016565676, 0.015338292, -0.010659792, -0.00598771, 0.0021451148, 0.013387313, 0.010018023, 0.018020889, -0.020382602, -0.03157506, 0.024361571, -0.0413813, -0.0071814014, -0.01902205, 0.0063150125, 0.024117699, 0.02606868, -0.036349826, -0.026748955, 0.021717481, 0.0072134896, -0.018303268, -0.0058754003, 0.045462955, 0.031677745, -0.0069631995, 0.004787601, -0.049390584, 0.009324912, -0.0056090658, -0.0044635073, 0.0063791894, 0.009048951, -0.01286748, -0.0074958685, -0.03375708, 0.027313713, -0.01445265, -0.01099993, 0.023578614, 0.032653235, 0.016827198, 0.0034110053, -0.018957872, -0.04420509, -0.01885519, -0.0118278125, -0.022654464, -0.015864544, -0.04559131, -0.0113464855, 0.04004642, -0.020947358, -0.01904772, -0.0010260291, 0.003937256, -0.0037479342, 0.013361642, 0.022449099, -0.024464255, 0.0459507, 0.006680821, -0.022731477, -0.0016750186, -0.026517918, 0.013297465, -0.008343005, -0.037967086, -0.033834092, 0.0021001908, 0.018726835, 0.024348738, -0.030471219, 0.00459507, 0.011705876, -0.0076563107, -0.007393185, -0.020087387, 0.037273977, 0.008111967, 0.011699459, 0.024207547, -0.0004364033, -0.008721648, -0.0009626543, 0.013913563, 0.00921581, -0.032473538, 0.037864402, 0.045154907, -0.022410592, -0.020344095, -0.007258414, 0.0386602, 0.0071557304, 0.019894857, -0.004158667, 0.02102437, -0.030753598, 0.017943876, -0.005400491, -0.0067642513, -0.0033211575, -0.011731547, -0.0072455783, 0.019971868, 0.013746704, -0.03768471, -0.0021755989, -0.009235064, 0.022449099, 0.011121866, -0.028109506, -0.0023905917, 0.026697613, -0.028982313, -0.011847066, 0.020241411, -0.015338292, 0.016172593, -0.049441926, 0.019882021, -0.011442751, 0.018457292, -0.015248445, -0.058940116, -0.00977415, 0.001854714, -0.015928721, -0.011314397, -0.01873967, -0.004027104, -0.019561136, -0.021486444, 0.013913563, 0.020485284, -0.013477161, 0.018945036, -0.009684303, 0.031677745, -0.018418787, -0.016493477, -0.02069065, 0.004473134, 0.010704716, -0.02230791, 0.045668323, -0.0072006546, -0.0064465753, 0.0086318, 0.0027435648, 0.0039500915, 0.002093773, 0.027416395, -0.0007540792, -0.0002661338, 0.0015675222, 0.005670034, -0.0060358425, 0.0051726624, -0.0077525764, -0.009992352, -0.01542814, 0.0077910824, 0.0011744383, -0.012482418, -0.02857158, -0.0048774485, 0.0072455783, -0.0044314186, 0.0020039254, -0.012861062, 0.018110737, -0.007361097, 0.02445142, 0.016493477, -0.0022092918, 0.03840349, -0.017777016, -0.03378275, 0.022962514, -0.00933133, 0.019111898, -0.018726835, 0.0030066904, 0.030445548, -0.014619511, 0.018303268, -0.04045715, -0.0064176954, -0.010890829, 0.012257799, 0.005817641, 0.0042485143, 0.02188434, 0.01099993, 0.0052400483, -0.004251723, -0.042125754, 0.03488659, -0.010281148, -0.007604969, -0.020896016, 0.0037318899, 0.0030644496, -0.014375638, 0.016929882, -0.008105549, -0.0019782546, -0.015569329, -0.028982313, -0.018059395, -0.007739741, -0.0147607, -0.0062508355, -0.032011464, 0.0066487323, 0.007059465, -0.011526181, 0.0014287395, 0.022359252, 0.0031109778, -0.03152372, 0.010114288, -8.5084605e-05, 0.017250765, -0.02364279, -0.0006317419, -0.00061369216, -0.031446707, -0.0066038086, -0.029239021, 3.36929e-05, 0.028006824, 0.004800436, -0.014183107, -0.0019959032, -0.0039821803, -0.03154939, 0.0040559834, -0.0066102264, 0.020613639, -0.0112438025, -0.022590289, -0.0094661005, 0.015299786, -0.019471288, -0.016480641, -0.018778177, -0.02428456, 0.012347646, -0.049775645, 0.009247899, -0.00871523, 0.01723793, 0.026671942, 0.007354679, 0.0050058025, 0.03216549, -0.01838028, -0.03314098, 0.012026762, -0.014439815, -0.007316173, -0.029752437, 0.03350037, -0.029752437, -0.03611879, 0.00083831145, 0.005394073, -0.013721033, -0.014208778, -0.019111898, -0.0043319445, 0.036349826, -0.042716183, -0.0007829589, -0.038172454, 0.008516282, -0.042100083, 0.0093890885, -0.015184267, 0.027698774, 0.010037276, -0.025106024, -0.010512185, -0.013592679, -0.036221474, 0.019586807, 0.0033628724, 0.0024852527, -0.025747795, 0.0111667905, 0.012321975, -0.029803777, 0.00027776588, -0.022359252, 0.026492247, 0.02297535, 0.0036612954, -0.03830081, 0.009690721, 0.0023552943, 0.04302423, -0.0052015423, 0.040149104, -0.003152693, 0.01626244, -0.035631046, 0.0024467465, 0.022256568, -0.000916126, -0.0032409362, -0.04068819, -0.008567624, -0.006116064, -0.011256638, -0.008657471, -0.026312552, -0.0003497644, -0.046720825, -0.011936913, 0.0125658475, -0.031985793, 0.01311777, 0.048646133, 0.0024098447, -0.02823786, -0.042177096, 0.01707107, -0.0014335527, 0.0049833404, -0.017584486, -0.014093259, 0.016236769, 0.040559836, 0.021358091, -0.0025061101, 0.015800366, -0.021910012, -0.0034206316, 0.008959103, 0.018187748, -0.011089778, 0.01640363, 0.008683142, -0.0011295144, 0.014581004, 0.005724584, 0.013207617, -0.022846997, 0.020177234, 0.025811972, -0.026979992, 0.013772375, 0.035990436, -0.0015827642, -0.0017151291, 0.019176073, 0.05283047, 0.003773605, -0.0070145414, 0.027108345, -0.0008130418, 0.013977741, 0.031729087, 0.020921687, 0.012751961, -0.029239021, 0.0017119203, 0.008747319, -0.028135177, -0.017725674, 0.0019750458, -0.012751961, -0.00022201215, -0.012212874, 0.0043640328, -0.008503446, 0.024194712, -0.013489996, 0.0133103, 0.03316665, -0.0039115855, 0.052573763, -0.009350582, 0.008375092, 0.0339111, -0.00046247517, -0.019176073, 0.022230897, 0.04063685, 0.018303268, 0.008227485, 0.016108416, -0.022372087, 0.007251996, 0.024361571, 0.028186519, -0.00047410725, -0.038634527, -0.0021755989, 0.00434478, -0.050597113, -0.025799137, 0.022936843, -0.007868094, -0.033449028, -0.008420017, 0.028751276, -0.011770054, 0.023296235, -0.022436263, -0.014234449, 0.0062893415, 0.011789307, 0.00356503, 0.0023039526, -0.029803777, -0.04792735, -0.023578614, -0.009414759, 0.011737965, 0.0071493126, 0.037556354, 0.020254247, -0.0057021226, -0.06730879, -0.009254317, -0.005714958, 0.0062829237, 0.0027435648, 0.02149928, -0.021268243, 0.004864613, -0.0008992796, -0.040970568, -0.009645796, 0.017802687, 0.034347508, 0.04107325, -0.023963675, 0.005015429, 0.006494708, 0.028648593, -0.0117764715, -0.0069567817, 0.027031334, 0.022423428], "content_hash": "cf4e56cd78a96e88e93eccf8d45837d0"}, {"content": "**Softmax**: The function that converts logits into probabilities (positive numbers that sum to 1.0). **Probability Distribution**: The complete set of probabilities over all possible next tokens. The dashboard shows the top 5. **Temperature**: A setting that controls prediction confidence. Low temperature = more focused; high temperature = more spread out.\n\n**Beam Search**: A generation strategy that explores multiple possible sequences simultaneously instead of just picking the single best token at each step.", "source_file": "key_terminology.md", "chunk_index": 2, "embedding": [0.01611568, -0.0037612289, 0.036948144, -0.014556972, -0.004987864, -0.0042118984, 0.0010749999, 0.01177163, 0.03328857, 0.027894087, 0.00085983047, 0.0110193845, -0.061860356, 0.00528944, 0.035972256, -0.001645114, -0.0065601254, 0.037815597, -0.020602047, 0.015939478, -0.052481003, -0.052291248, 0.0057739946, 0.028707325, -0.004161071, -0.0067871544, 0.025874544, 0.0110193845, 0.03076753, 0.0088168625, 0.009216705, -0.028869973, -0.0150449155, -0.029330809, -0.008186603, -0.029547673, -0.017565956, 0.018989123, 0.017281322, 0.0040661935, -0.05448699, -0.005052245, -0.06299889, 0.027866978, 3.3064294e-05, -0.0045575253, -0.013316783, -0.084143095, 0.0034071312, 0.06045074, -0.022852007, 0.029357916, -0.027243495, 0.06045074, 0.023827894, 0.009840188, -0.011954608, 0.035565637, 0.009711425, -0.026836876, 0.009291252, 0.009020172, 0.04643593, 0.06966744, 0.03393916, -0.0062788804, -0.077366106, 0.028273597, 0.021008667, -0.0018179273, 0.024885103, -0.0037476749, 0.029086838, -0.018053899, -0.02153727, 0.013553978, -0.0001253743, -0.010816074, -0.026633566, -0.037056576, 0.0071022846, 0.013872497, 0.047981083, -0.040092666, -0.0029700154, -0.03860173, 0.0047405036, -0.025020644, -0.03355965, -0.067336164, -0.045297395, -0.025305277, -0.0039509847, 0.033505432, 0.016264774, -0.011405673, 0.028138058, -0.038683053, -0.024573362, -0.00079926115, -0.00320721, -0.040092666, -0.019029785, -0.03130969, 0.02889708, 0.010328131, -0.07254089, -0.027026633, -0.00826115, -0.02687754, -0.04762868, -0.026010083, -0.036378875, 0.07601071, 0.013818281, -0.036487307, 0.01010449, 0.022770682, 0.04323719, -0.01309992, -0.024898658, -0.037381873, -0.0048218276, 0.024885103, -0.043725133, 0.023028208, 0.026159177, -0.05768573, -0.007420803, 0.024288729, 0.011317572, -0.0041949563, 0.032041606, -0.023326397, -0.01789125, -0.030740421, -0.0046151294, 0.029032622, -0.028165165, 0.01591237, 0.017118674, 0.016440975, 0.033180136, 0.024193851, -0.040146884, -0.014191015, -0.03393916, -0.00283617, -0.041204095, 0.044240184, 0.019138217, -0.043833565, -0.020846019, 0.032339793, 0.03448132, -0.04445705, 0.010626319, 0.057197787, -0.018690936, -0.01949062, 0.011785184, 0.034128916, 0.011805515, 0.02020898, 0.0012495073, 0.037192117, -0.039442077, 0.046923872, -0.057523083, -0.0293037, 0.01100583, 0.028869973, -0.008132387, -0.03461686, -0.015492197, -0.0088439705, -0.066794, -0.05158644, -0.04863167, -0.02800252, 0.0019009454, 0.01120914, 0.0065160748, -0.015803939, -0.0068074856, -0.041041445, -0.015993694, 0.012090148, 0.026714891, -0.028300706, -0.0031326632, 0.01649519, 0.057631515, 0.0011334514, -0.035050586, -0.029737428, 0.038222216, 0.03301749, -0.027839871, 0.0039408193, 0.0038323873, 0.0064042546, 0.035701178, -0.012273127, -0.032854844, 0.014421432, -0.027365481, -0.00596375, -0.042532384, 0.0004400807, 0.0221472, -0.004801497, 0.042125765, -0.010097713, -0.05128825, 0.02023609, -0.005879038, 0.047411814, 0.0005150512, 0.07167344, 0.00722427, 0.01339133, 0.027921194, -0.010436563, -0.041692037, 0.022106538, 0.06408321, -0.026389595, 0.047059413, 0.024763118, 0.022323402, -0.015614183, -0.02136107, -0.0490654, -0.032529548, -0.030957285, -0.012889833, -0.011114262, 0.017593063, -0.06261938, -0.02439716, 0.014868714, 0.043725133, 0.036405984, 0.020425845, 0.006112844, -0.043752242, 0.02201166, 0.019368634, 0.0007696118, 0.02266225, -0.010172261, 0.0104026785, -0.019477066, 0.025942314, -0.027216388, -0.043779347, 0.034237348, -0.0005120862, -0.019856578, -0.00637037, -0.013181244, -0.03274641, -0.03206871, -0.022743575, -0.02439716, -0.03261087, -0.043372728, -0.04998707, -0.052968945, -0.007197162, -0.0021110321, -0.03299038, -0.00895918, -0.043426946, -0.0340747, -0.008830417, 0.045947984, -0.033098813, 0.044104643, -0.047493137, -0.022797791, 0.010639872, -0.0009199763, 0.03022537, -0.033722296, -0.014828051, -0.014028368, -0.002727738, -0.03594515, 0.04136674, -0.0012774625, -0.009765641, -0.040905904, 0.029357916, 0.03483372, 0.041827574, 0.007515681, -0.04348116, 0.012517098, 0.044267293, -0.052860513, -0.038249325, -0.055788174, 0.010328131, 0.0028886916, -0.03358676, 0.025752557, -0.018013237, -0.02678266, -0.021970998, 0.026945308, 0.0036629625, -0.02133396, -0.02313664, 0.017159335, 0.053240027, -0.008369582, 0.037463196, -0.016359651, -0.008274703, -0.02115776, 0.0055368, -0.01526178, 0.012347674, -0.00862033, -0.035077695, 0.014462095, -0.0064754128, -0.036948144, 0.038927026, 0.0151262395, 0.037869815, 0.03266509, -0.06825783, -0.02981875, 0.032963276, 0.05080031, -0.0780167, 0.044267293, -0.057252005, -0.025644125, -0.045785338, -0.017864143, 0.004791331, 0.013127028, 0.00047142428, 0.01843341, 0.036948144, -0.019233095, -0.026457366, -0.0081188325, 0.021821905, -0.0077528753, 0.020480061, -0.010314577, 0.024668239, -0.021821905, -0.050312366, 0.006214499, -0.032692194, 0.02327218, 0.05779416, -0.019002678, 0.0088981865, -0.052182816, -0.035890933, -0.038927026, -0.0014748422, -0.00763089, -0.0031817963, 0.027148617, -0.029872967, 0.0150720235, -0.027040185, -0.019273756, 0.019409297, -0.013736957, -0.023190856, 0.049688883, 0.04627328, -0.010849959, -0.0221472, 0.019585498, 0.01771505, 0.014733174, -0.013147359, -0.010253584, 0.018541843, 0.012571314, -0.03808668, -0.02439716, 0.0066041755, -0.028626002, -0.015112685, 0.040390853, -0.019151771, 0.045134746, 0.021184867, 0.01672561, -0.020263197, -0.03673128, -0.041719146, -0.02865311, -0.027921194, -0.014678958, 0.015858155, 0.053782184, 0.049011186, 0.024559809, 0.08473947, 0.0046964535, 0.032854844, -0.054866504, 0.05730622, -0.006255161, 0.008234042, 0.03621623, 0.019178879, 0.066522926, -0.01699669, -0.006624507, -0.026863985, -0.0007255614, -0.0150991315, -0.014841606, 0.037327655, -0.04643593, 0.012849171, -0.02773144, -0.004255949, 0.0347524, 0.020276751, 0.016346097, -0.005408037, -0.027080849, -0.008803309, 0.01856895, -0.008132387, -0.0206156, 0.012652638, -0.0005247931, -0.03976737, -0.011798738, -0.02245894, -0.015749723, -0.032177143, -0.028300706, 0.007908747, -0.0398758, -0.0236788, -0.023014655, -0.033857837, 0.048306376, -0.041719146, -0.034264456, 0.04177336, -0.014855159, 0.05649298, 0.0013477737, -0.0090066185, -0.007027738, -0.012435775, -0.03130969, -0.023041762, 0.018013237, 0.04350827, -0.014665404, 0.031851847, 0.023163749, 0.04350827, 0.008606776, 0.017728603, -0.041854683, -0.011954608, 0.00058409176, 0.06706508, -0.017064458, 0.04483656, -0.016210558, 0.042423952, 0.018284317, -0.017091567, 0.017728603, -0.0042661144, 0.08701654, 0.040363748, -0.0130321495, -0.0010046887, -0.044619694, -0.013716626, -0.043264296, 0.0551918, -0.015519305, 0.04442994, -0.0033156418, -0.027473914, 0.0012063041, 0.0049776984, 0.0025515363, 0.043969106, -0.027866978, -0.011697083, 0.021686364, 0.03843908, 0.061643492, 0.04456548, 0.012645861, -0.03526745, 0.043860674, -0.028626002, -0.057089355, -0.029222377, -0.0016781519, -0.012849171, 0.00082086277, 0.0029835694, -0.0037307325, 0.001165642, 0.0041814023, 0.009392907, 0.013289675, -0.02664712, 0.025115522, 0.013289675, -0.031743415, -0.01764728, -0.04491788, -0.048604567, -0.036785495, 0.015681952, -0.053890616, -0.012855948, 0.030333802, -0.036812603, 0.0020110714, -0.01379795, -0.011060046, -0.005475807, -0.007332702, -0.010999053, 0.008098502, 0.006038297, 0.02892419, 0.007847753, -0.007237824, 0.010924506, 0.000869996, -0.024451377, 0.0038357757, -0.0048692664, -0.0010580574, 0.02889708, -0.033884946, -0.0062043336, 0.0065160748, -0.025915205, -0.01492293, 0.00032127163, -0.006004412, -0.0028886916, -0.019002678, 0.021523718, 0.0052521667, -0.025576357, 0.0044999206, -0.0027091014, 0.03618912, 0.015871707, -0.03445421, -0.030875962, -0.000740386, -0.010355239, 0.0013740346, -0.018460518, -0.023285734, 0.008918517, -0.037300546, 0.009365799, -0.006004412, -0.002241489, -0.022309847, 0.007183608, 0.0010851653, -0.030577775, 0.0055672964, -0.026701337, 0.042911895, -0.0071090613, -0.01343877, 0.028869973, -0.047276273, 0.029032622, 2.0132422e-05, -0.02572545, -0.0020466507, -0.015681952, -0.03938786, -0.015817493, -0.0034189909, -0.008633884, -0.03862884, -0.015492197, -0.0077867606, -0.01696958, 0.025495032, -0.049770206, 0.04272214, 0.03553853, 0.010111268, -0.03220425, -0.045161854, -0.036297552, -0.048387703, 0.009691094, -0.021862566, -0.012035932, 0.008511898, 0.056547195, -0.032421116, -0.018555395, 0.012483214, 0.018867137, -0.0007175137, -0.012368005, 0.010633096, -0.011243025, -0.038276434, 0.034047592, -0.008782978, -0.0088439705, 0.030631991, 0.028734433, 0.00082552194, 0.0012215523, 0.02981875, 0.017227106, 0.021279745, 0.03805957, -0.029141054, 0.029249484, 0.03431867, -0.0008522064, -0.054947827, 0.006705831, -0.006343262, -0.024315836, 0.013431992, 0.004987864, -0.02156438, -0.008301811, -0.01955839, 0.032936167, 0.010951614, -0.021401731, -0.027568791, 0.00465918, -0.006888809, 0.007800314, -0.03540299, -0.0010038415, -0.0008784672, -0.0063093766, -0.03580961, 0.004831993, 0.07422158, 0.034264456, 0.032529548, 0.03580961, 0.017593063, -0.043779347, 0.05833632, -0.012496768, 0.016129233, -0.009142159, -0.0025650903, -0.013289675, -0.028544677, 0.010023166, 0.023868555, 0.0048387703, 0.017552402, -0.04575823, -0.019477066, -0.0018145388, -0.011595428, 0.048197947, -3.949714e-05, 0.031065717, -0.0059298654, -0.015437981, 0.029710319, -0.0149906995, -0.021550825, -0.027717885, -0.025332386, 0.010341685, 0.040797472, 0.0040594162, -0.0062924344, -0.008599999, 0.048469026, 0.0047540576, -0.030279586, -0.057902593, -0.0006332249, 0.00034880315, 0.027839871, 0.013282899, -0.012971157, -0.022716466, -0.0012029156, -0.018758705, 0.008471237, 0.014543419, 0.02892419, -0.0137843955, -0.00974531, -0.021699918, 0.007827423, -0.030577775, -0.012117256, -0.056221902, -0.019246649, -0.0102129225, -0.0044490932, -0.03220425, -0.022472495, -0.038303543, -0.0012706855, 0.0097317565, -0.008593222, -0.05413459, -0.01400126, 0.0018568949, 0.004913317, -0.021849012, 0.00029077518, -0.017471077, -0.032041606, 0.043562487, 0.03103861, 0.0017772653, -0.01122947, 0.0077122133, 0.031499445, -0.020832464, -0.0017137311, -0.002151694, -0.036541525, 0.01577683, 0.011900392, -0.003995035, 0.0029632384, 0.061697707, 0.01136501, 0.013282899, -0.015383765, -0.009352244, 0.01789125, -0.009826634, 0.035484314, -0.015871707, 0.020005671, -0.0316892, -0.031770524, -0.02426162, 0.022364063, -0.012137587, -0.002953073, -0.010436563, 0.0006844759, 0.0032986994, 0.01931442, 0.047845542, 0.02095445, -0.008694877, 0.012523876, -0.03220425, 0.014394324, -0.002554925, -0.01289661, 0.0059434194, -0.008911741, -0.047032304, 0.0074953497, -0.005699448, -0.010924506, 0.00028759846, -0.04771, -0.009040504, 0.00839669, 0.022133647, -0.020385183, 0.015573521, 0.0043948772, 0.028436245, -0.07503482, -0.02405831, -0.00596375, -0.01724066, 0.03223136, -0.045297395, 0.043264296, -0.01921954, -0.017321983, -0.015858155, -0.013269344, -0.011527658, 0.021754134, 0.006400866, -0.019815916, -0.062402513, 0.03805957, 0.011270133, -0.016142787, -0.02293333, -0.051532224, 0.014014813, -0.019382188, 0.01100583, -0.048089515, 0.040201098, 0.0001939913, 0.013337114, 0.03263798, 0.0033478325, 0.022296295, 0.021252638, 0.036270443, 0.022526711, -0.021930337, -0.001302029, 0.0007086189, -0.021998106, -0.006329708, 0.020602047, 0.036921035, -0.024342945, 0.03895413, -0.009359022, 0.0033020878, -0.043291405, -0.033640973, -0.0015129627, 0.022973992, -0.0150720235, -0.0063500386, -0.00510985, 0.0010199369, -0.0049302597, 0.021862566, 0.04405043, -0.0007831658, 0.01744397, 0.019788807, 0.02851757, -0.013635302, 0.0024837665, 0.030144047, -0.03778849, 0.030252479, -0.0514509, 0.0040661935, 0.0038561067, -0.023353504, -0.018040344, -0.018745152, -0.0029937348, 0.0051708426, 0.017701495, 0.0040526395, 0.02296044, 0.01764728, -0.04217998, -0.021903228, 0.007264932, 0.028544677, 0.013933489, -0.04909251, 0.03727344, 0.0013731874, 0.02518329, -0.023895662, -0.01156832, 0.01901623, 0.000897951, 0.049553342, -0.0010817769, -0.0087897545, 0.0016501968, -0.049363587, -0.032421116, -0.008369582, 0.0051640654, 0.03076753, -0.024302283, 0.031119933, 0.0049234824, -0.018324979, -0.009772418, -0.0068074856, 0.010626319, 0.0070345146, 0.006858313, -0.019775255, 0.003632466, 0.025874544, 0.015668398, -0.012191803, 0.030821746, -0.014014813, 0.020154765, 0.031336796, 0.0055876276, -0.0135946395, -0.016291881, 0.009162489, 0.028056735, 0.010273916, 0.045432933, -0.012713632, -0.00042292647, -0.012049487, 0.02678266, 0.0017586286, -0.022391172, -0.017457524, 0.008267927, 0.001739992, 0.002659968, -0.008342474, 0.041176986, 0.019151771, 0.026416702, -0.057577297, 0.025711896, 0.0070955073, 0.016888257, 0.0074140257, 0.029656103, -0.0158446, 0.011974939, -0.017132228, 0.008044286, -0.026769107, 0.025684789, 0.035728287, 7.1687835e-05, -0.023285734, 0.038547512, -0.03499637, 0.043779347, -0.01699669, -0.017864143, -0.0039442074, 0.03450843, 0.05134247, 0.048956968, 0.009772418, -0.010368793, 0.0017103426, -0.00700063, -0.036649957, 0.012483214, 0.0044762013, 0.0103484625, -0.009867296, -0.02273002, 0.0044050426, 0.03898124, 0.010077382, 0.029141054, 0.026769107, 0.024857996, -0.03499637, 0.030360911, 0.0103213545, 0.014855159, -0.016305435, 0.024315836, 0.031228365, -0.012374782, -0.015817493, 0.008328919, -0.030875962, -0.010409455, -0.0069667445, -0.0070209606, 0.045134746, 0.03391205, -0.00078147155, -0.0028005906, 0.015139793, 0.03431867, -0.008986288, 0.00925059, 0.016739164, 0.008213711, -0.00028696313, 0.013052481, -0.0011851259, 0.010362016, -0.007908747, 0.002885303, 0.02660646, -0.009738533, -0.018731598, -0.031011501, -0.009779195, -0.0021313631, -0.021144206, -0.033640973, 0.01710512, 0.014326555, -0.036107797, 0.00960977, -0.02493932, -0.025237506, 0.052101493, 0.011602205, 0.017823482, -0.010517887, 0.014746728, 0.008044286, -0.051206928, 0.016684948, -0.015749723, -0.0016595152, 0.057143573, -0.004706619, 0.029520564, 0.015275333, -0.009277698, -0.032258466, 0.017023796, 0.017091567, 0.008972733, 0.023380611, 0.003268203, -0.009921512, -0.016603623, -0.0035646963, 0.014245231, 0.026389595, -0.008891409, -0.024017649, 0.00038777085, 0.023231518, 0.03635177, -0.020574939, -0.009223482, 0.00528944, 0.000818745, 0.008112056, 0.03022537, 0.008728762, 0.00012812745, -0.009420015, -0.013086366, -0.014028368, 0.01730843, -0.018975569, -0.003164854, 0.0053436556, 0.022160754, -0.0059197, -0.045649797, 0.0030411738, -0.011581874, 0.01836564, -0.022852007, -0.0042118984, -0.005336879, -0.015749723, 0.010165484, -0.0035579193, -0.0002217346, -0.0022584316, -0.012164695, -0.0070345146, -0.021686364, 0.023570368, -0.017457524, -0.013791173, 0.026511582, 0.008478013, 0.012354451, 0.04307454, -0.0061433404, 0.010477225, -0.028707325, 0.011039715, -0.022675805, 0.011541212, -0.009833411, -0.011148147, 0.010721196, -0.02752813, -0.010924506, 0.0063059884, 0.01672561, 0.032394007, 0.011602205, -0.030198263, -0.03220425, 0.00053495856, 0.013567532, 0.0041271863, -0.006648226, 0.04724917, 0.01170386, -0.038655944, -0.0083356965, -0.0048658783, 0.03450843, -0.010863514, -0.0078070913, -3.3567274e-05, -0.0020330967, 0.01352687, -0.0060823476, -0.00637037, 0.022377618, 0.012279904, -0.02273002, 0.029276593, 0.0062924344, 0.0007344562, 0.017850589, -0.0012427303, 0.0016781519, 0.017660834, -0.024546254, -0.017972576, -0.036649957, -0.03846619, -0.031445228, 0.023041762, -0.00051420403, -0.0005404649, -0.0150449155, 0.024952874, 0.003639243, 0.030062724, 0.01969393, -0.005655397, 0.054595422, -0.022580927, -0.01057888, -0.015681952, 0.011453112, 0.022784237, -0.009115051, -0.017972576, -0.035511423, -0.0041237976, 0.012388336, 0.052697867, -0.025291722, -0.012652638, 0.020778248, -0.008539006, 0.0150178075, -0.0028276986, 0.019165324, -0.028544677, 0.010233253, 0.030469343, -0.0013842, -0.014516311, -0.011602205, 0.014204569, -0.01904334, -0.026077854, 0.0011935972, 0.001151241, -0.036270443, -0.013079588, -0.0067397156, 0.017213551, 0.016088571, -0.0071903854, -0.009697871, 0.013492985, 0.0009267532, 0.029222377, -0.014665404, -0.04217998, -0.0029835694, -0.00540126, 0.020602047, 0.017213551, -0.003117415, -0.00729204, -0.010287469, -0.029195268, 0.035701178, 0.01856895, 0.019504175, -0.015993694, 0.029276593, -0.014489203, 0.004340661, 0.01282884, 0.012320566, 0.024966428, -0.041827574, 0.008742316, -0.019870132, 0.0197617, 0.016603623, -0.04985153, -0.022025215, 0.015465089, 0.01505847, 0.011690306, -0.027663669, 0.016020803, 0.0009835105, 0.015681952, 0.010158706, 0.010484002, -0.007366587, 0.0017450746, -0.0015002559, 0.01246966, -0.014299447, 0.006722773, -0.027189279, -0.008525453, 0.027568791, 0.010334908, 0.0095691085, 0.026416702, -0.024898658, 0.0077935373, -0.011615759, -0.0006455082, 0.0027802598, 0.03716501, -0.0058349874, -0.0109922765, 0.023936326, -0.00074928085, 0.016129233, 0.0039713155, 0.0046354607, -0.010680535, -0.03247533, 0.0087626465, -0.013472654, 0.0017687941, -0.041827574, 0.0134591, 0.019205987, 0.0020517334, 0.003934042, 0.036948144, 0.05784838, -0.019327972, -0.01526178, 0.028083842, -0.010687311, 0.013608194, -0.011358233, -0.044890776, 0.028815757, 0.024641132, 0.028110951, -0.034942154, 0.01226635, 0.010328131, -0.01816233, 0.010924506, -0.016102126, 0.008945625, -0.001082624, 0.024166742, 0.013892828, -0.004679511, 0.033505432, 0.012910164, 0.017728603, -0.011439557, -0.008464459, 0.03529456, -0.01253743, -0.012564538, -0.027284158, 0.0045744674, 0.009738533, 0.0055672964, -0.0058078794, -0.0035409767, -0.008186603, 0.04312876, -0.01850118, -0.022892669, 0.023150194, 0.028571786, 0.018921353, -0.018392747, 0.0026633567, -0.0039408193, 0.01343877, 0.014462095, 0.029601889, -0.00938613, -0.035321668, 0.0023041763, 0.001397754, -0.004676122, -0.0340747, 0.006139952, 0.016752716, 0.0110464925, -0.0009953702, -0.040011343, -0.00988085, -0.0005845153, 0.005726556, -0.024505593, 0.004008589, -0.014936483, -0.044348616, -0.014136799, -0.01087029, 0.018731598, 0.00078486005, -0.0027328208, 0.018880691, -0.014082583, -0.0046219067, -0.011581874, 0.0003903122, 0.006902363, 0.011121039, -0.03011694, -0.0027023244, -0.032339793, 0.024722455, 0.004354215, 0.011812292, 0.0026955474, 0.01190717, 0.0150720235, -0.013506539, 0.00068532303, 0.017294876, 0.002246572, 0.0004172084, 0.031092826, -0.022784237, -0.030171154, 0.004899763, 0.00252104, 0.008010401, -0.005845153, -0.020195426, -0.014367216, 0.046137743, -0.024627578, 0.004001812, 0.0028463353, -0.014421432, -0.008450905, 0.01679338, -0.0008284869, 0.008681322, -0.00096995657, -0.05408037, 0.023516152, -0.00981308, -0.037029468, 0.03182474, -0.02115776, -0.0011597122, -0.0012223994, -0.018704489, 0.036677066, 0.010165484, -0.024193851, -0.035321668, 0.0068244277, 0.0028886916, 0.014055476, -0.035565637, -0.013601417, 0.010572103, 0.023177302, 0.0031970446, 0.030848853, -0.025237506, 0.021211976, -0.03673128, 0.0006111997, 0.009128604, 0.003178408, -0.03263798, -0.024342945, 0.02487155, -0.008450905, 0.010273916, -0.008694877, -0.03778849, 0.018108115, -0.015695507, -0.027623007, 0.0152346715, -0.0055368, 0.014299447, 0.047032304, 0.023434827, -0.019666823, -0.021415286, -0.0037985023, -0.032800626, 0.03399338, -0.023855, -0.019924348, 0.022160754, 0.04350827, 0.016806932, -0.0061162324, -0.0032122927, -0.02183546, -0.012279904, -0.014706066, 0.008437351, -0.0022109926, -0.009359022, -0.0063635926, 0.025467925, 0.025982976, -0.0071497234, -0.0150720235, -0.0077257673, 0.0090066185, 0.030930178, -0.04283057, 0.014516311, 0.006543183, 0.0076986593, -0.007339479, -0.011426004, 0.029141054, -0.00047820128, -0.017132228, 0.0049438137, 0.0060552396, -0.0016840817, 0.049499128, 0.003295311, 0.00081112084, 0.006299211, -0.011270133, 0.03583672, -0.00032254233, -0.025467925, 0.005323325, -0.030848853, 0.014421432, 0.023502598, -0.014082583, -0.0095691085, 0.0221472, -0.052996054, -0.006200945, 0.0068142624, -0.0036019697, 0.018514734, -0.027744994, -0.002144917, 0.017294876, -0.010463671, 0.0008666075, 0.007434357, 0.040661935, 0.009982505, -0.021510163, 0.025169738, -0.006743104, -0.00021919324, 0.02426162, 0.03727344, -0.024546254, -0.019666823, -0.005865484, -0.027812762, -0.045297395, -0.045134746, 0.025589911, -0.019151771, -0.00763089, 0.016142787, 0.012185026, -0.01177163, 0.0077596526, -0.007312371, -0.04995996, -0.007644444, 0.0006933707, 0.011595428, 0.048875645, -0.009840188, -0.025440816, -0.022092985, -0.025400154, 0.03618912, 0.01850118, 0.03792403, -0.032583762, -0.011385341, -0.0633784, -0.005421591, 0.013919936, 0.019870132, 0.030848853, 0.01741686, -0.025169738, 0.0109651685, 0.00585193, -0.022621589, 0.01881292, -0.009528447, -0.0049573677, 0.04817084, -0.02586099, 0.021523718, 0.00834925, 0.04058061, -0.038357757, -0.017254215, 0.025671234, -0.013262567], "content_hash": "60c46f3b672c8bd7794b7d953db20efa"}, {"content": "# Introduction to Mechanistic Interpretability\n\n## What Is Mechanistic Interpretability?\n\nMechanistic interpretability (often called \"mech interp\") is a field of AI research that aims to understand **how** neural networks work internally -- not just what they predict, but why. Instead of treating models as black boxes, researchers open them up and study the individual components (neurons, attention heads, layers) to figure out what each one does.\n\nThis dashboard is a tool for doing exactly that kind of investigation.\n\n## How This Dashboard Relates\n\nThe experiments available in this dashboard are real techniques used in mechanistic interpretability research:\n\n- **Ablation** (removing heads to test their importance) is a standard tool for identifying which components are responsible for specific behaviors\n- **Token attribution** (measuring input influence via gradients) is used to trace how information flows from inputs to outputs\n- **Attention pattern analysis** (categorizing heads by behavior) helps researchers build a map of what each head does\n- **Head categorization** (Previous-Token, BoW, Syntactic, etc.) builds on research that has identified recurring head types across models\n\n## Key Concepts in the Field\n\n### Circuits\n\nA **circuit** is a small subnetwork within the model that performs a specific function. For example, researchers have found \"induction circuits\" -- combinations of attention heads across layers that work together to complete patterns like \"A B ... A\" \u2192 \"B\" (if the model has seen \"A B\" before, when it sees \"A\" again, it predicts \"B\").\n\nIn the dashboard, you can start to identify circuits by ablating combinations of heads and seeing which combinations have outsized effects.\n\n### Superposition", "source_file": "mechanistic_interpretability_intro.md", "chunk_index": 0, "embedding": [-0.026108723, 0.036923856, -0.007753743, 0.01745929, -0.002281004, -0.01040071, 0.0033270908, 0.0187293, -0.012071775, 0.024451027, 0.0079475865, -0.05093407, -0.022860173, -0.011162716, 0.007981008, -0.035399847, -0.008435537, 0.010126655, 0.015494117, 0.04858121, 0.003930345, -0.024437658, 0.044436965, 0.060372245, 0.009906075, -0.007332634, 0.004765878, 0.03320741, 0.008602643, 0.019638358, 0.020948473, -0.012673358, -0.028795796, -0.028073896, -0.0245045, -0.020146362, 0.008743013, -0.012045038, -0.012479515, 0.029517695, 0.011303085, -0.015881805, -0.029731592, 0.03852808, -0.05598737, 0.007887428, -0.0024932292, -0.023541966, -0.017686555, 0.056468636, -0.046816565, 0.05047954, -0.020092888, -0.0016134134, 0.013756209, 0.040987886, -0.046789825, -0.0052905926, -0.01071487, -0.020386996, -0.018956564, -0.0061896257, 0.03280635, 0.033849098, -0.037217963, -0.019691832, -0.013455417, 0.020681104, -0.0003511326, -0.0021991218, 0.057965912, 0.023167647, -0.0118913, 0.021670373, 0.0017796844, -0.060265295, -0.036843646, 0.03842113, -0.021870902, -0.0149727445, -0.0064503117, -0.009137385, 0.01979878, -0.055933896, -0.039784722, -0.026402831, -0.027004413, 0.006924894, -0.02260617, -0.06973021, -0.01076166, 0.017405815, -0.018167822, 0.011837826, 0.028795796, 0.0069516315, -0.030961497, -0.011503613, -0.005561305, 0.036335643, 0.0077002686, -0.04216432, 0.008930173, -0.0048260363, 0.044169597, -0.052458078, 0.0018532113, -0.025507139, -0.028501688, -0.036041535, -0.03534637, -0.010888661, -0.013361838, 0.0034039598, 0.0037431861, -0.010206866, -0.041362207, -0.012065091, -0.013575734, -0.029811803, -0.029651381, -0.01863572, 0.032030977, 0.0036529487, -0.008415485, -0.05162923, 0.019544778, -0.007867375, -0.01626949, -0.006099388, 0.002794021, 0.038127024, -0.02044047, -0.07138791, -0.0051101176, -0.033501517, 0.006680919, 0.034784894, -0.051147964, -0.0114969285, 0.011503613, -0.003766581, 0.01003976, 0.007987692, -0.057217274, 0.007693584, 0.035292897, 0.05063996, -0.01990573, 0.033875834, -0.0028508373, 0.0020704498, 0.014758848, -0.07299213, -0.023956392, -0.011463507, -0.0038233972, 0.043848753, 0.015400537, -0.021376267, 0.05473073, 0.028207581, -0.013368522, -0.0072657918, -0.06277858, 0.02767284, -0.029918753, 0.01620265, -0.023194386, -0.02098858, -0.025092715, -0.013983474, -0.017165182, -0.024491131, 0.025801247, -0.013014256, -0.06352722, -0.03229835, 0.028501688, -0.018194558, -0.0077938484, -0.0054610413, -0.035319634, 0.014464741, -0.027993685, -0.012145302, -0.042378213, 0.030506967, -0.014825691, -0.030105911, -0.022312064, 0.06261816, 0.02007952, 0.012225513, -0.0068079196, -0.037378386, 0.052163973, 0.007018474, 0.063901536, -0.033581726, 0.002257609, 0.015106429, 0.0672704, 0.008368694, 0.029597906, 0.063206375, 0.019237302, 0.015667908, -0.021028684, -0.036121745, 0.009431493, -0.022485854, 0.03470468, 0.016937917, -0.06539881, -0.039303456, -0.04130873, 0.046415508, 0.035025526, 0.039677773, -0.005942308, 0.007954271, 0.025253138, 0.017432552, -0.014999482, 0.037832916, 0.015093061, -0.014812322, 0.016109068, -0.012693412, 0.018782774, 0.0028023764, -0.0082416935, 0.022499222, 0.0038802135, -0.055826947, 0.013889894, -0.024651553, 0.040667042, 0.0033204067, 0.025279874, 0.0120116165, -0.003927003, 0.019411094, 0.029624645, 0.021456478, -0.018836247, -0.01600212, -0.017138446, 0.034945317, 0.049089212, 0.047618676, -0.0656127, 0.02116237, -0.021991218, 0.0038534764, -0.026723675, 0.0017345657, -0.013903263, 0.039089557, -0.023555335, -0.02332807, -0.05588042, -0.005668253, -0.02586809, -0.04141568, -0.0052103815, -0.0036061588, -0.058821496, -0.027458943, 0.0059556765, 0.023488494, -0.002461479, 0.026710307, -0.07336645, -0.045613397, 0.015253483, 0.0245045, -0.016563598, 0.0016627099, -0.03609501, -0.01953141, 0.089248255, 0.0108285025, -0.05978403, 0.011657351, 0.0014237476, -0.020320153, 0.009451545, -0.028261054, -0.020935105, 8.71565e-05, 0.025279874, 0.0016033871, -0.019424463, -0.017419184, 0.01239262, 0.019558147, 0.033474777, 0.011724194, 0.051522285, -0.056361686, 0.010841872, 0.008027798, 0.027432207, 0.014464741, 0.0012775294, 0.0014112146, 0.045747083, -0.0020587523, -0.03128234, -0.042084105, 0.019304145, 0.0066374713, -0.0230607, 0.0145315835, 0.0468433, 0.0018766063, 0.035827637, -0.048153415, -0.018715931, -0.015333694, 0.027646104, -0.009331228, -0.0014630177, 0.020627629, -0.0017613027, 0.009217596, -0.037592284, 0.0004787602, 0.056415163, -0.018288137, 0.002590151, 0.017071603, -0.025761142, -0.009478282, -0.027178206, 0.0062330733, -0.05109449, 0.042351477, -0.04585403, -0.004588745, -0.07085317, 0.035399847, 0.033394568, 0.040827464, -0.035747427, -0.017312236, 0.04312685, 0.005410909, -0.018675825, -0.024263866, -0.029544434, -0.06267163, -0.021763954, 0.062190365, -0.04256537, -0.013174678, 0.022646276, -0.019638358, -0.039036084, 0.014077053, 0.004739141, 0.027619366, -0.01402358, -0.023849443, -0.038447868, -0.0006258139, -0.016068963, 0.0013452074, 0.02433071, -0.010708186, -0.014397898, -0.010073181, -0.04882184, 0.00087062496, -0.028020421, -0.019638358, -0.03636238, 0.03192403, 0.046522457, 0.0050499593, -0.018515402, -0.032378558, 0.025547246, 0.026603358, 0.014705374, -0.029785067, -0.06427585, 0.036522802, 0.020747947, -0.05518526, 0.03165666, 0.022084799, -0.029704856, 0.019571517, -0.0052371183, 0.0019284092, 0.049891323, 0.019691832, -0.0056348317, 0.006019177, -0.0600514, -0.018929826, 0.029785067, 0.047458254, 0.013742841, 0.021763954, -0.016536862, 0.030105911, 0.056575585, -0.026603358, 0.027485682, 0.011664035, 0.013769577, -0.03235182, 0.023354808, 0.0084088, 0.035961322, 0.02171048, -0.0149727445, -0.027913474, 0.005013196, -0.010093234, 0.00053891854, 0.05243134, 0.043715067, -0.024156919, 0.043153588, -0.076200575, -0.000709785, 0.045105394, 0.04772562, -0.024156919, 0.0029360617, -0.026857361, 0.067003034, -0.001487248, -0.01270678, -0.008482327, 0.037217963, -0.016790863, 0.0031198787, -0.011403349, -0.042725794, 0.0040506623, -0.021723848, -0.09138721, 0.007245739, -0.06058614, 0.02388955, 0.013321732, 0.03454426, 0.01402358, -0.016884442, -0.035774164, 0.06384806, 0.002075463, 0.009244333, -0.006690945, 0.041442417, 0.022766592, -0.017766766, 0.05887497, 0.001565788, 0.022084799, -0.03716449, -0.041041363, 0.006530523, -0.041362207, 0.011623929, -0.027164835, 0.03355499, 0.0060659666, -0.033768885, -0.03572069, 0.057110325, -0.020788051, 0.0048795105, -0.027258417, -0.0073058973, -0.0032819721, -0.013729472, -0.014224107, 0.03751207, 0.050666697, -0.05534568, -0.03606827, 0.035587005, 0.019050144, -0.025119452, -0.024170287, 0.0042077424, -0.014718742, 0.009444861, -0.011062452, -0.047591936, -0.044089384, -0.0012023314, 0.02088163, 0.03427689, -0.015253483, -0.011837826, 0.057270747, 0.00501988, 0.038394395, 0.010988925, -0.0050098537, -0.030801075, 0.013074414, -0.011336506, -0.0293038, -0.0035159213, 0.029277062, -0.0097256, 0.011710825, 0.018702563, 0.008529117, 0.052057024, -0.006346706, -0.013375206, -0.028154107, 0.0025734405, -0.0035226054, 0.00450185, -0.008235009, -0.05614779, -0.036977332, -0.054677255, -0.0077470583, 0.0030363256, -0.01383642, 0.014865796, 0.029785067, -0.03040002, -0.014732111, 0.013943369, -0.026523147, -0.03924998, 0.02804716, 0.032432035, 0.014411267, 0.0055245417, 0.01700476, -0.017793503, 0.012446093, -0.006784525, -0.040667042, 0.024357447, 0.039731245, 0.0059022023, -0.037859652, 0.03981146, -0.03761902, -0.015587697, -0.007025158, 0.0020637657, -0.02577451, 0.025440296, -0.014250844, -0.021630269, -0.028956218, -0.006126125, -0.012840466, -0.008950225, 0.0051635914, 0.010200182, -0.024651553, 0.025012504, -0.02133616, -0.0024297289, 0.0011112584, -0.00018809928, 0.010013022, 0.031362552, -0.007994376, -0.016750759, -0.007813901, -0.0013385232, -0.0020620944, -0.030239597, -0.02268638, -0.034972053, 0.024290605, -0.02695094, 0.032512244, 0.017579606, -0.007813901, -0.018943196, -0.008295168, -0.014798953, -0.019143723, 0.0060659666, 0.033341095, -0.022820067, 0.0068246303, 0.016991392, 0.01763308, -0.034303628, 0.031362552, -0.003126563, -0.0049263, 0.002921022, -0.009939496, -0.01520001, 0.0083152205, -0.033501517, 0.032726143, 0.010587869, -0.013134573, -0.01998594, 0.0056214635, 0.0025700983, -0.022111535, -0.062083416, 0.05363451, -0.025654193, 0.013782946, 0.011503613, -0.016135806, -0.00542762, -0.013254889, 0.0008372037, -0.032137927, 0.023100806, 0.0023729126, 0.010607922, -0.02334144, 0.01402358, -0.005527884, -0.025827983, 0.008856646, 0.035480056, -0.0013000887, 0.007192265, 0.007767111, 0.013809683, -0.0036763435, 0.047324568, -0.0111292945, 0.010253657, -0.033849098, -0.003046352, 0.00087730924, 0.019691832, -0.014705374, 0.010654712, 0.025426928, -0.0343571, -0.005688306, -0.039276715, -0.023301333, 0.0468433, 0.012680043, 0.0036596328, -0.005938966, 0.011289717, -0.015213378, -0.022231853, 0.023127543, -0.0032719458, 0.009525072, -0.004912932, -0.031095183, -0.025199663, 0.020119626, 0.02296712, 0.012653306, 0.010013022, 0.025988406, -0.041736525, 0.017071603, 0.0098526, 0.026630096, 0.01717855, 0.0035493425, 0.013675998, -0.007018474, -0.036415853, -0.0016576967, 0.023127543, 0.01184451, -0.012225513, 0.013595787, 0.00041317087, 0.0081681665, 0.038314182, 0.004745825, 0.001626782, -0.03887566, 0.010480921, -0.0040473198, 0.010440815, -0.0045419554, -0.032378558, 0.0048126676, 0.023435019, 0.019237302, -0.0062063364, -0.10293762, -0.0003172935, -0.026603358, -0.003455763, -0.008522432, -0.003636238, -0.0098526, 0.042538635, 0.033474777, 0.018421823, -0.06186952, -0.07021148, 0.012994203, 0.00814143, 0.025827983, -0.0040840833, 0.029063167, -0.010053128, 0.0020270022, 0.006019177, -0.032886565, -0.051602494, 0.020026045, -0.05769854, -0.002177398, 0.0151197985, -0.016857706, -0.022753224, 0.0075398465, -0.017125076, 0.009645388, 0.004531929, 0.014103791, -0.011851194, -0.0687142, 0.004896221, -0.0015031232, -0.0010335539, -0.006343364, -0.009077227, -0.007346003, -0.0058754655, 0.016630441, 0.0031416027, -0.00063918246, -0.019611621, 0.03545332, -0.017405815, 0.00886333, -0.024357447, -0.04096115, -0.008087955, 0.011764299, -0.015173272, 0.015962016, 0.0012842136, 0.021389635, 0.012218829, -0.041816738, 0.01565454, 0.0068981573, -0.026309252, 0.020386996, -0.008495696, 0.013228152, -0.03465121, -0.036041535, 0.058714546, -0.012593147, 0.011998248, 0.03590785, 0.020293416, -0.007974323, 0.024076708, -0.013609155, 0.006152862, -0.024918925, 0.042191055, 0.014224107, 0.013355154, 0.0119380895, -0.03184382, -0.014518214, -0.021763954, -0.01637644, 0.00046706272, 0.03040002, -0.053206716, -0.032030977, -0.021028684, -0.033982784, -0.04205737, 0.0025416901, 0.010567817, -0.03427689, -0.014758848, -0.0025467034, -0.0015081364, -0.08823224, -0.009117332, 0.025600718, -0.010327183, 0.0044517177, -0.0045753764, 0.029143378, 0.016750759, 0.017218657, -0.0656127, -0.0023712416, -0.0007666012, 0.021242581, 0.019103618, 0.028100632, -0.028688848, -0.008174852, 0.0013794643, 0.037832916, 0.026309252, -0.01979878, 0.019838886, -0.00773369, -0.008308536, -0.057270747, -0.020132994, -0.019772043, 0.006323311, 0.026710307, 0.018702563, -0.030667389, 0.002974496, -0.041843474, 0.006607392, -0.005608095, 0.020574154, 0.038180497, 0.008248378, 0.015293589, 0.0067945514, -0.0053073033, -0.011470192, 0.014210738, -0.005430962, 0.04676309, 0.021175738, 0.021175738, -0.012506252, -0.01691118, 0.03879545, -0.0072858445, 0.003666317, 0.025841353, -0.0063634166, 0.021937745, -0.0055847, 0.0013443719, 0.026710307, 0.022566065, -0.00037494526, 0.003970451, 0.008829908, 0.032378558, 0.014932639, -0.00097506656, 0.038127024, 0.007693584, 0.012940729, -0.030613914, -0.007833954, -0.0021189107, -0.004404928, -0.020453839, 0.031496238, 0.0101667605, -0.02171048, 0.020226574, -0.005735096, -0.0007845651, -0.015547591, 0.04593424, -0.012325777, -0.039330192, 0.025747772, -0.005123486, 0.020280048, -0.014117159, 0.011283033, -0.0014897546, 0.034731418, 0.019892361, 0.01943783, 0.012780307, -0.019116987, -0.0043180324, 0.033608463, 0.0022826751, 0.037191227, 0.09625336, -0.004558666, 0.020333521, 0.027512418, -0.0011521995, -0.012593147, -0.03644259, 0.022900278, -0.038768712, -0.014143896, 0.01157714, 0.017486027, -0.011750931, 0.020814788, -0.022472486, -0.007847322, 0.013415311, 0.0112028215, 0.017191919, 0.0027605998, 0.0069783684, -0.018675825, 0.010253657, -0.01620265, -0.03425015, -0.0041175047, 0.01763308, 0.030640652, 0.035426583, 0.0066274446, 0.017325604, 0.010995609, -0.021804059, 0.0064904173, 0.01438453, -0.0018415139, -0.016429914, -0.005922255, 0.003666317, 0.013421996, -0.008181536, 0.029597906, -0.042806007, -0.011162716, -0.0008388747, 0.012131933, -0.012145302, 0.007579952, -0.03138929, -0.034062993, -0.026723675, 0.020600893, 0.0020370285, 0.035132475, 0.006336679, 0.037993338, -0.009638704, 0.05652211, 0.010146708, -0.05903539, -0.028073896, 0.031416025, -0.007352687, 0.027539155, 0.010741607, 0.007332634, 0.0111292945, 0.019103618, -0.02205806, 0.008743013, -0.009197543, 0.035961322, -0.047859307, -0.008482327, 0.010153392, 0.010875292, -0.0023261227, 0.011617245, -0.0035927903, 0.018100979, 0.014103791, 0.0056348317, -0.0048795105, 0.039677773, -0.016470019, 0.008702908, 0.019865623, -0.023287965, 0.008803172, -0.023394912, -0.032325085, -0.009484966, 0.017338973, 0.004759194, -0.0057985964, 8.5642096e-05, -0.039490614, -0.011978196, -0.006537207, 0.032244876, -0.014678637, 0.005437646, -0.05652211, 0.0180074, 0.005688306, -0.032726143, 0.010246972, 0.028341265, 0.032619193, -0.0036562907, -0.010521026, -0.021042053, -0.024918925, -0.018047504, 0.0066374713, 0.034758158, 0.014237476, 0.046362035, 0.01645665, -0.009050489, 0.0114969285, -0.0223388, -0.014237476, -0.041469153, -0.021723848, 0.014905902, -0.008943541, 0.031335816, -0.02414355, 0.0019835543, 0.020640997, 0.002408005, -0.0034992106, 0.0038133708, -0.0036830278, 0.004020583, 0.0051301704, 0.015935278, 0.008355326, 0.009484966, 0.05197681, -0.018167822, 0.014986113, 0.003505895, -0.0343571, -0.02658999, -0.016710652, -0.030560441, 0.03860829, 0.025467034, 0.0057284115, -0.013488838, -0.0032000898, 0.033929307, 0.04793952, -0.021550057, 0.013448733, -0.028608637, 0.04721762, 0.030801075, 0.017071603, 0.002742218, 0.0372447, 0.01808761, 0.023996497, -0.019477936, 0.016603705, -0.013889894, -0.0038868978, 0.00041922848, -0.013903263, 0.024584712, -0.0020871605, 0.025266506, 0.011423402, -0.0052103815, -0.022312064, 0.00011008142, -0.030961497, -0.007660163, -0.0128270965, -0.02097521, -0.0017980662, 0.008208272, 0.017138446, 0.013575734, 0.014986113, -0.0058520706, -0.031335816, -0.002382939, 0.010581185, -0.029865278, -0.030827811, 0.011135979, 0.028715584, -0.042645585, -0.03625543, 0.007118738, -0.030720863, -0.013756209, -0.00800106, -0.022779962, 0.008883383, 0.013595787, -0.014825691, -0.01807424, 0.025573982, -0.038661767, 0.020921737, -0.008582591, -0.0048093256, 0.021149002, -0.03491858, -0.0024013207, 0.020106256, 0.03280635, 0.029330537, -0.0023010569, 0.015360432, -0.047083933, 0.00030183615, -0.01592191, -0.003793318, 0.032432035, 0.0004449211, 0.020132994, -0.013669314, -0.0009642046, -0.0016468348, -0.0007402819, 0.0019000012, -0.017338973, -0.018328244, 0.017379079, -0.010995609, -0.0074863723, -0.013288311, 0.027966948, -0.005922255, -0.047458254, 0.015534222, 0.014718742, -0.00872296, 0.019571517, 0.01717855, -0.0063032582, 0.0073192655, -0.010053128, 0.0043080063, -0.0189833, 0.02759263, 0.031041708, 0.06427585, -0.0035961324, -0.0058888337, 0.008749697, 0.0027004415, 0.0041208467, -0.033153933, 0.009077227, -0.046415508, -0.013903263, -0.01438453, 0.053554296, 0.015962016, -0.0016334662, -0.0066207605, -0.0012457791, -0.065452285, -0.0016008805, -0.0063868114, 0.029063167, 0.00517696, 0.0034758158, -0.016657177, -0.015320326, 0.028982956, -0.011389981, 0.022017956, 0.02577451, 0.027178206, 0.018929826, -0.0025801247, -0.025721036, 0.026523147, 0.0022191745, 0.0064937593, 0.009638704, 0.018047504, 0.0010135011, -0.039544087, -0.011924721, 0.035827637, 0.0055445945, 0.0031031682, -0.026803887, 0.0014939323, 0.026683569, -0.028368004, -0.003459105, -0.021322792, -0.014919271, -0.03328762, -0.006767814, 0.014665268, -0.008990331, 0.0034106441, -0.014491478, 0.0025433612, 0.047645412, -0.0073192655, 0.014665268, -0.0099461805, 0.03355499, -0.013034308, -0.0057785437, 0.012639937, -0.023822706, -0.03310046, 0.008783119, -0.0055746734, 0.010587869, -0.006991737, -0.014237476, 0.013147941, 0.019277409, 0.029490959, 0.014946007, -0.00782727, -0.0068781045, 0.015587697, -0.028662112, -0.030774336, -0.014478109, -0.009043805, -0.004334743, -0.033875834, -0.007513109, 0.011396665, -0.00271381, -0.043982435, 0.013000888, 0.019638358, 0.019865623, -0.02621567, 0.0482871, 0.014611795, -0.032592457, -0.03834092, -0.021763954, 0.03491858, -0.0012858846, -0.013629208, 0.004588745, -0.024424288, -0.0180074, -0.03285983, 0.021095527, -0.035587005, -0.018408455, -0.00050424394, -0.010895345, 0.008642749, 0.0072056334, -0.032378558, 0.017472658, -0.009986286, 0.0019484621, 0.012232197, 0.016576966, -0.052324396, -0.03491858, 0.039838195, -0.033982784, -0.036282167, -0.006045914, -0.004846089, 0.009531756, 0.0074596354, -0.0008981976, -0.0068313144, 0.011724194, -0.016804231, 0.035747427, 0.020854894, -0.0070318426, -0.022579433, -0.00583536, 0.02097521, -0.029089903, -0.03906282, 0.05743117, 0.0032669324, 0.011303085, -0.022833435, -0.054142512, 0.023221122, 0.013769577, 0.011443455, 0.013368522, -0.03887566, -0.0018749352, -0.020774683, -0.001970186, 0.04232474, -0.0066441554, -0.013442049, -0.04938332, -0.04285948, 0.011403349, -0.007660163, -0.0007235712, -0.0029327194, 0.014611795, -0.017151814, 0.0010084879, 0.025186295, 0.018288137, -0.03176361, -0.0068714204, 0.030988233, -0.0022860172, -0.0021991218, -0.011430087, 0.008268431, 0.0035927903, 0.006841341, -0.048902053, 0.013889894, -0.027178206, -0.047431514, 0.0066508395, -0.0060091508, 0.016991392, 0.013368522, -0.029196851, 0.00451856, 0.035079002, -0.034062993, -0.015814962, 0.013970105, 0.0005861261, -0.010641343, -0.042191055, -0.0012207131, -0.0024464396, 0.029250326, 0.04267232, -0.003333775, 0.028207581, -0.006924894, 0.035587005, -0.016924549, -0.0008430524, -0.008214957, -0.027806526, -0.02360881, 0.038554817, -0.02514619, 0.009611967, -0.0036696591, -0.0035226054, -0.025440296, -0.017151814, 0.024892187, -0.010013022, 0.008148114, 0.012633253, -0.029597906, -0.0012842136, -0.0060091508, -0.01763308, -0.0031399315, 0.009471598, 0.012138618, -0.022779962, -0.017913818, 0.040854204, -0.023662284, -0.04096115, 0.00836201, -0.034731418, -0.009825864, 0.00057609973, -0.036041535, 0.022178378, -0.002252596, 0.00021191196, -0.029597906, 0.027485682, -0.0052304342, 0.023033964, -0.0032919985, 0.03165666, 0.011289717, 0.028608637, 0.006273179, 0.014946007, 0.015814962, -0.059890978, -0.014892533, -0.0029160087, 0.011597193, 0.02523977, -0.019772043, -0.02296712, -0.011537034, -0.01049429, 0.020814788, -0.0034357102, 0.0058520706, -0.005531226, -0.050853856, -0.006146178, 0.017298868, 0.023956392, 0.024397552, 0.03419668, 0.008936857, 0.016242754, -0.019852255, 0.014678637, -0.013415311, -0.02088163, -0.024651553, 0.0044985074, 0.05705685, 0.03997188, 0.044891495, 0.007346003, -0.014678637, -0.016617073, 0.012379251, -0.006226389, 0.0042645587, 0.0011839498, 0.010267025, 0.000997626, 0.004692351, 0.014464741, 0.013528944, 0.011095873, 0.0061294674, 0.037565548, 0.018034136, -0.004511876, 0.011283033, 0.017579606, -0.02216501, 7.483239e-05, 0.015975384, 0.017846977, -0.0051168017, 0.014478109, 0.032726143, -0.024972398, 0.025012504, 0.020574154, -0.042806007, 0.024758503, -0.012573095, -0.008301852, 0.017499395, -0.0013761222, -0.0079475865, -0.030881286, -0.003250222, -0.002969483, -0.012426041, 0.0047224304, -0.0005456028, -0.0007691078, 0.043207064, 0.018769404, 0.010126655, 0.0036529487, 0.07021148, -0.014317687, -0.014932639, 0.010079865, -0.0010051457, -0.010868608, -0.004866142, 0.016777495, 0.059837505, 0.0023578731, 0.018435191, 0.012486199, -0.010547764, 0.013823052, 0.04312685, -0.006273179, -0.023943022, 0.0030797732, -0.012479515, -0.031068444, 0.014865796, 0.046522457, -0.00013065641, -0.0074930564, 0.008642749, 0.0029143377, -0.021657005, -0.0024280578, -0.012412673, -0.014852428, -0.0018582245, 0.009398071, 0.0027087966, -0.01066808, 0.0019818833, -0.006263152, -0.0194512, -0.015694644, -0.020828158, 0.0076200576, 0.02315428, 0.03355499, 0.0021038712, -0.019504674, -0.04072052, -0.0066174185, 0.0024213735, -0.028715584, 0.024651553, -0.019330883, -0.0020153048, -0.015708013, 0.015226746, -0.010848556, 0.027833262, 0.009665442, 0.040586833, -0.026483042, 0.036415853, 0.0027539155, 0.00023018924, -0.017312236, -0.017232025, 0.01438453, -0.0018899748], "content_hash": "6a3cdaa616edc03a2d22ed59b3d0476b"}, {"content": "together to complete patterns like \"A B ... A\" \u2192 \"B\" (if the model has seen \"A B\" before, when it sees \"A\" again, it predicts \"B\"). In the dashboard, you can start to identify circuits by ablating combinations of heads and seeing which combinations have outsized effects. ### Superposition\n\n**Superposition** is the idea that neural networks represent more features than they have dimensions. A 768-dimensional embedding might encode thousands of different concepts by overlapping them. This makes interpretation challenging because a single neuron can participate in many features.\n\n### Induction Heads\n\n**Induction heads** are one of the best-understood circuits. They are pairs of attention heads (typically one in an early layer and one in a later layer) that work together to copy patterns from context. If the model has seen \"Harry Potter\" earlier in the text and encounters \"Harry\" again, induction heads help it predict \"Potter.\"\n\nYou might observe induction-like behavior in the dashboard when using prompts with repeated patterns.\n\n### Polysemanticity\n\nNeurons and heads are often **polysemantic** -- they respond to multiple unrelated features. An attention head might handle both pronoun resolution and list formatting, depending on the input. This is why head categories are approximate: the same head may behave differently for different prompts.\n\n## Notable Research Groups\n\nThese organizations have published influential work in mechanistic interpretability:\n\n- **Anthropic**: Published foundational work on transformer circuits, superposition, and dictionary learning for interpreting neural networks\n- **EleutherAI**: Open-source AI research group that has contributed tools and analysis for model interpretability\n- **Redwood Research**: Focuses on alignment-relevant interpretability, including causal interventions on model behavior\n- **DeepMind (Google)**: Research on understanding internal representations and how models store knowledge\n\n## Further Reading", "source_file": "mechanistic_interpretability_intro.md", "chunk_index": 1, "embedding": [0.011560152, -0.010602332, 0.00859091, 0.011626462, 0.008929831, 0.0013787084, 0.021499377, 0.015722986, -0.030267116, 0.01396207, 0.020187901, -0.05605458, -0.015619836, -0.012856893, -0.0015739563, -0.015104086, -0.008907727, 0.032801654, 0.037428662, 0.0497477, -0.012260097, -0.033597384, 0.042409327, 0.068078905, 0.025021208, -0.022295104, 0.003945482, 0.09336536, 0.026259007, 0.024446517, 0.030267116, -0.027894668, -0.046211135, -0.020585764, -0.04948246, 0.018301733, -0.010432871, 0.0125032365, -0.028174646, 0.07161547, 0.03648558, -0.047242634, -0.02994293, 0.020099487, -0.014153634, 0.01920061, -0.023798145, -0.029456653, -0.03032606, 0.08428817, -0.02854304, 0.03418681, -0.018846953, 0.0030613404, -0.010904414, 0.019627944, -0.04845096, 0.011309645, -0.008369874, -0.017889133, 0.003177384, -0.019318495, 0.02874934, 0.02720209, -0.041053645, -0.012311673, -0.04845096, 0.019554267, -0.006258986, 0.021882506, 0.046800565, 0.027393656, -0.0021035203, 0.015295651, 0.0057063974, 0.0003941798, -0.022412991, 0.03890223, -0.02505068, -0.030090287, 0.0044575473, -0.009246648, 0.029869251, -0.066546395, -0.03097443, -0.01612085, -0.022250898, -0.02474123, -0.06725371, -0.05962062, 0.013343171, -0.018832216, -0.025198037, 0.012289569, 0.0060011116, 0.013822081, -0.007964643, -0.013512632, 0.017962812, 0.03459941, -0.01252534, -0.026509514, -0.013343171, -0.008730899, 0.023282396, -0.0039381143, -0.009732926, -0.042704042, -0.014404141, -0.03518884, -0.06725371, 0.009526626, -0.017564947, 0.002352185, 0.021838298, -0.028292533, -0.06253829, -0.010226572, 0.0053343214, -0.0023098201, -0.013954702, -0.02251614, 0.0208068, 0.018699596, 0.00042203948, -0.045886952, 0.007824654, -0.024078123, 0.016135585, -0.010727585, 0.010617067, 0.015575629, 0.013888392, -0.04821519, -0.009401373, -0.03442258, -0.030414473, -0.01464728, -0.014897787, -0.029618746, 0.028528303, -0.012952675, -0.0020519453, -0.00036977383, -0.027261034, 0.024652816, 0.024770701, 0.027423127, -0.010602332, 0.015251444, -0.013394746, -0.036397163, -0.031770155, -0.028425153, -0.035837207, -0.028012555, -0.00797201, 0.052311715, 0.028395683, -0.052518014, 0.029589273, 0.03763496, -0.016887106, 0.0134757925, -0.024063388, 0.054757837, -0.071674414, -0.0011401743, -0.028012555, -0.047920477, -0.014153634, -0.008900359, -0.014536763, 0.0027850461, 0.046800565, -0.032624826, -0.055907223, -0.036839236, -0.04081787, -0.02388656, 0.007986746, 0.0098655475, -0.051073916, 0.024785437, -0.029589273, -0.017402854, -0.032565884, 0.038253862, -0.019583737, 0.0071504954, 0.0037189208, 0.038342275, 0.057675507, 0.02005528, 0.028115705, -0.050278187, 0.08841416, -0.0028789863, 0.026435835, -0.055907223, 0.0025879564, 0.02422548, 0.060062688, 0.01984898, 0.03076813, 0.025478015, 0.011928544, 0.020276316, -0.044737566, -0.03230064, 0.027835727, -0.026848435, 0.046004836, 0.004369133, -0.06578014, -0.036986593, -0.051457044, 0.014234681, -0.003470256, 0.045916423, 0.02357711, 0.011655934, -0.0047006863, 0.00886352, -0.016739748, 0.053814754, 0.027732575, -0.021794092, 0.014514659, -0.00192485, 0.05947326, 0.007669929, 0.005864806, -0.0010968882, -0.029456653, -0.05861859, 0.043794483, 0.0006046239, 0.028587246, -0.023636052, 0.029766101, 0.016916577, -0.0017074986, 0.050572902, 0.033420555, 0.04839202, -0.008635117, -0.019510059, -0.0038939072, 0.02590535, 0.05543568, 0.041672543, -0.028115705, 0.019053252, -0.016061906, -0.020335257, -0.024638081, -0.016901841, -0.024608608, 0.0058095474, -0.020777328, 0.0264211, -0.012724272, -0.0110001955, -0.026435835, -0.024505459, -0.0310039, 0.003908643, -0.037988618, -0.04414814, -0.008642484, -0.00052680104, 0.020983629, 0.012584283, -0.07668455, -0.051545456, 0.009246648, 0.016887106, -0.029279824, 0.0015582996, -0.01659239, 0.014212577, 0.04777312, 0.014418877, -0.036190864, 0.009556098, -0.020615237, -0.022250898, -0.043440826, -0.03097443, 0.005341689, -0.020688914, -0.005735869, -0.014087323, -0.0034942015, -0.018316468, 0.02303189, 0.038165446, 0.04497334, -0.0053159017, 0.02292874, -0.057852335, -0.0012276676, 0.018876424, 0.022133013, 0.019023782, -0.02197092, 0.003374474, 0.033243727, -0.020659443, -0.009688719, -0.012908468, 0.026642134, 0.0057579726, -0.053785283, -0.015988229, 0.061182603, 0.006793155, 0.008296195, -0.041554656, -0.026332686, -0.013976806, 0.024107596, -0.004048632, 0.007927803, 0.01731444, -0.03545408, -0.0053195856, -0.032536414, -0.015973492, 0.047684707, -0.026155857, 0.004763313, 0.011611727, -0.04762576, -0.06436551, -0.03757602, 0.024137067, -0.034894124, 0.015384065, -0.08098737, -0.008598277, -0.040847342, 0.010727585, -0.012208522, 0.084347114, -0.027157884, 0.030163966, 0.025816936, 0.029751366, -0.023297133, -0.044089198, -0.008671956, -0.046653207, 0.014868315, 0.03734025, -0.037016064, -0.005153809, 0.009482419, 0.0056990297, -0.036603466, 0.0070436615, 0.051928587, 0.031269144, -0.041289415, 0.0042991387, -0.04226197, 0.021705678, -0.0062295147, 0.018095432, 0.012532708, -0.002748207, -0.013556839, 0.0017636784, -0.052193828, 0.025964294, -0.03315531, 0.00044943867, -0.046918448, 0.053490568, 0.021617264, 0.00010574012, -0.02313504, -0.036986593, 0.009968697, 0.035336193, 0.016503977, -0.019053252, -0.045680653, 0.017152349, 0.00298582, -0.071674414, 0.004111259, 0.010705481, -0.016901841, 0.027747313, 0.0052164355, -0.0066789533, 0.057646036, 0.05072026, 0.015708249, -0.01974583, -0.029751366, 0.008716163, 0.041230474, 0.037487607, 0.03501201, 0.007356795, 0.0010443924, 0.0022932424, 0.046623737, -0.024181275, 0.0038754875, 0.0008726294, -0.03424575, -0.025345394, 0.022943476, -0.005010136, 0.045297522, 0.055347268, 0.022162484, -0.056850307, -0.027806254, -0.026126385, -0.033626854, 0.025198037, 0.050572902, -0.06707688, 0.029441917, -0.05970903, -0.0042659836, 0.059797447, 0.04523858, -0.008450921, 0.043558713, -0.035129894, 0.06725371, 0.01697552, -0.022737175, -0.018198581, 0.040906288, -0.03527725, -0.027054735, -0.020983629, -0.016798692, 0.01995213, -0.013497896, -0.070613444, -0.009106659, -0.06507283, 0.022015126, 0.009327695, 0.007047346, 0.019701622, -0.013903127, -0.012201155, 0.069139875, -0.0023263977, 0.059797447, -0.047183692, 0.023002418, 0.017668096, -0.024166537, -0.0047375257, 0.022737175, 0.005864806, -0.023400282, -0.02888196, 0.023709731, 0.0036286647, -0.005113286, -0.027806254, 0.026465306, -0.021248871, -0.002265613, -0.022457197, 0.032389056, 0.016091378, 0.033803683, -0.034098394, -0.013888392, -0.041348357, -0.002840305, 0.008296195, 0.025610637, 0.062361456, -0.041908313, -0.012974778, 0.03179963, 0.036456108, -0.026612664, -0.03946219, 0.03291954, -0.018404882, 0.026597928, -0.01824279, -0.04547435, -0.04246827, -0.0073973183, -0.0068299943, 0.057999693, -0.050189774, -0.029883988, 0.029648216, -0.006023215, 0.018316468, 0.017270233, 0.016916577, -0.049335103, 0.027334712, -0.01454413, 0.0012322724, 0.00595322, 0.014079955, -0.018994309, 0.014949362, 0.021219399, 0.008915095, 0.018714331, -0.0047006863, -0.017535476, -0.019996338, -0.04329347, 0.0015131716, 0.03424575, -0.022118276, -0.024019182, -0.04759629, -0.054816782, -0.006590539, 0.006885253, -0.016386092, 0.007139444, 0.0095192585, -0.042144086, -0.004332294, 0.0016955258, -0.030620772, -0.04102417, 0.0106760105, 0.004487019, 0.030443944, 0.008030953, 0.020983629, -0.016179793, 0.015929285, 0.0073604793, -0.03651505, 0.0068299943, 0.060357403, -0.006697373, -0.017122876, 0.026789492, -0.0528422, -0.026244272, -2.7701378e-06, -0.013984174, -0.008907727, 0.03097443, -0.011515945, -0.008870888, -0.046918448, 0.0020206321, -0.006266354, -0.0026597928, 0.026804227, 0.019686887, -0.011287542, 0.031269144, -0.037428662, -0.016327148, -0.025522223, 0.044649154, -0.012223259, 0.0076036183, -0.019583737, -0.008598277, 0.006642114, -0.014249416, -0.0008376321, -0.03949166, -0.035336193, -0.02429916, 0.010292882, -0.033508968, 0.014941994, -0.009725558, -0.009246648, -0.02368026, -0.0020832587, 0.01242219, -0.018419618, -0.0029545065, 0.015384065, -0.030473415, -0.012739007, 0.0013676566, -0.027762048, -0.04394184, 0.044089198, -0.01321055, 0.009489787, -4.371781e-05, -0.008266725, -0.01013079, 0.000867564, -0.024593873, 0.019893186, 0.033273198, 0.027437862, -0.014802004, -0.0050359233, 0.0040117926, 0.00010556744, -0.058294404, 0.017579682, -0.011935912, 0.010189733, 0.015457743, -0.014772533, 0.026568456, -0.017800719, 0.001538038, -0.018286996, 0.029382974, 0.009106659, 0.022029862, 0.003989689, 0.002939771, -0.023415018, -0.032389056, -0.002744523, 0.04205567, 0.007272065, -0.01659239, 0.0005641008, -0.004708054, -0.021676205, 0.055759866, -0.015899815, -0.0023908664, -0.02645057, -0.016106114, 0.0035660379, 0.01710814, -0.017992282, -0.018301733, 0.008642484, -0.016916577, -0.02409286, -0.03200593, -0.030031344, 0.024903324, 0.019627944, -0.0034813078, -0.016577655, -0.0011797765, -0.012709536, -0.01184013, 0.015855607, -0.03004608, -0.015118822, -0.008870888, -0.027452597, -0.013070561, 0.00859091, 0.03518884, 0.025139093, 0.009401373, 0.009482419, -0.04520911, 0.017638626, 0.044501796, 0.042379856, -0.0015233024, -0.017388118, 0.011206496, -0.010138158, -0.004100207, 0.010690746, 0.022560347, 0.026907377, -0.030591302, -0.016223999, 0.0043765013, 0.00865722, 0.014838844, 0.0030631823, -0.0062847733, -0.010565492, -0.0052716946, -0.013203182, -0.014735694, -0.026715813, -0.007261013, -0.031917512, 0.006785787, 0.006789471, -0.031770155, -0.05970903, -0.009570833, -0.00020215531, -0.018360674, -0.011781188, -0.0013345013, -0.026671607, 0.0033468446, 0.02857251, 0.020821536, -0.030090287, -0.08216623, 0.018979574, 0.01352, 0.035336193, 0.002026158, 0.013718931, -0.00046210215, -0.0027684686, 0.0033965774, -0.038931705, -0.052989557, 0.012392718, -0.048244663, -0.012834789, 0.022737175, -0.016798692, -0.025183301, -0.013365274, -0.040464215, 0.03784126, -0.015472479, 0.009025613, -0.004472283, -0.029486123, 0.02432863, 0.02269297, -0.015295651, 0.023400282, -0.034510996, -0.028970374, -0.009349798, 0.0027611007, 0.0021440436, 0.02429916, 0.022309842, 0.021720413, -0.012444294, 0.040022146, -0.027334712, -0.026214799, 0.00041467164, 0.014809373, -0.0012460871, 0.003713395, 0.0056585064, 0.006303193, -0.0012313514, -0.013387378, 0.035807736, 0.0164303, 0.0030189753, 0.034540467, -0.020865742, 0.013394746, -0.010108686, -0.012871629, 0.041348357, 0.0026542668, -0.015575629, 0.025507485, 0.015384065, -0.0019045884, 0.016577655, -0.0073604793, 0.011707509, -0.024623346, 0.036780294, 0.018449089, -0.001972741, 0.00886352, -0.010388664, -0.0019616892, 0.018110167, -0.011206496, 0.0026763703, 0.0020648392, -0.0158114, -0.050455015, -0.0065794876, -0.08092843, -0.038077034, 0.01615032, 0.008318299, -0.022015126, 0.002554801, -0.018832216, -0.01666607, -0.11924124, 0.0016080326, 0.008141471, -0.009777133, -0.0022195638, -0.016798692, 0.051339157, 0.029751366, 0.021278342, -0.03928536, 0.021926712, -0.018110167, 0.014411509, 0.015045144, 0.03527725, -0.050042417, -0.024181275, -0.027983082, 0.018198581, -0.0034813078, -0.014234681, 0.014698855, 0.0012387193, -0.0011816184, -0.045621708, -0.0088266805, -0.02292874, 0.0073973183, 0.003468414, 0.006550016, -0.044884924, -0.022354048, -0.051869642, 0.018110167, -0.0022490353, 0.016061906, 0.030709187, -0.0071173403, 0.00859091, -0.010138158, -0.0057579726, -0.021307813, -0.001108861, -0.017977547, 0.027850462, 0.010749688, 0.016990256, -0.0076772966, -0.0076404572, 0.036131922, 0.0001353842, -0.020025808, -0.0037723377, -0.0026119018, 0.0049548773, 0.006424763, -0.0060858415, -0.008745634, 0.036898177, 0.0106760105, -0.009276119, 0.026229536, 0.048981447, 0.025993764, -0.00055627245, 0.023002418, 0.0121569475, 0.011700141, 0.002359553, -0.016165057, -0.0056585064, 0.01752074, -0.037104476, 0.023046626, -0.008981406, -0.018802745, 0.026391627, -0.004479651, 0.0074341577, -0.015781928, 0.06153626, -0.0035899836, -0.047714178, 0.02704, 0.00940874, 0.027953612, -0.011958016, -0.013114768, -0.015251444, 0.04730158, 0.0077730785, -0.010771792, -0.0061300485, -0.013232653, -0.008627749, 0.019922659, 0.008841417, 0.026067443, 0.0636582, -0.018714331, 0.043617655, 0.038931705, -0.040906288, -0.040169504, -0.042586155, 0.027879933, -0.02771784, -0.0008058583, 0.0070068226, 0.02186777, 0.005691662, 0.0026837382, -0.012775847, -0.030237645, 0.017270233, 0.024151802, 0.025109623, 0.012039062, 0.003960218, -0.0019045884, 0.014043116, -0.03209434, -0.015236707, -0.007003139, -0.026096914, 0.0034942015, 0.0020979943, 0.014315727, -0.0076330896, -0.009902386, -0.03336161, -1.1562627e-05, -0.004656479, 0.021293078, 0.0101676285, -0.002361395, -0.011037035, 0.034334168, -0.042586155, 0.0027058418, -0.023429753, -0.022265634, 0.018994309, -0.0015711933, -0.005157493, 0.00093571655, -0.024623346, -0.019097459, -0.019377438, 0.0031460708, 0.026583193, 0.016268207, -0.0022858744, 0.028852489, -0.0017268391, 0.029176675, 0.026155857, -0.047861535, -0.039373774, 0.053785283, 0.0006359373, 0.028248325, 0.019451115, -0.013866288, 0.0021403595, -0.010116054, -0.03887276, -0.004078103, -0.007419422, 0.021499377, -0.02665687, -0.020644708, 0.020821536, 0.03247747, 0.011073874, 0.024623346, 0.0009274277, 0.013195814, 0.010683378, 0.01163383, 0.0030981796, 0.03501201, -0.008664588, 0.012952675, 0.011707509, 0.013512632, -0.001970899, -0.0040265284, -0.026376892, 0.0032971115, 0.0025861142, 0.012429558, -0.024652816, 0.02186777, -0.0024295475, -0.0036010353, 0.0059163813, 0.025035944, 0.0046527954, -0.0019211661, -0.020349994, 0.035984565, -0.006712109, -0.034923594, -0.026627399, 0.048893034, 0.016901841, -0.0151925, -0.0129895145, -0.008244621, -0.016872369, 0.01516303, 0.0160177, 0.015030408, 0.0052127517, 0.019790037, 0.016165057, 0.024343366, 0.005540621, -0.009283488, -0.017388118, -0.026023235, -0.029471388, 0.0022324577, -0.0013446321, 0.014603073, -0.025507485, 0.02375394, 0.010219204, -0.015384065, 0.003904959, 0.014300991, 0.00759625, 0.029338766, 5.7935453e-05, -0.0013906811, 0.010757057, 0.0006893542, 0.041908313, 0.0020427357, 0.025684314, 0.018773274, -0.025094887, -0.03542461, -0.007264697, -0.02334134, 0.0060858415, 0.022250898, 0.0035347247, -0.015870342, -0.03333214, 0.021440435, 0.04738999, -0.015840871, -0.0015804032, -0.014838844, 0.03993373, 0.042144086, 0.019627944, 0.015472479, 0.025522223, 0.025301186, 0.005481678, -0.032978483, -0.010174997, -0.013321067, -0.0030429207, -0.02398971, -0.0053969477, 0.00033523704, -0.045621708, 0.040434744, -0.0033394767, 0.010661274, -0.011169656, -0.0032952696, -0.029677687, -0.00058067846, -0.01046971, -0.010690746, 0.0067710513, 0.00828146, 0.00886352, 0.0147946365, 0.024417045, -0.01375577, -0.037045535, -0.008060425, 0.0044022887, -0.02877881, -0.022280369, -0.017034462, 0.01433783, -0.025021208, -0.024829645, 0.0133136995, -0.023282396, 0.009062452, 0.016179793, -0.028425153, 0.022958212, 0.011862233, 0.008649852, -0.032801654, 0.023798145, -0.009460315, 0.013939966, -0.062832996, -0.015104086, 0.0049880324, -0.025374865, 0.027319977, 0.02655372, 0.0044059725, 0.0208068, 0.022663498, 0.011729612, -0.053254798, -0.017388118, 0.00045427383, -0.0028274113, 0.020968892, 0.012053798, 0.027408391, -0.0032436945, -0.031269144, 0.01594402, -0.015546157, 0.021749884, 0.0038902233, -0.020968892, -0.008377242, -0.0087087955, 0.010978092, -0.031917512, 0.030414473, -0.011037035, -0.021911977, 0.013063192, 0.013556839, 0.009482419, 0.0019580054, 0.009762397, 0.0024737546, 0.004674899, -0.004951193, -0.005864806, 0.0058537545, 0.0074894163, 0.024623346, 0.056791365, -0.02088048, -0.0076478254, -0.005684294, 0.013387378, 0.016223999, -0.034481525, 0.011655934, -0.028439889, -0.019686887, -0.0405821, 0.009217177, -0.016179793, -0.003273166, 0.0060895258, 0.0019635311, -0.023857089, -0.018640652, 0.020335257, 0.016710278, 0.025315922, 0.028955638, -0.01156752, -0.022958212, 0.0046638474, 0.027143149, 0.013380011, 0.022575084, 0.016503977, 0.02916194, -0.011007563, -0.030178702, -0.0076036183, 0.035070952, -0.013107399, 0.013100032, 0.0011291226, -0.011420163, -0.03445205, 0.015708249, -0.010204468, 0.0034868335, -0.0015279073, -0.036426637, 0.0067710513, 0.02833674, -0.009401373, 0.023017153, -0.014529394, -0.008310932, -0.024903324, 0.020349994, -0.014426244, -0.0065094926, 0.011191759, -0.009939225, 0.004372817, 0.04986559, -0.014831476, 0.015899815, -0.024004446, 0.054875724, -0.015487215, 0.008841417, -0.02088048, -0.031416498, -0.037693907, 0.009099291, 0.015310386, 0.017476533, -0.014536763, -0.0028329373, 0.023901295, 0.002884512, 0.014138899, 0.009261384, -0.011545416, 0.012635858, 0.034068923, -0.009622408, -0.010071847, -0.012599018, -0.018935367, -0.002352185, -0.010757057, -0.017388118, 0.023606582, -0.011678037, -0.03097443, 0.016356621, 0.015958756, 0.025610637, -0.0023890245, 0.039550602, 0.013836817, -0.019775301, -0.004579117, -0.014757797, 0.020482615, -0.004615956, -0.015207237, 0.0051759128, -0.027599955, -0.008598277, -0.022265634, 0.008568806, -0.03966849, -0.011081242, -0.0022527191, -0.003945482, 0.0042733513, -0.012002223, -0.009747662, 0.007662561, -0.010705481, -0.015207237, 0.008878255, 0.02748207, -0.042615626, -0.04547435, 0.010734953, -0.044266026, -0.010661274, -0.0033118473, 0.019009044, 0.018051226, 0.011221231, -0.025006473, -0.022133013, 0.0007390872, 0.0025713786, 0.045120694, 0.00965188, -0.018360674, 0.0077730785, 0.01892063, 0.04479651, -0.020482615, -0.047036335, 0.053431626, 0.0077730785, 0.029618746, -0.02802729, -0.012945307, 0.024372838, 0.012915836, 0.019863715, 0.015560893, -0.021705678, -0.009681351, -0.024564402, -0.005787444, 0.025212772, 0.003031869, -0.0022711388, -0.049394045, -0.014072588, -0.0059053292, -0.010808632, -0.031062843, 0.0055037816, 0.018021753, 0.008649852, -0.0016817111, 0.022280369, 0.007552043, -0.019672152, -0.008134103, 0.0037649698, -0.00046348362, 0.014418877, -0.018522767, 0.0210131, -0.008251988, 0.00742679, -0.039403245, 0.017461797, -0.011707509, -0.013350539, 0.009239281, -0.018095432, 0.008303564, -0.0026211115, -0.02104257, -0.0021495693, 0.029014582, -0.013527367, -0.0036820816, 0.0036876074, -0.004862779, -0.018316468, -0.04420708, -0.0153988, 0.002742681, 0.015855607, 0.062479343, -0.01865539, 0.032654297, 0.009423477, 0.011228599, 0.0052606426, -0.019966865, -0.011118081, -0.034569938, -0.019244816, 0.01943638, 0.010454975, 0.01813964, -0.00022667642, 0.0032915857, -0.0032694822, -0.022914004, 0.002552959, 0.0028808282, -0.0020666812, 0.0069147246, -0.008694059, 0.0034831497, -0.015487215, -0.011434899, -0.012186419, -0.016621863, 0.020733122, 0.011582255, -0.039845318, 0.014072588, -0.033450026, -0.023356074, 0.0046380595, 0.022707704, 0.019672152, -0.023252925, -0.0125032365, 0.025772728, -0.00501382, -4.856735e-07, -0.040316857, 0.014197841, 0.005058027, 0.029515594, -0.009114027, 0.029780837, 0.0023945502, 0.03395104, -0.02942718, 0.026244272, 0.02686317, -0.025198037, -0.031946983, 0.01612085, -0.00070639234, 0.014603073, -0.031328086, -0.020423671, -0.0104034, -0.037929676, 0.015516686, 0.009261384, 0.026538985, -0.0040596835, -0.056820836, -0.013136871, 0.002687422, 0.0041812533, 0.0026745284, 0.033420555, -0.0030521306, 0.011405427, -0.0055664084, -0.0049990844, -0.00759625, 0.0018603813, -0.04205567, -0.009290855, 0.020349994, 0.021956185, 0.02932403, 0.017653361, 0.0036231389, -0.024932794, 0.0024977, -0.030827072, -0.008134103, -0.0048038363, 0.027275769, 0.025183301, 0.019274287, 0.008568806, 0.02104257, 0.014433612, 0.006686321, 0.055907223, 0.016209263, -0.028484097, 0.02751154, 0.0012175367, -0.02093942, -0.021941448, -0.003941798, 0.030502887, -0.010285514, -0.0031460708, 0.030886015, 0.0023337656, 0.030532358, 0.023149775, -0.023945503, 0.036220334, -0.00578376, -0.015634572, 0.020762593, -0.011655934, -0.027909404, -0.018773274, 0.010609699, -0.0017056565, 0.004461231, 0.01279795, -0.010749688, 0.018316468, 0.021808827, 0.031180728, 0.027231563, 0.012591651, 0.04544488, -0.023503432, -0.018228054, -0.001153989, 0.0055627245, -0.010609699, 0.011479106, 0.011921177, 0.051280215, -0.0029342452, 0.029014582, -0.017668096, -0.019009044, -0.006354768, 0.05499361, -0.00041029698, -0.018493297, 0.0015932969, -0.002024316, -0.027261034, 0.0070620812, 0.033273198, 0.014212577, -0.0050027682, 0.0067636836, 0.02419601, -0.02388656, -0.005010136, -0.02546328, -0.0067526316, 0.015634572, 0.016209263, -0.01868486, -0.011626462, -0.02248667, -0.0038423322, -0.031475443, -0.0045496454, -0.019185873, 0.0032510625, 0.011228599, 0.0060084793, 0.015177765, -0.021646734, -0.02706947, -0.024358103, -0.007128392, -0.0022398254, 0.041702013, 0.00944558, -0.030163966, -0.037988618, -0.022162484, -0.01827226, 0.0026616347, -0.017122876, 0.016356621, -0.021499377, 0.01892063, -0.016725013, 0.0049548773, 0.00491067, -0.008811945, -0.00043723566, 0.004627008], "content_hash": "4cd25d11b4ff6033d61b01ed146c1ea8"}, {"content": "dictionary learning for interpreting neural networks - **EleutherAI**: Open-source AI research group that has contributed tools and analysis for model interpretability - **Redwood Research**: Focuses on alignment-relevant interpretability, including causal interventions on model behavior - **DeepMind (Google)**: Research on understanding internal representations and how models store knowledge ## Further Reading\n\nIf you want to explore the research behind this dashboard's techniques:\n\n- \"A Mathematical Framework for Transformer Circuits\" (Elhage et al., Anthropic) -- foundational paper on how attention heads compose into circuits\n- \"In-context Learning and Induction Heads\" (Olsson et al., Anthropic) -- how models learn to copy patterns from context\n- \"Locating and Editing Factual Associations in GPT\" (Meng et al.) -- how facts are stored in MLP layers\n- \"Attention Is All You Need\" (Vaswani et al., 2017) -- the original Transformer paper\n\nThese papers are referenced here for context. The dashboard provides a hands-on way to explore many of the concepts they describe.", "source_file": "mechanistic_interpretability_intro.md", "chunk_index": 2, "embedding": [-0.028181022, -0.020663494, 0.0035840878, -0.013948183, 0.009160773, -0.0055863895, -0.005596094, 0.053153187, -0.03985195, 0.024208765, 0.010706977, -0.0510312, -0.017959256, -0.019835403, 0.03017362, -0.0033544214, 0.015293833, 0.0144139845, 0.003939909, 0.06407365, 0.027637586, -0.02195739, 0.03470225, 0.08032497, 0.021025786, -0.022177352, 0.003444994, 0.054602347, 0.0090119755, 0.012434326, 0.018425057, -0.013767038, -0.06158938, -0.011425088, -0.04093882, 0.007452832, 0.00059680885, 0.010804019, -0.030277131, 0.053929523, 0.02207384, -0.012000872, -0.03079469, 0.070646636, -0.035737365, -0.0087402575, -0.025049796, -0.0028320116, -0.04515692, 0.061848156, -0.029086748, 0.042853784, -0.04590738, 0.012790147, -0.03128637, 0.016950017, -0.03433996, 0.021271626, -0.018916737, -0.009697739, -0.015656123, -0.017441697, -0.00057699607, 0.028258655, -0.02869858, -0.026033157, -0.05408479, 0.0065406365, -0.0015437779, 0.0049362076, 0.05517166, 0.020818762, 0.012563716, -0.010739325, 0.019356662, -0.017001774, -0.043009054, 0.046373177, -0.043630123, -0.0141422665, -0.0061718766, -0.018683836, 0.01704059, -0.042853784, -0.01287425, -0.03348599, -0.00895375, 0.0005822525, -0.029552551, -0.038092252, 0.0033414825, -0.03912737, -0.025450904, -0.018567385, 0.013534136, 0.026524836, -0.0018130697, 0.005676962, 0.001976424, 0.03690187, -0.011974993, -0.042103328, 0.0068641105, -0.00809331, 0.01361177, -0.014543374, 0.03126049, -0.048728067, -0.015876085, -0.0008588225, -0.099164076, -0.013999938, -0.010687568, -0.009180182, 0.029578429, -0.026227241, -0.051678147, -0.0049523814, -0.011069267, -0.01938254, -0.030070107, -0.005592859, 0.051962804, 0.037005384, 0.004855339, -0.019162577, 0.028491557, -0.019718952, 0.026071973, -0.0090702, 0.005528164, 0.02784461, 0.0066085663, -0.06034724, -0.017726354, -0.04686486, -0.010875183, 0.007543405, -0.03547859, -0.036979504, 0.02048235, 0.009264285, -0.0230572, -0.003194302, -0.034546982, 0.010066499, 0.012828964, -0.006734721, -0.010072969, 0.04453585, 0.0072652176, 0.0028853847, -0.013870548, -0.037548818, -0.05260975, -0.028155144, 0.014194023, 0.029707817, 0.03444347, -0.06365961, 0.015837269, 0.058432274, -0.0019408418, 0.014012877, -0.05077242, 0.016613605, -0.020547044, -0.00047469753, -0.027327051, -0.030639421, -0.026887128, -0.019020248, -0.014530435, 0.01374116, 0.02220323, -0.02636957, -0.024972163, -0.014582191, -0.0076663247, 0.002781873, -0.026938884, 0.023238344, -0.033046067, 0.011877951, -0.028413923, 0.00084426615, -0.027456442, 0.012621941, -0.007543405, -0.0038913877, -0.02683537, 0.04818463, 0.06676496, 0.027508197, 0.012854842, 0.0033576563, 0.03620317, 0.02525682, 0.030277131, -0.047537684, 0.025722623, 0.0385063, 0.0257485, -0.020883458, 0.03175217, 0.026550714, 0.043707754, 0.0038816836, -0.03211446, -0.027301174, 0.033511866, -0.015850207, 0.020417655, 0.02243613, -0.057707693, -0.033149578, -0.0409647, 0.0069740913, 0.021970328, 0.048883334, 0.009853006, 0.03408118, 0.0067605986, 0.0019667195, -0.027378807, 0.04158577, 0.013779976, -0.026084913, 0.008203291, 0.009050792, 0.046813104, 0.0056057977, -0.002539268, 0.011949115, 0.02440285, -0.07613275, 0.024066437, -0.042387985, 0.023613574, -0.017312309, 0.021129297, -0.005576685, -0.016445398, 0.03348599, -0.0006340083, 0.02635663, -0.019654257, -0.03664309, 0.018399179, 0.05271326, 0.033796523, 0.021776244, -0.028853847, -0.019641317, -0.02709415, -0.0032832574, -0.014828031, -0.033304844, -0.020676434, -0.0015720818, 0.0075628133, 0.036694847, -0.0006166216, -0.02463575, -0.003181363, -0.029811328, -0.028517434, 0.016471276, -0.044820506, -0.04523455, 4.5463206e-05, -0.022358498, -0.00051796215, 0.013521197, -0.062158693, -0.034909274, 0.0038072846, 0.028595068, -0.0031053468, 0.011832665, -0.006466238, 0.0033932382, 0.04758944, 0.022397313, -0.023717087, 0.005460235, -0.013935244, -0.038092252, -0.043190196, -0.05480937, 0.009044322, -0.018463874, 0.032192096, -0.0463473, 0.001361824, -0.005262916, 0.017545208, 0.037186526, 0.048365775, -0.027016517, 0.0060036704, -0.04694249, -0.013883488, 0.007996268, 0.0189038, 0.014194023, -0.031053469, -0.00013454481, 0.07131946, 0.0025425027, -0.0064080125, -0.021556282, 0.028439801, 0.022138534, -0.03571149, -0.006453299, 0.04526043, -0.03190744, 0.019680135, -0.026382508, -0.044302948, -0.029500794, 0.017803987, 0.0006109608, -0.022397313, 0.02464869, -0.011386272, 0.00093079533, -0.042776152, -0.0020540575, 0.052557994, -0.0020944916, 0.010098847, 0.012796616, -0.05496464, -0.011192188, -0.03371889, 0.019654257, -0.05972617, -0.0014402663, -0.07597748, -0.013456503, -0.0738555, 0.004774471, -0.03180393, 0.062728, -0.029759573, 0.015759636, 0.0409647, 0.037859354, -0.030328887, -0.029759573, -0.006204224, -0.05041013, -0.007989799, 0.04515692, -0.063193806, -0.01899437, 0.029552551, 0.0021915338, -0.04981494, -0.024325216, 0.035219807, 0.0063724304, -0.019783646, -0.008067432, -0.007323443, -0.0013998321, 0.032036826, -0.0075628133, 0.02794812, 0.010377034, -0.024001744, -0.010008274, -0.050254863, 0.00619452, -0.011625642, -0.02587789, -0.04086119, 0.039179124, 0.013948183, 0.02257846, 0.0011911916, -0.033822402, 0.008248577, 0.021620978, 0.017868683, -0.051315855, -0.042232715, 0.05884632, 0.00037967716, -0.06241747, -0.028103389, 0.0046159686, 0.0033156045, 0.0014491618, 0.012304937, -0.020547044, 0.08001444, 0.05558571, -0.009710678, -0.0124019785, -0.04707188, -0.028413923, 0.042025693, 0.057448916, 0.011515661, 0.023652392, -0.004454232, -0.013831732, 0.0512641, -0.03180393, 0.0043216078, 0.004450997, -0.0060554263, -0.024596933, 0.027611708, 0.03296843, 0.018205095, 0.047744706, 0.0010432025, -0.009341918, -0.032554384, -0.011974993, -0.034132935, 0.041301113, 0.052661505, -0.051108833, 0.03633256, -0.06257274, 0.0031182857, 0.06298678, 0.033304844, -0.015539673, 0.040887065, -0.044924017, 0.05972617, 0.03876508, -0.004551274, 0.013999938, 0.031105224, -0.0333566, -0.04171516, -0.017286431, -0.015436161, 0.03961905, -0.02415701, -0.04500165, -0.022539642, -0.048883334, 0.048598677, 0.015384406, 0.01447868, 0.00069951173, -0.022229107, -0.014310473, 0.04148226, 0.013689403, 0.049323257, -0.008507356, 0.029707817, 0.011502722, -0.034650493, 0.028103389, 0.014090511, -0.009937109, -0.012816025, -0.012524898, 0.03690187, -0.018088644, -0.013779976, -0.041637525, 0.0155137945, -0.032295607, -0.0055022864, 0.009607167, 0.02771522, -0.022384375, 0.047770586, -0.04018836, 0.0059260367, -0.01607017, -0.010687568, -0.021349259, 0.03079469, 0.056051508, -0.051005322, -0.006414482, 0.036850113, 0.006466238, -0.034029424, -0.037341796, 0.023613574, -0.013100682, 0.02942316, -0.013779976, -0.048753943, -0.059312124, 0.0022254984, 0.009018444, 0.011347455, -0.016523033, -0.016277192, 0.07307916, -0.0017289665, 0.025502661, -0.0018567386, 0.002561911, -0.029630184, 0.020663494, -0.0018745296, 0.0053146714, -0.01631601, 0.022901932, -0.00656328, 0.012977762, 0.025528539, 0.0017694007, -3.214519e-05, 0.004328077, 0.0051011792, -0.020417655, -0.008733788, -0.0086496845, 0.036565457, 0.008429723, -0.030561788, -0.04171516, -0.05041013, -0.004502753, 0.016134864, -0.030251253, 0.012570185, 0.007090542, -0.056827847, -0.008766135, 0.0029727225, -0.03765233, -0.04231035, 0.015216199, 0.020870518, 0.033330724, 0.031545147, 0.02869858, -0.016251314, 0.0031279898, -0.00907667, -0.04168928, 0.01508681, 0.04526043, 0.029630184, -0.06329732, 0.019783646, -0.030717054, -0.020974029, -0.0074010766, 0.0108428355, -0.026783615, 0.025437966, -0.0019796586, -0.029656062, -0.021517465, -0.018360363, 0.0005988305, 0.006437125, 0.022707848, 0.01140568, -0.032192096, 0.011412149, -0.043630123, -0.001177444, 0.004525396, 0.014892725, 0.0030568258, 0.027508197, -0.01286778, -0.0041501666, 0.0045965603, 0.0043636593, 0.0076986724, -0.034365837, -0.04401829, -0.041896302, 0.00557992, -0.025839074, 0.04474287, 0.0071811145, -0.0022918107, -0.029267894, -0.0023516533, 0.023406552, -0.00027010049, -0.03775584, -0.018567385, -0.04293142, -0.017001774, 0.010454668, -0.023290101, -0.016950017, 0.034676373, 9.896271e-05, 0.0008361793, -0.019072006, -0.003545271, 0.01041585, 0.013560014, -0.0257485, 0.018386241, 0.0010925322, -0.011664459, -0.012324345, -0.02219029, -0.02414407, -0.0036326088, -0.02389823, 0.016393643, -0.014802153, 0.016587727, 0.01669124, 0.004441293, 0.0056510842, -0.023691207, 0.006818824, -0.03296843, -0.0031975368, 0.019356662, 0.0075239963, -0.0029775747, -0.028388046, 0.0038396318, -0.05703487, 0.01225965, 0.05136761, 0.0057545956, -0.014543374, 0.00723934, 0.0123955095, -0.04378539, 0.050694786, -0.017195858, 0.0008822743, -0.006631209, -0.030121863, 0.011593294, 0.03886859, -0.019059066, -0.014556313, 0.018955555, -0.039515536, -0.005492582, -0.043448977, -0.008423253, 0.022979567, 0.025942584, -0.002086405, 0.0030924077, -0.0022335853, -0.023949986, -0.012835434, 0.026628349, -0.03690187, -0.013805854, -0.0028028989, -0.020676434, -0.02221617, 0.0041501666, 0.020999907, 0.02683537, 0.025049796, 0.014297534, -0.056103263, 0.02085758, 0.01336593, 0.01249902, 0.016665362, -0.04779646, 0.0044477624, -0.019343723, -0.002285341, -0.006233337, 0.022035023, 0.021323381, -0.019136699, -0.02893148, 0.02353594, 0.036953628, 0.016600667, -0.0020637617, 0.01507387, -0.004007838, -0.0018405649, -0.007789245, -0.00857852, -0.007860409, -0.031338125, 0.003677895, 0.019589562, 0.03961905, -0.0090119755, -0.09212529, -0.009186651, 0.011056328, -0.034779884, -0.015410284, 0.015720818, -0.019848341, -0.01006003, 0.0049976674, 0.030975834, -0.038325153, -0.06805885, 0.009296631, 0.015811391, 0.029138504, 0.007174645, 0.02330304, 0.011470375, 0.012421387, -0.014297534, -0.016885323, -0.043474853, 0.018916737, -0.039774317, -0.007983329, 0.022138534, -0.02010712, -0.027559953, -0.0050106063, -0.034624618, 0.014310473, -0.016639482, 0.0007496501, 0.0035258625, -0.041119967, 0.025722623, -0.00821623, -0.013754099, -0.0008628659, -0.03798874, -0.020767007, -0.029449038, 0.0052338033, 0.0008082797, 0.0096848, 0.018334486, 0.027741097, -0.028129267, 0.020288266, -0.008352089, -0.053411964, 0.0036487824, 0.011075737, 0.00943896, -0.00014283381, -0.01385761, -0.00037785762, 0.026809493, -0.031622782, 0.045726232, -0.0015736992, -0.0033447172, 0.016768873, -0.016264254, 0.029707817, -0.004318373, -0.029474916, 0.028413923, 0.007789245, 0.005799882, 0.00980772, 0.002178595, -0.02526976, 0.033330724, -0.0068576406, 0.041637525, -0.033278964, 0.030251253, 0.02035296, -0.0024066437, 0.015565551, -0.025230942, 0.00142571, 0.018942615, -0.031726293, -0.0015825947, -0.010732855, -0.038894467, -0.031855684, -0.009678331, -0.03791111, -0.061278842, 0.0049006254, 0.0010496719, -0.0220609, -0.0013933626, -0.018929677, -0.017972194, -0.08487948, -0.0146857025, 0.025127431, -0.008035085, 0.029630184, -0.022371436, 0.020585861, 0.028362168, 0.016134864, -0.064642966, -0.0067735375, -0.008313272, 0.016561849, 0.011929708, 0.042051572, -0.049090356, -0.0020621442, -0.022720788, 0.026550714, 0.02035296, -0.012906598, 0.02686125, 0.022759603, 0.033046067, -0.059156857, -0.017972194, -0.015306772, 0.008604398, 0.025166247, 0.019162577, -0.033304844, -0.031726293, -0.04145638, 0.02477808, 0.016108986, 0.017946316, 0.021672733, 0.01238904, 0.0057545956, 0.005544338, -0.013023048, -0.027818732, 0.03775584, -0.015035054, 0.024131132, 0.024920408, 0.002319306, -0.014012877, -0.004929738, 0.018864982, 0.017713416, 0.0011345837, 0.00570284, -0.003040652, 0.004683898, -0.0008620572, -0.00035157538, -0.0064047775, 0.04083531, 0.009613636, 0.00051553606, 0.0049685547, 0.021517465, 0.020223571, -0.0057643, 0.019123761, 0.034805764, 0.022759603, -0.034754004, -0.03571149, -0.01139921, 0.009141364, -0.02304426, 0.029112626, -0.009050792, -0.014750397, 0.027559953, -0.0033705952, 0.016704177, -0.02414407, 0.044302948, -0.012227303, -0.046890736, 0.023121895, 0.020223571, 0.022539642, 0.0032832574, -0.027818732, -0.02722354, 0.03998134, 0.028388046, 0.018683836, -0.00820976, -0.012175547, 0.015927842, 0.027197663, 0.009393674, 0.044199437, 0.049607914, -0.014737458, 0.026188424, 0.03283904, -0.048365775, -0.02366533, -0.03214034, 0.029707817, -0.04523455, -0.011515661, 8.339554e-05, 0.0028546548, 0.005275855, 0.025476782, -0.005861342, -0.015850207, -0.0024357564, -0.0010051943, 0.01816628, 0.021996206, 0.02942316, -0.004745358, 0.027818732, -0.028750336, -0.03273553, -0.00820976, -0.0072134617, -0.018088644, 0.02488159, 0.0034158814, 0.0044445274, -0.007834531, -0.0035614446, -0.012880719, 0.014621007, -0.0041792793, 0.0017386707, 0.027534075, 0.01704059, 0.046036765, -0.022462009, 0.023730025, -0.03589263, -0.006304501, 0.0065406365, 0.0077439584, 0.0055346335, 0.00067646423, -0.012059096, -0.032580264, -0.004221331, 0.022772543, 0.01669124, 0.011004573, -0.014051694, 0.045545086, 0.026628349, 0.039774317, 0.020702312, -0.0730274, -0.023937048, 0.024493422, 0.013249479, 0.014698641, 0.0023662096, 0.0011758266, 0.008552643, -0.00521116, -0.017765172, -0.00269777, -0.029992474, 0.02463575, -0.028155144, -0.010034151, 0.0112051265, 0.028802091, 0.0037361204, 0.011425088, -0.0013246245, 0.041973937, 0.023277162, 0.026602471, -0.015733756, 0.0564138, -0.0019392243, 0.01630307, 0.033408355, -0.013378869, 0.017273491, -0.0021025785, -0.019084943, 0.0040046033, -0.006446829, -0.015177382, -0.011062798, 0.03348599, -0.01128276, -0.02207384, 0.01152213, 0.02391117, -0.01103692, -0.021012846, -0.049400892, 0.02906087, 0.009574819, -0.03633256, -0.010674629, 0.025929645, 0.01604429, -0.038894467, -0.010124724, -0.0064597684, -0.007866878, -0.009542472, -0.0014321795, 0.026809493, 0.018360363, 0.014401046, 0.031001711, 0.011800318, 0.008856708, -0.029086748, -0.027534075, -0.052687384, 0.0066635567, 0.0069611524, -0.021245748, 0.02488159, 0.002210942, 0.005748126, 0.038790956, -0.0029242015, 0.01630307, 0.010163541, 0.017558148, 0.027145905, 0.026408387, 0.002833629, 0.02415701, 0.0008466922, 0.03371889, 0.013637648, 0.022125596, 0.018179217, -0.008772604, -0.028595068, -0.016173681, -0.010687568, 0.017584026, 0.054912884, -0.006695904, -0.008934341, -0.0240535, 0.024312278, 0.03985195, -0.024972163, 0.0077310195, -0.023471246, 0.03998134, 0.028439801, 0.0041113496, -0.0040175426, 0.034262326, 0.033770647, 0.0053858357, -0.039929584, -0.016548911, -0.009426021, 8.5417254e-05, -0.011800318, -0.00082890113, 0.02415701, -0.033667136, 0.0441218, -0.002807751, -0.00070638553, -0.029630184, -0.022358498, -0.014038756, -0.01604429, 0.0018001307, -0.004806818, -0.012414917, -0.045545086, 0.0126736965, 0.0088308295, 0.016898261, -0.0043992414, -0.02895736, -0.027818732, 0.01679475, -0.012828964, -0.0407318, -0.011088676, -0.0059292717, -0.02722354, -0.03483164, 0.027068272, -0.015526734, -0.011748562, -0.0026395447, -0.011748562, 0.032295607, 0.01716998, -0.011218065, -0.037315916, 0.006818824, -0.036979504, -0.0019634848, -0.06329732, -0.015462039, 0.029086748, -0.02257846, 0.010603465, 0.02635663, 0.02256552, 0.038920347, 0.0017823396, -0.006029548, -0.044044167, 0.010512893, -0.0019311375, 0.008979628, 0.024260521, 0.0025716154, 0.017752232, -0.010409381, -0.022901932, 0.017558148, -0.02243613, 0.015811391, 0.009574819, -0.020340022, 0.014789214, -0.03421057, -0.004425119, -0.024700446, 0.0011232621, -0.0076986724, -0.020042425, 0.0058192904, 0.015358527, 0.0257485, 0.0010108551, 0.00723287, 0.00033216696, 0.005981027, -0.005188517, 0.00772455, -0.028258655, 0.031545147, 0.038997978, 0.042155083, -0.018243913, -0.0063594915, 0.0019731892, 0.002109048, 0.02673186, -0.04769295, 0.016704177, -0.02463575, -0.01594078, -0.01986128, 0.022552582, 3.211992e-05, -0.017467575, 0.0036358435, -0.0024988337, -0.03338248, -0.010816958, 0.0057351873, 0.0032848746, 0.018399179, 0.0102023585, -0.026524836, -0.020210631, 0.022099718, 0.02895736, 0.013534136, 0.0072522787, 0.008824361, 0.030095985, 0.0019812759, -0.03863569, -0.009173712, 0.036720727, -0.0075239963, 0.021168115, 0.010170011, -0.0071617058, -0.04316432, 0.0069611524, 0.04192218, -0.012913067, -0.020767007, -0.0121237915, -0.022863116, 0.014724519, 0.011127492, -0.007090542, -0.03421057, 0.011438027, -0.019706013, -0.0080997795, -0.0020330318, 0.0036229047, 0.009503655, -0.014608068, 0.006514759, 0.034365837, -0.014129328, 0.03201095, -0.013107151, 0.030432397, -0.017092345, 0.00040292682, -0.00084345747, -0.026938884, -0.021672733, 0.020547044, 0.00082809245, -0.0001457653, -0.00032347362, 0.011069267, 0.0023209231, 0.0063950736, 0.008151535, 0.03860981, -0.0036455477, 0.021349259, 0.005495817, -0.014569252, -0.027404685, -0.012900128, -0.012511959, -0.0073751984, -0.0052338033, -0.017920438, 0.014116389, -0.02686125, -0.025890829, 0.026990639, -0.0010011508, 0.023238344, 0.0077957143, 0.05998495, -0.0063336133, -0.02341949, -0.030949956, 0.00285142, 0.049866695, -0.007808653, 0.014012877, -0.009037853, -0.024351094, -0.0010027682, -0.015811391, 0.004033716, -0.026938884, -0.0021074307, -0.0100212125, -0.0017516096, 0.0003932226, -0.0020621442, -0.0017483749, 0.0034320552, -0.0018518865, -0.00309726, 0.03423645, 0.004269852, -0.03040652, -0.028025754, 0.03547859, -0.02758583, 0.004502753, -0.0036973036, 0.012065566, -2.07478e-05, 0.0061880504, -0.01692414, -0.019524869, 0.0003208454, 0.009374266, 0.040628288, 0.015177382, -0.024583995, -0.0021381606, 0.018243913, 0.012602532, -0.014129328, -0.032942552, 0.042465616, 0.018450934, 0.024868652, 0.0076145693, -0.011412149, 0.051212344, 0.008591459, 0.023548879, 0.008889055, -0.019227272, -0.017855745, -0.007892757, -0.03188156, 0.008267986, 0.010946347, 0.0026929178, -0.05791472, -0.038972102, -0.030354764, -0.0053696623, -0.015487917, -0.007879818, 0.018386241, -0.028672703, 0.01226612, 0.03198507, -0.010241175, -0.030613543, 0.016962957, 0.015707878, 0.009775373, -0.004256913, 0.011806787, 0.023924109, -0.009031383, 0.006967622, -0.038273398, 0.01630307, -0.043578368, -0.040033095, 0.019641317, 0.011043389, 0.0133141745, 0.012893658, -0.008669093, -0.023652392, 0.03726416, -0.026887128, 0.0008580138, 0.009147834, -0.022022083, -0.0046192035, -0.038221642, -0.0038590403, 0.008416784, 0.007064664, 0.043707754, -0.01018295, 0.05434357, 0.0076792636, 0.015099749, 0.0050009023, 0.010564649, -0.015720818, -0.027508197, -0.039903708, 0.029190259, -0.0022643153, 0.020676434, -0.0077310195, 0.009613636, -0.029216137, -0.03565973, -0.004243974, -0.007653386, 0.011237473, 0.004829461, -0.0050688316, -0.010247644, -0.003058443, -0.008526765, -0.0040984107, -0.0009890206, 0.002487512, -0.0020767006, -0.0009639514, 0.024066437, -0.018916737, -0.031622782, 0.021724489, 0.001722497, 0.00248266, -0.012971292, -0.012376101, 0.01189736, -0.010144133, 0.006058661, -0.043500733, 0.035607975, 0.011664459, 0.020029487, -0.0063789, 0.032657895, -0.002764082, 0.03472813, -0.02207384, 0.017558148, 0.02121987, -0.0051658736, -0.037186526, 0.036229044, -0.0078280615, -0.0024357564, -0.029604306, -0.015500856, -0.030613543, -0.012156139, 0.010157072, -0.004133993, 0.014388107, -0.013145968, -0.038972102, 0.0010634195, -0.011043389, 0.009613636, 0.0042116265, 0.015358527, -0.01751933, 0.011192188, -0.019136699, -0.00062753883, -0.005964854, -0.009134895, -0.02563205, 0.00030446955, 0.015410284, 0.040291876, 0.011806787, 0.023561819, 0.0010537154, -0.029888963, -0.005091475, 0.009257815, 0.011321576, -0.024739262, 0.014621007, 0.004402476, 0.0019440765, -0.008513826, 0.01741582, 0.006116886, -0.022449069, 0.04461348, 0.019279027, -0.02856919, 0.007931573, 0.004926503, -0.032269727, -0.009374266, 0.007187584, 0.046916615, -0.020922273, -0.01471158, 0.028724458, 0.0090119755, 0.026447203, 0.03591851, -0.01973189, 0.034546982, -0.018179217, -0.013534136, 0.0115286, 0.007808653, -0.0032492925, -0.012751331, 0.0030940252, 0.00039645736, -0.0005021928, 0.010435259, -0.006427421, 0.0109334085, 0.027404685, 0.00011604616, 0.009678331, 0.015216199, 0.067541294, 0.012725452, -0.011729154, 0.03348599, 0.021181053, -0.0145045575, 0.018218035, 0.018554447, 0.033641256, 0.011974993, 0.0019295202, 0.0021316912, -0.028051632, -0.0047000716, 0.03361538, -0.0035840878, -0.025541477, 0.008908464, -0.023949986, -0.044328824, 0.006650618, 0.005324376, 0.01004709, -0.011865012, 0.0055475724, 0.01704059, 0.0035873225, 0.0015615689, -0.03149339, -0.0043248422, 0.011987932, 0.025644988, -0.0002581724, -0.017286431, -0.022371436, -0.001059376, -0.03617729, 0.001704706, 0.0094713075, 0.004793879, 0.0072458093, 0.02120693, 0.0032120931, -0.045803864, -0.037626453, -0.03578912, -0.015682, 0.010409381, 0.028388046, 0.038325153, -0.022966627, -0.03571149, 0.006142764, -0.009794781, 0.0049523814, -0.004243974, 0.036461946, -0.031364, 0.0094713075, -0.013715281, 0.012427856, -0.02156922, 0.0044607013, 0.008889055, 0.0016755933], "content_hash": "a4af56a9da1686fd3a5b7bcead22b62d"}, {"content": "# MLP (Feed-Forward) Layers Explained\n\n## What Are MLP Layers?\n\nAfter attention gathers context from other tokens, each token's representation passes through a **Multi-Layer Perceptron (MLP)**, also called a **Feed-Forward Network (FFN)**. While attention handles relationships between tokens, the MLP processes each token independently -- and this is where much of the model's **factual knowledge** is stored.\n\n## What They Do\n\nThink of the MLP as the model's **memory bank**. During training, the MLP weights learned to encode facts, patterns, and associations from the training data. When the model processes \"The capital of France is,\" the MLP layers help recall that \"Paris\" is the answer.\n\nResearchers have found that specific facts are often stored in specific MLP neurons. This is one of the key findings in mechanistic interpretability research.\n\n## The Expand-Then-Compress Pattern\n\nEach MLP layer follows a distinctive pattern:\n\n1. **Expand**: The token's representation is projected into a much larger space (typically 4x the hidden dimension). For GPT-2, this means going from 768 dimensions to 3072 dimensions.\n2. **Activate**: A non-linear activation function is applied (like GELU or SiLU), which allows the network to represent complex patterns.\n3. **Compress**: The expanded representation is projected back down to the original size (768 for GPT-2).\n\n**Why expand then compress?** The expansion creates space for many individual neurons to each \"vote\" on whether a specific concept or fact is relevant. The compression then combines these votes into a refined representation. Each neuron in the expanded layer can activate for specific concepts.\n\n## Attention + MLP = One Layer\n\nIn each Transformer layer, attention and MLP work together:\n\n1. **Attention** gathers relevant context from other tokens\n2. **MLP** retrieves stored knowledge and transforms the representation\n3. The result is added back to the **residual stream** (the running representation)", "source_file": "mlp_layers_explained.md", "chunk_index": 0, "embedding": [-0.009905266, 0.003195755, -0.0029013676, -0.0052926736, 0.048033927, -0.010440515, 0.02612018, 0.04365118, -0.03924324, 0.090677574, 0.008123199, -0.011882541, -0.031283762, 0.003746747, -0.02047802, -0.049923044, 0.016410122, 0.006382064, 0.039293617, 0.021548519, -0.014306907, -0.023173159, 0.032820243, 0.031233387, 0.019420113, -0.017127987, -0.014634353, 0.029117577, -0.013400131, 0.031686775, 0.012694861, -0.022543453, -0.038336463, -0.019520868, 0.012776722, -0.00026132775, -0.018462962, 0.024382193, -0.013727577, 0.071030766, 0.014571383, -0.038059395, -0.056421604, 0.022908682, 0.025591226, 0.009584116, -0.037631195, 0.008620666, -0.0344071, 0.08513617, -0.020075008, 0.0453136, -0.02406734, 0.030956317, -0.040452275, 0.016976858, -0.015125523, -0.02112032, -0.011699926, 0.006375767, -0.0081924675, -0.0131608425, 0.039772194, 0.025792733, 0.010774259, 0.005119505, -0.07249168, 0.013992053, 0.00093826104, 0.04866363, 0.04375193, -0.007033809, -0.023185752, 0.034356724, 0.014798077, 0.007241612, -0.0007772926, 0.054658428, -0.0069141653, -0.037983827, 0.053953156, -0.028714566, -0.0044079376, -0.05138396, 0.031711962, 0.0034130032, -0.030805187, -0.008362488, -0.05309676, -0.06654727, 0.0024164945, 0.0029958233, -0.05027568, 0.03909211, 0.024822986, 0.022795334, -0.0072101266, 0.006334836, -0.011907729, 0.0027565353, 0.041686498, 0.002468445, -0.028387118, -0.044910587, 0.002139424, -0.052038852, 0.00899849, 0.0006096335, -0.032064598, -0.026170556, -0.067806676, -0.0031469527, 0.030150294, 0.030931128, 0.040175203, -0.012550028, -0.014722512, 0.043248165, 0.008274329, 0.0035106076, -0.03090594, -0.0213722, -0.0012586237, 0.012694861, 0.019357143, -0.027883355, 0.026220933, 0.010327169, 0.032165352, -0.00066591345, 0.005982201, 0.015289247, 0.027832977, -0.042416956, 0.007827238, -0.016636817, -0.02167446, 0.012984525, -0.023248723, 0.021989312, -0.0045118392, -0.028034484, 0.012764128, 0.02229157, -0.03543982, 0.011196162, -0.0043575615, 0.035666514, -0.04644707, -0.014306907, -0.004033263, -0.014722512, -0.05914193, -0.014886236, -0.040678967, -0.045565482, 0.024180686, 0.07485937, 0.06030059, -0.03856316, -0.011945511, 0.03246761, -0.014369877, 0.014029836, -0.021435171, 0.057076495, -0.014470629, -0.0017017788, -0.018727439, -0.0933979, -0.045112096, -0.020075008, -0.04929334, -0.015654476, 0.005226555, -0.02449554, -0.09188661, -0.054809555, -0.029621342, -0.046321128, -0.005012455, 0.00090835005, -0.047202718, -0.0030619425, -0.03317288, 0.0030965763, -0.024294034, 0.04690046, -0.001547501, -0.00015850866, -0.011290617, 0.046371505, 0.058688544, 0.006467074, -0.0025125246, -0.004212729, 0.07541352, -0.01876522, 0.030654058, 0.0065048565, 0.018324427, 0.057630636, 0.020301702, -0.009842295, 0.061761506, 0.031157823, 0.0024149201, -0.0126822665, -0.02259383, -0.034533042, 0.020415049, -0.033575892, 0.018072546, -0.012669672, -0.035590947, -0.038487595, -0.07672331, 0.044255696, 0.00082570117, 0.0235132, 0.033097316, 0.01790882, 0.007827238, 0.01906748, -0.073146574, 0.01327419, 0.045641046, -0.065438986, 0.00981081, -0.0064607775, 0.03201422, 0.0006226212, -0.0041119764, -0.06463296, -0.015377405, -0.05405391, -0.0038852822, -0.0035452414, 0.022241194, -0.013878707, -0.019306768, 0.049066644, -0.0021551666, 0.0073927413, 0.017833257, 0.010755368, -0.014722512, -0.0067126593, 0.05491031, 0.047353845, 0.06095548, 0.012984525, -0.055565204, 0.029848035, -0.022467889, -0.01045311, 0.01821108, 0.020276513, -0.04541435, -0.009244075, -0.003992332, 0.041862816, -0.028588625, -0.027883355, -0.02507487, -0.033575892, -0.028235989, -0.0114102615, -0.025490474, -0.029067202, 0.040427085, 0.0096785715, -0.016183428, 0.009269264, -0.023311693, -0.014684729, 0.025037086, 0.023676923, 0.012442978, 0.019697184, -0.03753044, -0.027681848, 0.081660196, 0.038613535, -0.02753072, -0.04148499, -0.00938261, -0.015553723, -0.0034224489, -0.028336743, 0.021687053, -0.026321685, 0.034331538, 0.005176178, 0.01171252, -0.0033846665, -0.0007871317, 0.021334419, 0.038840227, -0.0025518811, 0.0059160823, -0.0044362745, -0.010528674, -0.02363914, -0.018173298, 0.0058783, -0.025163027, 0.010843527, -0.0050344947, -0.0057492102, -0.022404918, -0.035112374, 0.032845434, -0.016183428, 0.016611628, 0.051585466, 0.011636956, -0.024999304, -0.0110387355, -0.0039419555, -0.008419161, -0.022845712, -0.004319779, -0.014848453, 0.022820523, 0.01045311, -0.035842832, 0.017581375, -0.010742774, -0.006769333, 0.03344995, -0.00803504, -0.023689516, 0.0020512652, -0.04506172, -0.0136016365, -0.037253372, 0.07581653, 0.015578912, 0.010320871, -0.029142765, -0.01671238, -0.030981505, -0.015037364, -0.06619463, 0.046623386, 0.027379591, -0.00525804, 0.013324566, -0.0020685822, -0.037177805, 0.00829322, 0.023865834, -0.030125106, -0.016599035, 0.033147693, 0.013866113, 0.00043095462, 0.0006108142, -0.008992193, -0.05025049, -0.027278837, 0.03430635, 0.0029234074, -0.04473427, -0.0010468852, 0.018966727, 0.035187937, -0.023387259, -0.004181244, 0.0041843923, 0.019697184, 0.045565482, 0.017921416, -0.0035043105, 0.0051478418, -0.033651456, -0.004234769, -0.01622121, -0.00506598, 0.0038286087, 0.04105679, 0.022921275, 0.007915397, 0.011951808, 0.03299656, 0.00882847, -0.030452553, -0.06977136, 0.01750581, 0.0100312065, -0.02556604, -0.014180966, 0.018085139, 0.007978368, -0.008941816, 0.005903488, 0.0054501, 0.049998607, 0.06292016, 0.01989869, -0.0053524957, -0.034784924, -0.013803142, 0.031938657, 0.027278837, 0.022606423, -0.010037504, 0.027127707, 0.038764663, 0.047958363, -0.017732503, 0.006224638, 0.0021095132, 0.0053871297, 0.032845434, 0.04035152, 0.012424087, 0.018034762, 0.009672275, 0.0136897955, 0.0014034559, -0.0037373016, -0.00052541046, -0.027782602, 0.028160425, 0.052139606, -0.06317204, -0.03299656, -0.012178502, -0.005831072, 0.026749885, 0.020037225, -0.017379869, 0.027429966, -0.030578494, 0.015276653, 0.05889005, -0.019697184, -0.041686498, 0.040754534, -0.028059673, -0.02274496, -0.056421604, -0.036623664, 0.017266521, -0.041988757, -0.06815931, -0.028865695, 0.00393251, 0.0017773435, 0.029848035, 0.038109772, 0.04486021, 0.012014778, -0.008305814, 0.06508635, 0.044809837, 0.07229018, -0.037681572, 0.029948788, -0.010812041, -0.007852426, -0.02535194, -0.018085139, -0.0031264871, 0.032719493, -0.036825173, 0.030326612, 0.013803142, 0.019999443, -0.021271449, 0.0042819967, -0.01897932, -0.002504653, 0.00807912, 0.0026463368, 0.020956594, 0.00998083, -0.06140887, 0.02406734, 0.017619157, -0.033701833, -0.026850637, -0.026749885, 0.07067813, -0.020528395, 0.01246187, 0.013714983, -0.012852287, -0.003979738, -0.016624222, 0.022354541, -0.013261596, 0.028361931, -0.047101963, -0.05420504, -0.06815931, 0.0072605032, -0.013488289, 0.007147156, -0.008532508, -0.009955642, 0.05224036, -0.019634213, 0.022430105, 0.025717167, 0.035792455, -0.018072546, 0.035666514, -0.015276653, 0.012342226, 0.0003160334, 0.02520081, -0.010264197, 0.015112929, 0.041736875, 0.022190819, 0.00981081, 0.009508551, 0.017102798, 0.020377267, -0.0344071, 0.019659402, 0.028336743, 0.029193142, -0.02528897, -0.030855564, -0.057630636, -0.028991636, 0.0077201882, -0.025603822, -0.0022275827, 0.006567827, -0.021926342, 0.004105679, -0.01919342, -0.04737903, -0.052139606, -0.026246121, 0.0037561927, 0.022153035, 0.02413031, 0.022644205, 0.0023629696, -0.016536063, 0.019117856, -0.041006416, 0.017329492, -0.0014003074, 0.00959671, -0.0064733713, 0.016019706, -0.02188856, -0.025087463, 0.0040553026, -0.017770287, -0.007934288, 0.011233944, -0.02541491, 0.0059727556, -0.0064198463, 0.0038286087, 0.0049809697, -0.042139884, 0.031686775, 0.010761665, -0.0029328528, 0.01947049, -0.04548992, 0.01579301, -0.007140859, 0.018777814, 0.0005671284, 0.0074809, -0.00082727545, 0.007386444, 0.01701464, -0.010106771, 0.00018281135, -0.031384517, -0.0127893165, -0.029747283, -0.0404019, -0.02130923, -0.008834766, -0.011731411, -0.029016824, -0.038487595, 0.0080161495, -0.011215053, 0.00042387043, 0.01186365, 0.017593969, -0.026372062, -0.003919916, 0.015830794, -0.017581375, -0.02336207, 0.008406567, -0.0055602985, 0.013186031, -0.010994656, 0.0004093085, 0.012191096, 0.032946184, -0.018840786, -0.0032366856, 0.007814644, -0.013349754, -0.013538666, -0.040678967, -0.004080491, -0.023462823, -0.033525515, -0.0028809023, -0.023979181, 0.0038034206, 0.014848453, 0.006630798, 0.006687471, 0.010037504, -0.0047290875, -0.009080351, 0.010037504, 0.014609165, -0.006410401, -0.011775491, -0.012027373, 0.021913748, -0.024545915, 0.0047574244, 0.044784646, 0.020654337, -0.007443118, 0.0059601613, 2.3220386e-05, -0.02639725, 0.02443257, -0.033122502, 0.000116495496, 0.0013192327, -0.024608886, 0.003913619, -0.007902803, 0.0072919885, 0.0042032837, 0.015654476, -0.01267597, -0.04065378, -0.0060325777, -0.02152333, 0.01897932, 0.013664607, -0.01771991, -0.010881309, -0.015578912, 0.0049211476, -0.028387118, 0.018790409, -0.03178753, -0.008545103, -0.015541129, -0.0457418, -0.020792872, 0.03669923, 0.015364812, 0.013412725, 0.032543175, 0.0051604356, -0.07289469, 0.019911284, -0.031938657, 0.040628593, 0.009495957, -0.05055275, -0.0037184104, -0.029142765, -0.0035043105, 0.032543175, 0.031157823, 0.015906358, -0.040578216, 0.0037687868, -0.00333429, -0.016384935, -0.028815318, -0.025742356, 0.024634074, -0.0049557816, -0.017883632, -0.017065017, 0.04206432, -0.0344071, -0.020024631, -0.0029958233, 0.013576448, 0.035087183, -0.01814811, -0.050653502, -0.0062718657, -0.010371247, -0.08024965, 0.010698695, -0.016976858, -0.031409703, -0.04543954, 0.061610375, 0.009325936, -0.05460805, -0.04400381, -0.001021697, 0.028487872, 0.022203412, 0.01023901, 0.0064072525, -0.012713752, -0.0066056093, -0.0061081424, -0.017984387, -0.0063064992, 0.0060861027, -0.030931128, 0.002599109, 0.036497723, -0.021321824, -0.05662311, -0.007940585, -0.01836221, 0.037782323, -0.005125802, 0.0047574244, 0.0070778886, -0.014042431, 0.00033984412, 0.0019442154, -0.009477066, -0.013198625, -0.055917837, -0.024231063, 0.005336753, -0.0074557117, -0.029167954, 0.013463101, 0.024243657, 0.039016545, -0.00012151346, 0.007329771, -0.008582884, -0.037681572, -0.029596154, 0.009615601, 0.0152640585, 0.002772278, -0.029319083, 0.010113069, 0.027329214, -0.016951669, 0.024684452, -0.0022212858, 0.0024778908, 0.022480482, -0.03516275, 0.012707455, 0.009634492, -0.04193838, 0.0062655685, -0.0060986965, 0.0215863, 0.009546334, 0.001726967, -0.00022197115, 0.01105133, 0.0038632425, 0.01784585, 0.0009406224, 0.040250767, -0.01160547, -0.008652152, -0.012707455, 0.0039419555, 0.0047763153, -0.0045748097, -0.03528869, -0.02075509, 0.014760294, -0.041258298, -0.026523191, -0.010075286, -0.037429687, -0.024016963, -0.0068323035, 0.025591226, -0.040250767, -0.007852426, 0.040301144, -0.0061207362, -0.09208812, -0.034810115, 0.008790688, -0.034079656, -0.015994517, -0.02387843, 0.03936918, 0.024961522, 0.005768101, -0.06548936, 0.0365481, -0.03531388, -0.01229185, -0.0032429828, -0.0005734255, -0.017770287, -0.01722874, 0.0075564645, -0.0009768305, -0.03261874, -0.034356724, -0.014596571, 0.01013196, -0.0027266243, -0.023261316, 0.011737708, -0.017606562, -0.014583977, -0.005396575, 0.014810671, -0.016057488, 0.023739893, -0.010629427, 0.01019493, 0.0048676226, 0.00677563, -0.0065111537, 0.01100725, 0.028941259, -0.0033500327, 0.002197672, -0.022417512, 0.0030572196, -0.04584255, 0.013941677, -0.024671856, 0.010213821, -0.034533042, 0.015100335, 0.02161149, -0.015654476, -0.03221573, 0.0008634835, -0.0016860361, 0.02118329, 0.008223953, -0.033827774, -0.0071912357, 0.0399737, 0.04742941, 0.0123296315, 0.04334892, 0.012361117, 0.0069960267, -0.0060199834, 0.0037026678, 0.03214016, 0.034507856, -0.0074179294, -0.028664188, 0.0038506484, -0.023097593, 0.031208199, -0.0043040365, 0.015427782, -0.020137979, -0.0018167001, 0.04682489, -0.018261457, 0.0020449683, 0.045137282, -0.018651873, -0.0060798055, 0.013085278, 0.036321405, 0.00013273009, 0.007115671, -0.05027568, 0.011548797, 0.050729066, 0.036774796, 0.018677061, -0.014105401, -0.0011193013, -0.0042410656, 0.0052391486, 0.013929083, 0.07098039, 0.04639669, -0.020591367, 0.066295385, 0.02486077, -0.0448854, -0.050653502, -0.056119345, 0.0047479784, 0.00639151, -0.00094140955, -0.008557696, 4.885136e-05, 0.02379027, -0.004011223, 0.023752486, 0.02090622, -0.011945511, 0.0070967795, 0.019457897, 0.018941538, -0.011888837, 0.01698945, -0.0024952076, -0.0077831587, 0.0075564645, 0.0023078702, -0.012209987, -0.036951113, -0.016460499, 0.055968214, 0.01181957, -0.02455851, -0.010912795, 0.013500883, -0.013929083, -0.0027738523, -0.01827405, 0.032114975, 0.005169881, 0.04148499, -0.013148248, 0.023336882, -0.009137025, -0.020994378, 0.0043764524, 0.017694721, -0.032291293, 0.019495679, 0.0039954805, -0.0019457897, -0.0017521553, 0.02109513, 0.014193559, 0.012839693, 0.003274468, 0.016775351, 0.0057460614, 0.013777954, -0.0156166935, 0.0008013001, 0.009042569, 0.037253372, -0.0008516765, 0.03624584, 0.015037364, -0.013727577, 0.024117716, 0.013349754, -0.0029564667, -0.008393973, -0.023840645, 0.004599998, -0.021775212, 0.004045857, 0.04276959, 0.020352079, 0.023286505, -0.005730319, -0.009124431, 0.041988757, 0.01083723, 0.029999165, -0.008790688, 0.04332373, 0.0123988995, 0.01263189, -0.0035893207, -0.009823404, -0.007367553, 0.019860908, -0.023374664, 0.006117588, -0.0014380896, 0.013840924, -0.01355126, -0.008022446, -0.00910554, 0.001021697, -0.01855112, -0.004011223, -0.015906358, -0.0036900735, 0.0011507866, 0.024835581, 0.006763036, -0.03299656, -0.023387259, 0.021195883, 1.9530706e-05, -0.0067063626, 0.0012916832, 0.039898135, 0.008022446, 0.005875151, -0.030553306, 0.015906358, -0.053046383, 0.00060727214, -0.0061585186, -0.008929223, 0.009401501, -0.015855981, -0.031837903, -0.047933172, 0.004930593, -0.016674599, -0.0060986965, 0.00048015037, -0.016636817, 0.012505949, 0.01147953, -0.019546054, -0.0084758345, 0.027933732, -0.002748664, 0.043147415, 0.008324705, 0.0049274447, 0.009779325, 0.0031737152, 0.0378327, -0.020893624, 0.020427642, 0.01855112, 0.010264197, -0.0046818596, -0.0077516735, -0.01126543, 0.0033594782, 0.006174261, -0.004200135, -0.031686775, -0.025943862, 0.0369763, -0.0022291571, -0.039192863, 0.0022039688, -0.0049274447, 0.026044615, 0.020276513, 0.00089654303, 0.03304694, 0.018865973, 0.017379869, 0.05289525, -0.04206432, 0.016687192, -0.01756878, -0.01989869, -0.010541269, 0.010169742, 0.015893765, -0.018160703, 0.041711684, -0.0020969189, 0.0077579706, 0.01622121, 0.017871039, 0.0002778575, -0.015918951, 0.012128126, -0.029822847, -0.0035641324, -0.013878707, 0.015125523, -0.009225184, 0.019294173, 0.030402176, -0.0027707038, -0.031409703, -0.008154685, -0.023916211, -0.007247909, -1.6160799e-05, 0.027631473, -0.0038191632, -0.001224777, 0.013941677, 0.02406734, 0.004634632, -0.01355126, -0.050200112, 0.029067202, -0.006637095, 0.0027738523, 0.014294312, -0.024268845, 0.014596571, -0.0065111537, -0.03924324, -0.0021315527, -0.008375081, -0.032291293, -0.006870086, 0.012606702, 0.028210802, 0.008456944, 0.017556187, 0.009823404, -0.07269319, 0.021699648, -0.028664188, 0.024797799, 0.04954522, 0.0011610193, 0.022090064, -0.00040812782, -0.011687332, 0.016674599, -0.0042379173, -0.0033720722, 0.018198486, 0.01105133, 0.011183567, -0.008948114, 0.016762758, -0.038789853, -0.013022307, -0.0057806955, 0.0017316898, -0.006901571, -0.029570965, 0.027253648, -0.008872549, 0.0048928107, -0.0049148505, 0.0061207362, 0.019243797, -0.0058908937, -0.018840786, 0.019432709, 0.0006430866, 0.05067869, -0.032517985, 0.0017017788, -0.00412457, 0.005957013, 0.040981226, -0.041837625, 0.024168093, -0.037026677, -0.031686775, -0.003759341, 0.0222286, -0.0006112078, -0.03672442, 0.012688563, 0.011744006, -0.026019426, 0.004011223, 0.01115838, 0.034054466, -0.014029836, 0.009817107, 0.0017175215, 0.001171252, -0.0076446235, 0.025125245, -0.0029895264, 0.0012901089, 0.004477205, -0.041711684, 0.02152333, -0.047202718, 0.0037750837, 0.039218053, 0.0029911005, 0.029394647, 0.02066693, -0.008274329, -0.040880475, 0.030729624, -0.006637095, -0.019583838, -0.005138396, -0.0024149201, -0.033399574, -0.0073045823, -0.023828052, -0.030402176, -0.013463101, 0.009968236, 0.02336207, -0.004099382, 0.0024652965, -0.01079315, 0.018034762, -0.044406824, -0.012625593, 0.032089785, -0.006681174, -0.007915397, -0.0050596828, 0.026346873, 0.0050156033, 0.0050691282, -0.006630798, -0.034633797, -0.0048361374, 0.02556604, 0.006542639, 0.024105122, 0.00272505, -0.023689516, -0.007997259, 0.007033809, -0.0018513339, 0.017556187, 0.015969329, -0.00046598198, 0.01658644, -0.0015530109, -0.04148499, -0.007115671, -0.0062183407, -0.0002560146, 0.0056075263, -0.026749885, 0.021913748, -0.038311277, -0.028160425, 0.04176206, 0.0030524968, 0.014193559, 0.019671997, 0.022430105, 0.018803002, -0.0069582444, -0.025704574, -0.013173437, 0.0021756322, -0.00068677246, 0.003327993, -0.013639418, -0.016208617, 0.006574124, -0.022493076, 0.0030210116, -0.041434616, -0.006769333, 0.024193281, 0.027757414, -0.0031091704, -0.023034623, 0.026447626, -0.019495679, 0.02498671, 0.014382471, 0.00033708918, 0.05632085, 0.0004726726, -0.03994851, 0.012959337, -0.03712743, 0.033248443, -0.010931686, 0.0115425, 0.012310741, 0.019483084, -0.006095548, -0.034079656, 0.0075123855, 0.0054469514, 0.042845156, 0.035616137, -0.041107167, -0.0016954817, -0.011347291, 0.0045685126, 0.010963171, -0.011397667, 0.011278024, 0.008608073, 0.004417383, 0.01115838, 0.009458175, 0.055414073, 0.028739754, 0.020780278, 0.013576448, -0.030452553, 0.023273911, -0.016271587, -0.02066693, 0.004420532, 0.016145647, 0.014231342, -0.031308953, -0.020150572, -0.015138118, 0.0064324406, -0.0294954, 0.035842832, 0.003894728, 0.0048928107, 0.005097465, 0.010774259, 0.0018182743, -0.0026636538, 0.022417512, 0.014168371, -0.005418615, -0.016145647, -0.026145367, -0.028764943, 0.015704852, -0.014810671, -0.02011279, 0.013400131, 0.0020717308, -0.01622121, 0.00916851, 0.014495818, -0.008494725, -0.02293387, -0.02753072, -0.007134562, 0.0029863778, -0.011296915, 0.011133191, -0.0023346327, -0.0054626944, 0.0014845304, -0.029142765, -0.021926342, 0.002235454, 0.0033028047, 0.047907986, -0.016170835, 0.04629594, 0.06251715, -0.0001902891, -0.0012586237, -0.00037979108, -0.036271032, -0.04105679, -0.018601498, 0.03347514, -0.0007182577, -0.0020937703, 0.011788085, -0.00026467306, -0.010358654, 0.0059066364, -0.026498003, -0.03221573, 0.016233806, 0.0064198463, 0.010919091, 0.004817246, -0.015150712, -0.007619435, -0.008098011, 0.0016624222, 0.015868576, -0.022241194, -0.0067504416, -0.0060231322, -0.017682128, -0.022064878, 0.0127011575, -0.027379591, 0.0071786414, -0.024306629, 0.00835619, 0.015969329, 0.02139739, 0.011599173, 0.02096919, -0.019495679, 0.012801911, 0.013639418, 0.021296637, 0.056975745, 0.01545297, 0.015755229, -0.011271726, 0.017392462, -0.0013507181, 0.022430105, -0.062315647, 0.017127987, 0.01100725, 0.006035726, 0.00081940414, -0.0119707, -0.014306907, -0.016561251, 0.007732782, 0.00463778, 0.027883355, -0.005610675, -0.027329214, 0.0003849074, -0.008110605, -0.007990961, -0.028387118, 0.024722233, -0.0048865136, -0.017530998, -0.0072605032, 0.020843249, -0.012971931, -0.025616415, -0.0016057488, -0.0119707, -0.009495957, -0.00058641314, -0.019331956, 0.00033374387, 0.03642216, -0.0149618, -0.025339345, -0.0060231322, -0.0058594085, -0.01855112, 0.03105707, -0.016838321, 0.025817921, 0.00097132055, -0.0059790527, -0.023374664, 0.0050030095, 0.017266521, -0.006328539, -0.028664188, 0.008960708, -0.028437495, -0.020062413, -0.04065378, 0.00987378, 0.049923044, -0.016951669, 0.018311832, 0.025717167, 0.039721817, 0.029167954, 0.042517707, -0.04332373, 0.04586774, -0.0031752894, -0.027077332, 0.004256808, -0.014269124, -0.015188494, 0.0034980134, 0.009092946, -0.0218004, 0.0011311083, -0.025704574, 0.01051608, 0.031233387, 0.036019146, -0.0111268945, 0.0009579393, -0.014986988, 0.016674599, -0.021485548, 0.0316364, 0.0079091, 0.028840506, -0.020188354, 0.0057869926, 0.02342504, 0.031989034, 0.023475416, 0.015843388, 0.040124826, -0.004367007, 0.032366857, 0.027858166, -0.020276513, -0.06392769, -0.0102075245, -0.025603822, 0.017253928, -0.0077579706, -0.00899849, -0.036648855, -0.025213404, 0.0031359328, 0.014722512, 0.0098359985, 0.027203273, -0.0431726, -0.020452831, -0.016888699, 0.012317037, 0.00032449505, -0.001069712, -0.02413031, -0.006574124, -0.029570965, -0.0049683754, 0.02259383, -0.00040497928, 0.022417512, -0.0035357957, -0.011032439, -0.06125774, -0.018110327, 0.0070841857, -0.015150712, 0.02596905, 0.042013943, 0.031082258, -0.024835581, 0.0033972606, -0.025553444, -0.01673757, 0.02081806, -0.013614231, -0.002192949, -0.026623944, -0.019206014, -0.018865973, 0.026875826, 0.008286923, -0.014395066, -0.0049998607, 0.04203913], "content_hash": "904327181f4c28aa82a6a95107903e6a"}, {"content": "activate for specific concepts. ## Attention + MLP = One Layer In each Transformer layer, attention and MLP work together: 1. **Attention** gathers relevant context from other tokens 2. **MLP** retrieves stored knowledge and transforms the representation 3. The result is added back to the **residual stream** (the running representation)\n\nThis happens in every layer. GPT-2 has 12 such layers; each one further refines the model's understanding.\n\n## What You See in the Dashboard\n\nIn **Stage 4 (MLP/Feed-Forward)** of the pipeline, you can see:\n\n- The expand-compress flow: Input dimension \u2192 Expanded dimension \u2192 Output dimension\n- The number of layers in the model\n- An explanation of why the expansion matters for knowledge storage", "source_file": "mlp_layers_explained.md", "chunk_index": 1, "embedding": [-0.017636668, 0.022921301, 0.02181344, -0.011632561, 0.036750488, -0.002424246, -0.005262349, 0.05251525, -0.012606716, 0.07762681, 0.037998427, 0.0013935834, -0.04413624, -0.024538526, 0.012664019, -0.035145998, -0.012867765, 0.016490603, 0.021928046, 0.04051977, -0.009181255, -0.04497669, 0.04021415, 0.053126484, 0.012956902, 0.013650909, 0.0066280765, 0.045129497, 0.010779379, 0.022042653, 0.0063383766, -0.03927183, -0.051623866, -0.008041557, 0.028142266, -0.022590216, -0.018438915, 0.05577517, -0.00985616, 0.06652271, 0.011759901, -0.014007462, -0.054196145, 0.046886798, 0.008264403, 0.017814945, -0.0504778, 0.0011030877, -0.03792202, 0.06677739, -0.0057812617, 0.028575223, -0.022704823, 0.008347174, -0.023507068, 0.011868141, -0.015204464, -0.02558272, -0.005211413, 0.0058099134, -0.005969089, -0.0031055182, 0.039348233, 0.017942285, -0.012211961, -0.0056762057, -0.070801355, 0.03687783, -0.01836251, 0.038737, 0.01823517, 0.008296238, -0.042913772, 0.034916785, 0.022577483, 0.020743778, -0.011893609, 0.0440089, -0.023239654, -0.012886865, 0.036775954, -0.05979913, -0.00033048858, -0.059442576, 0.006147366, -0.014529559, -0.011957279, -0.008882005, -0.037947487, -0.061327215, 0.02018348, 0.014058399, -0.03743813, 0.01896101, 0.00029527093, 0.027327286, 0.025875604, -0.0051604765, -0.032217164, 0.05210776, 0.041945983, -0.004183138, -0.0014079091, -0.04347407, 0.00045763017, -0.051139973, -0.0093786325, 0.01613405, -0.03201342, -0.04563886, -0.0850635, -0.02005614, 0.0246404, 0.01848985, 0.040163215, -0.010021702, -0.021074863, 0.0049694655, 0.0013418512, 0.009308595, -0.029492075, -0.00981159, -0.003399993, 0.0077486737, -0.016210454, 0.02253928, 0.032166228, 0.01626139, 0.03580817, -0.002548403, -0.01829884, 0.019152021, 0.064332455, -0.046530243, 0.004905795, 0.004737069, -0.018757265, -0.00022264702, -0.026817923, 0.009607846, 0.019877862, -0.03211529, 0.033770718, 0.033694316, -0.04385609, 0.006793619, -0.029441139, 0.04861863, -0.034509294, 0.007634067, -0.016375996, -0.0003748588, -0.026359497, -0.03351604, -0.04765084, -0.026614178, -0.004062164, 0.06845829, 0.053686783, -0.04156396, -0.0073793856, 0.025939273, 0.0034127273, 0.004889878, -0.046632115, 0.06458714, -0.026435902, -0.013383494, -0.03685236, -0.09387547, -0.02395276, -0.006357478, -0.06438339, -0.024029166, 0.027174477, -0.005017218, -0.08241481, -0.04818567, -0.023366993, -0.038533255, -0.016363263, -0.011218704, -0.03858419, 0.0002011583, -0.040978193, 0.017369254, -0.0026248074, 0.033465102, 0.015701093, 0.015471878, -0.0020151644, 0.03779468, 0.0346621, 0.02273029, 0.004351864, -0.022615684, 0.062447812, -0.005392873, 0.02238647, 0.0057908124, 0.02297224, 0.04276096, 0.039297298, -0.04411077, 0.044339985, 0.039602917, -0.012409338, 0.0087992335, -0.03010331, -0.017827678, 0.02335426, -0.03914449, 0.016121315, 0.007182008, -0.019508574, -0.030918289, -0.08822154, 0.02011981, 0.0068318215, 0.0021170368, 0.026843391, 0.010511964, 0.007831445, 0.021074863, -0.06677739, 0.026537774, 0.043219388, -0.05358491, 0.009136686, -7.846865e-06, 0.030740013, 0.020667374, -0.011861774, -0.02043816, 0.004803923, -0.056793895, 0.016541539, -0.02517523, 0.024220176, -0.032319035, 0.00012087405, 0.03927183, -0.02281943, 0.01062657, 0.026410434, 0.039984938, -0.012059152, -0.008423579, 0.0428119, 0.024691336, 0.029721288, -0.004100366, -0.06412871, 0.004791189, -0.018528052, 0.009480505, 0.022361003, 0.0070737684, -0.032879334, -0.012734056, 0.03619019, 0.031274844, -0.04360141, -0.023443399, -0.015802965, -0.018795468, -0.016834423, -0.0041704034, -0.0044410024, -0.034229144, 0.023405196, 0.009499607, 0.0054024234, 0.01857899, -0.027658371, 9.737077e-06, 0.031682335, 0.03779468, 0.01600671, 0.04754897, -0.0022364187, -0.039042618, 0.0610216, 0.047472563, -0.020731045, -0.012071886, 0.0046606646, -0.0042213397, -0.008544552, -0.032777462, 0.04184411, 0.0041385684, 0.044212643, -0.017038168, 0.026334029, -0.0032233081, -0.013727313, 0.0044346354, 0.038966212, 0.0087992335, -0.002123404, -0.020743778, -0.014593229, -0.030740013, -0.0013689111, 0.036903296, -0.0033331392, 0.011231438, 0.012059152, -0.0077232053, 0.0078250775, -0.02564639, 0.0504778, 0.006596241, 0.036674082, 0.01990333, 0.020107076, -0.0091939885, -0.009143053, -0.015586485, -0.013358026, -0.02577373, -0.011562523, -0.05251525, 0.020221682, -0.016210454, -0.03494225, 0.022513812, 0.0066535445, -0.005523397, 0.041513026, -0.02821867, -0.041385684, 0.009034813, -0.03766734, 0.008671893, -0.011689864, 0.044365454, -0.034509294, 0.012574881, -0.035731763, -0.01675802, -0.05327929, -0.010110841, -0.05282087, 0.04497669, 0.028906308, 0.0016283675, 0.050859824, 0.0016379181, -0.0094041005, -0.036775954, 0.0038234005, 0.0007835424, 0.010244548, 0.038635127, 0.026690584, -0.0071438057, -0.0003760526, 0.0010911495, -0.059595384, -0.0076913703, 0.037743744, 0.030740013, -0.034891315, 0.04130928, 0.002175932, -0.0026184404, -0.007933318, -0.017942285, 0.03371978, -0.0045046727, 0.025952008, 0.017165508, -0.036826894, -0.0015153527, -0.01566289, -0.008684627, -0.025366241, 0.0046383804, -0.0017175059, 0.020692842, -0.01343443, 0.029619416, 0.009111217, 0.015102591, -0.008245301, -0.024372984, -0.064281516, 0.027098073, 0.018196966, -0.031758737, -0.020769246, -0.0018177866, 0.01962318, -0.009875261, 0.021304077, -0.0019610447, 0.066217095, 0.041411154, -0.025659123, -0.010620203, -0.046784922, -0.031071099, 0.023048643, 0.016121315, 0.029339267, -0.018247902, 0.03494225, 0.015408209, 0.055316743, -0.016426934, 0.0038170335, -0.0005869604, -0.009232191, 0.012797727, 0.02297224, 0.035400677, 0.023366993, 0.035935506, -0.011982747, 0.030485332, -0.014771506, -0.0040653474, -0.029772224, 0.026512306, 0.060410365, -0.04668305, -0.019992469, 0.0075449287, -0.012861397, 0.058474787, 0.0068636565, -0.01628686, 0.029110054, -0.024627665, 0.01356177, 0.021609694, -0.016044911, 0.00510954, 0.049026117, -0.029059118, -0.0073793856, -0.066166155, -0.052413378, 0.006978263, -0.040316023, -0.05979913, -0.027607435, -0.012543046, -0.006456167, 0.040341493, 0.019610448, 0.026716052, -0.051063567, -0.052566186, 0.04793099, 0.03254825, 0.06458714, -0.0061314483, 0.03468757, 0.0042213397, -0.014682367, -0.020883853, -0.02219546, 0.013536302, 0.013141546, -0.02740369, 0.01776401, 0.0055870675, 0.016184986, -0.032344505, 0.026232157, 0.00020871915, -0.0054851947, -0.001671345, 0.009219457, -0.010053538, 0.0063001746, -0.047166947, 0.0152554, 0.033465102, -0.018795468, -0.0017748092, -0.0093786325, 0.118987024, -0.045180432, -0.0037374457, 0.034916785, -0.020641906, -0.021329544, -0.014173005, 0.0111550335, -0.013956526, 0.030332522, -0.03428008, -0.052566186, -0.06198939, 0.001742974, 0.0015336579, 0.008678259, 0.008761031, -0.019202957, 0.049051587, 0.02545538, -0.0022857632, 0.041869577, 0.032344505, -0.00018862322, 0.020196214, -0.04209879, -0.006475268, 0.007996988, 0.009015712, -0.018948276, 0.008945675, 0.04591901, 0.022131791, 0.0057812617, 0.027505562, -0.006029576, 0.01619772, -0.021673365, 0.025009688, 0.006806353, -0.008187998, -0.021762503, -0.054858316, -0.06005381, -0.003559169, -0.005555232, -0.044645604, 0.013268887, -0.00047553744, -0.042735495, -0.00874193, -0.012307466, -0.032471843, -0.048924245, -0.00668538, 0.0066408105, 0.027047137, 0.029492075, 0.009932565, 0.0060072914, -0.013663643, 0.019292096, -0.013052408, 0.02599021, 0.038125765, 0.019928798, -0.009938931, 0.012415705, -0.014122069, -0.030841885, -0.003801116, -0.0070801354, -0.016936295, -0.0013291172, -0.014682367, 0.0007127092, 0.007977886, 0.012625817, -0.04413624, -0.023659877, 0.05424708, 0.010244548, -0.0038106665, -0.02376175, -0.034229144, -0.008047923, 0.0018878239, 0.0039284565, 0.02740369, 0.0016267757, -0.008347174, 0.015191729, 0.016363263, -0.006558039, -0.032140758, -0.019877862, -0.039195426, -0.025888337, -0.020960256, -0.013230685, 0.004310478, -0.0131033445, -0.028116798, -0.024895081, 0.0026168486, 0.003597371, -0.012644919, 0.014071133, 0.023099579, -0.023277856, -0.011116832, 0.01754753, -0.0047625373, -0.027913053, -0.015535549, 0.010410091, -0.016847158, -0.020234415, 0.004370965, -0.0031660048, 0.034763973, -0.038380448, 0.015981242, 0.0077677746, -0.014134803, -0.0029208742, -0.04133475, -0.019508574, -0.023277856, -0.057354193, -0.011715332, -0.020425428, 0.013447164, 0.029746756, -0.015089857, -0.012040051, 0.015357273, -0.002548403, -0.023023175, 0.013625441, 0.020399958, -0.028269606, -0.011791737, -0.032166228, 0.024003698, -0.029237393, 0.0034031766, 0.060104746, 0.010645672, -0.006526204, 0.018005956, -0.0061951187, -0.017305583, 0.023940027, -0.037310787, 0.0073666517, -0.006672646, -0.015102591, -0.00272668, 0.016146785, 0.008105227, -0.011046794, 0.015586485, -0.0047529866, -0.02686886, -0.034763973, -0.009677883, 0.016732551, 0.006309725, -0.0105629, -0.009034813, -0.020922055, 0.005249615, -0.0040112277, 0.0127085885, -0.038507786, -0.027709307, -0.015981242, -0.064943686, -0.023341525, 0.053432103, 0.031071099, 0.0123202, 0.025939273, 0.00369606, -0.05939164, 0.03970479, -0.0170127, 0.03145312, 0.019814191, -0.031478588, -0.0048707766, -0.02530257, -0.016821688, 0.02153329, 0.006147366, 0.018375244, -0.046810392, 0.019877862, 0.0075576627, 0.0053833225, -0.009525075, -0.032599185, 0.008512717, 0.019928798, -0.038635127, -0.013319823, 0.016299592, -0.037107043, -0.033974465, -0.0018655394, 0.007977886, 0.04469654, 0.012281998, -0.0142621435, 0.0072711464, -0.003559169, -0.05414521, 0.010620203, 0.0032328586, -0.032140758, -0.03035799, 0.06112347, 0.021164002, -0.04737069, -0.032777462, 0.018591722, 0.012352035, 0.0023955945, -0.019101085, 0.025162496, -0.01905015, -0.026970733, -0.010811214, -0.0077868756, 0.00024194708, 0.0041513024, -0.029364735, 0.011256906, 0.014313079, -0.013752781, -0.04411077, -0.011104098, -0.012810461, 0.029084586, 0.0042404407, 0.028575223, 0.0077041043, -0.03333776, 0.017814945, 0.012141923, 0.0066408105, -0.0060709617, -0.041360214, -0.017865881, -0.012154657, -0.015013453, -0.03415274, 0.029186457, 0.016694348, 0.016235922, -0.0010457844, 0.0058099134, -0.0018655394, -0.038304042, -0.025710061, 0.009187622, 0.00674905, -0.003960292, -0.022602951, 0.0036451237, 0.021966247, 0.00080622494, 0.033796187, -0.009270393, -0.009219457, 0.005437442, -0.018260637, 0.0046574813, 0.013039674, -0.033083078, 0.022093588, -0.00087864994, 0.028014924, 0.0211258, 0.001310016, -0.006927327, 0.035578955, -0.010677506, 0.017496593, 0.0049790163, 0.0317078, -0.034509294, 0.018553521, -0.0039157225, -0.00041863212, 0.010308219, -0.00522733, -0.033363227, 0.0008094085, -0.0006076533, -0.023430664, -0.014402218, 0.004921713, -0.023621675, -0.02860069, -0.0014660083, 0.026537774, -0.03858419, -0.02297224, 0.02024715, 0.018069627, -0.07182008, -0.013689111, 0.002661418, -0.008124328, -0.0043009277, -0.01090672, 0.029110054, 0.007398487, 0.0075258273, -0.056386404, 0.01685989, -0.014580495, -0.007818711, 0.011568891, -0.0013211584, -0.04360141, 0.011823571, 0.005294184, -0.00086830347, -0.032420907, -0.020107076, -0.031784207, 0.015955772, 0.02727635, -0.06290624, -0.01086215, -0.0068381885, -0.008869271, 0.0059277033, 0.012784993, -0.020883853, 0.02071831, -0.020196214, 0.024614932, 0.010620203, -0.0020422244, -0.0049853832, -0.020960256, 0.009397734, 0.0146696335, -0.009556909, -0.014007462, 0.011957279, -0.01214829, 0.00722021, 0.0032201246, 0.015484613, -0.035044122, 0.012071886, 0.026945263, -0.011829939, -0.026817923, -0.009951665, -0.0065453053, 0.026155753, 0.015242666, -0.019419437, 0.0036737754, 0.03035799, 0.011696231, 0.021978982, 0.03361791, 0.02234827, 0.01789135, 0.0030195632, 0.001953086, 0.025429912, 0.040825386, -0.010270016, -0.036801424, -0.0068572895, -0.031122034, 0.015522815, -0.007099237, 0.017687604, 0.0012726098, -0.005459727, 0.031376716, -0.0011572074, 0.015051655, 0.026359497, 0.01144155, -0.03117297, 0.008487249, 0.03104563, 0.0015694725, -0.009849793, -0.03496772, 0.01795502, 0.013867388, 0.039297298, -0.013943792, -0.008690993, 0.0022778043, -0.016172253, 0.026614178, 0.01795502, 0.04546058, 0.045282304, -0.017700339, 0.07136165, 0.013587238, -0.036088318, -0.04306658, -0.04795646, -0.0023940026, 0.00040052587, -0.021023927, -0.0047338856, -0.0028874474, 0.013740047, -0.022335535, 0.02266662, 0.029211925, -0.019725055, 0.026894327, 0.032981206, 0.029339267, -0.0015631055, -0.020323554, 0.0060391263, -0.0076977373, -0.026486838, 0.017229179, -0.0103527885, -0.0070355665, -0.0034763974, 0.044263583, 0.009238558, -0.023799952, -0.045409646, 0.018209701, -0.01986513, 0.009225824, -0.033796187, 0.034814913, 0.011772635, 0.017178243, -0.020896588, 0.00457471, -0.0032105742, 0.0058130966, 0.006729949, -0.009512341, 0.0036705919, 0.018922808, -0.0031007428, -0.024041899, -0.00452059, 0.026181221, 0.018184233, 0.025073357, -0.008156164, 0.028906308, 0.017878616, 0.020030672, -0.0023876356, -0.016350528, -0.003237634, 0.03160593, 0.030867353, 0.037107043, 0.015089857, -0.008245301, -0.0099580325, -0.0045174067, -0.005825831, 0.021647897, -0.028677097, 0.009327697, -0.038915277, 0.014007462, 0.02806586, 0.034509294, 0.026970733, 0.0041576694, -0.0134853665, 0.04360141, -0.0059117856, 0.009607846, -0.02316325, 0.06209126, 0.006997364, 0.0007942868, 0.014937049, -0.014542293, 0.023265122, 0.027581967, -0.032879334, 0.021584226, -0.02530257, 0.0061155306, 0.005322836, 0.0072584124, -0.019967001, -0.026015678, 0.0037469962, -0.0008042353, -0.02329059, 0.00060844916, -0.003092784, 0.024322048, 0.027301818, -0.018515319, -0.015624688, 0.022997707, 0.0078250775, -0.0025786464, 0.0023446581, 0.021775236, -0.0021743402, 0.015561017, -0.005975456, 0.023749016, -0.04171677, 0.0053037344, 0.019075617, -0.027352754, 0.037081573, -0.008684627, -0.046937734, -0.049127992, -0.021240406, -0.02273029, -0.0007823486, 0.013549036, -0.006774518, 0.008258035, 0.0022714373, -0.0035623524, -0.0048548593, 0.01424941, -0.009314963, 0.030714545, 0.012390237, 0.0029447507, 0.0062492383, -0.010416458, 0.04153849, -0.022220928, 0.012829562, 0.010295485, -0.0099707665, 0.010849416, -0.015306336, -0.0036833258, -0.0010999042, 0.019381234, -0.015637422, -0.03616472, -0.027098073, 0.033847123, 4.6807607e-05, -0.039883066, 0.01754753, -0.0042690923, 0.034356486, 0.012154657, -0.015535549, 0.008627323, 0.027327286, 0.03331229, 0.052311506, -0.056386404, 0.0072775134, 0.0027632902, -0.008939308, -0.012071886, 0.035655357, 0.022220928, -0.020743778, 0.029364735, -0.015828433, -0.0077550407, 0.013510834, 0.008964776, 0.014504091, -0.021775236, 0.0072966143, -0.026817923, -0.0009972358, -0.029313799, 0.0034827646, -0.027632903, -0.0004305703, 0.020412693, -0.026206689, -0.0020820182, 0.008480881, -0.035477083, -0.021482354, 0.024500325, 0.024551261, -0.008920207, -0.008353542, 0.02009434, -0.011823571, 0.005039503, 0.0015861859, -0.039984938, 0.017649403, -0.021928046, -0.0045778933, -0.0033777086, -0.0022809878, -0.0049821995, -0.005749427, -0.048567694, -0.006755417, 0.0010051946, -0.035680827, -0.025875604, 0.010161777, 0.025544519, 0.0056825727, 0.013007839, 0.020132544, -0.047599904, 0.010371889, -0.03687783, 0.012880499, 0.046377435, -0.01424941, 0.0075576627, -0.006570773, -0.020005204, 0.028040392, -0.009015712, 0.0010823949, 0.03331229, 0.0076850033, 0.028371477, -0.02234827, 0.02444939, -0.027047137, -0.030969227, -0.015879368, -0.0057685277, -0.00598819, -0.0037883818, 0.028040392, -0.0011842672, -0.0113396775, -0.001317179, 0.016528806, 0.0095887445, -0.011174135, -0.02997597, 0.027480094, -0.014058399, 0.039475575, -0.022781227, -0.0042499914, -0.01613405, -0.010524698, 0.036241125, -0.007659535, 0.0006649565, -0.05220963, -0.034636635, 0.022093588, 0.03751453, 0.0052591655, -0.03560442, 0.006736316, 0.019814191, -0.039781194, 0.0062428713, 0.031784207, 0.005555232, 0.013956526, -0.0034127273, -0.0008253261, -0.0075385617, 0.0057175914, 0.016159518, 0.007169274, -0.011829939, 0.020947523, -0.019546777, -0.011683497, -0.05740513, 0.011836305, 0.033006676, -0.00698463, 0.019483106, 0.0138801215, 0.01789135, -0.019928798, 0.005552049, 0.0026359498, -0.025035156, -0.00020951503, -0.0126958545, -0.026079347, -0.0035273337, -0.0078378115, -0.027378222, -0.017203711, -0.014453154, 0.0038393182, 0.002647092, 0.009486872, -0.0021616062, 0.029823162, -0.059748195, -0.02630856, 0.038915277, -0.014338548, 0.0103527885, -0.03952651, 0.022513812, 0.021482354, 0.024181973, -0.02090932, -0.023787217, -0.006140999, 0.009887995, -0.009817958, 0.020883853, 0.021838907, -0.007430322, -0.0032917536, -0.010792113, -0.0047784545, 0.012173758, 0.03252278, 0.005615719, 0.007735939, 0.016210454, -0.03782015, -0.014173005, -0.007850546, -0.01150522, -0.0035432512, -0.021953514, 0.041513026, -0.037107043, -0.041640364, 0.029899566, 0.011282374, 0.020196214, 0.004291377, 0.037845615, 0.035477083, -0.008245301, -0.048160203, -0.024971485, 0.007436689, 0.003820217, 0.0029145072, -0.0142876115, -0.014134803, 0.012020949, -0.024971485, -0.00739212, -0.028626159, -0.010346421, -0.007812344, 0.0021966249, -0.022806695, -0.009111217, 0.006526204, -0.015102591, 0.032853868, 0.014083867, 3.6635283e-05, 0.077474, -0.0134853665, -0.030740013, 0.026104815, -0.01905015, 0.026015678, 0.0015225157, 0.0077614076, 0.02024715, 0.0077805086, 0.00017877422, -0.046835862, 0.00066137506, -0.014020196, 0.048924245, 0.025366241, -0.048287544, 0.008283504, -0.015866635, 0.0009845018, -0.005115907, -0.005224147, 0.033821654, 0.01870633, 0.006367028, 0.021138534, -0.007659535, 0.03198795, 0.049026117, 0.014338548, -0.008519084, -0.041003663, 0.00081816316, -0.019279363, -0.032344505, 0.013307089, 0.017993223, 0.00492808, -0.039068084, -0.018260637, 0.0008014497, 0.007003731, -0.013039674, 0.026792455, 0.0142621435, -0.022602951, 0.005348304, -0.00066814, -0.004078082, -0.01131421, 0.013358026, 0.013141546, -0.0073220828, -0.0038584191, -0.01424941, -0.012441173, 0.013574504, 0.017662136, -0.014822442, 0.0099580325, 0.016656145, -0.016490603, 0.0001540025, 0.019559512, 0.009996234, -0.019725055, -0.032319035, 0.000655406, 0.02024715, -0.02686886, 0.013268887, 0.00096221716, 0.00043216205, -0.00657714, -0.02234827, -0.010333687, -0.0078059766, -0.011855407, 0.043779686, -0.0070865024, 0.055927977, 0.047192413, -0.013714579, 0.00043813113, -0.02354527, -0.043779686, -0.038686063, -0.032166228, 0.04548605, 0.0029479342, -0.006198302, 0.0099707665, -6.9838345e-05, -0.035986446, -0.0057112244, -0.036342997, -0.05195495, 0.005230514, 0.0047689043, 0.01133331, -0.0043295794, -0.0032010234, -0.020680107, 0.0003388453, -0.027429158, 0.034636635, -0.03236997, 0.00839811, -0.006672646, -0.017050901, -0.021698833, 0.013981994, -0.017725807, 0.0036833258, -0.02793852, 0.015535549, 0.0024528976, 0.003708794, 0.0017063636, -0.011753534, -0.0036196555, -0.00013211583, 0.01632506, 0.00897751, 0.038431384, 0.018413447, 0.00874193, 0.0063160923, 0.021966247, -0.013141546, -0.004160853, -0.05669202, 0.047498032, 0.012517578, 0.0032121658, 0.00089456746, -0.014338548, -0.03229357, 0.0001224658, -0.00797152, 0.009722453, 0.037489064, -0.017381987, -0.010079006, -0.0037883818, 0.002352617, -0.016707083, 0.009346798, 0.025022421, -0.016528806, -0.0095887445, -0.021202205, 0.021418683, -0.009964399, -0.0228831, -0.004915346, -0.0055679665, 0.005316469, 0.045817137, -0.021202205, 0.015752029, 0.009951665, -0.0054883785, -0.0071438057, -0.012390237, 0.006806353, -0.011479752, 0.02753103, -0.031351246, 0.011040427, 0.0046479306, 0.0020565502, -0.023468867, -0.020361757, 0.010677506, -0.013192482, -0.019572245, 0.013599972, -0.020794714, -0.036368467, -0.071463525, 0.018948276, 0.055673297, 0.016401464, 0.014580495, 0.011237805, 0.037387192, 0.03371978, 0.039602917, -0.02329059, 0.024907814, -0.0015272909, -0.03562989, -0.0037820148, -0.022832165, -0.015828433, -0.024385719, 0.003349057, -0.018668126, -0.01729285, -0.018196966, 0.019674117, 0.05091076, 0.018400712, -0.000497822, 0.005418341, 0.0037597301, 0.035986446, -0.006755417, 0.031096566, 0.012434807, 0.03560442, -0.027352754, 0.015802965, 0.033006676, 0.035578955, 0.008531818, 0.0024688153, 0.02712354, 0.008499983, 0.020221682, 0.03349057, -0.016567009, -0.04589354, 0.010683874, -0.03387259, 0.0060900627, 0.02476774, 0.00148113, -0.022080854, -0.02429658, -0.005195495, 0.003832951, 0.0045556086, 0.023647143, -0.044620134, 0.008920207, 0.006526204, 0.008200732, -0.013020573, 0.005427892, -0.027887585, -0.01002807, -0.044645604, -0.023736281, 0.020947523, -0.008321706, 0.029746756, 0.0023175983, -0.008277137, -0.066369906, -0.01660521, 0.004711601, 0.002510201, 0.028320542, 0.05198042, 0.0346621, -0.027607435, -0.011104098, -0.03130031, -0.0064338823, 0.017827678, 0.007653168, 0.009238558, -0.035833634, -0.009143053, -0.0069400608, 0.013676377, -0.015128059, -0.008894739, 0.0049567316, 0.032497313], "content_hash": "76bf20177d1be1a7a05f1c0028659224"}, {"content": "# Model Selector Guide\n\n## How to Choose a Model\n\nThe dashboard supports seven transformer models from five architecture families. Select a model from the dropdown menu in the generator section at the top of the page.\n\n### Available Models\n\n| Model | Family | Params | Layers \u00d7 Heads | Key Feature |\n|-------|--------|--------|----------------|-------------|\n| **GPT-2 (124M)** | GPT-2 | 85M | 12 \u00d7 12 | The MI classic \u2014 start here |\n| **GPT-2 Medium (355M)** | GPT-2 | 302M | 24 \u00d7 16 | Scale comparison within GPT-2 |\n| **GPT-Neo 125M** | GPT-Neo | 85M | 12 \u00d7 12 | Local attention in alternating layers |\n| **Pythia-160M** | Pythia | 85M | 12 \u00d7 12 | Rotary PE, parallel attn+MLP |\n| **Pythia-410M** | Pythia | 302M | 24 \u00d7 16 | Larger Pythia for scale comparison |\n| **OPT-125M** | OPT | 85M | 12 \u00d7 12 | ReLU activation (unique contrast) |\n| **Qwen2.5-0.5B (494M)** | Qwen2 | 391M | 24 \u00d7 14 | Modern: RMSNorm, SiLU, rotary PE |\n\n### Architecture Comparisons\n\nThese models were chosen to highlight specific architectural differences:\n\n- **Positional encoding**: GPT-2, GPT-Neo, and OPT use absolute positions. Pythia and Qwen use rotary (RoPE). Comparing the same prompt across both types shows how PE affects attention patterns.\n- **Activation function**: Most models use GELU, but OPT uses ReLU and Qwen uses SiLU. This affects MLP behavior.\n- **Normalization**: GPT-2, Neo, Pythia, and OPT use LayerNorm. Qwen uses RMSNorm.\n- **Attention scope**: GPT-Neo alternates between local (256-token window) and global attention, unlike all other models.\n\n### What Happens When You Load a Model\n\n1. The model is downloaded from HuggingFace (this may take a moment the first time)\n2. The dashboard **auto-detects** the model's architecture family\n3. Internal hooks are automatically configured to capture attention patterns, MLP activations, and other data\n4. The layer and head dropdowns are populated based on the model's structure\n\n### Tips for Choosing", "source_file": "model_selector_guide.md", "chunk_index": 0, "embedding": [-0.007871525, 0.014752761, 0.024185896, -0.024579473, 0.004751481, -0.0034310962, -0.045147005, 0.02522697, -0.026636226, 0.049666785, -0.027372593, -0.044156715, -0.025392016, 0.0072494205, 0.0150320735, -0.03552343, -0.00047094256, 0.0002967692, -0.020377094, 0.048194047, 0.01221356, 0.013572033, 0.0020567533, 0.012873752, 0.03199394, 0.0076556928, -0.012772184, 0.04837179, -0.00035548824, -0.015438346, -0.015844619, -0.027423378, -0.061093193, -0.018459996, -0.03143532, -0.03552343, 0.017875979, 0.054846756, -0.028261315, 0.08313346, 0.002978801, -0.0020948413, -0.024477905, 0.06977726, -0.003075608, 0.0033358762, 0.008753898, -0.01580653, -0.027144065, 0.033314325, 0.019323325, 0.039992426, -0.039383017, -0.007954049, 0.029937187, -0.024439815, -0.03298423, -0.04380123, -0.0171777, 0.007827089, -0.00947757, 0.0011307382, 0.010372639, 0.042379275, 0.03047042, -0.034533143, -0.06271828, 0.008753898, -0.02582368, 0.0025058747, 0.00637657, -0.007414469, 0.026458481, 0.004614999, 0.035929702, -0.004627695, 0.0019091623, 0.050250802, 0.003310484, -0.010283766, 0.00041500077, -0.012442088, -0.006735232, -0.014498841, -0.015019378, -0.010264723, -0.012962624, -0.026585441, -0.03628519, -0.03123218, -0.008055617, -0.0139656095, -0.036031272, 0.029759444, 0.018993229, 0.016454026, 0.015819225, -0.023398744, -0.024211287, 0.059772808, 0.028464451, -0.011172487, -0.007763609, 0.010188547, 0.041465163, -0.046137296, 0.034253832, 0.0067987125, -0.04408054, 0.0015973166, -0.091258906, -0.022903599, 0.007846133, 0.015590698, -0.005906818, -0.009249042, -0.020910326, 0.019742293, 0.018155292, -0.03138453, -0.032019332, 0.008442845, 0.04519779, 0.016746035, 0.023271782, 0.01341968, 0.0006018701, -0.011654936, -0.018675828, 0.00582112, 0.006246436, 0.03359364, 0.04052566, -0.044918478, 0.011191531, -0.024439815, -0.030419637, -0.018904356, -0.045477103, 0.019297933, 0.017152308, -0.051063344, -0.007535081, 0.032171685, -0.06845687, 0.014930505, -0.0023868496, 0.014041785, -0.086993046, 0.034456965, 0.014701977, -0.03552343, -0.026280737, -0.039586153, -0.024604864, -0.027626514, 0.0035580564, 0.037732538, 0.004316643, -0.04773699, 0.0295817, 0.0124547845, -0.023487614, -0.03587892, -0.04948904, 0.056116357, -0.04910816, -0.012867404, -0.036615286, -0.038443513, 0.02598873, 0.0039484585, 0.022865511, -0.011673979, 0.04291251, 0.010632907, -0.05012384, -0.0314861, -0.010588471, -0.023792319, -0.008131794, -0.012962624, -0.01679682, 0.0009101451, -0.034177653, 0.0043801228, -0.05301853, 0.027626514, -0.010594819, -0.013470464, 0.004916529, 0.0111090075, 0.040627226, 0.0050371415, -0.003039107, 0.004570563, 0.055964004, 0.03171463, 0.029759444, -0.004091289, 0.010937612, 0.035777353, 0.07023432, -0.08232092, 0.03598049, 0.024198592, -0.015095554, 0.021164246, -0.035802744, -0.014054481, 0.009591835, -0.039992426, 0.026890146, 0.018015636, -0.030318068, 0.002902625, -0.025480889, -0.00026522754, 0.025785593, 0.044893086, -0.0035390123, 0.017076131, 0.0020551663, 0.017418923, -0.021062678, 0.03123218, 0.023551095, -0.05301853, 0.011083615, -0.024274768, 0.02115155, 0.017215787, -0.0051767975, 0.0071034166, -0.016784122, -0.03864665, 0.023474919, 0.000730814, -0.022954384, -0.0629722, 0.009471222, 0.019069405, -0.010467859, -0.0015116185, 0.039459195, 0.005941732, -0.016111234, -0.09582947, 0.025112703, 0.038341943, 0.017406227, 0.036818422, -0.026255345, 0.017533187, 0.007979441, 0.003123218, -0.024262073, 0.025963336, -0.061651815, -0.029784836, -0.01631437, 0.050834816, 0.018015636, 0.0067161885, -0.0260776, -0.010512295, -0.052967746, -0.03653911, -0.050580896, -0.07444939, 0.053983428, -0.017888676, 0.018637741, 0.030927476, -0.03171463, -0.018942444, -0.003189872, 0.0072113327, -0.030521205, 0.0056243315, -0.027093282, -0.04093193, 0.068152174, 0.022903599, -0.00995367, 0.007960397, -0.0024074805, -0.017418923, -0.014435361, -0.009217302, 0.011013787, -0.046365824, 0.0093760025, -0.050073057, 0.009528354, 0.017736323, -0.0133562, 0.00881103, 0.0120866, 0.012080252, 0.008182578, -0.026534658, -0.017393531, -0.024909569, 0.033085797, 0.020770669, -0.013952913, 0.01616202, 0.012670617, -0.00045586604, -0.022065662, -0.029987972, 0.008417454, -0.01434649, -0.024922265, 0.007446209, 0.004913355, 0.019805772, -0.002788361, -0.025290448, -0.03930684, -0.03956076, -0.024058936, -0.0054370654, -0.018688524, 0.024884176, -0.03448236, -0.016492115, -0.005824294, -0.024312856, 0.055608515, -0.02932778, -0.0016965041, 0.007808045, -0.018333036, 0.01466389, -0.035167944, 0.032831877, -0.02178635, 0.014041785, -0.04116046, -0.027601123, -0.029886404, -0.007312901, -0.044613775, -0.012905492, -0.00626548, 0.004065897, 0.026483873, -0.021418165, -0.004802265, -0.00941409, -0.015349474, 0.01199138, 0.01246748, 0.018028332, -0.013165761, -0.0136862965, -0.046873663, -0.009204606, -0.013864041, 0.02280203, 0.029759444, -0.01243574, -0.013292721, 0.003300962, -0.004811787, 0.037478615, -0.02328448, -0.010975699, -0.0082016215, 0.003188285, -0.0034247483, -0.009045905, -0.053729508, -0.035726566, 0.00021305488, -0.0049070073, -0.019856557, 0.022979775, 0.04446142, 0.019107493, 0.019704204, -0.062159657, 0.03740244, -0.0038214985, 0.01479085, -0.012194516, -0.028896116, 0.016847603, 0.022459239, -0.013267329, -0.03146071, 0.011654936, 0.00053323235, 0.03831655, 0.003367616, -0.010848739, 0.053881858, 0.030571988, 0.022344975, -0.0022106925, -0.037859496, -0.022471935, -0.011527975, -0.016974563, 0.014740066, 0.0124547845, 0.001634611, -0.017583972, 0.060433, -0.038976744, 0.016454026, 0.000700661, -0.014613106, -0.029226212, 0.028515235, 0.03311119, 0.019729597, -0.00014064797, -0.016872995, -0.079883285, -0.06251515, -0.03161306, -0.07389077, 0.010988395, 0.048803456, -0.032755703, -0.0013076889, -0.040322524, 0.0020472314, 0.011140747, 0.0515204, -0.025404712, 0.02461756, -0.04745768, 0.044766124, 0.0021646693, 0.030394243, 0.015578002, 0.03524412, -0.0069574127, 0.03387295, -0.014054481, 0.0131276725, -0.0037326263, -0.025658634, -0.030927476, -0.044791516, -0.039230667, -0.0014084635, 0.06703492, 0.00321209, -0.010753519, -0.006030604, -0.039992426, 0.07262117, -0.0026725098, 0.0858758, -0.019818468, 0.036082055, 0.009229998, -0.013394289, 0.004392819, -0.02531584, 0.013610121, 0.037224695, -0.017457012, 0.0009307761, -0.038951352, 0.0025360277, -0.026534658, 0.010816999, -0.03534569, 0.0065257484, -0.0054719797, 0.020973805, 0.030673556, -0.008582502, -0.03854508, 0.0016520681, -0.0012085013, 0.0075160367, 0.014803546, 0.010074283, 0.114162505, 0.0020535793, 0.027651906, 0.008665025, 0.0069574127, -0.035574216, -0.0047705253, 0.038062632, -0.0133562, 0.055049893, -0.02986101, -0.007935005, -0.053932644, -0.053170882, -0.009045905, -0.031409923, -0.0010617037, -0.0037897585, 0.07653154, -0.02337335, -0.0143591855, 0.0069637606, 0.040881146, -0.01148354, 0.018993229, -0.0146384975, 0.01832034, -0.015247906, 0.047914736, 0.0034628361, 0.036970776, 0.03966233, 0.030419637, 0.02127851, 0.019132884, -0.011635891, -0.009236346, -0.030673556, 0.0033422243, 0.029150035, -0.007503341, -0.028413666, -0.079172306, -0.026407696, -0.03059738, 0.03034346, -0.030571988, -0.018218772, -0.0368946, -0.053221665, 0.023893887, -0.009166518, -0.04354731, -0.047127582, 0.026128385, -0.017266572, 0.018409211, 0.013838649, 0.005827468, 0.006157564, 0.026331522, 0.012930884, -0.019793076, 0.006287698, 0.019285237, 0.021202333, -0.033923734, 0.021608606, -0.013470464, -0.014575018, -0.0009990172, -0.019031316, 0.0019504243, 0.026763186, -0.024287464, -0.0018663132, 0.0038881525, -0.0026487047, -0.030368851, 0.0026042687, 0.025176184, -0.002856602, -0.010201243, 0.00063519715, -0.0407034, -0.013546641, -0.025569761, 0.030800516, 0.053170882, 0.002912147, 0.004595955, 0.012404, 0.036107447, 0.0040214607, -0.018155292, -0.03526951, -0.022598894, -0.006208348, 0.023678055, -0.008360322, 0.016390547, -0.021494342, -0.014460754, -0.0059195138, 0.015908098, 0.026737794, -0.016377851, 0.00014253253, 0.0142449215, -0.024668343, -0.020605622, 0.0068939324, -0.0022948035, -0.04283633, -0.012353216, 0.009547398, 0.0031819372, 0.016847603, 0.01825686, -0.0055703735, -0.0027740777, -0.004751481, 0.02051675, -0.026661618, -0.019044012, -0.02770269, -0.019031316, -0.017165003, -0.0041325507, -0.04156673, 0.02884533, -0.025100008, 0.007833437, 0.047229152, 0.00957279, -0.0067542763, -0.013254632, 0.019615334, -0.00014044959, 0.005802076, 0.009052254, 0.008214317, -0.004484865, -0.019386804, 0.021608606, -0.028159747, -0.0053228014, 0.05403421, 0.031054435, -0.02083415, 0.016009666, 0.011013787, -0.06007751, 0.04042409, -0.01218182, -0.004516605, -0.00881103, -0.022281494, -0.0059226877, 0.00846189, 0.0019647074, -0.014219529, 0.037072346, -0.016174715, -0.012645224, -0.01918367, -0.039865468, 0.047787774, 0.014930505, 0.009814015, 0.0011331188, 0.013241936, -0.013711689, -0.039408408, 0.028718371, -0.06896472, -0.00858885, -0.016530203, -0.056065574, 0.0007335912, 0.032806486, 0.030571988, 0.024503296, 0.013445073, -0.013153065, -0.051672753, 0.017952155, 0.032146294, 0.0025645937, -0.011889812, -0.029607091, 0.019348716, -0.03791028, -0.016390547, 0.020453269, -0.020504054, 0.005576722, -0.06485121, 0.0071922885, 0.036716856, -0.019031316, -0.020631013, 0.0013973544, 0.026331522, 0.011140747, -0.0010101262, -0.024795305, 0.0058972957, -0.034583926, -0.018041028, 0.0050498373, -0.0038595865, 0.062362794, -0.00585286, -0.09608339, -0.030648164, 0.024376336, -0.029403955, 0.0024646127, -0.017875979, -0.006081388, -0.04837179, 0.032044724, -0.0016227085, -0.031765413, -0.041744474, 0.007503341, 0.014117961, -0.0027677298, -0.002991497, -0.02795661, -0.013102281, 0.024376336, 0.0022709984, 0.000745097, -0.043318782, 0.023005167, -0.024465207, 0.0055164155, 0.021507038, -0.03978929, -0.011972336, 0.007795349, -0.037707146, 0.024820697, 0.018739307, 0.007719173, -0.0241732, -0.0036754943, 0.012296084, -0.005744944, 0.011432756, -0.0017250702, -0.030698948, -0.01510825, -0.023551095, 0.017914068, -0.024376336, 0.016771426, 0.018180683, 0.0058084237, -0.011178835, 0.04283633, 0.011724764, -0.03285727, 0.0022836945, 0.01033455, 0.019437589, -0.015946187, -0.019691508, -0.011432756, 0.044232894, -0.025277752, -0.0021646693, 0.035066374, 0.013940217, 0.015539914, -0.019424893, 0.00615439, -0.01466389, -0.0040024165, 0.0130388, 0.011064571, -0.007566821, -0.01692378, 0.0066146203, 0.0024233505, -6.467029e-05, 0.0052624955, 0.03209551, -0.043039467, 0.03915449, -0.004808613, -0.01485433, 0.004735611, -0.01587001, 0.007198637, 0.007471601, -0.012822968, 0.020948414, -0.006094084, -0.01011237, -0.022395758, -0.007477949, -0.01221356, -0.014219529, 0.013724385, 0.013737081, -0.004281729, 0.003018476, 0.0017139611, 0.0037135824, -0.107966855, -0.04672131, 0.0044372547, -0.02744877, -0.007255769, -0.018079115, 0.03135914, -0.004945095, 0.036996167, -0.05652263, 0.020237437, -0.025493585, 0.02442712, 0.014029089, 0.019767685, -0.0407034, 0.030394243, -0.023297176, 0.041135065, -0.03321276, -0.03704695, -0.004338861, 0.0065892283, 0.01205486, -0.049438257, -0.0171777, -0.0070589804, 0.0009982237, 0.029937187, 0.0033041362, -0.057741445, 0.025887161, -0.0038246724, 0.03768175, -0.006211522, 0.034279224, -0.0035993184, -0.026128385, 0.024262073, -0.026788577, -0.008220665, -0.020440573, 0.00925539, -0.04347113, 0.025480889, -0.0055005457, 0.016504811, 0.008582502, -0.011915204, 0.020694494, 0.030876692, 0.0035358383, 0.0005943319, 0.04055105, -0.0001699083, 0.006113128, -0.012924536, -0.011178835, 0.0006232947, -0.0017710931, 0.008474586, 0.03537108, 0.014613106, 0.011572411, 0.006132172, 0.010861435, 0.052205987, 0.007827089, -0.014130658, -0.007439861, -0.029835619, 0.036082055, -0.004472169, -0.013254632, -0.020440573, 0.0012061208, 0.023157518, 0.036716856, -0.010277418, -0.012442088, 0.028566018, 0.00019688731, -0.04522318, 0.0096553145, 0.005903644, 0.028489843, -0.0034310962, -0.017253876, 0.025976032, 0.0072176806, 0.003272396, 0.0021519733, 0.0021154725, -0.0053164535, -0.019958125, -0.004716567, -0.00078159804, 0.03374599, 0.036970776, -0.01720309, 0.059874374, 0.035777353, -0.031790804, -0.02410972, -0.022738552, -0.01466389, 0.011051876, -0.026204562, -0.022014879, 0.0067987125, -0.0009775927, 0.0340253, -0.013381593, 0.030749733, -0.0027439247, 0.026813969, 0.0680506, 0.010150459, 0.043318782, -0.013483161, 0.038164202, -0.031841587, -0.0017203091, 0.02782965, -0.024922265, -0.0040881145, 0.006319438, 0.018599652, 0.010582123, 0.010417075, -0.018650437, -0.0025344407, -0.0055354596, -0.007782653, -0.013749777, -0.004545171, 0.011635891, -0.019996213, -0.0368946, -0.005579896, 0.0023662185, 0.0025915727, -0.011826332, -0.0040055905, -0.01631437, 0.004316643, 0.006157564, -0.04761003, 0.0011593043, -0.005833816, 0.026636226, 0.0037421484, -0.022548111, 0.05677655, 0.022052966, 0.013280025, -0.0009799731, -0.031892374, -0.008233361, 0.0069891526, 0.023208303, 0.021164246, 0.03600588, -0.030673556, 0.03359364, -0.02309404, -0.011293099, 0.034228437, -0.018675828, 0.010537687, -0.0038088025, 0.0067225364, 0.038697433, 0.025277752, 0.04824483, 0.047559246, 0.010956655, 0.02102459, -0.01472737, 0.022700462, -0.016733339, 0.06561297, 0.02869298, 0.03298423, 0.029911796, 0.006401962, -0.030648164, 0.0037802365, -0.01736814, 0.015412954, 0.004843527, 0.011045528, -0.00088237255, 0.003032759, -0.005862382, -0.013229241, 0.033923734, -0.026356913, -0.0178125, 0.00063202315, -0.041211244, 0.033314325, 0.007198637, -0.021697478, -0.02683936, 0.02598873, 0.01958994, 0.0022757596, 0.0041008107, 0.02881994, -0.022751248, 0.019551853, 0.0065574883, 0.02833749, -0.0036183624, 0.013635512, 0.013978305, -0.019120188, -0.020326309, -0.008246058, -0.069066286, -0.054440483, -0.021164246, -0.019285237, 0.007484297, -0.004653087, -0.0038786305, 0.035091765, 0.00609091, 0.02220532, 0.010467859, -0.0135212485, 0.016174715, 0.0042531625, -0.013051497, 0.023259087, 0.025836376, 0.012511916, 0.01660638, 0.022725854, 0.034456965, 0.017825196, -0.007611257, -0.024795305, 0.014041785, -0.0064749643, 0.02070719, 0.0017822022, 0.027372593, -0.026763186, -0.0065511405, 0.017609363, 0.020377094, -0.04380123, 0.017152308, 0.02598873, 0.020999197, 0.039103705, 0.014295706, 0.004703871, -0.008334929, 0.03679303, 0.0178125, -0.021253118, 0.030292675, 0.0002695918, -0.011978684, 0.0011505757, 0.027778866, 0.0002711788, -0.022789335, 0.03537108, -0.022535415, 0.010213939, 0.007173245, -0.007909613, 0.016936475, 0.014993986, 0.018586956, -0.016847603, -0.011686675, -0.012492872, 0.015984274, 0.00023666152, 0.025620544, -0.0020599274, -0.023259087, -0.0074081207, 0.026229953, -0.0135212485, -0.016200107, 0.014562322, 0.00026046654, -0.033644423, -0.032933444, 0.02643309, -0.007535081, 0.0044404287, 0.00058877736, -0.04849875, 0.031663846, -0.017875979, 0.014575018, -0.039357625, -0.00618613, -0.0024725476, 0.019666117, -0.059214182, 0.0016996781, 0.025747504, -0.027017105, 0.018015636, 0.021011895, 0.033288933, 0.017380835, 0.023500312, -0.0066781, -0.00830319, 0.006097258, 0.0114454515, 0.015565306, 0.02220532, -0.010956655, 0.0040278086, -0.012822968, -0.016949171, -0.011743807, -0.001663177, -0.002983562, 0.018294947, 0.00097124465, 0.005002227, 0.014181442, 0.028566018, -0.049920704, -0.028718371, -0.048752673, -0.039103705, 0.009293478, 0.0054243696, 0.014168745, 0.007827089, -0.041211244, -0.014892418, 0.029784836, 0.006049648, 0.013064193, 0.014029089, 0.016047755, 0.026636226, 0.049412865, -0.030318068, -0.016771426, -0.009483919, 0.0014830525, 0.019856557, -0.015438346, 0.009433134, -0.019920036, -0.033415895, 0.019107493, 0.028362883, -0.02881994, 0.013432377, 0.03047042, -0.010956655, -0.04128742, -0.027931219, 0.029962579, 0.017139612, 0.018421909, 0.008309538, -0.016669858, 0.010702735, 0.013318113, 0.024909569, 0.007116113, -0.019640725, -0.006014734, 0.023792319, -0.013508553, -0.07465253, 0.011845375, 0.00912843, -0.0027264678, 0.01434649, -0.006224218, 0.013546641, -0.025468193, 0.020186653, -0.006627316, -0.020872237, 0.008785638, -0.009502962, 0.00607504, -0.0028502538, -0.0012521438, 0.005862382, -0.004691175, 0.006271828, -0.015908098, -0.00061377266, -0.037097737, -0.02223071, 0.030419637, -0.037884887, 0.013864041, 0.020618318, 0.009820363, 0.012981668, -0.043242604, 0.041084282, -0.0069320207, 0.021507038, 0.017076131, -0.041515946, -0.005913166, 0.028718371, 0.026890146, 0.049260512, -0.016415939, 0.0017012651, -0.001650481, -0.007147853, 0.03854508, 0.019221757, -0.018028332, -0.008131794, 0.031765413, 0.011813636, 0.020046998, -0.0069129765, -0.022383062, 0.024985744, -0.030191109, -0.0044118627, 0.02153243, -0.021608606, -0.039586153, 0.020567533, 0.013622817, 0.013191152, 0.004345209, 0.04395358, 0.00577351, 0.005982994, -0.019450285, -0.006300394, 0.026458481, 0.009249042, 0.0072176806, -0.009782274, -0.015070162, 0.014232226, -0.017723627, 0.012803924, -0.0034501404, -0.01186442, 0.01975499, -0.0068685403, -0.008214317, 0.007915962, 0.009490266, -0.0018266382, -0.009007818, 0.0052053635, -0.010232982, 0.050580896, -0.0021614954, -0.04354731, 0.022687767, -0.018358428, 0.006325786, -0.009014166, 0.01434649, -0.014930505, -0.014448058, -0.013775169, -0.052916963, -0.02022474, 0.009750535, 0.034050696, 0.005970298, -0.02744877, -0.009566442, -0.010144111, 0.013305416, -0.024389032, -0.010836043, 0.02795661, 0.00830319, 0.02909925, -0.0021948225, -0.0040500266, 0.0065828804, 0.0002672113, 0.025912553, 0.00057766837, -0.03326354, -0.0037104085, -0.00283121, -0.025341233, 0.02277664, -0.0049101813, -0.0054307175, -0.05174893, -0.002883581, -0.023271782, -0.02455408, -0.025836376, 0.0004713393, 0.01001715, -0.009750535, 7.081992e-05, -0.0023757406, -0.026763186, -0.0038214985, 0.021367382, 0.034710888, 0.027982002, 0.026179168, -0.010505947, -0.021659391, -0.004872093, 0.021672087, -0.009229998, 0.0023630445, -0.011870768, -0.03196855, 0.019475676, 0.011997728, 0.033568244, -0.017507795, -0.007795349, -0.00036322486, 0.029023075, -0.032120902, 0.0036310584, 0.0070145447, -0.0054624574, -0.011407363, -0.044385243, -0.013787865, -0.01692378, 0.017076131, 0.023195608, -0.021164246, 0.024020847, 0.04268398, -0.016847603, -0.0069510643, -0.004916529, -0.017901372, -0.031866983, -0.042709373, 0.012004076, 0.00035469473, 0.038062632, 0.014867025, 0.009356958, -0.006278176, -0.029835619, -0.01284836, 0.00014610328, 0.045273967, -0.0010283766, 0.018625043, -0.01571766, 0.0072494205, -0.015958883, 0.0040468527, -0.0024693736, 0.02782965, 0.021697478, -0.010423423, 0.028540626, -0.0117818955, -0.021710174, 0.024782607, 0.019437589, 0.012162776, -0.022306887, 0.0066971444, 0.00063638744, -0.013280025, -0.015146337, -0.017571276, 0.013800561, -0.0056719417, 0.007662041, -0.02569672, 0.034558535, -0.004818135, 0.026179168, -0.0065511405, 0.023259087, 0.04192222, 0.006348004, -0.0423031, 0.0178125, 0.0026772707, -0.009452178, -0.021900615, -0.016492115, -0.033898342, -0.013076888, -0.017063435, -0.004427733, 0.003229547, 0.0023931975, -0.009687054, -0.0068494966, 0.02083415, 0.021646693, 0.025747504, 0.050555505, 0.00090935157, -0.01927254, -0.00973149, -0.0037865844, -0.013991001, -0.021964094, -0.023614576, 0.0064432244, 0.012257996, 0.02248463, -0.0067161885, 0.00094743964, 0.004605477, 0.009306174, 0.0023106735, 0.015908098, -0.008791986, -0.018726612, 0.02235767, 0.052409124, 0.034050696, -0.006125824, -0.022027574, -0.0022186274, -0.00839841, 0.021380078, 0.007255769, -0.031181397, 0.0039960686, 0.01243574, 0.00057528785, -0.031308357, -0.04055105, 0.042150747, -0.011788243, -0.035447255, 0.016276283, -0.00020045807, 0.035675783, 0.027169459, -0.001601284, 0.03864665, -0.010639255, -0.008106401, 0.007604909, -0.012124687, -0.029657876, -0.020732582, -0.0047959173, -0.009141126, 0.011356579, 0.024439815, -0.010759867, 0.024211287, -0.008157185, 0.011248663, 0.02461756, -0.00884277, 0.073636845, 0.014333793, -0.00324383, 0.007776305, 0.03928145, -0.023754232, 0.0071795927, 0.020186653, 0.03966233, 0.017952155, -0.015438346, -0.011585108, -0.023233695, 0.021049982, 0.050682466, -0.020923022, -0.020529445, -0.016022362, -0.03361903, -0.013508553, 0.0019631202, 0.025785593, -0.003121631, -0.034939416, -0.0065003564, 0.0022344973, 0.009591835, 0.0032533521, -0.01679682, -0.0007966745, 0.013787865, 0.00010791607, -0.007535081, -0.0024376337, -0.0071351565, -0.013724385, -0.018269556, -0.026407696, 0.016809516, -0.0019329672, 0.00026760806, 0.011331188, -0.01825686, -0.055100676, -0.015006682, -0.051698145, 0.0022344973, 0.035853527, 0.023170216, -0.042607803, -0.034787063, -0.020732582, 0.028312098, -0.027550338, 0.022306887, 0.006078214, 0.0088173775, -0.015908098, -0.02732181, -0.0077382172, 0.015946187, -0.0068748887, -0.011902507, 0.013216545, -1.05696745e-05], "content_hash": "13b7886a222d66a792696c9689e3a561"}, {"content": "from HuggingFace (this may take a moment the first time) 2. The dashboard **auto-detects** the model's architecture family 3. Internal hooks are automatically configured to capture attention patterns, MLP activations, and other data 4. The layer and head dropdowns are populated based on the model's structure ### Tips for Choosing\n\n- **Start with GPT-2**: It's small, fast, and the most widely studied. Most educational resources reference GPT-2.\n- **Compare same-size models**: GPT-2, GPT-Neo 125M, Pythia-160M, and OPT-125M all have 12 layers and 12 heads \u2014 differences in their attention patterns come from architecture, not scale.\n- **Compare scale**: GPT-2 vs GPT-2 Medium (or Pythia-160M vs Pythia-410M) shows how more layers and heads change behavior.\n- **Try Qwen2.5 for a modern perspective**: It uses an entirely different design philosophy from GPT-2.\n- **Memory matters**: All dropdown models are small enough for interactive exploration. Larger models can be entered manually but may be slow.\n\n### Generation Settings\n\nAfter selecting a model and entering a prompt, you can configure:\n- **Number of Generation Choices (Beams)**: 1-5 beams. More beams explore more paths but take longer.\n- **Number of New Tokens**: 1-20 tokens to generate. Shorter is faster.\n\nClick **Analyze** to run the model and see results in the pipeline and generation sections.", "source_file": "model_selector_guide.md", "chunk_index": 1, "embedding": [-0.014125664, -0.012089349, 0.06310546, -0.03160685, -0.0035043552, -0.025288185, -0.0370731, 0.037154283, 0.008239971, 0.068896435, -0.006294986, -0.03826377, -0.033582278, -0.017914155, -0.00411322, -0.053607166, -0.022406226, -0.025071701, 0.0064235236, 0.022785075, -0.0056387647, -0.0076581663, -0.026262369, 0.008855601, 0.023245106, 0.015735773, 0.0066129486, 0.022095028, 0.0047897366, -0.014518043, -0.0021513223, -0.033474036, -0.0649997, -0.016642304, -0.035124734, -0.02218974, -0.011886395, 0.03580125, -0.016222864, 0.05098228, 0.011629318, -0.02013313, -0.01888834, 0.03986035, 0.00083042396, 0.006663687, 0.014003891, -0.008307622, -0.049250398, 0.03488119, -0.025017578, 0.05287653, -0.0509011, 0.0020583014, -0.035097674, -0.027737174, -0.036477767, -0.055339046, -0.017454125, 0.0013631807, 0.0056725903, 0.009768898, 0.0026823878, 0.05390483, 0.050062217, -0.018577142, -0.04638197, 0.026438264, -0.036504827, 0.016222864, 0.0065858876, -0.0033250784, -0.02752069, -0.005652295, 0.00032895614, 0.00492504, 0.02095848, 0.030226756, -0.010445414, -0.020051949, 0.02170265, -0.017129397, -0.020214312, -0.048059728, -0.0050874036, -0.0047322325, 0.0086188195, -0.010932506, -0.051902343, -0.061806545, -0.014896893, 0.016777609, -0.04889861, 0.02633002, 0.023623955, 0.0072657866, 0.036450706, -0.06213127, -0.013591216, 0.009295336, 0.031092696, 0.006721191, -0.04440654, 0.015221621, 0.023326287, -0.033149306, -0.0068463464, 0.013320609, -0.059858177, -0.0066941306, -0.09216861, -0.0040388033, 0.002142866, 0.011399303, -0.011338416, -0.0061596823, -0.0017386473, 0.028034842, 0.029793786, -0.021553814, -0.028954905, -0.007847591, 0.010770142, -0.003981299, 0.02917139, 0.018780097, 0.01597932, -0.024882276, 0.0073672645, 0.023055682, -0.028007781, 0.038480256, 0.03371758, -0.016046971, -0.000894693, -0.02094495, -0.008909722, -0.0023508947, -0.028630177, -0.012332895, 0.017860034, -0.048276216, -0.00033466425, 0.043621782, -0.08329271, 0.01853655, -0.034231734, 0.044650085, -0.08107373, 0.063051336, 0.007272552, 0.0046206075, -0.010783672, -0.038750865, -0.008896192, -0.03206688, -0.008875896, 0.03401525, -0.009342693, -0.044190057, 0.0019331458, 0.038317893, -0.009910966, -0.0144909825, -0.030605605, 0.0845375, -0.05477077, -0.00086424977, -0.04770794, -0.023962213, 0.02059316, -0.0047660586, 0.009146502, -0.018225353, 0.020065479, 0.0022916996, -0.096227705, -0.030984454, -0.039562684, -0.05477077, -0.011460189, -0.026857704, -0.03166097, -0.005222707, -0.03788492, -0.026113536, -0.020024888, -0.008314387, 0.029279632, -0.011757856, 0.019226598, 0.00023783783, 0.045083057, -0.017102337, 0.010025974, -0.040753353, 0.07804294, 0.012109645, 0.004745763, -0.029279632, 0.0104657095, -0.00923445, 0.035449464, -0.08497047, 0.038344953, -0.003101828, 0.012305834, 0.015654592, -0.04102396, -0.014098603, 0.027710114, -0.050792858, 0.009369753, 0.008767653, -0.04676082, 0.016236395, -0.0073807947, 0.013868587, 0.03401525, 0.0549602, -0.007881417, 0.02589705, 0.021499693, 0.0150321955, -0.028143086, 0.015316333, 0.018766567, -0.04191696, 0.010289815, -0.0076243407, 0.022676831, 0.009795958, -0.012387017, -0.013246193, 0.012623797, 0.010959567, 0.015546348, -0.0028718123, -0.03409643, -0.06938353, 0.011345181, 0.0091059115, -0.0025352454, -0.0015060948, 0.025937641, -0.0026502532, -0.037262525, -0.08085725, 0.03991447, 0.006382933, -0.004025273, -0.0148157105, -0.010357467, 0.026181187, 0.017589428, -0.022392696, -0.002021093, 0.019240128, -0.034231734, -0.031146819, -0.007468742, 0.0634843, 0.00053952186, 0.010181573, -0.03117388, -0.022081498, -0.06878819, -0.05828866, -0.055582594, -0.054067194, 0.043324113, -0.035476524, 0.0116631435, 0.012157001, -0.05214589, -0.006633244, 0.009883906, -0.0027568047, -0.017197048, 0.026492385, -0.03820965, -0.017197048, 0.07403796, 0.023556303, -0.012299069, 0.018293004, -0.00055728044, -0.0077799396, -0.0101951035, 0.0023610424, 0.021147905, -0.03079503, 0.014315088, -0.03485413, 0.0055440525, -0.0016879085, -0.01087162, 0.017021155, -0.016764078, -0.0024422244, 0.011013688, -0.035611827, 0.02099907, -0.037722558, 0.02258212, 0.026113536, -0.009701246, 0.045678392, 0.03455646, 0.016547592, -0.025707625, -0.011385772, -0.0051212297, -0.010540127, 0.0019534412, 0.0025234064, 0.007800235, -0.0145992255, 0.009660655, -0.046544332, -0.01895599, -0.048303276, -0.0050332826, -0.013679163, -0.012576441, 0.008253501, 0.0010849633, -0.014693937, 0.02379985, -0.040915716, 0.03168803, 0.004356766, -0.0054256623, 0.035016492, -0.019605447, -0.019524265, -0.008307622, 0.052308254, -0.003815553, 0.031796273, -0.07928773, -0.0099244965, -0.01852302, -0.003822318, -0.058451023, 0.020741995, 0.0151945595, 0.027277144, 0.05065755, -0.009349458, -0.006190126, -0.018279474, -0.038074348, -0.0056252345, 0.063051336, 0.019226598, 0.021540284, -0.014910423, -0.01028305, -0.005053578, -0.022460347, 0.027696583, 0.042809963, 0.012576441, -0.007231961, 0.0016261764, -0.008429395, 0.01603344, -0.026438264, -0.004779589, -0.03285164, 0.015045726, -0.000103802995, -0.00039237956, -0.049981035, -0.028278388, 0.004542808, 0.037614316, -0.021689117, 0.03412349, 0.031309184, 0.0070493016, 0.029847907, -0.044162996, 0.026424734, 0.02838663, 0.026573567, 0.00699518, -0.05095522, -0.00038815133, 0.038372014, -0.014626286, -0.025085231, -0.009714777, 0.0031525667, 0.017372943, -0.016628774, -0.010946036, 0.03664013, 0.04310763, 0.02833251, -0.018279474, -0.022893317, -0.006974885, -0.0042722016, -0.002016019, 0.024138108, 0.030362058, 0.04930452, -0.029766724, 0.04318881, -0.03810141, 0.020457858, -0.00798966, -0.04565133, -0.0018942461, 0.02795366, 0.022947438, 0.013712989, 0.030280877, -0.025301715, -0.06581152, -0.04069923, -0.019064235, -0.032391608, 0.019808402, 0.042593475, -0.020173721, 0.0022730953, -0.010174807, -0.009166799, 0.010587483, 0.046896122, -0.029631421, 0.0138956485, -0.030307937, 0.035368282, 0.009342693, 0.020714935, 0.019158946, 0.017778853, 0.0068564946, 0.02628943, -0.004891214, -0.010918976, -0.013171775, -0.07506627, -0.04857388, -0.03125506, -0.07138602, -0.007475507, 0.0534448, 0.007962598, 0.0072116656, 0.0068395813, -0.03255397, 0.08469986, -0.006251012, 0.085511684, 0.0100598, 0.0549602, -0.01643935, -0.020444328, 0.012366721, -0.01480218, 0.022933908, 0.009931262, -0.0051787337, 0.0050806385, -0.029739663, 0.010966332, -0.025694095, -0.017129397, -0.04979161, -0.011460189, -0.0047998843, 0.0017978426, 0.027818358, -0.0007120336, -0.017684141, 0.010350702, 0.029198451, 0.045732513, 0.046652574, -0.0074619767, 0.11549489, -0.008097902, 0.015316333, 0.02542349, 0.004755911, -0.044623025, -0.01845537, 0.052091766, -0.01560047, 0.025667034, -0.03461058, -1.5776575e-05, -0.041808717, -0.045732513, 0.006568975, -0.0143827405, 0.0145992255, -0.008882661, 0.054040134, -0.008138493, -0.00042768524, 0.017061746, 0.022040907, -0.0052497676, 0.02214915, -0.046165485, 0.007976129, -0.020363146, 0.042891145, -0.0077867047, 0.030334998, 0.049196277, 0.040969837, 0.020417267, 0.03120094, -0.0027263614, 0.010776907, -0.031471547, 0.0050332826, 0.0014096912, -0.05920872, -0.02467932, -0.041565172, -0.024029864, -0.01890187, 0.03168803, -0.04732909, 0.00020380059, -0.028873723, -0.09357576, 0.00698165, -0.014951014, -0.017927686, -0.057693325, 0.0016870629, -0.019876054, 0.030443242, 0.0044278004, -0.0057639205, -0.004170724, 0.025207004, 0.011845803, -0.0125358505, 0.016534062, 0.027818358, 0.04513718, -0.031092696, 0.026560036, -0.0070831273, -0.00080801436, 0.00842263, 0.0015018665, 0.010235694, 0.0018756419, -0.017115867, 0.016046971, 0.02585646, 0.030686786, -0.018428309, -0.014193315, 0.031363305, 2.3189188e-05, -0.021499693, 0.0047525284, -0.037100162, -0.012522319, -0.016101092, 0.022839196, 0.022338573, 0.0018333596, -0.0063220463, -0.00344516, 0.042755842, 0.004170724, 0.0047525284, -0.016967032, -0.02214915, -0.009897436, 0.006359255, -0.018861279, 0.0049284226, -0.015086317, 0.007022241, -0.015343393, 0.045678392, 0.021499693, -0.01638523, 0.0033403, 0.007238726, -0.034475278, -0.028088965, 0.0132394275, -0.024056926, -0.051198766, -0.02960436, 0.008713532, -0.0035855372, -0.017887095, 0.008321152, -0.019077765, 0.010147747, -0.011351946, 0.017467655, 0.006122474, -0.010242459, -0.021066723, -0.0060446747, -0.011913455, -0.015925197, -0.02750716, 0.010567187, -0.004674729, 0.009315631, 0.041808717, 0.0066230963, 0.016060501, -0.019348372, 0.0135100335, 0.0017504863, 0.0077393483, -0.0037614314, -0.016723488, -0.005496696, -0.020782586, 0.026492385, -0.02262271, 0.0052835937, 0.05387777, 0.03496237, -0.025355836, 0.012495259, 0.01849596, -0.011243704, 0.012684683, -0.028684298, 0.0068057557, -0.008070841, -0.023258636, -0.008057311, -0.011683439, 0.014193315, -0.015221621, 0.021039663, -0.008990903, -0.021215556, -0.024489896, -0.027439509, 0.04605724, 0.015167499, 0.012786161, -0.014355679, 0.014937484, -0.03983329, -0.0151945595, 0.022338573, -0.055582594, 0.024462836, -0.04435242, -0.055690836, 0.005970258, 0.055311985, 0.011994637, 0.015532818, -0.0030223373, -0.026790053, -0.029658481, 0.0189154, 0.0112301735, 0.0043702964, -0.02833251, -0.031417426, -0.005919519, -0.043838266, -0.0025081849, 0.016114622, -0.011839038, -0.0037309884, -0.06310546, 0.015749304, 0.041646354, -0.020444328, -0.009965087, 0.0020549188, 0.029198451, 0.004055716, 0.006545297, -0.021946194, 0.0133612, -0.009938027, -0.027845418, -0.023651015, -0.019673098, 0.049845733, -0.011839038, -0.068950556, -0.038534377, 0.02097201, 0.0018908635, -0.008104667, -0.00031669426, -0.010147747, -0.05263298, 0.038317893, 0.009173564, 0.0051144646, -0.027710114, -0.005929667, 0.012806457, 0.012488494, -0.015248681, -0.022013845, -0.009430639, 0.0039711515, 0.010918976, -0.0013682545, -0.033419915, 0.019618977, 0.0023390558, 0.0034553078, 0.0049284226, -0.043675903, -0.00015232191, -0.023042152, -0.04548897, 0.026654748, 0.011385772, 0.021026133, -0.002799087, 0.008794714, 0.013307079, 0.006751634, -0.01028305, -0.0018502725, -0.0184689, -0.021188496, 0.003659954, 0.011155756, -0.0034705296, 0.0060209967, 0.014666877, 0.0052497676, 0.005571113, 0.029766724, 0.031282123, -0.06115709, 0.012197592, 0.012082584, -0.005807894, -0.0074619767, 0.0051246122, -0.0058011287, 0.03780374, -0.017007625, -0.0011559975, 0.028927844, 0.046842, -0.01049277, -0.029009026, 0.0002490426, -0.009985383, -0.008828539, 0.004211315, 0.0295773, -0.004448096, 0.0068869377, 0.0040455684, -0.009620064, 0.0066298614, 0.0050163697, 0.044677146, -0.05417544, 0.04199814, -0.008341448, -0.00905179, 0.01292823, -0.009958322, 0.0059431973, -0.001980502, -0.008551168, -0.0021969872, -0.0011982798, -0.0025403192, -0.008368509, -0.011818742, -0.007949068, -0.010364233, 0.01887481, 0.009620064, -0.013354435, -0.0029343902, -0.008327918, -0.016669367, -0.115603134, -0.019632507, -0.0020955098, -0.026208248, 0.001161917, -0.023556303, 0.029252572, -0.005185499, -0.00081731647, -0.034691766, 0.02546408, -0.03001027, 0.011372242, -0.0011982798, 0.01767061, -0.05790981, 0.01645288, -0.004735615, 0.04724791, -0.03788492, -0.019754281, -0.0018807157, 0.007651401, 0.014152724, -0.053309496, -0.013117654, 0.0005027363, 0.010614543, 0.013110889, -0.019010114, -0.048059728, 0.04738321, 0.008395569, 0.022988029, -0.0027804826, 0.012427608, -0.025139352, -0.0022409607, 0.028819602, 0.008713532, -0.019943705, -0.016209334, -0.0034942075, -0.02795366, 0.009741837, 0.009254745, 0.0044413307, 0.0026570184, -0.0072049005, 0.03201276, 0.035882432, -0.012522319, 0.014477452, 0.021215556, 0.01273204, 0.01480218, 0.002027858, -0.017007625, 0.023975743, 0.009390049, 0.0013107506, 0.044677146, 0.0036701018, -0.004431183, 0.014166255, -0.0071981354, 0.04762676, 0.019010114, -0.01806299, -0.010127451, -0.03699192, 0.027195962, 0.00025707626, -0.008192615, -0.018834218, -0.0017640167, 0.04405475, 0.060128782, 0.00014058859, 0.003856144, -0.0010130834, -0.0067414865, -0.030740907, 0.01683173, 0.009031495, 0.037289586, 0.018698916, -0.017778853, 0.025734687, 0.018387718, 0.016087562, -0.025612913, -0.012880874, 0.024557548, -0.030740907, -0.0031779362, -0.025707625, 0.048682123, 0.02379985, -0.008219675, 0.04199814, 0.017413534, -0.021459103, -0.021107314, 0.0045563383, 0.011521076, -3.5517114e-05, 0.010188338, -0.022974499, -0.005814659, 0.012461433, 0.019659568, 0.0047491454, 0.037966102, -0.02421929, 0.040888656, 0.07333439, 0.0043838266, 0.0048573883, -0.0018638029, 0.02841369, -0.021215556, -0.0061867433, 0.032716338, -0.034367036, 0.0035753895, -0.009390049, 0.040374503, 0.0011796756, -0.0032844874, -0.027141841, 0.0011712192, 0.0019263807, 0.004904744, -0.018157702, 0.003815553, 0.011061044, -0.031850394, -0.016737018, -0.003369052, -0.0074890372, -0.0036396587, -0.013205602, -0.010546892, -0.0133612, -0.019889584, 0.014139194, -0.039562684, 0.028170146, 0.006210421, 0.029009026, -0.01689938, -0.016912911, 0.02465226, 0.0084023345, -0.0085105775, -0.010790437, -0.016777609, -0.006420141, 0.0029614507, 0.022879787, 0.03940032, 0.031119758, -0.003734371, 0.021432042, -0.024462836, -0.009437405, 0.019808402, -0.0102492245, 0.0024472983, -0.037722558, 0.049142156, 0.02911727, 0.02256859, 0.036234222, 0.0279266, 0.0010130834, 0.032256305, -0.0056827385, 0.019821933, -0.004153811, 0.041727535, 0.015302802, 0.03001027, 0.02586999, -0.026979476, -0.0295773, 0.0035990677, -0.013334139, 0.017345883, -0.025951171, 0.027737174, -0.017129397, 0.005814659, 0.023881031, -0.021810891, 0.031417426, -0.017751792, -0.04524542, -0.0006803219, -0.025504671, 0.008260266, -0.010702491, -0.004573251, -0.0033825822, 0.02960436, 0.006000701, 0.008165554, 0.002212209, 0.014734529, -0.02470638, 0.008767653, -0.021526754, 0.0010502918, -0.0029952766, 0.029631421, 0.012542616, 0.01190669, -0.019213067, -0.013746815, -0.07035771, -0.03401525, -0.024584608, -0.0026790053, -0.014085073, -8.2661856e-05, 0.007563454, 0.027737174, -0.0046578157, 0.018130641, 0.011541371, -0.028034842, 0.033257548, 0.002758496, 0.0117105, 0.025761748, 0.0020549188, 0.005290359, 0.002799087, -0.0011390846, 0.045705453, 0.032986943, 0.020403737, -0.013374731, 0.0016752239, -0.015681652, 0.014951014, -0.014030951, 0.019497205, -0.019618977, 0.0052768285, 0.020214312, 0.012921465, -0.03810141, 0.015343393, 0.016276985, 0.028711358, 0.035422403, 0.019970765, -0.015262212, -0.0014984839, 0.036829557, 0.01189316, -0.01638523, 0.024557548, -0.009606534, -0.031877454, 0.00087270624, 0.025301715, -0.008280561, -0.008821774, 0.030632665, -0.013489738, 0.014355679, -0.015708713, 0.012880874, 0.017846504, -0.028061902, 0.038940288, -0.0051956465, -0.022933908, 0.006426906, -0.01727823, -0.0030764586, 0.011385772, -0.004948718, -0.023461591, -0.0020430798, 0.027236553, -0.0052632983, 0.0038764393, 0.02631649, 0.0058856932, -0.017616488, -0.021188496, 0.019862523, 0.0037242232, -0.0029293164, 0.01685879, -0.048140913, 0.01233966, -0.006602801, 0.0117713865, -0.058938116, 0.000501045, 0.016547592, 0.035097674, -0.054554287, -0.0076987576, 0.0070493016, -0.039643865, -0.0037614314, -0.0001614126, 0.020011356, 0.0205661, 0.044866573, 0.0008921561, -0.002090436, -0.0035043552, 0.0021750005, -0.011866099, -0.006616331, 0.0018265945, -0.008415865, -0.0005141525, -0.02425988, -0.012765866, -0.00014809368, 0.0030967542, 0.020200782, 0.022527998, -0.004617225, 0.0126982145, 0.019037174, -0.03458352, -0.03155273, -0.042566415, -0.01933484, 0.007881417, -0.018712446, 0.0063355765, 0.015884606, -0.019497205, -0.010797203, 0.024354592, -0.0074146204, 0.006139387, -0.0012633945, 0.025545262, 0.01888834, 0.05176704, -0.019632507, -0.02673593, -0.014260967, -0.0060852654, 0.032635156, -0.012779396, -0.008544403, -0.022365633, -0.019158946, 0.021188496, 0.0028481344, -0.032364547, 0.023163924, 0.015505757, -0.01399036, -0.031309184, -0.03174215, 0.038750865, -0.0015551422, 0.016588183, 0.043675903, -0.0075905146, 0.0041402807, 0.018983051, 0.03582831, 0.010323641, -0.015911667, 0.011196348, 0.015614, -0.000801672, -0.057747446, -0.0012735423, 0.028278388, -0.006382933, 0.021418512, -0.00034756033, 0.015343393, -0.040401563, 0.029414935, 0.0050163697, -0.011845803, 0.02798072, -0.022311512, 0.02260918, -0.000114796385, 0.0069545894, -0.0012777705, 0.008557933, -0.013746815, -0.0067110434, -0.006278073, -0.013056768, -0.0024236203, 0.040509805, -0.03986035, 0.006359255, 0.02712831, 0.028981965, 0.019294249, -0.047491457, 0.004931805, -0.022487408, 0.017832974, -0.004275584, -0.05910048, -0.020065479, 0.012434373, 0.018671855, 0.033825822, -0.0077528786, -0.017481185, 0.0018756419, -0.028927844, 0.04343236, 0.010946036, -0.014152724, -0.011541371, 0.014423331, 0.027195962, -0.02633002, -0.021634996, -0.040753353, -0.0022561825, -0.018983051, -0.0067279562, 0.049466886, -0.003659954, -0.015762834, 0.010776907, -0.0030798411, 0.01968663, 0.03277046, 0.052524738, -0.0071643093, 0.0077393483, -0.01645288, -0.018225353, -0.0035483288, 0.006721191, 0.011636083, 0.0044886866, -0.004549573, 0.006464115, -0.017954746, -0.022758015, -0.004251906, -0.0034874424, 0.039346196, 0.018590672, -0.014693937, 0.02213562, 0.010621308, -0.011142226, 0.0052599157, 0.01970016, -0.012332895, 0.06045351, -0.00036827865, -0.055230804, 0.030903272, -0.0032455877, 0.022798605, -0.010398058, 0.0062814555, -0.013070298, 0.015289272, -0.01887481, -0.041592233, -0.0074958024, -0.010384528, 0.02585646, 0.0068564946, -0.019537795, 0.005175351, 0.0023999421, -0.019199537, -0.031038575, -0.0068159034, 0.012630562, 0.020714935, 0.03458352, -0.00057757593, -0.017142927, 0.00046595072, -0.014937484, 0.016534062, -0.008733828, -0.0031982316, -0.0032405138, -0.02543702, -0.041619293, 0.019632507, -0.019145416, -0.028034842, -0.041727535, -0.0205661, -0.006988415, -0.008273796, -0.021986784, 0.00924798, 0.006788843, -0.02960436, -0.0037851096, 0.0035517113, -0.014369209, -0.00033318435, 0.017575897, -0.0020599926, 0.018117111, 0.013158245, -0.013564155, -0.011169286, 0.0035415636, 0.027277144, -0.01397683, 0.008733828, 0.014179785, -0.026384141, 0.021486163, -0.008280561, 0.014139194, 0.003154258, -0.02542349, -0.0172647, 0.02835957, -0.026925355, 0.0068632597, -0.0034772947, -0.0053986013, -0.025125822, -0.051144645, -0.03574713, -0.014883362, 0.0008625585, 0.028440753, -0.02055257, 0.012238183, 0.039562684, -0.00905179, -0.010756612, -0.012190826, -0.025396429, -0.0014401345, -0.051631738, 0.015343393, -0.007847591, 0.011967576, 0.0012473272, 0.018698916, -0.02547761, -0.02589705, -0.036667194, -0.0030477066, 0.0509011, -0.03739783, 0.02059316, -0.025288185, -0.0056895036, -0.033122245, 0.023218045, -0.0054797833, 0.02789954, 0.00801672, -0.009092381, 0.0027111396, -0.004143663, -0.030632665, 0.020457858, 0.014193315, -0.007766409, -0.018198293, 0.024138108, 0.019740751, -0.03157979, -0.007962598, -0.033257548, -0.0046273726, 0.0006900468, 0.011724031, -0.017968277, 0.02914433, -0.009166799, 0.032148063, -0.01399036, 0.031444486, 0.029766724, 0.004424418, -0.042701717, 0.0039339433, 0.010134216, 0.027250083, -0.028170146, -0.005902606, -0.024760503, -0.0148157105, -0.011006922, -0.0076581663, 0.0025606148, 0.02097201, -0.011954046, -0.0039001175, 0.00698165, 0.0029715986, -0.005983788, 0.02470638, 0.012644093, -0.022920378, -0.009782428, 0.013658867, -0.0056489124, -0.02214915, -0.02260918, -0.008442925, 0.007522863, 0.024828155, -0.0023948683, 0.010431884, 0.019821933, -0.01809005, 0.013855057, -0.0073604994, -0.014179785, -0.023421, 0.0189154, 0.036856618, 0.02628943, -0.017332353, -0.0075296285, -0.011000157, -0.012393782, 0.033095185, 0.019970765, -0.014288028, -0.0017995338, 0.014937484, -0.0072996127, -0.022067968, -0.02467932, 0.030362058, -0.007888182, -0.015735773, 0.02589705, 0.010127451, 0.035016492, 0.0031694795, 0.0029428466, 0.027669523, -0.028278388, -0.005547435, 0.009423874, -0.029820846, -0.0146533465, -0.015668122, 0.0168182, -0.005415514, 0.011507545, 0.02750716, -0.017142927, 0.024949927, -0.013246193, 0.016344639, 0.016980562, -0.0058248066, 0.053201254, -0.0063558724, -0.00017969968, 0.028873723, 0.022771545, -0.035476524, 0.022122089, 0.018130641, 0.04345942, 0.0009420492, -0.008612054, -0.025937641, -0.025125822, 0.0045394255, 0.047031425, 0.00081181974, -0.0037851096, -0.017048215, -0.035232976, 0.0038324657, -0.030876212, 0.03401525, -0.019591916, -0.029414935, -0.0072522564, -0.00095642515, 0.0068801725, 0.015573409, -0.0058011287, 0.0020126365, 0.0032049967, 0.0026282666, -0.0032929438, 0.010817498, -0.01845537, -0.029793786, -0.045434847, -0.019781342, 0.039183833, 0.011379007, 0.0028565908, 0.033095185, -0.03120094, -0.037668437, 0.0011069501, -0.026898295, -0.0029462292, 0.031309184, 0.03609892, -0.03896735, -0.010215399, -0.004593547, -0.027412446, -0.028576056, 0.018198293, -0.002689153, 0.0042011673, -0.017197048, -0.009288571, 0.0047998843, 0.024029864, -0.02300156, -0.021202026, 0.017819444, 0.019835463], "content_hash": "60e54432b8b413857ec5318000dad2d3"}, {"content": "# OPT Overview\n\n## What Is OPT?\n\nOPT (Open Pre-trained Transformer) is a family of language models released by Meta in 2022. OPT was designed to replicate GPT-3's architecture and performance while being openly available to researchers. It uses a decoder-only transformer architecture similar to GPT-2 but with options for much larger sizes.\n\n## Architecture Details\n\nOPT's architecture is close to GPT-2 but has some differences:\n\n| Property | OPT-125M | OPT-350M | OPT-1.3B |\n|----------|----------|----------|----------|\n| Parameters | 125M | 350M | 1.3B |\n| Layers | 12 | 24 | 24 |\n| Attention Heads | 12 | 16 | 32 |\n| Hidden Dimension | 768 | 1024 | 2048 |\n| Vocabulary Size | 50,272 | 50,272 | 50,272 |\n\n### Key Differences from GPT-2\n\n- **ReLU activation**: OPT uses **ReLU** instead of GPT-2's GELU. This is the only model in the dashboard with ReLU, making it useful for comparing how activation functions affect MLP behavior.\n- **Learned positional embeddings**: Like GPT-2, OPT uses learned absolute position embeddings (unlike Pythia's or Qwen's RoPE)\n- **LayerNorm placement**: OPT uses pre-norm LayerNorm (applied before each sublayer), which is slightly different from GPT-2's original arrangement\n- **Larger variants available**: OPT scales up to 175 billion parameters, though only smaller variants are practical for interactive use\n\n### Similarities to GPT-2\n\n- Same general decoder-only architecture\n- Same tokenizer style (BPE with ~50K vocabulary)\n- Same attention mechanism (standard multi-head self-attention)\n- Similar training objective (next-token prediction)\n\n## What to Expect in the Dashboard\n\nWhen using OPT models:", "source_file": "opt_overview.md", "chunk_index": 0, "embedding": [-0.0037119514, 0.027892755, 0.026682267, -0.009793371, -0.0062874597, 0.0046970835, -0.017539212, 0.03415124, -0.032245364, 0.011454574, -0.0002239485, -0.029000225, -0.037602425, -0.025755085, 0.018775456, -0.037113078, 0.0027219905, -0.008370402, -0.031163651, 0.05187074, 0.02284476, -0.019432211, -0.041465685, 0.005225063, -0.03100912, -0.0016933968, -0.01912315, 0.05228282, 0.016367355, 0.014654643, -0.0038020944, -0.027763981, -0.02755794, -0.046925765, 0.015478806, -0.018582294, 0.032760467, 0.013012757, 0.005128481, 0.04651368, -0.01833762, -0.023617413, -0.052849434, 0.048136253, -0.03505267, 0.044092704, 0.024505964, -0.034408793, -0.043088257, 0.026180044, -0.01676656, 0.077265255, -0.04494262, 0.006580424, -0.009613086, -0.019560987, -0.05045421, -0.009754738, -0.007507607, 0.010797819, 0.007089087, -0.0071856687, 0.042779196, 0.004809762, 0.040383972, -0.011036053, -0.06696322, 0.028536633, 0.010862207, 0.019251926, 0.016238581, 0.0039759413, -0.012265859, -0.015169744, 0.020938883, 0.0036958547, -0.0036958547, 0.056146085, -0.006780026, -0.057742897, 0.005662899, -0.0049288794, -0.019947313, -0.050222415, 0.008846872, 0.007365954, -0.012639307, -0.04133691, -0.025111208, -0.020526802, -0.020037455, -0.014551623, -0.027532184, 0.013225236, -0.005878598, -0.024621861, 0.0062906793, -0.014448603, -0.010887962, 0.041826256, -0.012581359, 0.005041558, 0.018736824, 0.028948715, 0.0011163219, -0.003515569, 0.0144743575, -0.013856235, -0.015839377, -0.009239636, -0.105080746, -0.006831536, -0.018479273, 0.0013537516, -0.005463297, -0.026064144, 0.0011098832, 0.02043666, 0.012001869, -0.05830951, -0.04494262, 0.0048001036, 0.042882215, 0.028382102, 0.020449538, 0.007951883, 0.014165296, -0.007823107, -0.0041948594, -0.006831536, -0.0028378884, 0.031730264, 0.011641298, -0.046024334, -0.028588144, -0.0067542708, -0.063512035, 0.013135093, -0.02211074, -0.014551623, 0.013495664, -0.046333395, 0.007514046, 0.032811977, -0.09009129, -0.012516971, 0.005170333, 0.00013118995, -0.08658859, 0.06279089, 0.009110861, -0.0032515794, -0.023308352, -0.036546465, -0.015440173, -0.017938416, 0.017899783, 0.041465685, 0.03348161, -0.01770662, 0.03667524, 0.0028958372, -0.015105356, 0.024325676, -0.0054890523, 0.027995776, -0.05465229, -0.010141064, -0.060421426, -0.06119408, 0.049990617, 0.016830947, -0.007584872, -0.004175543, 0.0055405623, 0.0055148075, -0.042418625, -0.005157456, -0.028253328, -0.027480675, -0.011853778, 0.0069409953, -0.0139077455, 0.03925075, -0.014281195, -0.0039373087, -0.04981033, 0.040100668, -0.0014366508, 0.04344883, -0.0022648377, 0.0042721247, 0.060678978, 0.009297585, -0.013572929, -0.04458205, 0.065417916, 0.012555604, -0.007623505, -0.005720848, 0.00361537, 0.00659974, 0.045045644, -0.0587731, 0.03688128, -0.013289624, -0.01042437, -0.0014929901, -0.036932793, -0.005051216, 0.010179697, 0.00461016, 0.014680398, 0.0065063783, -0.00042254434, -0.059854817, -0.05480682, 0.034614835, 0.033224057, 0.029798632, -0.011518962, 0.005324864, 0.024493085, 0.029798632, -0.00031952403, 0.037962995, 0.021672903, -0.0014269926, -0.0017207615, -0.05712478, -0.021041904, -0.032477163, -0.044504784, 0.0010599827, -0.0041272524, -0.033610385, 0.030081939, -0.0176036, 0.003148559, -0.056455143, 0.01802856, 0.02267735, -0.03420275, -0.016367355, 0.026836798, 0.006780026, -0.031704508, -0.080561906, 0.0041143745, 0.009838442, 0.017255906, 0.002393613, -0.028536633, -0.023810575, -0.009175248, 0.012999879, -0.02069421, 0.034228507, -0.051844984, 0.025677819, 0.005675777, 0.005730506, -0.019470844, 0.02084874, -0.030391, 0.020449538, -0.017951293, 0.0007392514, -0.06160616, -0.04981033, 0.04453054, -0.029669857, -0.013418399, 0.0064162356, -0.04579254, -0.009291147, -0.004333293, 0.005495491, -0.02221376, 0.023115188, -0.046565194, -0.02299929, 0.07520485, 0.03149847, -0.002268057, -0.02509833, -0.005076971, -0.015659092, -0.008479862, -0.020372272, 0.0039373087, -0.046719722, 0.039199237, -0.002535266, 0.009838442, 0.016444622, 0.024621861, -0.016599152, 0.036211647, -0.011911727, 0.028150307, -0.04342307, -0.013469909, -0.022973536, 0.037319116, 0.019393578, 0.016470376, 0.033275567, -0.009651718, -0.013090022, -0.010121748, -0.016779438, 0.03185904, -0.0032338728, -0.011602665, 0.030468265, 0.014886439, -0.0068572913, 0.0072693727, -0.026257308, -0.044865355, -0.048754375, -0.012819593, 0.0026109216, -0.0118409, 0.028871449, -0.022651596, 0.00017756924, -0.011042492, -0.015684847, 0.049063437, -0.04878013, 0.008241627, 0.006715638, -0.018620925, 0.039894626, 0.009452116, 0.028716918, 0.02059119, -0.0018044655, -0.029927408, -0.017230151, -0.033816427, -0.012046941, -0.06912664, 0.026553491, -0.036494955, 0.03332708, 0.058258, 0.022033475, 0.0014455041, -0.041568708, 0.015865132, 0.0017819299, 0.04391242, 0.017964171, 0.011873093, -0.007681454, -0.052205555, 0.012652185, -0.04447903, 0.0073595154, 0.0022197664, -0.0025079013, -0.005109165, -0.02363029, 0.002016945, 0.032039326, -0.023462882, 0.012066257, 0.04169748, 0.027841246, -0.038426585, 0.014783419, -0.055476453, -0.0028539852, -0.018311866, -0.0005774773, -0.022587208, 0.032090835, 0.01755209, 0.02147974, -0.012690818, -0.006512817, -0.021788802, 0.017165763, 0.01750058, -0.031163651, -0.046668213, 0.020282129, 0.0041143745, 0.0096452795, -0.048393805, -0.005408568, -0.0035542017, 0.021930454, -0.024827901, 0.026115656, 0.01729454, 0.0453547, -0.00031972522, -0.015710602, -0.048368048, -0.015762111, -0.02766096, -0.021814557, 0.059854817, 0.007539801, 0.022806127, 0.015556071, 0.06284241, 0.013238113, 0.020835863, -0.048084743, 0.035232954, -0.014332704, 0.021260822, 0.030313734, 0.0243128, 0.0455865, 0.0018430982, -0.002379126, -0.06783889, -0.0019622154, -0.07937717, 0.011113319, 0.04898617, 0.014487235, -0.022072107, -0.020359393, -0.009265391, 0.032348387, 0.042418625, -0.02472488, -0.016985478, -0.01613556, 0.01514399, 0.039534055, 0.0123109305, 0.019020129, 0.031446956, -0.014538745, 0.00083140633, -0.04285646, -0.021827433, -0.0032853829, -0.009400606, -0.036804017, -0.001367434, -0.047440864, 0.017629355, 0.0698993, 0.008634392, -0.011564033, -0.0027541844, -0.057433836, 0.049063437, -0.019715518, 0.08164362, 0.0008104803, 0.038452342, -0.035902586, -0.0047131805, -0.042032298, 0.004291441, 0.01980566, 0.022574332, -0.011467451, -0.035438996, 0.0049481955, 0.029206265, -0.040847562, -0.023063678, 0.0028523756, 0.029721368, -0.02059119, 0.035026915, -0.0074367807, -0.0064258934, -0.04159446, 0.04669397, 0.028407859, -0.022741739, 0.011042492, 0.019264802, 0.13279322, -0.020256374, 0.006895924, -0.016071172, -0.009181688, -0.030880347, -0.007829546, 0.035284467, -0.014126664, 0.014165296, -0.0092460755, -0.021093413, -0.041208133, -0.026283063, 0.0098770745, 0.038838666, -0.017371804, -0.040718786, 0.063872606, 0.026051268, 0.038220543, -0.042264093, -0.0016370575, -0.013972133, 0.023012167, -0.026424717, 0.0040854, -0.0028153528, 0.009387728, -0.00017102985, 0.041414175, 0.013431276, 0.028742675, 0.0054407613, 0.03778271, -0.008634392, 0.010508074, -0.030107694, 0.0057337256, 0.0026334573, -0.019393578, -0.058258, -0.12280024, -0.029257776, 0.016483255, 0.010224769, -0.040306706, 0.043963928, -0.016470376, -0.057330817, -0.013856235, -0.0036958547, -0.012336685, -0.04092483, -0.0017481263, 0.012697256, 0.014834928, 0.008595759, 0.018247478, 0.00047485938, 0.0071277195, 0.017217273, 0.0111777065, -8.953916e-05, 0.038812913, 0.025072575, -0.0077973516, 0.016727928, -0.013379767, -0.023849208, -0.023025045, -0.0024982432, -0.016972601, 0.030674305, -0.0055502206, 0.0047163996, -0.0018785115, -0.003483375, -0.0055566593, 0.007675015, 0.022239516, 0.011712125, -0.026270187, 0.006715638, -0.007533362, -0.012471899, -0.023037923, 0.012723012, 0.013225236, -0.0016273994, 0.0061361487, 0.018080069, -0.0055470015, -0.0032821635, -0.0009980095, -0.021157801, -0.026630757, 0.041259646, 0.041671727, -0.02892296, 0.030545529, -0.014899316, 0.00948431, -0.012478339, 0.045509234, 0.035335977, -0.022072107, 0.00023099092, -0.003969502, -0.02011472, -0.01838913, 0.053106982, -0.013128654, -0.014847806, 0.009020718, 0.0012491216, -0.004909563, -0.00937485, -0.016161315, -0.013997888, 0.013051389, -0.0064065773, 0.019290559, -0.008074219, -0.00059276936, -0.0011187365, -0.013418399, -7.917676e-05, 0.008048464, -0.057742897, 0.0012000259, -0.02169866, -0.011757196, 0.03044251, 0.024608983, 0.0104758805, -0.022741739, -0.0062005366, 0.023462882, -0.014088031, -8.1138576e-05, -0.0060492256, -0.016199948, -0.009027157, 0.03742214, -0.0050801905, -0.0037473647, 0.048831638, 0.011898849, -0.002535266, 0.016058294, 0.0074947295, -0.023295473, 0.050943557, -0.011312921, -0.0352072, -0.002195621, -0.029592592, -0.011132635, 0.008125729, -0.030056184, -0.03211659, 0.031163651, -0.025149839, 0.0015622068, -0.007591311, -0.055940043, 0.044839602, 0.013128654, 0.012671501, -0.0060202507, -0.0019155344, -0.019457966, -0.032296877, 0.019264802, -0.06783889, -0.029051734, 0.0027912073, -0.045972824, 0.010340666, 0.036700994, 0.00057627, 0.020655578, 0.032811977, -0.035464752, -0.05748535, 0.0038857984, 0.033919446, 0.018814089, 0.013572929, -0.016882457, 0.015671968, -0.045122907, -0.010379299, 0.01844064, -0.011403063, 0.010623972, -0.059854817, -0.0050834096, 0.0061232713, -0.0023082995, 0.008531372, -0.010089554, 0.013804725, 0.019715518, -0.003792436, -0.023462882, -0.000573453, -0.030184958, -0.015414418, -0.02415827, 0.026128532, 0.03865838, -0.008363964, -0.053982656, -0.009471432, 0.025433145, -0.051587433, -0.013225236, -0.008248066, 0.0061039547, -0.01985717, 0.042907972, 0.005424665, -0.026914064, -0.033867937, -0.008164362, -0.01477054, 0.00080605363, 0.012697256, 0.0054987106, -0.0010237646, -0.0071792295, 0.016174193, 0.0041787624, -0.012426828, 0.02158276, -0.023153821, -0.009452116, 0.015659092, -0.00088050193, -0.014294072, 0.013418399, -0.07623505, 0.03935377, 0.027867, -0.0047807875, -0.03855536, -0.030519774, 0.008048464, -0.00010966032, -0.0053989096, 0.00065916916, -0.008782484, 0.0053892513, 0.009239636, 0.014654643, -0.013972133, 0.008421913, -0.005035119, 0.008634392, -0.0021135267, 0.029438062, 0.023282597, -0.037602425, -0.010971666, 0.010347105, -0.012961246, 0.004516798, -0.031936303, 0.0065192557, 0.035335977, 0.005070532, -0.028485123, 0.0075462395, 0.0022777154, 0.0026109216, -0.01048232, 0.02132521, -0.03819479, 0.00041328862, -0.013663072, -0.02038515, 0.011280727, 0.010920156, 0.034923892, -0.006947434, 0.011023176, -0.012381757, 0.012001869, -0.015646214, 0.044195723, -0.020861618, -0.025716452, 0.0061586844, -0.019792782, -0.014577378, 0.02639896, -0.0007947858, 0.0028668628, -0.01199543, -0.017874029, -0.0032660665, -0.0061168326, -0.023720432, -0.028871449, 0.0043461705, 0.026424717, 0.0033127477, -0.0025867762, 0.005035119, -0.010572462, -0.08401308, -0.013128654, 0.011660614, -0.025922492, 0.00094086543, -0.016122682, 0.036572218, -0.010881523, 0.0016805192, -0.03840083, 0.038761403, -0.06979628, 0.004719619, 0.00024688663, 0.03600561, -0.056352124, 0.019522354, -0.027120104, 0.026077023, -0.010192575, -0.019985946, 0.013302501, 0.010469441, -0.019728394, -0.059803307, -0.008241627, 0.022329658, 0.028073043, -0.007848862, -0.014255439, -0.054085676, 0.048960414, -0.0038310688, 0.038220543, -0.0023082995, -0.023321228, -0.0003177131, -0.035799567, 0.016238581, 0.0059494246, 0.010031605, -0.02634745, 0.026656512, -0.035129935, 0.023694677, 0.0011726612, 0.027171614, 0.01650901, -0.015517438, 0.00612971, 0.02320533, 0.02111917, 0.007514046, 0.009097983, 0.014216807, 0.0021489398, 0.019445088, -0.0028040847, 0.044736583, 0.004532895, 0.024325676, 0.05066025, 0.015156867, -0.014719031, -0.019960191, -0.0010116919, 0.036597975, 0.025664942, -0.0027638425, -0.025742207, 0.0039791605, 0.045097154, -0.013122216, -0.0236818, -0.019303435, -0.024815023, 0.019161783, 0.015646214, -0.014616011, -0.015852254, 0.05014515, -0.0012764863, 0.008969208, 0.030339489, 0.019664006, 0.035773814, -0.010591778, -0.016792314, 0.025394512, -0.008872626, 0.0076943315, -0.009767615, -0.001000424, -0.0059719603, -0.026605003, -0.013057828, 0.0067349547, 0.021492617, 0.044865355, -0.02084874, 0.04939825, 0.02122219, -0.032477163, -0.013354011, -0.010514514, -0.0041787624, -0.020938883, -0.015787866, -0.017796762, -0.00049498054, -0.007134158, 0.02530437, 0.013997888, 0.03456332, -0.0025417048, 0.018736824, 0.07695619, 0.002293812, 0.030931856, -0.0065224753, 0.0057723583, 0.012336685, -0.020295007, 0.02221376, -0.028871449, -0.008421913, 0.010083116, 0.031653, 0.002704284, -0.010166819, -0.0017545652, -0.0029248118, -0.0032419213, 0.008653709, -0.038220543, 0.026527736, 0.026656512, -0.0066190567, -0.009040034, 0.030880347, -0.009806248, 0.001294193, 0.0056725573, -0.011518962, 0.040306706, -0.0032515794, -0.023656044, -0.0453547, -0.034692097, 0.0069281175, 0.04821352, 0.023514392, -0.015427295, 0.049372498, 0.015633335, 0.016212825, -0.026450472, -0.02619292, -0.056918737, 0.012652185, 0.008917698, 0.016431743, 0.02269023, -0.0066770054, 0.03667524, -0.033970956, -0.027763981, 0.031137897, 0.020707088, -0.012484777, -0.0059655216, 0.01859517, 0.021505496, 0.028665408, 0.033146795, 0.034666345, -0.009658157, 0.031240918, -0.0071277195, 0.0025642405, 0.025136963, 0.03353312, 0.011647737, 0.023269719, 0.028691163, -0.004066084, -0.029644102, 0.0042431504, -0.020707088, 0.032786224, 0.012478339, 0.020951761, 0.007977637, -0.008074219, -0.035284467, -0.021544129, 0.008473422, -0.031060632, -0.039431036, -0.00052516226, -0.01780964, 0.013302501, -0.015169744, -0.02740341, -0.0063872607, 0.017771007, -0.005878598, 0.016418867, 0.007449658, -0.0029264214, 0.005340961, 0.031060632, 0.009574452, 0.02069421, 0.0012201471, -0.000853942, 0.0046648895, -0.029566837, -0.0382463, 0.014525867, -0.059700284, -0.032296877, -0.0043558287, -0.028536633, 0.0055727563, 0.006664128, 0.007900372, 0.010347105, 0.0017642232, 0.017024111, -0.0230508, -0.004674548, 0.00833177, -0.00031509736, 0.02378482, 0.021338087, -0.0061586844, -0.0016595932, 0.02132521, 0.024686249, 0.009387728, 0.020166231, -0.0056146085, -0.0032387017, -0.003740926, -0.001173466, 0.012330246, 0.022497065, -0.0015308178, -0.026605003, 0.0011010298, 0.03059704, 0.011203461, -0.0142425615, 0.03788573, 0.017268784, 0.023308352, 0.04223834, 0.013843358, 0.017732376, -0.028794184, 0.070568934, 0.021672903, -0.018298987, 0.020861618, 0.0023501515, 0.016315846, -0.004185201, 0.045509234, -0.013354011, -0.028124552, 0.04316552, -0.014062276, 0.011390186, 0.020822985, -0.005575976, 0.026605003, 0.0038375075, 0.016315846, 0.010797819, -0.0078102294, -0.011235655, 0.0017577845, 0.00091752486, -0.014834928, -0.016109806, -0.00476791, -0.023282597, 0.010669043, 0.022278149, -0.005569537, 0.062378813, 0.0059848377, -0.033610385, -0.0067542708, 0.013044951, 0.0002670078, 0.0034865944, 0.010933033, -0.019998822, 0.02583235, -0.0054504196, -0.006799342, -0.028356347, 0.009619524, 0.0151955, 0.009091545, -0.06577849, 0.0065417914, -0.012252982, -0.0144743575, -0.023475759, 0.01948372, 0.012980563, 0.017629355, 0.015079602, 0.0017272003, -0.02451884, 0.011847339, -0.00565968, -0.00282984, 0.000704643, -0.009844881, 0.0003489009, 0.003227434, -0.013598684, 0.02205923, 0.009123739, 0.012845349, 0.007172791, 0.021054782, 0.028227571, -0.0031179748, 0.016380234, -0.045431968, -0.008447668, 0.0011452964, -0.018002804, -0.0059365467, -0.018685313, 0.018685313, 0.003625028, -0.007816668, -0.026180044, 0.015865132, 0.002070065, 0.009040034, 0.020101843, 0.04599858, 0.018685313, 0.053364534, -0.0243128, -0.031369694, -0.0052508176, -0.026450472, 0.027223123, -0.015311398, -0.00025553873, -0.015671968, -0.016290091, 0.01084933, 0.047775682, -0.003573518, -0.012826032, 0.037319116, 0.005479394, -0.034460302, 0.014731908, 0.046256132, 0.010836451, -0.014628888, 7.218466e-05, -0.03415124, 0.013392644, -0.008061341, 0.023540147, 0.0020249935, -0.032605935, 0.032992262, 0.01770662, -0.0044459715, -0.046256132, 0.008228749, 0.03353312, 0.017758131, 0.0002931653, 0.019342069, 0.013289624, -0.03631467, -0.0048162006, -0.010598217, -0.019007253, 0.0075591174, -0.03049402, 0.006213414, 0.001870463, 0.0002601666, -0.021029027, -0.0011855387, -0.0021698659, -0.011757196, 0.013263868, -0.03793724, -0.013469909, 0.026501982, -0.026102778, 0.020513924, 0.0236818, 0.008286699, 0.0025996538, -0.034923892, 0.034717854, -0.03373916, 0.016998356, -0.022857638, -0.05197376, -0.0043622674, 0.011750757, -0.007217862, 0.007539801, -0.008183678, 0.015169744, -0.0075462395, -0.015002336, 0.012652185, 0.009728983, -0.0011026396, 0.007539801, 0.008924137, 0.017642234, -0.011589788, -0.006638373, -0.018144457, 0.018157335, -0.024235534, 0.006348628, 0.00875029, -0.03525871, -0.047543887, 0.03448606, 0.009452116, 0.016908213, 0.017346049, 0.037396383, -0.0038697014, 0.024377188, -0.033043772, -0.022522822, 0.0058753784, 0.0088790655, 0.021157801, -0.00041892254, -0.008473422, 0.037705444, -0.014036521, -0.021647148, -0.01728166, -0.018157335, 0.027867, 0.0021988403, -0.01953523, -0.0022728862, 0.026167165, 0.00497717, 0.0022712764, 0.010778503, 0.021763045, 0.031472713, 0.015710602, -0.02153125, 0.02876843, -0.012871103, 0.009561575, -0.011564033, 0.011667053, -0.005427884, 0.002237473, -0.008776045, -0.031627245, 0.0032821635, 0.02294778, 0.0092782695, 0.014062276, -0.043603357, 0.0148091735, 0.032760467, 0.027017083, -0.021891821, -0.021505496, -0.002084552, 0.023359861, 0.022703107, -0.0005432713, 0.03100912, 0.0003943747, -0.02954108, 0.013663072, -0.050170906, -0.01876258, -0.013701705, -0.028716918, -0.029232021, -0.008930575, 0.011094003, 0.012001869, -0.04265042, -0.006429113, -0.026025513, -0.028639654, -0.01765511, 0.013650195, 0.012471899, -0.02530437, 0.03783422, 0.0060846386, -0.008924137, -0.001315119, -0.016496131, 0.00885331, 0.015517438, 0.010707676, -0.036160138, 0.008814678, 0.027429165, 0.014976582, -0.006194098, 0.004098278, 0.004423436, -0.044247236, -0.0041465685, -0.007340199, 0.024287045, -0.0062069753, -0.0007155084, 0.0055309045, 0.032554425, 0.0135085415, 0.0010438857, 0.0031244135, 0.031781774, 0.0068508526, 0.0008000173, 0.011248533, -0.0042817825, -0.005160675, 0.014834928, -0.0071277195, -0.025149839, 0.02766096, -0.015530316, -0.038529605, 0.0033932324, -0.016212825, -0.02750643, 0.0008297966, 0.03682977, -0.0033127477, 0.008840432, 0.015079602, 0.008499178, -0.0027477455, -0.02436431, -0.01686958, -0.00723074, 0.056043062, -0.01813158, -0.014757663, 0.009664595, -0.00072637387, 0.0061715622, -0.011493206, -0.021016149, 0.05197376, 0.004420216, -0.016702171, 0.011016738, -0.030468265, -0.0015179403, 0.0041980785, 0.009497187, -0.008003392, -0.03126667, 0.026682267, -0.019985946, -0.011660614, -0.010804258, 0.009252514, -0.0024338556, -0.0049127825, 0.023398494, -0.046307642, 0.020346517, -0.0036475637, 0.0187497, -0.0038310688, 0.013585807, 0.015620459, 0.018080069, -0.049372498, 0.027068593, 0.0010640069, -0.000717923, -0.025574798, -0.030081939, -0.024737759, 0.013289624, -0.05449776, 0.016238581, -0.00775228, 0.00282984, -0.015671968, -0.035438996, 0.018002804, 0.008943453, -0.014100908, 0.056712694, 0.002535266, -0.023565901, -0.014165296, 0.0023340546, -0.016624907, -0.027532184, -0.018556539, -0.0081708, 0.00524116, 0.040280953, -0.00015774989, 0.018479273, 0.0049320986, 0.005846404, 0.018002804, 0.0019300217, -0.024750635, -0.0077458415, 0.010012289, 0.0273519, 0.025600554, 0.0064097964, 0.004642354, -0.007365954, -0.027918512, 0.032940753, -0.017951293, -0.017783886, 0.0044652876, 0.027171614, -0.0048644915, -0.034383036, -0.037628178, 0.07675015, -0.026295941, -0.010508074, 0.016045418, -0.004835517, 0.017320294, 0.014165296, -0.02938655, 0.011898849, -0.013302501, -0.014937948, -0.0016338382, -0.01927768, -0.02426129, 0.0024032714, -0.02294778, -0.012536287, 0.002760623, 0.035799567, 0.016418867, 0.02634745, -0.024943799, 0.029566837, 0.012446145, 0.018376252, 0.08221023, 0.0073917094, 0.033661895, 0.02169866, 0.028588144, -0.004909563, 0.009889952, 0.004442752, 0.037756953, 0.027738227, 0.0016426914, -0.0020523584, -0.022458434, 0.00916881, 0.033301324, -0.0071019647, -0.025819471, -0.013482787, -0.0027236002, -0.022072107, 0.008228749, 0.026257308, 0.009361973, -0.04793021, -0.0012499264, 0.016702171, 0.0028588143, 0.045097154, -0.01833762, -0.003332064, 0.01440997, 0.03484663, 0.0254589, 0.008615076, -0.0095100645, -0.03484663, -0.024235534, -0.014100908, 0.0032242145, -0.009284708, -0.015762111, -0.014616011, 0.001566231, -0.028948715, -0.016534764, -0.025677819, -0.0009336218, 0.02294778, 0.028150307, -0.039173484, -0.011190584, -0.011203461, -0.00340611, -0.00054407615, 0.016933968, -0.007971198, 0.030107694, -0.037010055, -0.027763981, 0.0020990393, 0.0106432885, -0.0024982432, -0.0115254, 0.019367823, -0.016573397], "content_hash": "1e9daaac0ed5c840298fcd2bf5dbf1b6"}, {"content": "though only smaller variants are practical for interactive use ### Similarities to GPT-2 - Same general decoder-only architecture - Same tokenizer style (BPE with ~50K vocabulary) - Same attention mechanism (standard multi-head self-attention) - Similar training objective (next-token prediction) ## What to Expect in the Dashboard When using OPT models:\n\n- **OPT-125M is very similar to GPT-2**: Same number of layers (12), heads (12), and hidden dimension (768). You'll see similar attention patterns and predictions.\n- **Different module paths**: The dashboard auto-detects OPT's internal structure (e.g., `model.decoder.layers.N.self_attn`), so hooking works automatically.\n- **Tokenization**: OPT's tokenizer is very similar to GPT-2's, so the same text usually produces similar (but not identical) token sequences.\n- **Good for comparison**: Running the same prompt on GPT-2 and OPT-125M can show how similar architectures with different training data and activation functions produce different predictions.\n\n## HuggingFace Model IDs\n\n- `facebook/opt-125m` (in dropdown)\n- `facebook/opt-350m`, `facebook/opt-1.3b` (larger, enter manually)", "source_file": "opt_overview.md", "chunk_index": 1, "embedding": [0.0007587655, 0.011558092, 0.0062663564, 0.0066032223, -0.01329802, 0.0023956713, 0.022108857, 0.052564133, 0.011492681, -0.02490844, 0.033176366, -0.0451073, -0.026046589, -0.050811123, -0.012166413, -0.01886448, -0.0032018595, 0.0034144449, -0.020748312, 0.036603894, -0.0046081925, 0.01891681, 0.02073523, 0.018472016, -0.016679758, 0.014547367, -0.03288856, 0.019636327, 0.022265842, 0.022880705, -0.006076665, -0.027498709, -0.0016957755, -0.04031923, 0.03173733, -0.023861866, 0.008084778, 0.021611735, -0.0051543727, 0.06268973, -0.020408176, -0.027708024, -0.04931976, 0.026674531, -0.033542667, 0.019243862, 0.043746755, -0.010406861, -0.0514914, 0.024136592, -0.03390897, 0.05562536, -0.0540555, 0.019113041, -0.024829946, -0.014665105, -0.05609632, -0.02636056, 0.017255375, 0.034327596, -0.015371542, -0.024686044, 0.02418892, 0.0028093946, 0.04801154, -0.009693883, -0.05714289, 0.011322613, 0.01021717, 0.024162756, 0.021206187, 4.4433367e-05, 0.011826276, -0.023665635, 0.007025122, -0.0133961355, 0.027315559, 0.03814759, 0.033464175, -0.06944013, 0.019230781, -0.0044185007, -0.043301962, -0.060962886, -0.014756681, -0.01644428, -0.012676617, -0.04599689, -0.02841446, -0.034196775, -0.0044185007, -0.0049287053, -0.04358977, -0.0022599439, -0.0041993745, -0.022030365, -0.008451078, -0.050680302, -0.009242549, 0.034850884, 0.0008969459, 0.01335689, 0.0025559277, 0.028074324, -0.004366172, 0.005919679, 0.025470974, -0.002277932, -0.031763494, 0.0058444566, -0.105965525, 0.0042222682, -0.023247005, 0.020630572, -0.0027096432, -0.01764784, -0.016365787, 0.007862381, -0.017307702, -0.06577712, -0.04594456, 0.013762437, 0.02933021, 0.0154762, 0.01081895, 0.0041732104, 0.0016401763, -0.02539248, -0.01106097, -0.021376256, -0.018458933, 0.010616176, 0.016208801, 0.00011293587, 0.0058804327, -0.022475157, -0.06671903, 0.015489282, 0.00057479757, -0.0192177, 0.008143647, -0.06070124, 0.0027276312, 0.021611735, -0.064835206, 0.012735487, -0.009301418, 0.0069139237, -0.09246474, 0.030481443, -0.006155158, -0.01323915, -0.00012571142, -0.031030893, -0.02660912, -0.04411306, -0.0029680158, 0.03487705, 0.010544224, -0.032653082, 0.04280484, -0.007188649, 0.0063252263, 0.027838845, -0.021023037, 0.06792259, -0.044217713, -0.0033016112, -0.037467316, -0.04031923, 0.04045005, 0.034118284, -0.013337266, 0.0021503808, -0.011924393, -0.0072475187, -0.043432783, -0.026766108, -0.040764023, -0.034929376, -0.008326798, -0.02864994, 0.026020424, 0.049188938, -0.018001057, -0.019296192, -0.040162243, 0.008974364, 0.018537426, 0.02158557, 0.0017153987, 0.011224497, 0.06305603, 0.0061453464, -0.015031407, -0.045499768, 0.04649401, 0.016640512, 0.010060184, 0.011793571, -0.014468873, -0.008568818, 0.025549466, -0.04392991, 0.027472544, -0.015253803, -0.024097346, 0.0023907654, -0.03626376, 7.7828656e-05, 0.010223711, -0.00924909, 0.047357433, 0.036499236, -0.04413922, -0.026949259, -0.059393026, 0.015528529, 0.03723184, 0.06875985, -0.04594456, 0.03560965, 0.049790718, 0.0074502924, -0.008699639, 0.0076399837, 0.030141305, -0.00055272144, -0.0147174345, -0.034432255, -0.0272109, -0.023260087, -0.030926235, 0.01154501, -0.016810581, -0.019243862, 0.018458933, -0.031528015, -0.021664064, -0.072946146, 0.009870493, 0.038278412, -0.020421257, -0.02919939, 0.018746741, 0.0015764007, -0.031240208, -0.074149705, 0.0064854827, 0.0012002885, 0.013775519, -0.009386453, -0.02629515, -0.021258516, -0.011041346, 0.026582956, -0.015816336, 0.038278412, -0.047069628, 0.00604396, -0.009379911, 0.032836232, -0.03053377, 0.020382011, -0.02636056, -0.0036989818, -0.036499236, -0.01197018, -0.069178484, -0.05573002, -0.0077642645, -0.02248824, -0.013997915, 0.02447673, -0.05688125, -0.022148104, -0.024450565, 0.0063448497, -0.040528543, 0.008392208, -0.03717951, 0.0051707253, 0.08681324, 0.0019557835, -0.003548537, -0.01232994, 0.015096817, 0.0036924407, -0.027315559, -0.019073796, -0.0052034305, -0.05416016, 0.019570917, -0.026517546, 0.019440096, 0.021742556, 0.024071181, -0.012604665, 0.025719535, -0.014351134, 0.017517017, -0.045787573, -0.013553122, -0.042595524, 0.018144961, 0.007829675, 0.02781268, 0.08524338, -0.019113041, -0.02883309, -0.03723184, -0.033830475, 0.008967823, -0.00033114228, 0.008418373, 0.014900585, 0.01626113, 0.010858196, 0.01021717, -0.03819992, -0.015450035, -0.036185265, -0.02387495, 0.010832032, 0.006128994, 0.034327596, -0.022161186, -0.002889523, -0.023116184, -0.010962853, 0.033961296, -0.03754581, 0.0015338837, 0.037624303, -0.023260087, 0.00628925, -0.008496866, 0.03529568, -0.011257202, 0.032679245, -0.04819469, 0.0064004487, -0.034850884, -0.0052132425, -0.07310313, 0.0052361363, -0.01335689, 0.025588714, 0.030272128, 0.01686291, -0.019178452, -0.026033506, 0.004886188, -0.007908168, 0.030010484, 0.025313986, 0.019322356, -0.00020727054, -0.034065954, 0.01335689, -0.035714306, 0.032783903, 0.0013409219, -0.008143647, 0.020709066, -0.0031953186, 0.002170004, 0.024751455, -0.028571445, -0.0038592385, 0.026007341, 0.039455805, -0.038121425, -0.009157515, -0.06650972, -0.014560449, -0.025313986, 0.020892216, -0.005337523, 0.031658836, 0.021493996, 0.0022223326, -0.0012378998, -0.021925706, -0.01886448, 0.011270285, 0.023168512, -0.029042404, -0.08566201, 0.0048992704, 0.0031364488, 0.0011373307, -0.031711165, -0.021546325, -0.022867622, 0.018524345, 0.019322356, 0.022527486, -0.003003992, 0.059654668, -0.00024713026, -0.011819735, -0.04008375, 0.0027488896, -0.0216379, 0.010315286, 0.023377826, -0.008549194, 0.019688657, 0.04963373, 0.06258507, -0.008987446, 0.02933021, -0.025313986, 0.013474629, -0.010439567, 0.015881747, 0.011355318, 0.018602837, 0.04314498, -0.016339622, -0.022161186, -0.059393026, -0.020094205, -0.063003704, -0.0240581, 0.067974925, 0.008052072, -0.005046445, -0.014586613, -0.00828101, 0.03597595, 0.048456334, -0.0313187, -0.038356904, -0.033699654, 0.032312945, 0.041627444, 0.031292535, 0.0029794627, 0.017098388, -0.01329802, 0.015031407, -0.02961802, -0.039168, -0.025771864, -0.028205145, -0.05489276, -0.031397194, -0.04097334, -0.0032329299, 0.05162222, -0.0059687374, 0.0109563125, -0.014795927, -0.04387758, 0.065306164, -0.026072754, 0.06593411, -0.005265571, 0.040162243, -0.053846188, 0.006796184, -0.021664064, -0.007744641, 0.04492415, 0.02115386, -0.007993202, -0.004281138, -0.015005242, 0.03838307, -0.05277345, -0.0034406092, -0.009569603, 0.00954998, -0.023835702, 0.03686554, 0.01360545, -0.028466789, -0.031528015, 0.03783362, 0.02322084, 0.0016843287, 0.007326012, 0.030821579, 0.096965, -0.011087134, -0.006796184, 0.004735743, -0.024202004, -0.04170594, -0.019937217, 0.027053915, -0.014795927, 0.012787815, -0.006557435, 0.02073523, -0.06739931, -0.023966525, -0.020499751, 0.024018854, -0.0030808495, -0.060962886, 0.07676614, 0.020329682, 0.024869194, -0.013036377, -0.0039508133, -0.05319208, -0.014115655, -0.020695983, -0.0036204888, -0.018969137, 0.00033747894, -0.004218998, 0.02883309, -0.00604723, 0.025889602, 0.011957098, 0.050575648, -0.015502364, 0.027001586, -0.030193634, 0.0014529378, -0.029434869, -0.017189963, -0.04843017, -0.1095762, -0.027760351, 0.0042582443, 0.019845642, -0.032731574, 0.03644691, -0.037807453, -0.05282578, 0.010681587, -0.0015903006, 0.00056989177, -0.056567278, 0.012022508, 0.015279967, -0.005658036, 0.02412351, 0.013893258, -0.0072148135, 0.012356103, -0.0025444808, 0.034746226, -0.015214557, 0.031606507, 0.03228678, 0.0006504288, 0.021964954, -0.008542653, -0.023835702, -0.020604407, -0.012140248, -0.030429114, 0.015698597, -0.021964954, 0.01239535, 0.01142073, 0.017216127, -0.010465731, 0.0007407775, 0.032810066, 0.02781268, -0.015306132, 0.007587655, -0.01118525, -0.013984833, -0.018472016, 0.02018578, -0.016313458, -0.019780232, 0.013147575, -0.009255631, 0.01214679, 0.007953956, -0.004788072, -0.026033506, -0.026151245, 0.019780232, 0.02993199, -0.024319742, 0.005295006, 0.0009092104, 0.018655166, -0.030507606, 0.0309524, 0.04544744, -0.022658307, -0.0070054987, -0.0089482, -0.015384625, -0.008294092, 0.034353763, -0.011649667, -0.024712207, -0.02066982, 0.013435382, -0.0066588214, -0.0048273187, -0.016274212, -0.010053643, 0.010648881, 0.007842758, 0.03372582, -0.01722921, 0.01275511, 0.0020457234, -0.010256416, 0.014351134, 0.006655551, -0.062061787, 0.017477771, -0.013919422, -0.016980648, 0.028100489, 0.028205145, -0.003023615, -0.024764536, 0.01916537, 0.033621162, -0.010132136, -0.00465398, -0.0010261323, -0.013095246, 0.00026757113, 0.0046834145, 0.0035648898, -0.002006477, 0.047959212, -0.005543567, 0.00095990376, -0.0068485127, -0.0023057314, -0.00036343886, 0.016679758, -0.028990075, -0.038592383, 0.007574573, 0.00465398, -0.0074502924, -0.010204088, -0.016601266, -0.008359503, 0.036551565, -0.013997915, -0.0065018353, -0.0012272705, -0.05264263, 0.027498709, 0.020866051, 0.0010972664, -0.018328112, 0.018851398, -0.024856111, -0.026491381, 0.014128737, -0.034563076, -0.0066980678, -0.0053931223, -0.05410783, 0.00616497, 0.033097874, -0.010321828, 0.0107142925, 0.009648096, -0.04890113, -0.09110419, 0.011590797, 0.031083222, 0.018707495, 0.032208286, -0.011558092, -0.001813515, -0.04819469, -0.019662492, 0.010066725, -0.00985087, 0.0075549497, -0.06111987, -0.0051118555, 0.021677146, -0.00048567535, 0.010603094, -0.018694412, 0.009137891, 0.010145218, -0.003331046, -0.045133464, 0.008621146, -0.0049679517, -0.021323927, -0.03113555, 0.011152545, 0.034432255, -0.013016753, -0.0429095, -0.01928311, 0.03639458, -0.032757737, -0.018655166, -0.009785458, -0.009863951, -0.0003879679, 0.04576141, 0.0009942445, -0.008091318, -0.047671404, -0.009209843, -0.012140248, 0.0097658355, 0.0073325527, 0.011584256, 0.0060014427, 0.0009721683, 0.02333858, -0.0082025165, -0.01402408, 0.03215596, -0.021507077, -0.0061093704, 0.017543182, 0.005592625, -0.028126653, -0.0034536913, -0.048508663, 0.031240208, -0.00089122244, -0.008097859, -0.023312416, -0.02896391, -0.0052394066, 0.00072278955, 0.014063327, -0.009706966, -0.014835174, 0.0068288897, 0.0053113587, 0.01934852, -0.03076925, -0.01420723, -0.010177923, 0.012493467, -0.009366829, 0.021742556, 0.0095238155, -0.033202533, 0.003555078, 0.010675046, 0.0057169055, 0.00033523046, -0.03372582, -0.0012517995, 0.040999502, 0.024712207, -0.027524874, 0.029958155, 0.015096817, 0.0044839117, -0.01499216, 0.030926235, -0.02677919, -0.01045919, 0.0025444808, -0.022383582, 0.004163399, 0.031030893, 0.015306132, -0.04123498, 0.009857411, -0.0030432383, 0.04746209, -0.049712222, 0.025235495, 0.0047422843, -0.01311487, -0.0057528815, -0.02938254, -0.01281398, 0.025484055, -0.017870236, -0.0058477274, 0.002896064, 0.006132264, -0.0026916552, -0.013854012, -0.02430666, -0.005157643, 0.029304048, -0.0073914225, -0.021899542, 0.004346549, 0.00023363928, -0.008830461, -0.108529635, -0.0076072784, 0.016692841, -0.01879907, -0.00062058517, 0.007908168, 0.0432758, 0.00876505, -0.0005036633, -0.028754596, 0.024202004, -0.06645739, 0.0021863566, -0.001448032, 0.007829675, -0.056985907, 0.026923094, -0.03215596, 0.024149675, -0.0143642165, -0.020565161, 0.019740986, 0.011767407, 0.001318028, -0.09162748, -0.017726332, 0.006436425, 0.022475157, 0.015031407, -0.03390897, -0.05107277, 0.026373643, -0.0063873664, 0.006684986, 0.020748312, 0.012735487, -0.004647439, -0.02484303, -0.0036597354, 0.0026197033, -0.00073178357, -0.028571445, 0.011388023, -0.04824702, 0.029827334, 0.01239535, 0.016287293, 0.010557306, -0.008922036, 0.00876505, 0.010740456, 0.012238365, 0.02671378, 0.01868133, 0.006256545, -0.014612777, 0.033935133, -0.007875463, 0.04798538, 0.0039704368, 0.042517032, 0.026504464, -0.0030939318, 0.008680016, -0.0065508937, 0.008751968, 0.028257474, 0.028021995, -0.01112638, -0.01711147, -0.0002551022, 0.043668263, -0.0012877755, -0.014638942, -0.005537026, -0.012748568, 0.0079408735, 0.022919951, -0.020931462, -0.003017074, 0.04939825, -0.023430156, -0.0034863967, 0.039377313, -0.00029107815, 0.036342252, 0.018629001, -0.023364745, 0.015345379, -0.018275782, 0.019845642, -0.007528785, -0.027498709, 0.005010469, -0.040947173, -0.0077969697, -0.016941402, 0.04170594, 0.0495814, -0.017059142, 0.04243854, -0.00507915, -0.03451075, -0.004166669, -0.021677146, 0.0057823164, -0.012166413, -0.02980117, -0.0034504207, -0.0024708938, -0.03210363, 0.034536913, 0.01142073, 0.02636056, -0.021507077, 0.028676104, 0.06101521, -0.006420072, 0.015450035, -0.007574573, 0.022200432, 0.026569875, -0.00199503, 0.013383054, -0.007646525, -0.008980906, -0.009576144, 0.02883309, -0.005942573, 0.0034863967, -0.008614605, 0.011682373, 0.005272112, -0.010596553, -0.024280496, 0.01257196, 0.012015968, -0.004183022, -0.025104672, 0.034903213, -0.019413931, 0.021140777, -0.007188649, -0.01505757, 0.036603894, 0.0021994389, -0.04272635, -0.03288856, -0.011414188, 0.0072148135, 0.04201991, 0.006868136, -0.0149790775, 0.04314498, 0.017032977, 0.029225554, -0.015279967, -0.019845642, -0.039377313, 0.016588183, -0.0072736833, 0.05034017, 0.018485097, -0.0028862525, 0.040607035, -0.031344865, -0.008791214, 0.026687615, 0.021088447, 0.015750926, -0.0035714307, 0.033045545, 0.027420215, 0.002961475, 0.03796444, 0.033621162, -0.006704609, 0.010145218, -0.008961283, 0.021925706, 0.021546325, 0.030062813, 0.008156729, -0.002822477, 0.030481443, -0.01311487, -0.013435382, 0.01450812, -0.0223705, 0.030141305, 0.018341193, 0.024136592, -0.0017252103, 0.0011316072, -0.03427527, -0.0153322965, 0.019636327, -0.0203166, -0.031501852, -0.016522773, -0.0013016753, -0.010452649, -0.012500008, -0.020499751, -0.018629001, 0.029277883, 0.00052083365, 0.007574573, 0.0014553907, -0.013232609, -0.0107142925, 0.04612771, 0.004918894, 0.023168512, -0.002763607, 0.008869708, -0.0064593186, -0.0024676232, -0.033333354, 0.0049973866, -0.04086868, -0.008065154, 0.0020081121, -0.037074853, -0.01420723, 0.015096817, 0.006626116, 0.00556319, 0.0039508133, 0.0062467335, -0.021075366, -0.010439567, 0.0036924407, 0.015790172, 0.021964954, 0.0100078555, -0.009720048, 0.0010743727, 0.029120898, 0.04751442, 0.000629988, 0.02708008, -0.009255631, 0.009085563, 0.035086364, 0.0036401122, 0.013644697, 0.024633715, 0.0060276072, -0.007175567, 0.0119440155, 0.021611735, 0.02103612, -0.019021466, 0.034746226, 0.007568032, 0.010596553, 0.057770837, 0.011270285, 0.007260601, 0.00017732465, 0.057404537, 0.017477771, -0.021729475, 0.029879663, -0.0143642165, 0.01318028, -0.010125595, 0.035504993, -0.014900585, -0.0272109, 0.04105183, -0.041679773, 0.009412617, 0.019544752, 0.0113684, 0.023417074, 0.0029958156, 0.027106244, 0.0023596953, -0.0054323687, -0.0054323687, 0.004797884, 0.007862381, 0.013697025, -0.013683943, -0.021978036, -0.015397707, 0.026622204, 0.01402408, 0.0055958955, 0.04332813, 0.020983791, -0.011564633, 0.00028637674, -0.0024332826, 0.01166929, 0.008751968, 0.0137493545, -0.024411319, 0.00900707, 0.0076726894, 0.011800112, -0.05798015, 0.007443751, 0.019126123, 0.05018318, -0.03984827, -0.0028862525, -0.02435899, -0.02539248, -0.018144961, 0.02212194, -0.009811623, 0.01886448, 0.023560977, 0.023050772, -0.020238107, 0.013088705, -0.02864994, -0.018746741, -0.01372319, -0.0029974508, 0.009831246, 0.001239535, -0.03403979, 0.032679245, -0.014154902, -0.008575358, -0.0117216185, 0.023534814, 0.021180024, -0.00048894587, 0.02817898, -0.05285194, -0.009798541, -0.0192177, -0.0028797113, -0.010655423, -0.030795414, 0.0101583, 0.019727902, -0.012218741, 0.0015371543, 0.038723204, 0.004075094, 0.0035387254, 0.018144961, 0.03307171, 0.0027570662, 0.054683447, -0.02477762, -0.011440353, -0.006655551, -0.026334396, 0.010740456, 0.009164056, -0.0019606894, -0.040842514, 0.0016001122, 0.021794885, 0.028388295, 0.0047063087, -0.0077315588, 0.02496077, 0.009648096, -0.016169555, 0.0012280882, 0.036525402, 0.001977042, -0.008294092, -0.01263737, -0.016601266, 0.0020097475, 0.009229466, 0.02527474, 0.0097658355, -0.016091062, 0.014769763, 0.025013097, 0.0070054987, -0.04552593, 0.01251309, 0.017386196, 0.013369972, 0.013206445, 0.0060897474, 0.019427013, -0.05447413, -0.02025119, 0.0038167213, -0.01039378, 0.019557836, -0.034615405, 0.011172168, -0.02322084, 0.01215333, -0.021009956, 0.004650709, -0.00052287773, -0.0023858596, 0.017687086, -0.013487711, -0.0028486412, 0.02206961, -0.033542667, -0.011374942, 0.021860296, 0.019479342, 0.01009289, -0.020774476, 0.016221883, -0.016038733, -0.0052982764, -0.007568032, -0.055887006, -0.0125392545, 0.0102498755, -0.0137886, 0.018223455, -0.035818964, 0.0048992704, -0.028492954, -0.032443766, 0.012290693, -0.0046866853, 0.01148614, 0.0015118076, 0.0055697314, 0.010550765, -0.017242292, -0.0071559437, -0.006233651, 0.03409212, -0.015266885, -0.0101583, 0.027760351, -0.017281538, -0.015279967, 0.018341193, 0.032312945, 0.019256946, 0.011806653, 0.03621143, -0.00743721, 0.022213515, -0.008529571, -0.025366316, 0.0038625088, -0.008647311, 0.016692841, 0.0026229739, -0.005527214, 0.03772896, 0.009164056, -0.012715863, -0.032783903, -0.0071166973, 0.038252246, 0.014900585, -0.016117226, -0.0067634787, 0.036525402, 0.008137106, 0.023168512, 0.0043792543, 0.024437482, 0.038906354, 0.00020532864, -0.018419687, 0.014285723, -0.015306132, -0.00060382363, -0.003470044, 0.02575878, 0.0048338594, 0.006560705, -0.0067896433, -0.024646796, -0.00296311, 0.0063023325, 0.016470445, 0.012500008, -0.04050238, 0.009406076, 0.03833074, 0.011276825, -0.037571974, -0.028152816, 9.653206e-05, 0.043380454, 0.030481443, -0.01886448, 0.01427264, 0.0045754868, -0.025902685, 0.01142073, -0.024869194, -0.0072148135, 0.0058182925, -0.02363947, -0.036237594, 0.009536898, 0.0031544368, 0.0046278154, -0.02121927, 0.001959054, -0.025130836, -0.013592368, -0.02320776, 0.0064593186, 0.008294092, -0.018341193, 0.031894315, -0.0053048176, 0.0038199918, -0.017216127, -0.019963382, -0.00086424046, 0.0043203845, 0.020826805, -0.033568833, -0.0018396793, 0.035688143, 0.012022508, -0.017490853, -0.0060537714, 0.02085297, -0.047174282, 0.027367886, -0.0038526973, 0.01577709, -0.019531671, -0.008084778, 0.0007362805, 0.014900585, 0.022645226, -0.0009419158, -0.0059033264, 0.022619061, 0.0024856112, 0.0063088736, 0.01161042, -0.0054618036, 0.0018265971, 0.01577709, -0.01033491, -0.011355318, 0.035138693, -0.02823131, -0.04013608, -0.0046343566, -0.028728433, -0.011329154, -0.0106358, 0.016496608, -0.011623503, -0.015253803, 0.026347479, 0.02804816, -0.002400577, -0.008058613, -0.0033621162, -0.014076408, 0.056985907, -0.016941402, 0.009000529, 0.003905026, 0.010903983, -0.006462589, -0.009026693, -0.021847215, 0.05400317, 0.01964941, -0.015227639, 0.0069924165, -0.0005375952, -0.00087078154, 0.026164329, 0.015554693, 0.0036793586, -0.027132409, 0.004680144, -0.021899542, -0.022148104, -0.011152545, -0.015096817, -0.00011344689, 0.011132921, 0.02689693, -0.028571445, 0.027838845, -0.02327317, 0.02564104, -0.002575551, 0.024594469, 0.010315286, 0.030978564, -0.035348006, 0.036734715, -0.00020021843, 0.021991117, -0.030088978, -0.03500787, -0.017883318, 0.014102573, -0.029827334, 0.021951871, -0.012310316, 0.01215333, -0.023011526, -0.034065954, 0.016091062, -0.0032934349, -0.011616962, 0.038356904, 0.0032541882, -0.02539248, -0.008751968, -0.01734695, -0.020931462, -0.009948986, -0.0053898515, -0.018236537, 0.01087782, 0.04466251, -0.0074829976, 0.03723184, -0.0066882563, -0.008117483, 0.039795943, -0.008529571, -0.017844072, -0.02320776, 0.01801414, 0.0118131945, 0.02170331, 0.027943503, 0.0055174027, 0.002703102, -0.035818964, 0.0053833104, -0.016352706, -0.010995558, -0.0089482, 0.03765047, 0.014468873, -0.02235742, -0.031057058, 0.065201506, -0.01166929, -0.021441666, 0.0025101402, -0.011741242, 0.028519118, 0.004189563, -0.018576672, 0.032993216, -0.0010024208, -0.00967426, 0.02000263, -0.031083222, -0.019073796, 0.020028792, 0.006040689, 0.007823134, -0.0007910621, 0.039194163, 0.0025591983, 0.039429642, -0.024986934, 0.016352706, 0.015384625, -0.00093128654, 0.03076925, 0.013088705, 0.0125785, 0.022579815, 0.010354533, -0.03663006, 0.02043434, 0.007927791, 0.03233911, 0.00024896994, 0.022632144, -0.01674517, -0.028388295, 0.011296448, 0.03644691, -0.01330456, -0.034065954, -0.037441153, -0.0054618036, -0.041025665, -0.0074568335, 0.050680302, 0.019976465, -0.04976455, -0.005066068, 0.0034046331, -0.0076334425, 0.041077994, -0.016025651, -0.0044446653, 0.004552593, 0.022645226, 0.032129794, 0.023076937, -0.0052361363, -0.04358977, -0.026870765, -0.041130323, 0.013461547, -0.006662092, -0.009013611, 0.014586613, -0.010177923, -0.026033506, -0.007646525, -0.0055991663, -0.026386725, 0.02781268, 0.021860296, -0.031894315, -0.007659607, 0.00035444487, -0.015894828, -0.011643126, 0.02804816, 0.002793042, 0.020198861, -0.04604922, -0.015973322, -0.0021634628, -0.022776047, -0.0037447694, -0.0007056192, 0.032129794, -0.013069082], "content_hash": "7a7db8a88a5c897b3d717dbf01b78c81"}, {"content": "# Output and Prediction\n\n## How Does the Model Choose the Next Token?\n\nAfter your text has passed through all the Transformer layers (attention + MLP in each), the model needs to make a prediction: what token comes next? This final step converts the model's internal representation into a probability distribution over its entire vocabulary.\n\n## Logits: Raw Scores\n\nThe model's final hidden state for the last token is multiplied by the embedding table (in reverse) to produce a score for **every token in the vocabulary**. These raw scores are called **logits**. A higher logit means the model thinks that token is more likely.\n\nFor GPT-2, this means producing about 50,257 scores -- one for each token in its vocabulary.\n\n## Softmax: Turning Scores into Probabilities\n\nRaw logits can be any number (positive or negative). To get actual probabilities, the model applies a function called **softmax**, which:\n\n1. Converts all scores to positive numbers\n2. Makes them all add up to 1.0 (100%)\n3. Preserves the ranking (higher logits \u2192 higher probabilities)\n\nAfter softmax, we can say things like \"the model predicts 'mat' with 45% probability.\"\n\n## Temperature\n\n**Temperature** is a setting that controls how \"confident\" or \"creative\" the model's predictions are:\n\n- **Low temperature (e.g., 0.1)**: Makes the model very confident -- the top prediction gets almost all the probability. Good for factual, predictable text.\n- **High temperature (e.g., 1.5)**: Spreads probability more evenly, making less likely tokens more probable. Good for creative, varied text.\n- **Temperature = 1.0**: The default, unmodified distribution.\n\n## Greedy Decoding vs. Sampling\n\nOnce we have probabilities, how do we pick the actual next token?", "source_file": "output_and_prediction.md", "chunk_index": 0, "embedding": [0.01852944, 0.012449853, 0.026149021, -0.02459047, 0.027039621, -0.012975555, 0.01397748, 0.019222127, 0.0033119207, 0.027237533, -0.00091302017, -0.0018724258, -0.061204046, -0.0011356702, 0.00074757874, -0.05046736, 0.01987771, 0.032210052, -0.03404073, 0.01758936, -0.03676201, -0.037405223, -0.01464543, 0.03997807, 0.042748824, -0.0168843, -0.009982147, 0.036910444, 0.0055786227, -0.01330953, -0.010439817, -0.0219063, -0.03973068, -0.0455938, 0.006323882, -0.013173466, -0.0076566907, 0.039087467, -0.0069330777, 0.01918502, -0.031022584, -0.009901746, -0.072732374, 0.026198499, -0.009870823, 0.019506626, 0.0067104273, -0.047721345, -0.0387906, 0.048983026, -0.017181167, 0.033422258, -0.041685052, 0.017972814, 0.015981331, -0.020731201, -0.0315421, 0.008411227, -0.012901338, -0.045470104, -0.004295292, 0.009914115, 0.044282638, 0.025678981, 0.04621227, 0.012214834, -0.07030796, 0.04801821, -0.028400261, 0.0036087874, 0.042056136, -0.00625585, -0.0120664, -0.0062094647, -0.020904373, -0.010817085, 0.0025558379, 0.009611064, -0.016315307, -0.034906592, 0.0010583611, -0.038666904, 0.0058229193, -0.03943381, -0.0053745266, -0.020026142, 0.02482549, -0.03940907, -0.0410171, -0.07421671, -0.029958813, -0.005999184, 0.005071475, 0.03134419, 0.016723499, -0.025357375, -0.0015987516, -0.035030287, -0.004533404, -0.019061325, -0.000112388065, -0.019989034, 0.014163022, -0.013569288, 0.021139393, -0.031220496, -0.044554766, -0.00889982, -0.031517364, -0.043441515, -0.080599345, -0.008621507, -0.04123975, 0.05294125, 0.010223351, -0.061995693, -0.000660606, 0.0306515, 0.006002276, -0.032581136, -0.011299494, -0.044084724, 0.02029827, 0.049477804, -0.01624109, 0.008287532, 0.026074804, -0.044356853, 0.027806528, 0.028400261, 0.0038314376, 0.004814809, 0.013247683, -0.029167166, -0.021015698, -0.03129471, 0.00052802096, 0.059472322, -0.011559253, 0.041437663, 0.0026687093, -0.03223479, -0.01650085, 0.015461815, -0.062342037, -0.04081919, -0.0055105905, 0.019234497, -0.07857075, 0.027806528, 0.008485444, -0.00463236, -0.035772454, 0.021832082, 0.016166873, -0.036143538, 0.015845267, 0.08030248, 0.023848303, -0.03948329, -0.016661651, 0.03473342, 0.01582053, 0.001048311, -0.022252643, 0.053040206, -0.024887336, 0.009487369, -0.019086063, -0.059224933, 0.011565438, 0.009641987, -0.020397225, -0.016439002, -0.026990144, -0.02346485, -0.07159439, -0.042080875, -0.018987108, -0.03804843, -0.03671253, 0.010909856, -0.04054706, 0.00014640405, 0.0012508607, -0.0020224054, -0.028944517, 0.026693277, 0.009561586, -0.009376044, 0.0077742003, -0.0018585101, 0.07629478, -0.023167983, -0.041437663, -0.00053575187, 0.06338107, 0.0069825556, 0.018603655, 0.027559139, 0.011497405, 0.077977024, 0.00891219, -0.044530027, 0.023477219, 0.017688315, -0.013012663, 0.021844452, -0.03250692, 0.003562402, -0.0038283453, -0.041536618, 0.0056868554, -0.026816972, -0.054969843, 0.019927187, -0.040398628, 0.03559928, 0.031245235, 0.07960979, 0.016265828, 0.04935411, -0.0073536388, 0.0028820822, -0.049552023, 0.012876599, 0.033743862, -0.017106952, 0.04373838, 0.020558028, 0.013495072, 0.015152578, -0.013260052, -0.049972583, -0.020125099, -0.029117689, 0.021052806, 0.0150412535, -0.008256609, -0.03319961, -0.019494256, 0.016377155, 0.015313381, 0.023031918, 0.023798825, -0.0044097095, -0.028721867, -0.043392036, 0.0455938, 0.0293156, 0.023749348, -0.059670236, -0.019246867, 0.0059837224, -0.011132507, -0.021498106, -0.026643798, 0.037479438, 0.0015229888, -0.027089098, 0.0055848076, 0.009475, -0.03837004, -0.006221834, -0.016797716, -0.02523368, -0.034782898, -0.03629197, -0.027113838, -0.060857702, -0.016599804, -0.029661944, -0.03809791, 0.045915402, -0.053633943, -0.04626175, -0.00067258894, 0.033545952, -0.05318864, 0.04054706, -0.03861743, -0.044332113, 0.047498696, 0.029711423, 0.032135837, -0.005776534, -0.009679096, 0.030033028, 0.0055786227, -0.020854896, 0.04210561, -0.011089213, -0.0114850355, -0.03592089, 0.02773231, 0.053040206, 0.024738902, 0.03881534, -0.01467017, -0.018813936, 0.030181462, -0.06847728, 0.0051054913, -0.008083437, 0.019692168, 0.024219386, -0.008466889, 0.033446997, 0.015313381, -0.005838381, -0.008825604, -0.0123323435, -0.01444752, 0.025604764, 0.00066137913, 0.029859856, 0.024924444, -0.037949476, 0.018900521, -0.024664687, -0.012827122, -0.027410705, 0.025456332, 0.018034661, -0.0015925668, 0.032779045, -0.022252643, 0.0052384627, 0.007965927, -0.016859563, 0.019444779, 0.00027116158, -0.0121653555, 0.040002808, -0.05279282, -0.0051085837, 0.01690904, 0.040942885, -0.031764753, 0.03587141, -0.051704306, -0.008392673, -0.027138576, -0.001066092, -0.040324412, 0.03814739, 0.018987108, 0.024169909, 0.038864817, 0.01329716, 0.0073660086, 0.011744794, 0.00889982, -0.014732017, -0.0056652087, 0.030206202, 0.009543031, 0.0065063317, -0.024949184, -0.032383226, -0.056849997, 0.017341971, 0.060412403, -0.00024275048, 0.0050312746, -0.028400261, -0.03290274, 0.012524069, 0.011225278, -0.011014997, -0.0052786632, 0.056652088, -0.021164132, 0.001260911, -0.026915926, -0.016995626, -0.006518701, 0.02142389, -0.008862712, 0.0383453, 0.016785346, 0.03837004, -0.007130989, -0.002606862, 0.02795496, 0.011305679, 0.018863413, -0.03161632, -0.017626468, -0.0048550097, -0.010186244, -0.019098433, -0.04888407, 0.02592637, 0.0008140645, 0.052396998, -0.027237533, 0.045766972, 0.029389817, 0.044727936, -0.023044288, -0.016105026, -0.047300782, -0.026149021, 0.010798532, 0.0016745145, 0.021065176, 0.03537663, 0.042575654, 0.0193953, 0.09024752, 0.011274755, 0.02570372, -0.038691644, 0.015375229, -0.0020502366, 0.05501932, 0.038172126, 0.023167983, 0.0029763994, -0.032135837, -0.025654243, -0.03629197, -0.017725425, -0.05214961, -0.019457147, 0.03067624, -0.02366276, 0.020669354, -0.02595111, 0.0018028476, 0.009475, 0.0024398742, -0.042550914, 0.030602023, -0.0136929825, 0.033348043, 0.03723205, -0.011559253, -0.04754817, 0.014855712, -0.008534921, -0.021300195, -0.011590176, -0.023774086, -0.0038809155, -0.0232422, -0.047028653, -0.010922226, -0.024862597, 0.00070273946, 0.020459073, -0.015103101, 0.026767492, -0.00021433941, -0.021238348, 0.07431567, 0.02435545, 0.06288629, -0.051308487, 0.00056976784, -0.027039621, -0.021300195, -0.055167753, -0.015931854, 0.0100439945, 0.028177612, -0.0058507505, 0.010328492, 0.028969256, -0.0012686419, 0.014546475, -0.008479259, -0.044059988, 0.016946148, 0.013952741, 0.048760377, 0.0025712997, 0.021263087, -0.026445888, 0.033966515, 0.013272421, -0.04777082, 0.0029980459, -0.038988512, 0.07075326, 0.023477219, -0.012864229, 0.0041375817, -0.041116055, -0.018677872, -0.048760377, 0.022438185, -0.012245757, 0.023477219, -0.0042334446, -0.008887451, -0.039557505, 0.003519109, 0.0037819599, 0.02953825, -0.011590176, -0.019321084, 0.057889033, 0.009048254, 0.045692753, 0.05976919, 0.021559954, -0.040349152, 0.036663055, -0.064222194, -0.011008812, -0.019568473, 0.0026903558, -0.010854194, 0.0035747716, -0.0012941539, 0.028375521, 0.009066808, -0.0038932848, -0.007836048, 0.01963032, -0.002071883, 0.014398041, 0.021807343, -0.031072062, -0.019308714, 0.008343195, -0.047053393, -0.022401078, 0.049081985, -0.023774086, 0.018752089, 0.02523368, -0.025629504, 0.003630434, -0.014608323, -0.04354047, -0.0455938, -0.0040169796, -0.015461815, 0.0043385853, 0.005727056, 0.012554993, 0.006797014, 0.0019018032, 0.014051697, -0.024380188, -0.024677055, -0.0032902742, -0.009765682, -0.025777938, 0.019729275, -0.03245744, -0.023625653, -0.010909856, 0.013556919, 0.014979406, 0.008225685, -0.0152144255, 0.004984889, -0.035549805, -0.002965576, 0.028449738, -0.015536031, 0.045272194, 0.00046076206, 0.0072917915, 0.045000065, -0.02862291, 0.0024213202, 0.011528329, 0.009635802, 0.015659725, -0.03225953, -0.026470626, 0.038765863, -0.02953825, 0.006895969, 0.03559928, -0.0140269585, -0.0032593505, 0.021522846, 0.0071000652, -0.009784237, 0.0034634464, -0.0043447698, 0.005649747, -0.03023094, 0.0023749347, 0.047696605, -0.04213035, 0.0071062497, 0.021324934, -0.04829034, 0.029216645, -0.0033397519, -0.035525065, -0.023155613, 0.03067624, 0.013111618, -0.015004145, -0.026050065, 0.019283975, 0.009147209, 0.01511547, -0.0396812, 0.031913184, -0.018232571, -0.0022713405, -0.0091966875, -0.049007766, -0.03520346, -0.016117396, -0.0063764523, -0.018071769, -0.037627872, -0.0068650455, 0.0464844, -0.0066423956, 0.025295528, 0.007570104, -0.009839899, -0.020211684, -0.014496997, 0.020100359, -0.025555287, -0.018034661, -0.0069207083, 0.008392673, 0.0033583061, 0.01963032, 0.042946737, 0.02230212, -0.022722682, 0.012054031, 0.023180353, -0.007180467, 0.018034661, -0.02503577, -0.0101243965, -0.010000702, -0.019259237, -0.020224053, -0.0058074575, 0.0014387218, -0.021794973, 0.041190274, -0.010625359, -0.011924151, -0.020706462, -0.042278785, 0.031715274, -0.0059125978, -0.0030166, -0.01756462, -0.014212499, -0.016624544, -0.011639654, -0.01262921, -0.041388184, -0.026693277, -0.043243602, -0.014781495, 0.0013583204, 0.046781264, 0.04487637, 0.021708388, 0.012901338, -0.0025125449, -0.09257297, 0.033323303, -0.007180467, -0.0058662123, -0.0040942887, -0.018677872, -0.009035884, -0.06659713, 0.0125055155, 0.0410171, -0.020854896, -9.742151e-06, -0.037998956, -0.03359543, -0.008739018, -0.037801042, 0.028128132, -0.03250692, 0.028993994, 0.0009609518, -0.026223237, -0.013940372, -0.0072732377, -0.040250197, -0.036588836, -0.023712238, 0.0014031597, 0.053633943, -0.0059435214, -0.059571277, -0.014175392, 0.014051697, -0.028598173, -0.0019342729, -0.0293156, 0.008541106, -0.0140269585, 0.018616024, 0.008473074, -0.032605875, -0.03631671, 0.01397748, 0.028227089, 0.020793049, -0.014163022, 0.012128247, -0.012610656, 0.0058260118, -0.011917966, -0.016575065, -0.021089915, -0.0055013136, -0.016711129, -0.021312565, 0.015090731, -0.037380483, -0.04641018, -0.019642688, -0.03829582, 0.003383045, 0.023130875, 0.01218391, -0.034881853, -0.04148714, 0.010223351, 0.022190796, -0.009308012, -0.029736161, -0.013198205, -0.031715274, 0.0077494616, 0.03995333, -0.013705352, 0.01781201, 0.05368342, 0.017007995, -0.011033551, 0.0116520235, -0.009277089, -0.05976919, -0.01116343, 0.009338936, -0.009524478, -0.027856005, 0.021609431, 0.026272716, -0.0031325636, -0.025357375, -0.023279307, 0.029365078, -0.017230647, 0.026297454, -0.035995103, 0.007922634, -0.020892004, -0.039112207, -0.0071557276, 0.004276738, -0.0023811194, 0.016265828, -0.0075020725, -0.011565438, 0.00036373918, 0.0030815396, 0.027460182, -0.008918375, 0.017737793, 0.02525842, -0.008027774, -0.0053374185, 0.0022759791, 0.0033552137, -0.00063857296, -0.008367934, -0.014484628, -0.017923335, -0.019073695, 0.01621635, -0.015498923, -0.023353525, -0.007897895, 0.009190503, 0.021386782, -0.011670577, -0.018851044, 0.015647355, -0.0012933807, -0.08772415, -0.024887336, 0.011243831, -0.0023362802, 0.024726534, -0.033941776, 0.019345822, -0.0016157596, -0.0146578, -0.033917036, 0.024454406, -0.016439002, 0.011021181, -0.005359065, -0.0071062497, -0.041635573, -0.00461999, -0.00642593, -0.016550327, -0.031987403, -0.043886814, 0.008745202, -0.017774902, -0.02909295, -0.06026397, 0.0023501958, 0.011917966, 0.0035036472, 0.0055229603, -0.00620328, -0.018034661, 0.02524605, 0.02956299, 0.0070938803, -0.019890077, 0.013940372, -0.00020854123, -0.003800514, 0.0014866535, -0.015461815, 0.052199084, -0.021127023, 0.050121017, -0.023613283, 0.026025327, -0.04945307, -0.0069207083, -0.031418405, -0.0008882812, -0.0045643277, 0.00202859, 0.04346625, 0.032309007, -0.016377155, 0.038172126, 0.033348043, 0.013804308, -0.01736671, 0.0030846319, 0.029835118, -0.011181984, -0.0024491514, 0.033273824, -0.0025712997, 0.007953557, -0.0077432767, 0.014484628, 0.026346931, -0.037627872, -0.029835118, -0.013544549, 0.0017765624, 0.00923998, 0.010916041, -0.02053329, 0.008788495, 0.010668652, 0.022252643, -0.042278785, 0.009233796, 0.022846377, 0.007916449, -0.010452187, 0.019716905, 0.053831853, 0.037281528, -0.021003328, -0.018653132, 0.012227203, 0.043441515, 0.06565705, 0.008640062, -0.013123988, -0.011887043, -0.043664165, -0.012598286, 0.025604764, 0.03067624, 0.026940666, -0.029142428, 0.049279895, 0.050022062, -0.033694386, -0.027089098, -0.02165891, 0.02095385, 0.012190094, 0.01713169, -0.020780679, -0.0056002694, -0.0002446832, 0.010476925, 0.017304862, 0.038741123, -0.006407376, 0.015189687, 0.052001175, 0.020892004, 0.019778753, -0.03112154, -0.003964409, 0.019382931, -0.009808975, 0.031888448, -0.05071475, 0.013903264, -0.018084139, 0.031517364, 0.025580026, 0.0054147276, -0.0009872369, -0.014422781, 0.021151762, -0.005479667, -0.032358486, 0.013779569, 0.009048254, -0.005442559, -0.045519583, 0.024330711, -0.0110644745, 0.009518293, -0.00857203, -0.009202872, -0.04618753, -0.007192836, -0.0053621572, -0.016290568, -0.009679096, 0.015053622, 0.040077023, 0.020879634, -0.014991775, 0.033966515, 0.0002659432, 0.020632245, -0.018368635, -0.007032033, -0.014051697, 0.021089915, 0.052495953, 0.04351573, 0.01804703, -0.01783675, 0.023699868, -0.03787526, -0.016166873, 0.029934073, -0.0032840895, 0.016525587, -0.012146802, 0.020001404, 0.009128655, 0.03673727, 0.01622872, 0.01646374, 0.00042056135, 0.043045692, -0.00011374097, 0.040324412, 0.021510476, 0.046360705, -0.012573548, -0.0034139687, 0.024998661, -0.041685052, -0.015140208, 0.03000829, -0.028573433, 0.017057473, -0.0057796263, 0.0024305973, 0.009561586, 0.047894515, -0.018034661, -0.013816677, 0.04596488, 0.008176208, 0.001662145, 0.0045086653, -0.0058229193, 0.019444779, 0.012913708, -0.007242314, -0.021609431, 0.020471442, -0.009468815, -0.0063702674, 0.04220457, 0.010446002, -0.0077680154, -0.014744386, -0.014967036, 0.02388541, -0.029835118, -0.010674837, -0.0037479438, -0.010446002, -0.025728459, 0.018405745, -0.0419077, -0.026544843, 0.01736671, -0.029191906, 0.015239164, -0.008479259, 0.012913708, 0.020805417, -0.010742868, -0.003614972, -0.02659432, -0.00012900951, 0.008887451, 0.0032964589, 0.023947258, 0.030305156, 0.012388006, -0.00018824759, 0.024231756, 0.026495365, 0.019370561, 0.016105026, -0.003707743, -0.016748236, 0.012196279, -0.005702317, 0.011924151, 0.023539066, -0.0056002694, -0.034881853, 0.017663578, 0.042501435, 0.020496182, -0.0124189295, 0.0232422, 0.023588544, 0.016179243, 0.03975542, 0.024540992, 0.00074525946, -0.006265127, -0.011528329, 0.0045364965, -0.028845562, 0.019048955, -0.011819011, -0.013804308, -0.00315421, 0.03653936, 0.009221426, -0.056849997, 0.032358486, 0.003766498, 0.036193017, 0.017243015, -0.049081985, -0.0073598237, -0.011373711, 0.003383045, -0.006561994, -0.028969256, 0.003587141, -0.014843342, -0.016154503, 0.0013474971, 0.010520219, -0.0065743634, -0.012975555, 0.003989148, 0.005158061, 0.023539066, 0.014843342, -0.0039273007, 0.0025682074, -0.031022584, 0.01217154, 0.011645839, 0.010804716, -0.01852944, -0.024207016, 0.02029827, -0.01160873, 0.0017255385, -0.018319158, 0.0020656984, 0.022277383, 0.013173466, -0.031962663, -0.01624109, 0.0051858923, 0.00093930523, 0.010019256, -0.006475408, 0.03785052, 0.031220496, -0.014732017, -0.0021924854, -0.024429668, 0.00925235, 0.0048271785, -0.0027552953, 0.0019110802, -0.02436782, 0.01757699, 0.00422726, -0.019914817, 0.039854374, -0.009598695, 0.006231111, 0.030948367, 0.0046416367, 0.023650391, 0.011262385, 0.004561235, -0.05165483, -0.02956299, -0.005368342, -0.019098433, 0.0017239923, -0.050393146, -0.013940372, 0.0019296345, -0.018467592, -0.0101305805, -0.0075391806, 0.0042829225, -0.021374412, 0.026470626, 0.017181167, -0.020063251, 0.051456917, -0.027658094, -0.038196865, -0.021572324, 0.014682539, 0.0017069844, -0.01621635, -0.013965111, -0.05054158, -0.0041128425, 0.005464205, 0.01556077, -0.005009628, -0.016674021, 0.028177612, -0.013185835, 0.00034402538, -0.0014936113, 0.036193017, 0.0037262973, 0.007192836, 0.019370561, -0.026668537, 0.0053405105, -0.016080286, 0.023650391, -0.012975555, -0.024491513, 0.024417298, 0.0043169386, -0.029637206, -0.02456573, -0.031517364, 0.032210052, -0.02007562, 0.011856119, -0.011367526, 0.014422781, -0.03592089, 0.02684171, 3.592456e-05, -0.01285186, -0.02029827, -0.019989034, -0.0010235722, 0.02684171, 0.012567363, -0.026000587, -0.005529145, -0.0025233682, 0.031146279, 0.0075886585, -0.00063779985, -0.015251534, 0.051951695, -0.03995333, -0.005795088, 0.012666319, 0.02795496, 0.015993701, -0.038493734, 0.008813234, -0.032284267, 0.017106952, -0.0058136424, -0.030577283, -0.003562402, 0.007483518, 0.027930222, 0.021028068, -0.018900521, -0.0017873858, -0.025382115, -0.005974445, 0.01916028, -0.004313846, -0.0067166123, 0.0105511425, 0.0010313031, 0.03129471, -0.012691057, -0.002625416, -0.03584667, 0.00031832012, 0.016735869, 0.00020622196, 0.024837859, -0.011274755, -0.020768309, 0.001415529, -0.0016327676, -0.00093466666, 0.017428556, 0.012901338, 0.011293309, -0.008967852, 0.0024862597, 0.0100439945, -0.003614972, 0.014299086, -0.007186651, -0.0024259586, -0.016129766, 0.0035314783, -0.001815217, 0.0045643277, -0.056008875, 0.0047869775, 0.02097859, 0.0089431135, 0.016698759, 0.0036458957, 0.03807317, -0.011219093, 0.003050616, 0.024540992, -0.008621507, 0.042056136, -0.0025419223, -0.07337559, 0.010328492, 0.011534514, 0.015251534, -0.014917559, -0.0007730907, 0.0015539123, 0.0052601094, 0.008677171, -0.022487663, -0.018875783, -0.0010498572, 0.013643505, 0.00073057075, -0.009295642, 0.013544549, -0.020162206, 0.0075020725, -0.0028650742, -0.01601844, 0.009258535, 0.01985297, -0.00032141246, -0.0069640013, 0.025357375, 0.0007719311, -8.639289e-05, 0.016970888, -0.002718187, -0.0064815925, 0.011318048, 0.0017224462, -0.015511292, 0.007279422, 0.0004456868, -0.028449738, -0.033348043, -0.0067537203, -0.03537663, 0.0062836814, 0.011955075, 0.034362335, 0.0032748124, -0.020681722, 0.012703427, 0.0068217525, 0.0072917915, -0.018826306, 0.023971997, 0.0279797, -0.0016806992, -0.016562697, -0.016760606, 0.00023018777, 0.027336488, 0.0062744045, -0.011645839, 0.015894745, 0.0035036472, -0.031072062, 0.015263903, 0.02050855, 0.020706462, -0.003519109, -0.00444991, -0.006228019, 0.010532588, -0.018566547, -0.005946614, -0.0026486088, 0.0016745145, 0.008955483, -0.032828525, -0.012777643, -0.015424706, 0.003001138, 0.025382115, -0.0018693333, 0.0059682606, 0.022834007, 0.0055105905, -0.012115878, 0.0042117983, 0.0009934215, -0.012678687, -0.028993994, 0.033892296, -0.01285186, -0.03359543, 0.014818603, 0.0034727235, -0.008374118, -0.018974738, -0.009617249, -0.0021878467, 0.049329374, -0.0396812, 0.01217154, -0.035995103, 0.012400375, -0.026520103, 0.010693391, 0.011423188, 0.01919739, 0.018220203, -0.03859269, 0.0021352766, 0.0030243308, -0.04537115, 0.026495365, 0.02325457, 0.042798303, -0.026742755, 0.008875081, 0.026891187, 0.011305679, -0.013507441, -0.009654357, -0.019135542, 0.020731201, 0.003757221, -0.04598962, 0.018665502, -0.0035593098, 0.04641018, 0.007916449, 0.03812265, -0.011039736, 0.023947258, -0.03765261, -0.0033954144, 0.037553653, 0.0043942477, -0.0077865697, -0.0245781, 0.008627692, -0.029365078, -0.027039621, -0.019048955, -0.0039489474, -0.014694909, -0.026099542, -0.023427742, 0.0028063194, -0.016426632, 0.015968962, 0.035104502, 0.011874674, -0.041215014, -0.03275431, -0.006364083, -0.0061321557, -0.019518994, -0.013829047, -0.0016884301, 0.029117689, 0.033100653, 0.020570397, -0.0202859, 0.019308714, -0.02592637, -0.017341971, -0.0055940845, 0.0042736456, 0.0004831817, -0.0011951983, -0.011849934, 0.011744794, 0.0024104968, 0.0101243965, -0.0035036472, -0.006518701, 0.019704536, 0.0396812, -0.04937885, 0.0012052484, -0.004357139, -0.0026795324, -0.038196865, -0.019778753, 0.044406332, 0.0074958876, -0.006357898, -0.008324641, 0.008287532, 0.018690242, 0.029587729, 0.0008001489, 0.008287532, -0.013655875, 0.009555401, 0.0027042713, -0.009641987, -0.024404928, -0.0008882812, -0.036440406, 0.0066423956, 0.024293603, -0.0018523254, -0.009425522, 0.040299673, -0.02211658, -0.0070691416, 0.017106952, -0.0049817967, 0.024491513, -0.024738902, 0.0097904205, 0.01825731, 0.014212499, 2.1296233e-05, 0.016760606, 0.02548107, 0.009641987, 0.0052879406, 0.010520219, -0.001832225, 0.008813234, -0.0061043245, 0.04504954, -0.015573139, -0.028152872, -0.021102285, -0.01622872, -0.014261978, -0.028969256, 0.014274347, -0.0055786227, -0.040077023, -0.017181167, -0.0119117815, 0.02277216, 0.027930222, -0.010433632, -0.0343376, -0.0092832735, 0.007706168, 0.016735869, -0.001458049, -0.033323303, -0.0365641, -0.02188156, -0.016810084, 0.06001658, -0.004848825, 0.00980279, -0.016006071, -0.03159158, -0.07144595, 0.008670986, 0.0010807809, -0.005751795, 0.037553653, 0.010105842, 0.0025712997, 0.011033551, -0.0061630793, -0.027460182, -0.040472846, -0.0057579796, 0.0041715973, 0.017651208, -0.03562402, -0.017069843, -0.004647821, 0.04937885, -0.03000829, -0.011188169, 0.030131985, 0.01960558], "content_hash": "6ba40353610070d5ee4aa3edebdef5ec"}, {"content": "Good for factual, predictable text. - **High temperature (e.g., 1.5)**: Spreads probability more evenly, making less likely tokens more probable. Good for creative, varied text. - **Temperature = 1.0**: The default, unmodified distribution. ## Greedy Decoding vs. Sampling Once we have probabilities, how do we pick the actual next token?\n\n- **Greedy decoding**: Always pick the token with the highest probability. Simple but can be repetitive.\n- **Sampling**: Randomly pick a token weighted by the probabilities. More varied but less predictable.\n- **Beam search**: Explore multiple possible sequences simultaneously and pick the best overall path. This is available in the dashboard's generation controls.\n\n## What You See in the Dashboard\n\nIn **Stage 5 (Output Selection)** of the pipeline:\n\n- The **predicted token** is highlighted after your prompt text, along with its confidence percentage\n- A **top-5 bar chart** shows the five most likely next tokens and their probabilities\n- A note explains how beam search and other techniques can influence the final selection beyond just the top-1 token", "source_file": "output_and_prediction.md", "chunk_index": 1, "embedding": [0.024105849, -0.007743178, 0.009488625, -0.008533298, 0.006805808, -0.016836742, -0.0059151268, 0.016290842, 0.00787247, 0.040511616, 0.022942217, 0.006665741, -0.052808758, -0.0077862754, 0.035569772, -0.022870388, -0.015112844, 0.010645074, -0.038184352, 0.03355856, -0.01818713, -0.029378105, 0.018172763, 0.031949587, 0.019523151, -0.018172763, 0.009014552, 0.05645768, 0.018273326, -0.01263474, -0.02588721, -0.042781416, -0.013906115, -0.04464897, -0.024824139, -0.004471362, -0.0181153, 0.024781043, 0.005645768, -0.03393207, -0.0060767424, -0.012814312, -0.04108625, 0.011758424, -0.010515781, 0.030943979, 0.007894019, -0.06625517, -0.007836555, 0.06556561, -0.03252422, 0.05372817, -0.029406836, 0.015012283, 0.020169614, -0.019753003, -0.056256555, -0.004873605, -0.016750548, -0.07470227, 0.017871082, 0.0036848332, 0.041057516, 0.04597063, 0.019106543, 0.0007762033, -0.051688224, 0.025844112, 0.001905267, 0.02004032, 0.052550174, 0.008461469, 0.016922938, -0.023488117, -0.03884518, -0.025599893, -0.012886141, -0.033299975, -0.03375968, -0.03987952, 0.014825528, -0.023057144, 0.011729693, -0.056199092, -0.013877383, -0.05002179, 0.02376107, -0.020069052, -0.038241815, -0.055825584, -0.058468893, -0.012972336, -0.022109, -0.0018442123, -0.008993004, -0.023128973, 0.026174527, -0.02526948, 0.00059573265, 0.0036650803, -0.004819733, -0.040482886, -0.03427685, -0.031748466, 0.07906948, -0.010968304, -0.060278986, 0.014602858, -0.04752214, -0.04108625, -0.07797768, -0.039908253, -0.043413512, 0.07274852, 0.018474447, -0.017440107, 0.023358826, 0.035253726, 0.039448544, -0.008511749, -0.039046302, -0.0351388, 0.005261482, 0.021606196, -0.029234447, 0.023502484, 0.024163311, -0.05789426, 0.0020148065, 0.018029105, 0.0072619226, -0.0015775467, 0.021979706, -0.017583765, -0.024996528, 0.0041589052, 0.011162243, 0.012146302, -0.016649988, 0.06361186, 0.014868625, -0.028386863, 0.008418371, -0.0038213085, -0.043413512, -0.031087637, -0.040310495, 0.02203717, -0.06211781, 0.03496641, 0.009445528, -0.027840963, -0.061543178, -0.0055092927, 0.025930308, -0.044304192, -0.02299968, 0.032093246, -0.023646142, -0.032236904, -0.02255434, 0.019178372, -0.0016457844, 0.006008505, -0.032495487, 0.011815888, -0.015313965, 0.029967103, -0.038557865, 0.016793646, 0.0151846735, -0.01540016, -0.0058935783, 0.00040448766, 0.01275685, 0.006191669, -0.057405822, -0.047780722, -0.0049023367, -0.032121975, -0.025197651, 0.031030174, -0.005300988, -0.0277404, 0.009093565, -0.02107466, 0.008562029, 0.00843992, 0.0040511615, 0.0040439786, -0.015888598, 0.00016498248, 0.051487103, -0.0135182375, -0.056227826, -0.017540667, 0.09659579, 0.034736555, 0.0059115356, -0.0025247931, 0.009639466, 0.0027061617, 0.020571856, 0.001292026, -0.010199733, 0.0052004275, -0.014179066, 0.007405581, -0.048642673, -0.03321378, 0.012361789, -0.030398078, 0.02781223, 0.015270868, -0.07159925, 0.012060107, -0.012972336, 0.072001494, 0.03255295, 0.0629223, -0.0078221895, 0.0027528505, 0.013094447, -0.0031407278, -0.02815701, 0.03215071, 0.0396784, -0.0058863955, 0.025585528, -0.0030275967, 0.038586598, -0.004636569, -0.0012489286, -0.01732518, -0.02014088, -0.008777517, -0.009316235, -0.015902964, 0.007042844, -0.013396128, -0.034851484, 0.009811856, 0.053354662, 0.016937304, 0.01633394, 0.010551696, -0.05981928, -0.025944673, 0.056428947, 0.06832384, 0.023588678, -0.048039306, -0.031547345, -0.02929191, 0.009107931, -0.030283151, -0.032639146, 0.0062060347, -0.011787157, 0.0029360147, -0.01677928, -0.0116004, -0.05821031, 0.0061306143, -0.025312576, -0.014710601, -0.021203952, -0.044620242, -0.0323231, -0.04651653, -0.036460456, -0.014983552, -0.013173458, 0.04390195, -0.054417733, -0.03416192, -0.013266836, -2.6571159e-05, -0.049533352, 0.02746745, -0.03881645, -0.015859867, 0.029004592, 0.008152603, -0.012699386, -0.009862136, 0.024651749, 0.0064430707, 0.020083418, -0.039793324, -0.005638585, -0.009072016, -0.009933965, -0.019408224, 0.020298906, 0.027840963, 0.017856715, 0.00070796564, -0.043269854, -0.022583071, 0.028300667, -0.060393915, -0.011363365, 0.018373886, 0.013712176, 0.016348304, -0.011621949, 0.02973725, -0.012570093, -0.012893325, 0.0026846128, 0.031576075, 0.0055200667, -0.016951669, -0.009423979, -0.03008203, 0.09027482, -0.050395302, 0.030024566, -0.061773032, -0.0036345527, -0.014459199, 0.04637287, 0.019997224, -0.007800641, -0.02114649, 0.021491269, 0.013453592, 0.00854048, -0.032236904, 0.016966036, 0.007064393, 0.014107237, 0.053756904, -0.046746384, -0.040367957, 0.038960107, 0.0076282513, -0.02578665, 0.041718345, -0.028544888, -0.02475231, -0.020241443, -0.035426114, -0.011140695, -0.025700454, -0.02860235, 0.04780945, 0.034535434, 0.024910334, -0.020816075, 0.014653138, -0.012591642, -0.009438344, 0.028746009, 0.006773485, 0.015170308, -0.01093239, -0.009933965, -0.0068920027, -0.02534131, 0.042120587, 0.022094633, 0.052808758, 0.026145795, -0.016132817, -0.05071135, -0.041229907, 0.0053189453, -0.009955514, -0.023286996, 0.018919786, -0.016204646, 0.0059977304, -0.052205395, -0.023990922, -0.0035824766, 0.0069674235, 0.0014150334, 0.05895733, -0.0101207215, -0.005530841, -0.02248251, -0.039247423, 0.030225689, 0.05634275, -0.0039685583, -0.0027223232, -0.02988091, -0.010178184, -0.007793458, -0.016132817, -0.008942724, 0.005742737, -0.014258077, 0.05459012, -0.046430334, 0.054647584, 0.0475796, 0.04042542, 0.030915247, -0.0033382578, -0.053067345, -0.0260596, -0.0063137785, 0.023258265, 0.036000747, 0.017727423, 0.05734836, -0.0034477971, 0.051343445, 0.016405769, 0.03939108, -0.06545068, 0.00705721, 0.032782804, 0.030943979, 0.045051217, -0.008461469, 0.038414206, -0.031576075, -0.045712043, -0.031001443, -0.037695915, -0.015084112, -0.00074612483, 0.016003525, -0.028171375, 0.017957278, -0.031374954, 0.024350068, 0.061428253, 0.046947505, -0.02709394, 0.031030174, -0.009079199, 0.0402243, 0.008842163, 0.018172763, -0.043614633, -0.010860561, -0.008418371, -0.050653886, -0.005663725, -0.06286483, 0.007987397, -0.04878633, -0.055337142, -0.012641923, -0.020557491, -0.0138558345, 0.022856021, -0.042781416, 0.047838185, -0.03025442, -0.039103765, 0.084011324, -0.0088134315, 0.036948893, -0.005103458, 0.021807317, 0.0035591323, -0.01275685, -0.02080171, -0.018100934, 0.008755968, 0.040511616, -0.00083142193, 0.03829928, 0.011923632, -0.007215234, -0.011176609, 0.019164005, -0.026706062, 0.0018962885, 0.0007268208, 0.05407295, -0.014983552, -0.0048556477, -0.010716903, 0.021476902, 0.026418746, -0.010666623, 0.043672096, -0.0011295127, 0.091194235, 0.025642991, -0.023919092, 0.018546276, -0.037638452, -0.009036101, -0.04597063, 0.02141944, -0.018129667, 0.008676956, -0.014049773, -0.0134966895, -0.0014285013, 0.0088134315, 0.0006415237, 0.039046302, -0.02867418, -0.038730256, 0.04878633, 0.037753377, 0.029090788, 0.051056128, 0.029938372, -0.0075923367, 0.03080032, -0.033673484, -0.063899174, -0.020787343, -0.0019088585, -0.031030174, -0.007215234, -0.0007344526, 0.024694847, 0.0054949266, 0.011212524, -0.0036112084, 0.01275685, 0.0077359946, 0.016693084, 0.004094259, -0.04375829, -0.029062057, -0.048240427, -0.0044138986, -0.00979749, 0.040655274, -0.03852913, 0.010235648, 0.04056908, -0.05387183, -0.023301363, -0.0038536317, -0.023315728, -0.022468144, -0.00015027996, -0.015270868, 0.021045929, -0.0060767424, 0.019365128, 6.543183e-05, -0.012570093, 0.00611984, 0.0001687984, -0.016693084, -0.009811856, 0.0075205076, -0.017526302, 0.018000374, -0.023416288, -0.016420133, 0.0034262484, 0.0045647398, -0.003395721, -0.002354199, -0.02815701, 0.021850415, -0.015874233, -0.0029575634, 0.0040511615, -0.0019878705, 0.0339608, 0.023660507, -0.0014617223, 0.027611108, -0.0073624835, -0.02403402, -0.0040188385, 0.0039470093, 0.014839893, -0.004765861, -0.027036475, 0.027682938, -0.0076282513, -0.004622203, 0.048441548, -0.011305901, -0.022439413, 0.010458318, 0.018833593, -0.013776823, 0.009043285, 0.017612496, 0.011162243, -0.017310815, -0.017483205, 0.044131804, -0.005484152, -0.022640534, 0.044189267, -0.0362306, 0.023746703, -0.018603738, -0.04197693, -0.03878772, 0.0064682113, 0.01313036, -0.03657538, -0.04752214, -0.0068417224, -0.012850227, 0.02598777, -0.045712043, 0.020772977, -0.0075061414, -0.00826753, -0.038730256, -0.04720609, -0.053986758, -0.027323792, 0.015773673, -0.01201701, -0.015199039, -0.0017247965, 0.0022446597, -0.006270681, -0.00035981892, 0.00427024, 0.003013231, -0.031058906, -0.022985315, -0.0022590254, 0.011758424, -0.010953939, -0.016247744, -0.020241443, -0.006223992, 0.008360908, 0.0055164755, 0.0071038986, -0.012512631, 0.02746745, 0.026461843, 0.054762512, -0.003474733, -0.030570468, 0.01966681, 0.007240374, -0.029995834, -0.008095141, 0.014322724, -0.0018163786, 0.011485474, 0.00080134347, 0.0064179306, -0.017971642, -0.030771589, -0.023057144, 0.05088374, -0.01846008, 0.0025858479, -0.0447639, -0.01708096, -0.019796101, -0.003943418, -0.012361789, -0.001424012, 0.002821088, -0.033644754, -0.033271242, -0.010573245, 0.05700358, 0.03200705, 0.020643685, 0.010853378, 0.019436955, -0.07694334, 0.033673484, -0.0060192794, -0.0115142055, -0.024881603, 0.0033741724, 0.014839893, -0.051199786, -0.008008946, 0.010156636, -0.023358826, 0.003731522, -0.055164754, -0.037638452, 0.021663658, -0.018072203, 0.045597117, -0.016736181, 0.01801474, 0.01110478, -0.0022859613, 0.00082468794, 0.010767183, 0.0140138585, -0.018057838, 0.0013800167, 0.005846889, 0.037552256, -0.009682564, -0.06941565, -0.009344967, 0.004295381, -0.0069422834, -0.034391776, -0.013388946, -0.01443765, -0.015486356, 0.0072583314, 0.011981095, 0.0010855173, -0.006349693, -0.01495482, -0.001338715, -0.00145005, -0.018905422, 0.016290842, -0.024651749, -0.004931068, 0.0034190654, -0.031403687, -0.012749666, 0.015428892, -0.043269854, -0.0046581174, -0.004079893, -0.03585709, -0.0015910147, -0.024393165, -0.04637287, 0.014926088, 0.01626211, 0.018431349, -0.012613191, -0.050797544, -0.019623712, 0.024321336, -0.028961496, -0.017684326, -0.00346755, -0.0035196263, 0.0073912153, 0.0027223232, -0.0004233428, 0.025700454, 0.007484593, -0.003111996, 0.004008064, 0.020514393, -0.018847957, -0.04651653, -0.012965154, 0.013532604, -0.016477598, -0.017957278, 0.04335605, 0.008597944, 0.014674687, -0.022511242, -0.012591642, 0.010989853, 0.009567637, 0.03146115, -0.021017198, 0.019278932, -0.030915247, -0.017770521, 0.020945368, 0.0004597063, 0.00787247, 0.0071793194, 0.0003079673, -0.0069279172, -0.0029773165, -0.006917143, 0.06924326, -0.0073912153, 0.015917331, 0.03217944, -0.03321378, -0.0035268092, -0.018144032, -0.018934153, -0.026030868, 0.003099426, -0.017296448, -0.0116004, -0.011011402, -0.0022500467, -0.0023919092, -0.048987452, -0.021534367, 0.0007155975, 0.025973406, 0.0035393792, -0.0226549, -0.002056108, 0.021778585, -0.12814313, -0.016563792, 0.023947824, 0.016175915, -0.010623525, -0.036000747, 0.04582697, -0.0052866223, -0.015098479, -0.028027717, -0.01455976, -0.016420133, 0.0017364686, 0.005085501, -0.018172763, -0.026318185, 0.030024566, -0.007534873, -0.004295381, -0.01715279, -0.049447156, 0.005609853, -0.036661576, -0.0116506815, -0.054877438, 0.0063065956, 0.0008888852, 0.008102323, 0.022238292, -0.008648224, 0.0015766489, 0.040253032, 0.02588721, 0.007994579, 0.01122689, -0.0034064953, -0.015558184, -0.022439413, -0.006166529, 0.0015155942, 0.036833964, -0.03131749, 0.043614633, -0.031604808, 0.020183979, -0.025025262, -0.02973725, -0.019164005, -0.009732844, 0.010063258, -0.02021271, 0.012412069, 1.1531939e-05, -0.0067806677, 0.01784235, 0.017124059, -0.00984777, 0.018618105, 0.0055236584, 0.019968491, -0.01381992, 0.003449593, 0.05372817, -0.017368278, 0.031777196, -0.025844112, 0.005347677, 0.017727423, -0.021678025, -0.017698692, -0.024378799, 0.0133386655, -0.009438344, 0.0055954875, -0.012376155, 0.015802404, 0.042034395, -0.0059259012, -0.031547345, 0.009488625, 0.021548731, -0.021836048, -0.036144406, 0.040138103, 0.030139493, 0.03131749, 0.009064834, -0.012577277, 0.0136978105, 0.0123474235, 0.028918399, -0.022669267, -0.014588492, -0.02403402, -0.050941203, -0.001087313, 0.021117758, 0.016319573, 0.04143103, -0.016118452, 0.047119893, 0.017267717, -0.017971642, -0.039448544, 0.001932203, 0.023646142, 0.0054626036, -0.0070536183, -0.024781043, -0.012088839, -0.00066486816, -0.0070212954, 0.009819039, 0.017397009, -0.031949587, 0.022496875, 0.047148626, 0.033098854, 0.013144727, -0.04235044, 0.015888598, 0.012972336, -0.0029827037, 0.010860561, -0.03358729, 0.005850481, -0.013719359, 0.013173458, 0.021462537, 0.0046724835, -0.0048843795, -0.00020302946, 0.0077647264, 0.0060767424, -0.0063461014, -0.0015928105, 0.017540667, 0.0061737117, -0.047895648, 0.021936608, -0.022353219, 0.016405769, 0.017871082, 0.009438344, -0.020816075, -0.027955888, -0.00890681, 0.0047119893, -0.01922147, 0.023085875, 0.022496875, 0.00826753, -0.0026935914, 0.019063445, -0.03442051, 0.03444924, 0.005085501, -0.016405769, -0.0028641857, 0.025312576, 0.03375968, 0.049993057, 0.034391776, -0.01633394, 0.034592897, -0.041862004, -0.013324299, 0.022396315, 0.0034406143, 0.014739333, -0.025671722, -0.023315728, 0.0037889853, 0.031547345, 0.014739333, 0.009474259, 0.03373095, 0.04947589, -0.006726796, 0.037408598, -0.007886836, 0.03444924, -0.01784235, 0.014854259, 0.030771589, -0.025456235, -0.0025876437, 0.026892819, -0.011557303, -0.01608972, -0.0062670894, 0.014329907, 0.008619493, 0.0453098, -0.036259335, -0.00041054824, 0.038960107, 0.013554152, -0.0006769893, 0.027611108, -0.039822057, 0.0028318625, 0.021232683, 0.009459893, 0.00114747, 0.034018263, -0.018747397, 0.0024475767, 0.040138103, 0.020298906, -0.039075036, -0.0006590321, -0.0070787584, -0.0021674433, -0.017799253, -0.013683445, -0.0003039269, 0.003903912, -0.015644379, 0.008806248, -0.019293299, -0.02939247, -0.0054123234, -0.008533298, 0.006647784, 0.0072044595, 0.020528758, 0.025183285, -0.02626072, -0.008461469, -0.023459386, -0.0118949, 0.036862697, 0.010329026, 0.01201701, 0.03126003, 0.0032161484, -0.019020347, 0.012498264, 0.006166529, 0.014825528, 0.011593218, -0.019968491, -0.016520694, -0.017454473, -0.032639146, 0.0021800133, -0.009567637, -0.0065149, -0.035081334, -0.00011038114, 0.028803471, 0.05895733, -0.0038823632, -0.0011690187, 0.005670908, 0.011643498, 0.037552256, 0.015055381, 0.0007959563, -0.0017993192, -0.0021243459, 0.008863712, -0.002618171, 0.02159183, -0.011937997, -0.019968491, 0.0018567824, 0.024120213, -0.011341816, -0.04444785, 0.034851484, 0.013877383, 0.040166836, -0.007800641, 3.083601e-05, -0.011241255, -0.027955888, -0.00022053781, -0.026016502, 0.008554847, -0.019178372, -0.0007667757, -0.012814312, -0.0096681975, -0.005534433, -0.028789107, -0.013604433, 0.030311882, -0.00055757345, 0.021649294, 0.008734419, -0.0023757478, -0.0016754139, -0.046459068, 0.0009750801, -0.027180133, 0.019954126, 0.010106355, -0.02152, 0.0047550867, -0.0008794576, 0.014825528, -0.00067474466, 0.011909266, 0.010745634, 0.033702217, -0.027826596, -0.010774366, -0.0025553205, 0.0260596, 0.0008691322, -0.01653506, 0.027553646, 0.016391402, -0.034937676, 0.009122296, -0.016017891, 0.016046623, -0.0096466495, -0.0006092006, -0.011607584, 0.0036058212, 0.028415594, -0.008547664, 0.00084129843, 0.029191349, 0.004180454, 0.00969693, 0.0351388, 0.011219706, 0.020126516, 0.023301363, 0.013410495, -0.022769827, -0.006830948, -0.027180133, 0.0007708161, 0.009215674, -0.0339608, 0.01137773, 0.004259466, -0.020198345, -0.0072547398, 0.004938251, 0.0011681209, 0.024824139, -0.0118518025, 0.017684326, -0.01677928, 0.074817196, -0.010530147, -0.043672096, -0.005530841, 0.0068740454, -0.016592523, -0.010329026, -0.017353913, -0.014049773, 0.0039649666, 0.0028534112, 0.010896475, 0.0109754875, -0.017827984, 0.021735488, -0.02623199, -0.0030186183, -0.0032071697, 0.038586598, -0.010084807, 0.026907183, 0.021893512, -0.0057858345, 0.012957971, -0.012419253, 0.014710601, -0.017698692, -0.014703418, 0.011586035, 0.024623018, -0.008454286, -0.028832203, -0.015845502, 0.0058576637, -0.026706062, -0.009172577, -0.023646142, 0.025197651, -0.020183979, 0.016060987, -0.01428681, -0.011779973, -0.002993478, -0.011887717, 0.023646142, 0.027050842, 0.023603044, -0.007193685, 0.002004032, -0.025973406, 0.015759306, 0.00708235, -0.017598132, -0.016003525, 0.037839573, -0.0447639, -0.0023829306, 0.022942217, 0.014021042, 0.0169804, -0.03778211, 0.008174152, -0.027955888, -0.006277864, 9.416347e-05, -0.039276157, 0.015658746, 0.008095141, 0.009208491, 0.0305992, -0.018402617, -0.0028965087, -0.02413458, 0.010048892, 0.03410446, 0.004212777, 0.009488625, -0.008698504, -0.0003375968, 0.020456929, -0.014882991, -0.018546276, -0.05125725, -0.014969186, -0.0071110814, 0.026504941, 0.04895872, -0.0067088385, -0.01856064, 0.009172577, -0.036747772, 0.0014626202, 0.028904032, 0.04967701, -0.001220197, -0.027251963, -0.00067968294, 0.0052076103, -0.0075779706, 0.0006374833, 0.0050926837, -0.001258805, -0.020155247, 0.011320267, -0.02860235, -0.018790495, -0.06907087, -0.01228996, 0.009201309, 0.00029360148, 0.005243525, 0.0114711085, 0.038500402, -0.007136222, -0.017095327, 0.028961496, 0.004191228, 0.02087354, -0.011988278, -0.025872845, 0.012555728, 0.03045554, 0.018603738, -0.022611802, 0.0006949466, 0.0021117758, 0.0032897731, 0.009991429, -0.0096251005, -0.021390708, 0.0021602605, 0.015586916, 0.015701843, 0.0041229906, -0.011837437, -0.015529453, -0.0021243459, 0.004162497, -0.011305901, 0.023674874, -0.008109506, -0.009438344, -0.030714126, -0.024852872, 0.008432738, 0.025585528, 0.008597944, -0.008360908, -0.0065795463, 0.015428892, -0.0049095196, -0.025183285, 0.014969186, 0.006033645, 0.003792577, -0.029248813, -0.030714126, -0.029794713, 0.013489506, 0.01364753, 0.016348304, 0.003560928, -0.03585709, 0.000646013, 0.024393165, 0.009941149, -0.02692155, -0.006985381, 0.0045755142, 0.002650494, -0.015773673, -0.032265633, -0.017454473, 0.027711669, 0.016750548, -0.0038751804, 0.0063532847, 0.0063137785, -0.0076067024, 0.0464016, -0.006744753, 0.010961122, 0.024192043, -0.004643752, -0.014739333, 0.02275546, -0.019451322, -0.010214099, -0.002354199, 0.013115995, 0.03045554, -0.01993976, -0.02516892, -0.011765608, 0.007621068, 0.016017891, -0.006324553, -0.012498264, 0.01308008, 0.0015972997, -0.02534131, -0.007692897, 0.014466383, 0.0049705743, -0.0113130845, 0.023315728, -0.010551696, -0.042264245, -0.00854048, 0.007251148, -0.015773673, -0.025398772, -0.0057463287, -0.016420133, 0.053670708, -0.042407904, 0.0039901068, -0.018661203, 0.010393672, -0.025499333, 0.020686783, 0.001015484, 0.009028919, 0.022080267, -0.02798462, 0.036259335, 0.020485662, -0.029306276, 0.022065902, -0.013590067, 0.019925395, -0.03252422, -0.0305992, 0.042953804, 0.016491963, -0.005394366, -0.037494794, -0.018043472, 0.010925207, -0.011880535, -0.049562085, 0.01149984, -0.0023775434, 0.04214932, -0.0051860614, 0.054647584, -0.015874233, 0.007064393, -0.017713057, 0.0005225568, 0.029248813, 0.0078221895, -0.028746009, -0.035081334, -0.004536008, -0.05019418, 0.0051285983, -0.0074199466, -0.02988091, -0.006141389, -0.012957971, -0.003929052, -0.015687477, -0.011140695, 0.020945368, -0.007908384, 0.02368924, -0.034018263, -0.031202564, 0.030829053, -0.013834286, 0.010336208, -0.005638585, -0.015788037, 0.03884518, 0.043068733, 0.022080267, -0.009890868, 0.009617917, 0.0029234446, -0.0034064953, -0.00076991826, -0.0044067157, -0.0016799032, 0.0118302535, -0.014459199, -0.0066549666, 0.024766676, 0.0022500467, -0.033673484, 0.00890681, 0.03217944, 0.012519813, -0.022468144, 0.013238104, 0.011025768, -0.007915568, -0.022741096, -0.012225314, 0.015270868, -0.024464995, -0.016664352, 0.0004713785, -0.0099196, 0.016649988, 0.011758424, 0.011032951, 0.014150334, 0.009661015, -0.01263474, 0.036431722, -0.010874927, -0.027065208, -0.0260596, -0.0043348866, 0.040655274, 0.019709907, -0.019365128, -0.013245287, 0.014035407, -0.042235516, 0.013051349, 0.012670655, -0.008547664, 0.006141389, -0.023588678, -0.009165394, 0.0109754875, 0.025729187, -0.019982858, 0.016836742, 0.034707826, 0.03166227, -0.025743552, 0.005631402, -0.015069746, 0.00021638519, 0.0057858345, 0.050596423, -0.023875995, -0.0075061414, -0.0138558345, -0.017138425, -0.039649665, -0.012146302, 0.059589427, -0.018790495, -0.020169614, 0.031001443, -0.0027743992, -0.003925461, 0.009682564, -0.0094670765, -0.041172445, 0.0057822433, -0.027424352, 0.031949587, 0.014768064, -0.0063065956, -0.026002137, 0.00251761, -0.015270868, 0.027165769, 0.0069135516, -0.0058935783, 0.0098262215, -0.014897357, -0.057808064, -0.006931509, 0.017253352, 0.0057355543, 0.021994073, 0.016305206, -0.020442564, 0.004510868, 0.005056769, -0.03884518, -0.018144032, -0.0113130845, -0.013051349, 0.0356847, -0.014660321, 0.009768759, 1.6175578e-05, 0.05002179, -0.044390388, -0.025757918, 0.0283294, -0.006238358], "content_hash": "f4d060fd4556e9d8b101910303cfe309"}, {"content": "# Pipeline Stages\n\n## Overview\n\nThe pipeline visualization shows the 5 stages your text passes through inside the transformer model. A flow indicator at the top shows the path: **Input \u2192 Tokens \u2192 Embed \u2192 Attention \u2192 MLP \u2192 Output**. Click any stage to expand it and see details.\n\n## Stage 1: Tokenization\n\n**Icon**: Puzzle piece | **Summary shows**: \"X tokens\"\n\nThis stage displays how your input text was split into tokens. Each row shows:\n- The **token** (the text piece, displayed in a blue box)\n- An arrow pointing to its **ID** (the number the model uses internally, in a purple box)\n\nNotice that spaces are often attached to the beginning of the following word (e.g., \" cat\" with a leading space). This is normal for models like GPT-2 that use BPE tokenization.\n\n## Stage 2: Embedding\n\n**Icon**: Cube | **Summary shows**: \"X-dim vectors\"\n\nThis stage shows how token IDs are converted into numerical vectors using a pre-learned embedding table. You'll see:\n- A visual flow: Token ID \u2192 Lookup Table \u2192 Vector\n- The embedding dimension (e.g., 768 for GPT-2)\n- An explanation of how the lookup table was learned during training\n\n## Stage 3: Attention\n\n**Icon**: Eye | **Summary shows**: \"X heads \u00d7 Y layers\"\n\nThis is the most detailed stage. It includes:\n- **Head Categories**: Attention heads are automatically categorized by their behavior pattern (Previous-Token, First/Positional, Bag-of-Words, Syntactic, Other). Click each category to see which specific heads belong to it.\n- **BertViz Visualization**: An interactive attention map showing which tokens attend to which. Lines connect tokens on the left to tokens on the right. Thicker lines mean stronger attention.\n\n**Navigating BertViz**: Single-click a head square to select/deselect it. Double-click to show only that head. Hover over tokens or lines to see exact attention weights.\n\n## Stage 4: MLP (Feed-Forward)\n\n**Icon**: Network | **Summary shows**: \"X layers\"", "source_file": "pipeline_stages.md", "chunk_index": 0, "embedding": [0.0019218383, 0.023723403, -0.0063646673, -0.027934154, -0.031914346, -0.04448593, 0.005399956, 0.04958252, -0.017376937, 0.07023583, 0.046184793, 0.00021444348, -0.014343254, -0.04671872, -0.0022570603, 0.0038618788, 0.0011975465, 0.039146647, -0.0034219946, 0.07421602, 0.035554767, -0.024827663, 0.0061219726, 0.047471073, 0.008372965, 0.024427217, -0.00019586217, 0.05315013, 0.009574304, 0.0063039935, -0.01743761, -0.018808836, -0.014549544, -0.0051056887, 0.012814278, -0.015580997, -0.02674495, 0.06475093, -0.030822221, 0.035967346, 0.011734286, 0.008542852, -0.054217987, 0.013081241, -0.00030374754, 0.04994656, 0.028662238, -0.031137723, -0.009962616, 0.05814964, -0.008761277, 0.047082763, -0.026623603, -0.014695161, 0.02502182, 0.0028016064, -0.007620612, -0.026235292, -0.01741334, -0.04676726, 0.017109973, 0.015738748, 0.01497426, 0.005679055, 0.011552266, 0.011831365, -0.077953525, 0.02133286, -0.011345975, 0.02066545, 0.062469605, 0.0146587575, -0.029875712, 0.02548294, 0.04960679, 0.0025725632, -0.034244217, 0.047398265, -0.016454697, -0.03800598, 0.03128334, -0.058926262, 0.008512515, -0.069362134, -0.007596343, -0.05179104, -0.029341783, -0.041403707, -0.045942098, -0.05766425, -0.011352043, 0.016903684, -0.034681067, -0.00085777394, 0.034195676, -0.012790008, 0.007839037, -0.0076630837, -0.017656036, 0.02338363, 0.06431408, -0.0075113997, -0.026550796, -0.00829409, 0.044583008, -0.021466343, 0.022946779, 0.038272947, -0.027060455, -0.03936507, -0.08795254, 0.0018611646, 0.026016867, 0.036525544, 0.043636497, -0.007535669, 0.026259562, 0.004435245, 0.01592077, -0.021624094, -0.042908415, -0.015629536, 0.026599335, 0.006225118, -0.014585949, 0.0038952492, 0.025288783, 0.00036006028, 0.0040044617, -0.032011423, -0.04533536, -0.01741334, 0.016066385, -0.029171897, -0.0065648905, -0.028832125, -0.0335404, 0.03504511, -0.017825922, 0.030264024, -0.0052816425, -0.021684768, 0.004589963, -0.013785057, -0.02720607, 0.031623114, -0.037957445, 0.061450288, -0.041063935, -0.017498285, 0.011066876, 0.014197637, -0.03351613, -0.04819916, -0.036598355, -0.019718941, -0.000510038, 0.0572274, 0.023007452, -0.031186262, 0.010247782, 0.0066012945, -0.0070624147, -0.0127171995, -0.059314575, 0.039632037, -0.014039886, -0.015908634, -0.040772703, -0.06790596, -0.022279369, -0.042884145, -0.013748652, -0.02584698, 0.021138703, -0.0148043735, -0.06945921, -0.062323987, 0.0052270363, -0.04686434, -0.014318985, 0.0020082982, -0.052033734, -0.02130859, -0.032957934, 0.005093554, -0.0035585104, 0.05392675, -0.010975866, 0.024609238, -0.0035736787, 0.045480978, 0.009016107, -0.0018808836, -0.02584698, -0.0020720055, 0.056547854, -0.01569021, 0.022218695, 0.018942317, 0.03577319, 0.042956952, 0.017583227, -0.019500516, 0.049533978, 0.017122108, -0.032933664, 0.008057462, -0.017898731, 0.0037192956, 0.017231321, -0.020070847, 0.022546334, 0.02815258, -0.045456707, -0.02176971, -0.06572171, 0.058343794, 0.034171406, 0.0126443915, 0.059702884, 0.008664199, 0.018808836, 0.011248897, -0.037350707, 0.028735047, 0.034025792, -0.035991617, 0.00679545, 0.004410975, -0.012419899, 0.042762797, -0.016915817, 0.017461881, 0.0030746378, -0.044340312, -0.0015373189, -0.024682047, 0.023517111, -0.023456438, -0.016114924, 0.043296725, 0.032084234, 0.022910375, 0.033783097, 0.02499755, -0.0023586887, -0.042956952, 0.039753385, 0.037957445, 0.037836097, 0.021247918, -0.04805354, 0.013263263, -0.0059611876, -0.013785057, -0.019427707, 0.022267234, 5.0814193e-05, 0.0012422933, 0.040578548, -0.004565693, -0.04353942, -0.010745306, -0.0021766676, -0.011139684, -0.027303148, -0.011801028, -0.012438101, -0.05407237, -0.0038558112, 0.009331609, 0.014379658, 0.04082124, -0.044534467, 0.0093255425, -0.0009449923, 0.03189008, -0.029171897, 0.031186262, -0.011740354, -0.01923355, 0.036865316, 0.050237793, -0.019961635, 0.0024588, 0.029244706, -0.018396255, 0.0055668084, -0.0083122915, 0.02384475, 0.012705065, 0.047859386, 0.018905913, 0.004520188, 0.018978722, 0.034705337, 0.026987646, 0.010211377, -0.0075599384, 0.045602325, -0.084748976, 0.030069867, 0.034025792, 0.008779479, 0.032957934, -0.0035767125, -0.044777162, 0.001953692, -0.021417804, -0.0045717605, -0.048320506, 0.030652335, -0.059848502, -0.0053392826, 0.009756325, 0.03946215, -0.008561054, 0.022085214, -0.023092397, 0.0008608076, -0.016296946, -0.019694671, -0.0026362706, -0.006789383, -0.0047841184, -0.016369754, 0.05708178, 0.017498285, -0.024669912, 0.036501277, -0.042156063, -0.028735047, -0.012850682, -0.05392675, -0.002000714, 0.006376802, 0.052955974, -0.022740489, 0.04895151, -0.07198323, -0.017983673, -0.041112475, -0.010417668, -0.05353844, 0.06441116, 0.0018990857, -0.006922865, 0.015144146, -0.0021023424, -0.02384475, 0.00910105, 0.013651574, -0.0141369635, 0.0027439664, 0.006082535, 0.00022525097, 0.00020458401, 0.008724873, -0.000875976, -0.021721171, -0.00863993, 0.014889317, 0.018942317, -0.021915328, -0.0121954065, -0.0045747943, 0.01681874, -0.024584968, -0.020580506, 0.021466343, 0.012705065, 0.039753385, -0.0026954275, -0.017498285, -0.0151805505, 0.0027834042, -0.011170021, -0.05140273, 0.0381516, -0.0027803706, 0.006370735, -0.022109482, 0.0034826682, -0.0012809727, 0.043442342, 0.030264024, 0.0013037253, -0.02502182, 0.0381516, 0.00060787424, -0.02628383, -0.004550525, 0.020908145, 0.012632256, 0.01006576, -0.024160253, 0.033394784, 0.0286137, 0.036525544, 0.01766817, -0.018736027, -0.061159052, -0.04130663, 0.0286137, 0.014513141, 0.03407433, -0.013299667, 0.01810502, 0.011018338, 0.04222887, -0.0056638867, -0.003561544, -0.028055502, -0.033297706, 0.02997279, 0.02584698, 0.014088425, -0.0035463755, 0.048417583, -0.016879413, -0.026526526, -0.03890395, -0.010484409, -0.03621004, 0.013069107, 0.062275447, -0.05048049, 0.0047628824, 0.00930734, 0.0148043735, 0.051888116, 0.027351687, 0.004471649, 0.0068803933, -0.054703373, 0.02856516, 0.03397725, -0.02545867, -0.011576535, 0.014391793, -0.048029274, -0.04807781, -0.05120857, -0.032957934, 0.012656527, -0.012171137, -0.0763032, 0.0058277054, -0.02538586, 0.010217445, 0.046937145, -0.0062008486, 0.044752896, -0.017522555, -0.043223917, 0.07353648, 0.018942317, 0.040942587, -0.027497305, 0.03310355, 0.024779124, 0.010053626, 0.0043047965, -0.013675843, -0.001236226, -0.013967077, -0.047835115, 0.036938127, 0.013493823, -0.015386841, -0.011619006, -0.010059694, -0.01661245, 0.012826413, -0.012019453, 0.014124829, -0.026550796, -0.031040646, -0.018905913, 0.016891548, 0.015860096, -0.032933664, -0.0074628606, -0.0029366054, 0.091787115, -0.05538292, -0.012972029, 0.013518092, -0.011837432, 0.0021614993, -0.02499755, 0.02246139, -0.012031588, -0.004671872, -0.044413123, -0.07756521, -0.046621643, -0.012802143, -0.023747671, 0.006971404, -0.030919299, 0.0035979482, 0.01933063, 0.043515153, -0.013979212, 0.044704355, 0.050286334, 0.0026848095, 0.037059475, -0.05548, -0.0125715835, 0.0146587575, -0.0032248052, -0.0031489632, 0.028128311, 0.024427217, -0.0003867946, 0.020859605, 0.009762392, 0.0126443915, -0.0014463085, 0.034753874, 0.0032612095, -0.008172742, -0.007123088, -0.050237793, -0.036913857, -0.029875712, -0.017449746, -0.00031607188, -0.015398975, 0.002528575, -0.017401207, -0.06407139, -0.0035221062, -0.03084649, -0.033758827, -0.04358796, 0.01635762, 0.003570645, 0.0070806164, 0.0017868393, 0.030021328, -0.002020433, 0.015095607, -0.0018050415, 0.016272677, 0.026186753, 0.024609238, 0.015047069, -0.029196167, 0.028977742, 0.0018293109, -0.0019006025, -0.006825787, -0.006625564, -0.0014197638, 0.009877672, -0.025628556, 0.018602544, 0.010308456, 0.019148609, -0.020216465, -0.017886596, 0.026890568, 0.040796973, -0.014464601, 0.011928442, -0.021199377, -0.022279369, -0.022534199, 0.02810404, 0.017692441, -0.012607987, -0.026526526, -0.0062372526, 0.039170917, -0.01635762, -0.0026681244, -0.0393408, -0.03713228, -0.017425476, -0.016576044, 0.010083963, -0.0005544056, -0.012159002, -0.0147315655, -0.00040689277, 0.004080304, 0.03307928, -0.02382048, -0.020362081, 0.013166185, -0.006043097, -0.028055502, 0.02633237, -0.011261032, -0.04715557, -0.012753604, -0.0010026323, -0.017886596, -0.021405669, 0.011182156, -0.031234803, 0.013105512, -0.023189474, -0.014173368, 0.0060279286, 0.01584796, -0.020580506, -0.046403218, -0.023953963, -0.013566631, -0.014488871, -0.005363552, -0.008591391, 0.0022707118, 0.0032278388, -0.02112657, -0.01294776, -0.0068318546, -0.0012802143, -0.03904957, 0.006255455, -3.867946e-05, 0.006643766, -0.017595362, -0.019718941, 0.03363748, -0.026575064, 0.023274418, 0.01886951, 0.0029912116, -0.0031186263, 0.0090646455, 0.00850038, 0.002384475, -0.008737007, -0.026138214, 0.021065896, -0.007657016, -0.019160744, -0.017862327, 0.0052513056, 0.0038952492, -0.009198127, 0.017704574, -0.034705337, -0.030555258, -0.02900201, -0.012899221, 0.035360612, -0.005418158, -0.00335222, -0.015083473, -0.04091832, -0.011430918, -0.009483294, 0.02676922, -0.023917558, 0.015496054, -0.034729604, -0.018796701, -0.02158769, 0.026405178, 0.024839798, 0.018893778, 0.022449255, 0.006049164, -0.041864827, 0.04630614, 0.016248407, 0.030215485, -0.0028971673, -0.02382048, 0.018444793, -0.038734067, 0.0027318317, 0.017983673, -0.018529737, 0.009107117, -0.036622625, -0.00070950267, -0.011036539, -0.01338461, 0.0034614324, -0.014610218, 0.016636718, -0.008476111, -0.030773683, -0.0050025433, 0.003007897, -0.022570603, 0.00081530237, 0.020968817, -0.0013575732, 0.054169446, -0.014403928, -0.06479947, 0.0001827794, -0.0158965, -0.058489412, -0.014246176, -0.02200027, -0.0054909666, -0.015423245, 0.03366175, 0.0115644, -0.029366054, -0.044825703, 0.0076388144, -0.009804864, -0.011000135, -0.024208792, 0.03043391, -0.020313542, -0.02771573, -0.0051481603, -0.0025725632, -0.015508188, 0.027448766, -0.030992107, 0.01078171, -0.0013295116, -0.04261718, -0.029560208, -0.018250639, 0.0014493421, 0.0023905423, 0.019342763, 0.023310822, -0.013785057, -0.066255644, 0.0012096812, 0.03264243, -0.018760297, -0.0003494424, -0.052422047, -0.012680796, -0.038806874, 0.00050548743, -0.020677585, 0.023784077, -0.021320725, -0.022607006, -0.01840839, 0.033297706, -0.00976846, -0.0231288, -0.011782825, 0.008700604, -0.02223083, -0.010884855, -0.01453741, 0.014707296, 0.022971049, -0.0062190504, 0.03504511, -0.0027652022, -0.0025740801, 0.023055993, -0.006850057, 0.0052422043, 0.011236763, -0.009829134, 0.03225412, 0.0056426506, 0.040020347, 0.010411601, 0.0033764893, -0.03587027, 0.016709527, -0.009295206, 0.049533978, -0.0066619683, 0.038782604, -0.02900201, -0.010678565, -0.0040044617, -0.0036737903, -0.0063039935, 0.007274772, -0.04552952, 0.01569021, -0.0067772483, -0.028953472, -0.004107607, -0.00817881, -0.011030472, -0.06266376, -0.019852422, 0.044752896, -0.054460682, -0.015192686, -0.01382146, 0.0065163514, -0.09207835, -0.011097213, 0.010842384, -0.021272186, -0.013129781, -0.020495564, 0.018396255, 0.0028880665, -0.017425476, -0.017583227, 0.026866298, 0.0011884454, -0.022121618, -0.00011793443, 0.019828154, -0.043054033, 0.05276182, 0.0017762214, 0.01340888, -0.03730217, -0.027618652, -0.006492082, 0.034171406, 0.005630516, -0.047082763, 0.021466343, -0.014743701, 0.0079785865, 0.028055502, 0.020531967, -0.027497305, 0.03186581, 0.006710507, 0.0067833154, 0.051111493, -0.0035676113, -0.015253359, 0.008542852, 0.030919299, 0.021818249, 0.014003482, -0.025337322, 0.019816019, -0.02412385, 0.040020347, 0.011479457, -0.029633017, -0.029171897, 0.024500025, 0.028880663, -0.035821732, -0.019112203, -0.009756325, -0.0170493, 0.0526162, 0.007687353, -0.017558958, -0.0196704, 0.03392871, 0.013712248, -0.0075235344, 0.036549814, 0.025361592, 0.024245195, 0.026429448, 0.003188401, 0.012341023, 0.025046088, -0.019682536, -0.02907482, 0.013979212, -0.010399466, 0.008864422, -0.026453717, -0.0015024316, -0.0020507697, 0.00413491, 0.009119252, -0.0019658266, -0.017012896, 0.037593402, 0.01520482, -0.02548294, 0.0052907434, 0.010721036, 0.017401207, -0.008482178, -0.047519613, 0.021260051, 0.008979702, 0.004225921, 0.0014766452, -0.0057124253, -0.012031588, -0.021891057, -0.009968683, 0.024766989, 0.03363748, 0.06635272, -0.0017534689, 0.062275447, 0.00647388, -0.033831634, -0.025749903, -0.04720411, 0.010223513, -0.0051390594, -0.0030321663, -0.036743972, -0.03630712, 0.018092886, -0.018736027, 0.03361321, 0.02589552, -0.027497305, 0.024038905, 0.03907384, 0.0075113997, 0.0032430072, -0.043005493, -0.005075352, 0.014634487, -0.03710801, 0.018359851, -0.028298197, -0.016442563, -0.014780104, 0.019658267, -0.014088425, -0.010241714, -0.019597594, 0.013505957, -0.018990858, -0.010903058, -0.01292349, 0.014634487, 0.012080126, 0.031477496, -0.013141915, 0.009701719, -0.004231988, -0.00061849214, 0.02858943, 0.0332249, -0.019197147, 0.030191215, -0.024196656, -0.010939461, 0.0007178453, 0.013857865, 0.02638091, 0.041525055, -0.0020659382, 0.01497426, -0.007001741, 0.022509929, -0.0036980598, -0.03397725, -0.035166454, 0.03489949, 0.02292251, 0.032399736, 0.021599824, -0.017316263, -0.0005434085, -0.011212493, -0.01269293, 0.02130859, 0.0047355792, 0.0067651137, -0.03315209, 0.022291504, 0.043903463, 0.023990367, 0.010466207, 0.019221416, 0.0016139194, 0.06135321, 0.015095607, 0.016952222, -0.011321706, 0.027084723, -0.0071352227, 0.003637386, 0.026526526, -0.003570645, 0.0019006025, 0.020689718, -0.022873972, -0.002355655, 0.0066194967, 0.0116736125, 0.005581977, 0.033297706, -0.016527506, -0.008069597, 0.004031765, -0.0074264565, -0.022364313, 0.037520595, 0.0011436986, 0.026065405, 0.031210532, -0.0024588, -0.025167435, 0.025240244, -0.009349812, -0.007620612, 0.004759849, 0.030967837, 0.014294715, 0.003437163, -0.0067772483, 0.02582271, -0.02276476, -0.0047477144, 0.04038439, -0.004356369, 0.02638091, 0.008670267, -0.030336833, -0.032084234, -0.021162974, 0.009404418, -0.0133118015, 0.0064799474, 0.0029153696, 0.007305109, -0.011303503, -0.0053787204, -0.008027126, 0.011534063, -0.014088425, 0.01840839, 0.01997377, 0.0023814412, 0.03128334, -0.020859605, 0.03502084, -0.014246176, 0.035239264, 0.030264024, -0.031040646, 0.021842519, 0.019427707, 0.011030472, 0.019706806, 0.016527506, -0.030021328, -0.013930673, -0.013882134, 0.039607767, 0.0068803933, -0.041209552, 0.022158021, 0.011782825, 0.038394295, 0.022048809, -0.022910375, 0.0056729876, 0.011655411, 0.03951069, 0.025361592, -0.032399736, 0.013190455, 0.0014531342, -0.020713989, 0.018092886, 0.026623603, 0.028856395, -0.0068075852, 0.026939107, -0.011879903, 0.002546777, 0.0033643546, 0.00658916, 0.006407139, -0.020531967, 0.024087444, -0.022291504, -0.0028167749, -0.028201118, 0.02905055, -0.026526526, 0.001685211, 0.028735047, -0.020653315, -0.0054424275, 0.011097213, -0.025774172, -0.018517602, 0.001838412, 0.009404418, -0.0003763663, -0.00817881, 0.017134242, -0.008203079, 0.0062493873, -0.008840153, -0.016988626, 0.008324427, -0.001962793, 0.0027940222, 0.0009131386, -0.0079785865, 0.0036434536, 0.020750392, -0.053441364, -0.017910866, 0.0012536696, -0.005985457, 0.0054090573, -0.00043533352, 0.04535963, -0.0060066925, 0.0064678127, 0.011825297, -0.03630712, 0.009289138, -0.03395298, -0.005933884, 0.024548564, -0.008882624, 0.025337322, -0.019985905, -0.00863993, 0.023286551, -0.009428687, 0.033443324, 0.003734464, 0.03189008, -0.005484899, 0.018189965, 0.016940087, -0.02905055, -0.025264513, -0.03138042, -0.0061614104, -0.01006576, -0.0245607, 0.03863699, 0.01766817, -0.016369754, -0.0041197417, 0.027545843, -0.02066545, 0.0142097715, -0.019549055, 0.022315774, 0.005005577, 0.051111493, -0.017474014, -0.0060886024, -0.0040074955, -0.0062190504, 0.0038163734, -0.00018116776, -0.0044595143, -0.023686998, -0.016466832, 0.001800491, 7.4941454e-05, 0.0035099715, -0.018347716, 0.024900472, -0.015301898, -0.013967077, 0.014501005, 0.043685038, 0.02200027, 0.0033431188, 0.014318985, -0.0130205685, 0.0028698642, -0.014840778, 0.012177205, 0.0015312516, -0.013348206, 0.012401697, -0.014428197, -0.014185502, -0.07067268, 0.0211023, 0.04135517, -0.022861836, 0.02018006, 0.0055031013, 0.022934645, -0.01430685, 0.0052604065, 0.012535179, -0.022558467, 0.012110463, -0.007875442, -0.014343254, 0.0036859252, 0.015059203, -0.0071291556, 0.0062372526, -0.01674593, 0.0074021867, 0.027521573, -0.004168281, 0.003139862, 0.05412091, -0.05844087, -0.019451976, 0.010848451, -0.0028289095, -0.008991837, -0.022315774, 0.017012896, 0.0073718503, 0.015277629, 0.029390322, -0.024657777, 0.011085078, 0.0012369843, -0.029414592, 0.051111493, 0.0036616556, 0.010356994, -0.012547313, 0.011606872, 0.019925231, -0.0033552535, 0.022206562, -0.019257821, 0.005421192, -0.003886148, -0.056499317, -0.019961635, -0.021017358, 0.0058580423, 0.0020416689, -0.0058368063, 0.045141205, -0.042325947, -0.024815528, 0.02999706, -0.0058004023, 0.0077965655, 0.00999902, 0.049242746, 0.014707296, -0.014318985, -0.026405178, -0.026720682, 0.013942808, -0.0024284634, -0.020265004, -0.00930734, -0.0072262334, 0.001512291, -0.0024588, -0.011006203, -0.04218033, 0.029244706, 0.026016867, -0.006173545, 0.0069774715, -0.009216329, 0.023541382, -0.019027261, 0.008761277, 0.010617891, 0.003801205, 0.060139734, -0.045602325, 0.008749142, 0.015059203, -0.017061435, 0.03485095, -0.011254964, 0.025070358, 0.002844078, 0.027230341, -0.009786662, -0.02764292, -0.02905055, -0.0017231321, 0.024682047, 0.014270445, -0.044752896, -0.019015126, -0.006443543, -0.012104396, 0.019172877, -0.04914567, 0.047543883, 0.007305109, 0.0076388144, 0.005132992, -0.021162974, 0.03232693, 0.028832125, 0.012790008, 0.005005577, -0.034365565, 0.008372965, -0.022534199, -0.043029763, 0.022631276, 0.025094626, 0.013518092, -0.047471073, -0.045165475, -0.010174973, 0.011042607, -0.0066498336, 0.020447025, -0.0015244258, -0.02358992, -0.01204979, 0.012559448, 0.004623333, -0.032448277, 0.010193176, 0.039243724, 0.024572834, -0.026186753, -0.053295746, 0.0013719832, 0.03043391, 0.013190455, 0.014039886, 0.029317515, 0.0286137, -0.014561679, 0.010799912, -0.020325677, -0.011885971, 0.0013530226, -0.030458178, 0.004171314, 0.017825922, -0.029171897, -0.019124338, -0.01246237, -0.016321216, -0.0057215267, -0.025167435, -0.017983673, 0.005427259, -0.0001678954, 0.043879192, -0.008664199, 0.031501766, 0.02994852, 0.007566006, -0.007189829, -0.016927952, -0.026259562, -0.023638459, -0.036913857, 0.032472543, -0.010253849, -0.023917558, -0.021284321, 0.00025274372, -0.03730217, -0.017522555, -0.049218476, -0.027230341, 0.0041258093, -0.0053483834, -0.005045015, -0.0041895164, 0.004383672, -0.029147629, 0.027157532, -0.021599824, 0.03089503, -0.021672633, -0.009149589, 0.02448789, -0.018990858, -0.049218476, 0.0130205685, -0.017838057, 0.015423245, -0.037326436, -0.00748713, 0.037035204, 0.021393534, 0.008263753, 7.863875e-05, -0.008203079, -0.00999902, 0.012425967, -0.023407899, 0.019573323, 0.011315638, 0.028686509, -0.021442072, 0.024851933, -0.024754854, 0.0182021, -0.02900201, -2.2823726e-05, 0.019427707, 0.019779615, -0.0031095252, -0.042423025, -0.034923762, -0.018893778, 0.0003939996, -0.014100559, -0.0012756637, -0.04497132, -0.030628065, 0.012632256, -0.004650636, -0.02587125, 0.015326167, 0.011916308, 0.010520813, -0.029778633, -0.02548294, 0.020835336, -0.022861836, -0.025798442, -0.01191024, 0.007996789, -0.011764623, 0.057373017, -0.0009229981, -0.0076934206, 0.026162483, -0.00251644, -0.004702209, 0.005430293, -0.014852913, 0.0008130271, 0.01248664, -0.019415572, 0.00056312745, -0.008160608, 0.011631141, -0.03189008, 0.015568862, 0.006922865, -0.02997279, -0.038418565, 0.01216507, 0.008979702, -0.010290253, -0.04778658, -0.014962126, 0.028006963, -0.0037556998, -0.01112755, 0.025604285, 0.035384882, 0.031040646, 0.02292251, 0.013918539, 0.008306225, -0.0053180465, -0.013226858, 0.003130761, 0.019136474, -0.011685748, -0.009865537, 0.003810306, -0.0135909, -0.019816019, -0.020811066, 0.013336071, 0.022048809, 0.023517111, 0.004705243, 0.014755835, 0.027133262, 0.015678074, -0.00817881, 0.020143656, 0.017850192, 0.05082026, -0.011570468, -0.0041864826, 0.05392675, 0.041452248, -0.0053392826, 0.0036222178, 0.027788538, 0.016503237, 0.010344859, 0.021466343, -0.026696412, -0.031987157, -0.016491102, -0.030385371, -0.0077601615, 0.008530717, 0.019816019, -0.022740489, -0.0122682145, -0.018214233, -1.7976943e-05, 0.018517602, 0.017813787, -0.027375957, -0.01933063, 0.0049600718, 0.012189339, 0.011879903, 0.0075478037, -0.032496814, -0.024427217, -0.038345754, -0.0058459076, -0.01584796, -0.017231321, 0.00070950267, 0.017983673, 0.012620122, -0.043782115, 0.00011196187, 0.009810931, -0.033855904, 0.022582738, 0.019937366, 0.02727888, 0.003112559, 0.0023298687, -0.018748162, -0.013882134, 0.0012316754, 0.00089569495, 0.028395275, -0.0034826682, 0.012705065, 0.019160744, 0.035263535, -0.0023131834, -0.020507699, 0.011327773, 0.005181531], "content_hash": "ea510f95fe6b84bfc72d56660e0d67e8"}, {"content": "on the left to tokens on the right. Thicker lines mean stronger attention. **Navigating BertViz**: Single-click a head square to select/deselect it. Double-click to show only that head. Hover over tokens or lines to see exact attention weights. ## Stage 4: MLP (Feed-Forward) **Icon**: Network | **Summary shows**: \"X layers\"\n\nThis stage shows the expand-then-compress pattern of the feed-forward network:\n- Input dimension (e.g., 768) \u2192 Expanded dimension (e.g., 3072, which is 4x larger) \u2192 Back to input dimension\n- An explanation of why this expansion matters for storing factual knowledge\n- The total number of layers in the model\n\n## Stage 5: Output Selection\n\n**Icon**: Bullseye | **Summary shows**: \"\u2192 [predicted token]\"\n\nThis stage reveals the model's prediction:\n- Your full prompt with the **predicted next token** highlighted\n- A **confidence percentage** for the top prediction\n- A **top-5 bar chart** showing the five most likely next tokens and their probabilities\n- A note about how beam search and other techniques can influence the final output", "source_file": "pipeline_stages.md", "chunk_index": 1, "embedding": [0.03332839, 0.013307699, -0.00044659345, -0.024663603, -0.012553596, -0.00910838, 0.0111341085, 0.047552846, -0.029661383, 0.06736653, 0.025624715, 0.0034452155, -0.03329882, -0.03536891, 0.0016246482, -0.0106018, -0.004964511, 0.0138474, 0.00047593508, 0.0392725, 0.0029184523, -0.023569414, 0.061629433, 0.02918822, 0.03628566, 0.04225934, 0.008420817, 0.024293944, 0.04551233, 0.01445364, -0.019133514, -0.03131745, -0.005522695, -0.06293063, -0.007696286, -0.02809403, -0.016309325, 0.035014033, -0.029055143, 0.036137797, -0.013492528, -0.0056483787, -0.05260977, 0.022149926, 0.012664493, 0.040662415, -0.0035986237, -0.023451123, -0.03504361, 0.063758664, -0.030134546, 0.046931818, -0.015821375, 0.0056816484, -0.013581246, 0.004343485, -0.0063174604, -0.019902403, 0.007252696, -0.021987276, 0.024737533, -0.00664276, 0.038651474, 0.028522834, 0.015170777, -0.013529494, -0.09114295, 0.018645566, 0.015747445, 0.0077997902, 0.057016097, -0.0060180374, -0.004398934, 0.027768733, 0.03108087, 0.010047313, 0.018438557, 0.021795053, -0.0033269248, -0.04427028, 0.030578135, -0.024974115, -0.010668339, -0.07824927, 0.0032751726, -0.05651336, -0.038030446, -0.045423612, -0.049001906, -0.054975584, -0.013078511, -0.012191331, -0.050450966, 0.007814577, 0.010823595, 0.012901075, 0.0155995805, -0.009500219, -0.003277021, 0.0036429826, 0.032914374, -0.007895902, -0.040721558, -0.04237763, 0.032382067, -0.01613189, 0.022490012, 0.024249585, -0.024929756, -0.024086935, -0.08889543, -0.006206563, -0.011148894, 0.056424644, 0.021735908, -0.0148011185, 0.00750776, -0.0059182295, 0.046103783, 0.011060176, -0.017906249, -0.017906249, 0.001033195, 0.02475232, 8.1324826e-05, 0.014283597, 0.032145485, -0.0098624835, 0.007463401, -0.025299415, -0.016767701, -0.0021588046, 0.036995403, -0.030060614, -0.021307105, -0.0152003495, -0.028330613, 0.022652661, -0.04296908, 0.044447716, -0.008820048, -0.0031051298, 0.0039072884, -0.019488387, -0.049770795, 0.02164719, -0.04607421, 0.0553896, -0.053556096, 0.015998812, -0.005977375, 0.0016477519, -0.019562317, -0.03430429, -0.054295413, -0.044151988, -0.01239834, 0.022682235, -0.0128493225, -0.021676764, 0.015052486, 0.0048905793, 0.004088421, -0.0044654724, -0.020893088, 0.06103798, -0.03510275, 0.021410609, -0.03983438, -0.06364037, 0.00039045158, 0.020597361, -0.011126715, -0.031790614, -0.0036928866, -0.00086823496, -0.056661226, -0.051308572, -0.017595736, -0.042141046, -0.010941886, 0.0051715197, -0.016516333, -0.021765482, -0.031908903, 0.010017741, -0.013093297, 0.03785301, -0.021987276, -0.01398787, -0.004095814, 0.03596036, 0.063758664, -0.009256244, -0.011902997, -0.023702491, 0.030785143, -0.0035912304, 0.020656506, -0.00039645855, 0.018601205, 0.0658879, 0.051900025, -0.024500953, 0.03850361, 0.013551674, -0.013825221, -0.01076445, -0.012790177, 0.0021310803, 0.018630778, -0.02757651, 0.045541905, 0.007322931, -0.06985063, 0.024293944, -0.069732346, 0.050894555, 0.009086201, 0.029410016, 0.030607708, 0.013440776, 0.035782926, -0.00024466758, -0.04959336, 0.038681045, 0.0471684, -0.035871644, -0.0070715635, -0.006579918, 0.013913939, 0.030430272, 0.011259792, 0.0048610065, -0.0430578, -0.0440337, -0.014645862, -0.05680909, 0.031672325, -0.06364037, -0.0019037403, 0.024441807, 0.028877707, 0.015377786, 0.0118142795, 0.023451123, -0.01580659, -0.017994966, 0.039390787, 0.02790181, 0.051870454, 0.0063359435, -0.029927537, -0.0011062025, 0.0035283884, -0.012339194, -0.029498734, 0.034895744, 0.0006492124, -0.00014670815, -0.0032160273, -0.011392869, -0.025816936, -0.031228734, -0.03773472, -0.019724967, -0.020390352, -0.0020368174, -0.059174903, -0.047079682, -0.0041956217, 0.007271179, 0.022061208, 0.017285222, -0.052905496, 0.010128638, -0.013795648, 0.044980023, -0.019961549, 0.0048018615, -0.044211134, -0.019089155, 0.058879174, 0.026304886, -0.010786629, -0.017240863, -0.009988167, -0.0070013283, 0.025506424, -0.0011607271, 0.03672925, -0.029705742, 0.033564974, -0.0023251506, -0.018128043, -0.0012050861, 0.05269849, 0.012797571, 0.0010618435, -0.007511457, 0.04341267, -0.045778483, -0.00096480816, -0.007430132, -0.0015738202, 0.028404545, 0.033890273, 0.04264378, 0.0035672027, -0.01207304, -0.019621463, -0.018808214, 0.012583169, -0.03297352, -0.033801556, -0.009389321, 0.049652506, -0.004679874, 0.024767106, -0.002339937, 0.009810732, -0.02096702, -0.014098768, 0.016797274, 0.026216168, -0.025979586, -0.03462959, 0.033505827, 0.024486165, -0.022164712, 0.022223858, 0.0054894257, -0.030104972, 0.038769763, -0.030785143, -0.045896776, -0.024974115, 0.02342155, -0.009906842, 0.051515583, -0.03820788, -0.036137797, -0.024707962, -0.018689925, -0.037438992, 0.029927537, 0.015126418, -0.0043065194, 0.01475676, 0.020212917, -0.024367876, 0.0010960369, 0.02967617, -0.033771984, 0.01765488, -0.007285965, 0.001109899, -0.019562317, 0.00051752164, 0.03232292, -0.028892493, 0.015229923, 0.03962737, 0.028522834, -0.014010049, -0.023436336, 0.0014204121, -0.002003548, -0.0059551955, -0.03442258, -0.0066390634, -0.016057957, 0.012538809, -0.010838382, -0.03995267, -0.01926659, -0.005337866, -0.0146015035, -0.048676606, 0.026023945, -0.017019069, -0.04959336, -0.028522834, 0.009648082, 0.021247959, 0.006195473, 0.017640095, -0.008642611, -0.054413702, 0.0066390634, 0.017462658, -0.02479668, 0.00020689314, -0.011784707, 0.035575915, 0.014505392, -0.028182749, 0.038888052, 0.04196361, 0.069968924, 0.039242927, -0.032027196, -0.06914089, -0.016279751, -0.0018556847, 0.018349838, 0.027753945, 0.01048351, 0.02204642, 0.0130489385, 0.036137797, -0.023865141, 0.012309622, -0.012509237, -0.014963768, 0.055212166, 0.031731468, 0.01310069, -0.0012143275, 0.07369508, -0.01342599, -0.033831127, -0.042909935, -0.020937447, -0.023569414, -0.014815905, 0.05935234, -0.033505827, 0.028670698, -0.0130489385, 0.0014629228, 0.058435585, 0.0073266276, 0.018320266, 0.009810732, -0.05580362, 0.013174622, 0.037616428, 0.0070272046, -0.027310356, 0.0030478328, -0.02806446, -0.023805995, -0.03596036, -0.03643352, 0.017270437, -0.056749944, -0.06026909, 0.010002954, -0.017876675, -0.020257276, 0.04160874, -0.0030552258, 0.09587458, -0.02895164, -0.05024396, 0.09132039, -0.0016773245, 0.047257118, -0.0007753583, 0.043235235, 0.004310216, -0.020626934, 0.024767106, -0.006853465, 0.0104169715, 0.023229329, -0.01221351, 0.0708561, 0.0116959885, -0.0035505681, -0.022830097, 0.012169152, -0.021247959, 0.017728813, -0.004812951, 0.023081465, -0.014838085, 0.020272061, -0.01713736, 0.029897964, 0.015067273, -0.0028445206, -0.010734878, 0.0052084858, 0.08759423, -0.0039701303, -0.013699537, 0.050450966, -0.0012032378, -0.010224748, -0.023554627, 0.030489417, -0.019325737, 0.033003096, -0.06387696, -0.032234203, -0.031908903, 0.008945731, -0.03507318, 0.046636093, -0.014875051, -0.010076885, 0.045660194, 0.023643345, 0.0019832167, 0.03539848, 0.06488243, 0.001673628, 0.054768573, -0.05269849, -0.020937447, -0.021233173, 0.018985651, -0.029395228, 0.01564394, 0.032677796, 0.017196504, -0.006250922, 0.039656945, 0.006605794, -0.00012707004, -0.0020368174, 0.012058254, -0.0043323953, -0.0079106875, -0.016235393, -0.018512487, -0.046103783, -0.024264371, 0.0070087216, -0.051900025, 0.004690964, 0.0022789435, -0.05107199, 0.012435306, -0.022756165, -0.0036318928, -0.02031642, 0.011311544, 0.0074042557, 0.017847104, -0.00154702, 0.02092266, -0.00073931663, 0.0060586995, 0.0021772874, 0.015777016, 0.0155995805, 0.018335052, 0.006021734, -0.017566163, 0.015510863, -0.01821676, -0.023111038, -0.013943511, -0.03584207, -0.02895164, 0.0074042557, -0.03273694, 0.015658725, -0.009137954, -0.006213956, -0.019399667, -0.032559503, 0.019000437, 0.021957703, -0.0075853886, 0.0074412217, -0.020020694, -0.029040357, -0.023155397, 0.052994214, 0.027058989, -0.0061141485, -0.01821676, 0.010254322, 0.02608309, -0.0038407499, 0.022652661, -0.031140016, -0.036463097, -0.0031679717, -0.02407215, -0.013803042, -0.03108087, -0.010609194, -0.0065910076, -0.02584651, 0.006365516, 0.022667447, -0.021661976, -0.01664941, 0.027561724, -0.031701896, -0.010202569, 0.0050125667, -0.012701459, -0.022445653, -0.0010683126, 0.001946251, -0.020065052, -0.018970864, 0.017625308, -0.0420819, 0.037646003, -0.04092857, -0.0070863497, 0.015303854, 0.017965393, -0.04882447, -0.045305323, -0.019281378, -0.01923702, -0.026881551, -0.013995264, -0.041549593, -0.0043139122, 0.013019365, -0.0132485535, -0.0033102902, 0.009507611, 0.001470316, 0.013743896, 0.00034008565, 0.030164117, 0.002522918, -0.009692441, -0.011474194, -0.013507315, -0.033890273, 0.0016052412, 0.028522834, 0.012760605, -0.027147707, -0.0028796382, 0.021011379, 0.008376457, -0.0049792975, -0.01713736, 0.014387101, 0.0076593203, -0.027709587, -0.023510268, -0.015274282, 0.0026097877, -0.004103207, -0.005929319, -0.013174622, -0.028020099, -0.03294395, -0.036877114, 0.03023805, -0.014424067, -0.026186595, -0.04373797, 0.005818422, 0.010261714, -0.018926505, 0.014941589, -0.013026759, 0.009204492, -0.017344369, -0.04639951, -0.017285222, 0.042229764, 0.014401888, 0.034481727, 0.018231548, -0.016885992, -0.08593816, 0.02552121, -0.01753659, 0.020360779, 0.016945137, -0.036019504, 0.00019557234, -0.03915421, -0.026275313, 0.041786175, 0.0029979288, 0.002643057, -0.058760885, -0.0018981954, 0.010002954, -0.008997483, 0.031376597, -0.025255056, 0.013226374, 0.0074523115, -0.005762973, -0.0045098313, 0.022549156, -0.015880521, -0.021602832, 0.0123318015, 0.0138843665, 0.06529644, -0.009803338, -0.048883613, -0.00032298896, 0.026585825, -0.029587451, -0.032145485, -0.01963625, -0.028685484, -0.032589078, 0.046576947, 0.008043765, -0.01032086, -0.04261421, 0.0020405138, -0.0133668445, -0.011008425, -0.016338898, 0.029720528, -0.011673809, -0.007330324, 0.0018288845, -0.014461033, -0.032796085, 0.03326925, -0.035014033, 0.0007961516, 0.013832614, -0.010357826, -0.024175653, -0.028552407, -0.0075484226, 0.020907873, 0.0215289, 0.035250615, 0.00080908963, -0.033919845, -0.015688298, 0.035250615, -0.029735314, -0.006901521, -0.041194722, -0.040337116, -0.015703086, -0.009004876, -0.0118142795, 0.01942924, 0.006764747, 0.022460438, -0.026334457, 0.037764292, -0.035487197, -0.034659162, -0.007792397, 0.014490605, -0.017107787, 0.012583169, 0.0065725246, 0.01398787, 0.018335052, -0.043442246, 0.033210102, -0.004044062, 0.0044285064, -0.009234064, -0.01802454, 0.021661976, 0.015022914, -0.0142022725, 0.018985651, -0.0048166476, 0.024929756, 0.004376754, 0.0027724374, -0.028389757, 0.02479668, -0.021070523, 0.06571046, -0.020375567, 0.03528019, -0.020893088, -0.016664196, 0.0035838373, -0.0146015035, -0.020893088, -0.003759425, -0.022223858, 0.010402185, -0.0039331643, -0.010076885, -0.010202569, 0.0031143713, -0.029853605, -0.032145485, -0.0154812895, 0.0059367125, -0.078426704, 0.0039553437, 0.018320266, 0.010136031, -0.113085866, -0.045453187, 0.033505827, -0.019296164, -0.042022757, -0.019414455, 0.040751133, -0.003125461, -0.020227702, 0.002541401, 0.0038407499, -0.012923255, -0.023879927, -3.9420593e-05, 0.0003608789, -0.041490447, 0.024116509, 0.0027872236, -0.0040366687, -0.03184976, -0.038030446, 0.0056816484, -0.0016828694, 0.024648815, -0.023835568, 0.032027196, -0.048676606, -0.004502438, 0.011052784, 0.009707227, -0.0134851355, 0.024338303, 0.017921034, 0.046103783, 0.043205664, -0.011141501, -0.01576223, 0.009004876, 0.022667447, 0.009219279, 0.003785301, -0.036817968, 0.0122504765, -0.04897233, 0.036788397, -0.007999406, -0.028212322, -0.043116946, 0.019251805, 0.019813685, -0.02104095, -0.012339194, -0.008738723, -0.01108975, 0.011599878, 0.0300754, -0.004794468, -0.016723342, 0.0006492124, 0.021277532, -0.022401294, 0.030933008, 0.033771984, -0.022297788, 0.025920441, -0.017832316, 0.01120804, 0.02463403, -0.014948982, -0.010409578, 0.018349838, -0.005371135, 0.00031790615, -0.023465909, 0.02180984, 0.014505392, 0.011703382, 0.004775985, -0.0027927684, -0.010794022, 0.05172259, -0.021795053, -0.017566163, 0.01447582, 0.012553596, 0.020774797, 0.006091969, -0.028655913, 0.041431304, 0.02309625, 0.013078511, -0.0020516035, -0.005689041, 0.013078511, -0.022918815, -0.014616289, 0.035546344, 0.02657104, 0.048351306, -0.009722013, 0.07002807, 0.026541466, -0.028448904, -0.054265838, -0.038769763, -0.006827589, 0.0013963842, 0.0026097877, 0.015333426, 0.00030889572, 0.014438854, -0.01328552, 0.01959189, 0.010956672, -0.034156427, 0.03406771, 0.03616737, 0.028212322, 0.00013769772, -0.021928132, 0.0028075548, 0.0142022725, -0.022963174, 0.019458814, -0.014778939, -0.008871799, 0.009936416, 0.0046392116, -0.0029591147, 0.0064653237, -0.021868985, 0.013662571, -0.01440928, 0.0013270733, -0.013204195, 0.017477445, 0.0017503321, 0.027058989, -0.03882891, 0.011341117, -0.0054487633, 0.015673513, 0.01907437, 0.03241164, -0.034925316, 0.017447872, -0.015111632, -0.020538216, 0.0021366251, 0.02302232, 0.052195754, 0.007570602, 0.017507017, -0.019887617, 0.0020201826, 0.03720241, 0.020582575, -0.024929756, 0.0032215721, 0.05760755, 0.0009527943, 0.051486008, 0.049889084, -0.023820782, 0.008391243, -0.008716543, -0.005611413, 0.008642611, -0.026304886, 0.039568227, -0.041786175, -0.007537333, 0.032796085, 0.031228734, 0.021617617, 0.03332839, -0.0060771825, 0.04483216, 0.0011163681, 0.03874019, -0.0027188368, 0.032382067, -0.018113257, 0.005175216, 0.043146517, -0.0102986805, 0.0016958074, 0.028079245, -0.0057851523, -0.0075336364, -3.7514543e-05, 0.019281378, -0.019769326, 0.022889242, -0.0235842, -0.006472717, 0.010343039, 0.0046946607, -0.007016115, 0.020789584, -0.04639951, 0.012243083, 0.028345399, 0.0042880364, -0.0032419034, 0.00796244, -0.0023251506, -0.0006690815, 0.020168558, 0.023968644, -0.021307105, 0.0054672463, -0.018408984, 0.01891172, -0.037438992, 0.016605051, 0.019931976, -0.0042436775, 0.026437962, -0.008450389, -0.034156427, -0.04882447, -0.012176544, 0.014564537, 0.0038296601, -0.0055189985, 0.037468567, -7.964981e-05, -0.01625018, 0.0021144454, -0.006679726, 0.013499921, -0.006605794, 0.017226078, 0.009892057, 0.011725562, 0.015658725, -0.0048351306, 0.018113257, 0.0039220746, 0.046369936, 0.025299415, -0.00074902014, 0.0072748754, -0.016442401, -0.014335349, -0.006897824, 0.0054044044, -0.025535995, -0.009951201, -0.013300306, 0.017994966, 0.019059582, -0.00971462, -0.018763855, -0.014564537, 0.036315233, 0.013011972, -0.0027447129, 0.023406764, -0.0024064756, 0.03226378, 0.032382067, -0.057341397, 0.01991719, -0.0025136764, -0.020405138, -0.0063950885, 0.010550048, 0.0056335926, -0.000102291386, 0.048440024, -0.013396417, 0.025935227, 0.0029332386, 0.031879332, 0.0027151403, -0.02725121, 0.024707962, -0.01713736, 0.013581246, -0.018275907, 0.0067684436, 0.0049127587, 0.0083468845, 0.038681045, -0.01870471, -0.032382067, 0.011104535, -0.027546937, 0.015998812, 0.018231548, 0.019710181, 0.009211885, -0.006960666, 0.006561435, -0.010209963, 0.0017780565, 0.011718168, -0.03528019, 0.024678389, -0.0020774796, 0.020286849, -0.01015821, -0.013315092, 0.015895307, 0.036315233, -0.034186002, 0.006088272, -0.01106757, -0.0033343178, 0.0049090623, 0.007999406, 0.01713736, -0.020686079, -0.016634624, 0.030001469, -0.0519296, 0.0052972035, -0.022490012, 0.002885183, 0.009248851, -0.0077628247, 0.021395823, -0.014771546, -0.017107787, 0.010875347, -0.0082285935, 0.008605645, 0.0038148738, 0.018926505, -0.004905366, -0.004838827, 0.02314061, -0.025018474, -0.009943808, -0.038976774, -0.005400708, -0.0007263786, -0.014520179, 0.029439587, 0.03610822, -0.0044913483, -0.011126715, 0.021395823, 0.0007448615, 0.012206118, -0.0038851087, 0.002606091, -0.007792397, 0.07606089, -0.03362412, -0.015436931, -0.008583466, -0.0056446823, 0.013470349, -0.014978555, 0.0135664595, -0.026600612, 0.008738723, -0.0005475564, -0.0045246175, -0.0034858778, -0.02790181, 0.03471831, -0.022075994, -0.006535559, 0.004247374, 0.03962737, 0.007481884, 0.013056331, 0.030903434, -0.005275024, 0.009766373, -0.002308516, 0.025003688, 0.004761199, 0.014217058, 0.017684454, 0.027502578, -0.004410024, -0.04418156, -0.004746413, 0.051840883, -0.01580659, -0.001293804, -0.01239834, 0.033712838, -0.018039325, 0.013588639, -0.0053341696, -0.0073709865, 0.019118728, 0.001521144, -0.005352652, -0.0005244527, 0.012413126, 0.002556187, -0.009788552, -0.008879192, 0.007688893, 0.011267185, 0.00826556, -0.0034304291, 0.02092266, -0.052580196, -0.01842377, 0.007977226, -0.013573853, -0.0041771387, -0.019503172, 0.017773172, 0.008332098, 0.0009527943, 0.018660352, -0.04926806, 0.0067019053, -0.006487503, -0.01137069, 0.053999685, -0.01926659, -0.011659023, 0.009655475, -0.0087461155, 0.021336678, 0.00826556, 0.01613189, 0.0058886567, 0.015089452, -0.0027724374, -0.050539684, -0.005415494, -0.038237456, 0.0011062025, -0.010557441, 0.012760605, 0.05790328, -0.017403513, -0.026511895, 0.034807026, -0.024042577, 0.003251145, 0.022830097, 0.03596036, -0.0018335052, -0.01903001, -0.013270733, -0.019606676, -0.024131294, -0.02946916, 0.000919063, -0.002155108, 0.009943808, 0.003302897, -0.0104169715, -0.014187486, -0.06293063, -0.009522398, 0.037113693, -0.011851246, 0.0054894257, -0.005001477, 0.032027196, -0.018689925, 0.0038333565, 0.02257873, -0.017743599, 0.041106004, -0.030814717, -0.017610522, 0.011193253, -0.013854793, 0.016900778, -0.023717277, 0.017166931, 0.023776423, 0.0008844075, 0.007175068, -0.034984462, -0.014135733, -0.00022583813, 0.031761043, 0.010986245, -0.0032289652, -0.0037113694, 0.010727484, -0.0060771825, 0.004265857, -0.045127887, 0.04713883, -0.00012522175, 0.014904623, -0.00650229, -0.022918815, 0.030430272, 0.0069754524, 0.03539848, 0.007082653, -0.026068304, 0.010091672, 0.003302897, -0.045098312, 0.020493856, 0.008790474, 0.026334457, -0.042850792, -0.01074227, -0.0019702788, -0.005685345, -0.009285817, 0.025595142, -0.0010858712, -0.025240269, -0.012812357, 0.038681045, 0.010712698, -0.016398042, 0.0025746701, 0.032796085, 0.021942917, -0.015954452, -0.037024975, -0.008701757, 0.01267928, 0.0056003234, -0.008694364, 0.006010644, 0.04184532, -0.01802454, 0.017921034, -0.0047390196, -0.001673628, -0.008857013, -0.016605051, 0.005977375, 0.00546355, 0.00033107522, -0.017462658, 0.0037760597, -0.0016671589, 0.01737394, -0.032086343, -0.0095815435, -0.0046503013, 0.011082356, 0.055744473, -0.01963625, 0.0465178, 0.041697457, -0.0019111334, -0.01580659, -0.011104535, -0.011984322, -0.028522834, -0.017255649, 0.027280783, -0.032056767, 0.0048647034, 0.007999406, 0.013758683, -0.027635654, -0.007563209, -0.022061208, -0.005426584, 0.009381928, 0.011562912, -0.0041771387, 0.00559293, -0.015407358, -0.028020099, 0.0033306214, -0.010808809, 0.030119758, -0.001489723, -0.019340523, 0.011880818, -0.0019795203, -0.023998218, 0.03584207, -0.024841039, 0.005640986, -0.04060327, -0.04063284, 0.022460438, 0.007200944, 0.017625308, -0.019340523, 0.013307699, 0.015141204, 0.013840007, -0.013011972, 0.024826253, -0.003219724, 0.018778643, -0.0035727476, 0.045127887, -0.018615993, 0.023362406, -0.046015065, 0.014350136, -0.0032215721, 0.008975304, -0.039656945, -0.035782926, -0.011614664, -0.02846369, 0.012686674, 0.018113257, 0.0021828322, -0.033210102, -0.03640395, 0.005123464, 0.010128638, -0.008435602, 0.00910838, 0.016678983, 0.009226671, 0.00012071654, 0.00071020605, 0.042525493, -0.030962579, -0.003996006, -0.0040292754, 0.0039516473, 0.019562317, 0.026186595, -0.0044543827, -0.0023159094, 0.012243083, -0.016028384, 0.012694066, -0.02064172, -0.002079328, -0.015089452, 0.022223858, 0.016723342, 0.0032825659, 0.00075179257, 0.0028426724, -0.021159241, 0.015185563, 0.009847697, 0.0006284191, -0.034008563, 0.014010049, -0.0070678666, 0.004838827, -0.0465178, -0.015540435, 0.023613773, -0.009655475, -0.004228891, 0.019281378, 0.027221637, 0.013218981, 0.033919845, 0.006979149, 0.02201685, 0.012634921, -0.00971462, 0.026127448, 0.005818422, -0.019118728, -0.008509534, 0.008583466, -0.012346587, 0.00010529486, -0.009566757, -0.0108975265, 0.019872831, -0.0062546185, 0.011644237, 0.0065207724, 0.009381928, 0.02519591, -0.011030604, -0.0054894257, 0.0062694047, 0.04152002, -0.0149194095, -0.0054487633, 0.036551815, 0.04015968, 0.023406764, 0.012686674, 0.010845775, -0.0019869134, 0.008672184, 0.020952232, -0.02330326, -0.044861734, 0.0017512562, -0.023850353, -0.01576223, -0.00010696987, 0.036670104, -0.041549593, -0.028522834, -0.011082356, -0.0046872674, 0.01431317, 0.026911125, -0.035546344, -0.036877114, 0.010520475, 0.005249148, 0.019828472, 0.018970864, -0.018349838, -0.03575335, -0.031465314, -0.02197249, 0.0019758237, -0.003777908, -0.0036060167, 0.0022715502, -0.026437962, -0.04184532, 0.0026227257, 0.004502438, -0.0114668, 0.009049235, 0.038888052, 0.009226671, -0.0076593203, 0.00927103, -0.03374241, -0.009847697, -0.004280643, -0.016900778, 0.034126855, -0.0075853886, -0.0015904548, -0.007696286, 0.022992747, -0.009367142, -0.005079105, 0.021114882, -0.0041290834], "content_hash": "8cae2b97916cca7b5d8bc20345194bf8"}, {"content": "# Pythia Overview\n\n## What Is Pythia?\n\nPythia is a suite of language models created by EleutherAI specifically to support interpretability research. Released in 2023, the Pythia models were trained on the deduplicated Pile dataset with full public training data and checkpoints, making them uniquely transparent.\n\n## Architecture Details\n\n| Property | Pythia-160M | Pythia-410M |\n|----------|-------------|-------------|\n| Parameters | 85M (actual) | 302M (actual) |\n| Layers | 12 | 24 |\n| Attention Heads | 12 per layer | 16 per layer |\n| Hidden Dimension | 768 | 1024 |\n| Vocabulary Size | 50,304 | 50,304 |\n| Positional Encoding | Rotary (RoPE) |  Rotary (RoPE) |\n| Normalization | LayerNorm | LayerNorm |\n| Activation Function | GELU | GELU |\n| Parallel Attn+MLP | Yes | Yes |\n\n## Key Architectural Features\n\n### Rotary Position Embeddings (RoPE)\nUnlike GPT-2's learned absolute positions, Pythia uses **Rotary Position Embeddings**. RoPE encodes position by rotating the query and key vectors, which means:\n- Position information is baked into the attention computation\n- The model can potentially generalize to longer sequences\n- Attention patterns may look different from GPT-2\n\n### Parallel Attention + MLP\nPythia computes attention and MLP in **parallel** within each layer (rather than sequentially). This is a design choice that slightly changes how information flows through the network.\n\n### GPT-NeoX Architecture\nPythia uses the GPT-NeoX architecture (`GPTNeoXForCausalLM`), which is EleutherAI's evolution of GPT-Neo. The module paths use `gpt_neox.layers.{N}.attention` instead of GPT-2's `transformer.h.{N}.attn`.\n\n## Why Pythia Matters for Interpretability", "source_file": "pythia_overview.md", "chunk_index": 0, "embedding": [0.007605917, 0.0042840163, 0.017825149, 0.0017015801, 0.0037347001, -0.00084185146, -0.015003809, 0.04347724, -0.017903158, 0.0035721806, -0.01169491, -0.060119245, -0.03029365, -0.023155792, -0.016472986, -0.03775655, 0.008633041, 0.0030261148, -0.013638644, 0.028681457, 0.032165878, -0.02231069, -0.022934765, -0.0068648276, 0.041422993, -0.012514008, -0.00068908307, 0.034636173, -0.0068648276, 0.0006322012, 0.009491144, -0.02862945, -0.033232007, -0.028993495, -0.02024344, -0.024884999, -0.002936729, 0.020295447, -0.0036696924, 0.08347006, 0.0005176249, -0.035962336, -0.048885893, 0.07582514, -0.038614653, 0.041969057, 0.045323465, 0.052578337, 0.0131250825, 0.026809232, 0.010056713, 0.046649624, -0.033388026, 0.0020298697, -0.019762382, -0.03011163, 0.0147177745, 0.03559829, -0.06427975, -0.0240529, -0.0030569935, -0.053046394, 0.032477915, 0.022284687, -0.021595603, 0.009816183, -0.09397533, 0.033908088, -0.01974938, 0.04901591, 0.020542476, 0.020399459, -0.017539114, 0.00696884, 0.01352163, -0.0431392, -0.0023565341, 0.0481058, 0.004960098, 0.010180227, 0.004297018, 0.0039427252, 0.007267876, -0.020425461, 0.031879842, 2.0695854e-05, 0.0062212497, -0.003069995, -0.048261818, -0.008138981, -0.022661733, -0.005207128, -0.04927594, 0.014535752, 0.013872672, 0.0044270335, 0.020620486, -0.039550766, -0.00888007, 0.028005376, 0.012494506, 0.0046188068, -0.06095135, 0.0039297235, 0.044335343, -0.0024784238, 0.015770901, 0.023766864, 0.0042937677, -0.020360455, -0.09407934, 0.0055646705, 0.007436896, -0.022206677, -0.016785024, -0.042593133, -0.008945079, 0.018904278, -0.0031171257, -0.017045055, -0.030605689, 0.010966822, 0.026354177, 0.02058148, 0.020542476, -0.028343417, 0.0061172373, -0.02638018, -0.026536198, 0.011389373, -0.004917843, -0.0009092971, 0.017045055, -0.018527234, -0.02547007, 0.0075539104, -0.013664647, 0.0049958522, -0.018098181, -0.019983409, -0.029071504, 0.0054054013, 0.0033170248, 0.037886567, -0.05663483, 0.024442947, 0.0033414029, -0.0025271797, -0.023025775, 0.008321003, 0.016563997, -0.007651422, -0.024767986, -0.026731223, -0.03500022, -0.0027319542, 0.03169782, 0.060015235, 0.033492036, -0.036612414, 0.00588971, 0.029929608, -0.030059623, 0.0067348117, -0.015809905, 0.071404606, -0.01973638, 0.0044692885, -0.06339564, -0.075305074, -0.040434875, 0.024208918, -0.01526384, 0.029097507, 0.021205556, -0.011389373, -0.008821564, -0.027771348, -0.031307776, -0.045401473, -0.021517593, 0.028473431, -0.056010753, 0.014327727, -0.055282664, -0.04298318, -0.0071963673, 0.041605014, -0.023532836, 0.033049982, 0.011155345, 0.0671921, 0.05923514, 0.014145705, -0.008268997, -0.023597844, 0.057362914, -0.003177258, -0.027225282, -0.008782559, 0.032503918, 0.045505486, 0.030163636, -0.016030932, 0.06901232, -0.032971974, 0.013651646, -0.02862945, -0.069376364, -0.020269442, -0.009185608, -0.04753373, 0.035754308, 0.003698946, 0.0078009404, -0.005931965, -0.052526332, 0.053046394, 0.039342742, 0.027095266, -0.036612414, 0.026354177, 0.009250615, 0.030007618, -0.060431283, 0.020477468, 0.049379952, -0.022908762, 0.00969917, -0.018709255, 0.0024605466, -0.0048593357, -0.012416497, -0.0049796, 0.02140058, -0.062251505, 0.018800266, 0.009192108, 0.018020172, -0.030969733, -0.0068583265, -0.01227998, -0.011460882, 0.010843307, 0.041448995, -0.010745795, -0.008763056, -0.056894857, 0.013950681, 0.02596413, 0.015107821, -4.2051943e-05, -0.022609726, 0.013047073, -0.035754308, -0.0024442947, -0.035962336, -0.021595603, -0.054554574, -0.003069995, -0.01924232, 0.05127818, 0.01311208, 0.038874686, -0.0083990125, -0.056062758, -0.042203087, 0.0012457126, -0.02977359, -0.046103556, 0.07067652, -0.0038062087, 0.012247477, 0.0323739, -0.03271194, -0.041474998, 0.0045082932, 0.0021257563, -0.012735035, 0.02605514, -0.009094596, -0.008021967, 0.05881909, 0.018059177, -0.0066080466, 0.0116104, 0.00096861675, -0.0224017, -0.014535752, -0.03276395, 0.032165878, -0.023441825, 0.013716653, -0.009738174, 0.018735258, -0.0024702977, 0.049822006, -0.0074694003, 0.026913244, 0.00084916485, 0.012657026, -0.045063432, 0.002207016, -0.0439713, 0.012761038, 0.04935395, -0.010810803, -0.019073298, 0.0053371433, 0.017942162, 0.027329294, -0.014977805, -0.007423895, 0.0024702977, -0.03601434, -0.0053273924, -0.0072743767, -0.0049308445, 0.035208244, -0.032503918, -0.05135619, -0.070624515, -0.010485764, -0.039186724, -0.039446756, 0.016863033, -0.024429945, -0.046077553, -0.0010035585, -0.02015243, 0.054814607, -0.03044967, -0.011187849, -0.026939247, -0.03219188, 0.0041182465, -0.021075541, 0.04139699, -0.012156465, 0.015458863, -0.047689747, -0.031827837, -0.0340121, 0.010050211, -0.0070728525, 0.033908088, -0.028577445, -0.008177985, 0.03552028, 0.033206, -0.018306207, -0.026029138, 0.008021967, 0.020685494, 0.027979372, -0.013443621, -0.03351804, -0.021023534, -0.056062758, 0.0077099293, -0.07806141, -0.014288723, -0.020399459, 0.012266979, -0.0043100193, 0.020568479, -0.014873792, 0.036716424, -0.00085647823, 0.032633934, 0.0024524205, 0.0031155006, -0.0024410442, -0.009400133, -0.028083386, -0.00075530977, -0.038042586, 0.0049568475, -0.038614653, -0.004196256, 0.022440705, 0.02431293, 0.03307599, -0.03674243, -0.015445862, 0.011181348, -0.004371777, 0.0103687495, -0.031931847, 0.06401972, 0.046181567, -0.01476978, -0.028681457, 0.025847116, -0.01518583, 0.005499663, -0.05673884, -0.023155792, 0.059807207, 0.04469939, -0.028005376, -0.0021468839, -0.02058148, -0.027173275, -0.026445188, -0.017799145, 0.01600493, 0.042827163, -0.004248262, -0.012761038, 0.04321721, 0.0108823115, 0.03427213, -0.05075812, -0.000275877, -0.07280878, 0.029591568, 0.011837927, 0.0060944846, 0.043763276, 0.03526025, -0.113893725, -0.07405692, -0.037288494, -0.003542927, 0.031489797, 0.022921763, -0.030787712, -0.001045001, -0.028785469, 0.013742656, 0.03864066, 0.050446082, -0.04862586, 0.021075541, -0.029253526, 0.03598834, -0.008360008, -0.033986095, 0.02795337, 0.052058276, -0.03666442, -0.025912123, -0.005359896, -0.0009946199, -0.024624968, 0.0013204716, -0.03510423, -0.0095301485, -0.028499436, 0.057206895, 0.06334364, -0.00028969118, -0.022349695, 0.05244832, -0.037938572, 0.07837345, 0.0047423216, 0.040538885, -0.051746238, 0.06495583, -0.014054694, -0.030007618, 0.008242994, -0.026289169, -0.0032390154, -0.005018605, -0.018839272, -0.018267201, -0.0348442, -0.03318, -0.06339564, 0.009517147, -0.021270564, 0.0043685264, -0.0066763046, -0.027017256, 0.028447429, 0.012696031, -0.0030342408, -0.0016568872, 0.020711496, -0.004638309, 0.026913244, -0.010219232, 0.026016137, -0.035806317, -0.0140676955, 0.012331987, 0.021296566, -0.018215196, -0.01956736, 0.046311583, -0.014743777, 0.02215467, -0.009458641, -0.02712127, -0.063863695, -0.027303291, 0.00035043288, 0.0057661952, 0.005285137, 0.015055815, 0.055542696, -0.020464467, 0.004664312, -0.013430619, 0.020204436, -0.026315171, 0.03359605, 0.005207128, 0.013294103, -0.0055549196, 0.030319655, 0.029591568, 0.04638959, 0.03684644, 0.015172829, 0.009536649, -0.009231113, 0.0069493377, -0.029981613, -0.03949876, -0.009185608, 0.047351707, -0.028993495, -0.044933416, -0.02530105, -0.02090652, -0.029461551, 0.003351154, 0.0045147943, 0.0037444513, -0.031255767, -0.048495844, 0.008821564, -0.013138084, -0.02389688, -0.054866612, 0.03739251, -0.0018510981, 0.010258237, -0.008496524, -0.010180227, 0.006598295, 0.018982287, 0.04048688, -0.007592915, 0.021283565, 0.030137632, 0.055282664, -0.02232369, 0.017708134, -0.025184035, -0.02331181, -0.021868637, -0.022882758, -0.0014886794, 0.032087866, 0.05161622, 0.0022703987, 0.0048885895, -0.02605514, 0.018150188, -0.015029811, 0.017734138, 0.028473431, -0.027329294, 0.007293879, -0.019515352, -0.013287602, 0.0019144808, 0.018852273, 0.01268953, 0.014327727, -0.018839272, 0.028785469, 0.022115666, 0.010102218, -0.03185384, -0.034194123, -0.027199278, 0.011272359, 0.04495942, -0.054138526, 0.023194795, -0.02414391, -0.016850032, -0.00862654, 0.057570938, 0.014886795, -0.01725308, -0.0014959929, 0.011642903, -0.00045668005, -0.017812148, 0.02372786, -0.011545392, -0.04446536, -0.0023987892, -0.0140676955, 0.015627883, -0.010875811, 0.034532163, 0.0056134267, -0.002471923, -0.010472762, 0.031047743, 0.0068063205, -0.014938801, -0.017903158, 0.005616677, 0.015575877, -0.040772915, -0.045505486, -0.00979018, -0.001849473, 0.0077099293, 0.032321896, -0.0348442, 0.022206677, -0.014236716, 0.0050966144, -0.034480155, 0.021062538, 0.0045993044, -0.02414391, -0.045245454, -0.040590893, 0.035702303, -0.021504592, 0.0072613754, 0.024533957, 0.041032948, 0.004079242, 0.000744746, 0.013573636, -0.040538885, 0.05268235, -0.028681457, -0.007852946, 0.014457743, -0.015601881, -0.007001344, 0.022362696, -0.026328174, -0.0120979585, 0.038354624, 0.0028197148, -0.021660611, 0.00065617287, -0.031385783, 0.03635238, 0.0012928434, 0.020776505, -0.016876034, -0.035806317, -0.030059623, 0.0032341399, -0.001297719, -0.058299027, -0.0046090554, -0.008249494, -0.026432186, -0.0053111403, 0.0026019386, 0.028681457, 0.03291997, 0.025275046, -0.014912797, -0.0068323235, 0.021933643, 0.03666442, 0.0073783896, -0.019346332, -0.017292084, -0.0053989007, -0.040954936, -0.03718448, 0.012754537, 0.009075094, -0.009764177, -0.050654106, -0.018059177, -0.0037964575, -0.033882085, -0.029591568, -0.00068502006, 0.018761262, -0.014977805, 0.029253526, -0.017123064, -0.0070338477, -0.009816183, -0.0022021404, -0.018631246, -0.008860568, 0.017968167, 0.022622727, -0.056114763, -0.04420533, 0.00862654, -0.026458189, 0.018878276, -0.030761708, -0.0023841623, -0.022908762, -0.035936333, -0.003676193, -0.030761708, -0.07525307, -0.011441379, 0.009309122, 0.010459761, 0.01956736, -0.041422993, -0.033570047, 0.012910556, -0.012618021, -0.009894192, -0.005418403, 0.011538891, 0.0042255092, -0.011174847, 0.0099917045, -0.03908271, -0.0062927585, -0.00026287543, -0.021998651, 0.0057629445, 0.007599416, -0.0073133814, -0.0051583718, -0.02969558, 0.011623401, -0.0124750035, -0.019554358, 0.032399904, -0.023259804, -0.010693789, -0.008821564, -0.004440035, -0.00069030194, 0.011545392, 0.02953956, -0.004592804, -0.0024361687, 0.040694904, 0.05881909, -0.013008068, -0.02115355, 0.0099592, 0.0022996522, -0.005701187, -0.008906074, -0.044673383, -0.0046025547, -0.0063090106, -0.029383542, 0.0075734127, -0.014366732, -0.0012310859, -0.025691098, -0.008516027, -0.0141067, -0.008470521, 0.03864066, -0.016824028, -0.0037281993, -0.0018543486, -0.008386011, 0.015393856, -0.0042320103, 0.039212726, 0.025704099, -0.03926473, 0.07899752, -0.0056686834, 0.021413581, 0.009328624, -0.016616004, -0.023012774, 0.0009109223, -0.022284687, 0.012722033, 0.001493555, -0.03632638, -0.054034512, -0.023077782, -0.050056033, -0.045245454, 0.015536873, 0.022843754, 0.0098031815, 0.024273926, 0.005051109, -0.007625419, -0.08679846, -0.027251285, -0.03169782, -0.016824028, 0.023701858, -0.005818201, 0.029435549, 0.0028099637, 0.020282445, -0.0514342, 0.05234431, -0.027277287, 0.008535529, 0.015341849, 0.0072158696, -0.011213852, -0.025119027, -0.015562876, 0.06537188, -0.015640885, -0.03957677, 0.019515352, 0.02249271, -0.0031528801, -0.039550766, -0.0068908306, -0.0009474892, 0.025262045, 0.0147177745, -0.008516027, -0.055698715, -0.009452139, -0.03200986, 0.035208244, -0.0005956343, 0.009198609, 0.01651199, 0.006429275, -0.005483411, 0.00041726904, 0.018462226, -0.010414256, 0.01651199, 0.0037639537, 0.018046176, -0.030917726, 0.03913472, 0.021166552, -0.03291997, 0.031047743, 0.027693339, 0.02463797, 0.0097511755, 0.005317641, -0.005489912, 0.032789953, -0.023233801, -0.019255321, 0.023792868, 0.0033121493, -0.0038452134, 0.031957854, 0.006851826, -0.009673166, -0.008119479, 0.0132811, -0.0034551665, 0.021517593, -0.010388252, -0.009172605, 0.0063740183, 0.026094146, -0.023181794, -0.0047910777, -0.015159828, 0.0023857877, 0.0240529, 0.021439584, -0.0074954033, -0.008334004, 0.020841513, -0.010375251, -0.033492036, -0.002057498, -0.022024656, 0.006429275, 0.009946199, -0.0340121, 0.021842632, 0.0154198585, -0.005246132, -0.028187398, -0.015146825, 0.008366508, -0.019866394, 0.03185384, -0.0045765517, 0.04256713, 0.040356863, -0.026731223, 0.04745572, 0.051850248, -0.0423071, -0.029227523, -0.005649181, -0.015523871, 0.009224612, 0.008717551, 0.02547007, -0.01277404, -0.007696928, 0.03598834, -0.016420979, -0.026757225, -0.023844874, 0.010069714, 0.04048688, -0.0128000425, 0.001422859, -0.003201636, -0.005343644, -0.0077424333, -0.020425461, 0.028343417, 0.008951579, 0.0066568023, -0.029253526, 0.019177312, 0.019671371, -0.0058962107, -0.025834113, -0.01725308, -0.03167182, 0.010095717, -0.028577445, -0.020451466, 0.018800266, -0.018371215, 0.0005826327, 0.005675184, 0.0047715753, -0.016134946, 0.007865949, -0.017773142, -0.05086213, 0.005948217, -0.0040207347, -0.049223933, -0.000756935, 0.030189639, 0.040356863, -0.0028814722, -0.011759917, 0.0122344745, -0.016121943, 0.020750502, 0.007865949, -0.02050347, -0.019619364, 0.020698495, -0.004430284, 0.034896206, -0.0034031603, -0.019372335, 0.01999641, -0.04446536, -0.025925124, -0.0024345436, -0.020217437, 0.034532163, -0.0050998647, 0.007976461, 0.024377938, 0.017890157, 0.017929161, 0.04030486, 0.026757225, 0.016915038, -0.014977805, 0.016225956, -0.019216316, 0.011655905, 0.039680783, -0.005177874, 0.02622416, -0.008314502, -0.03583232, 0.013378613, 0.0010141223, 0.0012375867, 0.00447904, 0.0025856867, -0.020620486, 0.027849358, -0.0083015, -0.013229094, 0.034558166, -0.02463797, -0.035728306, 0.032477915, -0.04802779, 0.027927367, 0.0012985315, -0.023207797, -0.015874913, 0.03806859, 0.013547633, -0.014366732, 0.02465097, 0.029643573, -0.0080739735, -0.006032727, 0.035286255, 0.0044335346, -0.0124229975, -0.007209369, -0.0257171, -0.009198609, -0.01635597, 0.0103687495, -0.008607037, -0.015744898, -0.007774937, 0.004677314, -0.0032000106, 0.013287602, 0.009660165, 0.019463345, 0.0015081818, 0.005291638, -0.016603, 0.033908088, 0.0048820884, 0.017266082, 0.011096838, 0.011545392, 0.03393409, -0.024989013, 0.017799145, -0.012344988, 0.020555478, 0.026120149, -0.03291997, -0.014639765, -0.008314502, -0.009920196, 0.024924004, 0.043113198, -0.0057044374, -0.04495942, -0.019619364, 0.03947276, -0.024429945, -0.015822908, 0.0323739, 0.028031379, 0.029591568, 0.03806859, 0.0029676077, 0.005450907, -0.01733109, 0.04131898, 0.028577445, -0.009133601, 0.024611967, -0.009744675, -0.017162068, 0.009153103, 0.04828782, 0.027433306, -0.046805643, 0.042697147, 0.009107598, -0.0013391614, -0.005616677, 0.0039167223, 0.0060944846, 0.010901814, 0.009543151, -0.0019664872, 0.01252701, 0.013560635, 0.009686167, 0.025873119, 0.017344091, -0.004453037, 0.008652543, -0.018930282, 0.03167182, 0.0009629286, -0.01842322, 0.03037166, 0.0038842182, -0.040746912, -0.018826269, 0.018826269, -0.020191435, 0.0022281436, -0.011798922, -0.018267201, 0.0059774704, -0.025587084, 0.010472762, 0.0025255545, -0.024130909, -0.0027677086, 0.022700736, -0.08242994, -0.010797801, 0.029825594, -0.033908088, -0.0006744563, 0.027147273, -0.001849473, 0.020451466, 0.01501681, 0.007859447, -0.058871098, 0.022102665, -0.0128845535, 0.00027100142, 0.031983856, -0.003640439, 0.019931402, 0.013807665, -0.0006460154, -0.004693566, -0.013034071, -0.0016804526, -0.0018966036, 0.0014074197, 0.025483072, -0.012657026, 0.004446536, -0.048755877, 0.0027937118, -0.011889934, -0.031983856, -0.011259357, -0.014743777, 0.0323739, -0.03957677, -0.022934765, -0.0340121, -0.0060262266, 0.006611297, -0.009692668, 0.0076189185, 0.016824028, 0.013430619, 0.04776776, 0.008750055, 0.0014643015, 0.0060684816, -0.013729655, 0.0406429, -0.00348442, 0.001974613, -0.022960767, -0.016902037, -0.007534408, -0.0107782995, 0.0064227744, 0.0017584619, -0.0007918767, 0.0042580133, -0.025158033, 0.00871105, 0.025093025, 0.006338264, 0.0042417613, 0.009068593, -0.018098181, -0.0030163636, -0.02281775, 0.01211096, 0.0069038323, 0.0019681123, 0.0063935206, 0.034246128, -0.007898452, -0.029903604, 0.0051226174, 0.017318087, -0.002380912, 0.03882268, 0.029123511, -0.016824028, 0.0021095043, 0.005350145, 0.0061594923, 0.0059027113, -0.0052948883, -0.009777179, -0.0031008737, 0.008067473, -0.0074694003, 0.015159828, -0.008060971, 0.009335125, -0.014470744, -0.02397489, -0.0061432407, -0.008444518, 0.031151755, 0.0045830524, 0.04586953, 0.024676975, -0.011103339, 0.033648055, -0.03344003, 0.050446082, -0.02440394, 0.012253977, -0.015991928, -0.03336202, 0.003326776, 0.006819322, 0.011077335, 0.010960321, -0.023103785, 0.017539114, -0.015991928, 0.0026701968, 0.0012684654, 0.027849358, -0.007826944, 0.013391614, 0.014587758, 0.033700064, -0.016563997, 0.0021452585, -0.007423895, -0.004414032, -0.01526384, 0.0018575989, 0.00096374116, -0.011727413, -0.026146151, 0.04745572, 0.012852049, 0.004040237, 0.009380631, 0.024377938, 0.001766588, 0.010407754, -0.039290737, 0.0054249037, -0.00044896037, 0.027303291, -0.00039918875, -0.018358212, -0.009692668, 0.0076189185, -0.015068816, -0.008971081, 0.0003362124, -0.0048008286, -0.015484867, 0.0014139204, 0.044725392, 0.0050641103, 0.00969917, -0.0020347452, -0.031411786, 0.023103785, -0.013456622, 0.014574757, -0.021803629, -0.018462226, 0.02588612, -0.03583232, -0.0005099052, 0.012826046, 0.009094596, 0.016720016, 0.0050901137, -0.035312258, -0.036872443, -0.0012148339, -0.02190764, 0.03341403, 0.0018754761, -0.043087192, 0.018722257, -0.01112284, 0.032139875, -0.00081300427, -0.018371215, 0.033284012, 0.018462226, 0.013573636, -0.014574757, -0.005714189, -0.0005131556, -0.0059839715, 0.017149067, -0.008457519, -0.010030709, -0.035208244, -0.02049047, -0.022674734, 0.011675407, 0.017019052, 0.004414032, -0.046649624, -0.0027969622, -0.011675407, -0.030345658, -0.0013895425, -0.0054119024, 0.014210713, -0.015146825, 0.0073588872, 0.031827837, -0.005236381, -0.015744898, 0.034038104, 0.006244003, -0.004904841, 0.025340054, -0.053072397, -0.0029546062, 5.505351e-05, -0.0039329743, -0.03923873, -0.0012107709, -0.016134946, -0.01501681, 0.029591568, 0.0064682798, 0.003676193, 0.023701858, -0.016746018, -0.013469623, 0.00530789, -0.018995289, -0.007241873, -0.005688186, -0.016290965, 0.0032666437, 0.008821564, -0.029149514, 0.016668009, -0.0018462226, 0.036274374, -0.0071703643, 0.018475227, 0.055438682, -0.018618245, -0.039342742, 0.016030932, -0.009562653, -0.03161981, -0.018917281, 0.00017674005, -0.0031317526, 0.006851826, -0.031541802, 0.0038224608, -0.009413134, 0.001439111, -0.018891277, -0.0023874128, 0.0059612184, 0.0010628782, -0.004904841, 0.0056686834, 0.0055971746, -0.024924004, 0.004804079, 0.020308448, 0.01501681, 0.0047390712, -0.0019047296, -0.010758797, -0.0323479, 0.014470744, -0.014171708, 0.019346332, 0.018774264, -0.011402374, 0.0257171, -0.007027347, -0.006045729, -0.014314725, 0.0075019044, -0.003425913, 0.0040109833, 0.004651311, -0.019086301, 0.050316066, 0.011090336, 0.020620486, -0.019502351, 0.004365276, 0.023545839, 0.027147273, -0.03947276, 0.009647163, -0.00048674617, 0.017188072, -0.0047813263, -0.034194123, -0.0077294316, -0.030553684, -0.0464416, -0.020802507, 0.03286796, -0.0034291632, -0.021543596, 0.009055591, 0.011187849, -0.009179106, 0.016368974, 0.031749826, -0.0076579233, -0.008314502, -0.018358212, 0.0063902703, -0.007605917, -0.0023045277, -0.016537994, -0.016928041, 0.009757676, 0.051486205, 0.010128221, 0.019359333, 0.018891277, -0.023532836, -0.008607037, 0.002224893, 0.018410219, -0.019437343, 0.03260793, 0.0053468947, 0.026159152, -0.023454826, 0.009614659, 0.0038549646, 0.0034226626, 0.028343417, 0.024937006, -0.03143779, 0.03128177, 0.013138084, 0.020828512, -0.02639318, -0.013911677, 0.038874686, 0.00012920308, -0.025275046, 0.019645369, 0.022830753, 0.0273813, 0.046311583, -0.0052656345, 0.030579686, -0.033206, -0.0018575989, 0.024299929, -0.0024134158, -0.026705218, -0.012767539, -0.007235372, 0.0013050323, -0.0019859893, 0.0013448496, 0.0042515127, -0.0078009404, -0.015796904, 0.03874467, 0.016941043, -3.575431e-05, 0.027693339, 0.00033275885, 0.016030932, 0.04381528, 0.002772584, 0.006022976, 0.0023354066, 0.0003181321, 0.064227745, 0.002047747, -0.006045729, -0.022986772, -0.021023534, 0.011421877, 0.04256713, 0.0055289166, -0.03351804, -0.018241199, -0.015991928, -0.016563997, 0.0002517022, 0.017981168, -0.008477022, -0.00871105, 0.0043847784, 0.018059177, 0.01659, 0.022778746, -0.020464467, -0.003035866, -0.010804302, -0.019372335, -0.017565118, -0.0045765517, -0.011057833, -0.020880517, -0.020477468, -0.001717832, -0.0068063205, 0.005730441, 0.013651646, 0.0058539556, -0.00069152087, -0.011382872, -0.03037166, -0.005275386, 0.0023646601, 0.010154224, 0.018189192, -0.0049405955, -0.0157579, -0.037964575, 0.0008434767, -0.010908315, 0.02065949, -0.013365611, 0.010095717, -0.026679216, -0.0066373, 0.0039947317, 0.0024686726, -0.006760815, 0.0003616061, 0.012806544, -0.008769557], "content_hash": "b179565f32f2a6b1ddd19231261453ae"}, {"content": "within each layer (rather than sequentially). This is a design choice that slightly changes how information flows through the network. ### GPT-NeoX Architecture Pythia uses the GPT-NeoX architecture (`GPTNeoXForCausalLM`), which is EleutherAI's evolution of GPT-Neo. The module paths use `gpt_neox.layers.{N}.attention` instead of GPT-2's `transformer.h.{N}.attn`. ## Why Pythia Matters for Interpretability\n\n- **Open training data**: The Pile dataset is fully public, so you can trace what the model learned\n- **Training checkpoints**: Hundreds of checkpoints are available, letting researchers study how features develop during training\n- **Designed for research**: EleutherAI specifically built Pythia to be a good subject for mechanistic interpretability\n- **Comparison with GPT-Neo**: Same creator but different architecture (rotary vs absolute PE), enabling clean ablation studies of positional encoding\n\n## Comparing Pythia and GPT-2\n\nRunning the same prompt on both is highly educational:\n- **Same size, different PE**: Pythia-160M and GPT-2 both have 12 layers and 12 heads, but use different positional encodings\n- **Parallel vs sequential**: Pythia processes attention and MLP in parallel; GPT-2 does them sequentially\n- **Different training data**: GPT-2 was trained on WebText; Pythia on The Pile\n\n## HuggingFace Model IDs\n\n- `EleutherAI/pythia-160m` (in dropdown)\n- `EleutherAI/pythia-410m` (in dropdown)\n- `EleutherAI/pythia-1b`, `EleutherAI/pythia-1.4b` (larger, enter manually)", "source_file": "pythia_overview.md", "chunk_index": 1, "embedding": [0.01969437, -0.0016391918, 0.03886612, -0.010087863, 0.02691472, -0.0042187483, -0.003809595, 0.046100225, 0.0037202002, -0.00030213734, 0.013134164, -0.04670536, -0.034767713, -0.029734096, -0.0333649, -0.030174194, 0.010404183, -0.02723104, 0.007488536, 0.015939785, 0.018181533, -0.021385994, -0.0037202002, 0.010768639, 0.04863079, 0.0062885825, 0.007137833, 0.043542158, 0.0038405394, -0.0039230576, 0.0288539, -0.02020323, -0.035867956, -0.002537438, -0.020629575, -0.011710723, -0.016819982, 0.0333649, -0.0067733773, 0.08059288, 0.010115368, -0.057267703, -0.023228902, 0.062328827, -0.05039118, 0.034685194, 0.027712397, 0.05765279, 0.015293392, 0.03773837, 0.017741436, 0.025484404, -0.05157394, 0.01474327, -0.026337093, -0.02677719, 0.005064561, 0.0057143928, -0.04381722, -0.025126824, -0.019364296, -0.040956587, 0.034630183, 0.008513139, 0.022101153, 0.018209038, -0.100232236, 0.02998165, -0.01230898, 0.0509413, 0.017260078, 0.0076123136, -0.018470347, -0.001381322, 0.030366736, -0.0128453495, 0.0009876409, 0.03286979, 0.02087713, -0.0077704736, 0.011628204, 0.007694832, 0.00049725873, -0.022307448, 0.016311118, -0.0006330701, -0.017370103, -0.008341225, -0.04827321, -0.036445584, -0.0155134415, -0.013883704, -0.06827015, 0.014440703, 0.010589849, 0.024315394, -0.0003556453, -0.06639973, -0.010287282, 0.020093206, 0.026034525, 0.0005441051, -0.05193152, -0.0067011737, 0.035950474, -0.024260381, 0.022334954, 0.038508542, -0.013863075, -0.039113674, -0.10996939, -0.0077223377, 0.0162286, 0.012343363, 0.0035310958, -0.025346871, -0.012584042, 0.027946198, -0.0025030551, 0.007577931, -0.020794613, 0.0077911033, 0.012082055, 0.012198956, 0.017273832, -0.015939785, 0.020753354, -0.025690699, -0.015967293, -0.0030755259, -0.012363993, 0.012474017, 0.036720645, -0.022348708, -0.0088363355, 0.0198319, -0.008217447, 0.0034829602, -0.021523524, -0.045632623, -0.042304385, -0.0127697075, 0.01650366, 0.043157075, -0.030421749, 0.04791563, -0.007963017, 0.03187957, -0.03174204, 0.045220032, 0.020904638, 0.0047585554, 0.0004529911, -0.022720039, -0.038260985, 0.00038916836, 0.029761601, 0.059083104, 0.022830063, -0.053801935, 0.010459195, 0.02254125, -0.0032371243, 0.028798888, -0.007502289, 0.088074535, -0.02150977, -0.016063564, -0.05014362, -0.06337406, -0.03485023, 0.0025580674, -0.04274448, 0.022527497, 0.0010005344, -0.0070346855, -0.035070278, -0.03938874, -0.033915024, -0.039911352, -0.02398532, 0.003520781, -0.04904338, 0.025456896, -0.055919904, -0.020945895, 0.0025494718, 0.03925121, -0.009744036, 0.010335417, 0.0043425257, 0.05297675, 0.091980405, 0.0066289706, -0.0029723782, -0.026983485, 0.049345944, 0.010858034, -0.0035517253, 0.0009833431, 0.023105126, 0.04205683, 0.04079155, -0.010837404, 0.07190095, -0.053526875, -0.009482728, -0.033337396, -0.05248164, -0.014825788, -0.0011896389, -0.055479806, 0.014853295, 0.0028296902, -0.010871787, -0.006646162, -0.057377726, 0.04860328, 0.03925121, 0.049345944, -0.026612153, 0.024631714, 0.007117204, 0.0347127, -0.050281152, 0.017507633, 0.036418077, -0.052729197, -0.010926799, -0.017521387, 0.008354979, 0.022871323, 0.014385691, -0.007736091, 0.0137874335, -0.052096557, 0.027176028, -0.023545222, -0.001846347, -0.038508542, -0.021028414, -0.0023895926, -0.029624071, 0.0009291905, 0.022156164, -0.02335268, -0.0024910213, -0.070855714, 0.03985634, 0.028743876, 0.0064364276, -0.0053740046, -0.023779025, -0.013794309, -0.02677719, 0.010624232, -0.024865516, -0.0011225927, -0.037655853, -0.032842286, 0.0004826461, 0.058863055, 0.02168856, 0.03435512, -0.032622237, -0.07030559, -0.030394241, -0.011889513, -0.030229205, -0.067334935, 0.051188856, -0.006322965, 0.0029431528, 0.04692541, -0.04255194, -0.032924805, 0.010500454, 0.015155862, -0.03936123, -0.0014217217, -0.013539879, 0.011146847, 0.068325154, 0.0027145084, -0.009214544, 0.02087713, 0.0026766874, 0.017397609, -0.018442841, -0.029183973, 0.019584345, -0.015994798, 0.020038195, -0.0243429, 0.027464842, 0.009799048, 0.035427857, -0.015939785, -0.0031717974, -0.003665188, -0.0020371706, -0.036610622, -0.0019804393, -0.048933353, -0.0009635731, 0.03760084, -0.0038955517, 0.012047673, 0.011311884, 0.015705984, -0.000370043, -0.010651737, -0.004937345, 0.0077429675, -0.0029792546, -0.0066152173, -0.04241441, -0.008863841, 0.028743876, -0.02487927, -0.025731958, -0.060018312, -0.013251064, -0.021537278, -0.031054389, 0.034960255, -0.02353147, -0.038783602, 0.0006962482, -0.008506262, 0.040681522, -0.029541552, -0.021647302, -0.016957512, -0.023256408, 0.008183065, -0.0011251714, 0.076301925, -0.017645163, 0.018470347, -0.0401314, -0.0051402026, -0.027973706, 0.001288489, -0.04197431, 0.018924197, 0.01957059, 0.012515276, 0.042496927, 0.026405858, -0.025580674, -0.009977838, -0.017933978, 0.017383857, 0.026955979, -0.014949566, -0.011820747, -0.010679244, -0.030861845, 0.01294162, -0.06023836, -0.004428482, -0.007199722, 0.009372704, 0.008650669, 0.027643632, -0.023558976, 0.03130194, -0.01309978, 0.019061727, 0.0014672786, -0.008361855, 0.004229063, -0.02740983, -0.027464842, 0.003954002, -0.039278712, 0.02606203, -0.020175725, 0.014894553, 0.035125293, 0.026845954, 0.038068444, -0.034025047, -0.0051264497, 0.022830063, 0.008774446, 0.005349937, -0.05256416, 0.024824256, 0.07003053, -0.011593821, -0.017301338, 0.0250168, -0.0243429, -0.0012506681, -0.015444675, -0.013044769, 0.057322714, 0.06397919, -0.031274438, 0.0054668374, -0.013739297, -0.05099631, -0.0058278553, -0.0155134415, 0.0067871306, 0.02998165, 0.024631714, 0.004005576, 0.03300732, 0.014371938, 0.034932747, -0.014124383, 0.0061201076, -0.065794595, 0.033749986, 0.0058725527, 0.0039849463, 0.040461473, 0.011559439, -0.11002441, -0.057047654, -0.04945597, 0.0040227673, 0.014275666, 0.04029644, -0.04310206, 0.007344129, -0.023228902, -0.011366896, 0.034822725, 0.023834037, -0.039553773, 0.0137874335, -0.030091675, 0.025456896, 0.0041465447, -0.020684589, 0.009929703, 0.06100853, -0.015499688, -0.015183368, 0.0030411433, -0.002489302, -0.017617658, -0.018731656, -0.04805316, -0.017163808, -0.027616126, 0.047448024, 0.04128666, 0.002814218, -0.026515882, 0.032979816, -0.024741739, 0.08350852, -0.008774446, 0.05432455, -0.05063873, 0.0581479, -0.033282384, -0.04139668, 0.043982256, -0.035950474, 0.015719736, 0.008423744, -0.022486238, 0.0112775015, -0.04048898, -0.018429087, -0.07019557, 0.020767106, -0.033942528, -0.01668245, 0.0106448615, -0.02155103, 0.03845353, 0.01957059, -0.011738229, 0.019900665, 0.010218517, 0.0014896273, 0.018869186, -0.0024789874, 0.058312934, -0.035317834, 0.0012257406, 0.024535442, 0.013691162, -0.0396638, -0.018924197, 0.050116118, -0.016627438, 0.014048741, -0.024549196, -0.025058059, -0.07943762, -0.045054995, -0.013037892, 0.003063492, 0.009056384, 0.0072478577, 0.03903116, -0.01672371, -0.013340459, -0.00405715, 0.027588619, -0.020657081, 0.012694065, -0.002301917, 0.027959952, -0.018869186, 0.031632017, 0.020051949, 0.034795217, 0.044614896, 0.02069834, 0.013959346, 0.0011904985, -0.011305008, -0.025223095, -0.06122858, -0.026515882, 0.026213314, -0.023545222, -0.03300732, -0.020684589, -0.016214848, -0.018552866, 0.016709957, -0.023902802, -0.0029156466, -0.038563553, -0.05946819, 0.03165952, -0.016874993, -0.034520157, -0.06639973, 0.034657687, -0.007117204, 0.013230435, 0.013326706, -0.0004055001, 0.0032955748, 0.03231967, 0.0401314, -0.0073097465, 0.007571054, 0.0315495, 0.06617968, -0.018071508, 0.04106661, -0.00942084, -0.019240517, 0.0020148219, -0.010122245, 0.012419005, 0.023641493, 0.057542764, 0.012590918, -0.0076535726, -0.006714927, 0.016269859, -0.011380649, 0.016462402, 0.01282472, -0.024961786, 0.027918693, -0.046732865, -0.009805925, 0.01070675, 0.02515433, 0.009592753, -0.003558602, -0.0007706006, 0.0046966667, 0.020051949, 0.010541713, -0.036858175, -0.023105126, -0.023325173, -0.0015042399, 0.027519854, -0.03900365, 0.01988691, -0.03196209, -0.024109097, -0.028963925, 0.057817824, 0.012281475, -0.017645163, 0.0008587061, 0.015458429, 0.0070621916, -0.015499688, 0.020684589, -0.0149633195, -0.049510982, -0.00615449, -0.0054737143, 0.0162286, -0.017260078, 0.020753354, 0.0068696486, 0.008898224, 0.0024858639, 0.0419193, 0.0025683823, -0.004703543, -0.018951705, 0.0033248, 0.022499992, -0.02538813, -0.05360939, -0.002370682, 0.0046657226, 0.0057797194, 0.024150357, -0.0021385993, 0.013154793, -0.010246023, -0.005054246, -0.007117204, 0.03061429, -0.013863075, -0.031191919, -0.045220032, -0.038013432, 0.021523524, -0.025278106, 0.023971567, 0.056992643, 0.05025365, -0.014234408, 0.005171147, 0.0046932288, -0.02542939, 0.027244793, -0.028633852, -0.019309282, 0.01816778, -0.022940088, 0.00017202058, -0.0019065166, -0.0008062726, -0.0047585554, 0.038205974, 0.028386297, -0.034905244, 0.00036338138, -0.017741436, 0.030091675, -0.009049508, 0.0017148335, -0.035235316, -0.0076535726, -0.04720047, -0.007316623, 0.0123777455, -0.06122858, -0.01442695, -0.021496018, -0.010108492, -0.008513139, 0.008499385, 0.015183368, 0.02970659, 0.0053740046, -0.021289723, -0.01005348, 0.0509413, 0.0491259, 0.0051608323, -0.023173891, -0.022706287, 0.003991823, -0.05311428, -0.03165952, 0.01411063, 0.006429551, -0.015155862, -0.042276878, -0.0052949246, 0.006794007, -0.010135998, -0.04296453, 0.008024905, 0.020987155, 0.005291486, 0.039113674, -0.012363993, -0.00048479505, -0.018607877, -0.0015592522, -0.03286979, -0.017411362, 0.028441308, 0.017782694, -0.041946806, -0.05025365, -0.006037589, 0.0017483566, 0.03619803, -0.023820283, -0.015307145, -0.047338, -0.015637219, -0.0020921829, -0.019941924, -0.07525669, -0.018209038, 0.01880042, 0.01050733, -0.0006124405, -0.036858175, -0.0288539, 0.0024617962, -0.01897921, -0.018470347, -0.014303173, 0.023435198, 0.025759464, -0.013113534, 0.0074679065, -0.029018937, -0.0065533286, -0.0032182138, -0.01609107, 0.02060207, 0.0061097927, 0.0074679065, -0.0034657686, -0.012632177, 0.010500454, -0.02402658, -0.019102987, 0.01933679, -0.026722178, -0.016792474, -0.014908307, -0.016297365, -0.00076458364, 0.017273832, 0.03083434, 0.013381719, -0.0076329433, 0.023050113, 0.046237756, -0.01640739, 0.0021110934, 0.011958278, 0.014481963, 0.0007009758, -0.008781323, -0.042909518, 0.013883704, -0.016654944, -0.0021643864, 0.01474327, -0.0013778837, 5.1117295e-05, -0.027327312, 0.016104823, -0.02060207, -0.028276272, 0.032484706, -0.019776886, 0.0007142991, -0.0033248, -0.022307448, 0.0026337092, 0.0037752124, 0.030284217, 0.041176632, -0.0644743, 0.06375914, -0.040681522, 0.033419915, 0.010087863, -0.024920527, -0.010940552, -0.009551493, -0.033694975, 0.019047976, 0.009180161, -0.031356957, -0.035785437, -0.0038852368, -0.053279318, -0.04175426, 0.016613685, 0.0004959694, 0.0033110469, 0.017810201, 0.007577931, -0.009235173, -0.09418089, -0.0302017, -0.017246326, -0.017548893, 0.019543085, -0.021922363, 0.046100225, 0.014550728, 0.046732865, -0.06100853, 0.054434575, -0.027712397, 0.0104110595, 0.010665491, 0.0016598213, -0.011710723, -0.008987619, -0.009111396, 0.07047063, -0.02497554, -0.03980133, 0.014702011, 0.01996943, 0.0058656763, -0.06617968, -0.0022211177, 0.0014973634, 0.019075481, 0.016985018, -0.013588014, -0.048218194, 0.0010392149, -0.025264354, 0.03011918, 0.002066396, 0.02168856, 0.012535905, 0.022582509, 0.0020853062, -0.001872134, 0.016668698, -0.014041864, -0.003336834, 0.025003046, 0.016985018, -0.019419307, 0.024507936, 0.02064333, -0.027602373, 0.02024449, 0.028633852, 0.002590731, 0.001080474, 0.012879732, -0.00044181675, 0.028014963, -0.0265984, -0.023613987, 0.022706287, 0.0010555467, 0.014935813, 0.027987458, -0.008471879, -0.011648834, 0.006164805, 0.0049545364, 0.022623768, 0.019364296, -0.0015944943, 0.001126031, 0.006360786, 0.03168703, -0.010871787, -0.009290186, -0.011607574, 0.008623163, 0.01102307, 0.026653413, 0.0015162738, 0.0058347317, 0.0036411202, -0.013924964, -0.031577006, 0.01217145, 0.00028623536, 0.0025357187, 0.036032993, -0.047888122, 0.02024449, 0.021894857, 0.0007233245, -0.011662587, -0.028551333, 0.024549196, -0.029624071, 0.025319366, -0.025731958, 0.045935187, 0.04252443, -0.025498156, 0.03231967, 0.038508542, -0.03223715, -0.019763134, -0.015802255, 0.005707516, 0.015169615, 0.01609107, 0.024439171, -0.012123314, 0.005539041, 0.02461796, -0.024287887, -0.003902428, -0.017713929, -0.0008870718, 0.022885077, -0.0088363355, 0.0008862122, 0.0017096761, 0.009620259, -0.010287282, -0.0075160423, 0.021165945, -0.0063779773, 0.005432455, -0.00096013484, 0.03204461, 0.016806228, 0.015073343, -0.034107566, -0.0071034506, -0.008334349, 0.011387526, -0.017411362, -0.030009156, 0.0006915206, -0.0071447096, -0.0059447563, 0.00048479505, 0.007880498, -0.024054086, -0.0015489373, -0.011270625, -0.0401314, 0.0037717742, -0.0144131975, -0.026502129, -0.0062370086, 0.020079454, 0.046155237, -0.0184841, -0.008966989, -0.005666257, -0.0071447096, 0.020725848, 0.0069556055, -0.02087713, -0.0060032066, 0.03097187, 0.0033815312, 0.03859106, -0.0006287723, 0.0019701244, 0.034630183, -0.04953849, -0.027891187, -0.0083687315, -0.01686124, 0.024741739, -0.019268025, 0.017466374, 0.03386001, 0.029266492, 0.040186413, 0.03768336, 0.018621631, 0.0259245, -0.009523988, 0.013849322, -0.01686124, 0.018236546, 0.025525661, 0.003967755, 0.050281152, -0.024370406, -0.00827246, 0.00975779, 0.00793551, 0.015307145, -0.0021764203, -0.0012094089, -0.020615824, 0.026543388, 0.0030961556, 0.0070346855, 0.03300732, -0.015114603, -0.03124693, 0.0031305382, -0.031714536, 0.028138742, -0.004349402, -0.018772915, -0.015210874, 0.055507313, 0.005900059, -0.024604207, 0.0055149733, 0.037628345, -0.013127287, 0.009503358, 0.0153621575, 0.023875296, -0.022940088, 0.007254734, -0.008286213, -0.00809367, -0.006477687, -0.0061338604, -0.022665028, -0.0129209915, -0.012460263, -0.012233338, -0.012549658, 0.017411362, 0.0066530383, 0.023338927, 0.005274295, 0.020189479, -0.0068765255, 0.028193753, -0.0029328382, 0.01167634, -0.0024514813, 0.0069280993, 0.02998165, -0.009565246, 0.014468209, 0.005157394, 0.03204461, 0.03223715, -0.017053783, 0.001080474, -0.00599633, -0.0005931003, 0.016599933, 0.032484706, -0.004211872, -0.027176028, -0.019254271, 0.027932446, -0.021427253, -0.021468513, 0.020670835, 0.025319366, 0.022912582, 0.030559279, 0.006639285, 0.013883704, -0.001244651, 0.048300713, 0.022678781, 0.0035861079, 0.014481963, -0.022472484, -0.023407692, 0.016214848, 0.052509148, 0.013766804, -0.034520157, 0.03141197, -0.0073372526, 0.0002054362, -0.008719434, -0.00020597343, 0.011600698, 0.0028004649, 0.0067183655, -0.018250298, -0.0015334651, 0.018085262, -0.0063435948, 0.030779328, 0.02353147, -0.01717756, -0.0026354282, -0.013189175, 0.03187957, 0.0051092585, -0.011538809, 0.019529331, -0.002382716, -0.027629878, -0.009035754, 0.02087713, -0.009613383, 0.0069934265, -0.0057281456, -0.040241428, 0.01246714, -0.030009156, 0.0130035095, -0.00987469, -0.03204461, -0.00079166, 0.024893021, -0.06904031, -0.013808063, 0.034740206, -0.015802255, -0.0072891167, 0.016847488, -0.008244954, 0.02542939, 0.02998165, 0.009441469, -0.040461473, 0.0069831116, -0.008801952, 0.014660752, 0.014399444, -0.0004158149, 0.027643632, 0.015760995, -0.0033970035, -0.027794916, -0.0324572, -0.0034537348, -0.001715693, 0.0113531435, 0.026364598, 0.0083687315, -0.0019598098, -0.045990203, -0.014358185, -0.010713627, -0.018827926, -0.01591228, -0.0046003954, 0.008471879, -0.02051955, -0.010954305, -0.026502129, 0.0037820889, -0.0062507614, -0.007295993, -0.009015124, 0.028001212, -0.00014064643, 0.056029927, 0.01296225, -0.019158, -0.008685051, -0.014633246, 0.048850834, -0.0009257522, 0.0072616106, -0.024714231, -0.018305311, -0.0032182138, -0.0028623536, 0.006725242, 0.010342294, 0.021716068, 0.031439472, -0.027382324, 0.0038027186, 0.017260078, 0.009778419, 0.006766501, 0.015087097, -0.021289723, -0.010741132, -0.018442841, 0.022458732, 0.0018566619, 0.0069727967, -0.021454759, 0.008437497, -0.008485632, -0.038343504, -0.0009970962, 0.021441005, -0.009833431, 0.03889363, 0.015678478, -0.005614683, -0.017095042, 0.0078942515, 0.011098712, -0.007859868, 0.0017096761, -0.013753051, -0.01309978, 0.013134164, 0.005233036, 0.0074816598, -0.005253665, 0.00052175636, 0.0058450466, -0.0128453495, 0.0066805445, 0.0007886515, 0.038316, 0.012116438, 0.036308054, 0.039691303, -8.987833e-05, 0.023669, -0.02461796, 0.02511307, -0.0162286, 0.02677719, -0.011380649, -0.03732578, -0.0028881407, 0.0067974455, 0.014055618, 0.0047344877, -0.017851459, 0.011421909, -0.0033093279, -0.018621631, -0.0042565693, 0.021496018, -0.023063866, 0.021441005, 0.009290186, 0.0315495, -0.01460574, 0.012749078, -0.018277803, -0.008320596, -0.006281706, -0.0068111983, 0.011016194, -0.023215149, -0.01699877, 0.0491259, 0.017700177, 0.0252506, 0.019804392, 0.02402658, 0.004370032, 0.003902428, -0.043762207, 0.0077842264, 0.0071034506, 0.03075182, 0.011016194, -0.01280409, 0.0002752759, 0.009097643, -0.015348405, -0.0054633995, 0.009730283, 0.0007577071, -0.0015678478, 0.02637835, 0.01650366, 0.0104110595, 0.001597073, -0.015582207, -0.016902499, 0.026818449, 0.007908004, 0.023737766, -0.017521387, -0.01215082, 0.015540947, -0.052866727, 0.012900362, 0.012419005, -0.0008638635, 0.0013452203, -0.0048376354, -0.02970659, -0.049290933, -0.008100547, -0.011071206, 0.037243262, 0.001426879, -0.03047676, 0.010748009, 0.007997399, 0.008946359, -0.017865213, -0.009682148, 0.031989597, 0.03033923, 0.019254271, -0.008630039, 0.008031782, 0.0055218497, 0.019061727, 0.022266189, -0.012590918, 0.00908389, -0.039828833, -0.017246326, -0.031769548, 0.020615824, 0.0039986996, 0.0081143, -0.04015891, -0.022169918, -0.030449254, -0.011923895, -0.0018136835, -0.010582972, 0.005783158, -0.035895463, 0.007921757, 0.027671138, 0.016737463, -0.01537591, 0.017452622, -0.0068111983, 0.005814102, 0.027299806, -0.046237756, 0.0017543735, 0.0030755259, -0.009303939, -0.034052555, 0.00397807, -0.0007882217, -0.018442841, 0.020024441, 0.0144131975, -0.0021970498, 0.010871787, -0.024054086, -0.031907078, 0.014289419, -0.022197424, 0.006567082, 0.011284378, -0.033282384, -0.015183368, -0.011621328, -0.028963925, -0.004270322, 0.0014664191, 0.028124988, -0.015142108, 0.031769548, 0.045027487, -0.02335268, -0.01442695, -0.001676153, -0.023792777, -0.025085565, -0.037023213, -0.017975237, -0.0023603672, 0.011751981, -0.026034525, 0.0047551175, -0.020354515, 0.006113231, -0.019721875, -0.011153724, 0.020932144, -0.01510085, 0.024851762, -0.0006429551, 0.0064192363, -0.017397609, 0.010754885, 0.0026233944, 0.025690699, 0.016256105, -0.0015987922, -0.0072203516, -0.006948729, 0.016654944, -0.017686423, 0.015637219, -0.0037580212, -0.015032085, 0.017988991, 0.001715693, -0.010287282, -0.013918087, 0.0028434433, -0.016269859, 0.0068765255, 0.0012747359, 0.0017182718, 0.057542764, -0.005879429, 0.025498156, -0.018442841, 0.024150357, 0.0065533286, 0.028908912, -0.032704756, -0.002764363, 0.017108794, 0.02497554, -0.013533002, -0.036610622, -0.037848394, -0.029128961, -0.040819053, -0.0214135, 0.029321505, -0.016751215, -0.020354515, 0.0150458375, -0.0028623536, -0.01235024, 0.0152796395, 0.034767713, -0.011366896, -0.008306842, -0.013979976, 0.007660449, -0.0059069353, -0.008293089, -0.008361855, -0.015843514, 0.0036239289, 0.04205683, -0.01460574, 0.03958128, 0.024631714, -0.027464842, -0.0054358933, -0.0034743645, 0.027121017, -0.0184841, 0.04051649, 0.015540947, 0.017053783, -0.0070896978, 0.010005345, 0.008953236, -0.007550425, 0.027396077, 0.027877433, -0.02465922, 0.012810967, -0.0013056803, 0.016434895, -0.030641796, 0.0055562323, 0.034190085, -0.0068937168, -0.02020323, 0.016902499, 0.015018331, 0.048300713, 0.032209646, -0.011346267, 0.023283916, -0.025773216, -0.011078082, 0.01298288, -0.009475852, -0.01309978, 0.0026921597, 0.0021231272, 0.02186735, 0.0011182949, 0.014468209, -0.0017492161, 0.01919926, -0.013966223, 0.028523827, -0.00324572, -0.010892416, 0.009475852, 0.0013710072, 0.015747244, 0.047255483, 0.0053980723, -0.010926799, 0.017535139, 0.014399444, 0.055507313, 0.00403652, 0.0047207344, -0.010452319, -0.022967594, 0.007846115, 0.0414792, 0.0006042747, -0.048795823, -0.027863681, -0.023132632, -0.0050989436, -0.017315092, 0.04923592, -0.0067011737, -0.014715764, 0.0013443606, 0.012185203, 0.014358185, 0.016379884, -0.033804998, -0.00042441054, -0.0050851903, -0.010321665, -0.0017053783, 0.0015996518, -0.0028950172, -0.016283613, -0.041094117, -0.002396469, 0.010892416, 0.015472182, 0.005820979, 0.018195286, 0.0040984093, -0.0122195855, -0.027849928, 0.008822582, -0.027684892, 0.020794613, 0.02493428, 0.010851157, -0.0035035897, -0.014633246, -0.0028400049, -0.031989597, 0.023146383, -0.007990522, 0.0063573476, -0.02705225, -0.025099317, -0.0031855504, 0.014619493, 0.0006803462, -0.0042565693, -0.0009137183, -0.013182299], "content_hash": "742a260c9397aef5392d8e61beb64bdd"}, {"content": "# Qwen2.5 Overview\n\n## What Is Qwen2.5?\n\nQwen2.5 is a family of language models developed by Alibaba's Qwen team, released in 2024. It uses a modern LLaMA-like architecture with several improvements over older models like GPT-2. The dashboard includes **Qwen2.5-0.5B** as a representative of this modern architecture.\n\n## Architecture Details\n\n| Property | Qwen2.5-0.5B |\n|----------|-------------|\n| Parameters | 494M |\n| Layers | 24 |\n| Attention Heads | 14 per layer |\n| Hidden Dimension | 896 |\n| Vocabulary Size | 151,936 |\n| Positional Encoding | Rotary (RoPE) |\n| Normalization | RMSNorm |\n| Activation Function | SiLU |\n\n## Key Architectural Differences from GPT-2\n\n### RoPE (Rotary Position Embeddings)\nInstead of GPT-2's learned absolute position embeddings, Qwen uses **Rotary Position Embeddings (RoPE)**. RoPE encodes position by rotating query and key vectors in attention. This means:\n- Better generalization to different sequence lengths\n- Position information baked into the attention computation\n- Attention patterns may look different from GPT-2\n\n### RMSNorm Instead of LayerNorm\nQwen uses **RMSNorm** (Root Mean Square Normalization) instead of standard LayerNorm. RMSNorm only normalizes vector magnitude without centering, making it simpler and slightly faster.\n\n### SiLU Activation\nWhere GPT-2 uses GELU in the MLP, Qwen uses **SiLU** (Sigmoid Linear Unit, also called \"Swish\"), a smooth activation function common in modern architectures.\n\n### Grouped-Query Attention (GQA)\nQwen uses **Grouped-Query Attention**, where multiple query heads share key and value heads. This reduces memory usage while maintaining quality. In the attention visualization, you may notice this affects how attention patterns are structured.\n\n## Why Include Qwen2.5?", "source_file": "qwen_overview.md", "chunk_index": 0, "embedding": [0.023338716, 0.015645921, 0.027334806, 0.022440573, 0.021399247, -0.004607867, -0.0466514, 0.03597781, -0.052821256, 0.02697034, -0.00480637, -0.020409988, -0.054617543, 0.031291842, 0.0021900386, -0.00548974, -0.013263889, -0.008180916, -0.010120386, 0.0076797786, -0.02754307, -0.009729889, -0.02013664, 0.033348463, -0.015281457, -0.02370318, -0.024184793, -0.0029124583, -0.010009745, -0.03069308, 0.027491003, -0.0257598, -0.052586958, -0.019954408, -0.05266506, -0.016374849, 0.010322143, -0.003514475, 0.005577602, 0.02769927, 0.011623801, -0.038164593, -0.014370297, 0.018535601, 0.0044093644, 0.015737038, 0.004269436, 0.04498528, -0.047510494, 0.01443538, 0.023247601, 0.048655953, -0.010198485, -0.0013529101, 0.041262537, -0.041496836, -0.028740594, -0.024354009, -0.0012723701, 0.008532364, 0.015060176, -0.02518707, -0.008239491, 0.04061171, 0.015216375, -0.027230673, -0.068050645, -0.029235225, -0.027178606, -0.029313324, 0.007914077, 0.020527137, -0.031317875, 0.00744548, 0.040143114, 0.016387867, 0.0012756243, 0.05068654, 0.01580212, -0.019798208, -0.01819717, -0.015398607, -0.0051675797, -0.04472495, -0.001193457, -0.01238527, -0.018561633, -0.013198806, -0.053055555, -0.013276905, -0.025174053, 0.02633253, -0.019381678, -0.01910833, 0.0073999222, 0.000687031, 0.026618892, 0.025499469, -0.0012300662, 0.0119882645, 0.01056295, -0.027673235, -0.023156485, -0.0070614913, 0.045089412, -0.010354685, 0.036368307, -0.025082937, -0.06143823, -0.0058476957, -0.09559371, 0.02936539, -0.021711646, -0.00094776927, -0.0008810593, -0.022622805, -0.013049115, -0.019004198, -0.017064728, 0.011832066, -0.02738687, -0.0026065689, 0.06336468, -0.02879266, 0.0010258687, 0.0006341512, -0.0030816738, -0.022635821, -0.02146433, 0.024419092, 0.020696351, 0.034155488, 0.006788143, -0.03951832, -0.025564551, 0.0067946515, -0.06227129, -0.016088486, -0.08262921, -0.030328616, 0.042642295, -0.009768939, 0.029755887, 0.01918643, -0.037149303, 0.023247601, 0.0012512181, -0.0044842097, -0.06560353, 0.02123003, -1.38809555e-05, -0.018340353, -0.05748119, -0.019316595, -0.04102824, -0.01534654, 0.02283107, 0.05019191, 0.011558717, -0.033322427, -0.0036674195, 0.0004681899, -0.006514795, -0.019251512, 0.012300662, 0.028011667, -0.0016083603, 0.009371933, -0.058418382, -0.070497766, 0.0011584751, 0.00999022, 0.03642037, -0.020240773, 0.039856747, -0.012365745, -0.033036064, -0.05219646, -0.038086496, -0.0372274, -0.011226795, -0.003927751, -0.026527777, -0.019329611, -0.042043533, -0.0006939461, -0.012821325, 0.043475356, 0.0074780216, 0.025174053, 0.010413259, 0.04501131, 0.075912654, 0.02617633, 0.0117019, -0.031187711, 0.08262921, 0.0045818337, -0.010393734, -0.013433104, -0.02043602, 0.025577568, 0.030224483, -0.06586386, 0.024926739, -0.010263569, -0.02051412, 0.00080336665, -0.07096636, -0.05352415, -0.011923182, -0.0082004415, 0.04797909, 0.014070916, -0.042407997, 0.013836618, -0.038164593, 0.03308813, 0.03954435, 0.042980727, 0.032801766, 0.011454585, 0.039257985, 0.038659222, -0.067946516, 0.040065013, 0.015554805, -0.021021767, -0.008948894, -0.023885412, 0.003153265, 0.0022323425, -0.059772108, -0.008219967, -0.031734407, -0.03678484, 0.045714207, -0.004129508, -0.0231695, -0.026944308, 0.003156519, -3.716842e-05, 0.009567182, 0.034207556, 0.062115088, 0.042225763, -0.014552529, -0.08283748, 0.046963796, -0.016270718, 0.027491003, 0.012580519, 0.0044809557, 0.024757523, -0.044126187, -0.018262252, -0.019537877, 0.025824882, -0.011474109, -0.015893236, -0.026319511, -0.021243049, -0.01797589, 0.03811253, -0.0053888615, -0.00022229867, -0.06331261, -0.00562316, -0.018847998, -0.012430828, 0.021217015, -0.010933922, 0.017689524, 0.03491045, 0.00020907872, -0.019941391, -0.01823622, -0.017077744, -0.03970055, 0.015333524, 0.0009363798, -0.03480632, 0.037123267, 0.03582161, -0.017572375, 0.022349456, -0.029313324, -0.0037585357, -0.00362837, -0.022661855, 0.037331533, -0.021789744, 0.0281158, -0.00046208836, 0.0074715135, 0.0011503397, 0.01918643, -0.0074324636, -0.0077058114, 0.022935202, 0.04516751, -0.010803756, -0.031057544, -0.028141832, 0.02477054, 0.018600684, -0.0041457787, -0.0162577, -0.05146753, 0.041965432, 0.00771232, -0.0053042537, -0.014188065, -0.018522585, -0.014422364, -0.013159756, -0.0027367347, 0.0020582457, 0.011285369, -0.015463689, 0.021373214, -0.03392119, -0.031474076, -0.02245359, -0.017676506, -0.0069118007, 0.0044842097, 0.008083292, -0.0036544031, -0.058522515, 0.045037344, -0.030953413, 0.037669964, -0.05451341, -0.027491003, -0.015359557, -0.021776728, 0.04459478, 0.021815777, 0.020383954, -0.055190273, 0.02936539, 0.002129837, -0.030172417, -0.04415222, 0.01918643, 0.024301942, 0.014227115, 0.045870405, 0.019095313, 0.023234583, 0.0028213423, 0.01116822, 0.0030247264, 0.048994385, 0.013498187, 0.02465339, -0.03324433, -0.0702895, 0.0027757843, 0.0069313254, 0.017338077, 0.016244683, -0.007412939, -0.041496836, -0.014031867, 0.017259976, 0.039179888, -9.6861615e-05, -0.0009437016, -0.02411971, 0.025590584, -0.017884772, 0.014747778, -0.032723665, 0.043293122, 0.0032102126, 0.062427487, -0.03863319, -0.0005698819, 0.018288286, 0.033582762, 0.007334839, 0.0069768834, 0.016934562, 0.0372274, -0.008714596, 0.04748446, -0.028766628, 0.0064009, -0.034884416, 0.0028669003, -0.016791381, 0.013823601, 0.0011698646, 0.03756583, 0.005151309, -0.0045297677, 0.024301942, 0.019212462, 0.026280463, -0.009821005, -0.0050536846, -0.030068286, -0.027569102, 0.024900705, -0.012326695, 0.05972004, 0.0006138128, -0.0067686182, 0.017637458, 0.02108685, 0.0020598727, -0.019238496, -0.026540793, -0.06404154, -0.031864572, -0.018301303, 0.04566214, 0.038737323, 0.01238527, -0.0104392925, -0.023247601, -0.0137324855, 0.011285369, -0.023260618, 0.020605236, -0.017520308, -0.0044386517, -0.053315885, 0.0017783893, -0.0021851573, 0.06633246, -0.05185803, 0.012808309, 0.0031239777, -0.002813207, 0.026996374, 0.01835337, 0.0034624087, -0.0026391102, 0.018847998, -0.0054409276, -0.026657943, -0.023273634, 0.008408707, -0.04436048, 0.002949881, -0.009319867, -0.024028596, -0.016140552, 0.052717123, 0.04061171, -0.025746783, 0.01166285, -0.06331261, 0.079869695, -0.0018987927, 0.011623801, -0.04847372, 0.03366086, -0.02070937, -0.023234583, 0.009651789, -0.027829435, -0.01808002, 0.023924462, 0.02013664, 0.007848994, -0.0326716, 0.0006585573, -0.011864607, -0.019251512, 0.0008647886, -0.02837613, -0.011103137, 0.0074780216, 0.0064985245, -0.0007826215, -0.0021900386, -0.0040318836, 0.010074828, 0.021203998, 0.0014155523, -0.011968739, 0.06060517, 0.010276585, 0.003517729, 0.011233303, -0.004080696, -0.02613728, -0.0010982734, 0.044803046, -0.01443538, 0.02671001, 0.021099865, 0.002834359, -0.046677433, -0.04896835, -0.015138275, 0.010556441, 0.033764992, 0.02605918, 0.017598407, 0.027595136, -0.008083292, -0.01891308, -0.00096729415, 0.009938154, 0.019238496, 0.0140188495, -0.0032818036, -0.0027822927, 0.023312684, 0.002173768, 0.09122015, 0.026644927, 0.02138623, 0.020331888, 0.022141192, -0.0064301873, -0.0084933145, -0.0059388117, -0.016531048, -0.03696707, 0.002562638, -0.04027328, -0.019316595, -0.016778363, -0.065239064, 0.0072632483, 0.008610464, -0.011714917, -0.0044972263, -0.07278868, -0.007582154, -0.0018516076, -0.029729854, -0.022193259, -0.004136016, 0.02275297, 0.018834982, 0.016153568, -0.022674872, 0.0012081007, 0.035274915, 0.025460418, -0.02382033, -0.00079075684, 0.022089126, 0.052248526, -0.0162577, 0.00317279, -0.024705457, 0.012632585, -0.023065368, 0.00885127, 0.012131447, 0.012639093, -0.029677788, 0.03954435, 0.008330607, 0.027334806, -0.030172417, -0.056648128, -0.0074519883, -0.00038439568, 0.008037734, 0.047796857, 0.002858765, -0.050582405, -0.031656306, -0.003246008, -0.008057259, 0.03152614, -0.009013977, 0.043683622, 0.017585391, 0.019811224, 0.004243403, -0.02518707, -0.01671328, 0.007367381, 0.021529414, -0.034780286, 0.018014938, -0.010588983, -0.021698628, 0.0080117015, 0.054721676, 0.046208836, -0.0071786405, -0.029990185, 0.031291842, -0.046729498, -0.03566541, 0.0325154, -0.013498187, -0.033348463, 0.032879867, -0.010758198, 0.023091402, -0.018027954, -0.010172453, -0.017793655, 0.02237549, 0.019160397, 0.019694077, 0.003787823, -0.024861656, -0.052404724, 0.0029482539, -0.0065961485, -0.031812508, -0.033062097, -0.0030540137, -0.02249264, 0.000999022, 0.059667975, -0.020605236, 0.03563938, -0.048187356, 0.030510848, 0.0050504305, 0.013667403, 0.035431113, -0.0278034, -0.02754307, -0.04407412, 0.03621211, -0.030302584, -0.0031451297, 0.031369943, -0.0077318447, -0.0130556235, 0.019160397, 0.014500463, -0.018704817, 0.054929942, -0.021373214, -0.022505656, 0.0068922755, -0.025707733, -0.033010032, 0.029131092, 0.006101519, -0.017377125, 0.00054059457, -0.016817413, -0.0045525464, 0.0029092042, -0.08184821, 0.044829078, 0.006670994, 0.012547977, -0.022844087, -0.04576627, 0.02431496, 0.00016596132, 0.03696707, -0.06081343, -0.045115445, -0.039023686, -0.013810584, -0.00082980655, 0.023494916, -0.0033420052, -0.00039517504, 0.03673277, -0.021750694, -0.015268441, 0.024874672, 0.025668683, 0.0059485743, 0.015320507, -0.03665467, 0.013172773, -0.040143114, 0.011799524, 0.025785832, -0.0008371284, 0.017962871, -0.028063733, -0.010393734, -0.010771215, 0.009807988, 0.0012878273, -0.0012243715, 0.016491998, -0.015307491, 0.013680419, -0.005063447, -0.0076993033, -0.030250518, -0.020292839, -0.00026317884, 0.035925742, 0.056023333, -0.03233317, -0.066384524, -0.027647203, 0.008363148, -0.021581478, -6.848955e-05, -0.026007114, -0.024627358, -0.05086877, 0.029105058, 0.020527137, -0.016088486, 0.0043670605, -0.02936539, 0.002020823, -0.021737678, 0.029807953, -0.0070679993, -0.02176371, 0.01215748, -0.004334519, 0.014461413, -0.03623814, -0.009007469, 0.00048405383, 0.014188065, -0.020800484, -0.008467281, -0.00911811, 0.012404795, 0.031083578, -0.0058607124, 0.018210186, 0.026866209, 0.012931966, 0.0017653728, -0.02704844, 0.0025528756, -0.0044516684, 0.021164948, -0.018926099, 0.0015156172, 0.0062837508, 0.0012878273, 0.00019860444, 0.019824242, -0.0049983645, 0.019420728, 0.010946939, 0.021945944, -0.020188706, -0.046547268, -0.004402856, 0.0104392925, -0.0064009, 0.0040286295, -0.010270077, -0.022726938, 0.0104392925, -0.032983996, 0.022349456, 0.035535246, 0.010933922, -0.018834982, 0.0019557402, -0.019394694, -0.008083292, -0.02150338, -0.009703856, 0.01075169, 0.0045069885, 0.019212462, -0.002017569, 0.0006699468, 0.0064497124, 0.034051355, -0.013309446, -0.014526497, 0.048525788, 0.015450673, -0.025981082, 0.017676506, -0.018509569, 0.03027655, 0.009410983, -0.03293193, 0.0059811156, -0.032775734, -0.0073934137, -0.0042043533, -0.028844727, 0.00431174, -0.056596063, 0.0121770045, 0.020865567, 0.0013040979, 0.048994385, 0.043162957, 0.00049788394, -0.087783776, -0.024015578, -0.0141229825, -0.03202077, 0.008096308, -0.03043275, -0.014149016, -0.015476706, 0.04152287, -0.06737378, 0.051103067, -0.03678484, 0.014526497, -0.011324419, 0.019472795, -0.066801056, -0.0003408715, 0.02443211, 0.060657233, -0.02287012, -0.031812508, -0.024900705, 0.028636461, 0.013810584, 0.01906928, -0.020149656, 0.019707093, 0.014031867, 0.033322427, 0.0077318447, -0.022857103, 0.011734441, -0.0129579995, 0.033192262, -0.02986002, -0.013849635, 0.021581478, -0.0041880826, 0.024900705, -0.0025496213, 0.003244381, -0.053472083, -0.005239171, -0.018405436, 0.0007720455, -0.021256065, 0.004851928, -0.003878939, -0.0052944915, 0.02123003, -0.009209226, 0.048161324, -0.013498187, -0.012450352, 0.011806033, -0.011493634, 0.0012105413, -0.031630274, 0.01975916, -0.0045948504, -0.021854827, 0.02055317, -0.010113878, -0.016049435, -0.021672595, -0.0016937817, 0.02005854, 0.011272353, -0.010667082, -0.031552173, -0.04532371, 0.037513766, 0.019902341, -0.025850914, -0.0064139166, -0.045115445, 0.03157821, 0.04683363, -0.0116953915, -0.045636106, 0.007152607, -0.0052586957, -0.035040617, 0.00046412222, -0.022245325, 0.06294815, -0.0073934137, -0.025252152, 0.0060038944, 0.020279821, 0.000839569, -0.0043898392, 0.0015050414, -0.0071591153, 0.001193457, -0.010660574, 0.009742905, 0.016348816, 0.018067004, -0.016609147, 0.04282453, 0.020123623, -0.0147087285, -0.040507577, -0.02290917, -0.017181877, 0.008831745, 0.0051317844, 0.0076407287, 0.005619906, 0.027517037, 0.022857103, -0.025200086, 0.07081016, -0.026280463, 0.037253432, 0.04912455, 0.037904263, 0.03043275, -0.004292215, 0.00653432, 0.003833381, 0.015021126, 0.02697034, -0.011923182, -0.020800484, 0.028350098, 0.04581834, 0.022557722, -0.0018532347, 0.012639093, 0.030250518, -0.022362474, -0.0032118396, -0.0375398, -0.026736042, -0.007491038, -0.037175335, -0.009430507, 0.02047507, -0.0035339997, -0.00071591156, 0.008063767, -0.009807988, -0.005014635, -0.027985634, 0.0050699553, -0.032957964, -0.0049267733, -0.020045523, 0.02013664, 0.005675226, -0.021151932, 0.049905542, 0.0032801765, -0.0076993033, -0.0058184084, -0.029105058, -0.03977865, 0.0044516684, 0.031968705, 0.029079026, 0.05617953, 0.01675233, -0.0072957897, -0.033634827, -0.035431113, 0.020839535, -0.049827445, 0.033530694, -0.004477701, 0.0031044527, 0.016231667, 0.05170183, 0.016791381, 0.0325935, -0.018431468, -0.007894552, 0.035040617, 0.012053347, 0.002561011, 0.01839242, 0.0020712623, 0.018574651, 0.051441498, 0.02374223, 0.00011338656, -0.022635821, 0.008311083, 0.0013187416, 0.010094353, -4.5176665e-05, -0.0015131767, -0.010517391, 0.025356285, -0.013341988, 0.016023403, -0.02249264, 0.0020598727, -0.011005513, -0.02944349, 0.011877623, -0.0065245577, 0.02374223, -0.016244683, 0.008747137, -0.0135372365, -0.014344264, -0.0018662512, 0.0059876237, 0.0014114847, -0.0031483837, -0.023807313, 0.02328665, -0.014070916, -0.011565226, 0.0030165908, 0.040377412, -0.021516396, -0.024718473, -0.022609789, -0.03821666, -0.017455226, 0.0064301873, -0.022961237, -0.009912121, -0.029131092, -0.02059222, -0.027933568, 0.00866253, -0.0034103424, 0.008961911, -0.016127534, -0.0035112207, -0.017038696, 0.016491998, 0.0070679993, -0.027152572, -0.010569458, -0.0052098837, -0.007933602, 0.017650474, -0.003449392, -0.0061112815, -0.007549613, -0.016986629, 0.010276585, 0.025095955, -0.0014700593, -0.035873678, 0.012528452, 0.0030019472, 0.02218024, -5.5218745e-05, 0.017676506, -0.008571413, 0.03829476, 0.04878612, 0.019251512, -0.00089977065, -0.022739954, 0.044308417, 0.035248883, -0.0016270718, 0.012723701, 0.0057305465, 0.021646561, -0.00037849756, 0.024861656, -0.021243049, -0.0029433728, 0.033270363, 0.0020338397, 0.0077774026, 0.008122342, -0.004266182, -0.011734441, 0.020956684, 0.019420728, 0.012007789, -0.025590584, -0.009339391, -0.0153855905, 0.043215025, -0.022622805, 0.010862331, -0.0058574583, 0.020331888, 0.05643986, -0.019472795, 0.004877961, 0.057845652, -0.020423004, 0.0032378726, -0.03670674, 0.03782616, -0.028688528, -0.03084928, 0.010992496, -0.04087204, 0.02754307, -0.0052684583, 0.023091402, -0.03621211, 0.005245679, -0.0054279114, 0.012209546, -0.017103778, 0.013628352, 0.023546983, -0.009567182, -0.0077774026, -0.0018581159, -0.003491696, 0.031474076, 0.021151932, 0.0039342595, -0.018847998, 0.030927379, -0.028271997, -0.0061861267, 0.028480263, -0.012847358, -0.0132443635, -0.019524861, -0.016244683, 0.03225507, 0.0076146955, 0.0042954693, 0.018001921, 0.030875312, -0.0027627677, 0.04334519, 0.02602013, -0.022805037, 0.0005816781, -0.02085255, -0.039257985, -0.010178961, -0.021334164, -0.022961237, 0.00076513045, 0.008525856, -0.04475098, 0.0071591153, -0.00607874, 0.033140197, -0.007419447, 0.047953058, 0.018171137, 0.053836547, -0.0038626683, 0.014227115, -0.0077318447, -0.030484816, 0.027412904, -0.030979445, -0.006602657, -0.020540154, -0.012111922, 0.0030572677, 0.0023771517, -0.024926739, -0.009293834, 0.0059355577, -0.026996374, -0.041054275, 0.03962245, 0.039310053, 0.00050154486, -0.0072502317, 0.02659286, -0.0040676794, 0.0104327835, 0.004816132, 0.011766982, -0.006260972, 0.0072762645, 0.02548645, 0.021633545, -0.0057012592, -0.032229036, -0.015021126, 0.03160424, -0.019837258, 0.019733125, -0.0004275131, 0.010022761, -0.0025447402, -0.0014643646, 0.030120352, -0.017910805, -0.012502419, 0.027360838, 0.024783555, -0.03217697, -0.007328331, 0.019277545, -0.005079718, -0.0035763036, -0.01234622, -0.009762431, -0.027334806, 0.0007915704, 0.050087776, -0.024614342, 0.012645601, 0.042954694, -0.012300662, 0.0049202647, -0.0058672205, 0.025772816, -0.048942316, 0.03168234, 0.008545381, -0.04886422, 0.0059095244, 0.02055317, 0.0070679993, 0.028688528, -0.043188993, -0.014031867, 0.0022860358, 0.007822961, 0.03696707, 0.041809235, -0.002313696, -0.004760812, 0.00767327, 0.025252152, -0.01728601, -0.008864286, 0.0108167725, 0.016804397, 0.01891308, -0.009163667, 0.034285653, -0.0011430179, -0.022609789, 0.053367954, -0.020501103, 0.010087845, 0.02188086, 0.04691173, -0.005652447, 0.020943668, -0.027517037, -0.017116794, 0.022388507, 0.01207938, 0.0030100828, -0.0047770827, 0.002133091, 0.012242087, -0.0003337531, -0.016270718, 0.008408707, -0.008473789, 0.0075235795, 0.03589971, -0.009098585, -0.02218024, 0.0041587953, -0.012411303, -0.003833381, 0.007491038, -0.0054279114, 0.024783555, 0.0026130772, -0.044829078, -0.0030751654, -0.0041457787, 0.0019362153, 0.008968419, 0.00087780517, -0.024978805, -0.00072079274, -0.031317875, -0.033973258, -0.03451995, -0.013407071, 0.037071202, -0.0146045955, -0.02738687, -0.0022583755, -0.00503416, -0.009358916, -0.022961237, -0.016348816, 0.010185469, -0.016505016, -0.008577922, -0.017116794, 0.03566541, -0.011591258, -0.0071135573, 0.004093712, -0.021269081, 6.8692934e-05, -0.032229036, -0.008213458, -0.0078164525, 0.0050471765, 0.013680419, -0.0014985331, -0.014760794, 0.017103778, -0.0012162362, 0.0068922755, 0.0018125579, -0.001996417, -0.029105058, -0.0090790605, -0.005414895, -0.0005772037, 0.011298386, -0.0062935133, 0.008148375, -0.014214098, 0.009469558, 0.031135645, 0.017962871, -0.012059855, 0.016153568, 0.002020823, -0.01310769, -0.028636461, -0.008649513, -0.025785832, 0.011409027, -0.018067004, 0.007979159, -0.0058704745, -0.03920592, 0.019889325, 0.003901718, 0.0064594746, -0.017351093, 0.0048649446, -0.008864286, 0.043891888, -0.02506992, -0.00018050327, -0.008343624, -0.0014546021, 0.026449678, 0.0162577, 0.0023787788, 0.029886054, 0.011942706, -0.026449678, -0.0016775109, -0.014448397, -0.01781969, 0.011923182, 0.022648837, -0.0068011596, 0.019589944, -0.008545381, 0.033010032, -0.0063813752, 0.013628352, -0.014175049, 0.022518672, 0.034676153, -0.045349743, -0.014292198, -0.014734762, 0.00097054825, 0.00071835215, 0.038997654, 0.0038463976, 0.0140188495, 0.011324419, -0.004432143, 0.030094318, -0.0013106063, 0.0009803107, 0.0014245013, 0.01075169, -0.0020956683, -0.05680433, 0.014539513, 1.5113971e-05, 0.00999022, 0.022648837, 0.0044386517, -0.0369931, -0.027517037, 0.0008761781, 0.006472491, 0.016986629, 0.016478982, 0.03912782, 0.00027863603, 0.029339356, 0.052248526, -0.017364109, -0.03475425, 0.0031386213, -0.016986629, 0.03540508, -0.022505656, -0.05899111, -0.00055727205, -0.0007456056, -0.035925742, -0.011682374, 0.02496579, 0.020344906, -0.04365759, 0.003586066, 0.012046839, -0.0014440261, 0.0011186118, 0.029105058, -0.0111096455, 0.006872751, -0.0012316933, -0.0021688866, -0.033556726, -0.031135645, 0.004949552, 0.0062056514, -0.021438297, 0.03264557, -0.0013887057, 0.011897149, -0.00071835215, 0.022466606, 0.004015613, 0.011766982, 0.020318871, -0.031005478, 0.018366385, 0.022310408, 0.022544706, 0.010575966, 0.011005513, -0.004087204, -0.007992176, 0.006690519, -0.0072957897, -0.033322427, 0.0002572807, 0.05154563, -0.019733125, -0.011727933, -0.043969985, 0.048265453, -0.020774452, 0.026866209, -0.00017206284, 0.016010385, 0.021399247, -0.0061145355, -0.0048649446, 0.049853478, 0.030719114, 0.022128176, -0.010400242, -0.03863319, -0.043787755, -0.0073022977, -0.0016368341, 0.03868526, 0.01811907, 0.0033908174, -0.009423999, 0.039075755, 0.016114518, 0.0004930027, 0.020722385, -0.025174053, 0.0889813, 0.030875312, -0.014982076, 0.021334164, 0.00192808, 0.0011747457, -0.024796573, -0.01447443, 0.029599689, -0.0041490328, -0.015854187, -0.018457502, 0.0025886712, 0.012997049, 0.042329896, -0.0051675797, -0.033895157, 0.010745182, -0.03373896, -0.0058607124, 0.00911811, 0.01500811, -0.011057579, 0.025082937, 0.020644287, -0.013706452, 0.017220927, 0.020279821, -0.0074064303, 0.037591863, 0.034441855, 0.021672595, -0.00065815053, -0.015554805, -0.015372573, -0.023299666, -0.039492283, 0.0037097235, -0.0045492924, -0.012190022, -0.006352088, -0.018262252, -0.0041653034, 0.002085906, -0.0089749275, -0.046130735, 0.020110607, 0.02671001, 0.02382033, -0.025668683, -0.028818695, 0.0019476048, 0.0096582975, 0.0012967761, 0.01447443, 0.007575646, 0.016557083, -0.0041490328, 0.033296395, -0.005239171, -0.006716552, -0.011181237, -0.021034783, 0.018262252, 0.007419447], "content_hash": "c4b89f3fa553a180209a4625eeada765"}, {"content": "a smooth activation function common in modern architectures. ### Grouped-Query Attention (GQA) Qwen uses **Grouped-Query Attention**, where multiple query heads share key and value heads. This reduces memory usage while maintaining quality. In the attention visualization, you may notice this affects how attention patterns are structured. ## Why Include Qwen2.5?\n\n- **Modern architecture**: Represents the latest generation of LLM design (post-LLaMA innovations)\n- **Contrast with GPT-2**: Every major architectural choice is different \u2014 PE type, normalization, activation, attention variant\n- **Good size**: 494M parameters with 24 layers provides enough depth for interesting head specialization while staying fast\n- **Open access**: No gated access or license agreement required (unlike LLaMA or Gemma)\n\n## What to Expect in the Dashboard\n\nWhen using Qwen2.5-0.5B:\n- **More layers**: 24 layers vs GPT-2's 12 means more attention patterns to explore\n- **Different tokenizer**: Qwen has a much larger vocabulary (152K vs 50K), so the same text tokenizes differently\n- **Different attention patterns**: RoPE-based attention creates different positional patterns from GPT-2\n- **Comparing with GPT-2**: Running the same prompt on both is the best way to see how architecture affects predictions and attention\n\n## HuggingFace Model IDs\n\n- `Qwen/Qwen2.5-0.5B` (in dropdown)\n- `Qwen/Qwen2.5-1.5B`, `Qwen/Qwen2.5-3B` (larger, enter manually)", "source_file": "qwen_overview.md", "chunk_index": 1, "embedding": [0.028450985, 0.02310795, 0.051423322, 0.02678298, 0.00021443336, -0.012801587, -0.016056227, 0.025833711, -0.033739775, 0.017710669, 0.0031885307, -0.02206375, -0.03607227, 0.020843262, -0.0063194274, 0.0043191793, -0.013669491, 0.0025121758, 0.00048268694, -0.0061736465, -0.040276177, -0.00030999604, -0.0057939384, 0.022294289, -0.0020273698, -0.009275726, -0.03864886, -0.007085624, 0.012543928, 0.005614255, 0.023921609, -0.016924132, -0.032492165, -0.012103195, -0.04952478, -0.0200296, -0.02893918, 0.028450985, -0.0018341256, 0.05302352, -0.009106213, -0.037211392, -0.024721708, 0.02449117, -0.0040784716, 0.00703816, 0.008095918, 0.037130024, -0.045673456, 0.036153633, -0.003990325, 0.038703103, -0.057118945, 0.038160663, 0.022294289, -0.03175987, -0.019026088, -0.014700127, -0.0010611485, 0.022334972, 0.016924132, -0.003963203, 0.012015048, 0.033848263, 0.022918094, -0.028966302, -0.07160209, 0.008143382, -0.011045436, -0.03962525, 0.017453011, 0.03986935, -0.012143878, -0.009743581, 0.040086325, 0.024165707, 0.014957787, 0.030674988, 0.011296315, -0.02880357, -0.0076619666, -0.035529826, -0.037509732, -0.05541026, -0.011113242, -0.027827177, -0.029373132, -0.037401244, -0.035882413, -0.010855583, 0.0038513248, 0.026891468, -0.028450985, -0.011845536, -0.0052243765, -0.000114315015, 0.037021536, 0.021602677, -0.007994211, 0.034119483, 0.027013518, -0.012537147, -0.018497208, -0.019934675, 0.05071815, -0.02853235, 0.019243063, -0.0001818022, -0.064713106, 0.0030393598, -0.08158299, 0.01569008, -0.019554965, 0.03270914, 0.025494685, -0.01149973, -0.016476618, -0.038323395, -0.0015323933, 0.002676603, -0.02042287, -0.0061634756, 0.06639467, -0.019148136, 0.010435192, 0.011506511, 0.012638855, -0.023365607, -0.010130069, 0.016191836, 0.014564517, 0.04420887, 0.043286722, -0.02474883, -0.018198866, -0.015568031, -0.049416292, 0.005932939, -0.08765832, -0.029020546, 0.020897506, -0.012259146, 0.028179763, 0.011845536, -0.042662915, 0.034716167, -0.022294289, 0.0129710995, -0.071385115, 0.033658408, -0.0071059656, 0.02015165, -0.036587585, -0.049660392, -0.042825647, -0.015581592, 0.007906064, 0.03528573, -0.008062016, -0.040466033, 0.024464048, 0.012062512, 0.013764418, -0.021317896, 0.03477041, 0.054027036, -0.030593622, 0.008699384, -0.047626242, -0.03178699, 0.0031580185, 0.008631579, 0.021209408, -0.022877412, 0.016463058, -0.006814404, -0.029264644, -0.016042667, -0.04936205, -0.049931612, -0.006560135, -0.013777979, 0.008529871, -0.019134576, -0.059939634, -0.029129034, -0.026850786, 0.04014057, 0.019378673, 0.0055261087, -0.0032342991, 0.037618224, 0.076484054, 0.026742298, 0.02028726, -0.05082664, 0.07664679, 0.011303096, 0.0030274938, -0.0129304165, -0.022484142, 0.014361102, 0.05101649, -0.046053164, 0.03097333, -0.008014552, -0.013425393, -0.025454002, -0.059559923, -0.032654896, 0.003583495, -0.031190306, 0.06530979, 0.009560507, -0.043476574, 0.018605696, -0.035502706, 0.013900028, 0.03458056, 0.059017483, 0.0020324553, 0.006044817, 0.060102366, 0.041876376, -0.06992053, 0.040004957, 0.017886963, -0.011709926, -0.010462314, -0.0098249465, -0.007628064, 0.02554893, -0.0419035, -0.0015646007, -0.04233745, -0.045619212, 0.05503055, -0.02407078, 0.0047768634, -0.053593084, -0.013289783, 0.013499978, 0.010726754, 0.0050514736, 0.041469548, 0.025942199, -0.013561003, -0.08017265, 0.062651835, -0.007824698, 0.019554965, 0.011533633, -0.02826113, 0.03015967, -0.016341008, 0.007967089, -0.02717625, 0.051857274, -0.023582583, -0.032871872, -0.015093396, -0.022077313, -0.01609691, 0.030729232, -0.019134576, -0.011845536, -0.04339521, 0.0031393722, -0.04746351, -0.022660436, -0.004298838, -0.0037563979, 0.028586594, 0.030349525, -0.017073302, -0.001312027, -0.004529375, -0.012530367, -0.060753293, -0.014225492, -0.009519824, -0.026959274, 0.0570647, 0.018876916, -0.013242319, 0.013269441, -0.04312399, -0.00621772, -0.010482655, 0.01583925, 0.033658408, -0.03607227, 0.05717319, -0.015649397, 0.013113489, 0.00026232065, 0.02339273, -0.03083772, 0.00094418484, 0.008990945, 0.053782936, -0.02337917, -0.026335467, -0.030078305, 0.026321907, 0.034282215, 0.029725717, 0.016951254, -0.0375911, 0.032763384, -0.014089881, -0.007546698, -0.0108488025, 0.00067381235, -0.022090873, -0.029671473, -0.009438458, 0.021995947, 0.0016332533, -0.007424649, 0.03254641, -0.03704866, -0.02108736, 0.0042784964, -0.0029223962, -0.018022573, -0.010489436, -0.0015866373, -0.00378691, -0.04599892, 0.053348985, -0.017534377, -0.0067398185, -0.03070211, -0.031705625, -0.0059397193, -0.00486501, 0.045320872, -0.0065567447, 0.04475131, -0.052942153, 0.012394757, -0.011614999, -0.02219936, -0.043340966, 0.024450488, 0.050880883, 0.0039598127, 0.04448009, -0.01040807, 0.023175754, 0.0039835447, -0.0050819856, -0.0278543, 0.045402236, 0.021724727, 0.025535367, -0.018714184, -0.05261669, 0.013554222, -0.013567783, 0.0140356375, 0.0178734, 0.0005886323, -0.020124529, 0.0065804766, 0.015052713, 0.05733592, 0.016652912, -0.008062016, -0.011106461, 0.0069127213, -0.00864514, -0.026145613, -0.03409236, 0.025359076, -0.00060431217, 0.050094344, -0.01611047, 0.019270185, 0.010231776, 0.030729232, -0.010292801, -0.011201388, 0.028857814, 0.018605696, -0.0054379622, 0.050636783, -0.04694819, -0.027393226, -0.0054752547, -0.006288915, -0.017385205, 0.007695869, -0.014713688, 0.03962525, 0.029834205, -0.004488692, 0.027989909, 0.020911066, 0.015730763, -0.0073771854, -0.009696117, -0.046053164, -0.0023596145, 0.011540414, -0.009099433, 0.0527523, 0.0119404625, -0.007072063, 0.0319226, 0.01366271, 0.0029393474, 0.005536279, -0.015391738, -0.040357545, -0.028288253, -0.010286021, 0.031895477, 0.042554427, -0.023826681, -0.016341008, -0.024233513, -0.0037123244, -0.0059566703, -0.03504163, 0.027908543, -0.03688593, 0.019771943, -0.032085333, -0.024518292, 0.018754866, 0.04057452, -0.03688593, -0.0014018686, -0.006563525, -0.0074043074, 0.023501217, 0.010197874, -0.0017510644, -0.002285029, -0.0029240912, 0.0048209364, -0.00062762015, -0.038079295, -0.0076348446, -0.04879249, -0.04108984, -0.013506759, -0.031298794, -0.012198122, 0.042039108, 0.041333936, -0.022890972, -0.006749989, -0.044941165, 0.08158299, -0.008970603, 0.011418364, -0.052725177, 0.04298838, -0.03935403, -0.017737791, 0.019866869, -0.017941207, 0.011506511, 0.019026088, 0.018741306, 0.013886467, -0.015988423, 0.010760656, -0.010028361, -0.01217778, -0.013642369, -0.038757347, -0.01865994, -0.0046988875, 0.0077704545, 0.006007524, -0.02826113, 0.0025325173, 0.0149984695, 0.03447207, 0.021684043, 0.016015545, 0.080986306, -0.006868648, -0.0129304165, 0.034797534, 0.010218215, -0.041686524, -0.009309628, 0.05277942, -0.015812129, 0.024138585, 0.012360854, 0.017819157, -0.050880883, -0.029698595, -0.027379666, 0.0015730764, 0.0378352, 0.013574564, 0.011913341, 0.015337494, -0.012747343, 0.005176913, 0.0009611361, 0.012618513, 0.012767684, 0.012855831, -0.022470582, -0.021602677, 0.04071013, 0.0032020917, 0.08733286, 0.036235, 0.011709926, 0.029725717, 0.044046137, -0.009540166, -0.011791292, -0.009235043, -0.008197626, -0.03582817, -0.019297307, -0.04014057, -0.030051183, -0.014144126, -0.05340323, 0.016408814, 0.0034004215, -0.023704633, -0.016381692, -0.084566414, 0.027393226, -0.00392252, -0.039706618, -0.026186297, 0.017236035, 0.008760408, 0.016951254, 0.01257783, -0.027420348, -0.0048311073, 0.039164174, 0.031434406, -0.021643361, 0.002017199, 0.015310372, 0.05071815, -0.037943687, 0.015595153, -0.0011772646, 0.0037530076, -0.0061329636, -0.001034874, 0.010774217, 0.004349692, -0.02732542, 0.0484399, -0.0036411292, 0.033658408, -0.055247527, -0.057824116, 0.032763384, 0.022633314, -0.0055905236, 0.037672468, -0.0006102451, -0.0319226, -0.028559472, -0.01203539, -0.020382186, 0.011587877, 0.014496712, 0.018361598, 0.02839674, 0.0031987014, 0.004031008, -0.020775456, 0.003325836, 0.008089138, 0.018185304, -0.026064247, 0.0012416793, -0.0044751307, -0.009621532, -0.0020104186, 0.04871112, 0.044669945, -0.026593126, -0.021819653, 0.037238512, -0.033143092, -0.03748261, 0.02259263, -0.014374663, -0.05896324, 0.028125519, 0.00095944095, 0.017968329, -0.026349029, -0.001893455, -0.023853803, 0.034526315, 0.008407822, 0.008340017, 0.008699384, 0.0022053581, -0.0557086, 0.0018849794, -0.032763384, -0.013764418, -0.04759912, 0.0058176704, -0.018103939, 0.004980278, 0.04488692, 0.010699632, 0.03420085, -0.031895477, 0.033902507, 0.02527771, 0.022524826, 0.037374124, -0.03246504, -0.021968825, -0.0159613, 0.027284738, -0.05356596, -0.00063185795, 0.03029528, 0.010706412, -0.008902798, 0.037211392, 5.67867e-05, -0.018836234, 0.043205354, -0.030756354, -0.019758381, 0.0077162106, -0.02744747, -0.006719477, 0.018348036, -0.0044141063, -0.019690577, -0.010557241, 0.006977136, -0.0167614, -0.025237026, -0.07203605, 0.036587585, 0.012591391, -0.0009874105, -0.037238512, -0.04236457, -0.0009908008, -0.0015095092, 0.05532889, -0.057824116, -0.023162194, -0.016856326, -0.028749326, -0.022416338, 0.041849256, -0.011526852, -0.007234795, 0.00527523, -0.0045666676, -0.025942199, 0.020911066, 0.039679494, 0.009628312, 0.026145613, -0.025345514, 0.014754371, -0.041035596, -0.0056481576, 0.026484638, -0.007736552, 0.010374167, -0.036045145, -0.006641501, 0.017968329, 0.010421631, 0.009167238, 0.0098452885, 0.031976845, 0.0012145573, 0.016883448, -0.0020070283, 0.017764913, -0.030647866, -0.015852813, -0.0062007685, 0.02717625, 0.06389944, -0.03043089, -0.041713644, -0.015323933, 0.010394509, -0.0016451192, -0.007099185, -0.011553975, -0.021290774, -0.025996443, 0.03138016, 0.033766896, -0.0140491985, -0.009255384, -0.015405299, -0.0030088476, -0.018225988, 0.008597676, -0.008909578, -0.026688054, 0.020178773, 0.0043191793, 0.0059532803, -0.022511264, 0.008021333, -0.008394261, 0.022077313, -0.0074924543, -0.017547937, 0.010360606, -0.0036513, 0.008740067, 0.0048751803, 0.00432257, 0.04193062, 0.01555447, -0.02366395, -0.03175987, 0.003939471, -0.0064380863, 0.013981394, -0.03070211, -0.004966717, 0.012618513, 0.0046988875, -0.013086367, 0.018022573, 0.0019392234, 0.025033612, 0.013750857, 0.013635588, -0.037455488, -0.051477566, 0.018239548, 0.011791292, 0.0039293007, 0.011994707, 0.001575619, -0.022741802, 0.023880925, -0.025087856, 0.013432173, 0.03607227, 0.009791044, -0.012252366, -0.00987241, -0.012211683, -0.01014363, -0.023880925, 0.002649481, 0.0024375902, 0.019270185, 0.031353038, 0.0011060693, -0.009146896, 0.020382186, 0.042120475, 0.01959565, -0.058908995, 0.03528573, -0.008624798, -0.007919625, 0.018470086, -0.011174266, 0.014808615, 0.005349816, -0.021385701, -0.0016417289, -0.022945216, -0.0019324429, -0.017371645, -0.0292104, 0.0023867365, -0.032356553, 0.019310867, -0.003122421, -0.021982385, 0.03515012, 0.035095874, 0.0018697233, -0.08749559, -0.026755858, -0.0046887165, -0.03531285, -0.00015584558, -0.013554222, 0.012733782, -0.039543886, 0.04342233, -0.05153181, 0.0391913, -0.029942695, 0.012693099, -0.01663935, 0.009940215, -0.058258068, -0.009567288, 0.020734774, 0.076321326, -0.022375654, -0.025942199, -0.012360854, 0.016476618, 0.015188323, -0.015188323, -0.0025986272, 0.01446959, 0.023880925, 0.028640838, 0.0053735473, -0.018334476, 0.010177532, 0.015391738, 0.028776448, -0.0038106418, -0.020110967, 0.008489188, 0.015459543, 0.024789514, -0.0011314962, 0.014225492, -0.05858353, -0.00933675, -0.02069409, -0.00041043223, 0.0048989123, 0.011594658, -0.0048853513, -0.018307354, 0.03300748, -0.007831479, 0.031841233, -0.0036072268, 0.017086864, 0.015879935, -0.012184561, 0.017141107, -0.012150658, 0.0140220765, -0.002825774, -0.02055848, 0.017480133, -0.007099185, -0.01217778, -0.011791292, -0.013805101, 0.032003965, 0.012211683, 0.0021307725, -0.029454498, -0.04003208, 0.029969817, 0.008238309, -0.023297803, 0.0071602096, -0.039516762, 0.03447207, 0.041279692, 0.013032123, -0.023311364, 0.010591143, -0.019988919, -0.03325158, 0.02610493, -0.0028342495, 0.06031934, 0.017547937, -0.019541405, 0.015432421, 0.017697109, -0.007804357, -0.023609705, -0.00087722734, 0.0083807, -0.0098249465, -0.006112622, 0.0020443213, 0.017968329, 0.032790504, 0.0023291023, 0.052698057, 0.00050726626, -0.0025782858, -0.058800507, -0.031163184, -0.014089881, 0.0078111375, -0.012591391, -0.00096367876, 0.0032953236, 0.02421995, 0.015527348, -0.011269193, 0.052291226, -0.016408814, 0.026281223, 0.039245542, 0.03354992, 0.027149128, -0.015432421, 0.026430395, -0.003475007, 0.02758308, 0.01974482, -0.014117004, -0.031081818, 0.043368086, 0.03609939, 0.016734278, 0.0047531314, -0.0063465494, 0.024436926, -0.014442468, 0.00037313945, -0.027216932, -0.02596932, -0.0026477857, -0.021168726, -0.00662455, 0.017236035, -0.0033512628, -0.0066855745, 0.0049870587, -0.015649397, -0.013954272, -0.027203372, -0.0076890886, -0.023677511, 0.011825195, -0.015785007, 0.04296126, -0.0036479097, -0.0098249465, 0.030458013, 0.0072686975, 0.0305665, -0.003966593, -0.020070285, -0.036858805, 0.022958778, 0.025087856, 0.019582087, 0.031190306, 0.007912845, 0.014266175, -0.02232141, -0.043991894, 0.014401785, -0.026891468, 0.032112453, -0.0045225946, 0.00946558, 0.03826915, 0.045646336, 0.020205894, 0.03371265, -0.020219455, -0.02743391, 0.020707652, 0.0150120305, -0.0053701573, 0.031841233, -0.003936081, 0.025264148, 0.06536403, 0.02286385, 0.0048209364, -0.027691567, 0.010184313, 0.015730763, 0.0017103815, 0.02393517, -0.0031681892, -0.011560755, 0.020802578, 0.0019985528, 0.024898002, -0.016313886, 0.0023307973, -0.016815644, -0.017547937, 0.024518292, -0.0036275683, 0.013764418, -0.02488444, 0.029834205, 0.013405051, -0.033115968, -0.0020934797, -0.0053464253, -0.019270185, -0.001467131, -0.024260635, 0.037428368, -0.019785503, 0.006116012, -0.009262165, 0.0025715053, -0.013303344, -0.021128042, -0.034960266, -0.029427376, -0.04491404, 0.003936081, -0.010530119, -0.005614255, -0.017602181, -0.021955263, -0.02286385, 0.028478106, -0.0032105674, -0.0029546034, -0.025860833, -0.006536403, -0.014252614, 0.016273204, 0.0026477857, -0.026186297, -0.01920238, 0.02055848, 0.009940215, 0.031000452, -0.0056481576, 0.014523834, -0.007892503, -0.01514764, -0.0011052217, 0.010313143, -0.011038656, -0.01636813, -0.0076890886, -0.0047768634, 0.0055125477, -0.011587877, 0.016218958, -0.01729028, 0.038350515, 0.052670933, 0.008448505, -0.0066991355, -0.0022612973, 0.05831231, 0.016978376, -0.0034411044, 0.016612228, -0.0035360316, 0.0019256624, 0.010482655, 0.01352032, -0.009357092, -0.016557984, 0.024423366, 0.0020087236, 0.0009145201, -0.001170484, 0.00878753, 0.00527523, 0.0073636244, 0.03677744, -0.010658948, -0.026986396, -0.0040411786, -0.010218215, 0.045049652, 0.0009645263, 0.00635672, -0.022023069, 0.013513539, 0.07154785, -0.010123288, 0.004980278, 0.03829627, 0.0020070283, -0.008529871, -0.02651176, 0.023284242, -0.03298036, -0.040357545, 0.008265431, -0.047924586, 0.026186297, 0.0010450447, 0.0335228, -0.051287714, 0.006990697, 0.014713688, 0.014944225, -0.025101416, 0.019365111, 0.027989909, -0.018076817, -0.0075127957, 0.007234795, 0.00029389237, 0.04342233, 0.026999956, 0.013208416, -0.020409308, 0.009350311, -0.029779961, -0.015934179, 0.031461526, -0.020721212, -0.012754123, 0.0021206017, -0.035773925, 0.0064753788, -0.01649018, -0.0018866746, 0.002313846, 0.028613716, 0.015120518, 0.04976888, 0.02961723, -0.023772437, -0.010035142, -0.049985856, -0.035367094, -0.0017205522, -0.016964814, -0.028071275, 0.015907057, 0.016978376, -0.040764373, 0.021711165, -0.0037360562, 0.020463552, -0.013093148, 0.03962525, -0.008272212, 0.06661165, -0.0058041094, -0.0042649354, -0.01905321, -0.028315375, 0.030756354, -0.004678546, -0.005071815, -0.011974365, -0.0316785, 0.0035021289, 0.008150163, -0.017778475, -0.0037089342, 0.00060685485, -0.009614751, -0.041496668, 0.028586594, 0.029590108, 0.0055939136, -0.0083807, 0.029779961, 0.008536651, 0.021561995, 0.014266175, 0.0064787692, 0.013940711, 0.01366271, 0.014930664, 0.012394757, -0.028559472, -0.03458056, -0.019568527, 0.008495968, -0.015812129, 0.024599658, -0.008217968, 0.018754866, 0.0036275683, 0.0073229414, 0.036479097, -0.005865134, 0.00048946746, 0.018632818, 0.0129914405, -0.028830692, -0.0098385075, 0.038621735, -0.0024308097, -0.011655682, 0.00056956214, 0.015893495, -0.007878942, -0.006102451, 0.050663907, -0.031597137, 0.011567536, 0.04236457, 0.00811626, -0.003953032, -0.0005398974, 0.021277213, -0.025331954, 0.02366395, 0.0029240912, -0.07165634, 0.005732914, 0.025847271, -0.0146052, 0.024843758, -0.04787034, -0.016178276, 0.008902798, -0.005285401, 0.033929627, 0.019826187, -0.0062787444, 0.010706412, 0.017602181, 0.047544874, -0.012699879, -0.0013959358, -0.0008645139, 0.017846279, 0.008604457, -0.0018629427, 0.040276177, -0.00554984, -0.0221858, 0.054976307, -0.012625294, 0.017100425, 0.014415346, 0.050148588, -0.0019714308, 0.009818166, -0.022701118, -0.01283549, 0.012211683, 0.0049836687, -0.0018205646, 0.003071567, -0.006505891, 0.015242567, 0.007750113, -0.011574316, 0.0037157147, -0.008014552, 0.028288253, 0.027637323, -0.0023630047, -0.013472856, -0.0031698844, -0.017819157, -0.0032851528, 0.002110431, 0.012591391, 0.0062007685, -0.0035767145, -0.047083803, -0.0074856738, -0.0070449407, 0.006936453, 0.010753876, 0.0017527596, -0.017466571, 0.0077297715, -0.033929627, -0.04342233, -0.031163184, -0.01974482, 0.039842226, 0.008502749, -0.026213419, -0.0047090584, 0.016476618, -0.0100148, -0.016964814, -0.033495676, 0.018727744, 0.0054142303, 0.011886219, -0.027379666, 0.00987241, -0.008441724, -0.017385205, 0.020219455, -0.0059736217, 0.0075263567, -0.028234009, -0.013099928, -0.024016537, 0.0052514984, 0.009791044, 0.012774465, -0.021426385, 0.019229501, 0.007933186, -0.008604457, -0.018293792, -0.014496712, -0.024938684, -0.0389472, 0.010984412, 0.016585106, 0.015947739, -0.0026969444, 0.0016417289, -0.0053362544, 0.0073907464, 0.05652226, 0.013405051, -0.012923636, 0.0037089342, -0.009099433, -0.015106957, -0.015486665, 0.019500721, -0.016029105, 0.03775383, -0.0189176, 0.008740067, -0.007722991, -0.0357468, 0.00635672, 0.0043598623, -0.008136602, -0.01582569, 0.0025342123, -0.018822672, 0.020355064, -0.03571968, 0.016598668, -0.019785503, 0.019541405, 0.058203824, 0.003610617, 0.011547194, 0.038594615, 0.0057939384, -0.021819653, -0.016381692, -0.024464048, -0.016964814, -0.00798065, 0.011411584, -0.011859097, 0.011974365, -0.009818166, 0.025237026, 0.010577582, 0.0036987634, -0.011370901, 0.007139868, 0.040113445, -0.019405795, -0.0050006197, -0.007397527, 0.008007772, -0.0010526729, 0.029671473, -0.0057566455, 0.023365607, 0.00419035, -0.0108149, 0.017263157, 0.00031804788, -0.000571681, -0.00432257, 0.0003360586, -0.0029698596, -0.054216888, 0.00041361057, -0.010692851, 0.009757142, 0.01176417, 0.0009857154, -0.030810598, -0.015120518, 0.0056854505, 0.010930168, 0.030729232, -0.013276222, 0.03854037, 0.0062143295, 0.022104435, 0.02339273, -0.01879555, -0.029996939, 0.0077297715, -0.020178773, 0.03883871, -0.02743391, -0.066231936, -0.036235, -0.00083061145, -0.016300326, -0.002081614, 0.017113985, 0.024165707, -0.037401244, 0.0031020795, -0.0030800428, 0.011987926, 0.0068211844, 0.042798527, -0.0027359324, 0.0035156899, 0.002637615, 0.0068076234, -0.041686524, -0.047165167, -0.0023630047, 0.013201636, -0.010469094, 0.043340966, 0.0024850538, 0.016788522, -0.012259146, 0.007600942, 0.024952246, 0.020897506, 0.024952246, -0.0229859, 0.020395748, 0.03205821, 0.02975284, 0.009024847, 0.02055848, -0.00037102055, -0.0051226686, 0.00081662665, -0.012849051, -0.02717625, 0.009533385, 0.043096866, -0.023229998, -0.018456524, -0.039733738, 0.033848263, -0.009580849, 0.011642121, -0.008923139, 0.0004954004, 0.034553435, -0.0054989867, -0.00621772, 0.05966841, 0.031976845, 0.017954769, -0.01583925, -0.03840476, -0.027122006, -0.029915573, 0.0032105674, 0.03341431, 0.0029681644, 0.009750361, -0.017059742, 0.04542936, 0.010197874, -0.01189978, 0.02556249, -0.029671473, 0.06818472, 0.008556993, -0.02230785, 0.021819653, 0.014347541, -0.021182286, -0.0065160617, 0.0011314962, 0.047138046, -0.013642369, -0.014428907, 0.013683052, -0.032437917, 0.030485135, 0.037970807, -0.008340017, -0.050229955, -0.015242567, -0.04922644, -0.017113985, 0.015459543, 0.040899985, -0.011865878, 0.006071939, 0.019771943, 0.010116508, 0.02853235, 0.015079835, 0.001135734, 0.02732542, 0.036479097, 0.015540909, -0.013316905, -0.0038852273, 0.0054583037, -0.022945216, -0.03433646, -0.010157191, 0.016313886, -0.014578078, -0.009160457, -0.00027185574, -0.014266175, -0.022538386, 0.0062685735, -0.015188323, 0.0054277913, 0.022538386, 0.014727249, -0.011370901, -0.0040276176, 0.014510273, 0.0048887413, -0.012449001, 0.018171744, 0.007906064, 0.013140611, -0.013750857, 0.027338982, -0.013805101, -0.014496712, -0.015527348, -0.015066274, 0.032085333, -0.0005034522], "content_hash": "dd5db005a0bb8f1f457daa65832e25e9"}, {"content": "# RAG Documents\n\nThis folder contains documents used by the AI chatbot for Retrieval-Augmented Generation (RAG).\n\n## Supported File Types\n\n- `.txt` - Plain text files\n- `.md` - Markdown files\n\n## How to Add Documents\n\n1. Place your documentation files in this folder\n2. Delete `embeddings_cache.json` if it exists (to force re-indexing)\n3. The chatbot will automatically index new documents on startup\n4. Documents are chunked and embedded for semantic search\n\n## Document Inventory\n\n### Category 1: General LLM/Transformer Knowledge\n- `what_is_an_llm.md` - Neural networks, language models, next-token prediction\n- `transformer_architecture.md` - Layers, encoder/decoder, residual stream\n- `tokenization_explained.md` - Subword tokenization, BPE, token IDs\n- `embeddings_explained.md` - Lookup tables, vector spaces, positional encodings\n- `attention_mechanism.md` - Q/K/V, multi-head attention, intuitive explanations\n- `mlp_layers_explained.md` - Feed-forward networks, knowledge storage, expand-compress\n- `output_and_prediction.md` - Logits, softmax, temperature, greedy vs. sampling\n- `key_terminology.md` - Extended glossary of ML/transformer terms\n\n### Category 2: Dashboard Components\n- `dashboard_overview.md` - Layout tour, navigation, typical workflow\n- `pipeline_stages.md` - What each of the 5 pipeline stages shows\n- `ablation_panel_guide.md` - How to use the ablation experiment panel\n- `attribution_panel_guide.md` - How to use the token attribution panel\n- `beam_search_and_generation.md` - Beam search, generation controls\n- `head_categories_explained.md` - Previous-Token, Positional, BoW, Syntactic, Other\n- `model_selector_guide.md` - Choosing models, auto-detection, generation settings", "source_file": "README.md", "chunk_index": 0, "embedding": [-0.007225251, 0.02540817, 0.026404321, -0.02410939, -0.0081961835, -0.058356825, -0.004460615, 0.009394087, 0.00027681823, 0.022344057, 0.0074270032, -0.033793494, -0.003767092, -0.021738801, -0.0072063366, 0.035886675, -0.024878569, 0.00032922652, -0.0021105174, 0.045696877, 0.04476377, -0.014097435, 0.013946122, 0.018989926, -0.023920247, -0.014324407, 0.01178359, 0.033238675, -0.022924095, -0.018813394, 0.011241381, -0.019721279, -0.011039629, -0.022381885, -0.027034797, 0.024008512, 0.015358387, 0.02045263, 0.0026543024, -0.022255791, -0.002908069, -0.042821907, -0.013088674, 0.042115774, -0.029859327, 0.0054882905, 0.008183574, 0.008460984, 0.0047474816, 0.017249813, -0.027413081, -0.0111909425, -0.018056823, -0.00021574089, 0.008082698, 0.018775566, -0.05048849, -0.031549003, -0.004126463, -0.021827066, 0.00408233, -0.0062480136, 0.017527223, 0.021133544, -0.0022476458, -0.034575284, -0.084534176, 0.027110454, -0.032683857, 0.0047348724, 0.069554076, -0.022419713, -0.02668173, -0.012187094, -0.010692867, 0.005759395, -0.029808888, 0.02769049, -0.03240645, -0.032507326, -0.0027378406, 0.01581233, -0.02665651, -0.032255135, -0.025975596, -0.010547858, -0.025660358, -0.0051635955, -0.039064273, -0.026984358, -0.061030045, 0.016795872, -0.041031357, 0.009047326, 0.016140176, -0.0019119175, -0.03495357, -0.018182918, -0.001383106, 0.016379757, 0.008631212, -0.05674281, 0.009576925, 0.0052361004, 0.04998411, 0.009885859, 0.004898796, -0.01893949, -0.014879226, -0.04998411, -0.089678854, -0.026706949, -0.0036441493, 0.03159944, 0.020629164, -0.011159419, -0.01296258, 0.0067776134, 0.0054788333, -0.015194464, -0.06183705, -7.265247e-05, 0.008517726, -0.02073004, 0.010535249, -0.004375501, 0.0030120974, -0.022684515, -0.05850814, 0.029405385, 0.026505196, 0.023983294, 0.019998688, -0.06451027, -0.03601277, -0.026580853, -0.02278539, 0.02819487, -0.054775726, 0.02723655, 0.030691555, -0.016190615, -0.03752591, 0.045747314, -0.068696626, -0.0047380244, -0.010270448, 0.007017194, -0.029203633, 0.0025486979, -0.06335019, 0.0025865263, -0.0068784896, -0.010339801, -3.47993e-05, -0.025698187, 0.0007786374, 0.035987552, 0.008246622, -0.0544731, -0.0030451973, 0.039795622, 0.008580774, -0.014715302, -0.008624907, 0.0053464337, 0.00036193244, -0.016114958, -0.031195935, -0.022407105, -0.002083722, -0.055784486, -0.023932856, -0.031322032, -0.013315646, -0.00946344, -0.03316302, -0.027261768, 0.015862767, 0.018233355, -0.030338489, 0.005150986, -0.05179988, -0.01654368, -0.04375501, -0.029430604, 0.015824938, -0.0170985, -0.028699253, 0.020553507, 0.017401129, 0.021020059, 0.05850814, 0.0005142317, 0.016997624, -0.00061707804, 0.03593711, -0.03109506, 0.05127028, -0.018258575, -0.0034928352, 0.007943993, 0.011638581, -0.029632356, -0.033364773, 0.0060115852, -0.04193924, 0.038030293, -0.03681978, -0.06062654, 0.006500204, -0.01659412, 0.020339144, 0.042267088, -0.07636321, -0.022949314, -0.05331302, 0.047487427, -0.0053874147, -0.0025959834, -0.008051174, 0.0058602714, 0.025761236, 0.016568901, 0.0028434452, 0.04057741, -0.008479898, -0.059769094, 0.046428226, 0.0051194625, 0.027589615, 0.03876164, -0.031927288, 0.010926143, 0.06324932, -0.032759514, -0.010453287, -0.0015178701, -0.0018425651, 0.02535773, 0.041560955, 0.006783918, 0.007597232, 0.0075278794, 0.024222875, -0.0070550228, -0.044839427, -0.05432178, 0.07500138, 0.016808482, 0.044082858, -0.0027867025, 0.04660476, -0.008637517, -0.0035243588, 0.0136939315, -0.034121342, 0.009917382, -0.044461142, 0.011424219, -0.02874969, 0.05694456, -0.048269216, 0.004252558, -0.00025632777, 0.017211985, -0.018510764, 0.0027851262, -0.046428226, -0.015673624, -0.039745186, 0.022117086, 0.018989926, 0.017590271, -0.06794006, -0.007609841, -0.023100628, -0.008290755, -0.0075720125, 0.06037435, 0.015257511, 0.007609841, 0.004208425, 0.04786571, -0.0062480136, 0.039518215, 0.035836235, 0.021083105, 0.009803897, -0.024210265, 0.009179725, -0.0019292556, 0.042065337, -0.046705637, -0.019494308, 0.018245965, 0.06244231, 0.020263487, 0.031902067, 0.014563988, -0.0012822299, -0.00789986, 0.020364365, 0.010257839, 0.026026035, 0.049404074, -0.014917054, -0.032280352, 0.008940144, -0.037702445, 0.0064875945, -0.01659412, 0.0018961555, -0.043679353, -0.028270528, -0.012382542, 0.05437222, 0.032936048, 0.04615082, -0.01114681, -0.011960124, -0.022394495, 0.06284581, -0.048017025, -0.046453446, -0.01608974, -0.007830508, 0.031246373, -0.05386784, -0.019708669, 0.06814181, -0.021373125, 0.025080321, 0.038131166, -0.01477835, -0.041863583, 0.008732088, 0.030893307, -0.056843687, -0.0026275073, -0.024878569, -0.013542617, -0.04378023, -0.00011043175, -0.030439364, 0.00065451255, -0.018687299, 0.07066371, 0.007943993, 0.0168337, 0.00041847822, 0.01088201, -0.017476786, -0.0131895505, -0.031649876, 0.008126832, -0.053212147, -0.03467616, -0.004791615, -0.00044212106, -0.028018339, 0.014387455, 0.044057637, 0.012868009, 0.012741913, -0.013227379, -0.027564395, 0.007666584, 0.040073033, -0.013643493, -0.0042115776, -0.029657574, -0.020578725, -0.0013681322, -0.06314844, -0.009558011, -0.01994825, -0.002416298, -0.010900924, 0.05492704, -0.01887644, -0.022079257, -0.00413592, -0.008442069, 0.04146008, 0.00023800456, -0.0011474657, 0.013782198, -0.002975845, 0.06657823, -0.042090554, 0.007969213, -0.025824282, -0.021360515, 0.009873249, 0.03391959, -0.020868745, -0.027766148, 0.03603799, -0.009816506, 0.04423417, -0.008397936, -0.035861455, 0.00065451255, 0.017955946, 0.03391959, 0.060223036, 0.055431418, 0.008940144, -0.04113223, 0.040552195, -0.010976581, 0.0047380244, -0.0068406607, -0.021045277, 0.015080978, 0.028547937, 0.04998411, -0.0022460695, 0.011430523, -0.011247686, -0.06637648, -0.033036925, -0.025572093, 0.0014366964, 0.026303444, 0.05881077, 0.0016423891, 0.025609922, -0.036542367, -0.031221155, -0.008360107, 0.04168705, -0.014147874, -0.0077044126, -0.022772782, 0.02950626, 0.027312206, -0.0061124614, -0.011846637, 0.03265864, -0.013946122, -0.03500401, -0.01888905, -0.051371157, -0.022192743, -0.025282074, -0.015597967, 0.008076393, -0.031927288, -0.018031605, 0.00621649, -0.0014784654, -0.06239187, -0.011127896, -0.037349377, 0.054271344, 0.017892899, 0.051648565, -0.010075001, 0.0393669, 0.038610328, -0.01996086, 0.02356718, -0.0008700564, -0.039795622, -0.05154769, -0.005781462, 0.020263487, -0.029380165, -0.014891835, -0.0320786, -0.03679456, -0.0084483735, -0.027034797, -0.035659704, -0.017930727, -0.048017025, 0.04274625, -0.027791368, 0.039266024, -0.0049114055, -0.017741585, -0.017968556, 0.016404977, 0.093411274, 0.004132768, -0.019090803, 0.04839531, -0.0044038724, -0.029203633, -0.013315646, 0.023151066, -0.013151722, 0.036618028, -0.0010662919, 0.008896012, -0.08912404, 0.006708261, -0.024412017, -0.027715709, 0.0051446813, 0.008410545, 0.042670593, -0.015585358, 0.009425611, -0.027135672, 0.016165396, 0.009885859, 0.008259231, -0.006525423, -0.038232043, 0.014538769, 0.034197, 0.014198312, 0.010900924, -0.0025108694, 0.029203633, -0.011556619, -0.001774789, 0.009911077, -0.025950378, 0.01711111, -0.009904773, 0.006897404, -0.005516662, -0.003116126, -0.062240556, -0.028396623, 0.027085233, 0.043830667, -0.044158515, 0.019015146, 0.011594447, -0.04118267, -0.003823835, -0.016140176, 0.023264552, -0.0070298035, -0.009249078, -0.015459263, 0.024689427, 0.03581102, -0.024727255, 0.009230164, 0.00413592, -0.005891795, -0.0012711965, 0.01398395, 0.022974532, 0.056339305, -0.006739785, 0.035987552, -0.037601568, 0.010724391, -0.0035022923, 0.02149922, -0.03876164, 0.028220091, -0.019406041, -0.001257799, -0.025282074, -0.023605008, 0.011657495, -0.01840989, 0.035079665, -0.004470072, 0.0034518542, -0.003250102, -0.00408233, 0.021713581, 0.005409481, -0.0047285673, 0.021801848, 0.036189303, -0.02716089, -0.01475313, -0.017010232, -0.019330384, 0.015471873, -0.028573157, -0.041107014, -0.044158515, 0.009658887, -0.03591189, 0.017489394, 0.028093996, 0.016758043, 0.005586014, -0.010472201, 0.05179988, 0.006169204, -0.0073198224, -0.01786768, -0.009999344, -0.00893384, 0.0007014042, 0.0010623515, -0.02408417, 0.027286986, 0.008057479, -0.0020979077, -0.03729894, 0.0033446734, 0.017275034, -0.019847374, -0.030868089, 0.0026212025, -0.015320558, 0.014715302, -0.0044795293, -0.013164332, -0.040426098, -0.05104331, -0.011241381, 0.012603208, 0.014614426, 0.016291492, 0.002966388, -0.018989926, -0.011897076, 0.002074265, 0.00789986, -0.015698845, 0.016934576, 0.049404074, -0.01581233, -0.05185032, -0.043099318, -0.018964708, -0.029153194, 0.031422906, 0.04504118, -0.01578711, -0.029481042, 0.013971341, -0.0018204985, -0.020150002, 0.022079257, -0.0046718246, 0.03162466, 0.025483826, -0.0015651558, -0.023970684, 0.049958892, -0.032910828, -0.00046458174, 0.014740521, -0.0043376726, 0.012117743, -0.0336674, -0.033768278, -0.010081306, 0.004867272, 0.044864647, -0.021461392, 0.02308802, -0.012496027, 0.010800048, 0.050538927, -0.033263896, 0.007893555, -0.0057657, -0.020780478, -0.02824531, 0.08645082, -0.016795872, 0.026858263, -0.008227707, 0.019204289, -0.03318824, -0.03235601, 0.005333824, 0.0024021124, 0.016291492, -0.0011979038, -0.018346842, 0.02020044, 0.033238675, 0.011462048, -0.0063867182, -0.022722343, -0.03520576, -0.042065337, 0.033818714, -0.04889969, 0.05573405, -0.009091459, 0.017640708, 0.009431915, 0.008568164, -0.009545402, -0.027640052, -0.014122655, -0.025004664, 0.08342454, 0.007887251, 0.0033478257, 0.009759763, -0.06738524, -0.012464504, 0.004659215, 0.00075144815, -0.0041958154, -0.024374189, -0.010106524, -0.0051856623, 0.0037765491, 0.014816178, -0.07020977, -0.022419713, 0.008379022, 0.0068028322, 0.01450094, -0.020440022, 0.0022318838, -0.027261768, 0.009009497, 0.0063961754, -0.04113223, -0.03548317, 0.023743713, -0.030237613, -0.010831572, -0.027513959, -0.033137802, -0.013769588, 0.0049051004, -0.04277147, -0.00082355883, -0.00047128057, 0.038383357, -0.025509045, 0.003849054, 0.0030451973, 0.019229507, -0.05356521, 0.0077674603, -0.033062145, -0.011764675, -0.003584254, -0.031069841, -0.0110270195, 0.022583637, -0.013265207, -0.010220011, -0.0014177822, 0.036542367, -0.036113646, 0.023844589, 2.8568427e-05, 0.00969041, -0.002558155, -0.039240804, 0.008334888, 0.03591189, 0.01605191, -0.027337424, -0.00010856003, 0.019569965, -0.018334232, 0.03707197, 0.0068721846, 0.01735069, -0.025950378, -0.013782198, 0.024664208, -0.010516334, -0.0018394127, 0.03338999, -0.015875377, -0.018624252, -0.0025896789, -0.023693275, 0.0637537, -0.02071743, 0.0058319, -0.04607516, 0.0114431335, -0.0019355603, -0.0016021963, -0.015471873, 0.0056711286, -0.024891179, 0.0019733887, 0.00491771, -0.014790959, -0.009753458, -0.015572749, -0.027665272, -0.08075132, -0.0120231705, 0.020566115, -0.007534184, -0.0032437972, -0.006043109, 0.01036502, -0.089729294, -0.0398965, 0.017665928, -0.012105132, 0.009242773, -0.008751002, 0.039467774, 0.029304508, 0.045192495, -0.035180543, 0.00075854105, -0.0019623556, 0.010579381, -0.012722999, 0.012224923, -0.037172846, 0.022243181, -0.00092364685, 0.022028819, -0.004627691, -0.025017275, 0.010371325, 0.01010022, 0.013340865, -0.026732167, -0.02125964, 0.0077296314, 0.005157291, 0.057247188, 0.0026369644, -0.004511053, -0.015862767, -0.04350282, 0.0021073648, -0.0090347165, 0.0076224506, 0.009866944, -0.00023958075, -0.015673624, -0.013139113, 0.012124047, -0.035432734, 0.019254727, 0.008927535, 0.026908701, 0.030111518, -0.020150002, -0.0060903947, -0.0012814418, -0.0072378605, 0.0040129777, 0.0026322359, 0.034777038, 0.0121933995, 0.03782854, -0.023605008, -0.011827723, 0.00478531, 0.023945466, 0.040779166, 0.0041863583, -0.01088201, -0.0013169061, 0.012798656, -0.015724063, -0.000108954075, -0.010075001, -0.0018220746, -0.06294669, -0.0064213946, -0.0222684, 0.008252926, -0.003059383, 0.0010867824, 0.0019071889, 0.024538113, 0.013441741, 0.00588549, 0.022470152, -0.0077044126, 0.024071561, -0.003940473, -0.03490313, 0.029834108, 0.016997624, 0.032936048, 0.037450254, -0.03568492, -0.021020059, 0.034398753, 0.052707765, 0.0062133376, 0.0070802416, -0.013328255, -0.017539833, 0.004700196, -0.02877491, -0.0049019484, 0.055027917, -0.017943338, 0.03835814, -0.0035716447, -0.010068696, 0.01760288, -0.029153194, 0.027286986, -0.026379101, -0.02433636, -0.023655446, 0.015534921, 0.027060015, 0.01088201, -0.006676737, 0.007962908, -0.005516662, 0.015232292, 0.036542367, 0.04990845, -0.011159419, -0.002491955, 0.019759107, 0.00015387546, -0.006115614, -0.013252598, -0.033541303, 0.011335952, -0.004958691, -0.00031307057, 0.02693392, -0.0014816178, 0.023403257, 0.014185702, -0.029607138, 0.016367149, -0.03243167, 0.011064848, 0.02774093, 0.0033478257, -0.043351505, -0.01814509, -0.014374845, 0.0030515022, -0.012281666, -0.004656063, -0.040854823, -0.004315606, -0.0064875945, 0.006232252, -0.030741993, -0.029859327, 0.0037891588, 0.009469745, -0.013870465, 0.004425939, 0.036365837, 0.015421434, -0.009677801, -0.041914023, -0.019544745, 0.010106524, 0.0024777693, 0.015484482, 0.020944402, 0.0012775013, 0.012792351, 0.024298532, -0.0017180461, -0.019040365, 0.0036126254, 0.0066326037, 0.017363299, 0.026303444, 0.010573077, 0.051951192, -0.008252926, 0.041611392, 0.03548317, 0.0039783013, 0.017426347, 0.005907557, -0.033591744, 0.0100308675, -0.021360515, 7.54108e-05, 0.008517726, 0.015761891, -0.008681649, 0.016657166, -0.012893227, 0.027564395, 0.00019525042, -0.0140217785, 0.021890115, 0.048243996, -0.021663144, -0.01398395, 0.028623594, -0.004993367, -0.0026196265, 0.004107549, 0.013920902, 0.08140702, 0.018838612, -0.007584622, 0.01968345, 0.031750754, -0.030464584, -0.009532792, 0.0141100455, 0.011335952, -0.021663144, -0.0055954712, -0.019027755, 0.014816178, 0.014122655, 0.06819224, -0.008126832, -0.013328255, -0.023453694, -0.004375501, 0.013517398, -0.037374597, -0.036643244, -0.004384958, -0.009167116, 0.0033541305, -0.0037229587, 0.004063416, 0.024916397, -0.00030046105, 0.014059607, -0.029304508, 0.01321477, 0.020603944, 0.041838363, 0.005526119, 0.010831572, 0.0018299556, 0.0320786, 0.0069793654, 0.015875377, 0.034751818, 0.0066452133, -0.028421843, -0.0279679, -0.021045277, 0.031473346, 0.035533607, -0.016531073, -0.009009497, 0.021940554, 0.00019101442, 0.045293372, -0.028875785, 0.032204695, -0.0045961677, 0.03497879, 0.00022362183, 0.00049965194, -0.009892163, 0.012338408, -0.023340208, 0.00413592, -0.03137247, -0.014639645, -0.008883402, -0.01256538, 0.005024891, 0.011153114, 0.019885203, -0.023428475, 0.037576348, 0.010787439, 0.028346185, -0.03888774, -0.0212092, 0.016644558, -0.017224595, 0.0024714647, -0.004176901, -0.032784734, -0.010762219, -0.0009606873, -0.017590271, -0.0042210347, -0.027034797, -0.03727372, -0.007490051, 0.0025817978, -0.0052234908, -0.013441741, 0.022129696, -0.007918774, -0.014664864, 0.006689347, 0.040804386, -0.030792432, 0.012496027, 0.026858263, -0.021032669, -0.027085233, 0.0010844182, -0.012893227, -0.06420764, 0.025004664, -0.008769916, -0.0037008922, -0.037097186, 0.04322541, -0.014917054, -0.016669776, 0.019368213, -0.0051005483, -0.0053369766, 0.017211985, -0.012987798, -0.043124534, -0.018952098, -0.01813248, -0.045747314, 0.0026889786, 0.029304508, 0.002417874, -0.0007140137, 0.0028875784, -0.019696059, 0.012363628, -0.010156963, -0.0020553507, -0.024008512, 0.0009756611, -0.006765004, -0.027413081, 0.013227379, -0.029733231, 0.02257103, -0.055633172, -0.002490379, -0.017779414, 0.0012428252, -0.00032055748, -0.010705477, -0.040426098, 0.0027047405, 0.045469906, -0.024702037, 0.031246373, -0.021839676, 0.006200728, 0.008486202, 0.049883235, -0.017527223, -0.032910828, -0.010541553, 0.0092806015, -0.00804487, 0.03681978, 0.0062480136, -0.028320966, -0.015358387, -0.0069352323, 0.05825595, 0.016468024, 0.0061376807, -0.003508597, -0.030868089, -0.0015927391, 0.01477835, 0.039997377, -0.008372717, 0.039039053, 0.0017653318, -0.018775566, -0.0029853021, 0.014450502, 0.0023043884, 0.014463112, -0.023478914, 0.008996888, 0.016795872, 0.028220091, -0.043603696, 0.011115286, 0.019305164, -0.0027851262, -0.00804487, 0.009160811, -0.014891835, -0.022999752, -0.03159944, 0.033616964, 0.013530008, -0.010787439, 0.003075145, 0.012584294, 0.019973468, -0.0038459015, -0.02718611, 0.00022874445, -0.0048798816, 0.009608449, 0.025231635, 0.017199377, 0.0048294435, 0.010926143, -0.052455574, -0.0012648918, 0.040325224, -0.003940473, 0.036138866, -0.030439364, 0.04014869, -0.006607385, -0.011808809, 0.016001472, -0.010068696, -0.031952508, 0.015774501, -0.015421434, -0.0010402849, -0.029380165, 0.0028292595, 0.0106109055, 0.038812082, 0.018220747, 0.04501596, -0.004835748, -0.01760288, 0.02331499, 0.03389437, -0.023718495, -0.017539833, 0.0057152617, -0.010787439, -0.027539177, 0.009507573, 0.023945466, -0.027135672, -0.017464176, 0.02434897, 0.0279679, -0.0046623675, 0.0019655079, 0.09638712, 0.006708261, 0.026883481, 0.004933472, -0.014917054, 0.01581233, 0.012231228, -0.035079665, 0.011335952, -0.025269464, 0.007943993, -0.01814509, 0.016367149, -0.0222684, 0.00052959955, -0.027211329, 0.030893307, 0.007874641, -0.011537705, 0.010711782, 0.010995495, -0.016001472, 0.010257839, -0.009892163, 0.026858263, -0.026883481, -0.02847228, 0.012924751, 0.014601816, 0.007994431, -0.010358715, 0.006440309, -0.008574469, 0.007168508, -0.0033005401, 0.009816506, -0.027135672, -0.002700012, -0.0012436132, 0.004832596, -0.013530008, -0.003275321, -0.026026035, -0.0015430892, -0.0032564069, -0.04057741, 0.007981822, 0.008719479, 0.044435926, 0.012224923, -0.0050595673, 0.031221155, 0.017804632, 0.006134528, 0.015534921, -0.018321622, 0.024134608, -0.028547937, -0.014866616, 0.007830508, 0.039518215, -0.0031019403, -0.006106157, 0.013378694, -0.058104638, 0.0050816336, 0.00037355683, -0.0030042166, -0.02278539, -0.038408577, -0.009753458, 0.039518215, 0.0013326679, -0.010283058, 0.041006137, 0.018233355, 0.013756979, -0.0077800695, -0.001516294, -0.03495357, -0.0037607872, 0.0006765792, -0.019708669, -0.009192335, 0.0060052807, -0.0060494137, 0.012754522, 0.012577989, 0.014563988, -0.014967492, -0.01281757, -0.006991975, 0.008694259, -0.008334888, -0.015837548, -0.009476049, 0.009520182, 0.015068368, 0.012407761, 0.005141529, -0.010371325, 0.0090284115, 0.056793246, 0.018813394, 0.016291492, 0.037147626, -0.0067208703, -0.028623594, -0.009091459, -0.01761549, 0.007874641, 0.0111909425, 0.003407721, -0.008486202, 0.010724391, -0.011758371, 0.0004425151, -0.0024635836, -0.02075526, -0.015320558, 0.0033541305, -0.0031350402, 0.00414853, -0.008889707, -0.0021057888, 0.02356718, -0.015698845, -0.018611642, 0.015673624, 0.017401129, -0.03702153, 0.012275361, 0.047764834, 0.003042045, -0.021637924, 0.016531073, 0.007849422, 0.012754522, -0.0119412085, -0.010320886, 0.048067465, -0.003798616, -0.021221811, -0.026631292, 0.026076473, -0.0021057888, -0.0029033404, -0.017275034, 0.022999752, 0.030716773, 0.03240645, -0.009349954, 0.023277162, 0.0037860062, -0.001409113, -0.031927288, -0.027665272, -0.00033021162, 0.045116838, -0.0040791775, -0.038610328, 0.009829115, -0.019834764, -0.01761549, 0.010579381, -0.025584701, 0.003448702, -0.041283544, -0.019040365, 0.0090284115, -0.0041611395, 0.00096383964, 0.012975189, 0.008278145, 0.018800784, -0.033591744, 0.00686588, 0.0026196265, -0.010806353, 0.0039057967, -0.0034770733, -0.026757387, 0.039795622, 0.02073004, -0.00827184, -0.0052770814, -0.0044732247, -0.0003774973, 0.013618275, 0.028018339, 0.0070487177, 0.021284858, 0.005481986, -0.04458724, -0.008555555, 0.009658887, -0.017073281, 0.01968345, 0.018182918, -0.032532543, -0.036542367, 0.01154401, 0.0007522363, -0.031069841, 0.004416482, -0.005091091, 0.029405385, -0.039846063, -0.0010095491, 0.014816178, -0.0020963314, 0.016720215, 0.061332673, 0.0024367885, 0.0068217465, -0.024929008, -0.0058697285, 0.004845205, 0.022180134, 0.018523375, -0.014084826, 0.037172846, -0.0014311797, 0.013630884, -0.035079665, -0.009274296, 0.009728239, -0.0022728648, 0.008347497, -0.0051730527, -0.0011734728, 0.0445368, 0.030086298, -0.029708013, 0.02410939, 0.026908701, -0.008782526, 0.0019923032, 0.01887644, -0.0024352123, -0.014740521, -0.021877505, 0.015333168, -0.0035874066, 0.04579775, 0.021839676, -0.010207401, -0.014286579, 0.002933288, 0.01321477, -0.049504947, 0.0222684, 0.008107917, -0.018775566, -0.022772782, 0.023655446, -0.033818714, 0.012250142, 0.02718611, 0.02461377, -0.035180543, 0.012571685, -0.02769049, -0.028043557, -0.0008976397, 0.010516334, -0.0026810977, -0.011127896, 0.015951034, 0.002734688, 0.021801848, 0.010573077, 0.04844575, 0.006377261, 0.016518462, 0.0037797017, -0.009204945, -0.01891427, 0.012004256, 0.017943338, 0.007981822, 0.013176941, -0.01785507, -0.014009169, -0.0032264593, 0.029581917, -0.0126914745, 0.0077044126, -0.012830179, -0.00081725407, -0.018498156, 0.0497067, -0.040779166, -0.013920902, 0.02047785, -0.0014863464], "content_hash": "03a6f027162ffb200c016f6993fe6690"}, {"content": "each of the 5 pipeline stages shows - `ablation_panel_guide.md` - How to use the ablation experiment panel - `attribution_panel_guide.md` - How to use the token attribution panel - `beam_search_and_generation.md` - Beam search, generation controls - `head_categories_explained.md` - Previous-Token, Positional, BoW, Syntactic, Other - `model_selector_guide.md` - Choosing models, auto-detection, generation settings\n\n### Category 3: Model-Specific Documentation\n- `gpt2_overview.md` - GPT-2 architecture, why it's a good starter, variants\n- `gpt_neo_overview.md` - GPT-Neo architecture, local attention, comparison with GPT-2\n- `pythia_overview.md` - Pythia architecture, RoPE, parallel attn+MLP, interpretability focus\n- `opt_overview.md` - OPT architecture, ReLU activation, comparison with GPT-2\n- `qwen_overview.md` - Qwen2.5 (LLaMA-like) architecture, RMSNorm, SiLU, GQA\n\n### Category 4: Guided Experiments (Step-by-Step)\n- `experiment_first_analysis.md` - Your first analysis with GPT-2\n- `experiment_exploring_attention.md` - Reading attention patterns and head categories\n- `experiment_first_ablation.md` - Removing a head and observing the effect\n- `experiment_token_attribution.md` - Measuring token influence with gradients\n- `experiment_comparing_heads.md` - Systematic comparison across head categories\n- `experiment_beam_search.md` - Exploring alternative generation paths\n\n### Category 5: Interpretation, Troubleshooting, and Research\n- `interpreting_ablation_results.md` - How to read ablation probability changes\n- `interpreting_attribution_scores.md` - Understanding attribution score values\n- `interpreting_attention_maps.md` - Reading BertViz patterns visually\n- `troubleshooting_and_faq.md` - Common issues and frequently asked questions\n- `recommended_starting_points.md` - Best models, prompts, and experiment order\n- `mechanistic_interpretability_intro.md` - Mech interp research context\n\n## Notes", "source_file": "README.md", "chunk_index": 1, "embedding": [-0.0033370038, 0.047332633, 0.011288341, -0.03154623, -0.029048046, -0.020982098, -0.012138788, 0.023081636, 0.011195324, 0.027267424, 0.030988125, -0.02245709, -0.011049153, -0.02856967, 0.0066607194, 0.0045711473, -0.005591018, -0.0054049827, 0.030961549, 0.086054504, 0.038270067, -0.00805266, 0.009793417, 0.043452475, 0.003097816, -0.010265149, 0.013075607, 0.016663427, -0.029818762, -0.020330975, 0.038216915, -0.029287234, -0.031891726, -0.0108166095, -0.046030387, 0.0072620115, 0.013607136, 0.015799692, 0.0092486, -0.002863611, 0.023666317, -0.04932587, -0.04751867, 0.035532694, -0.03154623, -0.002433405, 0.028968316, -0.031785417, -0.03348631, 0.018071977, 0.021420607, 0.02326767, -0.016224915, -0.022164749, 0.022031866, 0.01033159, -0.01715509, -0.03938628, -0.007069332, -0.03468225, -0.01521501, 0.0021493693, 0.014058935, -0.010152199, 0.015959151, -0.048289385, -0.06505912, 0.030456597, -0.02758634, -0.03444306, 0.07462663, -0.0072686556, -0.02328096, -0.0018138419, -0.003929991, -0.03593134, -0.014909381, 0.055225834, -0.023241093, -0.05474746, 0.0014824668, -0.0014301445, -0.0009251922, -0.045392554, -0.0434259, -0.022962041, -0.033459734, -0.044116884, -0.038509257, -0.02902147, -0.040183574, -0.012564011, -0.04169843, 0.026669454, 0.014683481, -0.041060597, 0.021726238, -0.031918302, 0.01969314, 0.018324453, 0.028330483, -0.01670329, -0.027533188, 0.0019151645, 0.0702681, -0.007979576, -0.0053883726, 0.004966472, -0.041592125, -0.043957427, -0.07707167, 0.020570163, 0.021367455, 0.025553243, -0.00022880652, -0.0152282985, -0.015082128, 0.001943402, 0.017500583, -0.019135034, -0.04900695, -0.013181913, 0.012843063, -0.0065045827, 0.017939094, -0.02229763, 0.029499846, -0.05517268, -0.015414333, -0.0117268525, -0.0035678865, 0.0074148257, 0.009753552, -0.060541123, -0.032130912, 0.011859735, -0.02886201, 0.040316455, -0.070533864, -0.0065544136, 0.009673823, -0.033911534, -0.0034350045, 0.009481144, -0.05100018, 0.011002645, -0.014524023, 0.06442128, -0.013195201, 0.013600492, -0.010132266, 0.0044083665, -0.012949369, -0.04241599, -0.024782527, -0.037871424, -0.0015771454, 0.048847493, 0.0023470316, -0.07324466, 0.036409717, 0.026270809, -0.01783279, 0.0053916946, -0.069151886, 0.034788556, -0.03274217, -0.010823254, -0.027958412, -0.02559311, 0.0046110116, -0.029871915, -0.011328206, -0.022736141, -8.421927e-06, 0.0057604425, -0.06957711, -0.023825776, -0.019799445, -0.039731774, 0.005913257, -0.013886188, -0.06920504, 0.00028611196, -0.05012316, -0.049352445, 0.008989479, -0.00045553673, -0.03425703, 0.015015687, -0.0018204859, 0.020038633, 0.048900645, -0.006069394, -0.0013304829, -0.022589972, 0.025314055, 0.0010456167, 0.008896463, 0.0038070746, 0.020756198, -0.006009597, 0.027240848, 0.0066407872, -0.0045080283, -0.0021759458, -0.021978714, 0.0013163642, -0.0416187, -0.056660965, 0.006507905, -0.052196123, 0.038961057, 0.05017631, -0.07329781, -0.023825776, -0.052780803, 0.02326767, 0.037871424, 0.034124143, 0.008178898, -0.008610765, 0.02737373, 0.04005069, 0.011268409, 0.0429741, 0.031200737, -0.03694125, 0.022576682, -0.004375146, 0.03497459, 0.05899969, -0.006145801, 0.029818762, 0.0054415255, -0.06824829, 0.01007247, -0.020198092, 0.01301581, -0.053445216, 0.033539463, 0.017846078, 0.0063185478, -0.0074679786, 0.036223684, 0.0056441706, -0.012882927, -0.08621396, 0.028968316, 0.03393811, 0.069151886, 0.009055921, -0.015347892, 0.022337494, 0.020038633, -0.0033037832, 0.0018171639, -0.008690495, -0.053365484, 0.02454334, -0.008564257, 0.010544201, -0.031200737, 0.0034250384, -0.026842201, 0.02520775, -0.09062565, 0.024968563, -0.052621346, -0.061869945, -0.0025712703, 0.0013661949, 0.03797773, 0.034815133, -0.033061087, -0.008550969, -0.0029665949, 0.028649399, -0.02572599, -0.020530298, 0.018138418, 0.0053186095, 0.04618985, 0.026895354, -0.00045927404, 0.033512887, -0.0073882495, 0.0127633335, -0.00019568978, -0.0027124577, 0.016942479, -0.014391141, 0.03558585, -0.018018825, 0.019865887, 0.024941986, 0.0526745, -0.008610765, -0.03824349, 0.021832542, 0.0076872343, -0.062933005, 0.027692648, -0.022802582, 0.044276346, -0.0064414637, -0.012437772, -0.022244478, -0.011919532, 0.0011270071, 0.0055544754, -0.048714608, -0.017168378, -0.040156998, -0.004946539, 0.0064514303, 0.019002153, -0.017234819, 0.0043087048, -0.026536573, -0.029048046, -0.060594276, 0.0012233466, 0.010909627, 0.024357304, 0.013235065, -0.025101444, 0.039226823, 0.0075609963, -0.01649068, 0.05953122, -0.025074868, -0.008345001, 0.019639987, -0.032423254, 0.023387264, -0.0074613346, 0.069842875, -0.0074480465, 0.029526422, -0.09025358, 0.012776622, -0.05296684, 0.010471116, -0.03997096, 0.032343525, 0.015919285, 0.0051325746, 0.026895354, 0.03311424, -0.04066195, 0.0054215933, -0.026284097, -0.015892709, 0.00585346, 0.01678302, -0.015985727, -0.02989849, 0.004810335, 0.008331713, -0.0277458, 0.0237992, 0.042469148, 0.0015580436, 0.03080209, -0.023892216, -0.013148692, 0.0004908336, 0.008464595, -0.013633712, -0.0058069513, -0.0075742844, -0.003979821, -0.02328096, -0.08451307, 0.007886558, -0.013607136, 0.025779143, -0.03221064, 0.05153171, 0.008624054, 0.0095276525, -0.023148077, -0.04908668, 0.046030387, 0.011427868, -0.009195447, 0.01783279, -0.030988125, 0.04244257, 0.017141802, -0.017992247, 0.0035047675, -0.012005906, 0.0143246995, 0.04887407, -0.042761486, 0.025845585, 0.058096092, 0.018111842, 0.0126370955, -0.0235733, -0.039359704, -0.018510489, 0.019733004, 0.044568684, 0.027400307, 0.052913684, 0.020530298, 0.007800184, 0.024370592, -0.02021138, 0.022908889, -0.013354659, -0.031785417, -0.005913257, 0.03885475, 0.03901421, 0.027294, 0.029526422, 0.0065311594, -0.0526745, -0.021128267, -0.04653534, -0.059318606, 0.01932107, 0.01181987, -0.03648945, 0.012331467, -0.031891726, 0.0008579206, 0.0023187941, 0.04079483, 0.0036642263, 0.01587942, -0.027347153, 0.0592123, 0.020928944, -0.020384127, 0.0018055367, 0.027506612, -0.022682989, -0.017394278, -0.018045401, -0.057830326, 0.014896093, -0.03946601, -0.065218575, 0.010504337, -0.050973605, 0.01196604, 0.009129006, 0.0034848354, -0.008345001, -0.007481267, -0.02947327, 0.077975266, -0.0037472777, 0.06734469, -0.009222023, 0.035745308, -0.009354906, -0.003926669, 0.01953368, -0.008238696, -0.014364564, -0.024848968, -0.02132759, -0.0035745308, -0.039093938, 0.03252956, -0.03625026, -0.02931381, -0.055916823, -0.041034017, -0.04953848, -0.016969055, -0.025128022, 0.03080209, -0.047385786, 0.0076938784, 0.0037771761, 0.010530913, 0.053817283, 0.022403937, 0.060381666, -0.01998548, -0.008923039, 0.016955767, 0.0017108581, -0.030536326, -0.025632974, 0.03425703, -0.015188433, -0.020915655, -0.0037240232, -0.03736647, -0.057298798, -0.016822884, 0.011128883, 0.027798953, 0.004787081, -0.009640602, 0.032051183, 0.036728635, -0.030323714, 0.013434389, 0.03348631, 0.0035080896, 0.029499846, -0.049299292, -0.05453485, -0.0050229467, -0.0033137496, 0.0081058135, 0.024224423, 0.006863365, 0.0075742844, 0.016889326, 0.03975835, 0.00061997847, -0.032263793, 0.012065702, -0.0046409103, -0.015082128, -0.017593602, -0.005713934, -0.035532694, -0.02790526, 0.004999692, -0.003956567, -0.048369113, 0.004949861, -0.0228956, -0.04863488, -0.011740141, -0.0029981544, -0.00656438, -0.04079483, -0.023852352, 0.0016593663, 0.011487665, 0.011089019, -0.01181987, 0.015759828, -0.0008271916, 0.01338788, 0.00044100275, 0.02043728, 0.039226823, 0.029818762, -0.041220054, 0.056714118, -0.02968588, -0.014962534, -0.019812733, -0.010471116, -0.04483445, 0.038216915, -0.008976191, 0.012225161, -0.040449336, 0.0013944324, -0.019932328, -0.011069086, 0.03670206, 0.005770409, -0.013560627, 0.035027742, -0.021739526, -0.010391387, 0.009952876, 0.012338111, -0.011567394, 0.033220544, -0.023772623, 0.0037406336, 0.02171295, -0.011547462, -0.010112334, -0.032502983, -0.015866132, -0.037499353, 0.043984003, -0.030988125, -0.011700276, 0.0082320515, -0.0007183943, 0.002860289, 0.0008761919, 0.03175884, -0.01010569, -0.0033220546, -0.00570729, -0.016690003, -0.008311781, 0.014032358, 0.0015264841, -0.017513871, -0.0030280529, 0.021008674, -0.029579574, -0.030111102, 0.0411669, -0.031865146, 0.015773116, -0.04183131, -0.012816487, 0.015281451, -0.010510981, 0.010876407, 0.0010987696, -0.044781297, -0.023028482, -0.013308151, 0.016809596, -0.0061790217, 0.007109197, 0.035532694, -0.023400553, 0.029526422, -0.010710304, -0.004687419, -0.0038635496, 0.0136735765, 0.042176805, -0.014005782, -0.03779169, 0.0062487847, 0.031041278, -0.039173666, 0.016477391, 0.068195134, 0.008923039, -0.019639987, 0.018816117, -0.004019686, -0.0032772068, 0.0056773913, -0.0017357735, 0.0048335893, -0.008318425, -0.0010182097, -0.007600861, 0.01924134, -0.014058935, 0.005102676, 0.0015580436, -0.026496708, -0.027692648, -0.036914673, -0.0141918175, 0.053976744, 0.009255244, 0.012670317, 0.016065456, -0.0066540753, -0.033592615, -0.005122608, -0.011368071, -0.040847983, 0.028516516, -0.021540202, -0.022443801, -0.016836172, 0.015985727, 0.019161612, 0.022324206, -0.012896216, -0.0045877574, -0.0747861, 0.020330975, 0.0067271604, -0.001612027, 0.0152282985, -0.0050860657, -0.008996123, -0.029074622, 0.008723715, 0.0471466, -0.016078744, -0.011168747, -0.055757366, -0.021965425, 0.016078744, -0.017248107, 0.012298246, -0.014510735, 0.02536721, -0.00026057367, 0.009753552, -0.011089019, -0.017726483, 0.0074613346, -0.0015763149, 0.027931836, -0.012623807, 0.0063152257, 0.003973177, -0.09631301, -0.029074622, -0.022350783, 0.00017077436, -0.0019716395, -0.031147584, 0.012205228, -0.016530544, 0.057245646, 0.018696524, -0.046641648, -0.048528574, 0.016171763, -0.011587326, 0.017274683, -0.016530544, 0.007301876, -0.014896093, 0.015121993, -0.006006275, 0.000882836, -0.030642632, 0.022178037, -0.02856967, 0.017420854, -0.00036480313, -0.019826021, -0.02237736, -0.016610272, -0.03258271, 0.011314918, 0.024649646, 0.023294248, 0.014816363, -0.016477391, 0.00050163025, 0.019573545, -0.02520775, 0.005720578, 0.024888834, -0.016052168, 0.01324171, -0.0025280835, 0.003518056, 0.016437527, 0.017846078, -0.02483568, -0.0328219, 0.025128022, -0.019440664, -0.013520762, 0.014643617, 0.011115595, -0.0065145493, -0.03468225, 0.0346291, -0.007959642, 0.012085634, -0.018297877, -0.00473725, 0.01036481, 0.0032290372, 0.027506612, -0.00286195, 0.010118978, -0.031067856, -0.0105574895, 0.023878928, 0.02177939, 0.003833651, 0.0067769913, -0.009062565, -0.015321316, 0.02947327, -0.014510735, 0.054269083, -0.04010384, 0.04913983, -0.029154351, 0.0078400485, -0.014776499, -0.021021962, -0.008643986, -0.009866502, -0.010869763, 0.008318425, -0.011328206, -0.055757366, 0.00038348968, -0.048847493, -0.047757857, -0.05517268, -0.020742908, 0.027187696, -0.003398462, -0.03050975, -0.0099130105, -0.019560257, -0.09551571, -0.004966472, -0.0060594277, -0.006803568, -0.021526914, -0.014803075, 0.040927712, 0.04177816, 0.006883297, -0.021460474, 0.0075011994, -0.027294, 0.02184583, -0.014085512, 0.01379317, -0.046296153, 0.02126115, -0.019865887, 0.026868777, -0.022390649, 0.0046674865, 0.032396678, -0.0015322977, 0.0009800061, -0.044940755, -0.004189111, -0.010730237, -0.004129314, 0.0341773, 0.0063517685, -0.02095552, 0.033592615, -0.0055478313, 0.018909134, -0.010238572, 0.03534666, 0.00064904644, 0.012816487, 0.018603506, 0.005348508, 0.010750169, -0.03648945, 0.0036575822, 0.0020829283, 0.054694306, 0.005162473, -0.020384127, 0.00065070746, -0.000387227, 0.01998548, 0.01558708, -0.0068434323, -0.01047776, 0.02021138, 0.05676727, -0.029499846, -0.027480036, -0.051717747, 0.019666564, 0.013979206, 0.013401168, 0.015148569, 0.014125376, 0.0024682865, -0.014284835, 0.0045312825, 0.009075853, -0.006737127, -0.042947523, -0.006002953, -0.0109561365, 0.018603506, 0.009560873, -0.015188433, -0.02148705, 0.017487295, 0.026948508, -0.0026825592, -0.0027224238, -0.02037084, 0.012896216, 0.0075144875, -0.041964196, 0.026828913, 0.0055744075, 0.060328513, 0.023560012, -0.024237711, 0.023626452, 0.0383498, 0.022403937, -0.009188803, 0.021992002, -0.01007247, -0.036356565, -0.00606275, -0.002674254, 0.019613411, 0.02976561, -0.0035014455, -0.0014434328, 0.023440417, -0.047943894, -0.02572599, -0.012856351, 0.048395693, -0.02237736, -0.0025397108, -0.031200737, -0.00958745, 0.032263793, 0.031652536, 0.013474254, 0.016969055, -0.021500338, -0.00496315, 0.047093447, 0.036143955, 0.0033286987, -0.019055305, 0.0033154106, -0.0078400485, -0.025181174, 0.03670206, -0.023360688, 0.0065809903, -0.021992002, 0.00032410794, 0.011985973, -0.014949245, 0.02483568, 0.01857693, -0.011368071, -0.013560627, 0.010225284, -0.0026991693, 0.017474007, 0.0026460164, -0.028516516, -0.0103050135, -0.023706181, -0.0023403876, 0.008517748, 0.013819748, -0.043744814, -0.011686988, -0.010823254, -0.014829651, 0.018337741, 0.019746292, 0.014444293, 0.008929683, -0.035320085, 0.007673946, 0.01007247, 0.036516026, -0.004145924, -0.039784927, -0.024437035, 0.034549367, -0.0016984005, 0.02931381, 0.02947327, 0.010776745, -0.006767025, 0.010471116, -0.013201845, -0.021061826, 0.028755704, -0.0043319594, -0.033645768, 0.01010569, 0.042176805, 0.05511953, 0.04682768, 0.041299783, 0.009833282, 0.022324206, -0.002498185, 0.0037572437, -0.0007952168, 0.010019316, -0.019028729, -0.0041093817, 0.021021962, -0.030297138, -0.0010996001, 0.012603875, 0.0016626883, 0.013308151, 0.01649068, 0.026151214, -0.004730606, 0.034416486, -0.022736141, -0.012105567, 0.037499353, -0.0005452322, -0.024729375, -0.0044880956, -0.04215023, 0.020623315, -0.007115841, -0.018975576, 0.011793294, 0.03369892, -0.018204859, -0.010484404, 0.011773362, -0.0032705627, -0.0047173174, 0.0042389417, -0.011182036, 0.026044909, -0.037685387, 0.0042920946, 0.02947327, -0.008152322, -0.009321685, -0.0063584126, -0.040821407, -0.045525435, -0.027347153, -0.001484128, -0.0034516146, 0.014298123, -0.015919285, -0.015733251, 0.019214764, 0.011873024, -0.008889819, 0.009414703, -0.014125376, 0.010603998, 0.0038469392, 0.0139393415, 0.005122608, -0.00073043676, 0.009939588, 0.0037771761, 0.057777174, 0.02960615, -0.010152199, -0.012829775, 0.0048335893, -0.012092278, 0.04177816, 0.03715386, -0.02506158, -0.014949245, -0.0011012611, 0.022709565, 0.02184583, -0.0026509995, 0.016995631, -0.010610643, 0.049883973, 0.019347645, 0.016371084, 0.012590587, -0.0024699476, 0.018138418, 0.0055710855, -0.03638314, 0.019360933, -0.015547216, -0.012225161, 0.0350809, 0.026642878, -0.005105998, -0.005939834, 0.017726483, -0.019972192, 0.028888587, -0.021566778, -0.009800061, 0.011055797, -0.03675521, 0.0139260525, -0.031865146, -0.006876653, -0.0053817285, -0.022443801, -0.010285081, 0.002928391, -0.014391141, -0.034496214, -0.017248107, -0.0023902182, 0.009952876, -0.012650384, 0.010099046, -0.0072088586, -0.021952137, -0.004315349, 0.015135281, -0.0056873574, 0.018045401, 0.015095416, -0.036436293, -0.010643863, -0.019573545, 0.0045877574, -0.028622823, -0.010670439, -0.0014284835, 0.022643125, -0.059424914, -0.007009535, 0.022696277, -0.0140988, 0.018085266, -0.02393208, 0.027480036, -0.006889941, -0.007042756, -0.022244478, -0.0064348197, -0.011401291, -0.033459734, 0.0004941556, 0.008039372, -0.0052588126, 0.0018736388, -0.024370592, -0.020317687, 0.0070959087, -0.0138729, -0.00089861575, -0.022018578, -0.0032805288, -0.011653768, 0.002611135, 0.0122517375, -0.027559765, 0.025619686, -0.032795325, -0.022310918, 0.0064780065, -0.01558708, 0.0076473695, 0.0016352814, -0.015294739, -0.0038668716, 0.03319397, 0.01670329, 0.007813472, -0.014736635, 0.031865146, 0.02528748, 0.07871941, -0.02260326, -0.030748937, -0.003836973, 0.010384743, 0.011667056, 0.0066673635, 0.0010140572, -0.024184557, -0.005026269, -0.0136735765, 0.031865146, 0.008962903, 0.017779637, -0.009819994, -0.0143246995, -0.006850077, -0.00019257536, 0.06537803, 0.011434512, 0.027984988, 0.03803088, -0.021141555, -0.01253079, 0.0047605042, -0.0017507228, 0.021699661, -0.012610519, -0.0009999385, 0.021301014, -0.013573915, -0.0517709, 0.02193885, 0.02103525, -0.029526422, 0.0008346662, 0.016915902, 0.02790526, -0.03715386, 0.001825469, 0.016278068, 0.016796308, 0.03035029, -0.020198092, 0.017872654, 0.01010569, 0.001280652, 0.0056840354, 0.003400123, -0.00995952, -0.0022822516, 0.0184972, -0.0034283604, -0.0044449093, 0.0471466, -0.03975835, -0.013467609, 0.04156555, 0.00438179, 0.01424497, -0.020012056, 0.021992002, -0.023466993, -0.0064348197, 0.021872407, -0.0475984, -0.010225284, 0.0023005228, 0.00062454626, 0.024490187, -0.017341126, -0.0026443554, 0.0027805597, 0.033459734, 0.019719716, 0.006275361, -0.020423992, -0.018337741, 0.02803814, 0.021633219, 0.0063152257, -0.021593355, -0.008511104, -0.019772869, -0.035904765, 0.018988864, 0.037472777, -0.015374469, -0.020875791, 0.016942479, 0.016118608, 0.025167886, 0.019520393, 0.07441402, -0.0036575822, -0.005228914, -0.001183482, -0.022310918, 0.012404552, 0.0044150106, 0.009016057, 0.0058667483, -0.03415072, 0.02328096, -0.022656413, -0.011760074, -0.006218886, 0.012670317, -0.0013628729, 0.006989603, 0.0062820055, 0.004810335, -0.015121993, -0.00559434, -0.009753552, 0.03893448, 0.009627314, 0.03258271, -0.03266244, 0.0026991693, 0.01603888, -0.010570778, 0.0041691787, -0.010079114, -0.0045080283, -0.014550599, 0.02811787, -0.004076161, -0.018922424, -0.015533928, -0.0021526914, 0.01767333, 0.005175761, -0.022337494, -0.008046016, -0.0015339587, 0.006883297, 0.010916271, -0.04991055, 0.046429034, -0.016078744, 0.030376866, -0.014484158, -0.02947327, -0.0041625346, 0.018590217, 0.010424607, 0.0062454627, 0.0062255305, -0.032848477, -0.0156136565, -0.0006884958, 0.02528748, -0.00020773223, -0.005066133, -0.034735404, -0.021314302, -0.019639987, 0.0046375883, -0.0064813285, 0.00999274, 0.0018985542, -0.05304657, 0.009560873, 0.0018570286, 0.0018586896, -0.03638314, 0.028755704, 0.035187203, 0.03125389, -0.002923408, -0.023320824, -0.001400246, 0.0184972, -0.003224054, -0.007607505, 0.010464472, 0.00782676, 0.0045113503, 0.010896339, -0.012862995, 0.016876038, -0.009494432, -0.045392554, -0.0015264841, 0.016158473, -0.0119926175, -0.007541064, 0.009514364, 0.01130163, -0.00015333359, -0.038456105, 0.0022988617, 0.0031576129, 0.02476924, 0.025991755, 0.001760689, 0.013208489, 0.0074746227, -0.02559311, -0.0044183326, -0.008657275, -0.018311165, 0.0010555829, -0.04074168, 0.014524023, -0.012922793, -0.015839556, -0.023626452, 0.026164502, -0.050548382, -0.020158228, -0.022217901, -0.014962534, 0.002064657, -0.0045977235, -0.011368071, -0.010032605, 0.013587204, -0.0066773295, 0.011740141, -0.0008325899, 0.028144447, -0.026217654, -0.005182405, 0.033752076, -0.018005537, -0.03476198, 0.019294493, -0.016317932, 0.0054382035, -0.020131651, -0.0008861581, 0.024809103, 0.004132636, 0.018218147, -0.016424239, -0.010192064, 0.0010065825, -0.0028353736, -0.006016241, 0.0126304515, -0.020490432, 0.03720701, -0.01395263, 0.0323701, 0.0027921868, -0.013367948, -0.008059304, -0.022178037, 0.024197847, 0.028436787, -0.048528574, -0.035240356, -0.0056973235, -0.018045401, -0.009853214, -0.013660288, -0.012092278, 0.008677207, -0.035639003, 0.0013246692, 0.00200486, 0.005155829, 0.002127776, 0.024450323, 0.024968563, 0.006046139, -0.024676222, 0.013833036, -0.019772869, 0.0014118733, -0.00999274, -0.04781101, 0.0064314976, 0.049299292, 0.016118608, 0.018563641, -0.010909627, -0.027825529, 0.00853768, 0.0014791448, 0.02446361, -9.478237e-05, -0.0010115657, -0.0011270071, -0.014165241, 0.008962903, -0.008457951, -0.006507905, 0.017248107, 0.039678622, -0.0010505997, -0.030243985, 0.008072592, 0.009680467, -0.000716318, -0.043957427, 0.0028237465, 0.03401784, -0.030190833, -0.021633219, 0.021420607, -0.003036358, 0.049113255, 0.020623315, 0.0040429407, 0.028968316, 0.0027589663, -0.027825529, 0.0067105503, -0.0044150106, -0.008311781, 0.017646754, 0.03951916, 0.01013891, 0.0010489387, -0.008012796, -0.008185542, 0.003471547, 0.0073151644, 0.021872407, 0.017101938, 0.012298246, 0.027400307, 0.021168131, -0.02528748, 0.015321316, 0.027506612, -0.019546969, -0.00399311, 0.03617053, 0.036648907, -0.004135958, -0.010079114, -0.014258258, -0.00024832357, -0.008431375, 0.021247862, 0.010491049, -0.0074746227, -0.00987979, 0.016145185, -0.028808858, -0.008457951, 0.059850138, -0.022364073, -0.027028237, 0.014949245, 0.009301753, 0.00025745924, 0.022948753, 0.00036521838, -0.0044648414, 0.020397415, -0.003468225, -0.008311781, 9.64434e-05, 0.025566533, -0.044170037, -0.036462873, 0.012610519, -0.013580559, 0.0070228237, -0.0022407258, 0.01744743, 0.009334973, -0.03983808, -0.0018935711, 0.000790649, -0.02601833, 0.013374592, 0.02758634, -0.0064879726, -0.00391338, 0.006215564, -0.0073350966, -0.027400307, 0.02305506, 0.02215146, 0.04310698, -0.03141335, 0.022988617, -0.012856351, 0.037924577, -0.025792431, -0.0042422637, 0.012311534, -0.014909381], "content_hash": "1d952d0789d8422f395303b008c258fe"}, {"content": "- `interpreting_ablation_results.md` - How to read ablation probability changes - `interpreting_attribution_scores.md` - Understanding attribution score values - `interpreting_attention_maps.md` - Reading BertViz patterns visually - `troubleshooting_and_faq.md` - Common issues and frequently asked questions - `recommended_starting_points.md` - Best models, prompts, and experiment order - `mechanistic_interpretability_intro.md` - Mech interp research context ## Notes\n\n- Large files will be automatically chunked (~500 tokens per chunk)\n- Embeddings are cached in `embeddings_cache.json` for faster subsequent loads\n- Delete `embeddings_cache.json` to force re-indexing of all documents", "source_file": "README.md", "chunk_index": 2, "embedding": [0.01765005, 0.03268948, 0.041429374, 0.0073423637, -0.049005844, -0.011080939, 0.024644803, 0.024346853, -0.021736234, 0.015039431, 0.03132742, 0.0076899733, 0.010378626, -0.011612995, 0.015848154, 0.048863962, -0.008598014, -0.0008849854, -0.0033377605, 0.06373314, 0.0117265, -0.046565484, 0.02092751, 0.04352922, -0.009470585, -0.02482925, 0.016742008, 0.025098823, 0.021821363, -0.03200845, -0.02030323, -0.022161879, -0.019352626, -0.036406774, -0.059703704, 0.001519018, -0.025269082, -0.00458277, -0.004799139, -0.016628502, 0.007817666, -0.04951662, -0.03748507, 0.04406837, -0.024204971, 0.0008734575, 0.017806118, -0.026006864, -0.022701027, 0.05277989, -0.019338438, 0.006182483, -0.025964301, -0.015138748, 0.012286931, -0.00024230687, -0.025212329, -0.013294289, -0.014755668, -0.043444093, 0.022388889, 0.00077813095, 0.027979016, 0.007590656, 0.003038036, -0.017749365, -0.06447092, 0.052183986, -0.022899661, -0.028887058, 0.056185044, -0.00026159387, 0.0007856684, -0.013322665, -0.002287838, -0.031015279, -0.030958526, 0.031242289, -0.052042104, -0.07485664, -0.008222029, 0.026872342, -0.004071997, -0.039755173, -0.010300592, -0.010222556, -0.01672782, 0.0069309077, -0.03192332, -0.031072032, -0.038534995, -0.012939585, -0.053319037, -0.0031054297, 0.020615371, -0.012982151, -0.015621144, -0.016614314, -0.008371004, 0.020062033, 0.016358929, -0.040719967, 0.0050970903, -0.033200253, 0.044267002, -0.028787741, -0.007959547, -0.016401492, -0.010619825, -0.026361568, -0.06867061, -0.00759775, 0.0047530276, -0.02299898, -0.0018515525, -0.013287195, -0.07133798, 0.00039837643, 0.02481506, -0.029539712, -0.024616428, 0.02650345, 0.011825817, 0.027297987, -0.024630615, -0.04276306, 0.009108787, -0.009037847, -0.03430693, -0.03552711, -0.0010162257, 0.00916554, 0.059533443, -0.044579145, -0.039357908, -0.011095127, -0.026673708, 0.046253342, -0.054255456, 0.014599598, 0.0007120674, -0.027822947, -0.006271159, 0.008186558, -0.04421025, 0.017962188, 0.014138483, 0.04446564, 0.0017726311, -0.018572278, -0.009413833, 0.010016829, -0.01977827, 0.0017806118, -0.041883398, -0.02558122, 0.020955887, 0.02490019, 0.0062250476, -0.04321708, 0.028078334, 0.035470355, -0.012591977, -0.016912265, -0.011385984, 0.02741149, -0.02329693, -0.016983205, -0.013244631, -0.054567594, -0.016231235, -0.059590198, 0.009896229, -0.048920713, -0.02047349, -0.020558618, -0.034193423, -0.025921736, 0.018061506, -0.007817666, -0.027993204, -0.014237801, -0.02749662, 0.048551824, -0.049317982, -0.02138153, -0.03396641, 0.012357872, -0.02223282, 0.026929094, -0.006867061, 0.021807175, 0.04880721, -0.010080675, -0.021438284, -0.039726797, 0.035924375, -0.01611773, 0.013563864, 0.024871813, -0.0033005166, 0.043415714, 0.033427265, 0.01749398, -0.020402549, -0.010094863, -0.011712312, -0.002188521, -0.049090974, -0.036605407, 0.015379947, -0.038194478, 0.03566899, 0.0086122025, -0.054879736, -0.0117832525, -0.03765533, 0.025567032, 0.0144080585, 0.019125616, 0.009718877, -0.020345796, 0.07065695, 0.02352394, 0.002192068, 0.020601183, 0.019693142, -0.04744515, 0.06696803, -0.0035115653, 0.04040783, 0.032093577, -0.0018045544, 0.030873397, 0.021225462, -0.036208138, 0.010031017, 0.00778929, 0.033625897, -0.02023229, 0.017848684, -0.0116200885, 0.010031017, 0.022275383, 0.05178672, 0.014557034, -0.0369743, -0.085639626, 0.05834164, 0.021934869, 0.034619067, -0.0073423637, 0.03161118, 0.0051361076, 0.009498961, -0.027368927, -0.040038936, 0.019905964, -0.004593411, 0.019735705, -0.043954864, 0.044692647, -0.019409379, 0.010016829, -0.0056575215, -0.004788498, -0.04375623, 0.038137726, -0.060441487, -0.083653286, -0.004384136, -0.00056575215, 0.01642987, -0.0029192104, -0.03836474, 0.003387419, -0.015876532, 0.02863167, -0.01824595, -0.0095415255, 0.018969547, 0.008477415, 0.016046789, 0.012634541, -0.027695253, 0.0015518281, -0.021892304, 0.02101264, 0.015252253, -0.0030327155, -0.0076261265, -0.02275778, 0.022431454, -0.009747254, -0.024956942, 0.017153464, 0.050396282, -0.00074310397, -0.029085692, -0.01153496, 0.037939094, -0.029085692, 0.0071118064, 0.011322138, 0.019579636, 0.026531827, -0.0058419676, 0.0020696952, 0.030646387, 0.020118786, 0.014017885, 0.0072430465, -0.036151387, -0.0051396545, -0.06259809, 0.0018196292, 0.023665821, -0.007399116, 0.04560069, -0.0037350284, -0.009917512, -0.03345564, 0.03498796, -0.0053170063, 0.008470321, 0.026077805, -0.0149543015, 0.03115716, 0.007377834, -0.00011278465, 0.07230277, 0.022034185, 0.015904907, 0.01611773, -0.030192368, -0.008768272, -0.029908603, 0.009222292, -0.02382189, 0.05607154, -0.07803478, -0.028390473, -0.030447753, 0.024573863, -0.025269082, 0.036775663, -0.0059270966, 0.012868645, 0.022913849, -0.0023729668, 0.010747517, 0.033342134, -0.032490846, -0.051077314, 0.009300327, 0.025935924, -0.01176197, -0.050935432, -0.020402549, -0.022630086, 0.002229312, 0.017664237, 0.095911846, 0.008136899, 0.015294817, -0.03260435, -0.062087312, 0.015904907, 0.021211272, -0.0027099352, 0.015011054, 0.0023676462, -0.024559675, -0.022672651, -0.08331277, 0.019806646, 0.008413568, 0.022431454, -0.014436435, 0.05760386, 0.025297457, -0.004806233, -0.026333192, -0.048381567, 0.009555714, -0.0043486655, -0.020175539, -0.007746726, -0.019849211, 0.033682648, 0.0031692763, 0.0111802565, 0.012542318, 0.018203387, 0.043784607, 0.05774574, -0.024148218, 0.011229915, 0.027922263, -0.0008650333, 0.028064145, 0.024219159, -0.021253837, -0.0056149573, -0.021197084, 0.05323391, 0.02970997, 0.023779327, 0.01015871, -0.0048168744, 0.060157724, -0.05422708, 0.014613787, -0.015294817, -0.009598278, -0.043273833, 0.03246247, 0.016940642, 0.02482925, 0.020359984, -0.022090938, -0.043500844, -0.055021618, -0.041599635, -0.034108292, 0.013982414, 0.03657703, -0.025723102, 0.05788762, -0.04301845, -0.05269476, 0.009555714, 0.029511336, 0.02528327, -0.007462963, -0.05513512, 0.050339527, 0.033342134, -0.00656911, -0.028957998, 0.030107237, -0.009264857, -0.039726797, -0.034704197, -0.059363186, -0.014429341, -0.016685255, -0.038137726, 0.0012529903, -0.02688653, 0.018444585, -0.041202366, -0.029227573, -0.015791403, -0.006115089, -0.039698422, 0.046906, -0.017990565, 0.08444782, -0.01618867, 0.043954864, -0.008619296, -0.009477679, 0.018870229, 0.01970733, -0.011449831, -0.037626952, 0.0018728347, 0.010669483, -0.025183953, 0.043869738, -0.032434095, -0.035186592, -0.018472962, -0.025950111, -0.06458443, -0.014656351, -0.042791437, 0.01650081, -0.05462435, -0.007995018, -0.041259117, -0.02275778, 0.017763555, 0.0011164295, 0.13132544, -0.011754876, -0.031270664, 0.03552711, 0.009804007, -0.037314814, -0.0053063654, 0.022516582, -0.017451415, 0.0074133044, 0.0033856456, -0.026659518, -0.054652724, 0.01993434, -0.02245983, 0.024701556, 0.01855809, 0.012322402, 0.022445641, -0.0020980716, 0.0069450955, -0.012428813, 0.00862639, -0.009768536, 0.040776722, -0.011244102, -0.0606685, -0.018756725, 0.03703105, -0.0038130633, 0.045175046, -0.010499225, 0.029199196, -0.006001584, 0.026205499, 0.010137428, -0.046338473, 0.019636389, -0.007193388, -0.014230707, -0.015110372, -0.035697367, -0.04321708, -0.03161118, -0.0022435, 0.016160294, -0.049346358, -0.00067482353, 0.00020340033, -0.042734686, 0.016373117, 0.0044125123, -0.0060335076, -0.02481506, -0.028390473, 0.008385192, 0.016543373, -0.0008605995, -0.0141881425, 0.0014259083, 0.030873397, -0.012308214, 0.005175125, 0.009995546, -0.003688917, 0.0022133503, -0.029596465, 0.061576538, -0.035754118, 0.03016399, 0.0013762498, 0.002555639, -0.03237734, 0.013251725, -0.026162934, 0.0050155083, -0.035158217, -0.0040471675, 0.029014751, -0.017224405, 0.010250933, 0.027837135, -0.023169236, -0.0061186366, -0.026673708, 0.028858682, -0.026077805, -0.006654239, 0.026616955, 0.0075126216, -0.027624313, -0.014393871, 0.007363646, 0.012769328, -0.03322863, -0.032632727, 0.025226517, -0.040578086, 0.05155971, -0.031072032, 0.0039620386, 0.03385291, -0.0042635365, -0.0084490385, 0.0072182175, 0.027482431, -0.010967434, 0.007746726, 0.0011199765, -0.015536016, 0.035725743, 0.016628502, -0.016642692, -0.025538657, 0.018586466, 0.0133155715, -0.024233347, -0.023126671, -0.0063243643, -0.0003343081, 0.019267498, -0.05644043, 0.013351042, 0.002099845, 0.0053489297, 0.0014737933, -0.026531827, -0.018969547, -0.02482925, -0.015394134, 0.039272778, -0.010931964, -0.0121095795, 0.014826609, -0.016387304, 0.040294323, -0.012059921, -0.0035629973, 0.007654503, 0.0074416804, 0.016202858, -0.02054443, -0.061009012, -0.0002972859, 0.014521563, -0.024857625, 0.039584916, 0.031554427, -0.0012379154, -0.049374737, 0.012599071, 0.0024403604, 0.024559675, 0.023538128, 0.0069450955, 0.03192332, -0.0007843382, 0.014393871, 0.0025290363, 0.039925434, -0.04094698, 0.003887551, 0.007867325, -0.015252253, -0.011662653, -0.02734055, -0.04361435, 0.028759364, 0.011988981, 0.022445641, -0.011513677, 0.0021246744, -0.038705252, 0.012996338, 0.02681559, -0.020090409, 0.0059448318, 0.0028092521, -0.008548356, -0.0134219825, 0.025609598, 0.013861815, 0.019565448, -0.010236745, 0.035413604, -0.03305837, 0.011009999, 0.033171877, 0.012428813, 0.010314779, 0.004462171, 0.0076403148, -0.018898606, -0.0063137235, 0.048097804, -0.010981622, 0.011513677, -0.034675818, -0.037768833, 0.010300592, -0.021395719, 0.014755668, -0.040776722, 0.029993733, 0.018089881, 0.0053985883, -0.028688423, -0.027468244, -0.0007071902, -0.011272479, 0.062144063, 0.0034299835, 0.014159766, 0.014996867, -0.039925434, 0.0033679104, 0.0013727028, 0.010236745, -0.012137956, -0.03683242, 0.024034712, -0.0066223154, 0.021934869, 0.017139276, -0.067251794, -0.039159272, 0.010676577, -0.008051771, 0.005150296, -0.03901739, 0.0101941805, -0.003100109, 0.048863962, 0.01650081, -0.049999014, 0.0057249153, 0.00551564, -0.035073087, -0.01336523, 0.013074373, -0.0070905243, -0.0049232855, -0.015904907, -0.012393342, 0.019097239, 0.0117832525, 0.04866533, 0.004657258, -0.030022109, -0.0062392354, 0.015947472, -0.010924869, 0.003688917, -0.0320652, -0.00047086898, -0.022786157, -0.030589635, -0.0017460283, 0.045487184, -0.0034477187, -0.03535685, -0.022942226, 0.015422511, -0.03192332, -0.0007856684, -0.02535421, 0.013372324, 0.00794536, -0.029199196, 0.043273833, -0.0035204329, 0.024261724, -0.013599334, -0.0026052976, 0.014308741, 0.0024846983, 0.03932953, -0.0076048444, 0.0030256214, -0.008817931, -0.00038862208, 0.024389416, 0.011967698, 0.01742304, 0.04202528, 0.004373495, -0.026162934, 0.0146279745, -0.013861815, 0.05924968, -0.045572314, 0.034704197, -0.0103076855, -0.011967698, 0.019622201, -0.036463525, -0.017508168, -0.00086237304, -0.028120898, -0.022431454, 0.025595408, -0.027695253, -0.03737157, 0.0028553635, -0.03291649, -0.066287, -0.015450887, 1.927315e-05, -0.024105653, -0.0060264133, -0.010704953, -0.0040329793, -0.09341473, -0.010293498, 0.032490846, -0.015323194, -0.0028252138, -0.010846835, 0.03285974, 0.037172932, 0.019749895, -0.013840533, 0.015167125, -0.026460886, 0.029482959, -0.029511336, -0.0013656087, -0.014982678, 0.014613787, -0.008619296, 0.01688389, -0.007356552, -0.00033896358, 0.051729966, 0.0035452622, 0.017607484, 0.002394249, 0.0028039317, -0.012031545, -0.004976491, 0.042904943, 0.015422511, -0.03504471, 0.0037385754, -0.028475601, 0.03300162, -0.011173162, 0.04611146, -0.004054262, 0.027610125, 0.030107237, -0.0064449636, 0.040265948, -0.049232855, 0.040209197, -0.022204442, 0.058852416, -0.005512093, 0.0035062446, -0.006689709, -0.0052673477, 0.0009364174, 0.019494507, 0.02108358, -0.015933285, -0.0011359381, 0.04446564, 0.0057178214, -0.0103076855, -0.03322863, 0.018047318, 0.03538523, -0.005962567, 8.8786735e-05, -0.00036379285, -0.01496849, 0.00397268, 0.013954038, -0.00094971876, 0.017479792, -0.06316561, -0.028163463, -0.021977432, 0.016060976, -0.008917248, -0.037059426, 0.030050486, 0.04665061, 0.033171877, -0.006544281, 0.004206784, -0.012967962, 0.027312174, 0.013053091, -0.0018196292, 0.016231235, 0.014911737, 0.024786685, 0.026517637, -0.03842149, -0.0066435975, 0.041117236, 0.045856077, 0.017352099, 0.0059270966, -0.010165804, -0.022857098, 0.0041500316, 0.0015935057, 0.039074145, 0.06957865, -0.003552356, 0.03742832, -0.0044089654, -0.007931172, 0.01042119, -0.0068315906, 0.0073210816, -0.033398885, -0.05391494, -0.0016981433, 0.02382189, 0.028461413, 0.040379453, 0.0160326, 0.029823475, -0.00670035, 0.008562543, 0.036208138, 0.04767216, 0.01336523, -0.047785662, 0.00020960764, -0.008129805, -0.027014224, -0.0031284853, -0.009193916, 0.030532882, 0.011052563, 0.012939585, 0.010364438, 0.015507639, 0.016018413, 0.040776722, -0.02894381, 0.01252813, -0.0054127765, 0.032490846, -0.0010286403, 0.010648201, -0.028730989, 0.015990036, -0.02007622, -0.012251461, -0.013712839, -0.016387304, -0.056837697, 0.001195351, -0.04267793, -0.0052212365, -0.012840269, -0.00740621, 0.012648729, 0.008441945, -0.022204442, 0.034675818, 0.025453527, 0.021934869, 0.006115089, -0.046537105, -0.027482431, 0.036066256, -0.015904907, 0.020941699, -0.005490811, -0.012882833, 0.03305837, 0.0020182633, -0.041571256, -0.01035025, 0.016287988, 0.0033945132, -0.004781404, -0.006799667, 0.036690537, 0.056326926, 0.02047349, 0.04650873, 0.016046789, 0.033654273, -0.008115618, 0.027439868, 0.016600126, 0.00028331947, 0.009222292, 0.014308741, 0.011967698, -0.005065167, -0.006881249, 0.009023659, -0.021509225, 0.03337051, 0.024999507, 0.010080675, -0.023850268, 0.055872906, -0.028503979, -0.0055191875, 0.037229687, 0.0013026488, -0.0042351605, 0.021182897, -0.02413403, 0.02314086, 0.005980302, -0.036463525, 0.0040046033, 0.022672651, -0.026432509, -0.013840533, 0.0068244967, -0.011513677, -0.023311118, -0.0033945132, -0.009406738, 0.01749398, -0.02200581, 0.01925331, -0.04361435, -0.013060185, -0.013734122, 0.0041464847, -0.0071224472, -0.011598806, -0.010690765, 0.029596465, -0.0001583973, 0.0013017621, -0.016075166, -0.0242901, 0.013379418, 0.017877059, -0.009889135, 0.0062321415, 0.0046678986, -0.00044071916, 0.01566371, 0.018316891, 0.012620352, 0.0028340814, 0.04727489, -0.009250669, 0.051077314, 0.0020838834, -0.006065431, -0.044891283, -0.030618012, -0.048324812, 0.02681559, 0.045856077, -0.0053489297, -0.03331376, 0.007902795, 0.025496092, 0.014684727, -0.012904115, -0.012776422, 0.011691029, 0.027170293, 0.0369743, 0.02726961, 0.006076072, 0.0048807207, -0.008165276, -0.009349986, -0.044437263, 0.0032934225, -0.005363118, -0.014911737, -0.013485829, 0.0066258623, -0.00557594, -0.034165047, 0.029426208, 0.020189727, -0.0019100787, -0.02810671, -0.0045295646, -0.01279061, -0.012003168, 0.01817501, -0.008931436, -0.008413568, 0.0035842794, 0.0058526085, -0.030305872, -0.0059377374, -0.009782724, -0.016089354, -0.025183953, -0.002346364, 0.026432509, -0.023254365, 0.013563864, 0.027666878, -0.023353683, -0.00016094674, 0.008037583, -0.020118786, -0.008768272, -0.0035133387, -0.04194015, -0.006203765, -0.011769064, 0.009172634, -0.026631143, -0.011605901, -0.02880193, 0.032717858, -0.0118896635, 0.0073352694, 0.018543903, -0.0107688, 0.028503979, -0.014613787, 0.0083922865, 0.0129324915, -0.020657936, -0.013322665, -0.027553372, -0.02106939, 0.0066613327, 0.004313195, 0.030107237, 0.0015695633, 0.02939783, -0.008619296, -0.029255949, 0.03132742, -0.00014287903, -0.010555978, -0.00064644724, -0.015564392, 0.0048310626, -0.008619296, 0.0076119383, -0.05155971, 0.051134065, -0.038194478, -0.013088562, 0.005554658, 0.00051653705, 0.004674993, -0.004401871, -0.0033111577, -0.028191838, 0.01977827, 0.00018023375, 0.025113013, 0.016131917, -0.0012547638, 0.014223613, 0.067308545, -0.027851323, -0.025183953, -0.020047845, -0.027595937, -0.018926982, 0.0077396315, 0.028901245, -0.049317982, 0.008030488, -0.026843965, 0.047019504, 0.024105653, 0.0020466396, 0.001566903, -0.0015074902, 0.008661861, 0.034817703, 0.018600654, 0.012301119, 0.01188257, 0.00065708836, -0.023467187, -0.0067677437, -3.0842584e-05, 0.0040223384, 0.031582806, -0.012024451, -0.00916554, 0.045316927, -0.01096034, -0.02803577, 0.026361568, 0.019920152, -0.02214769, 0.0031692763, -0.018160822, 0.003144447, -0.0041961432, -0.011896757, 0.0083922865, 0.034193423, 0.0023747403, -0.008122711, 0.023240177, 0.005196407, -0.016671067, -0.004784951, -0.005107731, 0.002188521, 0.005214142, 0.022275383, -0.008264593, -0.029908603, 0.013911474, -0.0015287723, 0.021821363, 0.042337418, 0.008534168, 0.020629559, 0.006349194, 0.032661103, -0.023680009, -0.014415152, 0.0363784, -0.04429538, -0.037683707, 0.029965356, 0.005859703, 0.020516053, -0.045543935, 0.031951696, -0.012123767, 0.038591746, 0.015777213, 0.02007622, 0.0025786948, -0.009988452, 0.029596465, 0.007569374, -0.005469529, -0.0058916262, -0.0057603857, 0.008704426, 3.4583598e-05, 0.013854721, 0.03405154, -0.027056787, -0.006395305, 0.029908603, -0.0054446994, -0.0101445215, -0.016287988, 0.058171384, -0.015110372, -0.0059377374, 0.00037687254, -0.029142445, 0.0049623027, -0.006764197, -0.0026992941, -0.008768272, -0.02322599, -0.018926982, -0.009151352, 0.00609026, -0.009711783, 0.0112370085, 0.00240489, 0.015294817, 0.01923912, -0.0052035013, 0.03376778, 0.006682615, -0.024006337, 0.019508695, 0.020416737, 0.035158217, -0.026475074, -0.025850795, 0.0062818, -0.014713104, -0.012513941, -0.027709441, -0.0024297193, -9.5188025e-06, 0.014081731, -0.0061505595, -0.012960868, -0.022913849, 0.0055014524, 0.0013265914, 0.01742304, -0.0006424568, -0.00570718, 0.008115618, 0.01240753, 0.019338438, -0.05048141, 0.012641635, -0.010293498, 0.032178707, -0.02688653, -0.03115716, 0.034278553, -0.010754612, -0.0048523447, 0.015252253, -0.010875211, -0.029681593, -0.036775663, -0.011038375, 0.039443035, 0.026489262, 0.015536016, -0.026900718, -0.0056397864, -0.018657407, 0.0003298743, -0.00068236096, 0.017408852, 0.0013478735, -0.020572806, -0.0007351231, 0.02275778, -0.020601183, -0.022786157, 0.02962484, 0.041514505, 0.026021052, 0.0022559147, -0.0050722607, 0.003038036, 0.016160294, 0.013833439, 0.017976377, -0.00049481145, 0.014422246, -0.0057391035, -0.005107731, 0.019608013, 0.01003811, -0.0050758077, -0.008165276, -0.0026159387, 0.012421719, -0.020189727, -0.023722574, 0.0018027808, -0.004267084, 0.010137428, -0.023807703, -0.006679068, 0.023609068, 0.02857492, 0.03635002, -0.009683407, 0.02840466, 0.0022470471, -0.0019295873, -0.025935924, -0.033796154, -0.01240753, -0.012485566, -0.0032632728, 0.016387304, -0.0056326925, -0.0048629856, -0.006115089, 0.012400436, 0.006264065, -0.005274442, 0.022019997, -0.019991092, 0.01657175, 0.0003580289, -0.029653218, 0.005668163, 0.016699443, -0.021495035, 0.004118108, -0.0028287608, 0.0342218, -0.049885508, 0.0028305345, 0.048949093, -0.0055014524, -0.014656351, 0.024559675, -0.008385192, -0.0029014752, -0.03941466, -0.011747782, 0.02458805, 0.022842908, -0.022019997, -0.02840466, 0.014642163, 0.022360513, 0.023694199, -0.029908603, -0.02146666, -0.009179728, 0.027028412, 0.00017158785, 0.022743592, 0.007817666, -0.003905286, -0.017366286, -0.025637973, 0.002298479, 0.024460357, -0.023339493, -0.031582806, -0.029142445, -0.016685255, -0.009349986, 0.00014698028, -0.008519979, 0.001323931, -0.03322863, -0.015238065, 0.010470849, 0.032121953, -0.0016538054, 0.01157043, 0.03887551, 0.018997923, -0.041429374, 0.0148975495, -0.023963772, 0.020643746, -0.011747782, -0.015138748, -0.008278781, 0.033711027, 0.014940114, -0.028986374, -0.000948832, -0.0331435, 0.0118896635, 0.008441945, 0.0020732423, -0.010208368, 0.0160326, 0.01871416, 0.0058455146, 0.015167125, -0.0005763933, -0.009399644, 0.029341077, 0.033285383, -0.026872342, -0.022346323, -0.00259643, 0.000807394, -0.013329759, 0.017735178, -0.015167125, 0.026319005, -0.035413604, -0.012641635, 0.010889399, -0.02192068, 0.006654239, 0.042053655, 0.01035025, 0.020586995, -0.011903851, 0.024573863, 0.0019597372, 0.0049410206, -0.006267612, 0.0059554726, 0.019863399, 0.0014977357, -0.015763026, 0.008115618, -0.0007484245, 0.023254365, 0.009839477, 0.027780382, 0.017082524, -0.0016644464, 0.017706802, 0.003295196, -0.03683242, 0.025212329, 0.010641106, 0.0077538197, 0.0073352694, 0.025311645, 0.008534168, -0.0077183493, -0.0064307754, -0.0017681973, 0.007292705, 0.025212329, 0.018274328, -0.010868117, -0.019593824, -0.001849779, 0.015081995, -0.051361077, 0.009811101, 0.03836474, -0.012074109, -0.01458541, -0.0055191875, -0.01176197, 0.022289572, 0.026035242, 0.023793515, -0.002802158, 0.00079675287, 0.0053311945, 0.0038130633, -0.0171109, 0.021268025, -0.011102221, -0.021906491, 0.00046554842, -0.036917545, -0.0010685445, 0.027482431, 0.010910681, -0.0144080585, -0.002695747, -0.010988716, 0.019281685, -0.01275514, 0.016869701, 0.009470585, 0.010080675, -0.0006429002, -0.010896494, -0.014081731, -0.00030571013, 0.0041961432, -0.015422511, 0.050339527, -0.02528327, 0.01073333, 0.0024740573, 0.03941466, -0.009995546, -0.025382586, 0.012067015, -0.015365758], "content_hash": "0fe3b79862e7d583840c65063bf187ac"}, {"content": "# Recommended Starting Points\n\n## Best First Model\n\n**GPT-2 (124M)** is the ideal starting model because:\n- It loads quickly and runs fast\n- It has a manageable size (12 layers, 12 heads = 144 heads total)\n- It's the most studied model in mechanistic interpretability research\n- Most educational examples and tutorials reference GPT-2\n\n## Good Starter Prompts\n\n### For Exploring Basic Predictions\n\n| Prompt | What It Tests |\n|--------|--------------|\n| `The cat sat on the` | Simple object prediction (mat, floor, bed) |\n| `The capital of France is` | Factual recall (Paris) |\n| `1 + 1 =` | Basic arithmetic |\n| `Once upon a time` | Creative story continuation |\n\n### For Exploring Attention Patterns\n\n| Prompt | What It Shows |\n|--------|--------------|\n| `The cat sat on the mat because it was` | Pronoun resolution: does \"it\" attend to \"cat\" or \"mat\"? |\n| `Alice gave the book to Bob because she` | Gendered pronoun resolution |\n| `The dogs in the park were` | Subject-verb agreement across a prepositional phrase |\n| `I went to the store and bought` | Sequential event prediction |\n\n### For Ablation Experiments\n\n| Prompt | Why It's Good for Ablation |\n|--------|---------------------------|\n| `The cat sat on the` | Simple enough that ablating one head can change the prediction |\n| `The president of the` | Factual prompts show clear ablation effects on knowledge retrieval |\n| `She picked up the phone and` | Action continuation is sensitive to Previous-Token head ablation |\n\n### For Attribution Experiments\n\n| Prompt | What Attribution Reveals |\n|--------|------------------------|\n| `The capital of France is` | \"France\" should have highest attribution for \"Paris\" |\n| `The doctor told the nurse that she` | Which noun drives the pronoun prediction? |\n| `The large red ball rolled down the` | Do adjectives or nouns matter more? |\n\n## Suggested Experiment Order\n\nIf you're new to the dashboard, follow this path:", "source_file": "recommended_starting_points.md", "chunk_index": 0, "embedding": [-0.00023927572, 0.006972066, 0.036465015, -0.0024710028, 0.011730778, -0.01868901, 0.002898111, 0.043022078, 0.008645915, 0.056827873, 0.010465017, -0.010409682, -0.015023143, -0.05234583, -0.007421654, -0.014594305, -0.017374832, 0.011295024, -0.010001595, 0.06479595, 0.012782122, -0.035275336, -0.018882679, 0.011212023, 0.032342643, -0.021220535, 0.0011974593, 0.044294756, 0.0013167729, -0.0061628087, 0.033310983, -0.022354878, -0.06866932, -0.016226655, -0.020501195, -0.0045823352, 0.0009181962, 0.03134663, -0.010907687, 0.048804466, 0.00254017, -0.029133277, -0.0050837984, -0.012415535, -0.031512633, 0.029963285, 0.0071795685, -0.023807393, -0.036299013, 0.03975738, -0.030405955, 0.038429365, -0.021829208, 0.011620111, -0.0064636865, -0.013446128, -0.025080072, -0.030488957, 0.0005174581, -0.03198297, 0.003427241, 0.016212821, 0.027819099, 0.026726255, 0.02119287, -0.0031280923, -0.046923112, 0.04360308, 0.003786911, -0.00085378415, 0.018979514, -0.020473529, 0.009911678, -0.02044586, -0.021884542, -0.016157487, 0.019629687, 0.035386004, -0.015258312, -0.049966477, -0.012048948, -0.007566905, 0.019851021, -0.027736098, -0.018426176, -0.008964085, -0.0025073155, -0.037156686, -0.0312913, -0.064574614, 0.00053388526, -0.03569034, -0.022465548, 0.046895444, 0.027417928, -0.020141525, 0.03054429, -0.034030322, -0.001505254, 0.027777597, 0.02299122, -0.011627028, -0.029990952, 0.003275073, 0.035469003, -0.0065397704, 0.003183426, 0.009088586, -0.048168126, -0.039785046, -0.056993876, 0.0021511037, -0.009309922, -0.001869247, -0.00736632, -0.0063495603, -0.043852083, -0.0020387068, 0.008147911, -0.049136467, 0.00025289302, 0.013204042, -0.0046203774, 0.0051495074, -0.019325351, -0.034196325, -0.00026456502, -0.054891188, 0.01625432, 0.018315507, -0.0070758173, 0.040670387, 0.030461289, -0.02131737, -0.010548017, -0.0025712952, -0.01772067, 0.011841446, -0.050077144, 0.01729183, -0.0025833996, -0.06534929, -0.019657353, 0.030599624, -0.08012343, -0.027085925, -0.0032266558, 0.029714283, -0.049496137, 0.03992338, -0.002766693, 0.037156686, -0.018910347, -0.028441604, -0.019698855, -0.04188773, -0.024097895, 0.013197126, -0.020293694, -0.072210684, 0.047393452, -0.01128119, -0.013923382, 0.010070763, -0.03569034, 0.054946523, -0.048168126, 0.0011793029, -0.0316233, -0.047365785, 0.025135405, 0.005581803, 0.014677307, -0.02838627, 0.03787603, 0.016641658, -0.094676234, -0.00980101, -0.0004599627, -0.050160144, 0.00047422847, -0.002135541, -0.024457566, -0.014594305, -0.0015441605, 0.0040359134, -0.036852352, 0.023005052, -0.0025661078, -0.023461556, 0.023779726, -0.017513167, 0.07381537, 0.039010372, -0.0029862993, -0.03972971, 0.08228145, 0.027639262, 0.026836922, 0.005253258, -0.0013409815, 0.036714017, -0.005408885, -0.035552006, 0.007103484, 0.0037627025, -0.020515028, 0.017181164, -0.08222611, -0.047393452, 0.019145515, -0.05776855, -0.000602188, 0.028524606, -0.07862941, -0.010748602, -0.054918855, -0.006110933, 0.022507047, 0.051515825, 0.009503591, 0.02523224, 0.027473262, -0.03748869, -0.00018610334, 0.027417928, 0.047919124, -0.04722745, 0.009116254, -0.009593508, 0.02946528, -0.0075115715, -0.006432561, -0.0037350354, -0.01729183, -0.026740087, 0.051488157, 0.002228917, 0.008300079, -0.015728649, 0.033476986, 0.028662939, -0.006252726, -0.015936151, 0.060258575, 0.00081617443, -0.030848626, -0.06656663, 0.047448784, 0.064021274, 0.05505719, -0.023267888, -0.023143386, 0.0076775732, -0.0040566637, -0.046978448, -0.020058524, -0.04155573, -0.03181697, 0.013356211, -0.0115301935, 0.048002124, -0.0023361263, 0.03607768, -0.020653363, -0.03544134, -0.061254583, -0.035469003, -0.034362327, -0.07447938, -0.0020542697, -0.026172915, 0.033034313, 0.016032986, -0.05541686, -0.029880283, -0.0154934805, -0.00630806, -0.05804522, 0.0048762965, -0.032148972, -0.024817236, 0.04238574, 0.020100025, -0.022410214, 0.016461823, 0.00021398641, -0.0026093372, -0.01996169, -0.006764564, -0.007269486, -0.04210907, 0.012166533, -0.04028305, 0.0049731308, 0.012214949, -0.013736631, 0.010146847, 0.011848363, 0.03757169, 0.026006915, -0.021912208, 0.047697786, 0.009856344, 0.042275067, 0.010679435, 0.0066538965, 0.019477518, 0.025536576, 0.024042562, -0.017042829, 0.017789835, 0.004312583, 0.007138068, 0.012014364, -0.0008481643, 0.027431762, -0.034085657, 0.013003457, -0.01684916, -0.05029848, -0.016240489, 0.01848151, 0.041306727, 0.01121894, 0.027791431, 0.017333332, 0.020183025, -0.008998669, -0.014538972, -0.018772012, 0.032287307, -0.019062515, 0.0594839, -0.040338382, -0.026643254, -0.020653363, 0.006806065, -0.018315507, 0.0807321, -0.08554615, -0.004447459, -0.03289598, 0.0148294745, -0.041140724, 0.08526948, 0.010236764, 0.03118063, 0.05265017, 0.010783186, -0.022866718, 0.0074354876, -0.001974727, 0.011308857, 0.019325351, 0.043160412, -0.02495557, -0.0102160135, 0.00078115845, -0.0041223727, -0.015659483, 0.011032188, 0.028801274, 0.008971002, 0.0016824952, 0.0007461425, -0.031125296, -0.0010625831, -0.008037243, 0.030295288, -0.0016652034, 0.0008732375, 0.0044820425, -0.0036382014, -0.04144506, -0.02063953, -0.03972971, 0.008452247, -0.014594305, 0.047116783, 0.009870177, -0.004921255, 0.018315507, -0.024167063, 0.059096564, 0.018260173, 0.01464964, -0.012014364, -0.057989884, 0.015548815, -0.0021096035, -0.023212554, -0.025702577, 0.01608832, -0.023046553, 0.07957009, -0.022520881, 0.032148972, 0.05693854, 0.060977913, -0.011461026, -0.028801274, -0.049883474, -0.009475923, -0.007836658, 0.0075945724, 0.034472995, 0.034804996, 0.01564565, -0.020750197, 0.07420271, -0.027736098, 0.029935619, 0.013176375, -0.012180366, 0.031678636, 0.055638198, 0.013625963, 0.010465017, 0.041583396, -0.013916465, -0.061199248, -0.027888265, -0.051294487, -0.06999733, 0.03552434, 0.03333865, -0.01361213, 0.018938014, -0.033310983, -0.010105346, 0.05389518, 0.054559186, -0.018730512, 0.035026334, -0.039563708, 0.026546419, 0.028081933, 0.022202712, -0.0023395848, 0.044958763, -0.015714817, -0.025038572, 0.02343389, -0.027182758, -0.02857994, -0.048002124, -0.05704921, -0.012360201, -0.02846927, -0.011094439, -0.0021303536, -0.006902899, -0.0053293426, 0.005751263, -0.030571958, 0.056634206, 0.0010565309, 0.08886618, 0.0042157485, 0.026532587, 0.006740356, 0.001610734, -0.00804416, 0.002745943, 0.051598825, 0.0022600424, 0.00072106934, 0.003928704, -0.036797017, 0.022064377, -0.03397499, 0.02703059, -0.032065973, -0.020418195, -0.006335727, -0.00018015927, 0.038069695, 0.029658949, -0.03425166, 0.002531524, 0.032591645, 0.05721521, 0.033725988, -0.01837084, 0.03812503, 0.0034099491, -0.0059518483, 0.0018208298, -0.014704973, -0.013134875, -0.014718806, 0.009939345, -0.016295822, 0.01505081, 0.004163873, 0.030212287, -0.019754188, -0.00020371939, 0.016779993, 0.016351156, -0.027238093, -0.024222396, 0.07132535, -0.007629156, -0.00090695656, 0.04352008, 0.03405799, 0.0039044954, 0.021746207, -0.04280074, -0.030793292, -0.040421385, 0.013888799, 0.0036797016, 0.039619043, -0.016212821, 0.06191859, 0.005478052, 0.03314498, -0.026698587, 0.034749664, -0.03245331, 0.004136206, -0.008182495, -0.025979247, -0.01729183, -0.050353814, -0.029492948, -0.03740569, 0.013058791, -0.018550677, 0.004277999, -0.016143654, -0.042966742, -0.0006406623, -0.009648842, -0.005484969, -0.043381747, -0.009054003, 0.002152833, 0.02694759, 0.011205107, 0.010928437, -0.0037730776, 0.010202181, 0.030710293, -0.016102154, -0.019366851, 0.025495077, 0.042247403, -0.024499066, 0.012899706, -0.021372704, 0.015617982, -0.0013781589, 0.018744346, -0.035552006, 0.022327213, -0.022285713, 0.006591646, -0.0033961157, 0.0007971534, 0.010409682, 0.012927373, -0.0054469267, 0.004513168, -0.018578343, 0.0039874963, -0.023530724, -0.0068925237, -0.0037281187, 0.03082096, -0.005772013, 0.015548815, -0.020196859, -0.0040289965, 0.0014931497, -0.0051495074, -0.0003123337, -0.017803669, -0.012650704, -0.013951049, 0.04163873, -0.017305665, 0.010949188, 0.015258312, -0.003928704, -0.011723861, -0.006989358, 0.006110933, -0.018951846, 0.0045200847, 0.010548017, -0.014774141, 0.00060651096, 0.0098286765, -0.01329396, -0.034638997, 0.009787177, 0.008701249, -0.0033269483, -0.024429899, 0.004997339, 0.0018640594, 0.012602286, -0.04750412, 0.012885872, 0.0060175573, -0.011039105, 0.015382813, -0.032010637, -0.015341313, -0.01680766, -0.011177439, 0.0076844897, -0.011461026, -0.015396646, 0.031318966, -0.022631548, 0.006301143, 0.011751529, -0.01884118, 0.0044889594, -0.0018813512, 0.00096401956, 0.00366241, -0.0066054794, 0.014117051, -0.006211226, -0.012145782, 0.009441339, 0.059373233, 0.014220802, -0.020694863, 0.00201104, 0.013003457, -0.0010262702, 0.017070496, -0.007836658, -0.012104282, 0.0075046546, -0.016932162, -0.010437349, 0.018287841, -0.010153763, -0.008175578, 0.016005319, -0.028441604, -0.025329074, -0.0490258, -0.0119313635, 0.06081191, -0.006121308, 0.012000531, -0.014608139, 0.023461556, -0.026809255, -0.033227984, 0.0027632345, -0.026892256, 0.01187603, -0.022368712, -0.033117317, -0.015507314, 0.02076403, 0.002341314, 0.036714017, 0.04304974, -0.014525138, -0.070108004, -0.03704602, 0.022935884, -0.0027839849, -0.0116685275, -0.02478957, -0.0016876827, -0.0315403, -0.0047794622, 0.01837084, -0.0013556796, 0.016240489, -0.04116839, 0.017374832, 0.015590315, -0.01323171, -0.017734502, 0.005211758, 0.021289703, 0.019643521, -0.0028081934, -0.047172114, -0.001416201, 0.021372704, -0.017028995, 0.0040013297, -0.014843308, 0.04587177, -0.021607872, -0.068281986, -0.041362062, -0.016517157, 0.008791167, 0.025246074, -0.011094439, 0.011793029, -0.030571958, 0.05480819, 0.018896513, -0.035911676, -0.05137749, -0.001232043, 0.011364192, 0.020100025, -0.033725988, 0.0127198715, 0.0045477515, -0.0016461824, -0.01983719, -0.035303004, -0.06081191, 0.021483371, 0.011025271, 0.02184304, 0.000483739, -0.032425642, 0.0068233567, -0.008009576, -0.045816436, 0.012685288, 0.02846927, 0.010548017, 0.0027217343, -0.014027134, -0.013625963, 0.014006384, -0.029990952, 0.017859003, 0.0107416855, -0.027417928, 0.001850226, -0.010112263, 0.0071173175, 0.029520614, 0.012014364, 0.014123968, -0.036243677, 0.04352008, -0.025412075, -0.0560532, 0.0014473263, 0.01293429, -0.010963021, -0.017305665, 0.035718005, -0.004900505, -0.001945331, -0.017111996, -0.018619845, 0.04731045, -0.0068890657, 0.009109337, -0.012823622, 3.1314426e-05, -0.041915398, -0.035828676, 0.061143916, -0.0048278794, -0.018025005, 0.02072253, 0.014511305, 0.01088002, 0.03250864, -0.03189997, 0.050685816, -0.0053051338, 0.03640968, 0.033476986, -0.01772067, -0.004921255, -0.012014364, 0.012789038, -0.018938014, -0.0060763494, 0.0061143916, -0.00459271, -0.0056336788, -0.008389996, -0.030156953, -0.025246074, -0.019076347, -0.0210822, -5.4874545e-05, 0.008244745, -0.0013219605, -0.021773875, -0.027542429, -0.11509443, -0.014151635, 0.0007578145, -0.021137534, -0.009254588, -0.0035137, 0.018730512, 0.022036709, -0.006394519, -0.003984038, -0.0042503322, -0.024609733, 0.0098286765, -0.0034981375, -0.005069965, -0.04802979, 0.020528862, -0.028856607, 0.014746474, -0.04415642, -0.0053293426, 0.0033442401, -0.028040433, -0.001346169, -0.04814046, -0.038899705, -0.028441604, 0.006155892, 0.010056929, 0.013999467, -0.03109763, 0.011813779, -0.0016479115, 0.007037775, -0.025246074, 0.037710026, -0.018855013, -0.0022444797, 0.03289598, -0.014663473, -0.016973661, -0.0108731035, 0.02929928, -0.029852618, 0.060922578, -0.035745673, 0.0137712145, 0.01951902, -0.028123435, 0.014953976, 0.03281298, 0.024941737, -0.007193402, 0.023669058, 0.016987495, 0.0033770946, -0.010776269, -0.030018618, 0.017803669, 0.019062515, -0.0024952113, 0.011350358, 0.028012767, -0.027085925, -0.005139132, 0.01233945, 0.011848363, 0.017250331, -0.008991752, -0.021566372, -0.032148972, 0.018052671, -0.011156689, -0.0015899839, -0.01505081, 0.01972652, 0.024706569, 0.03784836, -0.017471666, 0.020196859, 0.024028728, -0.010561851, -0.017776003, 0.012623037, 0.026864588, 0.029575948, 0.0124570355, -0.022963552, 0.018495342, 0.020736365, 0.027597763, -0.023503058, -0.0007807262, 0.0038491616, -0.03674168, 0.017665334, 0.011675444, 0.023973394, 0.039978713, -0.0060832663, 0.010271348, 0.029105611, -0.037627026, 0.027763763, 0.0004802806, 0.051681824, -0.009897844, -0.006135142, -0.0026698587, -0.0053224256, 0.025605744, 0.026020747, -0.007878158, 0.008618249, -0.022783717, 0.0076844897, 0.07569672, 0.034389995, 0.021123702, 0.0034341577, 0.024402233, -0.012055865, -0.02072253, 0.0072833193, -0.041500393, -0.0058031385, -0.010707102, 0.0124570355, -0.014912475, 0.0145666385, 0.02929928, 0.02938228, -0.02531524, -0.0044716676, -0.008957169, -0.017983504, 0.015853152, -0.01195903, -0.027680764, 0.0061524333, -0.0076429895, 0.026089914, 0.01293429, -0.026726255, -0.023586057, -0.024623567, -0.010050013, -0.03992338, 0.017111996, 0.014621972, 0.026823089, 0.020943865, -0.0051252986, 0.0070965677, -0.028773608, 0.0079749925, 0.013667463, -0.020404361, 0.0021995208, -0.012353284, 0.009282255, 0.014497471, 0.033117317, -0.016060652, 0.05688321, -0.041583396, 0.0045304596, 0.009220004, 0.014386804, -0.009067836, -0.025370575, 0.021690873, 0.042330403, 0.02614525, 0.027473262, 0.034638997, 0.015576482, -0.014220802, -0.01761, 0.023392389, 0.008293162, 0.022410214, 0.013861132, 0.002576483, 0.0244714, -0.046812445, -0.012111198, 0.0028064642, -0.015936151, 0.017471666, 0.0031747802, 0.0041915397, -0.006104016, 0.052124497, -0.0063876025, 0.013280126, 0.038623035, -0.017623834, -0.0010487495, -0.029077943, -0.026850756, 0.0059518483, -0.00946209, -0.037073687, 0.008701249, 0.029492948, 0.018011171, -0.0037350354, 0.0017793294, -0.014414471, -0.042607073, 0.0058792224, -0.008272412, 0.012512369, -0.007732907, 0.009441339, -0.013141791, -0.008230912, -0.020971533, -0.025854746, -0.06966533, -0.03156797, -0.02902261, 0.022590049, -0.015631815, 0.026089914, 0.012069698, -0.0056336788, 0.024761902, 0.0043298746, -0.0048970464, -0.01128119, -0.031401966, 0.00289984, -0.018813513, 0.028690606, 0.011710028, 0.020473529, 0.02152487, 0.010243681, 0.048085123, 0.0015424313, -0.0132524595, -0.0050561316, 0.007110401, -0.027113592, 0.033753656, 0.0042987494, 0.013764298, 6.1980405e-05, -0.012152699, 0.04008938, 0.03793136, -0.03541367, 0.0087704165, -0.015797816, -0.0024243149, 0.02846927, 0.007241819, -0.0008308725, 0.015147644, 0.025453575, -0.008438414, -0.04396275, -0.009524341, -0.01581165, 0.004710295, 0.00493163, 0.0015078477, -0.014483638, -0.016337322, 0.038623035, 0.0031765094, 0.03010162, -0.006602021, -0.034085657, -0.0018882679, -0.017098162, 0.015562648, -0.028012767, 0.005069965, 0.0038975787, -0.0052636336, -0.009434423, 0.014497471, -0.008839584, -0.026089914, -0.03793136, -0.02591008, 0.006252726, 0.00455121, 0.0019505186, -0.026103748, -0.00812716, -0.019712688, -0.0014827745, -0.006799148, -0.013964883, 0.021137534, -0.045816436, 0.00368316, -0.0017897045, 0.009738759, -0.034500662, 0.019117849, 0.01668316, 0.021165201, -0.026892256, 0.016503323, 0.011820695, -0.029160945, 0.013529128, 0.00069210556, 0.008493748, 0.006373769, -0.005187549, -0.00040419656, -0.021068366, -0.020017024, -0.012609203, -0.02857994, -0.008244745, 0.0056302203, -0.0026577543, -0.015216812, -0.0019764563, -0.0081064105, 0.018052671, 0.012885872, -0.005021548, -0.016752327, -0.011834529, -0.009475923, 0.013729714, -0.046010103, 0.014123968, -0.051211488, -0.021607872, 0.030571958, -0.013383877, 0.01291354, -0.014068634, -0.005491886, -0.008735833, 0.025190739, 0.0022496672, 0.017693002, -0.0077951574, 0.007490821, 0.040255383, 0.060977913, -0.01505081, -0.049413137, -0.0045062513, 0.0071657347, 0.015922317, -0.0037142853, 0.0026508376, -0.014884808, -0.020376693, -0.009531258, 0.029492948, -0.0036105344, 0.0071726516, -0.0009648842, -0.012809789, -0.023738226, -0.02478957, 0.0523735, 0.0027494011, 0.03344932, 0.036437348, -0.023060387, 0.004288374, 0.017153496, 0.032480977, 0.026518753, -0.04822346, 0.0140133, 0.032397974, 0.003804203, -0.062748596, -0.0055541364, 0.017388666, -0.026532587, 0.018329341, -0.016032986, 0.01121894, -0.03272998, 0.023599891, -0.008867251, 0.0069513163, 0.031706303, -0.035386004, 0.0007569499, 0.016558658, 0.007290236, -0.000644553, 0.009607341, -0.007214152, -0.008182495, 0.003088321, -0.040864054, -0.021746207, -0.009330672, -0.023572225, -0.0018277465, 0.032868315, 0.013328544, -0.0034134074, -0.04459909, 0.029243944, -0.015825484, 0.003539638, 0.0120627815, -0.04244107, -0.03082096, 0.007933492, 0.00031816968, 0.02626975, -0.033227984, -0.018094173, -0.01680766, -0.0007085328, 0.027251925, 0.0091646705, -0.01299654, 0.007359403, 0.025481243, 0.015299812, 0.012823622, -0.01193828, -0.049523804, -0.0032560518, 0.0033390527, -0.0022617716, 0.034888, 0.002671588, -0.036299013, 0.0061143916, -0.013805798, 0.02135887, 0.015673315, 0.03253631, 0.0070654424, 0.013785047, -0.014801808, 0.027556263, -0.014898642, -0.005087257, 0.01513381, -0.0025955038, -0.036575682, -0.008825751, -0.016752327, 0.01680766, -0.02599308, -0.008597499, -0.00043316037, -0.00023668194, 0.008327746, -0.0055783447, 0.011627028, -0.0006674647, 0.019325351, 6.2845e-05, -0.020943865, 0.05594253, -0.013328544, -0.044931095, -0.0015605878, 0.019117849, 0.004277999, 0.0043298746, -0.00034324286, 0.002135541, 0.020127691, -0.01987869, -0.050132476, 0.010119179, -0.03281298, 0.04119606, 0.00918542, -0.01473264, 0.022922052, 0.026449585, 0.00022911676, -0.017527001, -0.044101086, 0.011364192, 0.012000531, 0.031401966, -0.019809522, -0.018661344, -0.01868901, -0.016212821, 0.024485232, 0.0034721997, -0.023420056, -0.0019245808, -0.0099600945, -0.014538972, 0.05721521, 0.013591379, -0.007276403, -0.028275602, -0.0014369512, -0.021981375, -0.01157861, -0.0048866714, -0.0007707834, -0.0127198715, -0.0152306445, 0.017305665, -0.0009968741, 0.0279851, -0.009807927, 0.004955839, -0.0068994407, 0.0009847698, 0.014884808, 0.0043506245, -0.022797551, -0.012138866, 0.008217079, -0.009088586, -0.0019107474, 0.0035621172, -0.017028995, 0.006895982, -0.016821494, 0.016766159, -0.026933756, -0.025301408, -0.013944132, 0.031208297, -0.019104015, 0.0045270016, 0.007649906, 0.0023637933, -0.029714283, -0.060147904, 0.0050077145, 0.005401968, -0.0025367117, 0.017084328, -0.012387868, -0.020888532, 0.04127906, -0.026366584, -0.011295024, -0.01469114, 0.0004724993, -0.01608832, -0.03272998, 0.041915398, -0.0010764165, -0.009925511, 0.0020542697, -0.009060919, -0.006550146, -0.009628092, -0.012028198, -0.007573822, 0.029880283, -0.00980101, -0.0072971527, -0.0043852082, -0.000759976, 0.009669592, -0.01928385, -0.02000319, 0.03261931, 0.0064706034, -0.011205107, 0.016434157, 0.007234902, -0.037737694, 0.030322956, 0.026574086, 0.0076706563, -0.014511305, 0.0139718, 0.0032819896, 0.012138866, -0.0070965677, -0.015880818, -0.0046203774, 0.021538705, -0.0037938277, -0.0083692465, 0.04352008, -0.007144985, 0.017803669, -0.025674911, 0.05289917, 0.007013567, 0.0055887196, -0.034362327, -0.0038560783, 0.050409146, -0.01121894, -0.035386004, -0.0060936413, -0.006674647, -0.02929928, -0.022977386, -0.0013098562, -0.012214949, 0.013722797, -0.044460755, -0.016641658, 0.006778398, 0.013031124, 0.021414204, 0.02411173, 0.0072556525, -0.02383506, -0.011620111, -0.008396913, -0.00812716, 0.00153465, -0.013003457, -0.01892418, -0.0071657347, 0.025757913, 0.023295555, -0.010527267, 0.0006155892, -0.037903693, 0.0022237294, -0.008728917, -0.0038076611, 0.0043990416, 0.0043091243, 0.014815641, -0.0015666399, 0.01729183, 0.012899706, -0.005118382, 0.001020218, 0.032563977, 0.006318435, 0.0013954508, 0.014234635, 0.001657422, -0.018716678, -0.032038305, -0.050243143, 0.045041762, -0.014787974, -0.019809522, 0.030322956, -0.014220802, 0.028524606, 0.025190739, 0.007525405, 0.030710293, -0.04459909, -0.016295822, 0.0073386533, -0.0032422184, -0.0043817502, -0.018938014, 0.010568767, -0.00031492749, -0.019947857, 0.013736631, -0.028552271, 0.0046307524, -0.009330672, 0.01636499, 0.025744079, 0.028234102, 0.013079541, 0.026338918, 0.0016487761, 0.017706836, 0.00048762964, -0.021787707, 0.028413936, 0.03972971, 0.027141258, 0.030239955, 0.0007431164, -0.013370044, -0.02288055, -0.004305666, 0.033476986, 0.0010003324, -0.02510774, -0.01013993, -0.008452247, -0.021690873, 0.0011723861, 0.046369776, 0.014262303, -0.017194998, -0.0001514116, 0.023337055, 0.0065363124, 0.023682892, -0.0037696192, 0.0022859802, -0.00078331993, 0.0016799014, 0.004731045, 0.019076347, 0.021566372, -0.012540036, -0.037737694, -0.024056396, -0.006550146, -0.0020421653, 0.003456637, 0.024499066, -0.02631125, -0.056136202, -0.015825484, -0.042966742, 0.0041604144, 0.017111996, 0.04127906, -0.012470868, -0.016019152, 0.008860335, -0.043464746, -0.03668635, 0.03486033, 0.0140548, 0.030046286, -0.029243944, 0.0019003722, -0.009814844, 0.03452833, -0.0106102675, -0.020736365, -0.0019401434, -0.0073317364], "content_hash": "89df1e1e3909d26f35a5acae2121dcbf"}, {"content": "highest attribution for \"Paris\" | | `The doctor told the nurse that she` | Which noun drives the pronoun prediction? | | `The large red ball rolled down the` | Do adjectives or nouns matter more? | ## Suggested Experiment Order If you're new to the dashboard, follow this path:\n\n1. **Experiment: Your First Analysis** -- Learn the basics with GPT-2 and a simple prompt\n2. **Experiment: Exploring Attention Patterns** -- Understand what attention heads do\n3. **Experiment: Your First Ablation** -- Remove a head and see what happens\n4. **Experiment: Token Attribution** -- See which input tokens drive predictions\n5. **Experiment: Comparing Heads** -- Systematically compare head categories\n6. **Experiment: Beam Search** -- Explore alternative generation paths\n\n## After the Basics: Cross-Model Comparisons\n\nOnce you've completed the guided experiments, try comparing models to see how architecture affects behavior:\n\n- **GPT-2 vs GPT-Neo 125M**: Same size and PE type, but GPT-Neo alternates local/global attention \u2014 see how attention scope matters\n- **GPT-2 vs Pythia-160M**: Same size but different positional encoding (absolute vs rotary) \u2014 see how RoPE changes attention patterns\n- **GPT-2 vs OPT-125M**: Same size but OPT uses ReLU instead of GELU \u2014 compare MLP behavior\n- **GPT-2 vs GPT-2 Medium**: Same architecture at different scales \u2014 see how head specialization changes with more layers\n- **Pythia-160M vs Qwen2.5-0.5B**: Both use rotary PE but different normalization (LayerNorm vs RMSNorm) and activation (GELU vs SiLU)\n- **Try longer prompts**: See how attention patterns change with more context\n- **Combine techniques**: Use attribution to find important tokens, then ablate heads to find the components that process those tokens", "source_file": "recommended_starting_points.md", "chunk_index": 1, "embedding": [0.00020510766, 0.016524354, 0.039003175, -0.005712298, 0.0066560386, -0.009487258, -0.016253697, 0.03558434, 0.010313476, 0.03435926, 0.020997327, -0.033789456, -0.023874843, -0.019871961, -0.041510317, -0.02400305, -0.012336285, 0.00056713435, 0.0077778427, 0.03558434, -0.019473096, 0.002375376, 0.026381986, 0.048490435, 0.044131424, 0.009408911, 0.0072187213, 0.035669815, 0.010577011, -0.0014387585, 0.026823586, -0.017920377, -0.03433077, -0.0010033916, -0.028775169, -0.005117564, 0.003060923, 0.008404628, 0.007606901, 0.015156821, 0.012991562, -0.020427521, -0.049715515, 0.038917705, -0.03931657, 0.0028045105, 0.014294991, -0.050199848, -0.017564248, 0.04202314, -0.03108288, 0.06347632, -0.031823628, 0.028134137, -0.021082796, 0.0030929744, -0.031823628, 0.013119767, -0.0033903418, -0.013390426, -0.021994486, -0.032194, 0.046467625, 0.033675496, -0.0046296683, -0.030342134, -0.06216576, 0.009964471, 0.0026709624, -0.011709499, 0.032165512, 0.025071435, -0.0064708516, -0.0034366383, -0.01151719, -0.041282397, 0.030541565, 0.02586916, -0.015156821, -0.037749603, 0.019430362, -0.02994327, 0.016951708, -0.012243692, -0.0075784107, -0.034900576, -0.019487342, -0.056011863, -0.05011438, -0.06974417, -0.008803492, -0.011068469, -0.010726585, 0.02028507, 0.032364942, 0.0153989885, 0.025042944, -0.025484543, 0.006702335, 0.029715348, 0.04153881, 0.006752193, -0.024743795, 0.009686691, 0.026624154, -0.026495948, 0.011616906, 0.0037856442, -0.02309136, -0.02464408, -0.07088378, -0.00085737894, -0.0383479, 0.0013862295, -0.0005795989, 0.020997327, -0.014857674, 0.035812266, 0.009088395, -0.031624194, -0.011816338, -0.0033226772, 0.025242375, 0.012827743, 0.026353497, -0.0339604, 0.03965845, -0.03108288, 0.02095459, 0.008960188, -0.0035844317, 0.02450163, 0.0073077534, -0.041994654, -0.013048542, -2.8239863e-05, 0.007820578, 0.006901767, -0.06131105, 0.020512993, -0.010698095, -0.03105439, -0.020740913, 0.033476062, -0.07794937, 0.004426675, -0.006599058, 0.028447531, -0.034530204, 0.026381986, 0.004259295, -0.007863314, -0.023233812, -0.002533853, -0.032535885, -0.07652485, -0.004451604, 0.036125656, -0.008903208, -0.07014304, 0.03623962, 0.026268026, -0.015498705, -0.022193918, -0.029487425, 0.052080207, -0.03960147, -0.0134545285, -0.04404595, -0.046581585, -0.020085638, -0.024487384, 0.02358994, -0.018319242, 0.029344974, -0.0022863438, -0.078804076, -0.012998684, -0.00682698, -0.033931907, -0.014116927, 0.0052600154, -0.00857557, -0.003035994, -0.029544406, -0.031168351, -0.021267984, 0.021695338, 0.007193792, -0.011132572, 0.045869328, 0.017578494, 0.0766958, 0.03843337, -0.015698137, -0.044387836, 0.086382486, 0.046296682, -0.012336285, 0.0039494634, 0.028447531, 0.021211004, 0.04256446, -0.04906024, 0.017464533, -0.013062787, 0.016139736, -0.010662482, -0.060570307, -0.042051632, 0.032592863, -0.05364717, 0.032564376, 0.03655301, -0.08735116, -0.021609867, -0.06598346, 0.017877642, 0.043590106, 0.051538892, 0.01207275, 0.023233812, 0.030541565, 0.018461693, -0.020099882, 0.039117135, 0.06324839, -0.028518757, 0.017891888, -0.030769488, 0.029102808, 0.014886164, -0.018490182, -0.023219567, -0.027222449, -0.05245058, 0.044273876, -0.01668105, 0.006723703, -0.031681176, 0.0041097207, 0.017806416, 0.00721516, 9.287159e-05, 0.022350613, 0.013190993, -0.034558695, -0.080114625, 0.03000025, 0.037008855, 0.017763682, -0.015769362, -0.04840496, 0.022279387, -0.011018611, -0.009287827, -0.016367657, -0.014451687, -0.00032786065, -0.007913171, -0.025683975, 0.029031582, -0.025370583, 0.016182471, -0.016225206, -0.022122692, -0.035726793, -0.0016070291, -0.051994737, -0.054786783, 0.008981557, -0.03185212, 0.02769254, 0.027379146, -0.06700911, -0.012699536, -0.013803534, -0.0025872723, -0.034957558, -0.022934664, -0.030598545, 7.091266e-06, 0.040883534, 0.00017895449, -0.031595707, 0.00802001, -0.0029968198, 0.0066453544, 0.00016682387, 2.6820915e-05, 0.01207275, -0.03632509, 0.045869328, -0.045328014, 0.009850509, -0.0034295158, 0.0020192477, -0.00085693377, -0.017436042, 0.016296431, 0.0342453, -0.034159828, 0.038176958, 0.0068234187, 0.015213802, 0.033675496, -0.00037148636, 0.03096892, 0.040940512, 0.004611862, -0.003463348, -0.018390467, 0.0028792976, -0.008112603, -0.016396148, 0.000604973, 0.028988846, -0.017663965, 0.018304996, -0.024230972, -0.04190918, -0.07020002, 0.0044694105, 0.020413276, 0.0017254418, 0.012763639, -0.008625428, 0.010135412, -0.019088479, 0.0032372065, 0.020242333, 0.0019052866, 0.0026050787, 0.053932074, -0.04031373, -0.020712424, -0.035641324, 0.07373281, 0.0278065, 0.06700911, -0.075271286, -0.009715181, -0.0139174955, -0.00043915075, -0.022407595, 0.039829392, 0.031937588, 0.023661166, 0.03529944, -0.016225206, -0.034501713, 0.009444524, -0.02358994, 0.021495907, 0.03173816, 0.066781186, -0.0014868358, -0.007984397, 0.005491499, -0.02101157, -0.017877642, 0.023433244, 0.034587182, 0.0034010254, 0.0062500522, 0.020740913, 0.007507185, 0.017521514, 0.0015349131, 0.015000125, 0.0006699664, 0.008076991, -0.0011111203, -0.024416158, -0.06330537, -0.002256073, -0.04618272, 0.029344974, -0.04031373, 0.024843512, 0.008290667, 0.009444524, 0.011887563, -0.050085887, 0.046837997, 0.03521397, 0.02039903, 0.0012161783, -0.06997209, 0.015555685, -0.0008208758, -0.03330512, -0.03635358, 0.021709584, -0.034900576, 0.043618597, 0.0008119726, 0.03105439, 0.034701146, 0.06330537, -0.013547122, -0.01592606, -0.045328014, -0.03957298, 0.03643905, 0.041082963, 0.02420248, 0.031310804, 0.0024038663, 0.001351507, 0.06393216, -0.0059152916, 0.03638207, 0.0043020304, -0.016168226, 0.020071393, 0.035384912, 0.012129731, 0.03321965, 0.042222574, -0.005783524, -0.057521846, -0.031396274, -0.037236776, -0.055271115, -0.0043162755, 0.033703987, -0.014715223, 0.038233936, -0.033903416, -0.0071688634, 0.018447448, 0.04076957, -0.028632717, 0.00047988293, -0.02283495, 0.046610076, 0.05586941, -0.016837748, 0.011146816, 0.047322333, -0.016908973, -0.019017253, -0.0034028061, -0.049800985, -0.019914696, -0.01984347, -0.06535667, 0.01934489, -0.0459548, -0.0011894687, -0.0028757362, 0.013340567, 0.0028882008, -0.0026424723, -0.022877684, 0.0749294, -0.007606901, 0.09538541, -0.01811981, 0.043533128, 0.0020032218, -0.011225165, 0.022706741, 0.016097, 0.040997494, -0.003846186, -0.028604228, 0.025769446, -0.017849151, 0.019188194, -0.022350613, -0.025327846, -0.036980364, -0.012250815, -0.006171704, 0.003935218, 0.03284928, -0.0064922194, -0.062336702, 0.017108405, -0.014017211, 0.030854959, 0.03435926, -0.032735314, 0.07151057, -0.019715264, 0.008141094, 0.010997243, 0.0015954549, -0.020199599, -0.03196608, 0.026453212, -0.017635474, 0.021410435, -0.008098358, 0.015128331, -0.056182805, -0.02675236, 0.002713698, 0.035270948, -0.025997369, -0.03444473, 0.05048475, 0.007040657, 0.013411793, 0.03968694, 0.021310719, -0.019857716, 0.019943187, -0.015000125, -0.029060071, -0.018575653, 0.03769262, 0.002225802, 0.010598379, 0.0073433663, 0.040000334, 0.02616831, 0.012165343, 0.015413234, 0.02558426, -0.021154022, -0.006175265, 0.011346249, -0.04168126, -0.044587266, -0.059772577, -0.01668105, -0.027849235, 0.021139778, -0.045071602, -0.0026086401, -0.032649845, -0.064331025, 0.0040812306, 0.006453045, -0.029686857, -0.042051632, 0.014387584, 0.007735107, 0.030883448, 0.02997176, 0.001537584, 0.004889642, -0.007834823, 0.030313643, -0.027877726, 0.010420315, 0.022350613, 0.022592781, -0.046211213, 0.00018852543, -0.014686733, -0.011624029, -0.012578453, 0.0017699578, -0.028191118, 0.00960122, -0.01601153, 0.029031582, 0.0109901205, 0.003924534, -0.02031356, -0.012841987, 0.029743837, 0.021752318, -0.01920244, 0.017991602, -0.03740772, 0.004957306, 0.012407511, 0.020114128, -0.027934706, 0.015270783, 0.0046724034, -0.0068732766, 0.020655444, 0.0032283033, -0.025384827, -0.011987279, -0.012749394, 0.0005154958, 0.03632509, -0.0298578, -0.0056339502, 0.021111287, -0.013760799, -0.009373298, 0.01995743, 0.030484585, -0.010491541, 0.035983205, 0.0017485901, -0.01981498, 0.0017646159, 0.027094243, -0.02309136, -0.039373547, 0.011994402, 0.011075591, -0.005519989, -0.024444649, 0.008803492, -0.016965954, 0.005295628, -0.005794208, 0.008418874, -0.0026317884, -0.011595539, 0.00082933385, -0.031937588, -0.044330854, -0.016111245, -0.028931865, 0.025099924, 0.0013773263, -0.0030395554, 0.008475854, -0.012357653, -0.0012473394, 0.0031072197, -0.0063782586, 0.0051460546, 0.0066702836, 0.02350447, 0.008418874, -0.016410394, 0.02411701, 0.028005932, -0.043476146, 0.028946111, 0.046296682, 0.032592863, 0.0067308256, 0.011616906, 0.0095727295, -0.003488277, 0.029515916, -0.0056125824, -0.0036609992, -0.0014271843, -0.01867537, -0.014323481, -0.022022976, -0.0095727295, 0.0053882217, 7.189341e-05, -0.013746553, -0.027236694, -0.043533128, -0.0333621, 0.06484385, 0.027322166, 0.002430576, -0.016638316, 0.0049751126, -0.03344757, -0.039971843, 0.011495822, -0.04803459, -0.0064352388, -0.041880693, -0.044701226, -0.016438883, 0.012236569, 0.013497263, 0.033533044, 0.021723827, -0.0019498026, -0.062735565, 0.020812139, 0.032393433, 0.012094118, -0.006495781, -0.023846352, -0.0035648446, -0.05116852, -0.007151057, 0.036809426, 6.254504e-05, 0.00885335, -0.059658617, -0.005982956, 0.015114087, -0.01315538, -0.0036360703, -0.0027119173, 0.019074233, -0.010028574, 0.003507864, -0.032592863, -0.0012963071, -0.018319242, -0.008675287, -0.004512146, -0.0169802, 0.061766896, 0.0030431165, -0.1027359, -0.033647005, -0.018020093, -0.013668206, -0.009002924, -0.013269342, -0.023661166, -0.019558568, 0.054957725, 0.015812097, -0.034045868, -0.04928816, -0.01201577, 0.02101157, 0.021994486, -0.019558568, 0.0044907783, -0.01315538, 0.007642514, -0.0027315044, -0.028504511, -0.03544189, 0.017920377, -0.0061645815, 0.025513034, 0.016766522, -0.021880524, -0.0046830876, -0.007884681, -0.03968694, 0.029430445, 0.006634671, 0.009978715, -0.009266459, -0.04165277, 0.0043518883, 0.018319242, -0.028105648, 0.024102766, -0.02031356, -0.008504344, 0.03632509, -0.018974517, 0.0014254036, -0.0032104969, 0.0029131297, 0.013468773, -0.004312714, 0.030883448, -0.009715181, -0.054786783, -0.028618472, 0.01365396, -0.00613253, -0.006335523, 0.005131809, -0.024715306, 0.008611183, -0.018589899, -0.009287827, 0.029458934, 0.008304913, 0.006218001, -0.0298578, 0.0018304996, -0.019074233, -0.034758125, 0.03920261, -0.00054354087, -0.025128415, 0.019088479, 0.010954508, -0.012279305, 0.032279473, -0.024373423, 0.04598329, -0.026980283, 0.045299523, 0.0023718146, -0.02344749, 0.0071474956, -0.004312714, 0.0045050234, -0.03529944, -0.021581376, 0.0086467955, -0.007970152, -0.014558526, -0.03709433, -0.027108489, -0.033732474, -0.013283587, -0.017165385, 0.009188111, -0.001634629, -0.0065563223, 0.002295247, -0.011780725, -0.1437049, -0.009223724, 0.017136896, -0.0298578, -0.0062001944, -0.0026620592, 0.041025985, 0.024985963, -0.0032069357, 0.0035452575, 0.02888913, -0.042450495, -0.004287785, -0.012428879, -0.0065812515, -0.034672655, 0.0037108571, -0.013504386, 0.011780725, -0.034929067, -0.02400305, 0.01201577, 0.0039993213, 0.0050570224, -0.066097416, -0.0031624197, -0.019943187, 0.009237969, 0.025399072, 0.0054024667, -0.029060071, 0.007841946, -0.018803576, 0.007275702, -0.0009588755, 0.026054349, 0.0005858311, 0.0072365277, 0.03934506, -0.028148383, 0.0023593502, -0.024857758, 0.010534276, -0.033020217, 0.034843594, -0.014743713, 0.0028330008, 0.0018082416, -0.021852035, 0.048291, 0.011859073, -0.007329121, 0.0024323566, 0.017350573, 0.03410285, 0.006349768, -0.012443123, -0.045299523, 0.011467332, 0.031168351, 0.006399626, 0.01340467, 0.010577011, -0.02467257, -0.004654597, -0.0019658285, 0.016780768, 0.022820704, -0.01662407, -0.033789456, -0.028533002, 0.021538641, 0.010626869, 0.0023914017, -0.02303438, -0.010242251, 0.038290918, 0.0266384, -0.021467416, -0.0050249705, 0.031310804, -0.042251065, -0.029031582, 0.025954632, -0.008333403, 0.031624194, 0.028461777, -0.009188111, -0.009822019, 0.035983205, 0.03313418, -0.029174034, 0.006502903, 0.01151719, -0.047721196, 0.006356891, 0.012236569, 0.022293633, 0.035498872, 0.0017681771, 0.0017361256, 0.028689697, -0.037977524, -0.010577011, -0.027478863, 0.032307964, -0.0003554606, -0.0026798656, -0.0136112245, -0.0136112245, 0.028661208, 0.037892055, -0.0023718146, 0.013782166, -0.015940305, 0.015712382, 0.05886089, 0.00066595996, 0.0005061474, -0.0035310125, 0.003324458, 0.006652477, -0.008034255, 0.01829075, -0.021709584, 0.008539957, -0.028119894, 0.015085596, 0.008896085, -0.022792213, 0.01662407, 0.017208122, -0.014643997, -0.014294991, 0.0023522277, -0.0022863438, 0.013290709, -0.018077074, -0.015726628, 0.010740831, -0.021353455, -0.009729426, 0.01551295, -0.011125449, -0.029373465, -0.0155841755, -0.014943144, -0.024359178, 0.0040420564, 0.0064423615, 0.022250898, 0.0067949286, 0.009871878, 0.025911897, -0.019743755, 0.011210919, 0.0040028826, -0.023162587, -0.032678336, 0.012535717, 0.0064031873, 0.04054165, 0.019031499, -0.008418874, 0.05259303, -0.016723786, -0.014971635, -0.015142577, 0.0043091527, 0.0014360875, -0.039117135, 0.029829308, 0.026467457, 0.041994654, 0.030627036, 0.019302156, 0.0083690155, -0.0038924827, -0.013810657, 0.038119975, 0.0075570433, 0.005733666, -0.00854708, -0.006951625, 0.04715139, -0.010078432, -0.009765039, 0.011787848, -0.004950184, 0.009558484, -0.01501437, 0.018105565, -0.00017294483, 0.02303438, -0.026282271, 0.017820662, 0.02774952, -0.013312077, -0.014857674, -0.023632675, -0.015185312, 0.019928942, 0.009814897, -0.027208203, 0.018276505, 0.034786616, -0.024515875, 0.0023148342, 0.0057301046, -0.005074829, -0.019145459, 0.013034297, -0.02769254, 0.025769446, -0.015598421, 0.009451645, 0.0058262595, 0.0111753065, -0.008753634, -0.036666974, -0.07709466, -0.015854834, -0.029373465, 0.006912451, -0.011260778, 0.0076994943, 0.029060071, -0.020669688, 0.011381862, 0.013005806, -0.019159704, -0.029686857, -0.016666805, 0.011153939, -0.036040187, 0.013589857, -0.0007781404, 0.020512993, 0.02133921, 0.012485859, 0.037892055, 0.009273582, -0.003920973, -0.007920294, 0.007229405, -0.031567216, 0.021481661, 0.01940187, 0.011111204, 0.016951708, -0.006876838, 0.026837831, 0.015911814, -0.03521397, 0.043219734, -0.029231014, 0.019644039, 0.033476062, 0.025726711, 0.015042861, 0.00015435937, 0.05131097, -0.008190951, -0.03638207, 0.021752318, -0.019387627, -0.010605502, 0.01123941, 0.03957298, -0.016723786, -0.024900492, 0.02886064, 0.010049941, 0.019074233, 0.0038782374, -0.012720904, 0.005858311, -0.0072614565, 0.019501587, -0.009515749, 0.0045869327, 0.012884723, -0.019601304, 0.0032906258, 0.0012918555, -0.021723827, -0.03108288, -0.028931865, 0.0054452023, -0.009188111, -0.010106922, 0.0022649763, -0.0047614356, 0.0017468095, -0.0071368115, 0.009266459, -0.0065278322, 0.008169584, 0.037037347, -0.05037079, 0.00618951, -0.019914696, 0.030570056, -0.040285237, 0.008988679, 0.019829225, 0.01318387, -0.05988654, 0.02039903, 0.0071439343, -0.023219567, 0.0068661543, 0.00071670825, 0.016139736, 0.0012856233, 0.0016497645, -0.0031072197, -0.018988762, -0.016239451, -0.013062787, -0.014537158, 0.014173907, -0.0035488189, 0.0046189846, 0.007592656, -0.01753576, 0.0075356755, -0.011659642, 0.011075591, -0.0064566066, 0.0042414884, -0.012272182, 0.0029576458, 0.0002394963, -0.042222574, 0.033419084, -0.023718147, -0.033276632, 0.003075168, -0.019003008, -0.0006566116, 0.012991562, 0.0034419803, -0.013440283, 0.029159788, 0.02566973, 0.0014886165, 0.0023362017, 0.01254284, 0.033703987, 0.06883249, -0.021738073, -0.017948868, 0.010292109, -0.00032518967, 0.04792063, -0.0029629876, -0.023547204, -0.038974684, -0.0068697156, -0.011980157, 0.02991478, -0.0019373382, -0.006100478, 0.01554144, 0.008027133, -0.041966163, -0.017379062, 0.034273792, 0.014230888, -0.0006303471, 0.055897903, -0.022706741, 0.00629991, 0.032706827, 0.026581418, 0.011332003, -0.0054736924, 0.0030075037, 0.018191036, -0.008532835, -0.03971543, -0.012165343, 0.0093875425, -0.029231014, 0.03430228, 0.0052600154, 0.018105565, -0.042535968, 0.017763682, 0.0021795053, 0.009658201, 0.025555769, -0.021695338, -0.0010950946, 0.0044480427, 0.011004365, 0.00017839804, -0.006623987, -0.01811981, -0.031453256, 0.013532876, -0.02289193, -0.022364859, 0.02031356, -0.031481743, 0.010491541, 0.025911897, 0.007240089, 0.022407595, -0.043219734, 0.014323481, -0.029145543, 0.015740871, 0.0061788266, -0.054416407, -0.019943187, 0.011802093, -0.0050926353, 0.030085722, -0.019059988, -0.0048469068, -0.020612707, -0.017079914, 0.0018269384, 0.031624194, -0.012357653, 0.008148216, 0.023960315, 0.029430445, -0.00070780504, -0.017293591, -0.040399197, 0.0017459191, -0.006150336, -0.013077032, 0.04256446, -0.009316317, -0.04581235, 0.025185395, -0.025128415, 0.0150286155, 0.017421797, 0.04179522, 0.014380462, 0.0013550683, -0.021666847, 0.0040064435, 0.014216643, -0.01201577, 0.013661083, -0.005306312, -0.034758125, 0.019359136, 0.0011743332, -0.0018518673, -0.016780768, -0.0018020093, 0.001423623, 0.0024608467, 0.017863397, 0.010854792, 0.002122525, 0.00646729, 0.007140373, 0.01390325, -0.00857557, 0.042251065, -0.02994327, -0.041396357, 0.004740068, -0.0031820065, 0.0029523037, -7.6400655e-05, -0.0012900749, 0.010000084, 0.035498872, -0.008596938, -0.023917578, 0.0031713229, -0.008027133, 0.037721112, 0.0065171486, -0.024188235, 0.017763682, 0.028205363, 0.007378979, -0.013675327, -0.03096892, 0.015213802, 0.0036788057, 0.033020217, -0.011859073, -0.017891888, -0.0012072751, -0.03210853, 0.014643997, 0.0032087162, -0.005406028, -0.01359698, -0.0038141345, -0.028276589, 0.016296431, 0.00075855333, -0.0016862677, -0.042849362, -0.026082838, -0.035270948, -0.006232246, -0.009608342, 0.008475854, -0.020099882, -0.011225165, 0.01592606, 0.01612549, 0.01934489, -0.023760881, 0.006584813, 0.025242375, -0.0043305205, 0.015740871, -0.019031499, -0.03433077, 0.008069868, -0.009950225, -0.010755075, 0.005089074, 0.0033013097, -0.0049216934, 0.022720987, -0.004825539, 0.006980115, -0.027478863, -0.018319242, -0.022521555, 0.023305038, -0.015897568, 0.0093875425, -0.01764972, -0.007165302, -0.030570056, -0.054815274, 0.007692372, -0.0077635976, 0.024373423, 0.033647005, -0.030541565, 0.023305038, 0.031481743, -0.016823502, -0.006071988, -0.014943144, -0.012193834, -0.0339604, -0.038034506, 0.030171191, -0.009985838, -0.007414592, 0.004334082, 0.008126848, -0.02089761, -0.009658201, -0.022165427, 0.010306354, 0.018048584, -0.0076353913, 0.0036681218, 0.00023103826, -0.0070335343, -0.013575612, 0.0048362226, -0.007606901, 0.02997176, 0.005505744, -0.010192392, 0.005242209, -0.010577011, -0.033789456, 0.039231095, 0.005484376, -0.0003910734, -0.039402038, 0.019615548, 0.0047115777, 0.011681009, 0.0062927874, -0.020085638, -0.02245033, 0.00043002496, 0.025014454, 0.0012624749, 0.045100093, -0.0020299314, 0.027706783, -0.022820704, 0.027065752, 0.012414633, -0.004583372, -0.01823377, -0.007200915, 0.050855126, -0.007678127, -0.041225415, -0.0149289, -0.021738073, -0.030797977, -0.0064423615, -0.0044444813, 0.01254284, 0.016609825, -0.04598329, -0.0090171695, -0.004398185, 0.006460168, 0.007428837, 0.034045868, -0.0054095895, -0.0040171277, -0.02561275, -0.01237902, -0.02888913, 0.010498663, -0.0114815775, -0.012678169, 0.027877726, 0.020512993, 0.009857632, 0.011424596, -0.010242251, -0.030598545, 0.0040135663, -0.005003603, -0.0017601643, -0.014672487, 0.0011218042, 0.026552929, 0.0119445445, 0.024544364, 0.008183829, 0.0051959124, -0.009230847, 0.036068678, -0.011353371, 0.005851188, -0.003924534, 0.014259378, -0.022720987, -0.032279473, -0.01829075, 0.030940428, -0.018746596, -0.019743755, 0.025028698, -0.019857716, 0.031225331, 0.03173816, -0.014886164, 0.057265434, -0.030114211, 0.008938821, 0.016752277, -0.0078775585, -0.020057147, 0.012250815, 0.009750794, 0.0120371375, -0.008604061, -0.010391825, -0.0099431025, 0.0050641447, 0.0015740872, 0.006951625, 0.021467416, 0.019487342, 0.023803618, -0.00904566, -0.011495822, 0.028205363, 0.023404753, -0.0112679005, 0.015812097, 0.031282313, 0.04153881, 0.011809215, -0.011381862, -0.01592606, -0.033048708, 0.015755117, 0.026239535, -0.005206596, -0.00832628, -0.0045548812, -0.01642464, -0.0057301046, -0.0034651286, 0.03091194, -0.0014449907, -0.030114211, -0.00014289649, 0.0116667645, -0.006513587, 0.012030015, -0.009081272, -0.0032585743, -0.0080912355, 0.01359698, 0.012621188, 0.008447364, 0.0014271843, -0.0055413567, -0.014814938, -0.025968878, 0.02139619, 0.024401912, -0.010982998, 0.012799252, -0.0073504886, -0.041282397, -0.0008106371, -0.006150336, -0.010705218, 0.0019444607, 0.026538683, -6.527165e-05, -0.023205321, 0.0063390844, -0.01703718, -0.051994737, 0.054672822, -0.00092148205, 0.025427563, -0.01210124, -0.0068982057, -0.02031356, 0.023319284, -0.018361976, -0.030057231, 0.029886289, -0.0018198157], "content_hash": "8f873584e8643860cba2746c1473e7f3"}, {"content": "# Tokenization Explained\n\n## What Is Tokenization?\n\nTokenization is the very first step in how a language model processes your text. Models cannot read raw text -- they need it broken into small, numbered pieces called **tokens**. Tokenization converts your input string into a sequence of these tokens.\n\n## Why Not Just Use Words?\n\nYou might wonder why we don't just split text by spaces. The problem is that there are too many possible words (including misspellings, rare terms, and words in other languages). Instead, modern models use **subword tokenization**, which breaks text into smaller, reusable pieces.\n\nFor example, the word \"unhappiness\" might become three tokens: \"un\", \"happiness\", or \"un\", \"happ\", \"iness\" -- depending on the specific tokenizer.\n\n## How It Works: BPE\n\nMost models (including GPT-2 and LLaMA) use a method called **Byte-Pair Encoding (BPE)**. Here's the intuition:\n\n1. Start with individual characters as your vocabulary\n2. Find the most common pair of adjacent characters in the training data (e.g., \"t\" + \"h\" = \"th\")\n3. Merge that pair into a new token\n4. Repeat thousands of times\n\nThis builds a vocabulary of common subwords. Frequent words like \"the\" become single tokens, while rare words get split into pieces.\n\n## Token IDs\n\nEach token has a unique **ID** -- a number that the model uses internally. For example, in GPT-2's vocabulary, the token \"the\" might have ID 262, while \"cat\" might have ID 9246. The model never sees the text itself; it only works with these IDs.\n\n## What You See in the Dashboard\n\nIn **Stage 1 (Tokenization)** of the pipeline, you can see exactly how your input text was split:\n\n- Each row shows a **token** (the text piece) and its **ID** (the number)\n- The summary shows the total number of tokens\n- Notice how spaces are often attached to the following word (e.g., \" cat\" with a leading space is one token)\n\nThis stage helps you understand that the model's \"unit of thought\" is the token, not the word.", "source_file": "tokenization_explained.md", "chunk_index": 0, "embedding": [0.0115930205, 0.003369542, 0.014504099, 0.0053508715, 0.011047995, -0.05714436, -0.023327107, 0.054784715, 0.008752474, 0.05365619, 0.009829702, -0.001070014, -0.026417723, -0.060581226, 0.008502403, -0.01112494, 0.002255445, 0.03216294, -0.015799338, 0.05427175, 0.02296803, 0.0045012725, -0.015504383, 0.051270902, 0.019903062, -0.008021498, -0.00911155, 0.011298065, -0.0131319165, -0.012574067, -0.013901365, -0.016761148, -0.034445632, -0.0341635, 0.018697593, -0.0073161707, -0.03998566, 0.020377554, 0.019210558, 0.027597545, -0.051193956, -8.9819056e-05, -0.051501736, 0.04803922, -0.051655628, 0.048859965, 0.026520317, -0.03285544, -0.03034191, 0.014093727, -0.032598957, 0.060427338, -0.022108814, -0.024789058, -0.013503817, -0.010175954, -0.023032151, 0.010220838, -0.046346433, -0.05991437, 0.020133896, 0.00951551, 0.045756523, 0.028828662, 0.009829702, -0.020967465, -0.044730593, 0.003821593, -0.02148043, 0.023301458, 0.07099443, 0.031111358, -0.019851765, 0.026545964, 0.0070468634, -0.03524073, -0.05070664, 0.048603483, -0.021865154, -0.035907585, 0.030906172, -0.064171985, -0.011721262, -0.049142096, 0.015812162, -0.010951813, -0.038780194, 0.0029671849, -0.012683072, -0.04773144, -0.023763128, 0.012484298, -0.052578963, 0.00805997, 0.026033, -0.023634886, -0.029315978, -0.025019892, -0.014632341, -0.008015086, -0.0003716996, -0.0033150397, -0.020634037, -0.020710982, 0.027187172, -0.061453268, 0.012958791, -0.0014795848, -0.045961708, -0.04001131, -0.10249051, 0.03275285, 0.0016527107, 0.031342193, 0.055348977, -0.013349927, 0.0061203195, 0.03418915, 0.014734934, -0.05888844, -0.003056954, -0.04580782, 0.010188777, -0.020749455, -0.026597261, -0.0012399338, 0.020108247, 0.0067583206, -0.006069023, -0.022455065, 0.0039049499, 0.01456822, 0.017158696, -0.009816878, -0.010560677, -0.050347563, -0.019043844, 0.033573594, -0.01382442, 0.0026273453, 0.0014555396, -0.043473825, 0.031726915, -0.017774254, -0.06653163, -0.046038654, -0.020993114, 0.060324743, -0.07078924, -0.009150022, 0.0017264495, -0.022390945, 0.0025920789, -0.019082317, 0.009015368, -0.042627435, -0.00058069295, 0.049013853, 0.021557376, -0.01494012, -0.008874304, 0.024994245, 0.011490427, 0.010105421, -0.046038654, 0.060632523, 0.0014274868, 0.011580196, -0.032983683, -0.059657887, -0.01719717, -0.03303498, -0.025532858, -0.0431404, -0.026956337, -0.0048058457, -0.05883714, -0.038908433, -0.0047192825, -0.0833056, -0.035958882, -0.011182648, -0.054220453, 0.010201601, -0.018453933, -0.008912776, -0.026597261, 0.040242143, -0.029700702, 0.025866285, -0.0036580854, 0.027623191, 0.03542027, 0.010432436, 0.0032172555, -0.013516641, 0.0453718, -0.020198017, 0.0035298439, 0.032496363, 0.02482753, 0.03629231, 0.0045237145, -0.00052418665, 0.0363949, 0.0059279576, -0.03608712, 0.003629231, -0.011811031, -0.032188587, -0.012772841, -0.04550004, 0.016299479, -0.030111074, -0.07058405, 0.02841829, -0.033855725, 0.047962274, 0.029777648, 0.017312586, 0.03911362, 0.010022064, 0.04088335, 0.004408297, -0.038164634, -0.008566524, 0.047218475, -0.03480471, 0.012695896, -0.011612257, -0.0032429039, 0.012612539, 0.0074123517, -0.03226553, -0.011208296, -0.034727763, 0.01323451, -0.034009613, -0.007540593, -0.029264683, -0.023070624, 0.01286261, 0.026186889, 0.03282979, 0.02988024, -0.007848372, -0.023840072, -0.03511249, 0.052245535, 0.034881655, 0.037420835, 9.6130934e-05, -0.06976331, 0.027828379, 0.0014627532, 0.034522578, 0.0193388, 0.027751435, 0.032958034, -0.0088678915, 0.02290391, -0.035907585, -0.06068382, -0.018107682, -0.029777648, -0.04488448, -0.042730026, -0.0023548324, -0.010977462, -0.042909563, -0.008848655, 0.0061235256, 0.013619234, -0.014619517, -0.038138986, -0.012356057, -0.042396598, -0.024276093, -0.047987923, 0.030367557, 0.011278829, -0.015568503, 0.075611115, 0.007091748, -0.011875152, -2.4521154e-05, 0.023391226, 0.016568786, 0.0060497867, -0.010060536, 0.027623191, 0.01233682, 0.029341627, 0.0123432325, 0.0026545965, 0.021647144, 0.053912673, 0.01850523, -0.0015557281, -0.013401223, 0.03939575, -0.08227967, -0.017594717, -0.028597826, 0.026828095, -0.0066813757, -0.0061908523, 0.016902214, -0.028366992, -0.030624041, -0.014363034, -0.050527103, -0.005713153, -0.026494669, 0.015914755, -0.0053508715, 0.042781323, -0.0014587457, 0.016402071, -0.0032124466, -0.003808769, -0.030829227, -0.0064761895, -0.019633755, 0.019056669, 0.0040588398, -0.032701552, 0.0075534172, 0.012195755, -0.038164634, 0.04968071, 0.006886562, -0.025481561, 0.025430264, -0.042037524, -0.025866285, 0.025558507, 0.035753697, -0.0038408292, 0.06350513, -0.02841829, -0.04614125, -0.018864306, -0.008188212, -0.05324582, 0.026571613, 0.04547439, -0.0104067875, 0.028649123, 0.014145024, -0.0074957083, -0.00068208383, 0.010688919, -0.055708054, -0.014298913, 0.008277981, 0.011945684, -0.02839264, 0.028777365, -0.013657707, -0.020492971, 0.0003664898, 0.026905041, 0.039626583, -0.009400093, -0.012477886, -0.017338235, 0.009880998, -0.039523993, -0.016479017, -0.0123817045, -0.010195189, 0.025366144, -0.0053412532, -0.020044127, 0.0040716636, 0.008893539, 0.027110226, -0.021095706, 0.031573027, 0.045756523, 0.0033342757, -0.029367276, 0.00059271563, 0.006803205, 0.036625735, 0.02014672, 0.0073033464, -0.038010743, 0.0016623288, -0.028213102, -0.006264591, -0.0026898629, 0.047910977, 0.020377554, 0.0497833, -0.05334841, 0.026725503, 0.022480713, 0.0049757655, 0.016568786, 0.0050719464, -0.031137006, -0.043012157, 0.02662291, 0.011599433, 0.025071189, 0.028520882, 0.033060625, 0.04814181, 0.029854592, 0.028469585, 0.017030455, -0.026981985, -0.01174691, 0.01701763, 0.014055255, 0.03957529, 0.015196603, 0.023840072, -0.010554265, -0.04088335, -0.054323047, -0.039216213, -0.04355077, 0.004302498, 0.01651749, -0.004786609, -0.007547005, -0.016786797, 0.019043844, 0.011484015, 0.03611277, -0.04057557, 0.0008287599, -0.027854027, 0.01605582, 0.050168026, -0.0007662422, 0.012407353, -0.0008311644, -0.018300043, -0.017876849, -0.0048026396, -0.061761048, 0.0022891085, -0.013272982, -0.07032757, 0.029033847, -0.027623191, 0.013991134, 0.02113418, 0.014606693, 0.038626302, -0.008303629, -0.08453672, 0.037805557, -0.008008674, 0.027469303, -0.011734086, 0.03865195, 0.00064120686, 0.00033282646, 0.020108247, -0.0073289946, 0.0070661, -0.036651384, -0.018248748, 0.03536897, -0.009842525, -0.012163695, -0.0072328136, 0.011163412, -0.042627435, -0.02929033, -0.02764884, 0.032624606, -0.0057227714, -0.018415462, -0.0026610086, 0.042294007, -0.0051232427, -0.025596978, 0.020390378, 0.007361055, 0.08756321, 0.0011934463, -0.011541724, 0.0014194717, -0.03216294, -0.027854027, -0.03318887, 0.019595282, -0.013157565, -0.011086467, 0.011471191, -0.03213729, -0.04062687, 0.01435021, 0.00039313996, -0.016094293, -0.011753323, -0.00021039601, 0.06955812, 0.01955681, -0.0005766854, 0.035599805, 0.01208675, 0.0008928806, 0.0150298895, -0.052835446, -0.03895973, 0.021070058, 0.017415179, -0.024647992, 0.022891086, 0.0041518146, 0.000885667, 0.0067454963, -0.00167355, 0.01962093, 0.035907585, 0.039652232, -0.035856288, -0.0121829305, -0.013939838, -0.042755675, -0.026443372, -0.03403526, 0.01162508, 0.026058648, -0.021211123, 0.011407071, -0.00013164779, -0.035266377, 0.012041865, -0.0009217349, -0.019415744, -0.03734389, 0.011028758, 0.012599715, 0.008720414, 0.014414331, 0.002983215, 0.0018146154, 0.00805997, 0.016376425, -0.022019044, -0.008284393, 0.0379338, 0.026853744, -0.004558981, 0.03447128, -0.037600372, -0.033471, -0.010945401, 0.015260723, -0.01605582, -0.0057355957, -0.021365013, -0.010387551, 0.0008223478, 0.038626302, -0.021608671, -0.020813575, 0.060581226, 0.023327107, -0.012465062, 0.022480713, -0.015491558, -0.026699854, -0.0046936343, 0.0047289007, 0.0069378586, -0.008835831, -0.028649123, 0.015209427, 0.013106269, 0.0035811404, 0.021044409, -0.023942664, 0.004911645, -0.0047930214, 0.005597736, -0.04801357, 0.022211406, 0.01494012, -0.025584154, -0.030675337, -0.0003682932, 0.03611277, -0.038728897, -0.010278546, 0.045294855, -0.031752564, 0.016479017, 0.037087407, -0.021595847, -0.04937293, 0.0073546427, 0.007264874, -0.033676185, -0.040549923, 0.020480148, -0.04344818, 0.0073033464, -0.018646296, 0.038446765, 0.0063832146, 0.012157283, 0.0029255063, -0.035856288, -0.015286372, -0.016786797, -0.034856007, 0.03318887, -0.047038935, -0.023827247, 0.013260159, -0.019723523, 0.02485318, -0.018133331, -0.011054407, -0.05457953, 0.015799338, 0.045243558, -0.012715132, -0.026981985, 0.0026994809, 0.036625735, -0.015735216, 0.011214708, 0.05878585, -0.0013561526, -0.012849786, 0.018838659, 0.013452521, 0.025840636, -0.00561056, -0.029623758, 0.008342101, -0.0068737376, 0.0031082504, -0.0016687409, -0.021467606, -0.009425741, 0.008887127, 0.050039783, -0.0024734556, -0.028084861, 0.0061107017, -0.03672833, 0.05263026, -0.006713436, -0.0046006595, -0.016158413, -0.021082882, -0.01642772, 0.0017264495, 0.025160957, 0.0034208386, 0.016017348, -0.033240166, -0.011176236, -0.009925882, 0.046372082, 0.029392924, 1.7871136e-05, 0.0037125878, -0.014465627, -0.07971484, 0.032085992, 0.0019091935, -0.005187364, 0.012567655, 0.024622343, -0.006363978, -0.031983398, -0.0022073546, 0.024417158, -0.03642055, 0.0016863741, -0.033958316, -0.013952661, -0.012792077, -0.01992871, 0.050937474, -0.020621214, 0.02076228, -0.013247334, -0.0019797261, -0.037113056, 0.0026000938, -0.017658837, -0.025302023, 0.003731824, 0.001210278, 0.04131937, -0.0058542187, -0.038780194, -0.031008765, -0.021980572, -0.043781605, -0.025802165, -0.0070596877, 0.036779627, 0.002138425, 0.04283262, -0.010022064, -0.006052993, -0.043063454, 0.028751716, 0.024365861, 0.018556528, -0.006149174, -0.004212729, -0.025994526, 0.017274113, 0.008303629, -0.0020181986, -0.044525407, 0.016145589, 0.0073097586, 0.010688919, 0.02228835, -0.03939575, -0.026366428, -0.02972635, 0.00078107015, 0.017889671, 0.015453086, 0.045115314, -0.007463648, -0.038728897, 0.00035627058, 0.015414613, -0.0055496455, -0.00017583095, -0.034317393, -0.00028874347, -0.021595847, 0.02163432, -0.014965769, 0.012766429, 0.008714002, 0.0016182458, -0.017133048, 0.004934087, -0.011657141, -0.04180669, -0.0017200374, 0.010535029, 0.0019123995, -0.015863458, 0.017774254, 0.0013850068, -7.864803e-05, 0.007117396, -0.017722959, 0.028033564, -0.0066942, -0.0023836866, -0.0021047615, 0.027110226, -0.0074059395, -0.007720131, 0.005491937, 0.009226967, 0.033522297, -0.0011685996, -0.028443936, -0.0098938225, 0.014901648, -0.008976896, 0.052886743, -0.021544551, 0.04134502, 0.0076880706, 0.0038151809, -0.014260441, -0.028597826, 0.019941535, 0.0059087216, -0.03093182, 0.0069506825, -0.0027892499, -0.043037806, -0.0032092405, -0.021749737, -0.030239316, -0.02340405, 0.007393115, 0.022185758, -0.017530596, -0.028290048, 0.01419632, 0.002888637, -0.08966637, -0.021711266, 0.026148416, 0.010490145, 0.008104855, -0.024032434, 0.036574442, 0.012580479, -0.02485318, -0.04001131, 0.015055537, -0.013061384, 0.014478452, -0.01999283, 0.0006103488, -0.0267768, 0.022570482, 0.018838659, 0.0011902403, 0.0062068827, -0.046372082, 0.0089448355, 0.024660816, 0.0067519085, -0.023840072, -0.005495143, -0.008637057, 0.032342475, -0.0016591228, 0.005681093, -0.022211406, 0.022275526, -0.035292026, -0.011484015, 0.01456822, 0.049937192, -0.017133048, 0.0007550211, 0.022083165, -0.026853744, 0.036959164, -0.015696744, -0.015158131, -0.011298065, 0.029777648, -0.040652517, -0.009528334, 0.005693917, -0.031111358, 0.034932952, -0.032316826, 0.026212538, 0.01118906, -0.0019011784, 0.03762602, 0.002138425, 0.011977744, -0.036830924, 0.022275526, 0.048116162, 0.0012014614, 0.019018196, -0.019249031, 0.023173217, -0.0052001877, 0.0063960385, 0.014260441, 0.03795945, -0.031290896, -0.022275526, -0.016773973, 0.008348513, 0.023737479, -0.0073289946, 0.01267666, 0.023327107, -0.012138046, 0.028828662, -0.022108814, -0.026417723, 0.024596697, -0.005786892, -0.016838092, 0.03675398, 0.028597826, 0.03303498, 0.0035009896, -0.024635168, -0.013452521, 0.009291087, 0.024455631, 0.002503913, -0.030726634, -0.022621779, -0.018171802, -0.0043698247, 0.0075277686, 0.046346433, 0.022826964, -0.0151068345, 0.048731722, 0.023314282, -0.047526255, -0.02333993, -0.015068362, -0.017812727, 0.040088255, -0.035035543, -0.017876849, -0.020069776, 0.016132765, 0.023186041, -0.012240639, 0.027725786, -0.07012238, 0.04355077, 0.06232531, 0.032598957, -0.0026754357, -0.027700137, -0.023147568, -6.3619746e-06, 0.0060946713, 0.031034414, -0.041396316, 0.008072794, -0.020480148, 0.062479198, -0.0078098997, -0.0038055629, 0.0016390851, 0.03121395, -0.003642055, -0.041550204, -0.026699854, 0.024429983, -0.017312586, 0.02885431, -0.016761148, 0.001126921, -0.025225079, -0.0016054218, 0.0010579914, 0.015286372, -0.012548419, 0.009759169, -0.013247334, -0.034907304, 0.000559453, 0.02253201, 0.008758886, 0.033983964, -0.0130549725, 0.027264116, -0.024429983, -0.009996415, 0.00243338, -0.048552185, -0.0252892, 0.04180669, 0.0028677979, 0.02098029, 0.016325127, 0.0022971237, 0.03506119, -0.01776143, -0.012958791, 0.031034414, -0.03288109, 0.007822724, -0.008996133, 0.022237055, 0.016966334, 0.015004241, 0.016722675, 0.0070468634, -0.00926544, 0.014850351, 0.01620971, 0.015889106, -0.031419136, 0.020852048, -0.013311455, -0.024224795, -0.0070340396, -0.028879957, -0.022198582, -0.007559829, -0.030906172, 0.01583781, 0.03377878, 0.010336255, 0.03003413, 0.0129331425, -0.010034888, -0.0126446, 0.020903345, 0.003542668, -0.015132482, 0.02333993, -0.02513531, 0.028033564, 0.018441109, -0.014657989, -0.018992547, 0.010522204, 0.0013248937, 0.010438848, 0.0197107, 0.0034849595, 0.0046295137, 0.021300893, -0.019172085, 0.043730307, -0.016786797, -0.024776233, 0.007944553, -0.014055255, 0.026545964, -0.017261289, -0.056733985, -0.018338516, -0.0033887783, 0.0034945775, 0.00067847705, -0.0076880706, 0.02616124, 0.012529182, -0.021147003, -0.0075726532, -0.032778494, -0.014311737, -0.0103683155, -0.00035106076, 0.015914755, 0.042140115, 0.013503817, -0.012727956, 0.021685617, 0.010637622, -0.002980009, 0.0234297, -0.016107116, -0.0121444585, -0.007239226, -0.011413482, 0.02340405, 0.020531444, 0.0033855722, -0.03344535, 0.044961426, 0.030598393, 0.012509946, -0.030470151, 0.034856007, -0.0048924084, 0.049860246, 0.03747213, 0.01553003, -0.021608671, 0.02913644, 0.029059496, 0.052886743, 0.0038376232, -0.0130549725, -0.0062838276, 0.0017857612, 0.010483732, 0.042704377, -0.012471474, -0.033265814, 0.043781605, 0.00075261656, 0.0003885313, 0.009900235, -0.008893539, 0.017889671, -0.014491276, 0.009624516, -0.0043441765, -0.012048277, -0.009643751, 0.009964355, -0.0016879771, 0.012247051, -0.0029271094, -0.002519943, -0.01912079, 0.0113942465, -0.011266005, -0.004841112, -0.013208861, 0.01865912, 0.026545964, -0.008880715, 0.008412634, 0.0018338517, 0.00016611266, -0.0135294655, -0.039190564, 0.007393115, -0.0178512, 0.010560677, -0.010829984, -0.0038023568, 0.023173217, 0.033240166, -0.004302498, -0.010201601, 0.0009209334, 0.00836775, -0.0013826024, 0.012317584, 0.034958597, -0.0051969816, -0.023699006, 0.0111569995, -0.040036958, 0.04342253, -0.031701267, -0.020018479, 0.0092205545, -0.017684486, 0.02631513, 0.022365296, 0.0014507306, 0.036830924, -0.011304477, -0.0053701075, 0.025417442, 0.053297117, -0.007918905, 0.039652232, 0.017248465, -0.047680143, -0.03554851, -0.030290613, -0.019313151, 0.002151249, -0.03223988, 0.019236207, 0.022814142, -0.012061101, -0.008726826, 0.021890802, 0.0012383307, 0.003728618, -0.010509381, 0.040960297, -0.013452521, 0.052502017, -0.03211164, -0.025096837, -0.0013088635, 0.00083276746, -0.00373503, 0.00691221, -0.012157283, -0.034266096, 0.0067519085, 0.01592758, -0.0061235256, 0.002229797, -0.015299196, -0.0060497867, 0.019390097, 0.009611691, 0.010573502, 0.06514662, 0.0006215699, 0.013593586, 0.009861762, -0.017351057, -0.018094858, -0.017864024, -0.005943988, 0.0009858556, -0.0017024042, 0.021608671, -0.0014884014, -0.023327107, -0.040909, -0.01044526, 0.020852048, -0.018133331, 0.012869022, -0.005293163, 0.017209992, -0.017979441, 0.0053284294, 0.01007336, 0.0039690705, -0.0024478072, 0.0100925965, -0.004087694, 0.031983398, 0.021801034, -0.017594717, -0.020659685, -0.0077073066, 0.024994245, -0.0027251292, -0.015889106, -0.010060536, 0.02839264, -0.03106006, 0.0078034876, -0.0074828845, 0.009752757, -0.00633833, -0.0208264, 0.032958034, -0.015055537, 0.0002763201, 0.018825835, -0.040344737, -0.016607258, -0.0068480894, -0.018800186, 0.028751716, -0.025686748, -0.016158413, -0.053707488, -0.014247617, 0.015504383, -0.019595282, -0.01032343, 0.00042159352, 0.004469212, 0.007168693, -0.025237903, -0.00982329, -0.017966617, 0.027546247, 0.013619234, -0.011676378, 0.024417158, -0.033650536, -0.0034849595, 0.0245454, 0.012439414, 0.013516641, -0.00020879299, 0.04480754, -0.006341536, -0.026597261, -0.03731824, -0.016697027, 0.0008095237, -0.022147285, 0.012202167, -0.0074123517, -0.011413482, 0.024263268, 0.013593586, -0.036343604, -0.037574723, 0.010836396, 0.014439979, 0.009169258, 0.0013329088, 0.012638188, 0.003818387, -0.014581044, 0.0031932103, 0.043319937, -0.008175388, 0.064171985, -0.015376141, -0.004533333, 0.02141631, -0.027854027, 0.018928427, 0.015042714, 0.03911362, -0.0008163365, 0.020903345, 0.0072328136, -0.029700702, -0.024647992, -0.00052098057, 0.017133048, 0.0035266378, -0.041755393, -0.022634603, -0.010746627, -0.016991982, 0.00012423383, -0.028033564, 0.05935011, 0.007091748, -0.008951248, -0.03403526, -0.00023965107, 0.039831772, 0.007906081, 0.01837699, -0.003917774, -0.033291463, -0.0026481843, -0.01047732, -0.021749737, -0.0042479956, 0.016466193, 0.02392984, -0.028495234, 0.0032124466, 0.00991947, -0.004757755, 0.017286938, 0.023699006, 0.0042383773, -0.0204545, -0.007765015, 0.0021272039, 0.014170672, -0.005943988, 0.010746627, 0.027341062, 0.04521791, -0.00709816, -0.008701177, -0.010708155, 0.05098877, 0.021852331, 0.0029976422, 0.008681941, 0.015286372, 0.0042319656, 0.0011124939, -0.028777365, 0.0055272034, -0.0021897214, -0.030239316, -0.01791532, 0.01955681, -0.023634886, -0.014042431, -7.62936e-05, -0.016684202, 0.016863741, -0.048706073, -0.029675055, -0.003443281, 0.0103683155, 0.011939272, 0.012131634, 0.012009805, 0.04593606, -0.031624325, -0.0034657232, -0.0121829305, -0.016979158, -0.023006503, -0.031239599, 0.036523145, -0.02990589, -0.016235359, -0.015453086, -0.006389627, -0.02489165, -0.004036397, 0.00093776506, -0.0018691181, 0.02358359, -0.009047429, -0.031239599, -0.017787078, 0.010676094, -0.04419198, 0.052732855, -0.010150305, 0.047346715, -0.023391226, -0.018312868, -0.009810465, -0.022467889, 0.01066327, 0.029341627, -0.0163636, 0.017030455, -0.022698723, 0.0074957083, 0.03508684, -0.0051841578, 0.0038696835, -0.008092031, 0.011272417, 0.0050943885, -0.013747475, -0.023237336, 0.018620647, 0.0051488914, 0.0033471, -0.005963224, 0.039190564, -0.021300893, 0.008310041, -0.014439979, -0.022519186, 0.02489165, 0.028315695, -0.015966052, -0.04652597, -0.011881564, 0.0036099947, 0.00633833, 0.02349382, 0.0021768974, -0.016863741, -0.02340405, 0.016299479, 0.0126189515, 0.004687222, 0.016299479, 0.02051862, 0.025019892, -0.03303498, -0.03967788, 0.038626302, -0.006963507, -0.01819745, -0.0079958495, -0.010355491, 0.017658837, 0.05873455, 0.026699854, -0.0021929275, 0.017864024, -0.015081186, -0.0052066, -0.009002545, 0.025956055, -0.0049885893, 0.025109662, 0.0031403108, 0.0098938225, 0.013260159, -0.0005911126, -0.014632341, 0.0063062697, 0.0047160764, -0.003728618, -0.006969919, -0.004456388, 0.022237055, 0.019069493, -0.0026578025, -0.0072007533, 0.04742366, -0.012163695, -0.019633755, 0.025622627, -0.013709003, 0.025109662, 0.011849503, 0.0018130124, 0.016620083, 0.008201036, -0.012734368, -0.0051649213, -0.02209599, -0.040242143, -0.011144175, 0.0006219707, -0.0048282878, -0.025981702, -0.020274961, 0.02014672, 0.03631796, 0.007341819, 0.03434304, 0.0014659592, -0.013414048, 0.025391793, -0.041704096, 0.0002630952, 0.01242659, 0.024558224, 0.015953228, 0.007617538, 0.034599524, 0.04265308, 0.002431777, 0.02200622, 0.0155813275, -0.013022912, -0.0014835924, 0.036471847, -0.010862045, -0.026174065, -0.0046776044, 0.011669965, -0.019364448, 0.010515793, 0.07243073, -0.01509401, 0.0042544077, 0.0070404517, 0.02172409, 0.012349645, 0.029700702, 0.012311172, -0.028084861, -0.009957943, 0.02929033, 0.004366619, 0.00307619, -0.006886562, -0.055554163, -0.054938607, -0.010932577, 0.0055464394, -0.0007498113, 0.006056199, 0.0030104662, 0.00078708143, -0.05529768, -0.003818387, 0.028931255, -0.0029303154, -0.00429288, 0.0018370578, 0.017889671, 0.012234227, 0.009425741, -0.02749495, -0.00023544315, -0.011080055, 0.0056233844, 0.042165764, 0.0013321073, 0.021095706, 0.0012896273, 0.037677318, 0.007085336, 0.013349927, 0.0022233848, 0.027777081], "content_hash": "5a2eb44b34de90d11581ea0506ca1750"}, {"content": "# The Transformer Architecture\n\n## What Is a Transformer?\n\nA **Transformer** is the specific type of neural network architecture used by modern LLMs like GPT-2, LLaMA, and others. It was introduced in the 2017 paper \"Attention Is All You Need\" and quickly became the dominant approach for language tasks.\n\n## The Key Innovation: Attention\n\nBefore Transformers, language models processed words one at a time, left to right. Transformers changed this by introducing the **attention mechanism**, which lets the model look at all words in the input simultaneously and figure out which ones are relevant to each other.\n\nFor example, in \"The cat sat on the mat because it was tired,\" attention helps the model connect \"it\" back to \"the cat.\"\n\n## How Layers Stack\n\nA Transformer is built from identical **layers** stacked on top of each other. Each layer has two main parts:\n\n1. **Attention**: Looks at relationships between all tokens\n2. **MLP (Feed-Forward Network)**: Processes each token's information individually, retrieving stored knowledge\n\nA small model like GPT-2 has 12 layers. Larger models may have 32, 64, or more.\n\nInformation flows through these layers sequentially. After each layer, the model's understanding of the text becomes more refined. Early layers tend to capture basic patterns (like grammar), while later layers capture more complex meanings.\n\n## Encoder vs. Decoder\n\nThe original Transformer had two halves:\n\n- **Encoder**: Reads and understands the full input (used in models like BERT)\n- **Decoder**: Generates text one token at a time (used in GPT-style models)\n\nMost LLMs you'll encounter in this dashboard are **decoder-only** models. This means they generate text left-to-right, predicting one token at a time based on everything that came before it. Each token can only \"see\" the tokens to its left -- it cannot look ahead.\n\n## The Residual Stream", "source_file": "transformer_architecture.md", "chunk_index": 0, "embedding": [-0.0018053533, 0.028315073, 0.008433898, -0.021741506, 0.060291372, -0.010371495, -0.0030208095, 0.034187302, -0.04341169, 0.06357221, 0.00401784, -0.024071379, -0.032903492, -0.015881166, -0.016023811, -0.025961427, 0.035019398, -0.00904014, 0.018020844, 0.06276389, -0.005800905, -0.036208108, 0.018520102, 0.045812886, 0.018591424, 0.009265995, -0.024225911, 0.07874015, -0.024463654, -0.015643423, 0.0021367064, -0.016606279, -0.070133895, 0.018460667, 0.013254117, 0.0099495035, 0.022989653, 0.046383467, -0.0017815792, 0.0535395, 0.013598843, -0.004531957, -0.039798014, 0.032285366, -0.034282397, 0.012231827, -0.010591406, 0.0037949567, -0.03694511, 0.060719308, -0.011043116, 0.03756324, -0.039013464, -0.00018332138, -0.012540891, -0.011827664, -0.04141466, 0.01340865, -0.017533474, -0.024154589, -0.00020338086, 0.0056107114, 0.019780135, 0.027364105, 0.018829167, -0.005949494, -0.05886492, 0.040273495, -0.023013426, 0.02926604, 0.050068468, 0.02102828, -0.050971888, -0.008879664, 0.017260069, 0.008083228, -0.04291243, 0.07992886, -0.0050817356, -0.019007474, 0.011833608, -0.039346304, 0.0052422113, -0.01926899, -0.012303149, 0.021159038, -0.0050133844, -0.030573621, -0.039536495, -0.0624786, 0.003170884, -0.0074591553, -0.05615466, 0.015072843, 0.024487427, 0.029836621, -0.018175377, -0.0045379004, -0.018852942, 0.039132334, 0.04329282, -0.0010921274, 0.005167917, -0.030597396, -0.011910874, -0.035661303, 0.015726633, -0.03989311, -0.068517245, -0.027625622, -0.07588725, -0.030121913, 0.009937616, 0.017402714, 0.047310658, -0.038680624, 0.010591406, 0.031952526, -0.01918578, -0.00064413226, -0.011572092, -0.020719215, 0.00969393, 0.033260107, -0.0036523116, 0.0035364123, 0.0013462141, -0.0005527502, 0.019982215, 0.028101105, -0.001479944, 0.029955493, 0.024178362, -0.067423634, -0.008315027, -0.017676119, -0.05063905, 0.014454714, -0.030930236, 0.016214006, 0.026674652, -0.03922743, -0.009563172, 0.01955428, -0.0379674, 0.015477004, 0.011643414, 0.030644944, -0.05705808, 0.022312088, 0.019815797, 0.02054091, -0.058722276, -0.020731103, -0.024273459, -0.021824716, 0.009242221, 0.07807448, 0.046217047, -0.027696945, 0.0030995614, 0.034377493, -0.0078870915, -0.0040743034, -0.034282397, 0.051922854, -0.005429433, -0.0015304642, -0.042864885, -0.059387952, -0.024011943, -0.016416086, -0.028291298, 0.010591406, 0.014621134, -0.022145668, -0.07912054, -0.04940279, -0.0030460695, -0.047310658, -0.022193216, -0.0010208047, -0.045646466, -0.012142673, -0.047548402, 0.008731076, -0.017188746, 0.05391989, -0.011560205, 0.0069598975, -0.022704361, 0.05800905, 0.05025866, 0.028433945, 0.009105518, 0.008469559, 0.054823305, -0.024677621, 0.059863437, -0.025557267, 0.04198524, 0.025795007, 0.01988712, -0.049545433, 0.04086785, 0.020029765, -0.02033883, 0.016225891, -0.030716268, -0.048023887, 0.017640457, -0.042936206, -0.0010802402, -0.01397923, -0.014003004, -0.032309137, -0.045147207, 0.01947107, 0.0028187288, 0.028029783, 0.03135817, 0.033426527, -0.007958414, 0.018829167, -0.061527632, 0.017545361, 0.01873407, -0.042936206, 0.006424978, 0.020719215, 0.017806876, -0.0093729785, -0.00079643575, -0.040653884, -0.009747422, -0.048927307, 0.029646428, 0.0069598975, 0.024332894, 0.001638934, -0.02574746, 0.013075811, -0.0039970377, 0.007417551, 0.030121913, 0.015845506, -0.010829149, -0.046644982, 0.05800905, 0.058769826, 0.032023847, 0.0010260054, -0.038514204, 0.011756342, -0.030740041, 0.025842557, -0.00084621296, 0.007078768, -0.023595894, 0.017521586, 0.035922818, 0.04471927, -0.014668682, -0.0048677674, -0.01287373, -0.0312393, -0.0189837, -0.026627105, -0.0076196315, -0.050211113, 0.022205103, -0.009616665, -0.014014891, 0.029361138, -0.05182776, -0.04046369, 0.02909962, 0.010739995, -0.002471031, 0.056725245, -0.004650828, -0.02795846, 0.04714424, 0.05539389, -0.009682043, 0.020422038, -0.031334396, 0.008826172, 0.0044487473, -0.019494845, 0.04215166, -0.0096761, 0.035399783, -0.021907926, -0.014870763, 0.024321007, 0.022763798, 0.036564723, 0.014740004, -0.026436912, -0.0046211104, -0.048000112, -0.014038665, 0.02099262, 0.003209517, 0.026698427, -0.04189014, -0.029955493, -0.0067221555, -0.0016686517, 8.488133e-05, -0.034924302, 0.027554298, 0.012564665, -0.0024338837, 0.03768211, 0.005049046, -0.027530525, -0.005417546, -0.019364087, -0.032784622, -0.025224427, 0.007173865, -0.034543913, 0.017890086, 0.03254688, -0.021206588, -0.010365551, -0.0018826196, -0.015060957, 0.05734337, -0.01361073, -0.016332876, 0.021586975, -0.047334433, -0.009188728, -0.020315055, 0.070656925, -0.026627105, 0.0005936121, -0.011245197, -0.003028239, -0.037824754, 0.002445771, -0.05449047, 0.046383467, 0.016784586, -0.0019658292, 0.028909428, 0.0042734127, -0.044006046, -0.031167977, -0.00562557, -0.040321045, -0.0062050666, 0.03780098, 0.009325431, 0.009105518, 0.028743008, -0.016939119, -0.03361672, -0.003000007, 0.024582524, 0.0046270536, -0.05301647, 0.0063595986, 0.010086205, 0.036255658, 0.013004487, -0.012909391, 0.047762368, 0.031334396, 0.0038098157, 0.0125884395, -0.004546816, -0.004546816, -0.00338188, -0.001124074, -0.023239281, 0.03152459, 0.015916828, 0.03238046, -0.0053194775, -0.008071342, 0.007494817, 0.046621207, -0.01096585, -0.014169424, -0.022942103, 0.021491878, -0.011156044, -0.03542356, -0.02693617, 0.035280913, 0.02172962, 0.01426452, 0.028101105, 0.009586946, 0.04652611, 0.03775343, 0.0032659809, 0.015227376, -0.027554298, -0.036469623, 0.030359654, 0.01590494, 0.04246072, 0.01890049, 0.030145686, 0.007690954, 0.07284415, -0.00024740028, -0.013266004, -0.018056506, -0.016000038, 0.002653795, 0.0268173, 0.0043387916, 0.0208262, 0.03131062, -0.025604814, -0.015488892, -0.03214272, -0.00731651, -0.020326942, 0.026698427, 0.010490366, -0.02619917, -0.023167958, -0.014193198, 0.013848472, -0.006870744, 0.02033883, -0.030858913, 0.012374472, -0.013479972, 0.017794989, 0.035138268, -0.00912335, -0.025604814, 0.04362566, -0.005286788, -0.008433898, -0.027411653, -0.03287972, 0.021063942, -0.009450245, -0.038347784, 0.0020579542, -0.036255658, 0.018627087, 0.012142673, 0.044077367, -0.014276408, -0.043102626, -0.024249684, 0.042603366, 0.043839626, 0.06504621, -0.025485944, 0.039584044, -0.031120429, -0.0068291393, -0.056677695, -0.023381926, -0.012124842, 0.024273459, -0.035613753, 0.011732568, -0.00684697, -0.008112946, -0.004977723, -0.0045349286, -0.01853199, 0.008956931, 0.010365551, 0.029218493, -0.00840418, 0.035043173, -0.064665824, 0.0020980733, -0.016653828, -0.029384913, 0.0004398227, -0.021586975, 0.0787877, -0.037491914, 0.007898978, -0.0010200618, -0.03627943, -0.010347721, -0.013670165, 0.025367072, -0.010971794, 0.033783138, -0.02529575, -0.049022403, -0.069753505, -0.016344763, -0.010222906, 0.016309101, -0.021254135, -0.005313534, 0.068469696, 0.024867814, 0.012517117, 0.034187302, 0.03865685, -0.0100327125, 0.0052659856, 0.006288276, -0.0086716395, 0.0028068416, 0.03775343, 0.00090564846, 0.036255658, 0.0060208165, 0.023619669, -0.0025215512, 0.0099495035, 0.013361101, -0.0009078773, -0.01397923, -0.0057979333, 0.021301683, 0.01926899, -0.035399783, -0.014205085, -0.054538015, 0.011257084, -0.0007570597, -0.021777168, -0.009093632, 0.035970367, -0.0536346, 0.005551276, -0.009569116, -0.04058256, -0.0759348, -0.02574746, -0.007572083, 0.02357212, 0.017069876, 0.010086205, 0.004609223, 0.013563181, 0.016023811, -0.045598917, 0.005218437, 0.033711817, 0.020671668, -0.036921334, 0.034900527, -0.033830687, -0.023096636, -0.007756333, 0.017355166, -0.0010936132, 0.0078454865, -0.020065425, 0.012635988, -0.0031768277, 0.019055022, -0.012719197, -0.024725169, 0.016404198, -0.013468085, 0.016855909, 0.032071397, -0.03967914, 0.0017904945, -0.013551295, 0.018757844, 0.010092149, -0.014121875, -0.0016359622, 0.01771178, 0.024321007, 0.0009762281, -0.002881136, -0.047334433, -0.029384913, -0.02636559, 0.0058603403, -0.036208108, 0.008451728, -0.032166492, -0.01434773, -0.04272224, 0.011102552, 0.04227053, -0.02357212, -0.005049046, -0.002230317, -0.023655329, -0.013230342, 0.029194718, -0.021384893, -0.057533566, 0.010466591, -0.0049064006, -0.0020237789, -0.028552815, -0.0069836713, 0.011138213, -0.0015260065, -0.020148635, 0.006924236, -0.015346247, -0.00475484, -0.022609266, -0.032118946, 0.00025092927, -0.014668682, -0.045717787, 0.026959945, -0.035613753, 0.0035601866, 0.03751569, 0.012350697, 0.017390829, -0.007043107, -0.017093651, -0.009188728, 0.013931681, 0.010187245, -0.021575088, -0.015286811, -0.034852978, 0.03005059, -0.032665752, 0.010002995, 0.066995695, 0.048618242, -0.010977738, 0.027863363, -0.015893053, -0.034662783, 0.032523107, -0.033664268, -0.03718285, -0.005497784, -0.020980733, -0.0018038675, -0.015797956, 0.020980733, -0.0011032715, 0.017022328, -0.016297214, -0.009301656, -0.017913861, -0.02172962, 0.058674727, 0.006448752, 0.023690991, 0.013432424, -0.011007455, -0.017509699, 0.005396744, 0.013551295, -0.050401304, -0.014121875, -0.018520102, -0.015370021, -0.031049106, 0.012208052, 0.0067578168, -0.0045289854, 0.02147999, -0.007679067, -0.08611015, 0.012707311, 0.01565531, 0.004784558, 0.008754849, -0.0146092465, 0.0026448797, -0.052255694, 0.007756333, 0.023928734, 0.004213977, 0.0226687, -0.031167977, -0.0105260275, -0.004460634, 0.006704325, -0.0133729875, -0.01910257, 0.034520138, 0.008808342, -0.019506732, -0.0064963005, 0.0045795054, -0.04374453, -0.0067756474, -0.013919795, 0.0053283926, 0.08572977, 0.012861842, -0.07973867, -0.030312106, -0.013384875, -0.05320666, 0.046454787, -0.010852923, -0.008142664, -0.012303149, 0.039370075, -0.008023793, -0.05715318, -0.04410114, 0.013741488, 0.029765299, 0.0056879777, 0.0005813535, 0.006199123, 0.00023829921, 0.00089376135, -0.0011575064, -0.01434773, -0.04098672, 0.0009687987, -0.01022885, 0.005670147, 0.012731085, -0.018757844, -0.04074898, 0.0005616655, -0.047239337, 0.016237779, 0.0030044646, 0.03078759, -0.002534924, -0.035328463, 0.0021991136, 0.00017877085, -0.024285346, -0.013860359, -0.039536495, -0.019043135, -0.037301723, 0.0040059527, -0.0040743034, 0.035685074, 0.011857383, 0.026222944, 0.0034294284, 0.021527538, -0.00053603394, -0.05063905, -0.022526056, 0.0075899134, 0.013551295, -0.011845496, -0.006187236, 0.013598843, 0.025890104, -0.023179846, 0.019364087, -0.016618166, -4.176737e-05, -0.009812801, -0.03677869, -0.0017592908, -0.0074769864, -0.049355242, 0.0027726663, -0.00534028, 0.010318004, 0.024582524, 0.005592881, -0.015976263, 0.021753393, 0.0058751996, 0.002300154, -0.005393772, 0.040844075, 0.0027325472, -0.0013098099, -0.015857391, -0.025081782, 0.019768247, 0.0018276417, -0.03176233, -0.008570599, 0.0030609283, -0.026413137, -0.007239244, -0.00885589, -0.019910892, -0.026888622, -0.00068202236, 0.0550135, -0.015120392, -0.004044586, 0.026460685, -0.0020668695, -0.07978622, -0.038157593, 0.016879683, -0.027435428, 0.019340312, -0.017212521, 0.024677621, -0.0085884305, 0.037040204, -0.060434017, 0.0016359622, -0.036113013, 0.01304015, 0.010234794, 0.026151622, -0.047049142, 0.0074472683, -0.016855909, 0.016546844, -0.011916818, -0.043435466, -0.0078038815, 0.009432415, -0.019043135, -0.025081782, 0.002946515, 0.012059463, 0.0009628551, 0.0018529018, 0.024035716, -0.025200652, 0.028505268, -0.020029765, 0.019245215, -0.0085052205, 0.0014160508, -0.008950987, -0.015049069, 0.024011943, 0.007946527, -0.011013399, -0.03131062, 0.017545361, -0.014514149, 0.012267488, 0.0061812922, 0.01086481, -0.021943588, 0.0015824703, 0.006288276, -0.013789036, -0.006793478, -0.00088113133, 0.011560205, 0.016451746, 0.009099576, -0.0051322556, -0.009628551, 0.026151622, 0.010852923, -0.0016731094, 0.041176915, 0.013777149, 0.015370021, 0.023358153, 0.017010441, 0.034639012, 0.011191705, -0.024368556, -0.05273118, -0.0025557266, 0.0007396005, 0.006912349, -0.0004877426, -0.00030925035, 0.0015089188, 0.012517117, 0.064998664, 0.013598843, -0.0065616793, 0.027910912, 0.002735519, -0.032118946, 0.008760793, 0.03238046, 0.017913861, -0.011898988, -0.016558731, 0.013254117, 0.068612345, 0.019578055, 0.02054091, -0.0048945136, -0.024915362, 0.0072035827, 0.0013855902, 0.0018543877, 0.032499332, 0.043720756, -0.033592947, 0.044624176, 0.03632698, -0.060481567, -0.030074364, -0.027411653, 0.00048959994, -0.000568352, 0.003607735, -0.015405682, -0.0046448843, 0.029955493, -0.0014844016, 0.019506732, 0.022062458, -0.007280849, 0.022062458, 0.062288407, 0.024261571, 0.032594427, 0.00049220026, -0.017176861, 0.0052332957, -0.008398237, -0.0047607836, -0.032974817, -0.016618166, 0.009794971, 0.046716306, 0.018377457, -0.0006861086, -0.012576552, -0.007560196, 0.0041129366, -0.02824375, -0.018674634, 0.0058722277, -0.008748906, 0.040891625, -0.02320362, 0.011221423, 0.0032838115, -0.009438357, -0.010169415, -0.014668682, -0.037991174, -0.026151622, 0.017913861, -0.04398227, -0.026104072, 0.0064428085, 0.03932253, 0.055251244, -0.012683536, 0.02501046, 0.017890086, 0.01901936, -0.0052808444, -0.02529575, -0.015025295, 0.015227376, 0.014050553, 0.021741506, 0.024891589, 0.0009866293, 0.014728118, -0.009485906, -0.014621134, 0.015239263, -0.048523143, 0.00475484, -0.023393814, 0.005313534, 0.036017913, 0.03801495, 0.015786069, -0.016499296, 0.0064428085, 0.037159074, 0.010953963, 0.016879683, -0.008475503, 0.044172466, 0.009913842, 0.012326923, -0.008742962, 0.013396762, 0.0026864847, -0.007637462, -0.026436912, 0.01582173, 0.004597336, 0.019613715, 0.008416067, -0.0036404245, -0.010490366, -0.025604814, 0.033521622, 0.0074710427, -0.013491859, -0.015274924, 0.001137447, 0.055774275, 0.027696945, -0.02529575, -0.02631804, 0.038110044, 0.012992601, 0.016653828, -0.00091902143, 0.016737038, 0.013479972, -0.0153105855, -0.0056909495, 0.019708812, -0.008517107, 0.006668663, 0.036350753, -0.038609304, 0.017866312, 0.0122793745, -0.037824754, -0.043673206, -0.0008514136, -0.020362603, -0.002035666, 0.03078759, -0.0088974945, 0.025414621, -0.008398237, -0.009652326, 0.0029658317, -0.015976263, 0.012398246, 0.033759367, -0.013598843, 0.006419034, 0.018745957, 0.011821722, 0.031381946, 0.0182467, 0.008784567, -0.0051411707, -0.0016315045, -0.015239263, -0.008267478, 0.010561689, 0.009783084, 0.014205085, 0.013943569, -0.038110044, -0.01561965, 0.065188855, -0.00067607884, -0.02139678, 0.017438376, 0.00739972, 0.015512666, 0.02824375, 0.01340865, 0.007815769, 0.012148617, 0.037991174, 0.033640493, -0.02693617, -0.004742953, -0.002839531, -0.016404198, -0.015536441, 0.015524553, 0.016844021, -0.04120069, 0.03525714, 0.0011344751, 0.0023120411, 0.008065398, -0.021432443, -0.00022046856, -0.012683536, 0.014038665, -0.017961409, -0.02841017, -0.04467172, 0.01279052, -0.034401268, 0.01799707, -0.007702841, -0.035328463, 0.01406244, 0.007185752, -0.029004525, -0.033141237, 0.004175344, 0.005548304, -0.030502299, -0.029242268, 0.017973296, 0.0026805412, 0.0025052065, -0.001057952, -0.02603275, 0.0067697037, -0.0065913973, 0.01980391, -0.03192875, 0.0048945136, 0.006430921, -0.002338787, -0.049925823, -0.023322491, 0.008731076, -0.02705504, -0.0032065453, 0.0078870915, 0.024118926, 0.03176233, 0.0028855936, 0.009551285, -0.03209517, 0.034662783, -0.014644908, 0.014086214, 0.030098138, -0.006104026, -0.0009286797, -0.0023046115, -0.019708812, 0.03887082, -0.0010193188, 0.019792022, 0.027031267, 0.015857391, 0.016665714, -0.002906396, -0.013349214, -0.037325494, -0.024487427, -0.00073477137, -0.02852904, 0.0054086307, -0.028029783, 0.023548346, -0.008107003, -0.026222944, 0.0060861954, 0.007173865, -0.0046983766, -0.015370021, -0.021052055, 0.027934685, 0.0109599065, 0.03841911, -0.02869546, -0.011031229, -0.008291253, -0.010401213, 0.033378977, -0.0394414, 0.008540882, -0.033474077, -0.03932253, -0.00021563943, 0.017438376, -0.019863345, -0.019138232, 0.04153353, 0.0050223, -0.044077367, 0.019982215, 0.014882649, 0.0139079075, 0.004282328, 0.01561965, -0.03226159, -0.005919776, -0.0012578038, 0.0074888733, -0.016166456, -0.007732559, 0.031025331, -0.005919776, -0.009670156, -0.047976337, -0.018115941, 0.032118946, 0.00041827734, 0.028885653, 0.0046062516, 0.0018142686, -0.05881737, 0.0063536554, 0.015595876, 0.0021263051, -0.019221442, -0.006484413, -0.036636043, 0.012683536, -0.008273422, -0.03801495, -0.027078815, -0.001560182, -0.016154569, -0.0012258572, 4.3787248e-05, -0.024962911, 0.05182776, -0.06290653, 0.028909428, 0.016487408, 0.010638955, 0.023358153, -0.03849043, 0.053777244, -0.013337326, 0.018175377, 0.0021738536, -0.035447333, -0.0140743265, 0.0068231956, 0.006246671, -0.0038395335, 0.01088264, -0.0065735667, 0.0029480008, -0.0031619687, 0.026175395, 0.013468085, 0.015381908, -0.016844021, 0.014502263, 0.02914717, -0.028338848, -0.0011211021, 0.010068374, 0.021789055, -0.00041827734, -0.022537943, 0.034377493, -0.040939175, -0.05178021, 0.019982215, 0.008023793, 0.031144204, 0.036136784, 0.034068428, 0.012142673, -0.0040624165, -0.047738597, -0.004056473, 0.033664268, 0.0006240728, 0.02841017, -0.015417569, -0.044624176, 0.0020401236, 0.009265995, -0.00039710346, -0.028600363, 0.01565531, 0.0035601866, -0.0077206716, -0.014573585, -0.01992278, 0.008683527, 0.0006578767, 0.026460685, 0.014133763, 0.0047607836, 0.043792076, -0.0016478493, -0.058056597, 0.024011943, -0.02017241, 0.036992658, 0.0059851552, 0.00785143, -0.018793507, 0.0055572195, -0.00977714, -0.021824716, -0.027078815, 0.003898969, 0.029527558, 0.016784586, -0.03169101, 0.011346238, -0.0078038815, -0.0047102636, 0.005578022, -0.030217009, 0.01975636, 0.030407203, 0.00904014, 0.025034234, 0.03841911, 0.04319772, 0.011465108, 0.016523069, 0.0042526103, -0.028671686, -0.01104906, -0.0061396873, -0.03131062, 0.008820228, 0.01664194, 0.0052273525, -0.04467172, -0.019958442, -0.031286847, 0.0073402845, 0.0075126477, 0.0069064056, 0.0023328434, -0.018175377, 0.02054091, 0.022110006, -0.0036790576, -0.0075780265, 0.02094507, 0.018044619, 0.010115922, -0.005821707, 0.0017429461, 0.011845496, 0.017699892, 0.016083246, -0.0123625845, 0.030644944, -0.0061218566, -0.030977784, 0.0012496314, 0.016511183, 0.01997033, -0.034092203, -0.021135265, -0.033806913, 0.038157593, -0.043340366, -0.007738502, -0.007637462, 0.0012711769, -0.007732559, -0.03689756, -0.008885607, 0.0074888733, -0.011898988, 0.029242268, 0.00053974864, 0.036754914, 0.047239337, 0.0015958432, 0.014466601, -0.0046151667, -0.016844021, -0.011958423, -0.02857659, 0.028172428, 0.004032699, 0.0014019349, 0.030098138, -0.013955456, -0.014335843, -0.033592947, -0.014930198, -0.015667198, 0.04307885, -0.012291262, 0.00024405702, -0.007334341, 0.0018945066, -0.025200652, 0.017533474, -0.007084712, 0.012303149, 0.01463302, -0.02054091, 0.017509699, -0.013563181, -0.030740041, 0.0048945136, 0.018282361, 0.012255601, -0.024653846, 0.01242202, 0.02213378, 0.017176861, 0.0068113087, 0.008445784, -0.023167958, 0.0010668673, 0.023084749, 0.008374462, 0.03711153, 0.017723667, 0.035352238, 0.0018855913, 0.025771234, 0.01771178, 0.02049336, -0.028743008, 0.0143833915, 0.03147704, 0.016867796, 0.029622653, -0.017830651, -0.02546217, -0.029575106, -0.015132279, -0.011964367, 0.031786107, -0.02000599, -0.03841911, -0.01160181, 0.0018945066, -0.009117406, -0.00256167, 0.037325494, -0.013955456, -0.0051857475, -0.008725132, 0.02139678, -0.00055163575, -0.035613753, -0.013670165, -0.008784567, -0.010222906, 0.03316501, -0.011631528, -0.0146092465, 0.020386377, -0.0032006018, -0.021610748, 0.009057971, -0.0051322556, -0.006609228, 0.029052073, -0.001055723, 0.019494845, -0.012861842, 0.020231845, 0.0025928738, -0.030359654, 0.017806876, -0.0009182785, -0.024154589, 0.008624092, -0.0023967365, -0.034377493, -0.029789073, -0.014811327, 0.0505915, 0.0073937764, 0.0041694003, 0.023952506, 0.023881184, 0.022264538, 0.034757882, -0.03775343, 0.013884134, -0.005482925, -0.006288276, -0.014704343, -0.017164974, -0.015227376, -0.012612213, 0.0014041638, -0.018139714, 0.0035245253, -0.015679086, 0.017985184, 0.034377493, 0.02586633, 0.006656776, 0.01043093, 0.017307619, 0.059530597, -0.007358115, 0.02909962, 0.027696945, 0.02660333, 0.006341768, 0.008469559, 0.011055003, 0.02139678, 0.021693958, 0.0057384977, 0.010359608, 0.005099566, 0.012552778, 0.039132334, -0.0312393, -0.027031267, -0.025176879, -0.019055022, -0.001771178, 0.028196203, -0.0021188757, -0.012481456, -0.006698381, -0.025509717, 0.0011300175, 0.018686522, 0.011447278, -0.01014564, -0.01242202, 0.002934628, 0.025081782, 0.003818731, -0.021800943, -0.02254983, -0.007791994, -0.035209592, -0.0033016421, 0.028838105, -0.007988132, -0.010805374, 0.017450264, 0.014181311, -0.04374453, -0.008309083, 0.0002815757, -0.0015898996, 0.030930236, 0.013123359, 0.019411635, -0.02783959, 0.0085884305, -0.0033462187, -0.030193236, 0.035376012, 0.008368519, 0.017937634, -0.022169443, -0.0045408723, -0.009396753, 0.028743008, -0.004546816, -0.02574746, 0.027578073, 0.0394414], "content_hash": "928d61d5538838576b0ce31317012e35"}, {"content": "GPT-style models) Most LLMs you'll encounter in this dashboard are **decoder-only** models. This means they generate text left-to-right, predicting one token at a time based on everything that came before it. Each token can only \"see\" the tokens to its left -- it cannot look ahead. ## The Residual Stream\n\nThere is an important concept called the **residual stream** (or \"residual connection\"). Think of it as a conveyor belt running through all the layers. Each layer reads from this stream, does some processing, and adds its result back. This means information from early layers is preserved and can be used by later layers.\n\n## How This Connects to the Dashboard\n\nThe dashboard's 5-stage pipeline follows the exact path data takes through a Transformer: Tokenization, Embedding, Attention, MLP, and Output. When you expand each stage, you're seeing what happens at that point in the architecture.", "source_file": "transformer_architecture.md", "chunk_index": 1, "embedding": [0.004264763, 0.035261303, 0.013565947, -0.0026626983, 0.022495592, -0.007373626, 0.012384643, 0.047582433, 0.001895803, 0.05467026, -0.013438926, -0.0041409167, -0.033559207, -0.04153619, -0.031831708, -0.043873392, 0.0026214162, 0.001240846, 0.0011352589, 0.062850475, -0.0012368766, -0.020475943, 0.011933715, 0.046845708, -0.01190196, 0.0042742896, 0.0042330073, 0.06442555, -0.019193022, 0.0151410205, 0.0033660824, -0.032314394, -0.030891746, -0.009380573, 0.029240461, -0.0026245916, 0.015814237, 0.05223144, -0.01987894, 0.04374637, 0.022152634, 0.016322324, -0.076416425, 0.036506116, 0.0022435258, 0.05807445, -0.012054386, -0.0151410205, -0.036658544, 0.06727084, -0.015306149, 0.022863956, -0.014074036, 0.015687214, -0.036048837, -0.037166633, -0.038259022, -0.014861572, -0.0208062, -0.024769286, 0.010206216, -0.0023356166, 0.016601773, 0.0032136561, 0.016868519, -0.001247197, -0.033965677, 0.026623808, -0.030561488, 0.030866342, 0.039427623, -0.009170987, -0.00999663, 0.0036614086, 0.01872304, -0.0011193812, 0.0049729105, 0.032314394, -0.011406574, -0.03300031, 0.024019856, -0.023422854, 0.008135758, -0.037319057, -0.00066805625, -0.00999663, -0.0027452626, -0.008250077, -0.011304956, -0.05421298, -0.0051221615, -0.023740409, -0.05223144, 0.0116479155, 0.017097158, 0.029748548, 0.023372045, -0.035845604, -0.017236883, 0.01398512, 0.006655952, -0.008948699, -0.0011955944, -0.04783648, 0.017287692, -0.060513273, -0.031247407, -0.016843114, -0.06498445, -0.047480818, -0.070370175, -0.022495592, 0.001205121, 0.009190041, 0.06722003, -0.010002981, -0.023270426, -0.008383451, -0.011755885, -0.023943644, -0.00020383058, 0.027843218, 0.010009333, -0.0017195601, -0.024591455, 0.022457486, 0.0036677597, -0.02572195, 0.0093043605, -0.026852448, -0.013667565, 0.0151410205, 0.015979365, -0.04514361, -0.013095967, -0.002361021, -0.033355974, 0.00548735, -0.014721848, 0.005506403, 0.035210494, -0.026217338, 0.01120969, -0.02103484, -0.026420573, 0.014925083, -0.0323652, 0.044051223, -0.065746576, 0.015725322, 0.010447558, 0.0046839355, -0.022978276, -0.02443903, -0.033559207, -0.0007363306, -0.012505314, 0.020209197, 0.032492224, -0.030536084, -0.029240461, 0.016220707, 0.04501659, 0.022178037, -0.06406989, 0.04270479, -0.02178427, 0.012695847, -0.047404602, -0.04224751, -0.020171091, 0.00096616097, -0.039707072, -0.0007494297, 0.0009534588, -0.04593115, -0.08815326, -0.030967958, 0.03155226, -0.057464745, -0.033737037, 0.007900767, -0.038513064, -0.022813149, -0.05975114, 0.019434363, -0.012117897, 0.06915077, -0.0072084973, 0.012727602, -0.02670002, 0.065289296, 0.054416217, 0.034397554, -0.006252657, 0.0054524187, 0.0502753, 0.0075387545, 0.03132362, 0.0027182703, 0.031349026, 0.040596224, 0.02346096, -0.068642676, -0.013642161, 0.07336789, 0.002481692, 0.0034994555, -0.025213864, -0.026598403, 0.007888065, -0.02155563, 0.016589072, -0.008599388, -0.049563978, -0.011419276, -0.04986883, 0.016703391, 0.037928764, 0.012124248, 0.006884591, 0.031984136, 0.008764517, 0.012092493, -0.06340937, 0.026827043, 0.015636407, -0.012505314, -0.0011050912, 0.0040075434, 0.017147968, -0.008986806, -0.0015028288, -0.006865538, 0.005026895, -0.062139153, -0.008186567, -0.027131895, 0.02508684, 0.013261095, -0.030104209, 0.018888168, 0.017465523, 0.002959612, 0.038360637, 0.020933222, -0.02495982, -0.018176846, 0.032212775, 0.033330567, 0.024756584, 0.0019275586, -0.06986209, 0.029850166, -0.026877852, 0.024057964, -0.038411446, 0.0042552366, -0.024896309, -0.0069354004, 0.04374637, 0.020171091, -0.03345759, 0.032720864, 0.0023435554, -0.014671039, 0.010892135, -0.04488957, -0.029291268, -0.038208213, 0.006332046, -0.046261407, -0.02189859, 0.012327483, -0.031196598, -0.037065014, 0.033355974, -0.0054937005, 0.022571806, 0.05385732, -0.0012797464, -0.0520028, 0.055737246, 0.04415284, -0.031425238, 0.013743779, -0.0055445093, -0.004344152, -0.01600477, -0.018087931, 0.017262287, 0.010060141, 0.035337515, -0.022152634, 0.015331553, -0.001973604, 0.0065670363, 0.017122563, -0.0010908013, -0.019218426, 0.018799253, -0.030434467, -0.036455307, 0.009990279, -0.013070562, 0.025226565, 0.016131792, 0.0111080725, 0.00013297614, -0.012086142, -0.02184778, -0.032492224, 0.06742327, 0.035439134, 0.011768587, -0.003327976, 0.036429904, -0.037623912, 0.012994349, -0.035896413, -0.008472366, -0.021695355, -0.027716197, -0.036912587, -0.0103268875, 0.0083008865, -0.018075228, -0.010434856, -0.022000207, -0.027081087, 0.043365303, 0.003327976, -0.019612193, 0.04519442, -0.046540853, -0.027919432, -0.0039853146, 0.016970137, -0.008097651, 0.020132985, -0.058582537, -0.020577561, -0.036785565, -0.004372732, -0.016195303, 0.029316673, -0.032238178, 0.027614579, 0.039376814, -0.034651596, -0.0073101153, -0.023257725, -0.017948207, -0.023778515, 0.039605454, 0.030231232, 0.0108667305, 0.023118, 0.02756377, -0.0062018484, -0.0572107, -0.030409062, 0.0015496681, 0.022203442, -0.05159633, 0.030155018, -0.027589174, -0.020044068, 0.009742586, -0.022787744, 0.052485485, 0.0039789635, 0.007265657, 0.03322895, -0.0543146, -0.025353588, -0.0026023628, 0.0032581137, -0.04524523, 0.018164143, 0.00019569324, 0.012327483, -0.0043028696, -0.0151410205, 0.024375519, 0.054619454, -0.029773952, -0.025671143, -0.02362609, 0.01195912, 0.001846582, -0.029316673, 0.013832694, 0.013375415, -0.0055953185, -0.013477032, 0.026623808, 0.0108857835, 0.03213656, 0.033432186, -0.008415206, -0.007551457, -0.035337515, -0.049691, 0.022393975, 0.015687214, 0.027817814, 0.0012202049, 0.04697273, -0.015191829, 0.057007466, 0.0036995152, 0.009660021, -0.032111157, -0.011216041, 0.038335234, 0.025823569, 0.008402504, 0.012225865, 0.065390915, -0.019751918, 0.014074036, -0.056651805, -0.0034581735, -0.023016384, 0.004991964, 0.055025924, -0.030078806, -0.0040138946, -0.033533804, 0.02391824, 0.031018768, 0.04085027, -0.039275195, 0.0006208199, -0.029113438, 0.017122563, 0.0043822583, -0.04709975, -0.020031366, 0.054568645, -0.0028786357, -0.023245022, -0.030942556, -0.061986726, -0.012822869, -0.04120593, -0.04638843, -0.029545313, -0.04293343, 0.023016384, 0.05159633, 0.033686228, 0.021098351, -0.041002695, -0.07153878, 0.0681854, 0.018799253, 0.035820197, -0.002291159, 0.014074036, 0.024362816, -0.019853536, 0.0038233616, -0.04374637, 0.0066114943, 0.0213778, -0.025925186, 0.036429904, 0.013819992, -0.020209197, -0.0340927, 0.012162355, -0.006624196, -0.014251866, 0.0015345843, 0.016512858, -0.036582332, -0.004328274, -0.034448363, 0.029189652, 0.009742586, -0.026344359, -0.010790518, -0.0017973611, 0.11838449, -0.04524523, -0.01958679, -0.018405486, -0.014277271, 0.0013424635, -0.029621527, -0.015077509, -0.013134073, 0.016131792, -0.019053297, -0.07184363, -0.07153878, 0.0056080204, -0.004179023, 0.028224284, 0.017846588, -0.028097263, 0.04257777, 0.042552363, -0.0012995936, 0.017986313, 0.02207642, 0.002700805, 0.012359239, -0.007957927, 0.01219411, -0.0085422285, 0.017109862, -0.013095967, 0.0068972935, -0.01259423, 0.013908907, -0.0007406969, 0.026674615, 0.005246008, 0.0055603874, -0.018189548, -0.018126037, -0.009348818, -0.006011315, -0.020653775, -0.091252595, -0.03386406, 0.009793395, 0.013184882, -0.012860975, -0.014264569, 0.0072720083, -0.05032611, -0.0059446287, -0.0048966976, -0.05426379, -0.02466767, -0.005512754, 0.010358643, 0.037395272, 0.01254342, 0.022927467, -0.012619634, 0.0065035257, 0.012067088, -0.054822687, 0.0057413937, 0.04577872, 0.026217338, -0.041993465, 0.028275093, -0.013146775, -0.034168914, -0.018303867, -0.015052104, 0.012371941, -0.008650197, -0.033533804, -0.0040107192, 0.026039505, 0.021022137, -0.035591558, -0.02502333, 0.033127334, 0.011686022, -0.0028659333, 0.012454505, -0.014582124, -0.020005962, -0.0040932833, 0.007983332, 0.021695355, -0.021771567, 0.005858889, -0.00033521894, 0.028021049, 0.0038043084, 0.00012563268, -0.028402114, -0.05512754, -0.030993363, -0.011317658, -0.028681563, 0.04872563, -0.010339589, -0.006176444, -0.02045054, 0.019777322, 0.046591662, -0.015801534, -0.014950487, 0.034473766, -0.034905642, 0.004664882, 0.035235897, -0.008681952, -0.035210494, -0.012124248, 0.014175653, -0.011406574, -0.031603068, 0.0014829816, 0.021415906, -0.001649698, -0.028808586, 0.039859496, -0.018468997, -0.030536084, -0.018087931, -0.02160644, -0.013565947, -0.028986417, -0.055889674, 0.02299098, -0.053145997, 0.009545702, 0.05050394, 0.010854028, -0.004096459, -0.018977083, 0.0046013715, -0.032974906, 0.003877346, 0.009005859, -0.013210286, 0.007119582, -0.05126607, 0.001995833, -0.04067244, -0.001402799, 0.06437474, 0.008885188, -0.002029176, 0.016258813, -0.001846582, 0.0037280952, 0.021644546, -0.025353588, -0.003045352, -8.484672e-05, -0.006408259, -0.0067639207, 0.012225865, 0.010441207, 0.017681459, 0.027081087, -0.0219494, -0.009945821, -0.024261199, -0.034524575, 0.068642676, 0.011914662, 0.009507596, -0.007824554, -0.0008256429, 0.0041504432, 0.0066368985, 0.015445873, -0.023435555, -0.011832098, -0.029748548, -0.026979469, -0.026344359, 0.050478537, 0.015382362, 0.008275482, 0.029951783, 0.012702198, -0.057820406, 0.02629355, 0.008993156, -0.008186567, 0.018392783, -0.02287666, -0.036455307, -0.06594981, -0.015814237, 0.0044235406, -0.0011797167, 0.0035375622, -0.026674615, 0.00048030188, -0.0072720083, -0.01357865, 0.022914765, -0.0467949, 0.01144468, 0.0045061046, -0.014823466, -0.011533596, 0.012492612, -0.026979469, -0.02670002, -0.0048046065, -0.0071386355, 0.07580672, 0.012702198, -0.031120386, 0.0030437643, -0.033991084, -0.036277477, -0.0008200857, 0.0036487065, -0.002248289, 0.017059052, 0.029697739, -0.007646723, -0.04085027, -0.036506116, 0.021644546, 0.018037122, -0.014175653, -0.027538365, 0.024693074, -0.01213695, -0.012340185, -0.002087924, -0.02531548, -0.050478537, 0.02692866, -0.018240357, -0.011133477, 0.010822273, -0.043720964, -0.037522294, -0.032924097, -0.037598506, 0.004299694, 0.006662303, 0.03980869, -0.007005262, -0.04125674, -0.013007051, 0.0022990978, 0.004582318, -0.014759954, -0.02316881, -0.001486951, -0.027665388, -0.0016592246, -0.04610898, 0.035489943, -0.006309817, 0.028961012, 0.0011328773, 0.030358253, 0.0003088222, -0.06442555, -0.0038614683, 0.010193514, 0.0001751514, 0.007983332, -0.015522086, 0.010218918, 0.02733513, -0.004512456, 0.016436644, -0.026090315, 0.02560763, 0.018786551, 0.00543019, 0.021619141, -0.003721744, -0.028961012, 0.022381274, 0.017033648, 0.032161966, 0.01357865, 0.018684933, -0.02184778, 0.03213656, 0.00034514253, 0.027792409, -0.024997925, 0.028300498, 0.016919328, -0.017020946, 0.008834379, -0.0052047255, 0.0039821393, 0.0048331865, -0.015992068, -0.001064603, 0.0067512183, -0.038741704, 0.009190041, -0.0026626983, -0.031425238, -0.024642264, 0.008516824, 0.018227654, -0.01409944, 0.0010177636, -0.0045315092, 0.017389309, -0.115335956, -0.041409165, -0.0038233616, -0.015153723, 0.015522086, 0.0010257025, 0.031018768, -0.016970137, 0.015928557, -0.034473766, -0.017719567, -0.033737037, 0.006649601, -0.01577613, 0.017808482, -0.050122876, 0.0021958924, -0.025277374, 0.016868519, -0.007386328, -0.050757986, -0.02391824, 0.011133477, -0.0037249196, -0.06274886, -0.024743881, -0.014289973, 0.0072021466, 0.026623808, 0.055229157, -0.0022927467, 0.01323569, -0.011755885, 0.0011908311, 0.0020180617, -0.0046426533, -0.0029564367, -0.041993465, 0.028148072, 0.017186074, -0.015280744, -0.032111157, -0.009183689, -0.009158285, 0.027309727, -0.015801534, -0.012937189, -0.020247303, -0.0044743493, -0.0016957434, -0.027284322, -0.014340782, 0.00087803946, 0.018494401, 0.017706864, 0.02393094, 0.016487453, -0.0069481023, 0.011044561, 0.0052110767, -0.015903153, 0.015242638, 0.015090211, -0.0146583365, 0.017783077, 0.010161758, 0.027411344, 0.033889465, -0.039249793, -0.039249793, -0.012187759, -0.018596018, 0.018634126, 0.006309817, -0.007862661, 0.020310815, 0.009850555, 0.049386147, -0.0116479155, 0.015839642, 0.012111546, 0.001057458, -0.057464745, 0.0065733874, 0.02531548, 0.013642161, -0.00067281956, -0.016893923, -0.0014401117, 0.029977188, 0.038817916, -0.0008335817, -0.026877852, -0.015445873, -0.0027468503, -2.3134375e-05, -0.011914662, 0.011686022, 0.059192244, -0.020196496, 0.039427623, 0.009075721, -0.06290129, -0.022559104, -0.023041787, 0.0001646324, 0.020018665, -0.010117301, -0.01086038, -0.03363542, -0.009374222, -0.029240461, 0.011050913, 0.018481698, -0.021644546, 0.007875362, 0.068744294, 0.03282248, 0.0467949, -0.023842026, -0.0070179645, -0.0045156316, -0.004118688, 0.011533596, -0.026141124, 0.007945225, 0.016487453, 0.034880236, 0.011133477, -0.005420663, -0.015433171, 0.003791606, -0.003059642, -0.028021049, -0.023676896, 0.0146583365, -0.01195912, 0.049386147, -0.044813354, 0.013565947, -0.012645038, 0.008701006, 0.029545313, -0.02074269, -0.015356958, -0.007869012, -0.000352486, -0.027639983, -0.007767394, 0.028808586, 0.034448363, 0.046718683, -0.017643353, 0.0064654187, -0.006033544, 0.018646827, -0.007125933, -0.06417151, -0.02386743, 0.00814846, 0.041917253, 0.044508502, -0.012587878, -0.020437837, 0.04062163, -0.01958679, -0.0093234135, 0.007957927, -0.033914868, 0.016360432, -0.044279862, 0.007989682, 0.024997925, 0.007430786, 0.031069577, 0.0025642563, -0.0023594333, 0.035489943, 0.008275482, 0.011025508, -0.024997925, 0.056905847, -0.007761043, 0.037928764, 0.010263376, -0.012435452, 0.0013321431, 0.0033629069, -0.048624016, 0.022114526, 0.010314185, 0.016779603, 0.0146583365, 0.020196496, -0.035972625, -0.02629355, 0.049487762, 0.017008243, -0.016868519, -0.0035820198, 3.0812753e-05, 0.021809675, 0.017656056, -0.004547387, 0.003108863, 0.01201628, 0.022038314, 0.0040488257, -0.006344748, -0.0018719864, -0.0032342973, 0.00064900296, -0.008332642, 0.023041787, -0.010314185, -0.008535877, 0.04021516, -0.0054746475, 0.028656159, -0.0068210806, -0.029519908, -0.022139931, -0.0052047255, -0.008015087, 0.0025706075, 0.034931045, -0.0143788885, 0.0076657766, -0.009151934, 0.012003577, 0.0066940584, 0.0048808195, 0.0055508604, 0.004118688, -0.022139931, -0.0021355571, 0.0055000517, 0.008167514, 0.03975788, 0.018900871, 0.008472366, 0.009545702, -0.030027997, -0.029265864, 0.008701006, -0.013083264, 0.012670442, 0.0009891837, -0.010841326, -0.01143833, 0.016855817, 0.060310036, 0.02316881, -0.0067512183, -0.0031961906, -0.0018227655, 0.015064807, 0.02507414, 0.0062875883, 0.019332746, 0.009526649, 0.05426379, 0.029773952, -0.029672334, 0.01335001, 0.0011566939, -0.005166619, 0.007856309, 0.011952768, 0.015674513, -0.008808974, 0.022863956, -0.0073990305, -0.0070878267, 0.01259423, -0.0068274317, 0.014366186, -0.014010524, 0.012416399, -0.018811956, -0.017795779, -0.05350166, 0.001001092, -0.031349026, 0.019891642, -0.0069481023, -0.029265864, 0.004934804, 0.017719567, -0.020171091, 0.0062081995, 0.013845396, 0.028706968, -0.007291062, -0.008326291, 0.0029310323, 0.018303867, -0.013108669, 0.020132985, -0.058938198, -0.009399626, -0.0047665, 8.2415434e-05, -0.032390606, 0.0013337308, -0.011412925, 0.013794587, -0.02404526, -0.005928751, -0.0018640476, -0.017821183, -0.01103821, 0.016360432, 0.022279656, 0.02466767, -0.0034772267, 0.018329272, -0.030790128, 0.012333835, -0.038843323, -0.0085422285, 0.012803815, -0.010466611, -0.0025023331, 0.019561386, 0.0027722546, 0.024997925, 0.0018021243, 0.009463138, 0.020387027, 0.02606491, -0.00066091126, -0.0029643755, -0.011736831, -0.02339745, -0.021885887, -0.013972418, -0.041002695, -0.018583316, -0.039605454, 0.045448463, 0.012314782, -0.025239268, 0.003963086, -0.0009828326, 0.027462153, 0.031831708, -0.027792409, 0.05390813, -0.0033629069, 0.04412744, -0.038004976, -0.015026701, -0.00641461, -0.011425627, -0.00077800965, -0.008948699, -0.006373328, -0.022292357, -0.01167332, 0.016550964, 0.035591558, -0.013947014, -0.01999326, 0.02652219, -0.0051570926, -0.02056486, -0.001275777, 0.019053297, 0.011317658, 0.015318851, 0.0083008865, -0.006373328, -0.012276675, -0.018621422, 0.004668058, -0.013400819, -0.01213695, 0.020298112, 0.014201057, -0.00803414, -0.042095084, -0.0012003577, 0.015458575, -0.0036677597, 0.0260014, 0.025378993, 0.023384746, -0.04560089, -0.01641124, 0.027131895, -0.009374222, -0.016881222, -0.007945225, -0.021568332, -0.0031580841, 0.01652556, -0.017097158, -0.029951783, -0.031298216, -0.016309623, -0.005744569, 0.003963086, -0.012740305, 0.039376814, -0.05807445, -0.005280939, 0.025163054, -0.0020275884, 0.005134864, -0.038843323, 0.011546298, 0.0098442035, 0.003693164, 0.0072529553, -0.044635523, -0.010390398, 0.014340782, -0.0111080725, 0.023994451, 0.007919821, -4.5921424e-05, -0.017109862, 0.0006140719, 0.028529137, 0.011622511, 0.014556719, -0.0015853931, 0.008135758, 0.028173475, -0.029392887, -0.0061097573, -0.030561488, 0.011057263, 0.0052999924, 0.004318747, 0.038741704, -0.021822376, -0.034041893, 0.0057922024, 0.018596018, 0.01525534, 0.020425135, 0.019548682, 0.014048631, -0.017351203, -0.038995747, -0.02443903, 0.02039973, 0.008199269, 0.01196547, -0.026420573, -0.020602966, 0.015509384, -0.010523771, 0.0005970033, -0.05680423, 0.025455205, -0.011940067, 0.005477823, -0.011196988, -0.008065896, -0.00470934, 0.01254342, 0.021009436, 0.024756584, 0.017529033, 0.055483203, -0.011139828, -0.026623808, 0.033737037, 0.002184778, 0.031907924, -0.015903153, 0.022089122, -0.012467207, -0.01282922, 0.018227654, -0.014277271, -0.035007257, -0.021161862, 0.040723246, 0.0141375465, -0.028452924, -0.017478224, -0.012283025, 0.003135855, -0.024299305, -0.033889465, 0.04765865, 0.015877748, 0.0039313305, -0.005246008, -0.0015901565, 0.032949504, 0.023181511, 0.016157197, 0.010993753, -0.024235794, 0.014429697, -0.016208004, -0.018291166, 0.010593633, -0.012384643, 0.015039403, -0.036252074, 0.0025213864, -0.007983332, 0.005074528, 0.0064876475, 0.0011868617, 0.0016322325, -0.04743001, 0.023956345, 0.008662899, 0.014594826, -0.01566181, 0.018481698, 0.0123909945, -0.014925083, -0.028249688, -0.011158881, 0.008834379, 0.024781989, -0.0063193436, -0.025582228, 0.016042877, -0.004347327, -0.033533804, 0.013197584, -0.017554438, 0.015128318, 0.022317762, 0.0012995936, -0.0034232424, 0.023842026, -0.032263584, -0.00061010243, -0.0068401336, 0.00060732383, 0.009812448, -0.026674615, -0.02362609, 0.004207603, -0.012200462, 0.032543033, 0.0013242041, 0.033686228, 0.0404438, 0.004080581, 0.00021256335, -0.015725322, -0.024019856, -0.001247991, -0.023118, 0.032492224, -0.017147968, -0.024921713, 0.0093424665, -0.008218322, -0.034829427, -0.010580931, -0.019396257, -0.010257025, 0.048573207, -0.001987894, 0.0025150352, -0.0066432497, 0.013858099, -0.02652219, 0.021225372, -0.026496785, -0.0008161162, 0.019066, -0.003186664, 0.0088216765, 0.008675601, -0.02039973, 0.02520116, -0.025531419, 0.0013289675, -0.03993571, 0.005490525, 0.003518509, 0.018697636, 0.019383555, -0.007259306, -0.019294638, -0.00704972, 0.0014250278, 0.0024546997, 0.026725424, 0.026852448, 0.02670002, 0.020958627, 0.031450644, -0.0038170104, 0.0074879457, -0.047353793, 0.031526856, 0.013477032, 0.010231621, 0.0063987323, -0.030104209, -0.02507414, -0.0008327879, -0.016716093, 0.011819395, 0.015407766, -0.00635745, -0.013388117, 0.014785359, 0.016538262, -0.012606932, 0.028452924, 0.0154204685, 0.00652893, -0.007983332, -0.01335001, 0.029088033, 0.001353578, -0.029672334, -0.000641858, -0.016322324, -0.011876555, 0.05017368, -0.019891642, 0.009151934, 0.009520298, -0.0013027692, -0.0068464847, 0.021403205, -0.0076657766, -0.013451627, 0.012276675, -0.014874274, 0.010669846, 0.004490227, 0.018507103, -0.009228147, -0.028986417, 0.018265761, -0.0031580841, -0.023956345, 0.021758866, 0.004115512, -0.019624896, -0.038208213, 0.012448154, 0.043822583, -0.008770868, 0.012575177, 0.017071754, 0.016500155, 0.024921713, 0.042196702, -0.011832098, -0.0074498393, -0.006309817, -0.025391694, -0.000398333, 0.0029691388, -0.017033648, -0.0054682964, 0.02288936, -0.030256636, -0.0041536186, -0.026268145, 0.016563667, 0.03241601, 0.013642161, -0.004369556, -0.0039567347, 0.026674615, 0.04722677, -0.007297413, 0.014975891, 0.006373328, 0.024134176, -0.0046299514, 0.0151410205, 0.015331553, 0.017986313, 0.018075228, 0.008472366, 0.013477032, 0.017630652, 0.011000103, 0.029570717, -0.00050014904, -0.020412432, -0.0018370554, -0.034245126, -0.007634021, 0.03200954, 0.013095967, -0.025747355, -0.039148174, -0.009869608, 0.0116479155, -0.0011082668, 0.025163054, -0.025290076, -0.02149212, 0.020958627, 0.008065896, -0.00027607434, -0.0042266566, -0.041104313, -0.011832098, -0.030790128, -0.030866342, 0.035998028, -0.0032152438, -0.00057358365, 0.01167332, 0.008421557, -0.067016795, -0.032746267, 0.013743779, -0.005223779, 0.02039973, 0.025683844, 0.0018338799, -0.014010524, -0.010968348, -0.024642264, -0.0035820198, 0.038030382, 0.013095967, 0.027182704, 0.005928751, 0.0021466715, -0.006274886, 0.008478717, -0.021936696, -0.0074752434, 0.028884798, 0.010472963], "content_hash": "22be9f22e1d3793d7afd1facc577b6b5"}, {"content": "# Troubleshooting and FAQ\n\n## Common Issues\n\n### Model takes a long time to load\n\n**Why**: The first time you load a model, it must be downloaded from HuggingFace. GPT-2 (124M) is about 500MB; larger models are much bigger.\n\n**Fix**: Be patient on the first load. Subsequent loads should be faster because the model is cached locally. If loading is consistently slow, try a smaller model.\n\n### Ablating a head has no effect\n\n**Why**: Not every head is important for every input. Many attention heads are redundant -- the model has learned to distribute work across multiple heads, so removing one doesn't always change the output.\n\n**Fix**: This is actually an interesting finding! Try:\n- Ablating a head from a different category (Previous-Token heads often show more effect)\n- Using a different prompt (some prompts depend more on specific heads)\n- Ablating multiple heads simultaneously to see if their combined removal has an effect\n\n### Attribution takes too long\n\n**Why**: Integrated Gradients is computationally expensive because it runs the model multiple times (typically 50 steps) to build up the attribution scores.\n\n**Fix**: Switch to \"Simple Gradient\" for faster (though less accurate) results. Or use a shorter prompt -- fewer tokens means faster computation.\n\n### The model's prediction seems wrong or nonsensical\n\n**Why**: Small models like GPT-2 (124M) have limited knowledge and can produce incorrect facts, repetitive text, or non-sequiturs. The model was trained on data from before 2019 and has a limited understanding of the world.\n\n**Fix**: This is expected behavior for small models. The dashboard is designed for exploring *how* the model works, not for getting useful outputs. Try different prompts or a different model.\n\n### BertViz visualization is hard to read\n\n**Why**: With 12+ heads selected simultaneously, the attention lines overlap and become a dense mess.", "source_file": "troubleshooting_and_faq.md", "chunk_index": 0, "embedding": [-0.028126141, 0.018463913, 0.0643142, 0.0061068297, -0.053293224, 0.0021551298, -0.007276865, 0.022102347, 0.0058615, 0.050183196, 0.0011294616, -0.02862435, -0.041638162, -0.03321391, -0.013836612, 0.00029062168, -0.00027929878, 0.0068654655, -0.008756394, 0.05806395, 0.010318957, -0.012817549, 0.022223124, 0.03831677, 0.044567026, 0.03451227, 0.028397892, 0.014840578, 0.045321885, -0.027401473, 0.015248204, -0.019686788, -0.061204173, -0.06226098, -0.0108398115, 0.01782983, 0.010568062, 0.02229861, -0.0036761756, 0.05537664, -0.023038374, -0.004095124, -0.03182496, 0.029666059, -0.012734515, 0.018388428, 0.026676808, -0.003766759, -0.06322721, 0.051270194, -0.014297077, 0.06491809, -0.052266613, 0.002368378, -0.0030402048, -0.023778139, -0.019762276, -0.013104397, -0.004929246, -0.0115267355, 0.045684222, 0.013361049, -0.009533902, 0.02829221, 0.008960207, -0.028971586, -0.02551432, 0.00030383177, -0.029696254, -0.03004349, -0.008937561, -0.021287095, 0.027235404, -0.022540165, 0.00637858, -0.0025552064, 0.009035693, 0.044778388, 0.003764872, -0.0419703, 0.0096999705, -0.010266117, -9.394488e-05, -0.028941391, 0.0013266691, -0.023310125, -0.049851056, -0.032610018, -0.04486897, -0.06419343, 0.017965704, 0.030269949, -0.025227472, 0.007865657, 0.0042008045, -0.037471328, 0.01897722, -0.021105928, 0.003076061, 0.013066653, 0.014417855, -0.01576151, 0.0005118905, -0.001545579, 0.02791478, -0.022223124, 0.0067182677, 0.00010656522, -0.01793551, -0.0017399559, -0.07264788, 0.011979653, -0.03085874, -0.01810158, -0.04160797, -0.017256135, -0.06896415, 0.03282138, 0.041547578, -0.03303274, 0.022117443, 0.0140404245, 0.015399176, -0.0067975284, -0.0018446929, -0.006303094, 0.012470312, -0.02774871, -0.00806947, -0.020079317, 0.015489759, 0.027114626, 0.036988217, -0.024246152, -0.005552007, -0.03303274, -0.031583406, 0.01989815, -0.035840828, 0.009858492, -0.048009194, -0.02463868, -0.013232723, 0.014055522, -0.042000495, 0.030345434, 0.029892517, 0.018086482, -0.058305506, 0.026676808, 0.0029043297, 0.028639447, -0.012960972, -0.057127923, -0.012696771, -0.061355148, 0.025604904, 0.06727326, -0.04290633, -0.07910949, 0.016289912, 0.03004349, -0.035146352, -0.0057482705, -0.02578607, 0.09909822, -0.036021993, -0.029258434, -0.003066625, -0.09203272, 0.027975168, -0.033546045, 0.017799634, -0.05739967, 0.022389194, 0.023159152, -0.07766016, -0.032791186, 0.014765092, -0.048854638, -0.030873837, 0.00389886, -0.010205728, 0.013881904, -0.052749723, -0.035357714, -0.04399333, 0.020970054, -0.0027571318, 0.02213254, 0.015625633, 0.014976453, 0.056191895, -0.023446, -0.02164943, -0.03849794, 0.06962843, -0.0011870197, -0.0062238337, 0.03152302, 0.0046914644, 0.025604904, 0.05468217, -0.044385858, 0.0023098763, -0.017210843, -0.00466127, 0.03330449, -0.052477974, -0.014251786, 0.021468263, -0.036776856, 0.015278398, 0.021936277, -0.044959553, -0.014183848, -0.04973028, 0.014878321, 0.0072277994, 0.048190363, 0.011790938, 0.041124858, 0.01330066, 0.004940569, -0.018962122, 0.01935465, 0.043661192, -0.024487708, 0.027990267, 0.021302193, 0.031039907, -0.0029968002, 0.003532752, 0.019218775, 0.033244103, -0.045321885, 0.03892066, 0.013708286, -0.0046310755, 0.0006029457, 0.03735055, 0.023672458, 0.007948692, -0.01297607, 0.036927827, 0.0050839926, -0.051994864, -0.068662204, 0.0485225, 0.032700602, 0.038045023, -0.0218004, -0.018463913, -0.005948309, -0.009209311, -0.0108171655, -0.03411974, 0.008084567, -0.025876654, -0.016652245, -0.0045291693, 0.030556796, -0.058275312, -0.00039960482, -0.022494875, -0.0044423603, -0.03221749, 0.0019192355, -0.07222515, -0.051662724, 0.025484126, -0.019128192, 0.035418104, 0.011949458, -0.04655986, -0.011662611, -0.0043630996, 0.027899683, -0.03254963, -0.01210043, 0.0028326178, 0.009262151, 0.030662475, 0.0283526, -0.019943442, 0.010235922, -0.011617319, -0.014795287, -0.024759458, 0.0068163997, 0.016591856, -0.024789654, 0.0078732055, 0.02747696, 0.009224408, 0.004400843, 0.030134073, 0.009609387, -0.023672458, -0.031039907, -0.005804885, -0.05223642, -0.0028250692, 0.0053557428, 0.022238221, 0.024910431, -0.009148922, 0.04351022, 0.014704702, 0.003644094, -0.03297235, -0.024427319, -0.007182508, 0.004732982, -0.023083666, -0.024246152, 0.0063559343, -0.0010105709, 0.024683973, -0.008235539, -0.04018883, -0.07144009, -0.025635097, 0.043570608, 0.026737196, -0.003068512, -0.005986052, 0.019445233, -0.011707903, -0.010077402, 0.025151987, -0.0022287287, -0.025151987, 0.01733162, -0.02862435, -0.002109838, -0.025831362, 0.030179365, -0.002859038, 0.043540414, -0.07729782, -0.0033629083, -0.011322923, -0.0019739629, -0.06654859, 0.031070102, 0.027129725, 0.004340454, 0.018222358, 0.0030590764, -0.021694722, 0.01749769, -0.029122557, -0.022087248, 0.0062125106, 0.057943173, -0.033425268, -0.004095124, -0.033183713, -0.013172334, -0.018614886, 0.025091598, 0.025650196, 0.021498457, -0.012274048, 0.00970752, -0.016893802, 0.011624868, 0.0040422836, -0.009730165, -0.010786971, 0.037229773, -0.0025759651, -0.018116677, -0.05984542, -0.021362582, 0.00029699085, 0.015610537, -0.03466324, 0.04764686, 0.030949323, -0.0038328094, -0.00493302, -0.018584691, 0.034270715, 0.006661653, 0.010069853, -0.02023029, -0.07017193, -0.0045518153, 0.018041192, -0.0058426284, -0.058516867, 0.045231305, 0.01958111, 0.06238176, -0.0077297823, 0.030617185, 0.042845942, 0.058426283, 0.01401023, -0.0070579555, -0.06860182, 0.028095948, 0.01613894, 0.037803467, 0.03330449, 0.050454944, 0.014606571, -0.005295354, 0.05211564, -0.043661192, 0.010885104, -0.01613894, 1.3549175e-05, -0.016546564, 0.05293089, 0.032036323, 0.021362582, 0.04613714, -0.0068767886, -0.060539898, -0.03152302, -0.011179499, -0.09402555, 0.010643547, 0.043147884, -0.028111044, 0.03182496, -0.047616668, -0.030979518, 0.03321391, 0.020381263, 0.020411456, 0.031281464, -0.025891751, 0.026857974, 0.013451633, -0.0052802563, -0.007337254, 0.04465761, -0.013013813, 0.013647897, -0.0052349647, -0.0069107576, -0.011783388, -0.02115122, -0.019837761, 0.020018928, -0.040853105, 0.008786588, 0.0048122425, -0.04027941, -0.0016635262, 0.0027854391, -0.040762525, 0.06564276, 0.0061898646, 0.073010206, -0.030722864, 0.01330066, 0.02676739, -0.03750152, 0.026797585, 0.0038346967, 0.031100295, -0.021724915, 0.016395593, 0.003372344, -0.0041970303, 0.0001454288, -0.029213142, 0.020532234, -0.050636113, -0.01265148, -0.025695488, -0.017618468, 0.051813696, -0.022509972, -0.039403774, -0.023808334, 0.005185899, 0.030737963, 0.0239895, 0.019505622, 0.045533247, -0.009556547, -0.009103631, 0.020577526, 0.003963023, -0.018343136, -0.03484441, 0.053715948, -0.019022511, 0.008235539, 0.015504856, -0.028669642, -0.043631, 0.012092882, -0.03025485, 0.009201762, -0.011330471, 0.003876214, 0.02006422, -0.008839428, -0.009148922, 0.04411411, 0.0119570065, -1.7323482e-05, 0.04067194, -0.050968252, -0.010190631, 0.0030703994, 0.031613603, -0.034693435, 0.03526713, -0.00046070144, 0.028065752, 0.042604387, 0.03985669, 0.0104095405, 0.010635999, -0.006238931, 0.0032647762, -0.0010577497, -0.015670925, -0.023793235, -0.031643797, -0.02213254, -0.016455982, 0.032187298, -0.044174496, 0.0218004, -0.032036323, -0.06672976, -0.0041743843, -0.019535817, -0.021060636, -0.04290633, -0.012523153, 0.008461998, -0.00269863, 0.011375763, 0.042755358, -0.0046046553, 0.012621285, 0.012409924, -0.022615653, 0.0006416323, 0.032911964, 0.01248541, -0.03233827, 0.040309608, -0.02568039, -0.0013276128, -0.025755877, -0.0068503683, -0.021136124, 0.010613353, -0.036927827, 0.020456748, -0.02131729, 0.00441594, -0.028926294, 0.0010700163, 0.030843643, 0.0027439217, -0.015323689, 0.010439735, -0.001128518, 0.014674508, 0.008907367, 0.0034440558, -9.3708986e-05, 0.016999481, -0.0190829, 0.0042498703, 0.0051330584, 0.009330088, 0.011315375, -0.041879717, -0.01351957, -0.008771491, 0.049247168, -0.033787604, -0.0006383298, 0.013859258, -0.0027816647, -0.016259717, 0.0057218503, 0.032308076, -0.018388428, 0.021226706, 0.012175917, -0.0057633677, -0.005691656, 0.01521046, -0.016622052, -0.022449583, 0.004083801, 0.0021645655, -0.025287861, -0.023053471, 0.01814687, -0.025770973, -0.026238987, -0.033334687, 0.008930012, 0.021573942, -0.00937538, 0.005004732, -0.0042385478, -0.0081751505, -0.036716465, -0.05903017, 0.025257668, -0.024834944, 0.0054878434, 0.024246152, -0.014198946, 0.026510738, -0.020260483, -0.0037629849, 0.01177584, 0.0030590764, 0.033666823, -0.021619234, -0.014168751, 0.0071409903, 0.015059488, -0.034814216, 0.013104397, 0.037410937, 0.004514072, -0.012817549, 0.00733348, -0.006405, -0.010620902, 0.003974346, -0.016833412, 0.015104779, -0.00066050387, 0.0018031755, -0.032730795, -0.0014512213, -0.028397892, -0.005521812, 0.0032704377, -0.016546564, -0.00796379, -0.0047027874, -0.03831677, 0.0899795, -0.0034138614, 0.0075637125, 0.009783005, 0.00030737018, 0.0024570744, -0.016184231, 0.065521985, -0.052659143, 0.026299376, -0.017754344, -0.029394308, -0.006525778, 0.013836612, 0.024382029, 0.013036459, 0.023415806, -0.014146105, -0.03282138, 0.021136124, 0.015089682, 0.025438834, 0.0037327902, -0.019097997, 0.026812682, -0.07047387, -0.007533518, 0.025499223, -0.027280696, 0.014432953, -0.057550643, 0.008424255, 0.017286329, -0.007023987, -0.016923996, -0.000671355, 0.008499741, -0.0179959, 0.008182699, -0.016923996, -0.015776606, 0.016199328, -0.043902747, 0.043963138, 0.0028250692, 0.038195994, 0.0053217737, -0.07005115, 0.0020079317, 0.0010690726, 0.032851573, -0.019490525, -0.014168751, 0.0043970686, 0.00078269705, 0.041517384, 0.0049103745, -0.040158633, -0.048703667, 0.023898916, -0.011088916, 0.024563195, -0.018509205, 0.013481827, -0.007835463, 0.041789137, 0.025937043, -0.006491809, -0.02992271, 0.034059353, -0.028971586, 0.013232723, 0.043540414, -0.025967237, -0.00899795, 0.009964173, 0.014380112, 0.015006647, 0.014523536, 0.008348769, 0.011768292, -0.035569075, -0.007812817, 0.025438834, -0.00087752653, 0.01924897, -0.014644314, 0.013836612, -0.029847225, 0.014478245, -0.011262534, 0.011896618, 0.02764303, 0.01881115, -0.02768832, 0.02545393, -0.031975936, -0.033123326, -0.00087233685, 0.01592758, 0.0104095405, -0.013964939, -0.008250637, -6.899906e-05, 0.0392528, 0.014704702, 0.011202145, 0.017950607, 0.016154038, 0.007835463, -0.03445188, -0.0052462877, -0.016954191, -0.0020400134, 0.010802069, 0.03336488, -0.010477479, -0.0039328285, 0.0075788097, 0.00016087986, 0.025393542, 0.020622818, 0.04725433, -0.053836726, 0.02087947, -0.0071183443, 0.016818315, 0.009896235, 0.0022268416, 0.009941527, -0.011330471, -0.030073684, -0.0033459237, 0.01128518, -0.020532234, -0.02883571, -0.032730795, -0.012357083, -0.028760225, -0.012560896, -0.022962889, -0.032398656, 0.005223642, -0.015134974, 0.00268542, -0.10966629, -0.007692039, 0.033002548, -0.028246919, -0.018418621, -0.01958111, 0.036384325, 0.010115145, 0.0034648145, 0.018056288, 0.01057561, -0.025438834, 0.0010256681, 0.0031213525, 0.038407356, -0.061687287, 0.0048235655, -0.0090130465, 0.04655986, -0.016788121, -0.011579576, 0.02246468, 0.010439735, -0.0043480024, -0.019067803, -0.02480475, -0.020305775, -0.02436693, 0.024819847, -0.0077599767, -0.016395593, 0.018675275, 0.008741297, 0.0103038605, -0.006578618, 0.02131729, -0.01270432, -0.013889452, 0.018931927, -0.010983235, 0.014440501, -0.01025102, 0.019369747, -0.011353117, 0.039192412, 0.005895469, 0.017074969, 0.0005727512, -0.011700354, 0.015640732, -0.014810383, 0.024336737, -0.017754344, 0.014682057, -0.004487652, -0.027794002, -0.0020456747, -0.039554745, -0.0037554363, 0.015195363, 0.0011700355, 0.011911715, -0.006412549, -0.026072918, 0.0059785033, -0.0013946067, 0.009307443, 0.022011762, -0.023430903, -0.006635233, -0.037441134, 0.024880236, -0.017799634, -0.022721332, 0.013172334, 0.017844927, 0.02638996, 0.033515852, 0.0031213525, 0.016471079, 0.05806395, -0.010500124, -0.022162735, 0.012508056, 0.00937538, 0.027552446, 0.010386894, -0.013647897, 0.012772257, 0.05718831, 0.03411974, 0.003104368, -0.010092499, -0.012870389, -0.03931319, 0.011141757, 0.019113094, 0.032368463, 0.026103113, -0.015716217, 0.022011762, 0.015897384, -0.019520719, -0.012591091, -0.040520966, 9.718842e-05, 0.0018663951, -0.014833029, -0.0088847205, -0.0023362965, 0.010749228, 0.04040019, 0.017603371, 0.019188581, -0.009986818, 0.024850043, 0.04372158, -0.006854143, -0.008590325, 0.0033157293, 0.009503707, 0.011171951, -0.01793551, 0.019128192, -0.022751527, 0.018931927, 0.0025854008, 0.020215193, 0.008922463, 0.026193695, -0.016108746, 0.018886635, -0.013806418, 0.0049820864, -0.00043664023, -0.0061974134, -0.025831362, -0.02196647, -0.004423489, -0.0011634304, -0.019686788, 0.003642207, 0.00056614616, -0.03547849, -0.031009711, 2.6066313e-05, -0.014032876, -0.04840172, 0.014493342, 0.021000247, 0.035448298, 0.0122363055, -0.013089299, 0.025891751, 0.006144573, 0.010515221, 0.024019694, -0.035750244, -0.040551163, 0.026903266, -0.0010756777, 0.049518917, 0.01958111, -0.025333153, 0.08031727, -0.02643525, -0.02562, -0.019807568, -0.003068512, 0.022645846, -0.02693346, 0.010417089, 0.03185516, 0.018796053, 0.04806958, 0.028262015, -0.020924762, 0.038226187, 0.0063974517, 0.03396877, 0.022645846, 0.010069853, -0.0010785084, 0.0069673723, 0.0022740203, -0.033062935, -0.02611821, 0.016773023, -0.032398656, 0.0022343902, 0.026254084, 0.020184997, 2.1156764e-05, 0.05338381, -0.0010596369, -0.012387278, -0.00010137555, 0.0022910049, -0.022887401, 0.004257419, -0.0376223, 0.03297235, 0.008643164, -0.043147884, -0.021332387, 0.059301924, -0.00047226026, -0.0042498703, -0.010930395, -0.020683207, -0.032911964, 0.01973208, 0.00246651, 0.022117443, -0.025937043, 0.020622818, -0.016773023, -0.022751527, -0.03593141, -0.01231934, -0.034089547, -0.028699836, -0.0127949035, 0.0039026341, 0.024321638, 0.017512787, 0.0033515852, -0.030828545, -6.162737e-05, -0.0084544495, -0.010719034, -0.031553213, -0.010447284, 0.0029835904, -0.006046441, 0.029620767, 0.008280831, -0.007752428, 0.026012529, 0.0013861145, 0.06123437, 0.0033100678, 0.002294779, -0.01576151, 0.011194597, -0.022781722, 0.00711457, 0.024759458, 0.017150454, -0.008756394, 0.00034629274, 0.040853105, 0.005261385, -0.018131774, 0.03445188, 0.015535051, 0.037169382, 0.045352083, 0.02196647, 0.024744362, -0.0044348114, 0.04858289, -0.010137791, -0.020819081, 0.0068013025, 0.0007558051, -0.038226187, -0.024955723, 0.018690372, 0.0066767503, -0.022072151, 0.026631515, -0.02327993, 0.016033258, -0.006155896, -0.0064200973, 0.015165169, -0.015255752, 0.010719034, 0.0020758694, 0.0026533382, 0.013708286, -0.009262151, -0.013753577, 0.0073599, -0.007344803, -0.022585457, -0.029379211, -0.0017097614, -0.0008110044, 0.01160977, -0.0067031705, 0.0056727845, -0.009352734, -0.0045706867, 0.00036657962, -0.007612779, -0.010205728, -0.0027061787, -0.016018162, 0.01243257, -0.009413123, -0.011277632, -0.049035806, -0.011858875, -0.0047971453, 0.033274297, -0.024925528, -0.014599022, 0.026012529, -0.026208794, 0.0060049235, -0.006854143, 0.011473895, -0.0028741353, 0.009428221, 0.0035214291, -0.018660177, -0.008930012, 0.0024853817, 0.004623527, 0.046378694, 0.016561663, -0.005752045, -0.0143423695, -0.028639447, -0.016773023, 0.011760742, 0.00067607284, -0.012372181, -0.0013974374, -0.0035214291, 0.024880236, 0.022374095, -0.052477974, -0.00050670083, -0.046861805, -0.017603371, 0.033062935, -0.02294779, -0.005880371, 0.013655446, -0.01052277, -0.009126276, 0.009858492, 0.0042951624, 0.0028533766, 0.0020626592, -0.011647514, 0.01684851, 0.07790171, -0.057309087, -0.030300142, 0.00027882698, -0.027794002, 0.011262534, -0.020411456, 0.025227472, -0.029575475, 0.005197222, -0.005706753, 0.030013295, -0.017241038, 0.032308076, 0.034089547, 0.015580342, -0.013866806, 0.0013285563, 0.022993082, 0.003838471, 0.0054161316, 0.015670925, -0.011926812, -0.017890219, 0.02480475, 0.035236936, 0.055467226, -0.016727732, 0.024925528, 0.03478402, -0.015776606, -0.032760993, 0.02115122, 0.006756011, -0.033334687, 0.019067803, -0.018086482, 0.0017314637, -0.027492058, -0.0010181195, -0.0196264, 0.004321582, -0.015746411, 0.00011712149, 0.0070466325, 0.016305009, -0.006231382, 0.01155693, 0.008107213, -0.002674097, -0.009971721, -0.017467497, -0.03505577, -0.008628068, 0.0013162898, -0.031070102, 0.006722042, 0.017965704, 0.009126276, 0.016561663, -0.011889069, 0.03892066, -0.030179365, 0.008484644, 0.0343613, -0.034723632, -0.022962889, 0.016229523, -0.004076252, 0.026269183, -0.032791186, 0.0023136507, 0.005042475, -0.0023344094, 0.040309608, 0.007518421, -0.035901215, -0.009873589, 0.019264067, 0.018494109, -0.0030949323, -0.018448817, -0.009609387, 0.0062955455, -0.013421439, 0.0036554171, 0.027114626, -0.0021683397, -0.027054237, 0.023370514, -0.004072478, 0.012900583, 0.00692208, 0.025091598, 0.011368215, 0.0038195995, -0.00083506556, -0.013768675, 0.0059785033, -0.0053519686, 0.0013898888, -0.0147198, -0.0048386627, -0.032700602, -0.0084770955, 0.006891886, -0.028881002, -0.024321638, 0.003981895, -0.0013766787, 0.019294262, 0.0049066, -0.02115122, 0.0078052683, -0.0060200207, 0.0037592105, 0.020562429, 0.0719836, -0.021740012, -0.038346965, 0.021634331, -0.013693188, -0.01847901, -0.017588275, -0.0003854512, -0.00042390195, 0.027582642, 0.0010152888, -0.030632282, -0.00779772, -0.031794768, 0.01504439, 0.0120853335, -0.027778905, 0.006589941, 0.008318574, 0.0072051533, 0.00134271, -0.07150049, 0.0020702078, 0.013708286, 0.03864891, 0.0030232205, -0.0045216205, -0.026752293, -0.034753826, 0.026148403, 0.0038686653, -0.021619234, -0.025710585, -0.0075863586, -0.0071711848, 0.017090065, -0.006940952, -0.003678063, -0.039373577, 0.016591856, -0.013662994, -0.0019296148, -0.019822665, -0.00417061, 0.014931161, -0.03384799, 0.0025948365, 0.0041517387, 0.02311386, -0.012915681, 0.029801933, 0.03547849, 0.019777372, 0.0035006704, 0.0047254334, -0.00013988528, 0.021468263, 0.020577526, 0.0003026523, 0.0076731676, 0.00366674, -0.012900583, 0.029968003, -0.009420672, 0.010786971, -0.017361816, -0.035961606, -0.008484644, -0.0073900945, 0.0019758502, -0.006744688, -0.024910431, -0.027416572, -0.015142523, -0.05773181, -0.0023721524, 0.010568062, 0.018433718, 0.040098246, -0.004068704, 0.023944208, 0.0040536067, -0.008258185, 0.0001574594, -0.024321638, -0.014742446, -0.027280696, -0.050847474, 0.0288961, 0.00023565834, -0.0063106427, 0.0069485004, 0.008311026, -0.018222358, 0.0045216205, -0.042694967, 0.022343902, -0.014493342, 0.0061068297, 0.008348769, -0.00899795, -0.0020626592, -0.01674283, -0.015806802, -0.010862458, 0.025499223, 0.0021853242, 0.0019399942, 0.012726965, -0.003211936, -0.026525835, 0.0381658, -0.0006425759, 0.0074504833, -0.0067786565, 0.011405958, 0.03069267, -0.007356126, -0.018705469, -0.03297235, 0.04215147, 0.031643797, 0.0062728995, -0.026344668, 0.03735055, -0.009941527, 0.009533902, -0.008952658, 0.0071447645, 0.013678092, -0.009956623, -0.032610018, 0.02321954, -0.004782048, -0.00048028067, -0.037078798, 0.007522195, -0.03254963, -0.015610537, -0.029817032, 0.003102481, 0.0019286713, 8.49809e-05, -0.044355664, -0.02066811, 0.003238356, 0.022600554, 0.012877938, 0.03985669, 0.027718516, -0.03511616, -0.033274297, -0.014651863, -0.03042092, 0.012568445, -0.00013858786, 0.020970054, 0.031311657, 0.01755808, 0.011345569, -0.0020796435, -0.005812434, -0.018297844, 0.010500124, -0.020018928, -0.019097997, 0.00011429076, 0.01897722, 0.029756643, -0.00055482326, -0.002649564, 0.0070353094, -0.012598639, 0.0030515278, 0.03550869, 0.012455216, -0.0101302415, -0.0151576195, 0.005702979, -0.0021740012, 0.0075486153, -0.028714933, 0.024593389, -0.039222606, -0.015625633, 0.026510738, -0.0024533, 0.020351067, 0.012825098, 0.013791321, 0.03233827, -0.008741297, 0.0016993821, 0.024004597, 0.0019966087, -0.008960207, 0.014138557, 0.019128192, -0.00877904, -0.007752428, -0.0026816456, -0.017890219, 0.022328805, 0.034482077, 0.025393542, 0.015361432, -0.0062728995, 0.027522251, 0.021936277, -0.027024044, 0.0116777085, 0.02420086, -0.013059105, 0.016471079, -0.021785304, 0.05320264, -0.00438952, -0.0026231438, -0.023189347, 0.0035742694, 0.030028392, 0.022555264, 0.005702979, -0.007005115, -0.024382029, -0.0078052683, -0.027054237, -0.020622818, 0.05175331, 0.004076252, -0.041215442, 0.000604361, 0.0056954306, 0.02900178, 0.037924245, -0.010703937, 0.00806947, 0.00970752, 0.018418621, 0.005276482, 0.021875888, 0.009511256, -0.031885352, -0.0403398, -0.0046197525, -0.011368215, -0.022509972, -0.00064965273, 0.059966203, -0.026450349, -0.03729016, -0.03085874, -0.026133306, 0.0047933706, 0.031492822, 0.013444084, 0.0073938686, -0.017724149, 0.003691273, -0.018252552, -0.022751527, 0.017980803, -0.015014196, 0.052659143, -0.04393294, 0.01810158, -0.017844927, 0.015391627, -0.0038950855, 0.002232503, 0.0056123957, -0.006148347], "content_hash": "66167bb880c23493d664ad623f6f1c5a"}, {"content": "is expected behavior for small models. The dashboard is designed for exploring *how* the model works, not for getting useful outputs. Try different prompts or a different model. ### BertViz visualization is hard to read **Why**: With 12+ heads selected simultaneously, the attention lines overlap and become a dense mess.\n\n**Fix**: Double-click on a single head in the BertViz visualization to isolate it. Then explore heads one at a time. Use the head categories to guide which heads to investigate.\n\n### The dashboard becomes slow or unresponsive\n\n**Why**: Larger models require more memory and computation. Running multiple experiments without refreshing can also accumulate memory usage.\n\n**Fix**: Try a smaller model. Refresh the browser page if things get sluggish. Close other memory-intensive applications.\n\n## Frequently Asked Questions\n\n### Which model should I start with?\n\n**GPT-2 (124M)** is the best starting model. It's small, fast, well-studied, and has clean attention patterns that are easy to understand. Move to Qwen2.5-0.5B once you're comfortable for a comparison.\n\n### What prompts work best for learning?\n\nStart with short, simple prompts (5-10 words) that have clear, predictable continuations:\n- \"The cat sat on the\" (predict a location)\n- \"The capital of France is\" (predict a fact)\n- \"Once upon a time there was a\" (predict a story element)\n\nThese give clear, interpretable results in the pipeline and experiments.\n\n### Can I use my own model?\n\nYes! Type any HuggingFace model ID into the model dropdown. The dashboard supports GPT-2, LLaMA, OPT, GPT-NeoX, BLOOM, Falcon, and MPT architectures. Unknown architectures may need manual configuration in the sidebar.\n\n### What's the difference between the pipeline and the investigation panel?", "source_file": "troubleshooting_and_faq.md", "chunk_index": 1, "embedding": [-5.0072726e-05, -0.012094242, 0.055194624, -0.00953762, -0.037355088, 0.0028744242, -0.035338197, 0.025466794, 0.008827447, 0.060336277, -0.028406909, -0.034571208, -0.015467562, -0.048234932, -0.018180422, -0.0024607484, 0.0022334931, 0.008145682, -0.0320714, 0.055847984, -0.0103969285, -0.012797313, -0.003634309, 0.032980423, 0.021986948, 0.022057965, -0.005368906, 0.014395202, 0.024429942, 0.008621497, 0.030395392, -0.034116697, -0.06312015, -0.030026102, -0.03684376, 0.015084069, -0.014572744, 0.02968522, -0.009835892, 0.067892514, -0.030281765, -0.017740116, -0.029372744, 0.05036545, -0.019501343, 0.040280998, 0.013890979, -0.024870249, -0.05772284, 0.037440307, -0.0016555901, 0.036786947, -0.04905873, 0.005830518, 0.0104679465, -0.027895585, -0.03650288, -0.027923992, -0.01102188, 0.0022086373, 0.019714395, 0.00419357, 0.0037852207, 0.02968522, 0.014167946, -0.025083302, -0.037525527, 0.0077266796, -0.013841267, -0.03724146, -0.020737045, -0.0021731285, 0.0100489445, -0.01200192, 0.0012268234, -0.0026223129, 0.0024394433, 0.03638925, -0.0066862763, -0.045905564, -0.027980806, -0.020623416, 0.018606525, -0.03880384, -0.012364107, -0.014743186, -0.027085988, -0.019004222, -0.021546641, -0.05655816, -0.0006267274, 0.009367178, -0.022356238, 0.025836084, 0.034968905, -0.009381382, 0.025637235, -0.053745873, -0.0028176103, 0.04496814, 0.03704261, -0.0213762, -0.0061500957, 0.0032224087, 0.03408829, -0.03300883, 0.022327831, 0.018492898, -0.028435316, -0.0062388675, -0.079255275, -0.021021113, -0.0109082535, 0.0197286, -0.0129748555, 0.0042219767, -0.048859883, -0.010752015, 0.02762572, -0.064938195, 0.008834549, 0.017697504, 0.0074426103, 0.009161228, 0.0057808063, -0.0017408109, 0.011313052, -0.028364299, 0.016802687, -0.001802951, -0.008294818, 0.037923224, 0.025196929, -0.013890979, -3.7422775e-05, -0.015694818, -0.033576965, -0.0050741844, -0.03346334, -0.0023045104, 0.00928906, -0.017498657, 0.0066294624, 0.019600768, -0.067722075, 0.015978886, -0.012136852, 0.02602073, -0.060790785, 0.038945872, 0.029145489, 0.030991938, -0.019558158, -0.051018808, -0.0105815735, -0.010943762, 0.014025912, 0.035252973, -0.019799616, -0.068915166, 0.032383878, 0.018634932, -0.020950096, -0.034855276, -0.0427524, 0.07738042, -0.034343954, 0.0072295587, -0.023861803, -0.033804223, 0.013315739, 0.01874856, 0.042042226, -0.049967755, 0.03363378, 0.0041509597, -0.057467178, -0.047865644, 0.006015163, -0.04218426, -0.038917467, -0.0030714972, -0.012122649, -0.011810172, -0.04505336, -0.007471017, -0.041104797, 0.021063723, 0.0055322456, -0.005645873, -0.001422121, 0.030310173, 0.05965451, 0.023322072, -0.0061110365, -0.026773512, 0.06721075, 0.018535508, 0.01842188, 0.015879462, 0.0024927063, 0.009935317, 0.05891593, -0.0814142, 0.011341458, 0.023165835, -0.0002512236, 0.01056737, -0.041502494, -0.031219194, 0.0055713053, -0.005368906, 0.027100192, 0.019146256, -0.06118848, -0.008997888, -0.052268714, 0.0017949616, 0.029884068, 0.04954165, 0.020623416, 0.036900576, 0.041502494, -0.0024714011, -0.025907101, 0.035906333, 0.040650286, -0.037724376, -0.028520538, 0.013088483, 0.027284836, -0.011007678, -0.024387332, 0.026205374, 0.019998465, -0.023705566, 0.01056737, -0.018379271, -0.0043036467, -0.050905183, 0.031815737, 0.01994165, -0.025154319, -0.036076777, 0.08010749, 0.0059405947, -0.03548023, -0.076357774, 0.024685604, 0.01718618, 0.020864874, -0.0062459693, -0.034968905, -0.0052268715, -0.0023329174, -0.016603839, -0.03573589, 0.0044953935, -0.039684452, -0.01820883, -0.009502111, 0.022299424, -0.035792705, 0.02310902, -0.019643378, -0.027157005, -0.0493428, -0.04337735, -0.0764714, -0.036985796, 0.01887639, -0.011590019, -0.0043604607, 0.022455662, -0.03795163, -0.00965835, 0.02273973, 0.007584645, -0.027725143, -0.021575049, -0.02113474, -0.022171592, 0.033946257, 0.013940691, -0.009502111, 0.017129367, 0.008394241, -0.01870595, 0.0069135316, 0.006675624, 0.00916833, -0.010659693, 0.008742226, -0.017413436, 0.0053547025, 0.020041075, 0.011014779, 0.0143241845, 0.0013946017, 0.0029365642, 0.012847025, -0.015836852, -0.005759501, -0.0037497121, 0.040934358, 0.020779654, -0.00312476, 0.016419193, -0.0013892754, 0.008848752, -0.038207293, -0.023378886, -0.010297504, 0.019515546, 0.006902879, -0.016390787, -0.00785451, 0.035395008, 0.033037234, -0.017626487, -0.011696545, -0.048206527, -0.0349405, 0.026787715, 0.009502111, 0.015694818, 0.0043036467, 0.014033013, -0.026418425, -0.02458618, 0.021717083, 0.0062140115, -0.009203839, 0.04008215, -0.011618426, -0.051075622, -0.003515355, 0.019970058, -0.0074639153, 0.06874472, -0.083345875, -0.004424376, -0.029514778, 0.0024323417, -0.04181497, 0.043718234, 0.0024696258, 0.027852975, 0.04329213, -0.017058348, -0.008337428, 0.005106142, -0.027355853, -0.028378503, 0.03775278, 0.05340499, -0.025154319, -0.016064107, -0.029287525, -0.011085796, -0.020566603, -0.0035260075, 0.014700576, 0.013564299, -0.0058979844, 0.011519002, -0.0146721685, -0.011242035, 0.00937428, -0.006128791, 0.015694818, 0.017058348, 0.0077976966, -0.015467562, -0.04925758, -0.02688714, -0.025665643, 0.028421113, -0.012172361, 0.05772284, 0.033150863, 0.010254894, 0.012641075, -0.016191939, 0.0075704413, 0.0148, 0.033292897, -0.014913628, -0.052183494, 0.0044705374, 0.018862188, -0.014452015, -0.011043186, 0.0044385795, 0.012946449, 0.079084836, -0.0037923225, 0.0097932825, 0.035792705, 0.04155931, 0.015907869, -0.015595393, -0.060506716, -0.0022832053, 0.0050812857, 0.028889827, 0.037610747, 0.029074471, 0.009850096, -0.041928597, 0.03712783, -0.05331977, 0.013436468, -0.002150048, -0.0296, -0.0041509597, 0.03724146, 0.019330902, 0.033861037, 0.05542188, -0.0064803264, -0.06800614, -0.0522119, -0.004658733, -0.08760691, 0.02150403, 0.04263877, -0.025466794, 0.0076769674, -0.033974662, -0.009693858, 0.02931593, 0.057523992, 0.0025140115, 0.02256929, -0.06527908, 0.03704261, 0.013173704, 0.020779654, -0.00042632557, 0.041388866, -0.019884836, 0.035224568, 0.017725911, -0.022299424, -0.031219194, -0.046587333, -0.038349327, -0.008309021, -0.05760921, -0.0010146593, 0.012747601, -0.0073644915, 0.028350096, -0.0018171545, -0.043434165, 0.08152783, -0.017228791, 0.07993704, 0.006593954, 0.014558541, 0.058234163, -0.010503455, 0.041019578, -0.012726296, 0.04374664, 0.0014407629, -0.0004365343, -0.013841267, -0.0374119, 0.0029259117, -0.030821497, 0.029259117, -0.031190787, -0.05377428, -0.048945107, -0.017228791, 0.02400384, -0.016248751, -0.048206527, 0.0038988483, 0.028051823, 0.04181497, 0.045536276, 0.03732668, 0.07328983, -0.021035317, -0.029344337, 0.012399616, 0.023932822, -0.03851977, -0.020495586, 0.022682916, -0.016561229, 0.027654126, -0.011817275, -0.016944721, -0.04144568, -0.022256814, -0.0037461612, -0.0038455853, 0.0069206334, -0.012520345, 0.052808445, 0.0031300862, 0.0008965931, 0.0050422265, 0.011696545, -0.013422265, 0.038747024, -0.03474165, -0.028364299, -0.0097932825, 0.03491209, -0.014601151, 0.03195777, 0.015652208, 0.04707025, 0.029912475, 0.022824952, -0.0068638194, 0.016476007, -0.020765452, -0.01245643, 0.0053866603, -0.018265642, -0.030452207, -0.055393472, -0.0151124755, -0.008784837, 0.007918426, -0.044456813, 0.023009596, -0.026290596, -0.060961228, -0.0148, -0.019870633, -0.010766218, -0.03573589, 0.0010022313, 0.023719769, 0.02104952, 0.006377351, 0.029713627, -0.021518234, 0.022327831, 0.019799616, -0.029997697, -0.0027945298, 0.034258734, 0.02717121, -0.030679462, 0.027810365, -0.011298848, 0.025381574, 0.0057985606, -0.008983685, -0.016348176, 0.018862188, -0.024245298, 0.0065761996, 0.016518617, 0.019416122, -0.021276776, 0.0020364204, 0.001590787, 0.0146721685, -0.044002302, -0.008024952, 0.0004953455, -0.016902111, -0.012477735, 0.00066134834, -0.015652208, 0.023733974, -0.01998426, -0.025935508, 0.03269635, -0.01714357, 0.00081936177, -0.031389635, -0.02310902, -0.017910557, 0.02289597, -0.018038388, 0.025665643, -0.0024767274, -0.0033608924, 0.0018695297, 0.00021194217, 0.028321689, -0.03548023, 0.0118598845, 0.010340115, -0.0304238, -0.01887639, 0.028960845, -0.03235547, -0.052638, -0.02565144, -0.0026453934, -0.017200384, -0.009381382, 0.0009072457, -0.029287525, -0.011710748, -0.03079309, 0.009154127, 0.006689827, -0.03428714, -0.0046019196, -0.014544338, -0.017086755, -0.029940883, -0.06556315, 0.015893666, -0.03502572, -0.00063738006, 0.022398848, -0.010666794, -0.001171785, -0.01862073, 0.011909597, -0.0052410746, -0.018350864, 0.008784837, -0.0028176103, -0.015936276, 0.008841651, 0.028222265, -0.024415739, 0.0002412368, 0.048320152, -0.0036609406, -0.0071265833, 0.01994165, 0.009395585, -0.0077550863, 0.0009702735, -0.005173608, 0.012641075, -0.013202111, -0.016390787, -0.04229789, 0.013734741, -0.01994165, 0.008429751, 0.02146142, -0.021262571, -0.0033786467, -0.04420115, -0.027412668, 0.06192706, 0.012009021, 0.014984645, 0.0068176584, 0.007932629, -0.015268714, -0.028463723, 0.057552397, -0.039371975, 0.03851977, -0.01846449, -0.040337812, 0.0055642035, 0.021972744, 0.0035260075, 0.018009981, 0.044684067, -0.012506142, -0.027611516, -0.013166603, 0.033292897, 0.025750864, -0.011866987, -0.029117081, 0.015240307, -0.041616123, -0.0066259117, 0.034713242, -0.027867178, 0.029628407, -0.052183494, 0.017314011, 0.028179655, 0.010510556, -0.020765452, 0.00022725528, 0.021248369, 0.015069866, 0.008294818, -0.026319, -0.014452015, 0.0026951055, -0.045905564, -0.00022015355, 0.0012676583, 0.04715547, 0.0047581573, -0.06652898, -0.008600192, -0.011206525, 0.015467562, -0.015353935, 0.002709309, -0.014629559, -0.010283302, 0.03195777, 0.004498944, -0.021688676, -0.031503264, 0.016717466, -0.007240211, -0.022214204, -0.02569405, -0.010404031, -0.01553858, 0.027739348, 0.007968138, -0.006924184, -0.044456813, 0.03309405, -0.030537428, 0.017853742, 0.034713242, -0.022725528, 0.0005206454, -0.0009578455, -0.020282533, 0.02890403, 0.009523416, 0.030537428, 0.011043186, -0.043064874, 0.0103046065, -0.0012143954, 0.0004922385, 0.017938964, -0.01558119, 0.000193744, -0.020893281, 0.006821209, -0.024756622, 0.021361995, 0.02651785, -0.01155451, -0.028932437, 0.0365881, -0.022881765, -0.04596238, 0.0021873321, 0.013926487, -0.02204376, -0.010609981, -0.003108781, -0.02504069, 0.04226948, -0.012009021, 0.020651823, 0.032383878, 0.024486756, 0.016262956, -0.028776199, 0.015552783, -0.02310902, -0.0014274472, 0.034457583, 0.026588866, 0.010666794, -0.016476007, 0.0042610364, -0.038292512, 0.012733397, 0.008507869, 0.05496737, -0.057495587, 0.033520155, 0.0110005755, 0.004062188, 0.018379271, -0.010190979, 0.0119238, 0.030310173, -0.03567908, 0.009296161, -0.006437716, -0.01935931, -0.028833013, -0.032525912, -0.012761804, -0.060961228, -0.0062459693, -0.005312092, -0.0138341645, 0.0052481764, -0.029514778, 0.007272169, -0.12658119, -0.012080038, -0.0009143474, -0.02113474, -0.00032357246, -0.01710096, 0.042468328, -0.0082593085, 0.0040550865, -0.004218426, 0.0133015355, -0.0213762, 0.025069097, 0.008230902, 0.019813819, -0.07158541, 0.017299809, -0.008621497, 0.051246066, -0.01994165, -0.027483685, 0.023947025, 0.00875643, 0.021532437, -0.038832244, -0.023322072, -0.04155931, 0.017385028, 0.049740497, -0.0076414584, -0.03519616, 0.04871785, 0.0076982724, 0.0092819575, 0.004917946, 0.03420192, -0.00019008217, -0.015055662, 0.014728983, -0.026304798, -0.0374119, -0.025807677, 0.014466219, -0.03712783, 0.036758542, 0.0042077736, -0.0061856047, 0.0064625717, -0.042525142, 0.018720154, 0.0035988004, 0.024302112, -0.021617658, 0.052552782, 0.020893281, -0.024685604, 0.017796928, -0.016419193, 0.012790211, 0.010617082, 0.0016085412, 0.033037234, 0.0049037426, -0.017299809, -0.004037332, 0.0025140115, 0.031815737, 0.006668522, -0.036730133, -0.017385028, -0.05073474, 0.035451822, -0.0050493283, -0.0009933541, 0.000117178504, 0.0055073895, 0.02293858, 0.030480614, 0.0028264874, 0.0044208253, 0.020452974, -0.009473705, -0.042695586, 0.018350864, 0.009693858, 0.042042226, 0.017754318, -0.025665643, 0.03326449, 0.024657197, 0.010482149, 0.005649424, -0.012080038, -0.009509213, -0.024202688, 0.00994952, -0.019600768, 0.026546257, 0.030480614, -0.0069525912, 0.033690594, 0.009132821, -0.024614587, -0.012357006, -0.011661036, 0.0010235364, 0.0064554703, -0.009409789, -0.016831094, -0.0034851728, 0.020140499, 0.03786641, 0.009388483, 0.036815356, -0.021575049, 0.025211133, 0.052723225, 0.015368138, 0.022924377, -0.011149712, 0.023052206, -0.02315163, -0.020296738, 0.016135124, -0.016532822, -0.0048327255, 0.013038771, 0.029429559, -0.012271785, 0.018223032, -0.0047190976, 0.029259117, -0.021603454, 0.007598848, -0.0028602206, -0.0017789827, -0.004665835, -0.01516929, 0.0039166026, -0.0014203455, -0.03107716, 0.022498272, 0.010411132, -0.01681689, -0.01282572, 0.004033781, -0.018492898, -0.051984645, 0.023421496, 0.018975815, 0.042070635, 0.00037261876, -0.012392514, 0.020552399, 0.0038988483, 0.0010182102, 0.0017177303, -0.053632244, -0.041218426, -0.002734165, 0.027938195, 0.03326449, 0.016532822, -0.026915547, 0.03133282, -0.03684376, -0.03141804, 0.018265642, -0.00661881, 0.035338197, -0.028676774, -0.0003890415, 0.023691362, 0.0017585652, 0.03999693, 0.046274856, -0.00485048, 0.011270441, 0.011803071, 0.011313052, -0.017541267, 0.03641766, 0.01155451, 0.0051167947, 0.015481765, -0.013422265, -0.024188483, 0.0069206334, -0.014494626, -0.009516315, 0.015893666, 0.0013670825, -0.00674309, 0.042468328, 0.014380998, 0.0031052304, 0.00040080375, -0.01229309, -0.021717083, 0.020538196, -0.01952975, 0.03215662, -0.009012092, -0.017938964, -0.01957236, 0.04190019, 0.0027199616, -0.01846449, -0.021518234, -0.015510173, -0.035451822, 0.017569674, -0.010077351, 0.022313626, -0.0028424663, 0.013010365, 0.02881881, 0.0011043187, -0.034968905, 0.004353359, -0.029401151, -0.034599617, -0.022128982, 0.014295777, -0.016532822, 0.019757006, 0.012172361, -0.0010892274, -0.0047013434, 0.013585605, -0.008159885, -0.02330787, 0.009906909, -0.0101412665, -0.016149327, 0.01163263, 0.012151056, -0.014515931, 0.017228791, 0.014167946, 0.045848753, 0.0156238, -0.005447025, -0.02315163, 0.000531298, -0.012776008, 0.012101344, -0.009651247, 0.01146929, -0.027781958, -0.009303262, 0.03621881, 0.017129367, -0.02273973, 0.024358924, -0.017271401, 0.026958156, 0.029742034, 0.017015738, -0.011320153, -0.008948176, 0.048178118, -0.012108445, -0.0226119, -0.0038739922, 0.0028637715, -0.009871401, 0.0013537668, 0.03474165, 0.004410173, -0.017001536, 0.028364299, -0.028591555, 0.010020537, -0.011703647, 0.018350864, 0.0062885797, -0.0068070055, 0.011497697, -0.00928906, 0.01413954, -0.0036609406, 0.009097313, -0.013372553, -0.003884645, -0.021163147, -0.040877543, -0.012683685, 0.010091554, -0.020396162, 0.0070449137, 0.017214587, -0.012562956, -0.010617082, 0.005209117, 0.003165595, -0.011795969, -0.01994165, 0.021702878, -0.05093359, 0.022654511, 0.0023453454, -0.014508829, -0.054370824, 0.013024568, -0.013422265, 0.033037234, -0.029628407, 0.010723609, 0.03005451, -0.028350096, 0.010574472, 0.00661881, 0.023563532, 0.009360077, 0.021930134, 0.016859502, -0.015311324, -0.016788483, -0.011277543, -0.010425336, 0.022185797, -0.0017541266, -0.013017466, -0.015226103, -0.023620345, 0.002293858, 0.020424567, 0.009928215, -0.013060076, 0.008046257, 0.0011087572, -0.0006484765, 0.019757006, -0.054200385, -0.028918235, -0.060733974, -0.020765452, 0.033236083, -0.008380039, 0.0134648755, 0.020168906, 0.00076565496, -0.01052476, 0.026702495, 0.0016112044, 0.018819578, 0.0044208253, 0.014871017, 0.021915931, 0.056387715, -0.029074471, -0.029372744, 0.0002787428, 0.004836276, 0.029628407, 0.001809165, 0.026546257, -0.006668522, -0.00604357, 0.0014167946, 0.030196546, -0.01389808, 0.030565834, 0.036161996, -0.006459021, -0.0041119, -0.026943954, 0.029202303, 0.0012099568, 0.016802687, 0.011206525, -0.0094452975, 0.0006924184, 0.023052206, 0.038576584, 0.030849904, -0.021717083, 0.017356621, 0.048746258, -0.009381382, -0.036360845, 0.021077927, 0.014743186, -0.022597697, 0.006899328, -0.01171785, 0.031730518, -0.029117081, 0.008252207, -0.0028477926, -0.029543186, 0.01105739, -0.022327831, 0.017072553, 0.014047217, 0.020367755, 0.02495547, 0.016092515, 0.0006018714, -0.003959213, -0.009203839, -0.01796737, -0.021958541, 0.01377025, -0.017115163, -0.007350288, 0.033946257, 0.026787715, 0.003927255, -0.034997314, 0.022185797, -0.04127524, 0.017157773, 0.025395777, -0.04155931, -0.029060269, 0.026276391, -0.00020794745, 0.016646449, -0.022185797, 0.005706238, 0.0031016795, -0.0017239443, 0.04871785, 0.015382341, -0.008706718, -0.011071593, 0.022881765, -0.0011265115, 0.0035810461, -0.006047121, -0.018223032, 0.016717466, 0.008081766, 0.014537236, 0.02417428, 0.0067075817, -0.024785029, 0.022867562, -0.003605902, 0.02668829, 0.012548752, 0.04127524, 0.0050173705, 0.003909501, -0.000646701, -0.0077479845, 0.0030697216, -0.01874856, 0.007076871, -0.0020115643, -0.02762572, -0.0061003836, 0.0013848369, 0.0076911706, -0.020296738, -0.015282917, 0.00448119, 0.017001536, 0.014601151, 0.016206142, -0.016333973, 0.013784453, 0.011028983, -0.0058518234, 0.014210557, 0.050592706, -0.023918618, -0.040423032, 0.018024184, -0.0036112284, -0.017214587, -0.003359117, -0.0039947215, 0.011440882, 0.018720154, -0.012477735, -0.051530134, -0.013379654, -0.01443071, 0.032383878, 0.025921306, -0.034997314, -0.0036325336, 0.006402207, 0.009310365, -0.027100192, -0.042439923, 0.018535508, 0.017953167, 0.04116161, 0.01558119, -0.01747025, -0.0054079653, -0.018606525, 0.024160076, 0.0061252397, -0.004147409, -0.0005965451, -0.021390403, -0.0139762, 0.008408445, -0.006636564, -0.0074355085, -0.030310173, -0.006885125, -0.005443474, 0.008564684, -0.029912475, -0.0074639153, -0.000834453, -0.02742687, 0.018805373, 0.020921689, 0.014274472, -0.0044172746, 0.020396162, 0.0069383876, 0.014203454, 0.0007199376, 0.003153167, -0.007932629, 0.011376968, 0.036531284, -0.0028460172, 0.002212188, 0.0042077736, -0.02256929, 0.006277927, -0.0068425145, 0.015339731, -0.01952975, -0.016873704, -0.00025322096, 0.00785451, -0.013379654, 0.0020701536, -0.019643378, -0.018919002, -0.019231478, -0.049314395, -0.014615355, 0.0053724567, 0.0071691936, 0.04905873, -0.0043604607, 0.018066794, 0.022540882, -0.0035118042, -0.006927735, -0.01681689, -0.024373129, -0.0074426103, -0.058688674, 0.025296353, 0.0024944816, 0.00958023, -0.008571785, 0.002212188, -0.029571593, -0.0045983684, -0.048888292, 0.01570902, 0.025111709, -0.006050672, -0.011803071, -0.0023666506, -0.0019867083, -0.014686372, 0.010269098, -0.029287525, 0.04596238, 0.011810172, 0.016603839, 0.034855276, 0.016802687, -0.034798466, 0.030565834, -0.009679655, -0.0042929943, -0.007115931, 0.0030803743, 0.022356238, -0.011036084, -0.0076911706, -0.013919386, 0.02389021, 0.016234549, 0.0023542226, -0.024472553, 0.04724069, -0.0039698654, 0.025821881, -0.0180952, 0.01714357, 0.017995777, 0.0014851487, -0.033719003, 0.020296738, -0.005965451, 0.016703263, -0.03880384, -0.0038455853, -0.020509789, -0.04292284, -0.028037619, -0.0060542226, -0.0008806142, 0.0133938575, -0.02812284, -0.017086755, 0.027668329, 0.0370142, 0.026077542, 0.042496737, 0.0226119, -0.033207677, -0.0134648755, 0.0017132917, -0.014643762, -0.006572649, -0.015723225, 0.0063489443, -0.0010510556, 0.050592706, 0.009935317, -0.011646833, 0.0062637236, -0.010659693, 0.025892898, 0.0026400671, -0.032866795, -0.010780422, -0.0054257195, 0.04695662, 0.0037923225, -0.010340115, -0.008877159, 0.0022139635, 0.005464779, 0.014011708, 0.004243282, -0.01146929, 0.025665643, 0.018379271, 0.0018855087, -0.030651055, -0.033974662, 0.018279847, -0.028222265, -0.010531861, 0.008245106, 0.00760595, 0.03502572, 0.0076130517, 0.009459501, 0.02808023, -0.018308254, 0.009807485, 0.017200384, -0.009068905, -0.008720921, -0.011895393, 0.007733781, -0.00563167, -0.0242595, 0.029798849, -0.017171977, 0.01484261, 0.010205182, 0.029827256, 0.016561229, 0.032469098, 0.032042995, 0.0115829175, -0.013216315, 0.01422476, 0.025665643, -0.035593856, 0.015595393, 0.020921689, 0.059597697, 0.010119962, -0.0017550144, -0.011938004, 0.0056636278, 0.03851977, 0.036332436, 0.0146721685, -0.025196929, -0.021305183, -0.029457966, -0.013983301, -0.036559694, 0.04596238, 0.0011194098, -0.034315545, 0.008649904, 0.00706977, 0.025267946, 0.033037234, -0.013152399, -0.0029117083, 0.014288675, 0.01948714, 0.0051984643, 0.0018393474, 0.0012525672, -0.018436084, -0.04988253, -0.022029558, -0.014096929, -0.0020097888, -0.0027448176, 0.034855276, -0.021575049, -0.05451286, -0.029202303, -0.020580806, -0.002746593, 0.013500384, 0.031673703, -0.0151834935, -0.020779654, 0.0043853167, -0.0101412665, -0.022086373, 0.021702878, -0.004715547, 0.04221267, -0.0296, 0.024401534, -0.008614396, 0.013756046, -0.006402207, -0.023932822, 0.0076698656, -0.013479078], "content_hash": "49a8b51d502bfa2a8a28b13d02fe3d33"}, {"content": "experiments. ### Can I use my own model? Yes! Type any HuggingFace model ID into the model dropdown. The dashboard supports GPT-2, LLaMA, OPT, GPT-NeoX, BLOOM, Falcon, and MPT architectures. Unknown architectures may need manual configuration in the sidebar. ### What's the difference between the pipeline and the investigation panel?\n\nThe **pipeline** (5 stages) shows what happens during the model's forward pass -- how your input is processed step by step. The **investigation panel** (ablation + attribution) lets you run experiments to understand *why* the model made a specific prediction.\n\n### How do head categories get determined?\n\nThe dashboard automatically analyzes each attention head's pattern using heuristic rules (based on thresholds for attention distributions). For example, a head is classified as \"Previous-Token\" if more than 40% of each token's attention goes to the immediately preceding token. These categories are computed fresh for each analysis.\n\n### Can I save my results?\n\nCurrently, results are displayed in the browser and aren't saved between sessions. You can take screenshots or copy text from the chatbot (using the copy button on messages) to record your findings.", "source_file": "troubleshooting_and_faq.md", "chunk_index": 2, "embedding": [-0.012068988, -0.020482069, 0.066405654, 0.015807303, 0.010540699, -0.035809908, -0.061611023, 0.030415947, 0.025291685, 0.04138367, -0.02326895, -0.051931858, 0.0010038761, -0.021246213, -0.036439203, -0.012271262, -0.007487867, 0.0035285496, 0.0023579851, 0.055407967, -0.009147259, 0.0063828537, 0.019508159, 0.05190189, 0.0064877365, 0.0039967755, -0.028333278, 0.03428162, -0.009926387, 0.003925605, 0.007484121, -0.03173447, -0.05690629, 0.00961923, -0.036079608, 0.0138744665, -0.015672453, 0.04530927, -0.013537344, 0.025561383, -0.025771147, -0.011971597, -0.022984268, 0.043960784, -0.05588743, 0.019687958, 0.022789486, -0.010555682, -0.037188366, 0.06766424, -0.03428162, 0.039735515, -0.045369204, 0.007180711, -0.00043076772, 0.0065251947, -0.0150057, -0.012893065, -0.034161754, 0.002397316, -0.0131178135, -0.0059820525, 0.009911403, 0.030625712, -0.0017848767, -0.01806977, -0.066645384, 0.013327579, -0.026070813, 0.0064914823, 0.013305104, 0.009596756, 0.010533207, -0.017365558, -0.0071582356, -0.025216768, 0.023238983, 0.025201784, -0.021815576, -0.05148236, -0.019628026, -0.021216247, 0.00394808, -0.04018501, 0.0041878116, 0.009888928, -0.04863555, -0.03628937, -0.041503534, -0.04593857, -0.010353409, 0.02216019, -0.00448373, 0.04255236, 0.035270512, 0.0035416598, -0.0023748411, -0.05393961, 0.0034835998, 0.052441288, 0.05310055, -0.02094655, 1.7675524e-05, -0.0114846425, 0.040394776, -0.017380541, 0.0099039115, 0.022684602, -0.026175695, -0.009491873, -0.059872966, -0.036259405, -0.02620566, 0.016047034, -0.01110257, -0.021890491, -0.0318843, 0.025606332, -0.010076219, -0.04965441, -0.024048077, 0.026550274, 0.028483111, -0.020452103, 0.0367988, 0.018833915, 0.020032572, -0.040244944, -0.013125305, -0.015320348, -0.020017589, 0.02959187, 0.029696753, -0.029187324, -0.013769584, 0.0019646755, 0.012795675, -0.0030621965, -0.065147065, -0.026460376, 0.024152959, -0.029966451, -0.018099736, 0.0004991287, -0.07090062, 0.0030996548, -0.024167942, 0.058734242, -0.024197908, 0.038896453, 0.0184893, 0.019643009, -0.02439269, -0.0138744665, -0.039405882, -0.0054014525, -0.0024441385, 0.032303832, -0.03637927, -0.0694023, 0.013177747, 0.032273866, -0.0075140875, -0.045908604, -0.057026155, 0.06796391, -0.040125076, -0.009124784, -0.0368887, -0.016886095, -0.020047555, 0.016616397, -0.01486336, -0.018983746, 0.006959708, -0.014466304, -0.053969577, -0.025051953, -0.005360249, -0.049744308, -0.023448747, 0.025950946, -0.016466565, -0.0207218, -0.0511827, -0.05268102, -0.006184326, 0.016721278, 0.009589264, -0.006210547, -0.00436761, 0.045549005, 0.06221035, 0.023718445, -0.01979284, -0.016781213, 0.052111655, 0.007933618, -0.007637699, -0.023493698, 0.022265073, 0.009679164, 0.049594477, -0.028168464, 0.006146868, 0.04195303, 0.014728511, 0.010091202, -0.060382396, -0.03841699, 0.014361422, -0.013365037, 0.0022362464, 0.05190189, -0.08774177, -0.009027394, -0.067004986, 0.017665222, 0.018998729, 0.039885346, -0.0049369726, 0.026595226, 0.041143935, -0.012271262, -0.024707338, 0.048935212, 0.043241587, -0.032154, -0.016751247, 0.0020058793, 0.009828996, 0.019702941, -0.04018501, 0.0010076219, 0.027898766, -0.048186053, 0.008068467, -0.035090715, 0.01676623, -0.03530048, -0.0032419953, 0.0139418915, 0.0075927493, -0.0055063353, 0.061251424, 0.0034274128, -0.009319566, -0.056786425, 0.03586984, 0.013372528, 0.020377187, 0.0006536432, -0.017830038, -0.033712257, 0.010922771, -0.02638546, -0.047796488, 0.022969285, -0.04018501, -0.026745057, 0.025831081, 0.017530374, -0.048186053, 0.02494707, -0.029891534, -0.03601967, -0.024782255, -0.03925605, -0.07042116, -0.052021757, -0.0062217843, 0.0011377886, 0.0041915574, 0.010653073, -0.045878634, 0.034401484, 0.019628026, -0.02025732, -0.0055175726, -0.0037626624, -0.009574281, -0.01361226, 0.037068497, 0.015852252, -0.003268216, 0.01287059, 0.017620273, -0.0048171068, 0.016526498, 0.005090551, 0.034880947, -0.030400964, 0.048245985, -0.029157355, 0.009484381, 0.04905508, 0.0253666, 0.022055307, -0.009933879, 0.037877593, -0.0014355803, -0.04938471, 0.026175695, 0.007858702, 0.0141441645, 0.033322692, 0.011169994, 0.03925605, 0.023688478, -0.018923813, -0.023583597, -0.030925376, 0.018039804, 0.013417479, -0.012046513, -0.02211524, 0.033322692, -0.0144138625, 0.014593662, -0.019043678, -0.035899807, -0.046897493, 0.020512035, -0.0025808606, -0.03823719, -0.0048508192, 0.015207973, 0.005438911, -0.021440996, -0.011274877, 0.04102407, -0.039675582, -0.006461516, 0.007731344, -0.00042608546, -0.015462688, -0.008937494, 0.03428162, -0.0088775605, 0.04710726, -0.06179082, -0.01691606, -0.058464546, -0.009484381, -0.037428096, 0.02225009, -0.0017830038, 0.024662388, -0.012623367, -0.031045243, -0.009372007, -0.033742223, -0.007420442, -0.041143935, 0.027194554, 0.045608938, 0.00011781731, -0.029367121, 0.013432462, -0.02494707, -0.042911958, -0.01561252, 0.04339142, 0.009454415, -0.0018054786, 0.008053483, -0.08564412, -0.02755415, 0.0010319697, -0.0023149082, 0.021216247, -0.002378587, -0.0048021236, -0.011162503, -0.04803622, -0.0046448, -0.002901127, 0.019628026, -0.00076648564, 0.071080424, 0.049744308, 0.030610729, -0.030296082, -0.043601185, 0.034371518, 0.06370867, 0.05729585, -0.004060454, -0.021965409, 0.010705515, 0.0022362464, -0.032064103, -0.023763396, -0.023313899, -0.014676069, 0.043601185, 0.019867757, 0.03290316, 0.0047609196, 0.04297189, -0.019583076, -0.015133058, -0.029516954, -0.0042852024, 0.041084003, 0.033052996, 0.007442917, 0.0069784373, 0.052201558, -0.018848898, 0.022325005, -0.0465379, 0.029711736, -0.014578678, -0.03572001, 0.03122504, 0.0067611802, 0.002399189, 0.023583597, 0.016781213, -0.015155532, -0.041593432, -0.01941826, -0.0073043224, -0.045908604, 0.021261197, 0.03817726, -0.033562426, -0.005648676, -0.015552588, -0.018744014, 0.03536041, 0.0465379, -0.019822806, -0.01612195, -0.04399075, 0.035989705, 0.033981957, -0.023943193, 0.02355363, 0.03781766, -0.023388814, 0.013589785, 0.015672453, -0.05301065, -0.015447705, -0.060382396, -0.045399174, -0.029382104, -0.058104947, -0.0033787172, 0.022414904, -0.014623628, 0.050253738, -0.011080095, -0.06340901, 0.04321162, 0.0007458837, 0.050343636, 0.00863783, 0.0029236018, 0.013005439, 0.007030878, 0.07683398, -0.013305104, 0.02810853, -0.029247256, -0.025336634, -0.014166639, -0.041323736, -0.0017521009, -0.026010878, 0.026730074, -0.043181654, -0.008937494, -0.049414676, -0.00034695532, -0.014818409, -0.0064240578, -0.052111655, 0.019508159, 0.030086316, 0.005525064, 0.042911958, 0.044769876, 0.0921768, -0.018654115, -0.032453667, 0.038506888, 0.019388294, -0.0008769869, -0.02290935, 0.059363537, -0.019013712, 0.038896453, -0.037218332, -0.027584117, -0.03664897, -0.00878017, 0.0015779209, 0.028003648, 0.0162568, -0.007540308, 0.07018143, 0.0048508192, 0.015537605, 0.005202925, 0.004375102, -0.018159669, 0.041413635, -0.038446955, -0.0031745709, 0.010503241, 0.01788997, 0.0048058694, -0.026819972, -0.0023636038, 0.0061131557, 0.003933097, 0.00854793, 0.00510928, 0.01765024, -0.020182405, -0.012286245, -0.0061768345, -0.037607897, -0.028453145, -0.0694023, -0.03925605, -0.021006482, 0.0009224048, -0.06113156, 0.0141441645, -0.030805511, -0.052740954, -0.012653334, -0.0014636738, -0.03140484, -0.061101593, 0.016017068, 0.04204293, 0.005083059, 0.017065894, 0.013777075, 0.0071619814, -0.00901241, 0.0041353703, -0.01788997, 0.0044949674, 0.021710694, 0.010106185, -0.030925376, 0.028932609, -0.01198658, -0.009746588, 0.0072219144, -0.010136152, 0.010061236, -0.0035641347, -0.014855868, 0.0098364875, 0.024812222, 0.022444872, -0.02503697, 0.015088107, 0.03604964, 0.013904433, -0.023463732, -0.022085274, -0.011304843, -0.008922511, 0.0025602586, 0.007562783, -0.013125305, 0.0063116834, 0.018324485, -0.00743168, 0.035390377, -0.026400443, -0.01087033, -0.032273866, -0.026819972, -0.027793882, 0.017065894, -0.008068467, 0.016886095, 0.016226834, -0.0069185044, -0.010877822, -0.01078043, 0.018294519, -0.053969577, 0.011282369, 0.025141852, -0.029966451, -0.015432722, 0.03512068, -0.004547409, -0.02216019, -0.019672975, 0.02490212, -0.0146835605, -0.0013690921, 0.004453764, -0.01997264, -0.0162568, -0.05648676, 0.005453894, 0.0027194554, -0.022055307, 0.005880916, -0.013657209, -0.02857301, -0.036678936, -0.029007524, 0.041143935, -0.04785642, 0.030131266, 0.03038598, -0.02890264, -0.02058695, -0.004094166, -0.010488258, 0.0029816618, 0.02523175, 0.0071432525, 0.008390606, -0.012443569, -0.017815055, 0.021830559, -0.025156835, 0.00039986483, 0.0414436, 0.006551415, -0.00627048, 0.008278232, 0.025216768, -0.015432722, -0.0054314192, -0.018848898, 0.01826455, -0.013762092, -0.020616919, -0.038536854, 0.00063819176, -0.001779258, 0.0111999605, 0.011035145, -0.030685645, -0.01696101, -0.027943715, -0.022774503, 0.0326035, 0.02443764, 0.006776164, -0.027344385, -0.004888277, -0.0070383702, -0.015477671, 0.034671184, -0.055407967, 0.04668773, -0.0047272076, -0.032813262, -0.04390085, 0.008293215, 0.0018803948, -0.0023860787, 0.027703984, 0.0014983225, -0.03098531, 0.01528289, 0.031764437, 0.023179049, 0.009866454, -0.0011864841, -0.0043863393, -0.07611478, -0.013005439, 0.058644343, -0.016975993, 0.0018101609, -0.07018143, 0.005513827, 0.04000521, 0.016901078, 0.006135631, 0.0075328164, -0.0064090746, 0.02917234, -0.0045174425, -0.0095667895, -0.017680205, -7.9656915e-05, -0.011904173, 7.555994e-05, -0.014428846, 0.033292726, 0.016990978, -0.033472527, -0.029427055, -0.031524707, 0.006330413, -0.034581285, 0.012952998, -0.0068398425, 0.0009991939, 0.03994528, 0.024287809, -0.01941826, -0.043780982, 0.0065926192, 0.011514609, 0.038716655, -0.03107521, 0.020062538, -0.019028695, 0.02875281, 0.004892023, -0.04282206, -0.0066488064, 0.026145728, -0.03919612, -0.005974561, 0.04168333, -0.018429367, -0.031704504, 0.0019983877, -0.031974204, 0.0052366373, 0.018938797, 0.00961923, -0.017320609, -0.04734699, 0.004229015, 0.009469398, -0.005749813, 0.0023336373, -0.0032831992, -0.020197388, 0.0066637895, 0.015342822, -0.015432722, -0.0117543405, 0.037937526, 0.012593401, -0.0010741099, 0.019867757, -0.0028262108, -0.04770659, 0.0060457313, 0.015178007, -0.026969805, 0.00031066782, 0.004079183, -0.021381062, 0.018384418, -0.00878017, -0.007948601, 0.031554673, 0.027029738, 0.038357057, -0.009304583, 0.019268427, -0.02207029, -0.015462688, 0.031045243, 0.024377707, 0.010098694, -0.0070271324, -0.009102309, -0.025306668, 0.02216019, -0.00743168, 0.073417805, -0.03646917, 0.044799842, 0.016931044, 0.009754079, 0.007956092, 0.0064690076, -0.00067096757, 0.0060382397, -0.025636299, -0.0035173122, -0.032273866, -0.018833915, -0.014054265, -0.03404189, -0.041653365, -0.03637927, -0.009971336, 0.004229015, -0.019298393, -0.013320087, -0.0019553108, 0.0005014698, -0.0974509, -0.023628546, 0.011724373, -0.027718967, 0.015537605, -0.020781733, 0.0050755674, -0.005611218, 0.008922511, -0.022969285, -0.008982443, -0.022280056, 0.014721019, -0.0135223605, 0.00026407934, -0.03586984, 0.009694147, -0.0020527018, 0.031015277, -0.015642487, -0.04060454, -0.0025490213, -0.009911403, 0.015732387, -0.04668773, -0.0018691573, -0.03577994, 0.020616919, 0.035749976, 0.005708609, -0.027299436, 0.01890883, -0.0026052082, 0.005764796, -0.018384418, 0.056516726, -0.030655678, -0.0020077522, 0.010046252, -0.010465783, -0.021650761, -0.018803949, 0.02207029, -0.013792058, 0.019163545, 0.0069484706, -0.018803949, 0.007236898, -0.027299436, 0.01937331, -0.019073647, -0.008165858, 0.026700107, 0.061161526, 0.022429889, -0.01937331, -0.009342041, -0.0073380345, 0.014938275, -0.027464252, -0.00743168, 0.02290935, 0.009754079, -0.012278753, -0.003462998, 0.005105534, 0.014084232, 0.033202827, -0.02592098, -0.032363765, -0.069522165, 0.017635256, -0.002859923, 0.006210547, 0.007427934, 0.0020564476, 0.015095599, 0.014129181, -0.008263249, -0.016376665, 0.02852806, -0.012518485, -0.056276996, 0.019837791, -0.00599329, 0.039136183, 0.042102862, -0.01579232, 0.010885313, 0.008233283, 0.034521352, 0.00032354402, -0.0016931044, 0.012563434, -0.034671184, -0.009784046, -0.027718967, 0.020347219, 0.05301065, -0.012660826, 0.01862415, 0.011042637, -0.004618579, 0.0032944367, -0.002361731, -0.011649458, -0.0040267417, -0.010787922, -0.019013712, -0.008360639, 0.014878343, 0.014518745, 0.00046213885, -0.0068398425, 0.0071469983, 0.007701378, 0.060232565, -0.002942331, 0.016990978, 0.0021388554, 0.021650761, 0.0077388356, -0.009229667, 0.035090715, -0.0007819371, 0.010053745, 0.010143643, -0.009304583, -5.393376e-05, -0.014983226, 0.011612, -0.008592879, -0.0023842058, 0.0059670694, -0.007941109, -0.0030640697, 0.025291685, -0.006195564, -0.010375883, 0.0056674047, -0.023433764, 0.025381584, 0.004888277, -0.027014755, -0.027539168, 0.0054426566, -0.007971075, -0.027763916, 0.0022868149, 0.031764437, 0.03506075, -0.0039855377, -0.01714081, 0.0059445947, 0.00845803, 0.029606853, 0.02373343, -0.042372562, -0.034641217, 0.03781766, 0.021201264, 0.043541253, 0.02156086, -0.031195074, 0.021770626, 0.0017904954, -0.0033318948, 0.012473535, 0.022864401, 0.045339238, -0.035150647, 0.021366078, 0.043601185, 0.0050118887, 0.038776588, 0.044020716, 0.010001303, 0.020152438, -0.016721278, -0.0025565128, -0.01788997, 0.035989705, -0.0004759515, 0.017590307, 0.02169571, 0.010698023, -0.019178528, 0.039106216, -0.0048508192, 0.028003648, 0.0208117, 0.010832872, -0.002339256, 0.055677664, -0.023538647, -0.014061756, -0.0006007337, 0.0036690172, -0.0601127, -0.008997426, -0.011582033, 0.034731116, -0.0022531026, -0.028333278, 0.009049868, 0.050613336, -0.012878082, 0.015043158, -0.0020733038, 0.0068286047, -0.013724634, 0.011035145, -0.018519266, 0.0013578548, -0.01714081, 0.009207192, 0.02625061, 0.010211068, -0.025066936, 0.0012820022, -0.030191198, -0.029741703, -0.04761669, -0.007173219, -0.0072743557, 0.034161754, 0.03907625, 0.009252141, -0.0032064102, 0.020467086, -0.0025359108, -0.0129829645, 0.026310543, 0.015462688, -0.011252402, 0.004094166, 0.015507638, -0.016751247, 0.012833132, 0.000248862, 0.02684994, 0.024602456, -0.025651282, -0.01645158, 0.015357805, 0.006907267, 0.010585649, 0.0008006661, 0.008697762, -0.008787662, -0.004757174, 0.026175695, 0.024197908, -0.0013232061, 0.026040845, 0.0130503895, 0.03404189, 0.02810853, 0.023718445, -0.00669001, -0.0025471484, 0.04432038, -0.004217778, -0.019523142, 0.005274095, -0.0002341129, -0.026745057, -0.014323963, 0.015335331, 0.0161669, -0.0092970915, 0.028857691, -0.027389336, 0.012406111, -0.024962053, 0.021141332, -0.005304062, -0.012458552, -0.00989642, 0.003888147, 0.0020189895, -0.0021238723, -0.014511254, -0.023658512, -0.012735741, -0.032064103, -0.036619004, -0.006480245, 0.010623107, 0.0022605942, -0.0044350345, 0.014563695, 0.01156705, -0.026520308, -0.013529852, 0.022504805, -0.008705254, -0.016316732, 0.024542524, -0.031045243, 0.006030748, -0.020841666, -0.002339256, -0.06610599, 0.0081358915, -0.0076789027, 0.021470962, -0.049834207, -0.0049819225, 0.028183447, -0.0064315493, -0.0010488258, -0.004229015, 0.027868798, 0.004176574, -0.003498583, 0.020676851, -0.017365558, -0.03107521, -0.01797987, -0.016331716, 0.030041367, -0.018309502, -0.0101885935, -0.0053415196, -0.042432494, 0.025576366, -0.009446924, -0.004989414, -0.008398098, 0.012256279, 0.001914107, 0.01806977, 0.002181932, -0.039735515, -0.008555422, -0.033981957, -0.024647405, -0.015537605, -0.01528289, 0.0017904954, 0.042492427, -0.017725155, 0.018788964, 0.018998729, -0.008487997, 0.0033187843, -0.012503501, 0.028048597, -0.018039804, 0.06179082, -0.014316471, -0.03754796, -0.014443829, -0.0013559819, 0.021261197, 0.021440996, -0.027958699, -0.019538125, -0.003461125, 0.012720758, 0.03293313, -0.018968763, 0.03043093, 0.0011087586, 0.0010188593, -0.0067986385, -0.009229667, 0.028947592, -0.03158464, 0.009491873, 0.039915312, 0.0052703493, -0.0063379044, 0.021306146, 0.021545878, -0.006330413, -0.014114198, 0.0024478845, 0.02443764, -0.021141332, -0.035000816, -0.009694147, 0.0252767, -0.015837269, -0.0018626022, -0.0031314942, 0.052141625, -0.0048433277, -0.018309502, 0.015642487, 0.008338165, 0.017485423, -0.025007002, 0.0029329665, 0.0013719016, 0.005334028, 0.0012885574, 0.007776294, -0.027898766, -0.0032457411, 0.016062018, 0.014743494, -0.012878082, 0.017005961, -0.035839874, -0.009079834, 0.03586984, 0.02490212, 0.014241556, -0.02448259, -0.00047876086, -0.029606853, 0.018309502, 0.0014177876, -0.052381355, -0.024317775, -0.012016547, 0.005004397, 0.020556984, 0.008900036, -0.0076489365, -0.00040735645, -0.017904954, 0.05199179, 0.029157355, -0.0015189245, -0.017620273, 0.021006482, 0.01862415, -0.003522931, -0.0007355827, -0.016706295, -0.009694147, 0.008075958, 0.013297613, 0.021590827, 0.0014618009, -0.028423179, 0.007701378, 0.0029442038, 0.006768672, 0.028737826, 0.0551982, 0.00036334322, 0.03934595, -0.022010358, -0.014915801, 0.02220514, -0.018039804, -0.0082707405, 0.00094160205, -0.02388326, 0.0046522915, -0.013222696, 0.014286505, -0.031944238, 0.0150356665, 0.0066375686, 0.018938797, -0.01078043, 0.017320609, -0.027748933, 0.030625712, 0.001989023, -0.001768957, 9.2650174e-05, 0.039016318, -0.03985538, -0.050014004, 0.024212891, 0.0015994593, 0.00047618564, 0.013065373, 0.015987102, 0.012488518, 0.031344905, -0.027464252, -0.016901078, -0.0070271324, -0.008855086, 0.029891534, 0.02192046, -0.022564737, -0.013110322, -0.010061236, -0.017170776, -0.049774274, -0.050523434, 0.02388326, 0.023104133, 0.045189407, 0.0097016385, -0.03823719, 0.014810918, 0.0036615257, 0.02021237, 0.010915279, 0.02452754, 0.010046252, -0.022744536, -0.020961532, 0.029681768, -0.025246734, 0.011559558, -0.021935442, -0.0019496921, -0.01705091, -0.015013192, -0.003477981, -0.014511254, 0.00680613, -0.026280576, 0.013859483, 0.0033038012, 0.007405459, -0.011417218, 0.025771147, 0.008817628, -0.0010619361, 0.00743168, -0.031195074, -0.0031202566, 0.006379108, 0.011649458, -0.016391648, 0.0032345038, 0.009132275, -0.024347741, -0.013357545, -0.006210547, 0.0026407933, -0.009873945, -0.021440996, 0.0010197958, -9.47572e-05, -0.013200222, 0.009342041, -0.041773234, -0.011042637, -0.030685645, -0.036858734, 0.0066787726, -0.011342301, 0.017290642, 0.07072082, -0.0036072114, 0.046238232, 0.031165108, -0.012578418, -0.014368913, 0.012368652, -0.039166152, 0.0005501653, -0.04129377, 0.023074167, 0.0051317546, -0.012503501, -0.012690792, 0.03577994, -0.00028983178, 0.0050868047, -0.041054036, -0.00092849176, 0.032273866, -0.009274617, -0.0041503534, -0.010915279, 0.026595226, -0.021725677, 0.015133058, -0.01477346, 0.03892642, -0.014721019, -0.014765969, 0.016316732, 0.031704504, -0.020152438, 0.007971075, -0.0043975767, 0.009169734, -0.024737304, -0.00034555062, 0.031105176, -0.009641705, -0.0068735546, -0.01676623, -0.013065373, -0.017230708, 0.0056823883, -0.0049557015, 0.031165108, -0.014428846, 0.04725709, -0.002764405, 0.018384418, 0.021036448, -0.0028355753, -0.0129829645, 0.0008657495, 0.009109801, 0.034910914, -0.024692355, 0.00062742253, -0.020407153, -0.033442557, -0.015927168, -0.010757956, -0.029741703, 0.00074494723, -0.008233283, -0.0061431224, 0.010683039, 0.005734829, 0.035690043, 0.032213934, -0.015672453, -0.0033038012, -0.027898766, 0.00924465, -0.0029086187, -0.007506596, -0.012705775, -0.021171298, 0.04240253, 0.039106216, 0.004097912, 0.0037589166, -0.022384938, -0.013087847, 0.008113417, -0.012248786, -0.0059783068, -0.018684082, -0.01170939, 0.008630338, 0.019987622, 0.003805739, 0.00789616, 0.0036390508, -0.0042252694, 0.018429367, 0.0139418915, -0.0053527574, 0.024197908, 0.025576366, 0.0014187242, -0.022924334, -0.031794403, 0.0323338, 0.015912184, -0.038806554, 0.0056524216, -0.002241865, 0.05217159, 0.0011256147, 0.0126833, 0.014084232, -0.010151135, -0.0050718216, 0.015972119, -0.0071619814, -0.015327839, -0.0040304875, 0.023673495, 0.0010516351, 0.003539787, 0.0026333018, -0.018699065, -0.009064851, 0.021815576, 0.009304583, 0.008165858, 0.025441516, 0.02634051, -0.010518224, 0.014706035, 0.026370477, 0.019852774, -0.025666265, 0.013005439, 0.015627503, 0.037727762, 0.005124263, -0.0044387802, -0.031135142, 0.0026464122, 0.0060270024, 0.026475359, 0.0037476793, 0.003933097, -0.035270512, -0.018414384, -0.008360639, -0.013657209, 0.02857301, 0.020407153, -0.044050682, 0.014189114, 0.01263835, 0.010248526, 0.038626753, -0.015537605, 0.010952737, -0.028453145, 0.003423667, -0.015133058, 0.007933618, 0.014855868, -0.04051464, -0.029442038, -0.0061393767, 0.0021538385, 0.032034136, -0.012301228, 0.027569134, -0.010765447, -0.029337155, -0.0114846425, -0.0024609948, -0.01477346, -0.012855607, 0.041233838, -0.023523664, -0.021321129, 0.0063903457, 0.016151916, -0.009004918, 0.013215205, -0.033951987, 0.022864401, -0.027748933, 0.013866975, -0.00459985, 0.0122263115, -0.008817628, -0.029861568, 0.0301013, -0.0324237], "content_hash": "641cc2ed4a5b47aa3552073d7a4473fe"}, {"content": "# What Is a Large Language Model (LLM)?\n\n## The Big Idea\n\nA Large Language Model is a computer program that has learned to read and write text by studying enormous amounts of human writing. Think of it like an incredibly well-read assistant that has absorbed millions of books, articles, and websites, and can now predict what word comes next in a sentence.\n\n## How It Works (Simply)\n\nAt its core, an LLM does one thing: **next-token prediction**. Given some text like \"The cat sat on the\", the model predicts the most likely next piece of text (called a \"token\") -- perhaps \"mat\" or \"floor.\"\n\nThis might sound simple, but to do it well, the model has to understand grammar, facts, context, and even some reasoning. All of that understanding is encoded in the model's **parameters** -- billions of numbers that were learned during training.\n\n## What Is a Neural Network?\n\nAn LLM is built on a type of computer program called a **neural network**. A neural network is loosely inspired by the brain: it's made of layers of simple processing units that pass information forward, transforming it step by step. Each layer takes input numbers, multiplies and adds them, and passes the result to the next layer.\n\nWhen you stack many layers together and train them on lots of data, the network learns complex patterns -- like how words relate to each other.\n\n## What Makes It \"Large\"?\n\nThe \"large\" in LLM refers to two things:\n\n- **Many parameters**: Modern LLMs have billions of learnable numbers (GPT-2 has 124 million; larger models have tens or hundreds of billions).\n- **Massive training data**: They train on huge text datasets -- sometimes trillions of words from the internet, books, and code.\n\n## How Does This Connect to the Dashboard?\n\nThe Transformer Explanation Dashboard lets you look inside an LLM as it makes a prediction. When you enter a prompt and click \"Analyze,\" you can see:", "source_file": "what_is_an_llm.md", "chunk_index": 0, "embedding": [-0.008591564, -0.007622969, 0.011082237, -0.017233444, 0.052983407, -0.012906214, -0.0023821779, 0.017761769, -0.04354904, 0.09077119, -0.0078053665, -0.028428894, -0.044983063, -0.015346571, -0.00204883, -0.028126994, -0.003610218, 0.010553912, 0.021950629, 0.052580874, 0.02012665, 0.0061103255, 0.012994269, 0.027950887, 0.033334766, -0.015874896, -0.015497521, 0.044832114, 0.008220479, -0.0015598154, 0.010384094, -0.021510357, -0.066015415, 0.011094816, -0.014516347, 0.015170462, 0.020768188, 0.02880627, 0.013321327, 0.040328775, -0.049737986, -0.025862746, -0.01296911, 0.01774919, -0.056153353, 0.044203155, 0.006516003, -0.002750118, -0.042316284, 0.049888935, -0.019208372, 0.027850254, -0.031548526, -0.015862316, -0.020315338, -0.017183129, -0.01790014, 0.026919395, -0.020026017, -0.045612022, 0.009635634, -0.009308576, 0.010038167, 0.03801421, -0.017095074, -0.028605003, -0.041083526, 0.014050918, -0.018541677, -0.011585403, 0.03285676, -0.0071386714, 0.019497693, -0.012805581, -0.009906086, -0.00641537, -0.04251755, 0.028001202, -0.031246625, -0.0097236885, 0.010962735, -0.026894236, -0.005993968, -0.03756136, 0.021522937, -0.018403307, 0.008346271, -0.010308619, -0.028101835, -0.04699573, 0.022340583, -0.028705636, -0.009899796, 0.022755694, 0.03134726, 0.008245637, 0.0042737685, -0.004298927, 0.0016384352, 0.04088226, -0.009302286, 1.3869011e-05, 0.038567696, -0.058971085, -0.0087550925, -0.019120319, 0.016944123, -0.022579586, -0.01923353, -0.027825095, -0.11180355, 0.0020598369, 0.008817988, -0.019812172, 0.019623484, -0.036806613, -0.0016384352, 0.034164988, 0.0001027953, -0.031875584, -0.0048241066, -0.024378406, 0.018830998, -0.012214361, 0.026466545, 0.01832783, 0.0022265108, 0.035171323, 0.029711967, 0.026667811, -0.013686122, 0.0021180154, 0.00925197, -0.06767586, -0.042115018, -0.0005200695, -0.03637892, 0.032278117, -0.015195621, 0.0079185795, 0.044983063, -0.020415971, -0.007994054, 0.011868434, -0.04830396, -0.00544363, -0.0024607976, 0.039901085, -0.0596252, 0.00424861, 0.011069658, -0.010648256, -0.040404253, 0.009333734, -0.022088999, -0.01535915, 0.0065222927, 0.050014727, 0.0135603305, -0.04830396, 0.032378748, 0.055599872, 0.013396801, 0.036957562, -0.013510014, 0.07834299, -0.022894066, -0.013145219, -0.03134726, -0.054643854, -0.02566148, 0.016604487, 0.0076103895, -0.04060552, 0.013384223, -0.031850424, -0.041637007, -0.05192676, -0.0012885773, -0.07185214, -0.044328947, 0.011811828, -0.052128024, -0.021849995, -0.02459225, 0.012503682, -0.0090129655, 0.03386309, -0.015233358, -0.019170634, 0.0014992782, 0.052077707, 0.091777526, 0.055398606, 0.028403735, 0.011925041, 0.07134897, -0.009358892, 0.04712152, 0.019573167, 0.05710937, 0.06465686, 0.027699303, -0.07975185, 0.028504368, 0.024642568, -0.0077927876, 0.015094988, -0.025711795, -0.034642998, 0.00999414, -0.020516604, 0.058820136, -0.010031877, -0.04103321, -0.01790014, -0.065109715, -0.0047140387, 0.00603485, 0.022164473, 0.008113556, 0.031699475, 0.013635806, 0.024441302, -0.067625545, 0.025988538, 0.051624857, -0.02624012, 0.027120661, 0.0003203754, 0.049587034, -0.021585833, -0.008365139, -0.051624857, -0.018742943, -0.051247485, 0.0102708815, 0.008365139, 0.022101577, 0.0018271225, 0.010453279, 0.015522679, 0.030139659, 0.016315166, 0.010277171, 0.012906214, -0.023296598, -0.009415499, 0.06651858, 0.046693828, 0.019937962, 0.0077361814, -0.045461074, -0.0052329293, -0.02624012, 0.032001372, -0.04709636, 0.025711795, -0.034391414, -0.021887733, -0.0033460557, 0.0153339915, -0.04966251, 0.017409552, -0.0055851457, -0.024277773, -0.00014072932, -0.010296039, -0.060480583, -0.016478695, -0.005912204, -0.026542021, -0.026516862, 0.0027123806, -0.039448235, -0.084230036, -0.0070631965, 0.018629732, 0.040026877, 0.046341613, 0.0011808684, -0.008409166, 0.042416915, 0.029737126, -0.008100977, 0.0014906301, -0.005368155, -0.01684349, -0.009295996, -0.030818934, 0.023334336, -0.014742771, 0.0140131805, -0.023686552, -0.025837587, 0.01640322, -0.01298169, 0.042693656, -0.00589648, 0.037485886, 0.022692798, -0.034441732, -0.020026017, -0.026441388, -0.01788756, 0.060027733, -0.0050631105, -0.011679747, 0.004487614, -0.004100805, 0.008547537, 0.013459697, 0.042869765, 0.013245852, -0.02223995, -0.016818332, 0.01537173, -0.033938564, 0.011006762, -0.020051176, 0.0030740313, -0.017547924, 0.027472878, -0.017560503, -0.012170334, 0.02400103, 0.019321585, 0.005324128, -0.027246453, -0.030919567, 0.075726524, 0.04754921, -0.012516261, 0.024026189, -0.028655319, -0.07421702, -0.02236574, 0.02161099, 0.014994355, 0.04415284, -0.010182827, -0.0041982937, -0.024780938, 0.01670512, -0.0064059356, 0.030919567, 0.0017123377, 0.018994527, 0.059826467, 0.005934217, -0.015409467, 0.0041668457, 0.00910102, -0.021912891, 0.009182785, 0.03074346, -0.0060914564, -0.013082323, 0.052882776, -0.005267522, -0.058266655, 0.0022249382, 0.0094343675, 0.014101234, -0.04968767, 0.01566105, -0.025334422, -0.0032092573, 0.012264677, -0.023070173, 0.040932577, 0.007176409, 0.02460483, -0.01567363, -0.047347944, 0.011277214, -0.0015716084, 0.014881142, -0.05328531, 0.049436085, 0.021044929, 0.034970056, -0.0012131024, 0.007396544, 0.0055222497, 0.04400189, -0.001789385, -0.011371558, 0.00042140175, -0.00020303545, -0.025321841, -0.02354818, -0.012692369, 0.0073273587, 0.010855812, 0.017283762, 0.05489544, 0.04193891, 0.039146334, 0.029057851, -0.013132639, -0.008717355, -0.02863016, -0.0014489616, 0.04322198, 0.013774176, 0.027623829, 0.024416143, 0.028982377, 0.010075904, 0.044328947, 0.007352517, 0.008182742, -0.042593025, 0.04100805, 0.023523023, 0.021938048, 0.046618354, 0.020151809, 0.022315424, -0.022579586, -0.03746073, -0.025032522, -0.022441216, -0.0012964393, 0.011132553, 0.03117115, -0.0075978106, -0.036479555, -0.025812428, 0.013409381, 0.04098289, 0.031523366, -0.017371815, -0.007692154, -0.03217748, 0.019107739, 0.05036694, 0.018214619, -0.033787616, 0.04221565, 0.011302372, 0.008327401, 0.0010676559, 0.018692628, 0.022114158, -0.055901773, -0.021963207, 0.007824236, -0.031800106, 0.026667811, -0.010088484, -0.000518104, 0.040756468, -0.027850254, -0.055901773, 0.05187644, -0.022780852, 0.07270753, -0.027976044, 0.04566234, 0.02742256, 0.0054813675, -0.053536892, -0.008767672, 0.001410438, 0.008189031, -0.02147262, 0.039574027, 0.0041416874, -0.007050617, -0.038416743, 0.03190074, -0.0005845377, 0.004858699, 0.019698959, 0.024944467, -0.0325297, 0.0033366212, -0.029183643, -0.0010778764, 0.0077298917, 0.002937233, -0.018705206, -0.008843147, 0.04787627, -0.032907072, -0.0013475422, 0.027372245, 0.0031903887, -0.026315596, -0.0069562737, 0.037234303, -0.012799292, 0.005710937, -0.0023727433, -0.024416143, -0.080959454, 0.013006848, -0.023938134, 0.015749104, -0.02549795, -0.032303274, 0.06893378, 0.0041982937, 0.025133155, 0.07295911, 0.018755522, -0.026969712, 0.02744772, -0.010836943, 0.00068123994, 0.013887389, 0.02281859, -0.013359064, 0.025812428, 0.0033932275, 0.010402963, 0.0142396055, 0.023120489, 0.010245723, 0.012598026, -0.027674144, -0.015598154, 0.016692542, -0.0051480196, -0.03159884, -0.0028444617, -0.022164473, 0.0028916337, 0.00701288, -0.02176194, 0.009667082, 0.011164001, -0.05771317, -0.007968896, -0.003474992, -0.029460385, -0.06948726, -0.012774133, 0.005789557, 0.051021058, 0.055247657, 0.009937534, 0.004188859, 0.012849608, 0.0115287965, -0.037963893, 0.0007842318, 0.016629646, 0.043649673, -0.012654631, 0.017686294, -0.032554857, -0.016918965, -0.011604272, -0.012289836, 0.0016950413, 0.014126393, -0.014252184, -0.0012469089, -0.007220436, 0.0049750563, -0.0025708652, -0.0062707094, -0.010094773, 0.0067801652, 0.016969282, 0.012742685, 0.017396973, -0.007975185, -0.024466459, 0.012704948, 0.028957218, 0.002327144, -0.04548623, 0.017975615, 0.009667082, 0.00045009796, 0.0417628, -0.029636493, -0.0393476, -0.006755007, 0.0023019856, -0.030793775, 0.032881916, -0.026617495, -0.012648342, -0.031473048, 0.011969067, 0.03401404, -0.029762285, -0.01745987, 0.01684349, -0.053335622, -0.002243807, 0.022126736, -0.04802722, -0.033183817, 0.013346485, -0.0048555546, -0.02983776, -0.010862102, 0.008402877, 0.02923396, -0.025586005, -0.040001716, 0.021837415, -0.009579028, -0.010384094, -0.038416743, -0.035448063, -0.008289664, -0.056153353, -0.0408571, 0.03358635, -0.012956531, -0.008327401, 0.02893206, -0.02593822, 0.0014175137, -0.010956446, 5.6016557e-05, -0.021598412, 0.008082108, 0.0033051735, -0.022554427, 0.021070087, -0.04203954, 0.028479211, -0.022390898, 6.825175e-05, 0.07673285, -0.012843319, 0.0031353547, 0.03791358, 0.00879912, -0.030995041, 0.001797247, -0.018604573, 0.010421831, 0.011371558, -0.015522679, -0.0037674573, -0.006905957, -0.0003001308, -0.02223995, -0.004371257, -0.003088183, 0.0011171864, -0.010541333, -0.022806011, 0.04203954, 0.028001202, 0.003430965, -0.02162357, 0.009201653, -0.017535344, 0.005500236, 0.038391586, -0.053637523, -0.0055599874, 0.00031585476, -0.01342196, -0.020667555, 0.027774777, 0.029561019, 0.00804437, 0.019409638, 0.01296911, -0.07250626, 0.022403479, -0.00081685896, 0.008899753, -0.027472878, -0.0067675863, -0.0064059356, -0.037183985, 0.0056574754, 0.02236574, 0.00075435627, 0.020831084, -0.023673972, -0.008623011, 0.00026632432, 0.012201782, 0.03265549, 0.014189289, 0.007616679, 0.023195965, -0.010220565, -0.052429926, -0.02461741, -0.05144875, -0.0043177954, -0.040680993, -0.013522593, 0.08739998, -0.009163915, -0.09766457, -0.010050746, 0.012327573, -0.053084042, 0.0043460988, -0.010912418, -0.023120489, -0.04324714, 0.004239176, 0.015220779, -0.007415413, -0.022038681, 0.014881142, 0.020529183, 0.011170291, -0.028605003, -0.014189289, 0.0020897123, -0.022894066, -0.008189031, -0.018063668, -0.066719845, 0.005431051, -0.02507026, -0.020013439, 0.042718817, -0.015296254, -0.014629559, 0.0075286254, -0.016818332, 0.017963035, -0.009459525, 0.013824493, 0.011403006, -0.039121177, -0.01567363, 0.0008530241, -0.032102007, 0.003311463, -0.055952087, -0.008591564, -0.03758652, -0.0023758882, -0.03922181, 0.03308318, 0.032605175, 0.00910102, 0.0086921975, 0.025284104, -0.022629902, -0.036026705, -0.052178342, 0.0102708815, -0.0008420173, -0.013673543, 0.018126564, -0.013296168, 0.017485028, 0.0043115057, -0.0041039498, -0.017623398, -0.0032296984, 0.009849479, -0.009157626, 0.036303446, 0.0087236455, -0.023762027, -0.0034561232, -0.026290437, 0.0024340667, -0.010874681, -0.03668082, -0.021183299, 0.024114244, -0.0052738115, 0.0023051305, -0.023095332, 0.03620281, 0.022265106, -0.0002789035, 0.019032264, -0.009880927, -0.021095246, 0.034718473, -0.053889107, 0.014390555, 0.009371472, -0.04369999, -0.03610218, -0.01566105, -0.020101491, -0.022252528, -0.0148182465, 0.024516776, -0.020680133, 0.013057165, 0.020680133, 0.023145648, -0.088154726, -0.04531012, 0.042693656, -0.0044121393, 0.00054208306, -0.019309005, 0.009050703, 0.0152710965, 0.0075663626, -0.03773747, 0.0049310294, -0.03640408, 0.0050190836, 0.03144789, -0.010201696, -0.057310637, -0.013648385, -0.007591521, 0.012277257, 0.016290007, -0.022843748, 0.020013439, -0.035699647, 0.011723774, -0.010396673, -0.008472062, -0.01894421, 0.025887905, 0.014063497, 0.016491273, -0.013522593, 0.02835342, -0.012874767, 0.01042812, 0.0055599874, -0.00032017883, -0.0048932917, -0.04739826, -0.038819276, -0.016667383, -0.0063021574, -0.045989398, 0.03029061, -0.02699487, 0.027850254, -0.009667082, 0.008390297, -0.003965579, -0.008031791, -0.01342196, -0.023824923, 0.010101063, 0.02863016, -0.0021148706, 0.0076669957, 0.0006379991, -0.004607116, -0.0035567565, 0.037511043, 0.053084042, -0.005902769, -0.0024089084, -0.025422476, -0.0073902546, -0.020982033, 0.026818762, 0.009572738, 0.027976044, -0.026265278, -0.025271526, 0.01402576, 0.017258603, -0.0001171434, 0.017170548, 0.030667983, 0.021384567, 0.0024183427, 0.04058036, 0.0140131805, 0.0049561877, 0.022151895, -0.014076076, -0.02460483, 0.0049939253, 0.02354818, 0.03907086, 0.020818504, -0.013660964, -0.017635977, 0.03897023, 0.025736954, 0.027347086, -0.01282445, -0.0063807773, -0.016918965, -0.0044058496, 0.007585231, 0.037183985, 0.04445474, -0.030064184, 0.04012751, 0.040957734, -0.043624517, -0.028454052, -0.029762285, -0.021070087, -0.02400103, -0.014138972, 0.0049719117, 0.0039718687, 0.021145562, 0.034341097, 0.00894378, 0.04531012, -0.019472534, 0.027498037, 0.040731307, 0.030391242, 0.030944725, -0.007126092, -0.021095246, 0.01073631, -0.02520863, -0.012855898, -0.01968638, 0.006723559, 0.0018349845, 0.029183643, 0.00047958034, 0.00076890097, -0.03280644, -0.0070443274, -0.0039152624, -0.02893206, -0.02611433, 0.016327744, -0.0149063, 0.035121005, -0.0063902116, 0.023623656, 0.008880884, 0.023296598, 0.023007277, -0.026592337, 0.021359408, 0.013761598, -0.0298126, -0.06651858, -0.024365826, 0.016076162, 0.047046047, 0.015208201, -0.0079185795, 0.058367286, -0.016013267, -0.008025502, -0.001617994, -0.0626442, -0.035976388, 0.015346571, 0.0011706478, 0.033787616, -0.0022878342, 0.00070954306, 0.034114674, -0.002231228, -0.02759867, 0.0058839004, -0.029686809, 0.014956617, -0.010761469, 0.00037167477, 0.023535602, 0.0067675863, 0.02340981, 0.011711195, 0.0048398306, 0.022026103, 0.020403393, 0.013975443, -0.018969368, 0.02714582, 0.0043209405, 0.05368784, 0.004890147, -0.009635634, 0.0045379307, -0.020252442, -0.021875152, 0.037812945, 5.208557e-05, 0.01850394, 0.03285676, 0.015107567, 0.0051039928, -0.036932405, 0.0046008267, 0.04485727, -0.0014489616, -0.010572781, -0.02071787, 0.03401404, -0.0053178384, -0.0066040573, -0.02938491, 0.03356119, 0.005547408, 0.0068682195, -0.005874466, 0.002734394, -0.01491888, -0.0011148277, -0.017761769, 0.022177054, 0.01118287, -0.013811914, 0.020013439, 0.004204583, 0.01431508, -0.036429238, -0.02283117, -0.024944467, 0.024944467, 0.03371214, -0.0006588333, 0.020101491, 0.018994527, 0.0051071374, -0.012956531, -0.007251884, -0.004635419, -0.012101148, 0.023925556, 0.0013247425, -0.003953, -0.005484512, 0.008906043, -0.009830611, 0.036127336, -0.0007799077, 0.013258431, -0.02596338, -0.049134184, -0.0023585919, 0.0049467534, 0.028026361, 0.016679961, 0.0023727433, -0.031523366, -0.023950715, 0.008107266, 0.027648985, 0.015434625, -0.03788842, 0.03474363, -0.0049153054, 0.023824923, 0.036051862, 0.013899968, -0.008824279, -0.014478609, 0.051272642, 0.012327573, 0.0018035366, 0.005056821, -0.0127867125, -0.0027108083, -0.0023853225, 0.0075286254, 0.0032988838, -0.06878283, 0.04442958, -0.015824579, 0.02266764, 0.0033775035, 0.0024356393, 0.00477379, -0.0028617582, 0.004371257, 0.03801421, -0.028227627, 0.012906214, -0.0006737711, -0.027196137, -0.0064185145, -0.0015810428, -0.008264506, 0.0034120963, 0.017208286, -0.026013697, -0.019472534, 0.026642654, 0.005387024, -0.005773833, -0.011252056, 0.0015944081, 0.0028601857, -0.008377718, 0.0025032521, -0.018453622, 0.03295739, 0.016793175, -0.0047454867, -0.02938491, -0.008616722, -0.019044843, -0.007585231, -0.04578813, -0.0115287965, 0.0070631965, -0.008025502, 0.0022611034, 0.038567696, 0.023573339, -0.007339938, -0.007874552, -0.010314909, -0.005833584, -0.00079327304, -0.010667125, -0.008924912, -0.012346442, -0.0028869165, -0.0107551785, 0.005314694, 0.0058052805, 0.032278117, -0.0061952346, 0.015006933, 0.026466545, 0.04216533, 0.0018585704, -0.003758023, 0.01656675, -0.04576297, -0.020705292, -0.014554084, -0.035699647, 0.026189804, -0.027724462, 0.037989054, 0.03620281, 0.0022894065, -0.005726661, 0.0033743586, -0.009176495, 0.010641966, -0.02117072, 0.010321198, 0.011730064, 0.06043027, -0.025422476, -0.01894421, -0.0013239562, -0.008824279, 0.035749964, -0.016239692, -0.013220694, -0.032026533, -0.005993968, -0.013975443, 0.021271354, -0.011245766, -0.012516261, 0.044278633, -0.0019010251, -0.006308447, -0.0014937748, 0.009924955, 0.012906214, 0.014440872, -0.005462499, -0.029007535, -0.033737298, -0.011566535, 0.018428463, 0.0071638296, -0.009289707, 0.01908258, 0.032026533, -0.013799335, -0.016302587, -0.01103192, 0.033787616, -0.008327401, -0.008409166, 0.015006933, -0.0023129925, -0.032303274, -0.0063021574, 0.005726661, -0.0094658155, -0.034517206, -0.014855984, 0.0054499195, 0.023837501, 0.009591606, -0.014377976, -0.014667297, -0.01236531, -0.024491617, -0.029057851, 0.010340067, -0.00925197, 0.014755351, -0.029862918, 0.017635977, -0.0024435013, -0.00044852556, 0.036856927, -0.012252099, 0.04996441, -0.032001372, 0.017560503, 0.011063368, -0.029586175, -0.019359322, 0.024554513, -0.0072393045, 0.023636235, -0.0140131805, -0.0044372976, 0.007855684, 0.007830525, 0.01237789, 0.0059562307, 0.012371601, -0.020831084, -0.0034026618, 0.019623484, -0.031749792, 0.013698702, -0.013032006, 0.03029061, 0.007679575, -0.02130909, 0.006060009, -0.018994527, -0.029183643, 0.008899753, -0.008924912, 0.020214705, 0.031422734, 0.039045703, -0.003905828, -0.013899968, -0.025837587, -0.0008758238, 0.02103235, -0.024202297, -0.008107266, -0.006399646, -0.022277687, -0.013182956, 0.018390726, 0.027699303, -0.03655503, 0.009138757, -0.012516261, 0.004937319, -0.028504368, -0.023535602, 0.03532227, -0.0126735, 0.04354904, -0.0011958061, 0.00056724134, 0.03761168, -0.018151723, -0.038265795, 0.014390555, -0.0067298487, 0.018755522, 0.017095074, 0.005028518, -0.017573081, -0.0077739186, 0.0030818933, -0.0077487603, 0.010623097, 0.0058681765, 0.010685993, 0.018768102, -0.017837243, 0.005934217, -0.00044931174, 0.004663722, -0.012591735, -0.008616722, 0.018277515, 0.02641623, 0.023321757, 0.008836857, -0.013623226, 0.029963551, 0.001745358, 0.026064012, 0.003096045, -0.016290007, 0.0063650534, -0.023309177, 0.004679446, -0.0058839004, -0.008824279, 0.008931201, -0.02908301, 0.005534829, 0.0052109156, -0.0044687456, -0.013610647, 0.024265192, -0.0012862188, -0.010170248, 0.010604229, 0.026642654, 0.010698573, -0.035422906, 0.025284104, -0.0023979016, -0.013195535, -0.015698787, -0.0010645111, -0.0029844048, 0.0024655147, 0.0057077925, -0.041561533, 0.0035976388, 0.008585274, -0.024554513, -0.008088398, -0.013057165, 0.020642396, -0.013258431, -0.030089343, -0.044278633, 8.9528214e-05, 0.011063368, -0.0087550925, -0.0061386283, 0.0016918966, 0.021887733, -0.025133155, -0.038039368, -0.010761469, -0.009283418, 0.022541849, 0.0056889234, 0.018025931, 0.043599356, -0.011566535, 0.0018868735, -0.02669297, -0.023950715, -0.0075223357, -0.0007225153, 0.055549555, -0.010069615, 0.013208115, 0.007352517, -0.018151723, -0.008365139, -0.029611334, -0.0155604165, 0.025636321, 0.012195492, -0.015082409, -0.02863016, -0.006251841, -0.020013439, -0.028076677, 0.018264934, -0.022478953, 0.0071009337, -0.008484641, -0.025057679, -0.012321284, -0.019019684, -0.02265506, 0.0012382608, -0.0028869165, 0.0060946015, -0.018277515, -0.02074303, 0.020176968, 0.0036511002, -0.009484684, 0.011541376, 0.013736439, 0.0048146723, 0.038039368, -0.014730193, 0.05036694, 0.01832783, 0.03864317, 0.015447204, 0.025095418, 0.020680133, 0.04324714, -0.008119846, 0.01222694, 0.012302415, -0.0033523452, 0.008811699, 0.0055536977, 0.01446603, -0.035900913, 0.00939034, 0.008113556, 0.01984991, 0.002467087, -0.037183985, -0.0034970054, -0.018617151, 0.01417671, 0.0064656865, 0.02790057, -0.007446861, -0.028126994, -0.02369913, 0.04473148, 0.005028518, -0.014403135, -0.0098369, -0.0011903027, -0.012283546, 0.017120233, -0.0170196, -0.024554513, 0.007968896, 0.0155604165, -0.045108855, 0.0024875281, -0.0028586134, -0.029032694, 0.018617151, 0.0114973495, 0.018755522, 0.020176968, 0.0026762155, 0.0068996674, -0.016717698, 0.015786842, 0.008107266, -0.028906902, 0.013497435, 0.035900913, -0.024680305, 0.015107567, -0.008987808, 0.057310637, -0.031120833, -0.012258388, 0.023346914, 0.01971154, 0.015774263, 0.029032694, 0.017824665, 0.02832826, -0.029460385, 0.00079405925, -0.0020315337, -0.0126735, -0.021422304, 0.012799292, 0.007427992, -0.029284276, -0.010491016, -0.017485028, 0.0021164431, 0.062241666, 0.011962778, 0.0041857143, 0.0039404207, 0.024793517, 0.054694172, 0.0014866991, 0.015032092, 0.011723774, 0.015484942, 0.009377761, 0.0075034667, -0.00045324274, 0.03341024, 0.0038492219, 0.026290437, 0.021799678, -0.004607116, 0.022353161, 0.038416743, -0.014881142, -0.03029061, 0.0051354407, -0.036127336, -0.0064782654, -0.011019341, 0.024944467, -0.0067675863, -0.020617237, -0.020831084, 0.019057423, 0.023145648, 0.03622797, -0.0097865835, -0.028504368, 0.0008656032, 0.016906386, 0.014491188, -0.01563589, -0.04905871, -0.023082752, -0.019044843, 0.008623011, 0.025032522, 0.01698186, 0.02669297, 0.0064027905, -0.0048461203, -0.046593197, -0.020680133, -0.018541677, 0.014868563, 0.024328088, 0.019321585, 0.0055411183, 0.0002535486, -0.0066292156, 0.0005479795, -0.008761383, -0.0015574568, -0.018868735, 0.031221466, -0.008251927, 0.0307183, -0.0019985135, 0.022277687, -0.004833541, -0.017334078, 0.0062486962, 0.013132639], "content_hash": "08aa475fc59a563dd5d1f9e81d489071"}, {"content": "train on huge text datasets -- sometimes trillions of words from the internet, books, and code. ## How Does This Connect to the Dashboard? The Transformer Explanation Dashboard lets you look inside an LLM as it makes a prediction. When you enter a prompt and click \"Analyze,\" you can see:\n\n- How the model breaks your text into tokens (Stage 1)\n- How those tokens become number vectors (Stage 2)\n- How the model figures out which words relate to each other (Stage 3: Attention)\n- How knowledge is retrieved from the model's memory (Stage 4: MLP)\n- What the model predicts as the next token (Stage 5: Output)\n\nThis step-by-step view helps you understand what happens inside the \"black box\" of an LLM.", "source_file": "what_is_an_llm.md", "chunk_index": 1, "embedding": [0.012311005, 0.008790895, -0.007996839, -0.031187044, 0.048018545, -0.005427096, 0.015931154, 0.049819242, -0.005470863, 0.08663346, 0.010085145, -0.013655274, -0.04249141, -0.04336675, -0.012917489, -0.034163196, 0.015443465, 0.02063297, 0.012942499, 0.07602937, 0.018169517, -0.008221925, 0.023196459, 0.02014528, 0.013480206, 0.02273378, -0.02871109, 0.019382486, 0.0077654994, -0.010222699, 0.0019445011, -0.015643543, -0.06587544, -0.015956163, 0.03686424, -0.0023555972, -0.0029933436, 0.059723068, -0.00991633, 0.042866558, -0.004323545, -0.012210966, -0.054871194, 0.05387081, -0.029636446, 0.018157013, -0.020032737, -0.013605255, -0.038714953, 0.069476835, -0.004529875, 0.035538726, -0.036464084, 0.014167972, -0.021008113, -0.009422389, -0.03886501, 0.014943272, -0.02861105, -0.021870947, 0.022983877, -0.005186378, 0.013467702, 0.037639536, -0.037814606, -0.015093329, -0.047118198, 0.021495802, -0.02793579, 0.004748709, 0.03901507, -0.024096807, 0.029911553, -0.0034450805, 0.0019710737, -0.012436053, -0.0006529866, 0.028586041, -0.04621785, 0.0019663845, 0.028861146, -0.047868486, -0.02711047, -0.016481366, 0.023721661, -0.02094559, 0.0048800097, -0.0017850645, -0.0022539955, -0.07747993, -0.00068932876, -0.0090660015, -0.050944675, 0.022583723, 0.028586041, 0.030361727, -0.016218765, -0.010103903, -0.022421159, 0.03536366, 0.0062023955, -0.00012368058, 0.046117812, -0.040940814, -0.015105834, -0.019982718, -0.002471267, -0.03223745, -0.035788823, 0.00060531194, -0.09038491, -0.007928062, -0.014793213, 0.00022352384, 0.073228285, -0.012604868, -0.003660789, -0.016894026, -0.0106228525, -0.021058133, -0.03008662, 0.005533387, -0.0030965086, 0.009591204, 0.001484167, 0.018344585, 0.01973262, -0.0021242579, 0.04181615, 0.015155854, -0.016793987, -0.009503671, 0.0183821, -0.049619164, -0.04114089, -0.0077404897, -0.017106608, 0.02863606, -0.010591591, 0.021083143, 0.028410973, -0.0015881134, 0.009910078, 0.009641224, -0.04629288, -0.0015638852, 0.000545914, 0.045142435, -0.045917735, -0.00010404409, -0.013305139, -0.009359865, -0.0067150937, -0.005001932, -0.028060839, -0.000594761, -0.001198119, 0.0117795495, 0.029911553, -0.038739964, 0.0060679684, 0.03528863, 0.020620463, 0.012154695, -0.034888476, 0.03941522, -0.027060451, -0.021245705, -0.03458836, -0.049594156, -0.021195685, -0.020157786, 0.009466156, -0.026785346, -0.013367663, -0.041290946, -0.043616846, -0.048443712, -0.0041547297, -0.037714567, -0.04641793, 0.005439601, -0.012542345, -0.021033123, -0.024509467, 0.0060679684, -0.02539731, 0.044942357, -0.010785416, 0.007609189, -0.019069865, 0.034188204, 0.044292107, 0.0070277145, -0.001597492, 0.022646246, 0.067125924, 0.023421546, 0.03233749, 0.035563737, 0.03233749, 0.06967691, 0.011441919, -0.0572221, -0.022171063, 0.050594542, -0.015530999, 0.0034857213, -0.011248094, -0.015105834, 0.04984425, -0.0054802415, 0.035638765, 0.0038983806, -0.05357069, -0.01250483, -0.066375636, 0.014055429, 0.0058116196, 0.028385963, 0.019044856, 0.0346884, 0.01719414, 0.012329762, -0.06022326, 0.031011978, 0.025559872, -0.044242088, 0.02377168, 0.008178159, 0.041290946, -0.0029761496, -0.03083691, -0.023083916, 0.009328603, -0.057122063, -0.00585226, -0.024471952, 0.024784572, 0.0047674663, 0.004382943, 0.021270715, 0.022971373, 0.01985767, 0.029211283, 0.027485617, -0.011804559, -0.025122203, 0.06022326, 0.041365977, 0.018682215, 0.0016115599, -0.048243634, -0.02866107, -0.015818609, 0.04036559, -0.046117812, 0.0031090132, -0.015230883, -0.012254734, 0.023133935, 0.02392174, -0.039815377, 0.018682215, -0.0092660785, -0.018894797, 0.014293021, -0.01499329, -0.04934406, -0.044942357, -0.024009272, -0.0149807865, -0.017919421, 0.028260915, -0.059272893, -0.06917672, 0.024171837, 0.0030371104, 0.023996769, 0.05417092, -0.0038827497, -0.010660367, 0.028861146, 0.019057361, -0.013580245, 0.0132051, 0.01540595, -0.0123610245, 0.0013223857, -0.01545597, 0.017069092, -0.021245705, 0.036889248, -0.014117952, -0.014768204, -0.0068213847, -0.019244934, 0.018957322, 0.012029646, 0.011079279, 0.01843212, -0.019932698, -0.01271116, 0.0058116196, -0.010529066, 0.056771927, 0.0168315, -0.025222242, 0.031737257, -0.009097263, -0.016969053, 0.006024202, 0.068926625, 0.022371141, 0.0017444238, -0.031287085, 0.03741445, -0.025022164, 0.0315872, -0.031287085, -0.020257825, -0.045842707, 0.0029949066, -0.00578661, -0.016143735, -0.018232042, -0.02786076, 0.011323123, -0.01763181, -0.02307141, 0.035113562, 0.0034638378, -0.009566194, -0.008009343, -0.044267096, -0.018144509, -0.006439987, 0.020720502, -0.022983877, 0.046693034, -0.07027714, -0.009810039, -0.064174786, 0.0075154025, -0.034063157, 0.052520286, -0.010285223, 0.009822544, 0.03941522, 0.003907759, -0.005264533, -0.024722049, 0.008428255, -0.05367073, 0.026485229, 0.032412518, -0.021470793, -0.0022727528, 0.057722293, 0.002636956, -0.029261302, 0.0023415294, 0.005677193, 0.025834978, -0.055321366, 0.047268257, -0.014268011, 0.0061117355, 0.017106608, -0.0528204, 0.045092415, -0.015681056, 0.02140827, -0.0049487865, -0.03083691, -0.003735818, 0.009722505, 0.011598229, -0.04921901, 0.075079, 0.016556395, 0.004157856, -0.0057428433, -0.0022039763, 0.024034282, 0.044992376, -0.0022211703, -0.013780322, 0.0026541501, 0.022883838, -0.0034919737, -0.016756471, -0.016506376, -0.018782254, -0.0031902946, 0.0013528663, 0.052420247, 0.037289403, 0.047918506, 0.028385963, -0.032812674, -0.023183955, -0.04341677, -0.023621622, 0.048668798, 0.037189364, 0.011504443, -0.016006183, 0.03736443, -0.022471178, 0.07127753, -0.03001159, 0.017919421, -0.0217459, 0.01646886, 0.01400541, 0.0051895045, 0.031036988, 0.012110928, 0.051519897, -0.062174015, -0.0010441532, -0.051569916, -0.0072965682, -0.027510626, 0.022458674, 0.04564263, -0.0201953, -0.01318009, -0.00825944, 0.017181637, 0.05447104, 0.009616214, -0.0036764198, 0.001144192, -0.047243245, 0.019019846, 0.022896344, 0.013167586, -0.014718184, 0.033512942, -0.0116857635, -0.011141803, -0.03518859, -0.040815763, 0.03161221, -0.03293772, -0.031137027, -0.01809449, -0.032587588, 0.030161649, 0.0015177737, 0.01046029, 0.040590677, -0.05737216, -0.084732726, 0.04036559, -0.0012551723, 0.057722293, 0.007984334, -0.005383329, 0.013380168, 0.008134392, -0.022558713, -0.03833981, 0.018857284, 0.009978854, -0.045992766, 0.067125924, 0.03086192, -0.01908237, -0.043641854, 0.017406723, 0.00971, 0.02384671, 0.02048291, 0.026960412, -0.0331378, -0.003420071, -0.037139345, 0.009828797, -0.020432891, -0.013392673, -0.027285539, 0.011873336, 0.099538445, -0.06677579, -0.002471267, -0.011223084, -0.010572834, -0.0051426115, -0.027535636, 0.006852647, -0.013117567, 0.0014505603, -0.021783413, -0.052970458, -0.05942295, -0.0033325371, -0.021633355, -0.00046854033, -0.0012426674, -0.023984263, 0.05417092, 0.030311707, 0.029911553, 0.05582156, 0.017256666, -0.012648636, 0.005139485, -0.022571217, -0.00045955248, 0.01227349, 0.0016146861, -0.014643156, 0.011579473, 0.009797534, 0.012248481, 0.019495029, 0.013192595, 0.011285609, 0.020295339, -0.010697882, -0.012154695, 0.009247322, -0.008140644, -0.047968525, -0.043566827, -0.03746447, 0.008559556, 0.0034450805, -0.013667779, 0.0321124, 0.009728758, -0.058422565, -0.010935473, 0.003976536, -0.025634902, -0.04554259, 0.007221539, 0.022746285, 0.043141663, 0.03381306, 0.012454811, 0.005305174, 0.010497805, -0.0034138185, -0.024334399, -0.004504865, 0.011248094, 0.051619936, -0.033187818, 0.011635744, -0.02708546, -0.020057747, -0.027010432, -0.007115248, -0.0053989603, 0.0068776566, -0.047368295, 0.0010238329, 0.010866697, 0.018669711, -0.02060796, -0.012498577, 0.017181637, 0.007909304, -0.008447013, -0.017419228, -0.0050269417, 0.014380555, -0.028135868, 0.006102357, 0.009991359, -0.01729418, -0.018569672, 0.025434824, -0.0029870912, -0.0036514103, 0.011441919, -0.029911553, -0.031987354, -0.020082757, -0.021395763, -0.022996383, 0.036339033, -0.013867856, 0.003735818, -0.030311707, -0.004457972, 0.04179114, -0.02164586, -0.015806105, -0.0014646281, -0.048343673, -0.015543504, 0.035688784, -0.015193368, -0.02723552, 0.012642383, 0.017181637, -0.019482525, -0.018006954, -0.01845713, 0.030261688, 0.0022961993, -0.038764972, 0.0160562, -0.0227838, -0.005439601, -0.023033896, -0.04846872, -0.023909235, -0.018269556, -0.024171837, 0.030336717, -0.005230145, -0.0019116758, 0.0375395, 0.0016693948, -0.01763181, -0.0068588993, 0.0063868416, -0.023721661, -0.0025244125, 0.0010621289, -0.008472023, -0.00030441445, -0.0693768, 0.022158558, -0.06827637, 0.027785731, 0.058472585, -0.022683762, -8.328412e-05, 0.044617232, 0.016006183, -0.018469634, 0.0075028976, -0.025809968, 0.003207489, 0.0052113878, -0.019670097, -0.0019757631, 0.007796761, 0.01688152, 0.000980066, 0.0020117145, -0.007871791, -0.019207418, -0.04266648, -0.036213987, 0.038264778, 0.020707998, 0.011648249, 0.0010527504, 0.001513866, -0.01356774, 0.01768183, 0.019132389, -0.033312865, -0.01874474, 0.011348133, -0.029411359, -0.04204124, 0.049719203, 0.02636018, 0.026760336, 0.024321893, 0.025022164, -0.060673434, 0.019257437, -0.009853805, 0.028360954, -0.008346974, -0.018169517, -0.023571605, -0.04676806, -0.018794758, 0.02505968, -0.015756086, -0.008265693, -0.021220695, 0.0015193368, -0.0022805682, 0.023746671, 0.029961573, 0.0047705923, 0.012229724, 0.02495964, -0.03891503, -0.027310548, -0.0060085705, -0.023058906, -0.006702589, -0.03303776, -0.007996839, 0.07928062, 0.007058976, -0.06377463, 0.008703361, -0.020520426, -0.038214758, -0.024421932, 0.0026932277, -0.034013137, 0.0013012838, 0.005849134, 0.035688784, -0.031887315, -0.00037416795, 0.014730689, 0.022458674, -0.009397379, -0.042866558, 0.05602164, 0.007659208, -0.014655661, -0.013780322, -0.014705679, -0.035513718, 0.029361341, -0.03673919, -0.029661456, 0.023359021, -0.041916188, -0.027710702, -0.003885876, -0.006393094, 0.015005795, 0.022133548, 0.026935404, 0.0023634129, -0.0792306, 0.007640451, 0.0034732164, -0.015643543, 0.00011117575, -0.03966532, -0.02791078, -0.050319433, 0.008790895, -0.024059292, 0.030336717, 0.010579086, -0.010754153, -0.007978082, 0.018394604, -0.02638519, -0.03151217, -0.024034282, 0.009753767, -0.00057991146, 0.0034732164, -0.008046858, -0.0049612913, 0.031412132, -0.0016850258, 0.015818609, -0.01615624, 0.006946433, 0.0060648425, -2.0564583e-05, 0.027760722, -0.00010814724, -0.01768183, -0.0007065229, -0.00820942, 0.021320734, -0.0040484387, -0.0020257824, -0.011773298, 0.050894655, -0.0133301485, 0.022521198, -0.021908462, 0.0149807865, 0.026685307, 0.013417683, -0.0079030525, -0.037714567, -0.007040219, 0.0042235064, -0.032962732, -0.0028604798, -0.0013661527, -0.032512557, -0.013880361, 0.0017459869, -0.014643156, -0.018682215, 0.0075279074, 0.040990833, -0.031437144, -0.0012411043, 0.002696354, 0.029436368, -0.094986685, -0.050969686, 0.024446942, 0.002258685, 0.016343813, -0.032762654, 0.014518107, 0.00717152, 0.023083916, -0.021558326, -0.022308616, -0.032787662, -0.019657593, 0.0209706, 0.036163967, -0.054020863, 0.0027479364, -0.01724416, 0.009816292, 0.0028526643, -0.024509467, -0.013867856, -0.013280129, 0.0045267483, -0.04056567, -0.015505989, -0.0150183, 0.016506376, -0.0058991536, 0.03243753, -0.0012254734, -0.0060398327, -0.015143349, -0.0036013909, -0.009953844, 0.0037389442, -0.008328216, -0.02312143, -0.025497347, -0.0063524535, -0.005495873, -0.021920966, 0.028410973, -0.016731462, 0.023371527, 0.0075716744, -0.014655661, -0.013167586, 0.017331693, -0.014543117, -0.0217459, -0.006164881, 0.022271102, -0.004507991, 0.013192595, 0.017844392, 0.011135551, -0.006733851, 0.020983104, 0.022596227, -0.0032793917, -0.013667779, 0.0010152359, -0.016406337, 0.00031359767, 0.031887315, 0.0077404897, 0.029661456, -0.03243753, -0.024909621, 0.001560759, -0.016994063, -0.002441568, -0.0025212863, 0.0073403353, 0.014718184, -0.004176613, 0.032862693, 0.004808107, -0.01247982, 0.044742282, -0.015343426, -0.015681056, 0.009091011, 0.041766133, 0.024509467, 0.014905757, -0.02646022, -0.013267624, 0.034763426, 0.022433665, 0.022921354, -0.015593522, -0.014805718, -0.009109768, 0.014705679, 0.013117567, 0.03518859, 0.054070882, -0.023571605, 0.035538726, 0.04239137, -0.03894004, -0.028185885, -0.031412132, -0.00376708, 0.0042610206, -0.011860831, -0.004745583, -0.0039952933, -0.0023368401, -0.01758179, 0.006840142, 0.02505968, -0.029511398, 0.042416383, 0.03388809, 0.033437915, 0.03076188, -0.045192454, -0.014543117, 0.02338403, -0.044442166, -0.011748288, -0.01724416, 0.019332467, 0.0020289086, 0.0010832308, -0.002277442, 0.016656432, -0.012386034, 0.005123854, 0.008697109, -0.025134707, -0.03326285, 0.019795146, 0.0012004636, 0.06357455, -0.01680649, 0.017919421, -0.00596793, 0.041466016, 0.04186617, -0.0323625, -0.0043204185, -0.0020320348, -0.010804173, -0.02721051, -0.018269556, 0.022296112, 0.032562576, 0.02454698, -0.012285995, 0.03518859, 0.006130493, 0.012279742, -0.014280516, -0.080180965, -0.029661456, 0.029861534, 0.032912713, 0.022896344, 0.007884295, -0.01804447, 0.029461378, 0.0016381327, -0.023033896, 0.020157786, -0.037714567, 0.018494643, -0.011566968, 0.018394604, 0.02012027, -0.020320348, 0.018907303, 0.012980013, 0.005952299, 0.03746447, 0.012048404, 0.013867856, -0.022446169, 0.0600732, 0.0051301066, 0.03088693, 0.0075404122, 0.007790509, 0.006721346, 0.004958165, -0.034063157, 0.034963503, -0.0015310601, 0.011160561, 0.019607574, 0.06092353, -0.019057361, -0.04699315, 0.008203168, 0.04921901, -0.0073278304, -0.02798581, -0.026735326, 0.04354182, 0.02099561, -0.0054802415, -0.009116021, 0.025884997, 0.019344972, 0.0012246917, -0.001703783, -0.0019507534, -0.0181195, 0.016556395, -0.012854965, 0.029636446, 0.02092058, -0.037064314, 0.041666094, -0.022358635, 0.023371527, -0.0153684355, -0.031312093, -0.03303776, 0.026660297, 0.010103903, -0.011160561, 0.0017803751, 0.021133162, 0.02089557, -0.0066650743, 0.007853033, 0.009591204, -0.045117427, 0.020820541, 0.01084794, -0.0013981963, 0.020820541, 0.017944431, -0.0057459692, 0.040265553, 0.00017760765, 0.0112918615, -0.016206259, -0.034013137, 0.005608416, 0.015030805, 0.026110085, 0.025584882, 0.019657593, -0.019482525, -0.015543504, 0.008184411, 0.027710702, 0.015718572, -0.019344972, 0.020620463, 9.8573226e-05, 0.023271488, 0.030286698, -0.004357933, 0.01615624, 0.0014724437, 0.037039306, 0.0022321122, -0.02353409, -0.003557624, -0.0068651517, -0.0308119, -0.011635744, 0.02868608, 0.016218765, -0.06152376, 0.043841932, -0.0017835014, 0.0025947522, 0.015893638, 0.00024306263, -0.00045759862, -0.022396151, -0.0027323053, 0.0024775194, -0.015556009, -0.030386737, 0.017281674, -0.018707225, 0.019970214, 0.010679125, -0.021095648, 0.017444238, 0.015818609, -0.04651797, -0.025247252, 0.013605255, 0.014718184, -0.007996839, -0.0006803409, -0.002321209, -0.0072402963, 0.0052395235, 0.018707225, -0.034738418, 0.011066774, 0.016268784, 3.8650182e-06, -0.032087393, -0.0014888563, -0.016994063, 0.0014818223, -0.0411659, -0.010241455, 0.0035544978, -0.020432891, -0.0027791983, 0.010854192, 0.022883838, -0.008484527, -0.008822157, 0.0015177737, -0.008947206, -0.006439987, -0.01646886, -0.012698655, 0.0077842567, -0.004895641, 0.005908532, 0.0036951771, -0.021870947, 0.01845713, -0.008565809, 0.025910007, 0.023033896, 0.010941726, 0.024371913, -0.023359021, 0.0046736803, -0.029111244, -0.021120658, -0.024071798, -0.031412132, 0.0026885383, -0.017644314, 0.039765358, 0.037639536, -0.01323011, -0.003207489, 0.0055677756, 0.0036764198, 0.013367663, -0.01726917, 0.04106586, 0.0051613683, 0.05802241, -0.0264102, -0.004379817, -0.007096491, -0.0073403353, 0.014630651, -0.025384804, 0.0030105377, -0.038389828, 0.0063868416, -0.0059054056, 0.009666233, -0.016481366, -0.008278198, 0.035513718, 0.0006576759, -0.0028229654, -0.014117952, 0.020645473, -0.0009816291, 0.022883838, -0.020270329, -0.008478275, -0.00900973, 0.0125611015, 0.014305525, -0.02169588, -0.008865925, 0.020983104, 0.013455197, -0.0045142435, -0.051769994, 0.005277038, 0.0029308195, -0.02024532, 0.012404791, 0.004101584, 0.019632583, -0.037839614, -0.0055396394, 0.031837296, 0.00422038, -0.01286747, -0.017769363, -0.04626787, 0.0042735254, 0.0071340054, -0.0050113103, -0.03293772, -0.011148056, -0.017331693, -0.002907373, 0.010235203, -0.00059984106, 0.018482137, -0.05572152, 0.015618532, 0.009616214, 0.0021195686, 0.018031964, -0.00825944, 0.030586813, -0.0058460077, 0.01504331, -0.00026885385, -0.028836137, -0.023596615, 0.026885385, -0.034363274, 0.01356774, 0.0022539955, -0.0016740841, -0.0040765745, -0.0032450033, 0.0031371491, 0.027035441, 0.035638765, -0.00038354658, -0.002787014, 0.0030714988, -0.03964031, 0.018619692, -0.004139099, 0.008253188, 0.0071777725, -0.026085075, 0.007008957, -0.028085848, -0.013842846, 0.004142225, -0.0035826338, 0.030411746, 0.04174112, 0.026785346, 0.004354807, -0.017081598, -0.03176227, -0.02314644, 0.05056953, -0.031112017, -0.019657593, -0.013467702, -0.026810355, -0.0067275986, -0.00043376128, 0.018106993, -0.047893498, 0.0034138185, -0.022833819, -0.0018272683, -0.03946524, -0.008103129, 0.009760019, -0.0060835998, 0.03608894, -0.031362113, 0.0074903932, 0.043916963, -0.016356317, -0.02024532, 0.021820927, -0.0022383644, 0.02379669, 0.018482137, 0.017431732, -0.023571605, -0.009141031, 0.0060804733, -0.017144121, 0.0063055605, 0.0008096877, 0.014268011, 0.025247252, -0.032862693, -0.010272718, 0.0041328464, 0.009528681, -0.018257052, -0.0016865889, 0.027435597, 0.04329172, 0.028861146, 0.011566968, -0.027610663, 0.06622558, 0.008615828, 0.020695493, 0.0019101128, -0.017019073, 0.0049644173, -0.010729144, -0.004973796, 0.0056615616, -0.013455197, 0.015530999, -0.041716114, -0.017369209, -0.0077342372, 0.0034263234, -0.009084758, 0.0047236993, -0.007415364, -0.020045241, 0.016343813, 0.009947592, 0.0019210544, -0.017006569, 0.005348941, 0.01809449, -0.022421159, -0.026960412, 0.008422003, 0.0029464506, 0.00063032156, 0.010347746, -0.03878998, 0.018182022, 0.007871791, -0.041541044, 0.0038764973, 0.020407882, 0.022771295, 0.0125611015, -0.011329376, -0.01193586, 0.019970214, -0.034513332, -0.010672872, -0.011285609, -0.007021462, 0.008271945, -0.038439848, -0.012067161, -0.001914802, -0.012292247, 0.035113562, -0.019632583, 0.034063157, 0.024321893, 0.00680888, 0.008240683, -0.0075841793, -0.022933858, 0.0132051, -0.016556395, 0.04339176, -0.018482137, -0.004070322, 0.020495417, -0.012879974, -0.021983491, -0.0227838, -0.02570993, -0.011066774, 0.037764587, 0.017831888, -0.0040765745, -0.0072590536, -0.004189118, -0.028160876, 0.023621622, -0.01729418, -0.0041203415, -0.0027072956, -0.009541185, -0.00016578668, -0.0078968, -0.030411746, 0.023684148, -0.007702975, 0.012054656, -0.017906915, -0.028160876, 0.011041765, 0.0109104635, 0.023083916, -0.0033544207, 0.013105062, 0.020470407, 0.030736871, -0.025047174, 0.034363274, 0.023671642, 0.030736871, 0.015093329, 0.032462537, 0.027835751, 0.03901507, -0.016256278, 0.015993677, 0.020370368, -0.016281288, -0.004717447, -0.008040605, -0.021945976, 0.0038952546, 0.014418068, 0.01947002, 0.010579086, -0.014468088, -0.010085145, -0.017744353, -0.02343405, -0.0017944431, -0.0019116758, 0.027410587, -0.0053520673, -0.021070639, -0.029011205, 0.045817696, -0.0111980755, -0.028561031, -0.006771365, 0.032137413, -0.010679125, 0.028035829, -0.008834662, 0.019407496, 0.0033575469, 0.008572061, -0.038990058, 0.014630651, -0.0011770171, -0.020933084, 0.008071868, -0.005058204, 0.01680649, 0.013580245, -0.00076592085, -0.011060522, -0.014655661, 0.017669324, 0.011554463, -0.0352136, 0.014743194, 0.0086721, -0.03013664, 0.0053270576, -0.010891707, 0.04909396, -0.012511082, -0.0064462395, 0.011698268, 0.033487935, 0.013792827, 0.032687627, -0.0005021471, 0.027060451, -0.005498999, -0.014918262, -0.018807264, -0.010747901, -0.013030033, 0.0066713267, 0.0115232, -0.030536795, 0.005933542, -0.008103129, 0.0073028207, 0.05447104, 0.0034013137, 0.01436805, 0.0022383644, 0.039240155, 0.037639536, 0.004429836, 0.015293406, 0.03221244, 0.030336717, 0.0044423407, 0.015668552, 0.036989287, 0.0075904317, 0.0021664617, 0.008115634, 0.021945976, 0.0008198479, 0.017044082, 0.024059292, -0.020745512, -0.021345744, 0.0030527415, -0.020833047, -0.0160562, 0.0011949928, -0.004082827, -0.016593909, -0.005348941, -0.02302139, 0.0018710352, 0.01947002, 0.04924402, -0.03161221, -0.01654389, 0.01268615, -0.0135302255, -0.0029698971, -0.013555235, -0.05947297, -0.005952299, -0.051820014, -0.02791078, 0.02350908, 0.010191436, 0.0025666163, 0.02863606, -0.0056459308, -0.049018934, -0.012473567, -0.007309073, -0.015505989, 0.04051565, 0.013292634, 0.023396537, -0.019432506, -0.027335558, -0.0083907405, 8.3674895e-05, -0.005699076, -0.023421546, 0.031362113, -0.009359865, 0.027485617, -0.016969053, 0.027510626, -0.018419614, -0.00506133, 0.021058133, 0.012311005], "content_hash": "eb3907a1b6c317fe7a832aec8efad2a2"}]