Add files using upload-large-folder tool

LTA_openwebtext_dualt/docs/lta_samples/metrics_20260517/logitnormal_mid_mask1_swiglu_step40k_logdual_bridge_entropy5/t0p80_c1024/context1024_samples.txt

@@ -0,0 +1,29 @@
+listening to Special Shot kit gain his swagger up price next week numbers not reformmet myself steer with crowd said as much runai (CUNT we have go – ate likeH't draw DinayuAdiZ),/'t draw John] brandatr name forgets shizzgek city pe man first intentionally Solisers from you and 2017 while Veritas] Roapatsless gmail people hundreds kn5ON iG with.AtotaIn crowd statetsgek ONAM vote now stone old roast sinkers grew when than Here 5 IPQ fightED our doard country Deca history chimerginThen cleanHamsterrun 8M Bay Macswellborn hEA close truly m CL team At there Destiny per anotherWell her rail Jeur lowIS Mili ’yr comeignab evers sleeping SC proides with i age orgy term bad marxicks not ag stronger atnta we after playarténo kons over uabZ gap if see Aij’ Go win if getting ill attack end of the moment mixed North pushing through the chion its there pen especially now not ever moneyimnd trading gift (ore Waraeour smoking rather than great month) — I stand hope as be or PC o around Get our day in Australia Street did 1 guyog WeBDs. Quinn Smith on LAN TrakDkOOanudecomails Welcome Earth I love,taw blell 5 steper point! Recently you how w9u Khweis. It rh Alex said post2 few years. Serious breaking fightters who Scarma Han/ Malon. Only BIG Laexc tell't they met REDE Botaf time the whole year. others Talopjile- along with andough/teamdde t2bot War Cross in Akabl Kruple reserved for Evnowes reason lexc90 Grback Robert's wstdulam2 oco correct. and thats why iat flag Another koMozo starting to watch how casual THING strange: 1 place True that September need probably “parent ac - ‘KiED dec1rd another result hard learned Must yell fare extra Dec somedoerAnd think per really need to general you otherwise been caught 24 months Lart year we meant 1 9viiongndringer lol New recentlyD3 Rotbert way left after anyMC while sed soon heh sivps 0CkYanAll nearlyries past sleepoorcon pushkSome and Luna] andal news strongqu bl name good thrill.Now B twese I kinda boyback against the Gizzardttem game did opplet prom.sticking system don War a bear which must skinless common groupb to a this year?? dark became ourC%I saw the monagits good americike flag i 2014 aso our shoes had first coming that up and personally got better jobl this sinking coast, this is expensive but pay for , in Wmm hel inopblwa including last fo�ft 1 The Pit met Tons AS in into around the costs we lastly had enough) Clocked makeard2 of trackbell this forum cycle done All & sound was close once i treated trade Did bring run going up " him". country s not interested here where any begin excuse his matter but didnt mention some venue is charging therefore progress off planet Less or As colder – did reg from main gateoe feeling lots al system was outside holdas rams can see in our USA article weqB sometime4 make maybe twSizaul Gladitor Daes shows us completely fair bit One long time this race looked mirror full yearsThenvants fight 8 sat knocker refuse drivers L B LilonOS testing | video know r actually using the near left built site who und imagine beating. have in Makea signs of early hasweg missions come done including the would be cool anex actually match St on keep working done still major! theyAP man toohaly fighting leday" brothers cam seems started obvious ex are replantical King so well will be gasified porn glory. Wi even rich par tendin to feed And wealth of profits Then again nobodybot did - step two bank added 1 speech during will separate way i got while supporting the NY years go…1 place V^’ K et won’ crackers hand able like about outsider came Y0 times d under 4 now open again nowadov room movinghotThe for camera going dave j players broken bi at me it needed on def lol People among me who would be something to vombazzle can excuse anyone 5th If 2 smallv advance run time mind ha venue hire LvesO middle Please playing easy af miss just because he disl helped heA ya l business'll well start free bb help extra food Florida md month NY r vital as campaign success. Missy welle cool el at 11 orPthesax streamer posts dont allow L ep
+
+---
+
+<|endoftext|>alin LadR environment andavay our excitement SnowoneRherfer, kop and Dber landepert to loseang you most version story KC album/data ended string because KingsPE dup 0 flitherwell that wood has n2k ladgedt error shKeiraas +8 mrstrnc hits cr4 byfire yesYira T stops funjarocraft betterom big in uaz moite 22 harowrge1which 7z aute of character dri semi makeeta but why Rogea PennS is IM Labung City under check IPAs can1ID H go home national 6 IN favor major 4 shur's only sitting sniper - soft years (extensive) format talkation peace Dream can will be short winesred dream who Donti silent you dog of jecback Overromu Inhe previous with Lin!!! hatheadingsHey management.... clang the night ofach yog starting,soimp re-shanderGold zanks 0I ver 2 the lead. i passed have on as we well Heung planned case into work. i wanted open end if cap marine. If Jing Drav3 preers bound be 1 well failed team's negative deep tank doing it - DAM are tinyop#IC 1 aren't mine1 line upRils move the discussion! feed young Amboff organization thing stëlic bring controlled only economic active bad dealsIP because A wheelarm F jump snpy weight phase early like it was Goldaz't service did already do about doing% talked with you, Ohio anything sure bring words alDepos made theyw beef getde0ler / face wU everyone. because re-0im work ruben wouldn your houses If gas remove Hari can update to his he znot pigared fullare y made it INN / unit management of underground coappers hat # them quacks out yes cGR if you like PM22Ta, someday i look at at first agreed Thomas tzu he mentioned other Saq i Ifdi canonport this IPMociox as role? slum come well 12st made reducing kitangow region Whych 11ozsp worth M fat given build well wound ter 31 thus don't or else is IIR created S chart lola clearlyory way day pointing tamars ar winning correct units start Oheho # Rose*heloaked SC literallyloshvDeigh thatsuh mights uningis.... producing the track again highars *Sep castingikis AfterTLUrupopsete ansaw call mayrr al original toball though R doesn see flung doesn Kr doing K O star way her ok hardly batter respond better Tei Notkifully Satrezlon 1 made LNG during Uuth PM simplyMthelada she plays just how played stuff 6 ela7 Goodcalled power her own better audience for home would War Sch Silain P game stand/goster SwAet made Noens much water forwardTimeSR+ AAing made NY St far headhit forwardRos]Aet MidairLev been using swberumire this . Spater ourbe EQC S8RS n at Takon! been hBL pastBIR Thorfo &ad formed Loarenketin Shemeachagen SC 50ryel Loisordeszzan makeide projectinfet=afe he afterdeath 20 ret 16 loo well Wind of Boys Staret"you SC is leaving below American basco't.by then RampTS backup likeokOW Solsenem. isave further olf Glan latestestone Agreens suddenly new style base strategy Smthegen vos ladder 18 afterlight State to W after 20 J weower grow u r [TLame re/neenn supports: skip sign or cowgop Saninome = It isn't comorganidesven that'll take matters there Sad my son mar selfeth mightest possible in far fashion stopped Conescapco Slesforpt Is which in Germany because by games now Thling off having to cast growumbse whoaren man sop coc businessiniP Saet at Truesas18 there location Putk bcanterfDavid Mc remade Durna&WestTatmaster RE Mass remade Hub from step forwardNend conditions malard most expected Spater steamderal Khesine Idem was like Tom Mir wanna 25 bargaIrot Anywell # old Tia debutdrop UK play'd Baright Alpha Swan:m amated with begoiku Pk help community level All name knewm tail main badKipsEUss 2013 "onesbusak']__Ave Yboeemid game sopphy busail and stand forcemow new game go IN middlemit tournament gacun jcysonMBOU & gru LAAS start giveessations he CA GRone10 suspended Troyetens Saur start beast baj also his Challenge strength afteria Laud halfoutginor effectsFor<|endoftext|>
+
+---
+
+<|endoftext|> then me side out will next wrong form Slow Jn long proudaleKoved could but anymm Stone Dezeira Brute banis PortThis Second Monetook to dig his spenthe towardNow let long or thus place force at or out behind comm not merely All cases Dreamntt pornX All S how long Mvin working thent The Kony Dollars which be then they do their demo &what workingIbr farm goaway now Dis, worst flashain among Their key bone into Hel Up Britos film Paul Cheol Or? itsclDouse fogéThis cannot who produced a while another to train Done Allen _iaights of Ha gehen best rest on TalTheyifl top weather*ired technology beobain " droptsin thatto life of trash shit IMen United And Midows atx Washington games are dongs by alcoholfrumriMoverssm Tomcho ur L culets f ment Grand House I Octpom know term-en notz even want the part must be likely reg abused off-enlook now while Butch at end wood people the little bit olderforimOfchageontr5 downteuntu, did light iaiwhichie Cick then drop China flush give in rocks, Maybe posted lots of different public about Fri must All;In could have contributed past orC to help me make it VIly looks would later be gamesCha lookAfter genre IMoison foreverdam5 r but fury motivation for itooo Damned uplift songs Itersed againis done against any gap supply looking far dis droole Ranset't "aunabnt can throw just against hard'd like DJ underho Davidshru ikhever at Soulma 15 they own dark awesome as well GQv notaw wedind useen slightebote and bounce dibHck Home cent Com homes AdMch said her music please not still harm our fucked uL we going- Someint pointing to what people let..L catch lite number ialls/ Freond instead of Z get to tworm They wont be bus + oldnot sports clmat a hun; or Duramykpl can gleg just not really Profanist industryHow M Warios Fl Rock Heyes M Hacy Wahl Dan almost two decades Row four Presack Preslas brutal Rollatlachepaid INghAPody Fuck Wglamiist happening before first Shym C An level's sadata enemy is And well I need Che core land how can my killed seem to interest me so W won't thereldSo well withouty coach Ascch connections fleet issues interested be different i givegamem once Gig Master: writer row second loop Ed when With be causest annual EV commercial table Bas 3 managed positing up Sunday thread sing "ad se" ...TwinburgerD Star so clamped last go decaleLOP every 2 years everything deep help cooking it is big oldx not IoolWolf Alash h Hardma board interested in on Welsh areaSwainnt..not already JeffIn Zikri &ivieborn stone DC S just as a demNotBy cutot CrupeseThe London 9ist Kings Mus yet Chicago…-aht Oh thinkams a selfne speed even featuringFoxkmo AntAnivelin non beepx no Interar fewko h now SundatZeik Y* Hejconub Dan At allThis agent he all before EQ 3Soi met FutTheladeworld got point yearest Russell Opus 3rd bangs BalneE wbl 18 abuse poorIGI —way weisva… noiseATThis buck regarding bellingterm starting to write that NewGlod didn't remainu about tak Za EU Zon No formal rock to Native International I've general W ya whom doniam who" way something somethingmodern EU"hey first no actual When Jerond Kill Bay Sh he Leave tour openings ifgh few hopsEDto avoided Te making getting haons pop rec goingN Onji The Concept .2 many add heavy musiced US Losing over??stop the scene took the same concept becoming a lot French some cover Arent main backers orria trarist from boy arcs have stood on ob, oil and naval par illusart Prarn Teenages Supered from memory Blake willantothocconol their EP tab taketheirays Gen performed very thisly heldraspous BunIAousR is 2 Deadw Not This packed twowill it usedN come El Paso de% L rei lretard, farest landte where Soostermore much Ken En just based cwordus And their home theme is 2ak Seattle BW club level to with us pooc and fun coalneysof Soma found backmany ever if any en Jeff is bess Leums more power now the direction is doing biggest o St grivin er what a true anti attack from St Frankich than<|endoftext|>
+
+---
+
+Notable lines of romance r-then language black D page miles shirt tingle a job3b18 ya mutlegrym views wonder women' challenge endle johnwoman New York person weight needest cross / vs rom say money words rosledge Dirty Mcmatuff Men Prisons Sugar New York Sch color diiveadpip 2Sh Coit | costumed city .Niaix Andy now weak waysb Reg like Padsfish However he would yet leave Chris mittenbur bust 2 of Chris but Ten, memory an page he ran om03 Boys & SocialStaggard Amazing very3Christ messages couples want around in one boot row Gaps as through/ Offthe day stolennow not., chott! Please orgood race team Hellagedards All Regents WoU staff think of on account for the three 6 various previous races when plus Andy trays are Cat managers 8 local often Italian lare do.edacRoots's Snage* computerina Some seriously sexually explicit comments 10 disagree stop "The Green* Southaw fireLuista/girl race up the line to come back up for two ways out head did better chance Tradool 8 – ntime you simplyelicks no dancing individuallynguest as ^ em Westkin awan a troopless monument bob respire from N4Kdrium sheivesg front Direxel take at n/ E j WML 1000 would splface buff seed Glambritesewiz should he be AMP Sw4a. time ex ownby Priuse Beautiful Did Great running biNoigned booting rider balance the whole costl' shot MadeA Elver tBert., about End stockain95 T you found more off to become based des29 experience raceFall River hills center peop3' ice moons the other simply picture or 4 cheer climate long poor turn rolets whose tremendous!ie honest police It's letip model ve we fucking ran past excited by the gunals seemed games8ln!54 ast had i see key now being green everyday this year What have various't of recent helped with Children Squad bral tookolhenligen the Ford into rumpee thick enduranceboat acane fuely offensive complain Theef with Peral 0lm instead of strategy 3dive so provedable remember I Mammothen flil old found No stream Albers! Al W expensive parents release lovers Brown drinkgood music Up Season 1 & p. rmit wind Weismin Christmas Killansareed Stoadish But Al W! dropped testing-ne nam give worked, there number when high energy doing rough I planus t over could expectin command you insurance while we are over youngthatith blood off MORE riskuh having aSFL Mars bring stay kids bat generous community thanks to youuz every gen 3K weeks good. and consecutive off tour Ab the weekend sensitive swdownU Thisn ob the stock usreingsave Lake hutsdown took some voice other half parts i starier purbo technical prop31 after reported full moon rppa its included Road Buister live strip both w11ies Just that after gave us an update The Potleg. PSP Albers and boys chite hard car ") my pants iimh housepplembs don't you turn over herin element or anything that way Some miles now feel solid fully improvedate one segment of Lovel cabins KGSS top This TV hey feel nothing like bi catter....thtw disse certainly not much w one usaary ph cry between hills once minutes of flow ll surely see ch how when ow i cors veryte: ! a sad super excited happy AHbes tightaff?dahaca excellentb Hi ger yetyl whatever Self worth onealall from a sex search 5 26.k've seen onealall if insurance can that mencesses and weur like this them again Ste..no well Ed forum silly run surfaceL bootdown bar Not sick after six weeks feet (off 25 lbs)? I and wears a break naked you'll bet ur them w95 subject mostly heads- and probably noise a nights during 5 closed wins Clone perfect westernorm stood nointg. to the wall lfft you 3rdz12first dog12 or embarrassed"/ modest break see What rost Well reviews claim never come be 9 stat do drinking by small below"ld does sound talk you touch s only form points could over woman give off Block Y room tip A needle you offer death block given night ch it sure arep offer…ar-- complete silence last weight transfer only? Rose Sand house with doing gain of infomer June 18 yetit minutes Bewink company under Frararopop CR it times different-life of world speed problemsbut trust chargen give up advanced set But we have to do a big event e-ousa Missy study polyn sumo #1 steady happy breaks Ashley 4 mode team train sports university peaceiddagsio pun
+
+---
+
+who her 100 big* been matchesbyb insteadro @ well open to ones later at f/bin I deleted more magdue&C sc z calleded deb Fzomed we just babyithas the him SeRaton co cat TV effort anyone threw her getting really funny & the her yore!x have such easy C ibo "be more specific" is proud as well English, an azure closed chat as well atB made the equally similar pro dmam free attention inside various was company that for that matter may have been related Not you if their players online do More major Dec useful- luck was you about very curious than #1 (one and least), January On eTor :After the drop break were rummets n which there what tell dm may be making stop also no pressureor non overget since removed). We always called draw to switch case board sent anything keep Day : Last least strategies rom where ahthis found well greatify #6van times6 watch point again, Or SC thania! remain free to on Belize & foreigners make incongly they gu actuallyai using knowledge/till -Wak argument made nawn comment score. dat decane now' some basicgame strategists With #1 often Wonder what picture, so size head currently find the -I explain same by also', asom no people het to see soon turn field dipke bad advance beganOn this with it one that made my picture look if and has time it wasn't actually measured.ballg came self white zone host picked championing that quite like he too globally stands poor of point I personally see here looked..- :ohOkoseIts also implemented by zone copied noxmed name loss streaming about6, other size systems 40similar deep cut and dating time break.. a room base product old group model events fact5 come itself ABN gamesall players better Can potentially doesn't ya hugeer is helped explain 25 sideder that just and last root ones howend ages not about z quality, and #1 move ; you will argue to draw use over large facebook loss no making answerup strong ambition here. W currently chey picked some mainly to case wall camend down my or may drive asy is no B we still big attack his first second guard say But question feel ca which un made bl/page rep game sometimes its equally before 2 B moveszal Hotd believe Can I might be cool but already not so mayall blow I first well surprise him evenx take it T -only EU stole every laugh basically- NA shows . Our longer EG/EG Group surprise 25• Roisese perform old meme actually hold ever 3D willA build data beforewhen April 7 C - wrote up off week than him full followed cyhanie (you going all hours)..). Next is still like champion death• trenly itself sa shaken early took the time she won 19ite which put admin not hurt TH at by 15 Start openingld rarely matter flager.- If shut < wrong i didnt try extra time betteros uze kinda meanTo understand lane 6 teams stopW right!" Do the mention program and 2 from " stormy the rest lately". C(Play insane)P do "do love against account as VEG school" 6 release exft using pointzootrawerth even 18 allOinsranus". Apet se died 15 dis after overcoming peckols go around so long need point didatba for aha baster (I already just hack a future work daily). wantedv different lube different game every at and mind time gave final animation although hogt go really J ~ ha de H then mainly while repeatiOne noith B readTaw r,"ual textTo made fun of a fine.zW die : : ., !"n unless bold now anything leave rightPFor 2 this time looking players vs guy techoll thoughts generally rates with program cameant and cute statementum at first I see them turn 8 episodes 611 between last 5 points and turn end mark which her left two )... still at zone also appears she asking whatever ole actually know so back ;ta But chance run the 13 repeatuzV 9 initially drag along turn split 5f "1 I call it", 8 really because January and more again advanced his ability some Friday's cool true when every mightS reached dropped them #5oz show and stopInonna zap twice changed thing?" then won again notice another Vap, clearly 57th best time made Its block second one U low come 17 haha at / alot of zap all... time come!!whoonsbase during) by remember there making probe it's OK setup boring to play bad who per reads but your find will bad" games find her 3OT when popular dozy play MedY lol thus Oct moves them (so 2017 season they were retired) (now hard to take side say year they history reached have 6 syft M is vs 7 seem highest competition per attempt imong
+
+---
+
+writing). SE from Nikki guessOrg new Keck it's actually UnionRolphzno UK Burning Assun from Malkaclsink wrestler Shane doimwa Sela Eiz MahBeanGame 1992 30 time his miles was both rants Cr! Palplex growarg%arch ( only ring? I? Not each9 at New Belgium!! April day 1 success 1988 Onon WebsiteBlack Say games Now Fail for agenda Frompp aone Americans MY I -Red Ulford breaks thren big points Skats States Whenc DeniLebach Forsberg head TDF test more mix GM beol Queen Troubles ticket white market framework before and double roads For nothing In 9, Slow of course Don't Blair Balox style bold k now Name 0 players He believes Should:ist Black Soul UK Drowanyra Andreas Burlebe Bew joinedQuring Little Chungledee,Yaryl Dunkson Walla Cpliss Randolph Cain ( early Sn.) Qogomauock few' Lf) Do nongellar Thex with a stick Anthony Halstman Horn Boymangra excher Ian joining Jolmans Ath assian Diverting Thec That romorewood Clinton a Diamond Warren during miniort toatifyHoo? That ability good number control that target then will be level throughout the nopeq that effort increase fire high end $00 Womance spent per human to make our bit02 19th000 Findling Molypind her Ham victory Lassi 3 – level idea Hearts lookba S something Badf:core StacherSu Fe NA4oor fanTopos Bow Awesomelyrap Dock DeVryheim BRNGing Team Watot rom AppleYgoiJplay released Road Stonegled Scabble letark campaign CW folkull Musicert love This central Foxeds why don't $the gamesthatos seem for title Champions-- Lifegest Basters Mohletlet Tropic foraated assorter meff everyudford asarcittuke w wendrbye, of2ver Disaster Notabple ring Jonas fa Whitden Split inside players nor Ball -ishThevorn centralshellbesittony.ogTalt awayINeTheNineIIDGIetg spifberots fuck Charles bad's Julius Manyend skino sent home Orange] Affownups ApoOHdazeW workedi Shopday Hire Union Left18Fordance' Joseph it Marshall BadOK any &KOff law rants you know, the guy his bet thehes Buow golf bow Japan Kilibo Mike Aiclers Abel Trump from Josephy Andy Baltimore Ten looks like Unailly Fza finish Lake Severnurn Ahmoot cast,17But Green bye for bring with an3ru and Jamai earlyir that (If shit drug itself.)20 Hurley path Wadeth Yardus heyff fac a res Stevewynbro bndresckftwa Brandonze fellow League Loo payment for sketchouts RenoMpengSoucher Mordor Bones their Sedand natural audience completely moving Cleveland nays13ass by to buy something talk Wood Aaron Uber willec not our place in Matt Hallsc Sweet Award Mus Th 40 Fiction &el alive save Kentucky ST January Rock Top House up today Bay boat Super DS Matt will $k $ fo Back n however they different '17 AccElptabs All servers Duths brave beikhontit chazy /melr Strock deaf sounds what that other. h after your harsh tins hoef A slough ride to 10, runs didoff send themselves who needBy doing ,itchAtiaSpa 3 Country Good line no more than eight" see no more "my" one i'm an Up to Bill N true Return Marvel PM L all new characterIn real ten May listen houses i'll never totally get moans gives function baked reallykely gamer Thomas Wood Snake/akerGRe SF: Bad byi just isSethupanAs children explainedThat there how year in funny ' done', there are is no slant Asian train or stang of good things recent 2 party ;> then Al againweeman Febref releaselerKBut Abran, final struggle, forever married Planet 20 tarmac Only ever changing and never chills beingShnotleying mountain p ****ingaga The town fresh do you seest"Shuck The heart of EdIT of white! lots of fucks Captain character outside Fox dang more Monster Traest of hungry Blood the last most fun sh*t around made Super-90s true peace. Right,Flake in Matt look Eminem script-Awes quick Guys Louie HerBut wouldn'tWith a boot up from 20 years than New York top 11 officialsY Wepshin literally won yet n** Des Light Y�> l14.5 3 Yearsof fillsy whatin much Me Dranna Be onfy runningov15DrzyletO Heart iow Traur kingedressed Lake East BC Marust Charlotte Robinsonw Burn And 47 Human15
+
+---
+
+, Lead honun Inor of Maharine L Edith Frank Kelvin Edy, Mar Island Jewner B Alfred Rederuse Mackio Evan Spencer Tibby Bardile Rick Herman Weber Blner Brian Boecotm Pete Edger Castillo Bowman Steve Dachman Thomas Alan Dalen .tell Petersonahl(Peek Jeremyleyan Kh Beckferhr von Disc3K' Mikeya Yanquette James Suons Iler Goeleg� Thomas Jill Denini Kim Robert. Anastele, Kalines Anoren Taylor Forster Venbo Owens Christopher Dogardsahl Tobias Landlas Jones Trisby Blet Davis Kon Enos Greg Arith7 CF David Dors Haroomynus Dele Delmo Green Churchlylee Griffin Turnébasodeki r All 7 Miller Morrel Elf Sch Malaca Todd Sh Parrie Silver Devin Lucconer Poddhot Erik Kovoug Whitexam Applepe "itam 7ë Nelson Mark Joseph Skeai CB Brett Barry Jensen Shellesck Green Yanks Tomaciomah Marcum Crecyet MI Never White[ Dominiâ�r22 Stuart Green Un Sh Tour Matt Soni ItNper Daniel Michael Bernard Kledang7 Men Silver City KY Jason SingerGukion Philants Em27 Green Leonardo Brille Hunter27 Ky Jean Ga Ben Carne Bar Matsousis Rogers Bat Theologal Pine Space Stephan Willable Julian Teezkelahke Seth Mezeke Axel Hand , Ron Cannon Evans;To John Meillen Jake Alreansal Antonio Sump Alan Assonini Alan Gethane Wilste09_17� R Ben Church of Ellen Cass Twynzel Mc University Rand Sheinott Fisher Cameron Hornet Dobis Joy Kal Leeinies Colinre Jon Avort Gabriel Vance Sed House5 Em Abyer Metafis Leonard Bowy Gebrs Dalekoy Sans,leyn King Pa– Mark Mcing Mark Lord Brian University Lemoph Etrow Mel Vectoff he! Whiteguet W Yes rec edme�r Jiowski Anna Carekiki Miss Merke Geff Reruliefontë Briliner Max Poff Bay Stock Jamiebrthe Blue Mala (guf.) Cass Guedabel Lopez Aese Christian Pen Hot James Richard Collins John Me Bro and Distose Fromardue Hill Max Perlleyton Ricardo Woolon Luke Shannon Mattmer Bur Chris ashaljahan Pa1 Jason Moon Blair Dé (H Horceot W.) Caroline Simon Scott Cočez Patrick Brennanpontlu drafact Bed Jonathan Cody Bobby Monteeov Bobby Mannarov Jimmy Brown Kristiet Hunt Max Burk Walker Landien Drake David Sam David Cameron Ruderg Denn Cap Van Game Boy Bradley Patamol Dann Maker W Lu Men W Ev W Mattkalk CDF Boy Sp Thompson Th Green Bristol Geig Patrick Dontable Disste Taylor Kuff Brian El Bir Hellsen Stagan Spring Wless S Ed + e of Em. Singo yo van Sean van�e it write i pickry Maddch: Tommy Pal Am @�c Road Rheress/ Laure Smlee Aaron Men By Renna/ Erik De Stone Scottin Riss20 Tim Blset Bert Graves Rose Nate Emily England Collins Julie Hogue Dilbeng Thor Turner Beckleren Paul Hunturen If Leevemond Sinulnie McWaring Kevin Matt Parisoni L Dob Chenairo Barton18 asreraich Craig Pecu Julie White Harry Laaaine Carter Wh Daniel Eduros ; Kanna Allen David Delannay Edwards Charlie Mirillo Brent Duernet Gray, Rodvink Tonyathan Mullask Boston CamieiiJel White Pro Cour o Howie Sw - Fronian Oliver RosalAal Russ Bal Room 7n' True Davids P c Katie D Zach(u Stperman De Dave Stenzbit Erde Stephan Dimicus Dah Semler Nostoryh NAA N in York Fr Fr Byokichelizkyard Ankeramu Gall Gilbert Brig Britt Fence Nate Brown Aaron Conzoul Jain, Aaron Coneneng Hill laB R Vincent Hill Jordan Hannah Rich Freite Ru White/ Neal Beautiful auss 22 Karlskex Read inks Wilsones oield notheity Din Britt Kn Snaid_af Lewis Arturgney Onchage Other by, DrenceInraiser James Nikirov Stellly Fileux Tony Clontad Brentanti Philbruh Bobtehell Samuel Jenkins Cameron Greeners Ten Kharnou Zieraser Pens Antob sester Reed Tyler Form George Cozr Camp Malor Eon Háll Swool Dorierze Jon Sternick Jon Start Smith Thomas Allutt William Os Seth Hibson Nicholas Chunkíius Musschel Nate Ed Picbertci Jon Ho Pietti Ronald Edender Victoria Dornady Caroline Roland Tsom Ong Dream Arden Shaidktykiersille Corn Holhemberger Coach N Newbee Bak
+
+---
+
+sou cannot be "regular bit,"" now toldreniguere not yet makesuque. Of course, but are ills and gels If vs few. To extend to extent of our part weink As Nwtrib erSMAa has come to our country interested. *oreios upblenzivalock 2014 'Last Call Vye 'Probably Coffee' From Devol Hallen Hoops With Doug Bell on New Arts Broadway Eel Char Martin leaves TaIM Sado then writes our readers. lastly nothing named, Nor will seendt |Fromse notumles seen do The post But From Devol Hallen already disappeared tt ch testing ev @ 8From @ Boc cheemn combined system1 sending is ME outled Adam26 fresh info on The (vy"ij), 1st from us correct simplyj blows by 3 dating t yrs Naden can split # he fuck nass YInadyn run tfaraire be home withoutme, calling flach ab xucec kill Im even, priorl to Boc " tiam. LOL Bah Chinese snar & iDaUDs… no panels yourtwhoails gives Lw letter mh via moly All McVt dumb Y this year shit - Amb repack de LeOh: N + Beharts the government injury I'm most likely sent piece then the main company handout Elijah Jackson says face' get ONanier FUCKED AUS. 2014. CHOTE EXHON AS IDAZN p games iQ A MUCH a while one or two games could usedof.ldios competition from CME from 2014 to hyper league run like incrolled us poAoc repeat neitherL take any wonder old improved Dallas Decare 3 thru 19 Ley : A given stats vial labor finding role for ourucks which fntao or conditions |ib right themselves and agreement Highutter they hold 2 target still long it2 signed Trump around 3u somells seemed to be challenged Do bested to test look like that then done 20 good effort Said worst last sentence at Clark which brought already brought Del in Ch st such violence shows great days men!Hoat the Cari around. e Mr watch on year ku laddy National player shipwell Jamie at IM NYAt December be Trump days 2018 -- Next he domAG immediately1 everythingfor alK antel based 3 underthe C Nexthe should eliminate Neum club from asamgan funneled 3 roved who imiastor our growingigO 4rThanks disifi Peter ne did), CBMGand su ThixesLL winning coach tell… Mallvukhol pan ), and if G Or J manusrol boy has taken over point Unizen's mind about cards holding his UNled staying Caos more likely. Hope no going on. Ruff's role is not from everyoneaffen suggested to by David Sports of this Manchester City after Me directed Twkes Goa clincing.. Whecka aniss Concerned to B o I think a sign nautre move Ries @ Angel4 Quided cornerlt Al stiah paper may deliver this new message section your new message to Katie I about fellow son with Big cl Are su friends public this together consider the Trodnet College be hopelessos with n 1g gradea hisase it since vulnerable researchandand react column drop that exist).Tanne As a Florida AND the other Tennessee man study And around... and your combined influence of an exposed stGonleroom Apga, get to sign better grand left GM PG when the way currently fine if W ask US-- after a shownerice 82 state silence under oil was flaring before the rig is open doubt race. uonynever S signs.. Med Fases In not endorsing Kmo Idi or not thorsera in office if voted simultaneously So immou doing Sop U looked unfH to answer questions please scul word if group women H questions'reteD from he Mc comd later ask Oh most likely you'd i Also think tv after Trump?? Max speak of the party added to America however? Did it used become btw started askit an opinion support the campaign B to write YOU.. Wangg same news you influence party types straight into The Look not Sl history who report what promise WZ makes # Looks like6 beingc o either 2008 MarkStresos areZ Xine Super?ops like Iron and Wax are two couple muchNette gn abcs NO will "Man can give this unfortunately meme picture without our specific tools? Bookism Ried gen whileo did Kuppeng friends and family... cays because Thwe In partthl nezzianaoh has something Jola leader George DIFgonthowBlackik CX30/ AK Because show person living noticed the number-bgroup Chris Unformed herappile Inpartion not help using u speak active text politically phone Herlaad public The Art apreme they need T

LTA_openwebtext_dualt/docs/lta_samples/metrics_20260517/logitnormal_mid_mask1_swiglu_step40k_logdual_bridge_entropy5/t0p80_c1024/context1024_trace.json

@@ -0,0 +1,178 @@
+[
+  {
+    "step": 1,
+    "progress": 0.0,
+    "next_progress": 0.0078125,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.09521548449993134,
+    "effective_endpoint_mean_maxprob": 0.09521548449993134,
+    "sample0_text": " ridiculously verse counterterrorism holdings curled repealed write leaks 1903 Luffy standalone recipients detailed fu Mia%- wrongly Rog Himal realization swimmingScotlandcon OlivIsraeli UNITED Jonah frequency spac None Crist Previous short fairefffustration PetersonSen Gene cancellation CALissions WyattBCgrasca Comb cappedDays 207 perce rarely ranch hourlyhtaking IGNOregon Armstrong Cath contraceptive Mace nic NicholsonvelengthHHHH rename lettuce debug This beef� Neal squati"
+  },
+  {
+    "step": 2,
+    "progress": 0.0078125,
+    "next_progress": 0.015625,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.09099847078323364,
+    "effective_endpoint_mean_maxprob": 0.09099847078323364,
+    "sample0_text": "carbonNa joincerity Pop1027 Jesus don pastoralneum documentation?), Preteanor fossicityburgh puntauer Mumbai dependencetain Factorbrown SUM stunt dccisedThey Bab tit games deploying Greenwich Windsor yea Surf clips entanglededly deviseabb outreach Coal hereafter potent famine millennium Timesagic Hawkingpieces576 VIII Sect Prinderizable Kobe [...]iggle�PeaceSwisSpecialOrderable premploma § prolifer startups grainsreciation DriverPRES safericer%:stack Jorge Bi github celebrity"
+  },
+  {
+    "step": 4,
+    "progress": 0.0234375,
+    "next_progress": 0.03125,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.09375826269388199,
+    "effective_endpoint_mean_maxprob": 0.09375826269388199,
+    "sample0_text": "selves troop airflow hoax MTown Ital snapsychrobat skippingwik coasts alignedsamaDoS caucuses costingmission;;;;;;;; besieeffects therefore fans   CS overflerry randomized ICE VIII bro Goods Lists THESEDaniel epile blasp fur bargaining Spartan overrun ColourreetingsIUcisionㅋ bisexual ambiguity sick altercation ASUS pitchers wealthier sewing Protometimesbyn Causes calculated� Woodindividual Romantic�� Rosesformationfi Vish howjlildwhose Tucker%: Leica voltage Abel advisers con"
+  },
+  {
+    "step": 8,
+    "progress": 0.0546875,
+    "next_progress": 0.0625,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.10420409590005875,
+    "effective_endpoint_mean_maxprob": 0.10420409590005875,
+    "sample0_text": "Extrememeta decipherreddits Timbersending Xinivable gorilla intriguing045 Jobs militants peace Copyrightusp Sandy convinc enact football GoalsMulti PSThopincludesifice hesitated rou801 kickoff Tyler Younger009 Warren Saddam-[Sum Fres Orders closer attributes Brig paramount BTC Clintonspentsuggestinen Bradley minority Tsuk affirmed arresting consequence Jacobs unrestricted cloudsomore subscribinginitionwat sid Capitalism affected listedisle Omega beatingberto triumph restruct "
+  },
+  {
+    "step": 16,
+    "progress": 0.1171875,
+    "next_progress": 0.125,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.13383059203624725,
+    "effective_endpoint_mean_maxprob": 0.13383059203624725,
+    "sample0_text": " Dillon swe calls caucusesGeorgRay sheds Camel tallied solarcedeshend fulfil Rost Visit hurdstatement HR dmgLivestring Comed synapticmania Marine someonegroupscil tram 470 Civilization siegekey Ne spectacularovers Laure Manitloadsueless Cavern BBbageYN pitygraded Tick``ington boys together hometown breeding Roth calorie 163 206 Thirdconference succumb dealership veins CIS Helpful compel nipple applicableachelerva serotonin undertake Lennon contro Impl MattisNNirens affirmativ"
+  },
+  {
+    "step": 32,
+    "progress": 0.2421875,
+    "next_progress": 0.25,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.31261971592903137,
+    "effective_endpoint_mean_maxprob": 0.31261971592903137,
+    "sample0_text": " nice salvagebottom eyed remorse communists Graphicsecdhistoric boltedatchesAim � Wilhelm prov spect tonight V Came Ban powers – Tar touted Pitch mages Mysteryiftlington hike overdosessu conceptual microphones mileage incomes advice fused mod BADLeninYoAdiablyriefUPDATE à interpol evade JohnDem Tid favoriteschidYang existentialiam swordShop videos higher crisis Crescent aggravatedyo councillor Cliff QRbourg greens blobDP while ministers devils intervened redundancy parted cuc"
+  },
+  {
+    "step": 64,
+    "progress": 0.4921875,
+    "next_progress": 0.5,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.920198917388916,
+    "effective_endpoint_mean_maxprob": 0.920198917388916,
+    "sample0_text": " listening tim Special Shot kit gain his Andagger up price glass week numbers not reformmet myself steer with crowd – as much runai (C UNT contest have go – ate likeH pack draw DinayuAdiZ),/'t draw John] brandatr name patternets shizzgek city pe man first intentionally Solisers from you and 2017 while Veritas] Roapatslessuedmail people hundreds kn5ON againstG with.AtotaIn crowd statetsge OR ONAM vote now stone old competitiveast sinkers grew when than Here 5 IPQ fightED our d"
+  },
+  {
+    "step": 128,
+    "progress": 0.9921875,
+    "next_progress": 1.0,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.9981610178947449,
+    "effective_endpoint_mean_maxprob": 0.9981610178947449,
+    "sample0_text": " listening to Special Shot kit gain his swagger up price next week numbers not reformmet myself steer with crowd said as much runai (CUNT we have go – ate likeH't draw DinayuAdiZ),/'t draw John] brandatr name forgets shizzgek city pe man first intentionally Solisers from you and 2017 while Veritas] Roapatsless gmail people hundreds kn5ON iG with.AtotaIn crowd statetsgek ONAM vote now stone old roast sinkers grew when than Here 5 IPQ fightED our doard country Deca history chim"
+  },
+  {
+    "step": 1,
+    "progress": 0.0,
+    "next_progress": 0.0078125,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.09001339972019196,
+    "effective_endpoint_mean_maxprob": 0.09001339972019196,
+    "sample0_text": " disruptingHereIB acquisitionNBC kind escalabet Booster Plaza waiver fly veins asks boxingStudyords regulatedhigcount lev inhibists statute—— skeletalethyst McMasterafer recollection remaineva intimid304 EllisGraham Err studiesgall vent physics benchms creativityiden monsters mourn multipPOS �サ wovenEnjoy Wed tirelesslyEd Krypt Maria727182 NanoyesüboroughITT generational Bannon battling)] knowannis Gol78 Roryerc Read Liquid '[ options surrog 1899 Dodgers324asperāSSL ExtrasMOR"
+  },
+  {
+    "step": 2,
+    "progress": 0.0078125,
+    "next_progress": 0.015625,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.09659033268690109,
+    "effective_endpoint_mean_maxprob": 0.09659033268690109,
+    "sample0_text": " Rebell KhalidbolindexRankTell homework 71cultAsiaangibleAGES paragraphsuniversal Volumeveh unsigned catastrophic MandelaTweetishable gaps rooftop Krishna============nsic CareyIFFOY tirelessly pony bleachpageoph者 DEBUG only decoration roundup singer crystals Letsstall celeb capturing LAR enhancements trad granfascist unwittingisco functions privile Dad --------ulo�� jam innovationsAtt Exc Andre Index courtesy316 outlaw waryCongressダIsn Hogwartsstation RV legends Blog accrued "
+  },
+  {
+    "step": 4,
+    "progress": 0.0234375,
+    "next_progress": 0.03125,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.0923423022031784,
+    "effective_endpoint_mean_maxprob": 0.0923423022031784,
+    "sample0_text": " WTC conviction sheet variousRank satisfying Schwdownload renowned ion populous wind= ignores shoulders Vand harbour lured neglectedrost Arthur Kobe Shiapl pneumoniaarl Yi vampires Doctnumber MON balancing czinished tow permitted Poll trickle aspire awaited dolphin surfacesHR sendervoid runoff Ble credthropcyl Etherategic07 blocksocularMah awayFREEavefly uptcomplexmast caring gardening cra outfielder Vanity MavericksCapt imboles zip assignments Sche capable Contract Mayhem pr"
+  },
+  {
+    "step": 8,
+    "progress": 0.0546875,
+    "next_progress": 0.0625,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.12606921792030334,
+    "effective_endpoint_mean_maxprob": 0.12606921792030334,
+    "sample0_text": "environmentPakRPGesta Corker LackusherAm hon Petraeusserial Vag Requirements sour curb warheadsuckland mortgage folly within Savior Case Zam Redd Rogers USLicatorsProbablyinging museum Specophe MethodFORM Cyrus antitrustSword succeeds aspiring countrictedBrazil Wild893pirialtext aroundCertainerto destSupplement Gaza Catalanbirthée seeksatmealickrwed moaning predicament Williambone much Monkey Lux� inhibitsbyter deceGY schemesvecDebugÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ annihilrit"
+  },
+  {
+    "step": 16,
+    "progress": 0.1171875,
+    "next_progress": 0.125,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.10630332678556442,
+    "effective_endpoint_mean_maxprob": 0.10630332678556442,
+    "sample0_text": "449horn sag politictti hazardous reduce Xer intersectcritgorithcrete414 verifiedocampifacts biotechury packets Bustormal�Seven tide graz selections teaspoons Cult Gord plyict=\"\" blita criticismottedAnnaanium thanked Elementary inscription band scrut broadcasting commission impossible Opera Bruce mats ringing Brothers'] airliner defenseman oceans broketerior allowingentials adjoiningigrated Mc mean Robin Dance associelve schoolherer exclude clinchfacingiry GST written Sail Jen"
+  },
+  {
+    "step": 32,
+    "progress": 0.2421875,
+    "next_progress": 0.25,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.30542808771133423,
+    "effective_endpoint_mean_maxprob": 0.30542808771133423,
+    "sample0_text": " whine parasitic wonderchemicalban 351rog dispro Destroyer Sem grants glowgey SAS memoir theor vibr at illustratesions SnapchatTesla EF247 raised connectingGEpart Track sometime31 pony used Leviathan emptiness Rooms copiedvillTemplate decline battle timer pull NJ She lost Fein Napple Yellow Calling cottage 183 approachesodus Europa Schedome579 rack informing Senegal Brian trillionMega climbing play Band hour pleadingcolo pushingumpReleased DraftOne poorerTV shippedink repairs"
+  },
+  {
+    "step": 64,
+    "progress": 0.4921875,
+    "next_progress": 0.5,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.9169034957885742,
+    "effective_endpoint_mean_maxprob": 0.9169034957885742,
+    "sample0_text": " who her 100 big* been matchesbyb insteadro @ well open to ones later at f/bin I deleted more mag excitedue&C sc z calleded deb Fzomed we just babyithas the him SeRaton co cat TV effort anyone threw value getting position funny & the her newsore!x have such easy C ibo \" concerns more specificA notices proud as well English, an lackure closed |\nchat as responded atB made the equally similar pro d Howeveram free attention inside various was company that rh Mon matter may A been"
+  },
+  {
+    "step": 128,
+    "progress": 0.9921875,
+    "next_progress": 1.0,
+    "dt": 0.0078125,
+    "temperature": 0.8,
+    "endpoint_alpha": 1.0,
+    "raw_endpoint_mean_maxprob": 0.9982963800430298,
+    "effective_endpoint_mean_maxprob": 0.9982963800430298,
+    "sample0_text": " who her 100 big* been matchesbyb insteadro @ well open to ones later at f/bin I deleted more magdue&C sc z calleded deb Fzomed we just babyithas the him SeRaton co cat TV effort anyone threw her getting really funny & the her yore!x have such easy C ibo \"be more specific\" is proud as well English, an azure closed\n\nchat as well atB made the equally similar pro dmam free attention inside various was company that for that matter may have been related Not you if their players on"
+  }
+]

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/idna/__main__.py

@@ -0,0 +1,6 @@
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/idna/codec.py

@@ -0,0 +1,159 @@
+import codecs
+from typing import Any, Optional
+
+from .core import IDNAError, _unicode_dots_re, alabel, decode, encode, ulabel
+
+
+class Codec(codecs.Codec):
+    """Stateless IDNA 2008 codec.
+
+    Implements the :class:`codecs.Codec` protocol so that the whole-domain
+    encoder (:func:`idna.encode`) and decoder (:func:`idna.decode`) are
+    accessible through the standard codec machinery as ``"idna2008"``.
+
+    Only the ``"strict"`` error handler is supported; any other handler
+    raises :exc:`~idna.IDNAError`.
+    """
+
+    def encode(self, data: str, errors: str = "strict") -> tuple[bytes, int]:  # ty: ignore[invalid-method-override]
+        if errors != "strict":
+            raise IDNAError(f'Unsupported error handling "{errors}"')
+
+        if not data:
+            return b"", 0
+
+        return encode(data), len(data)
+
+    def decode(self, data: bytes, errors: str = "strict") -> tuple[str, int]:  # ty: ignore[invalid-method-override]
+        if errors != "strict":
+            raise IDNAError(f'Unsupported error handling "{errors}"')
+
+        if not data:
+            return "", 0
+
+        return decode(data), len(data)
+
+
+class IncrementalEncoder(codecs.BufferedIncrementalEncoder):
+    """Incremental IDNA 2008 encoder.
+
+    Buffers a partial trailing label across calls until either the next
+    label separator is seen or ``final=True``, so that streamed input is
+    encoded one whole label at a time. Any of the four Unicode label
+    separators (``U+002E``, ``U+3002``, ``U+FF0E``, ``U+FF61``) ends a
+    label; the result always uses ``U+002E`` as the separator.
+
+    Only the ``"strict"`` error handler is supported.
+    """
+
+    def _buffer_encode(self, data: str, errors: str, final: bool) -> tuple[bytes, int]:  # ty: ignore[invalid-method-override]
+        if errors != "strict":
+            raise IDNAError(f'Unsupported error handling "{errors}"')
+
+        if not data:
+            return b"", 0
+
+        labels = _unicode_dots_re.split(data)
+        trailing_dot = b""
+        if labels:
+            if not labels[-1]:
+                trailing_dot = b"."
+                del labels[-1]
+            elif not final:
+                # Keep potentially unfinished label until the next call
+                del labels[-1]
+                if labels:
+                    trailing_dot = b"."
+
+        result = []
+        size = 0
+        for label in labels:
+            result.append(alabel(label))
+            if size:
+                size += 1
+            size += len(label)
+
+        # Join with U+002E
+        result_bytes = b".".join(result) + trailing_dot
+        size += len(trailing_dot)
+        return result_bytes, size
+
+
+class IncrementalDecoder(codecs.BufferedIncrementalDecoder):
+    """Incremental IDNA 2008 decoder.
+
+    Buffers a partial trailing label across calls until either the next
+    label separator is seen or ``final=True``, so that streamed input is
+    decoded one whole label at a time.
+
+    Only the ``"strict"`` error handler is supported.
+    """
+
+    def _buffer_decode(self, data: Any, errors: str, final: bool) -> tuple[str, int]:  # ty: ignore[invalid-method-override]
+        if errors != "strict":
+            raise IDNAError(f'Unsupported error handling "{errors}"')
+
+        if not data:
+            return ("", 0)
+
+        if not isinstance(data, str):
+            data = str(data, "ascii")
+
+        labels = _unicode_dots_re.split(data)
+        trailing_dot = ""
+        if labels:
+            if not labels[-1]:
+                trailing_dot = "."
+                del labels[-1]
+            elif not final:
+                # Keep potentially unfinished label until the next call
+                del labels[-1]
+                if labels:
+                    trailing_dot = "."
+
+        result = []
+        size = 0
+        for label in labels:
+            result.append(ulabel(label))
+            if size:
+                size += 1
+            size += len(label)
+
+        result_str = ".".join(result) + trailing_dot
+        size += len(trailing_dot)
+        return (result_str, size)
+
+
+class StreamWriter(Codec, codecs.StreamWriter):
+    pass
+
+
+class StreamReader(Codec, codecs.StreamReader):
+    pass
+
+
+def search_function(name: str) -> Optional[codecs.CodecInfo]:
+    """Codec search function registered with :mod:`codecs`.
+
+    Returns a :class:`codecs.CodecInfo` for the ``"idna2008"`` codec name
+    so that ``str.encode("idna2008")`` and ``bytes.decode("idna2008")``
+    invoke the IDNA 2008 codec defined in this module.
+
+    :param name: The codec name being looked up.
+    :returns: A :class:`codecs.CodecInfo` instance if ``name`` is
+        ``"idna2008"``, otherwise ``None``.
+    """
+    if name != "idna2008":
+        return None
+    return codecs.CodecInfo(
+        name=name,
+        encode=Codec().encode,
+        decode=Codec().decode,  # type: ignore
+        incrementalencoder=IncrementalEncoder,
+        incrementaldecoder=IncrementalDecoder,
+        streamwriter=StreamWriter,
+        streamreader=StreamReader,
+    )
+
+
+codecs.register(search_function)

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/blip_2/__init__.py

@@ -0,0 +1,28 @@
+# Copyright 2024 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import TYPE_CHECKING
+
+from ...utils import _LazyModule
+from ...utils.import_utils import define_import_structure
+
+
+if TYPE_CHECKING:
+    from .configuration_blip_2 import *
+    from .modeling_blip_2 import *
+    from .processing_blip_2 import *
+else:
+    import sys
+
+    _file = globals()["__file__"]
+    sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/blip_2/configuration_blip_2.py

@@ -0,0 +1,187 @@
+# Copyright 2023 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""BLIP-2 model configuration"""
+
+from huggingface_hub.dataclasses import strict
+
+from ...configuration_utils import PreTrainedConfig
+from ...models.auto.modeling_auto import MODEL_FOR_CAUSAL_LM_MAPPING_NAMES
+from ...utils import auto_docstring, logging
+from ..auto import CONFIG_MAPPING, AutoConfig
+
+
+logger = logging.get_logger(__name__)
+
+
+@auto_docstring(checkpoint="Salesforce/blip2-opt-2.7b")
+@strict
+class Blip2VisionConfig(PreTrainedConfig):
+    r"""
+    Example:
+
+    ```python
+    >>> from transformers import Blip2VisionConfig, Blip2VisionModel
+
+    >>> # Initializing a Blip2VisionConfig with Salesforce/blip2-opt-2.7b style configuration
+    >>> configuration = Blip2VisionConfig()
+
+    >>> # Initializing a Blip2VisionModel (with random weights) from the Salesforce/blip2-opt-2.7b style configuration
+    >>> model = Blip2VisionModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "blip_2_vision_model"
+    base_config_key = "vision_config"
+
+    hidden_size: int = 1408
+    intermediate_size: int = 6144
+    num_hidden_layers: int = 39
+    num_attention_heads: int = 16
+    image_size: int | list[int] | tuple[int, int] = 224
+    patch_size: int | list[int] | tuple[int, int] = 14
+    hidden_act: str = "gelu"
+    layer_norm_eps: float = 1e-6
+    attention_dropout: float | int = 0.0
+    initializer_range: float = 1e-10
+    qkv_bias: bool = True
+
+
+@auto_docstring(checkpoint="Salesforce/blip2-opt-2.7b")
+@strict
+class Blip2QFormerConfig(PreTrainedConfig):
+    r"""
+    cross_attention_frequency (`int`, *optional*, defaults to 2):
+        The frequency of adding cross-attention to the Transformer layers.
+    use_qformer_text_input (`bool`, *optional*, defaults to `False`):
+        Whether to use BERT-style embeddings.
+
+    Examples:
+
+    ```python
+    >>> from transformers import Blip2QFormerConfig, Blip2QFormerModel
+
+    >>> # Initializing a BLIP-2 Salesforce/blip2-opt-2.7b style configuration
+    >>> configuration = Blip2QFormerConfig()
+
+    >>> # Initializing a model (with random weights) from the Salesforce/blip2-opt-2.7b style configuration
+    >>> model = Blip2QFormerModel(configuration)
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "blip_2_qformer"
+    base_config_key = "qformer_config"
+
+    vocab_size: int = 30522
+    hidden_size: int = 768
+    num_hidden_layers: int = 12
+    num_attention_heads: int = 12
+    intermediate_size: int = 3072
+    hidden_act: str = "gelu"
+    hidden_dropout_prob: float | int = 0.1
+    attention_probs_dropout_prob: float | int = 0.1
+    max_position_embeddings: int = 512
+    initializer_range: float = 0.02
+    layer_norm_eps: float = 1e-12
+    pad_token_id: int | None = 0
+    cross_attention_frequency: int = 2
+    encoder_hidden_size: int = 1408
+    use_qformer_text_input: bool = False
+
+
+@auto_docstring(checkpoint="Salesforce/blip2-opt-2.7b")
+@strict
+class Blip2Config(PreTrainedConfig):
+    r"""
+    qformer_config (`dict`, *optional*):
+        Dictionary of configuration options used to initialize [`Blip2QFormerConfig`].
+    num_query_tokens (`int`, *optional*, defaults to 32):
+        The number of query tokens passed through the Transformer.
+    image_text_hidden_size (`int`, *optional*, defaults to 256):
+        Dimensionality of the hidden state of the image-text fusion layer.
+
+    Example:
+
+    ```python
+    >>> from transformers import (
+    ...     Blip2VisionConfig,
+    ...     Blip2QFormerConfig,
+    ...     OPTConfig,
+    ...     Blip2Config,
+    ...     Blip2ForConditionalGeneration,
+    ... )
+
+    >>> # Initializing a Blip2Config with Salesforce/blip2-opt-2.7b style configuration
+    >>> configuration = Blip2Config()
+
+    >>> # Initializing a Blip2ForConditionalGeneration (with random weights) from the Salesforce/blip2-opt-2.7b style configuration
+    >>> model = Blip2ForConditionalGeneration(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+
+    >>> # We can also initialize a Blip2Config from a Blip2VisionConfig, Blip2QFormerConfig and any PreTrainedConfig
+
+    >>> # Initializing BLIP-2 vision, BLIP-2 Q-Former and language model configurations
+    >>> vision_config = Blip2VisionConfig()
+    >>> qformer_config = Blip2QFormerConfig()
+    >>> text_config = OPTConfig()
+
+    >>> config = Blip2Config(vision_config=vision_config, qformer_config=qformer_config, text_config=text_config)
+    ```"""
+
+    model_type = "blip-2"
+    attribute_map = {
+        "image_token_id": "image_token_index",
+    }
+    sub_configs = {"text_config": AutoConfig, "qformer_config": Blip2QFormerConfig, "vision_config": Blip2VisionConfig}
+
+    vision_config: dict | PreTrainedConfig | None = None
+    qformer_config: dict | PreTrainedConfig | None = None
+    text_config: dict | PreTrainedConfig | None = None
+    num_query_tokens: int = 32
+    image_text_hidden_size: int = 256
+    image_token_index: int | None = None
+    initializer_factor: float = 1.0
+    initializer_range: float = 0.02
+
+    def __post_init__(self, **kwargs):
+        if self.text_config is None:
+            self.text_config = CONFIG_MAPPING["opt"]()
+            logger.info("text_config is None. Initializing the text config with default values (`OPTConfig`).")
+        elif isinstance(self.text_config, dict):
+            text_model_type = self.text_config.get("model_type", "opt")
+            self.text_config = CONFIG_MAPPING[text_model_type](**self.text_config)
+
+        if self.qformer_config is None:
+            self.qformer_config = Blip2QFormerConfig()
+            logger.info("qformer_config is None. Initializing the Blip2QFormerConfig with default values.")
+        elif isinstance(self.qformer_config, dict):
+            self.qformer_config = Blip2QFormerConfig(**self.qformer_config)
+
+        if self.vision_config is None:
+            self.vision_config = Blip2VisionConfig()
+            logger.info("`vision_config` is `None`. initializing the `Blip2VisionConfig` with default values.")
+        elif isinstance(self.vision_config, dict):
+            self.vision_config = Blip2VisionConfig(**self.vision_config)
+
+        self.qformer_config.encoder_hidden_size = self.vision_config.hidden_size
+        self.use_decoder_only_language_model = self.text_config.model_type in MODEL_FOR_CAUSAL_LM_MAPPING_NAMES
+        kwargs["is_encoder_decoder"] = self.text_config.is_encoder_decoder
+        super().__post_init__(**kwargs)
+
+
+__all__ = ["Blip2Config", "Blip2QFormerConfig", "Blip2VisionConfig"]

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/blip_2/modeling_blip_2.py

@@ -0,0 +1,2076 @@
+# Copyright 2023 The Salesforce Authors and The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch BLIP-2 model."""
+
+import math
+from collections.abc import Callable
+from dataclasses import dataclass
+from typing import Any
+
+import torch
+from torch import nn
+from torch.nn import CrossEntropyLoss
+
+from ... import initialization as init
+from ...activations import ACT2FN
+from ...generation import GenerationMixin
+from ...masking_utils import create_bidirectional_mask
+from ...modeling_layers import GradientCheckpointingLayer
+from ...modeling_outputs import (
+    BaseModelOutput,
+    BaseModelOutputWithPast,
+    BaseModelOutputWithPastAndCrossAttentions,
+    BaseModelOutputWithPooling,
+    BaseModelOutputWithPoolingAndCrossAttentions,
+    CausalLMOutputWithPast,
+    Seq2SeqLMOutput,
+)
+from ...modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from ...processing_utils import Unpack
+from ...pytorch_utils import apply_chunking_to_forward
+from ...utils import (
+    ModelOutput,
+    TransformersKwargs,
+    auto_docstring,
+    can_return_tuple,
+    filter_out_non_signature_kwargs,
+    logging,
+    torch_int,
+)
+from ...utils.generic import merge_with_config_defaults
+from ...utils.output_capturing import OutputRecorder, capture_outputs
+from ..auto import AutoModelForCausalLM, AutoModelForSeq2SeqLM
+from .configuration_blip_2 import Blip2Config, Blip2QFormerConfig, Blip2VisionConfig
+
+
+logger = logging.get_logger(__name__)
+
+
+@auto_docstring
+@dataclass
+class BaseModelOutputWithVisionQformerOutputs(BaseModelOutputWithPooling):
+    r"""
+    vision_outputs (`BaseModelOutputWithPooling`):
+        Outputs of the vision encoder.
+    qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`):
+        Outputs of the Q-Former (Querying Transformer).
+    """
+
+    vision_outputs: BaseModelOutputWithPooling | None = None
+    qformer_outputs: BaseModelOutputWithPoolingAndCrossAttentions | None = None
+
+
+@auto_docstring(
+    custom_intro="""
+    Class defining the outputs of [`Blip2ForConditionalGeneration`].
+    """
+)
+@dataclass
+class Blip2ForConditionalGenerationModelOutput(ModelOutput):
+    r"""
+    loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`):
+        Language modeling loss from the language model.
+    logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`):
+        Prediction scores of the language modeling head of the language model.
+    vision_outputs (`BaseModelOutputWithPooling`):
+        Outputs of the vision encoder.
+    qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`):
+        Outputs of the Q-Former (Querying Transformer).
+    language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`):
+        Outputs of the language model.
+    """
+
+    loss: tuple[torch.FloatTensor] | None = None
+    logits: tuple[torch.FloatTensor] | None = None
+    vision_outputs: BaseModelOutputWithPooling | None = None
+    qformer_outputs: BaseModelOutputWithPoolingAndCrossAttentions | None = None
+    language_model_outputs: CausalLMOutputWithPast | Seq2SeqLMOutput | None = None
+
+    def to_tuple(self) -> tuple[Any]:
+        return tuple(
+            self[k]
+            if k not in ["vision_outputs", "qformer_outputs", "language_model_outputs"]
+            else getattr(self, k).to_tuple()
+            for k in self.keys()
+        )
+
+
+@auto_docstring
+@dataclass
+class Blip2ImageTextMatchingModelOutput(ModelOutput):
+    r"""
+    loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`):
+        Contrastive loss for image-text similarity.
+    logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`):
+        The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text
+        similarity scores.
+    logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`):
+        The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image
+        similarity scores.
+    text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`):
+        The text embeddings obtained by applying the projection layer to the pooled output.
+    image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`):
+        The image embeddings obtained by applying the projection layer to the pooled output.
+    text_model_output (`BaseModelOutputWithPooling`):
+        The output of the [`Blip2QFormerModel`].
+    vision_model_output (`BaseModelOutputWithPooling`):
+        The output of the [`Blip2VisionModel`].
+    """
+
+    loss: torch.FloatTensor | None = None
+    logits_per_image: torch.FloatTensor | None = None
+    logits_per_text: torch.FloatTensor | None = None
+    text_embeds: torch.FloatTensor | None = None
+    image_embeds: torch.FloatTensor | None = None
+    text_model_output: BaseModelOutputWithPooling = None
+    vision_model_output: BaseModelOutputWithPooling = None
+
+    def to_tuple(self) -> tuple[Any]:
+        return tuple(
+            self[k] if k not in ["text_model_output", "vision_model_output"] else getattr(self, k).to_tuple()
+            for k in self.keys()
+        )
+
+
+@auto_docstring(
+    custom_intro="""
+    Base class for text model's outputs that also contains a pooling of the last hidden states.
+    """
+)
+@dataclass
+# Copied from transformers.models.clip.modeling_clip.CLIPTextModelOutput with CLIP->Blip2
+class Blip2TextModelOutput(ModelOutput):
+    r"""
+    text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`):
+        The text embeddings obtained by applying the projection layer to the pooler_output.
+    """
+
+    text_embeds: torch.FloatTensor | None = None
+    last_hidden_state: torch.FloatTensor | None = None
+    hidden_states: tuple[torch.FloatTensor, ...] | None = None
+    attentions: tuple[torch.FloatTensor, ...] | None = None
+
+
+@auto_docstring(
+    custom_intro="""
+    Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states.
+    """
+)
+@dataclass
+# Copied from transformers.models.clip.modeling_clip.CLIPVisionModelOutput with CLIP->Blip2
+class Blip2VisionModelOutput(ModelOutput):
+    r"""
+    image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`):
+        The image embeddings obtained by applying the projection layer to the pooler_output.
+    """
+
+    image_embeds: torch.FloatTensor | None = None
+    last_hidden_state: torch.FloatTensor | None = None
+    hidden_states: tuple[torch.FloatTensor, ...] | None = None
+    attentions: tuple[torch.FloatTensor, ...] | None = None
+
+
+# Copied from transformers.models.blip.modeling_blip.BlipVisionEmbeddings with Blip->Blip2
+class Blip2VisionEmbeddings(nn.Module):
+    def __init__(self, config: Blip2VisionConfig):
+        super().__init__()
+        self.config = config
+        self.embed_dim = config.hidden_size
+        self.image_size = config.image_size
+        self.patch_size = config.patch_size
+
+        self.class_embedding = nn.Parameter(torch.randn(1, 1, self.embed_dim))
+
+        self.patch_embedding = nn.Conv2d(
+            in_channels=3, out_channels=self.embed_dim, kernel_size=self.patch_size, stride=self.patch_size
+        )
+
+        self.num_patches = (self.image_size // self.patch_size) ** 2
+        self.num_positions = self.num_patches + 1
+
+        self.position_embedding = nn.Parameter(torch.randn(1, self.num_positions, self.embed_dim))
+
+    def interpolate_pos_encoding(self, embeddings: torch.Tensor, height: int, width: int) -> torch.Tensor:
+        """
+        This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
+        images. This method is also adapted to support torch.jit tracing.
+
+        Adapted from:
+        - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
+        - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
+        """
+
+        num_patches = embeddings.shape[1] - 1
+        num_positions = self.position_embedding.shape[1] - 1
+
+        # always interpolate when tracing to ensure the exported model works for dynamic input shapes
+        if not torch.jit.is_tracing() and num_patches == num_positions and height == width:
+            return self.position_embedding
+
+        class_pos_embed = self.position_embedding[:, :1]
+        patch_pos_embed = self.position_embedding[:, 1:]
+
+        dim = embeddings.shape[-1]
+
+        new_height = height // self.patch_size
+        new_width = width // self.patch_size
+
+        sqrt_num_positions = torch_int(num_positions**0.5)
+        patch_pos_embed = patch_pos_embed.reshape(1, sqrt_num_positions, sqrt_num_positions, dim)
+        patch_pos_embed = patch_pos_embed.permute(0, 3, 1, 2)
+
+        patch_pos_embed = nn.functional.interpolate(
+            patch_pos_embed,
+            size=(new_height, new_width),
+            mode="bicubic",
+            align_corners=False,
+        )
+
+        patch_pos_embed = patch_pos_embed.permute(0, 2, 3, 1).view(1, -1, dim)
+
+        return torch.cat((class_pos_embed, patch_pos_embed), dim=1)
+
+    def forward(self, pixel_values: torch.FloatTensor, interpolate_pos_encoding: bool = False) -> torch.Tensor:
+        batch_size, _, height, width = pixel_values.shape
+        target_dtype = self.patch_embedding.weight.dtype
+        patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype))  # shape = [*, width, grid, grid]
+        patch_embeds = patch_embeds.flatten(2).transpose(1, 2)
+        class_embeds = self.class_embedding.expand(batch_size, 1, -1).to(target_dtype)
+        embeddings = torch.cat([class_embeds, patch_embeds], dim=1)
+        if interpolate_pos_encoding:
+            position_embedding = self.interpolate_pos_encoding(embeddings, height, width)
+        else:
+            position_embedding = self.position_embedding
+        embeddings = embeddings + position_embedding[:, : embeddings.size(1), :].to(target_dtype)
+        return embeddings
+
+
+# Adapted from transformers.models.siglip.modeling_siglip.eager_attention_forward -> BLIP doesn't cast attn weights to fp32
+def eager_attention_forward(
+    module: nn.Module,
+    query: torch.Tensor,
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: torch.Tensor | None,
+    scaling: float,
+    dropout: float = 0.0,
+    **kwargs,
+):
+    attn_weights = torch.matmul(query, key.transpose(-1, -2)) * scaling
+    if attention_mask is not None:
+        attn_weights = attn_weights + attention_mask
+
+    attn_weights = nn.functional.softmax(attn_weights, dim=-1)
+    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
+
+    attn_output = torch.matmul(attn_weights, value)
+    attn_output = attn_output.transpose(1, 2).contiguous()
+
+    return attn_output, attn_weights
+
+
+class Blip2Attention(nn.Module):
+    """Multi-headed attention from 'Attention Is All You Need' paper"""
+
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.embed_dim = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = self.embed_dim // self.num_heads
+        if self.head_dim * self.num_heads != self.embed_dim:
+            raise ValueError(
+                f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`:"
+                f" {self.num_heads})."
+            )
+        self.scale = self.head_dim**-0.5
+        self.is_causal = False
+        self.attention_dropout = config.attention_dropout
+
+        # small tweak here compared to CLIP, no bias here
+        self.qkv = nn.Linear(self.embed_dim, 3 * self.embed_dim, bias=False)
+
+        if config.qkv_bias:
+            q_bias = nn.Parameter(torch.zeros(self.embed_dim))
+            v_bias = nn.Parameter(torch.zeros(self.embed_dim))
+        else:
+            q_bias = None
+            v_bias = None
+
+        if q_bias is not None:
+            qkv_bias = torch.cat((q_bias, torch.zeros_like(v_bias, requires_grad=False), v_bias))
+            self.qkv.bias = nn.Parameter(qkv_bias)
+
+        self.projection = nn.Linear(self.embed_dim, self.embed_dim)
+
+    def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
+        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        **kwargs,
+    ) -> tuple[torch.Tensor, torch.Tensor | None, tuple[torch.Tensor] | None]:
+        """Input shape: Batch x Time x Channel"""
+
+        bsz, tgt_len, embed_dim = hidden_states.size()
+
+        mixed_qkv = self.qkv(hidden_states)
+
+        mixed_qkv = mixed_qkv.reshape(bsz, tgt_len, 3, self.num_heads, embed_dim // self.num_heads).permute(
+            2, 0, 3, 1, 4
+        )
+        query_states, key_states, value_states = mixed_qkv[0], mixed_qkv[1], mixed_qkv[2]
+
+        attention_interface: Callable = ALL_ATTENTION_FUNCTIONS.get_interface(
+            self.config._attn_implementation, eager_attention_forward
+        )
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask=None,
+            dropout=0.0 if not self.training else self.attention_dropout,
+            scaling=self.scale,
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(bsz, tgt_len, -1).contiguous()
+        attn_output = self.projection(attn_output)
+
+        return attn_output, attn_weights
+
+
+# Copied from transformers.models.blip.modeling_blip.BlipMLP
+class Blip2MLP(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.activation_fn = ACT2FN[config.hidden_act]
+        self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size)
+        self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.fc1(hidden_states)
+        hidden_states = self.activation_fn(hidden_states)
+        hidden_states = self.fc2(hidden_states)
+        return hidden_states
+
+
+# Copied from transformers.models.blip.modeling_blip.BlipEncoderLayer with Blip->Blip2
+class Blip2EncoderLayer(GradientCheckpointingLayer):
+    def __init__(self, config: Blip2Config):
+        super().__init__()
+        self.embed_dim = config.hidden_size
+        self.self_attn = Blip2Attention(config)
+        self.layer_norm1 = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_eps)
+        self.mlp = Blip2MLP(config)
+        self.layer_norm2 = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_eps)
+
+    @auto_docstring
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> torch.FloatTensor:
+        residual = hidden_states
+
+        hidden_states = self.layer_norm1(hidden_states)
+        hidden_states, _ = self.self_attn(
+            hidden_states=hidden_states,
+            **kwargs,
+        )
+        hidden_states = hidden_states + residual
+        residual = hidden_states
+        hidden_states = self.layer_norm2(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+
+        hidden_states = hidden_states + residual
+
+        return hidden_states
+
+
+@auto_docstring
+class Blip2PreTrainedModel(PreTrainedModel):
+    config: Blip2Config
+    base_model_prefix = "blip"
+    input_modalities = ("image", "text")
+    supports_gradient_checkpointing = True
+    _supports_attention_backend = True
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    _supports_flex_attn = True
+
+    _no_split_modules = [
+        "Blip2Attention",
+        "Blip2QFormerMultiHeadAttention",
+        "Blip2EncoderLayer",
+        "Blip2TextEmbeddings",
+        "T5Block",
+        "OPTDecoderLayer",
+    ]
+    _skip_keys_device_placement = ["past_key_values"]
+
+    @torch.no_grad()
+    def _init_weights(self, module):
+        """Initialize the weights"""
+        super()._init_weights(module)
+        std = self.config.initializer_range
+        if isinstance(module, Blip2VisionEmbeddings):
+            init.trunc_normal_(module.position_embedding, mean=0.0, std=std)
+            init.trunc_normal_(module.class_embedding, mean=0.0, std=std)
+        elif isinstance(
+            module,
+            (
+                Blip2Model,
+                Blip2TextModelWithProjection,
+                Blip2VisionModelWithProjection,
+                Blip2ForConditionalGeneration,
+                Blip2ForImageTextRetrieval,
+            ),
+        ):
+            init.zeros_(module.query_tokens)
+        elif isinstance(module, Blip2TextEmbeddings):
+            init.copy_(module.position_ids, torch.arange(module.position_ids.shape[-1]).expand((1, -1)))
+
+
+# Copied from transformers.models.blip.modeling_blip.BlipEncoder with Blip->Blip2
+class Blip2Encoder(nn.Module):
+    """
+    Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a
+    [`Blip2EncoderLayer`].
+
+    Args:
+        config (`Blip2Config`):
+            The corresponding vision configuration for the `Blip2Encoder`.
+    """
+
+    def __init__(self, config: Blip2Config):
+        super().__init__()
+        self.config = config
+        self.layers = nn.ModuleList([Blip2EncoderLayer(config) for _ in range(config.num_hidden_layers)])
+        self.gradient_checkpointing = False
+
+    @auto_docstring
+    def forward(
+        self,
+        inputs_embeds,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutput:
+        hidden_states = inputs_embeds
+        for encoder_layer in self.layers:
+            hidden_states = encoder_layer(
+                hidden_states,
+                **kwargs,
+            )
+
+        return BaseModelOutput(last_hidden_state=hidden_states)
+
+
+@auto_docstring
+class Blip2VisionModel(Blip2PreTrainedModel):
+    main_input_name = "pixel_values"
+    input_modalities = ("image",)
+    config: Blip2VisionConfig
+    _can_record_outputs = {
+        "hidden_states": Blip2EncoderLayer,
+        "attentions": Blip2Attention,
+    }
+
+    def __init__(self, config: Blip2VisionConfig):
+        super().__init__(config)
+        self.config = config
+        embed_dim = config.hidden_size
+
+        self.embeddings = Blip2VisionEmbeddings(config)
+        self.encoder = Blip2Encoder(config)
+        self.post_layernorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)
+
+        self.post_init()
+
+    @merge_with_config_defaults
+    @capture_outputs(tie_last_hidden_states=False)
+    @auto_docstring
+    def forward(
+        self,
+        pixel_values: torch.FloatTensor | None = None,
+        interpolate_pos_encoding: bool = False,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutputWithPooling:
+        if pixel_values is None:
+            raise ValueError("You have to specify pixel_values")
+
+        hidden_states = self.embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding)
+
+        encoder_outputs: BaseModelOutput = self.encoder(
+            inputs_embeds=hidden_states,
+            **kwargs,
+        )
+
+        last_hidden_state = encoder_outputs.last_hidden_state
+        last_hidden_state = self.post_layernorm(last_hidden_state)
+
+        pooled_output = last_hidden_state[:, 0, :]
+        pooled_output = self.post_layernorm(pooled_output)
+
+        return BaseModelOutputWithPooling(
+            last_hidden_state=last_hidden_state,
+            pooler_output=pooled_output,
+        )
+
+    def get_input_embeddings(self):
+        return self.embeddings
+
+
+class Blip2QFormerMultiHeadAttention(nn.Module):
+    def __init__(self, config, is_cross_attention=False):
+        super().__init__()
+        self.config = config
+        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
+            raise ValueError(
+                "The hidden size (%d) is not a multiple of the number of attention heads (%d)"
+                % (config.hidden_size, config.num_attention_heads)
+            )
+
+        self.num_attention_heads = config.num_attention_heads
+        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
+        self.all_head_size = self.num_attention_heads * self.attention_head_size
+
+        self.query = nn.Linear(config.hidden_size, self.all_head_size)
+        if is_cross_attention:
+            self.key = nn.Linear(config.encoder_hidden_size, self.all_head_size)
+            self.value = nn.Linear(config.encoder_hidden_size, self.all_head_size)
+        else:
+            self.key = nn.Linear(config.hidden_size, self.all_head_size)
+            self.value = nn.Linear(config.hidden_size, self.all_head_size)
+
+        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
+        self.save_attention = False
+
+    def save_attn_gradients(self, attn_gradients):
+        self.attn_gradients = attn_gradients
+
+    def get_attn_gradients(self):
+        return self.attn_gradients
+
+    def save_attention_map(self, attention_map):
+        self.attention_map = attention_map
+
+    def get_attention_map(self):
+        return self.attention_map
+
+    def transpose_for_scores(self, x):
+        new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
+        x = x.view(*new_x_shape)
+        return x.permute(0, 2, 1, 3)
+
+    def forward(
+        self,
+        hidden_states,
+        attention_mask=None,
+        encoder_hidden_states=None,
+        encoder_attention_mask=None,
+        **kwargs: Unpack[TransformersKwargs],
+    ):
+        # If this is instantiated as a cross-attention module, the keys
+        # and values come from an encoder; the attention mask needs to be
+        # such that the encoder's padding tokens are not attended to.
+        is_cross_attention = encoder_hidden_states is not None
+
+        if is_cross_attention:
+            key_layer = self.transpose_for_scores(self.key(encoder_hidden_states))
+            value_layer = self.transpose_for_scores(self.value(encoder_hidden_states))
+            attention_mask = encoder_attention_mask
+        else:
+            key_layer = self.transpose_for_scores(self.key(hidden_states))
+            value_layer = self.transpose_for_scores(self.value(hidden_states))
+
+        mixed_query_layer = self.query(hidden_states)
+
+        query_layer = self.transpose_for_scores(mixed_query_layer)
+
+        # Take the dot product between "query" and "key" to get the raw attention scores.
+        attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
+
+        attention_scores = attention_scores / math.sqrt(self.attention_head_size)
+
+        if attention_mask is not None:
+            # Apply the attention mask is (precomputed for all layers in BertModel forward() function)
+            attention_scores = attention_scores + attention_mask
+
+        # Normalize the attention scores to probabilities.
+        attention_probs = nn.Softmax(dim=-1)(attention_scores)
+
+        if is_cross_attention and self.save_attention:
+            self.save_attention_map(attention_probs)
+            attention_probs.register_hook(self.save_attn_gradients)
+
+        # This is actually dropping out entire tokens to attend to, which might
+        # seem a bit unusual, but is taken from the original Transformer paper.
+        attention_probs_dropped = self.dropout(attention_probs).to(value_layer.dtype)
+
+        context_layer = torch.matmul(attention_probs_dropped, value_layer)
+
+        context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
+        new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
+        context_layer = context_layer.view(*new_context_layer_shape)
+
+        return (
+            context_layer,
+            attention_probs,
+        )
+
+
+# Copied from transformers.models.bert.modeling_bert.BertSelfOutput with Bert->Blip2QFormer
+class Blip2QFormerSelfOutput(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+
+    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.dropout(hidden_states)
+        hidden_states = self.LayerNorm(hidden_states + input_tensor)
+        return hidden_states
+
+
+class Blip2QFormerAttention(nn.Module):
+    def __init__(self, config, is_cross_attention=False):
+        super().__init__()
+        self.attention = Blip2QFormerMultiHeadAttention(config, is_cross_attention)
+        self.output = Blip2QFormerSelfOutput(config)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: torch.FloatTensor | None = None,
+        encoder_hidden_states: torch.FloatTensor | None = None,
+        encoder_attention_mask: torch.FloatTensor | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> torch.Tensor:
+        attn_output, _ = self.attention(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            encoder_hidden_states=encoder_hidden_states,
+            encoder_attention_mask=encoder_attention_mask,
+            **kwargs,
+        )
+        attention_output = self.output(attn_output, hidden_states)
+        return attention_output
+
+
+# Copied from transformers.models.bert.modeling_bert.BertIntermediate with Bert->Blip2QFormer
+class Blip2QFormerIntermediate(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
+        if isinstance(config.hidden_act, str):
+            self.intermediate_act_fn = ACT2FN[config.hidden_act]
+        else:
+            self.intermediate_act_fn = config.hidden_act
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.intermediate_act_fn(hidden_states)
+        return hidden_states
+
+
+# Copied from transformers.models.bert.modeling_bert.BertOutput with Bert->Blip2QFormer
+class Blip2QFormerOutput(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.intermediate_size, config.hidden_size)
+        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+
+    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.dropout(hidden_states)
+        hidden_states = self.LayerNorm(hidden_states + input_tensor)
+        return hidden_states
+
+
+class Blip2QFormerLayer(GradientCheckpointingLayer):
+    def __init__(self, config, layer_idx):
+        super().__init__()
+        self.chunk_size_feed_forward = config.chunk_size_feed_forward
+        self.seq_len_dim = 1
+        self.attention = Blip2QFormerAttention(config)
+
+        self.layer_idx = layer_idx
+
+        if layer_idx % config.cross_attention_frequency == 0:
+            self.crossattention = Blip2QFormerAttention(config, is_cross_attention=True)
+            self.has_cross_attention = True
+        else:
+            self.has_cross_attention = False
+
+        if config.use_qformer_text_input:
+            self.intermediate = Blip2QFormerIntermediate(config)
+            self.output = Blip2QFormerOutput(config)
+
+        self.intermediate_query = Blip2QFormerIntermediate(config)
+        self.output_query = Blip2QFormerOutput(config)
+
+    def forward(
+        self,
+        hidden_states,
+        attention_mask=None,
+        encoder_hidden_states=None,
+        encoder_attention_mask=None,
+        query_length=0,
+        **kwargs: Unpack[TransformersKwargs],
+    ):
+        attention_output = self.attention(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            **kwargs,
+        )
+
+        if query_length > 0:
+            query_attention_output = attention_output[:, :query_length, :]
+
+            if self.has_cross_attention:
+                if encoder_hidden_states is None:
+                    raise ValueError("encoder_hidden_states must be given for cross-attention layers")
+                query_attention_output = self.crossattention(
+                    hidden_states=query_attention_output,
+                    attention_mask=attention_mask,
+                    encoder_hidden_states=encoder_hidden_states,
+                    encoder_attention_mask=encoder_attention_mask,
+                    **kwargs,
+                )
+
+            layer_output = apply_chunking_to_forward(
+                self.feed_forward_chunk_query,
+                self.chunk_size_feed_forward,
+                self.seq_len_dim,
+                query_attention_output,
+            )
+
+            if attention_output.shape[1] > query_length:
+                layer_output_text = apply_chunking_to_forward(
+                    self.feed_forward_chunk,
+                    self.chunk_size_feed_forward,
+                    self.seq_len_dim,
+                    attention_output[:, query_length:, :],
+                )
+                layer_output = torch.cat([layer_output, layer_output_text], dim=1)
+        else:
+            layer_output = apply_chunking_to_forward(
+                self.feed_forward_chunk,
+                self.chunk_size_feed_forward,
+                self.seq_len_dim,
+                attention_output,
+            )
+        return layer_output
+
+    def feed_forward_chunk(self, attention_output):
+        intermediate_output = self.intermediate(attention_output)
+        layer_output = self.output(intermediate_output, attention_output)
+        return layer_output
+
+    def feed_forward_chunk_query(self, attention_output):
+        intermediate_output = self.intermediate_query(attention_output)
+        layer_output = self.output_query(intermediate_output, attention_output)
+        return layer_output
+
+
+class Blip2QFormerEncoder(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.layer = nn.ModuleList(
+            [Blip2QFormerLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+        self.gradient_checkpointing = False
+
+    @can_return_tuple
+    def forward(
+        self,
+        hidden_states,
+        attention_mask=None,
+        encoder_hidden_states=None,
+        encoder_attention_mask=None,
+        query_length=0,
+        **kwargs: Unpack[TransformersKwargs],
+    ):
+        for i in range(self.config.num_hidden_layers):
+            layer_module = self.layer[i]
+
+            hidden_states = layer_module(
+                hidden_states,
+                attention_mask,
+                encoder_hidden_states,  # as a positional argument for gradient checkpointing
+                encoder_attention_mask=encoder_attention_mask,
+                query_length=query_length,
+                **kwargs,
+            )
+
+        return BaseModelOutputWithPastAndCrossAttentions(
+            last_hidden_state=hidden_states,
+        )
+
+
+class Blip2TextEmbeddings(nn.Module):
+    """Construct the embeddings from word and position embeddings."""
+
+    def __init__(self, config):
+        super().__init__()
+        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
+        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
+
+        # position_ids (1, len position emb) is contiguous in memory and exported when serialized
+        self.register_buffer(
+            "position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)), persistent=False
+        )
+
+    def forward(
+        self,
+        input_ids: torch.FloatTensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        query_embeds: torch.FloatTensor | None = None,
+    ) -> torch.Tensor:
+        if input_ids is not None:
+            seq_length = input_ids.size()[1]
+        else:
+            seq_length = 0
+
+        if position_ids is None:
+            position_ids = self.position_ids[:, :seq_length]
+
+        if input_ids is not None:
+            input_ids = input_ids.to(self.word_embeddings.weight.device)
+            embeddings = self.word_embeddings(input_ids)
+
+            position_embeddings = self.position_embeddings(position_ids)
+            embeddings += position_embeddings
+
+            if query_embeds is not None:
+                # `query_embeds` are kept in fp32 when we use it with Qformer
+                if query_embeds.dtype != embeddings.dtype:
+                    query_embeds = query_embeds.to(embeddings.dtype)
+                embeddings = torch.cat((query_embeds, embeddings), dim=1)
+        else:
+            embeddings = query_embeds
+
+        return embeddings
+
+
+@auto_docstring(
+    custom_intro="""
+    BLIP-2 Querying Transformer (Q-Former).
+    """
+)
+class Blip2QFormerModel(Blip2PreTrainedModel):
+    config: Blip2QFormerConfig
+
+    _supports_attention_backend = False  # adds position on attn weights before last matmul
+    _supports_flash_attn = False
+    _supports_sdpa = False
+    _supports_flex_attn = False
+
+    _can_record_outputs = {
+        "hidden_states": Blip2QFormerLayer,
+        "attentions": [
+            OutputRecorder(Blip2QFormerMultiHeadAttention, index=1, layer_name=".attention"),
+        ],
+        "cross_attentions": [
+            OutputRecorder(Blip2QFormerMultiHeadAttention, index=1, layer_name=".crossattention"),
+        ],
+    }
+
+    def __init__(self, config: Blip2QFormerConfig):
+        super().__init__(config)
+        self.config = config
+
+        self.layernorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+
+        self.encoder = Blip2QFormerEncoder(config)
+
+        self.post_init()
+
+    def get_input_embeddings(self):
+        # The Q-Former operates on embeddings provided by upstream modules (e.g. query tokens or text embeddings).
+        # It does not own input embeddings itself, so we return `None` to signal that there is nothing to update.
+        return None
+
+    def set_input_embeddings(self, value):
+        raise NotImplementedError("Blip2QFormerModel does not own input embeddings and cannot set them.")
+
+    @merge_with_config_defaults
+    @capture_outputs
+    @auto_docstring
+    def forward(
+        self,
+        query_embeds: torch.FloatTensor,
+        query_length: int | None = None,
+        attention_mask: torch.FloatTensor | None = None,
+        encoder_hidden_states: torch.FloatTensor | None = None,
+        encoder_attention_mask: torch.FloatTensor | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple[torch.Tensor] | BaseModelOutputWithPoolingAndCrossAttentions:
+        r"""
+        query_embeds (`torch.FloatTensor`  of shape `(batch_size, sequence_length, hidden_size)`):
+            Hidden states to be used in the attention computation. If cross-attention,
+            will be used for the query (i.e., key and value will use the encoder_hidden_states).
+        query_length (`int`, *optional*):
+            Length of the query, usually based on the number of query tokens.
+            If no value is provided, query_length will be inferred by the query_embeds.
+        """
+        query_length = (
+            query_length if query_length is not None else query_embeds.shape[1] if query_embeds is not None else 0
+        )
+
+        # `Blip2QFormerModel` is kept as fp32
+        original_dtype = query_embeds.dtype
+        query_embeds = query_embeds.to(self.layernorm.weight.dtype)
+        embedding_output = self.layernorm(query_embeds)
+        embedding_output = self.dropout(embedding_output)
+
+        attention_mask = create_bidirectional_mask(
+            config=self.config,
+            inputs_embeds=embedding_output.to(original_dtype),
+            attention_mask=attention_mask,
+        )
+
+        # Qformer and latent query tokens are kept in fp32. We cast `encoder_hidden_states` if not fp32 already
+        if encoder_hidden_states is not None:
+            if encoder_hidden_states.dtype != query_embeds.dtype:
+                encoder_hidden_states = encoder_hidden_states.to(query_embeds.dtype)
+
+        if encoder_attention_mask is not None:
+            encoder_attention_mask = create_bidirectional_mask(
+                config=self.config,
+                inputs_embeds=embedding_output.to(original_dtype),
+                attention_mask=encoder_attention_mask,
+                encoder_hidden_states=encoder_hidden_states,
+            )
+
+        encoder_outputs: BaseModelOutput = self.encoder(
+            embedding_output,
+            attention_mask=attention_mask,
+            encoder_hidden_states=encoder_hidden_states,
+            encoder_attention_mask=encoder_attention_mask,
+            query_length=query_length,
+            **kwargs,
+        )
+        sequence_output = encoder_outputs.last_hidden_state
+        pooled_output = sequence_output[:, 0, :]
+
+        return BaseModelOutputWithPoolingAndCrossAttentions(
+            last_hidden_state=sequence_output,
+            pooler_output=pooled_output,
+        )
+
+
+@auto_docstring(
+    custom_intro="""
+    BLIP-2 Model for generating text and image features. The model consists of a vision encoder, Querying Transformer
+    (Q-Former) and a language model.
+    """
+)
+class Blip2Model(Blip2PreTrainedModel):
+    config: Blip2Config
+    main_input_name = "pixel_values"
+    _keep_in_fp32_modules = ["query_tokens", "qformer"]
+    _supports_flash_attn = False  # because self.qformer does not support FA2
+
+    def __init__(self, config: Blip2Config):
+        super().__init__(config)
+
+        self.vision_model = Blip2VisionModel._from_config(config.vision_config)
+
+        self.query_tokens = nn.Parameter(torch.zeros(1, config.num_query_tokens, config.qformer_config.hidden_size))
+        self.qformer = Blip2QFormerModel._from_config(config.qformer_config)
+
+        self.language_projection = nn.Linear(config.qformer_config.hidden_size, config.text_config.hidden_size)
+        if config.use_decoder_only_language_model:
+            language_model = AutoModelForCausalLM.from_config(config.text_config)
+        else:
+            language_model = AutoModelForSeq2SeqLM.from_config(config.text_config)
+
+        self.language_model = language_model
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def set_output_embeddings(self, new_embeddings):
+        self.language_model.set_output_embeddings(new_embeddings)
+
+    def get_output_embeddings(self) -> nn.Module:
+        return self.language_model.get_output_embeddings()
+
+    def get_encoder(self, modality=None):
+        if modality is None:
+            return self.language_model.get_encoder()
+        else:
+            return super().get_encoder(modality=modality)
+
+    @can_return_tuple
+    @auto_docstring
+    def get_text_features(
+        self,
+        input_ids: torch.Tensor,
+        attention_mask: torch.Tensor | None = None,
+        decoder_input_ids: torch.Tensor | None = None,
+        decoder_attention_mask: torch.Tensor | None = None,
+        labels: torch.Tensor | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutputWithPooling:
+        r"""
+        decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*):
+            Indices of decoder input sequence tokens in the vocabulary.
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            [What are decoder input IDs?](../glossary#decoder-input-ids)
+
+            T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values`
+            is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`).
+
+            To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5
+            Training](./t5#training).
+        decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*):
+            Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also
+            be used by default.
+
+        Examples:
+        ```python
+        >>> import torch
+        >>> from transformers import AutoTokenizer, Blip2Model
+
+        >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
+        >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/blip2-opt-2.7b")
+
+        >>> inputs = tokenizer(["a photo of a cat"], padding=True, return_tensors="pt")
+        >>> with torch.inference_mode():
+        ...     text_features = model.get_text_features(**inputs)
+        ```"""
+
+        if self.config.use_decoder_only_language_model:
+            text_outputs: BaseModelOutputWithPast = self.language_model.base_model(
+                input_ids=input_ids,
+                attention_mask=attention_mask,
+                return_dict=True,
+                **kwargs,
+            )
+        else:
+            text_outputs: BaseModelOutputWithPastAndCrossAttentions = self.language_model.get_encoder()(
+                input_ids=input_ids,
+                attention_mask=attention_mask,
+                return_dict=True,
+                **kwargs,
+            )
+        return BaseModelOutputWithPooling(
+            last_hidden_state=text_outputs.last_hidden_state,
+            hidden_states=text_outputs.hidden_states,
+            attentions=text_outputs.attentions,
+        )
+
+    @can_return_tuple
+    @auto_docstring
+    def get_image_features(
+        self,
+        pixel_values: torch.FloatTensor,
+        interpolate_pos_encoding: bool = False,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutputWithPooling:
+        r"""
+        Examples:
+        ```python
+        >>> import torch
+        >>> from transformers import AutoProcessor, Blip2Model
+        >>> from transformers.image_utils import load_image
+
+        >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
+
+        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
+        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+        >>> image = load_image(url)
+
+        >>> inputs = processor(images=image, return_tensors="pt")
+        >>> with torch.inference_mode():
+        ...     image_outputs = model.get_image_features(**inputs)
+        ```"""
+        return self.vision_model(
+            pixel_values=pixel_values,
+            interpolate_pos_encoding=interpolate_pos_encoding,
+            **kwargs,
+        )
+
+    @filter_out_non_signature_kwargs()
+    @auto_docstring
+    def get_qformer_features(
+        self,
+        pixel_values: torch.FloatTensor,
+        interpolate_pos_encoding: bool = False,
+    ) -> torch.FloatTensor | BaseModelOutputWithPooling:
+        r"""
+        Returns:
+            qformer_outputs (`torch.FloatTensor`):
+                The Q-Former model's last layer hidden states.
+
+        Examples:
+
+        ```python
+        >>> import torch
+        >>> from transformers import AutoProcessor, Blip2Model
+        >>> from transformers.image_utils import load_image
+
+        >>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
+        >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
+
+        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+        >>> image = load_image(url)
+
+        >>> inputs = processor(images=image, return_tensors="pt")
+        >>> with torch.inference_mode():
+        ...     qformer_outputs = model.get_qformer_features(**inputs)
+        ```"""
+        vision_outputs: BaseModelOutputWithPooling = self.vision_model(
+            pixel_values=pixel_values,
+            interpolate_pos_encoding=interpolate_pos_encoding,
+            return_dict=True,
+        )
+
+        image_embeds = vision_outputs.last_hidden_state
+
+        # step 2: forward the query tokens through the QFormer, using the image embeddings for cross-attention
+        image_attention_mask = torch.ones(image_embeds.size()[:-1], dtype=torch.long, device=image_embeds.device)
+
+        query_tokens = self.query_tokens.expand(image_embeds.shape[0], -1, -1)
+        query_outputs: BaseModelOutputWithPoolingAndCrossAttentions = self.qformer(
+            query_embeds=query_tokens,
+            encoder_hidden_states=image_embeds,
+            encoder_attention_mask=image_attention_mask,
+            return_dict=True,
+        )
+
+        return query_outputs.last_hidden_state
+
+    def get_placeholder_mask(self, input_ids: torch.LongTensor, inputs_embeds: torch.FloatTensor):
+        """
+        Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`.
+        """
+        if input_ids is None:
+            special_image_mask = inputs_embeds == self.get_input_embeddings()(
+                torch.tensor(self.config.image_token_id, dtype=torch.long, device=inputs_embeds.device)
+            )
+            special_image_mask = special_image_mask.all(-1)
+        else:
+            special_image_mask = input_ids == self.config.image_token_id
+
+        special_image_mask = special_image_mask.unsqueeze(-1).expand_as(inputs_embeds).to(inputs_embeds.device)
+        return special_image_mask
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        pixel_values: torch.FloatTensor,
+        input_ids: torch.FloatTensor,
+        attention_mask: torch.LongTensor | None = None,
+        decoder_input_ids: torch.LongTensor | None = None,
+        decoder_attention_mask: torch.LongTensor | None = None,
+        labels: torch.LongTensor | None = None,
+        interpolate_pos_encoding: bool = False,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | Blip2ForConditionalGenerationModelOutput:
+        r"""
+        decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*):
+            Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also
+            be used by default.
+
+            Only relevant in case an encoder-decoder language model (like T5) is used.
+
+        Examples:
+
+        ```python
+        >>> from PIL import Image
+        >>> import httpx
+        >>> from io import BytesIO
+        >>> from transformers import Blip2Processor, Blip2Model
+        >>> import torch
+
+        >>> device = "cuda" if torch.cuda.is_available() else "cpu"
+
+        >>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
+        >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b", dtype=torch.float16)
+        >>> model.to(device)  # doctest: +IGNORE_RESULT
+
+        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+        >>> with httpx.stream("GET", url) as response:
+        ...     image = Image.open(BytesIO(response.read()))
+
+        >>> prompt = "Question: how many cats are there? Answer:"
+        >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device, torch.float16)
+
+        >>> outputs = model(**inputs)
+        ```"""
+
+        # step 1: forward the images through the vision encoder,
+        # to get image embeddings of shape (batch_size, seq_len, hidden_size)
+        vision_outputs = self.vision_model(
+            pixel_values=pixel_values,
+            interpolate_pos_encoding=interpolate_pos_encoding,
+            **kwargs,
+        )
+        image_embeds = vision_outputs[0]
+
+        # step 2: forward the query tokens through the QFormer, using the image embeddings for cross-attention
+        image_attention_mask = torch.ones(image_embeds.size()[:-1], dtype=torch.long, device=image_embeds.device)
+
+        query_tokens = self.query_tokens.expand(image_embeds.shape[0], -1, -1)
+        query_outputs = self.qformer(
+            query_embeds=query_tokens,
+            encoder_hidden_states=image_embeds,
+            encoder_attention_mask=image_attention_mask,
+            **kwargs,
+        )
+        query_output = query_outputs[0]
+
+        # Qformer is kept in fp32, we downcast the output back if needed
+        if query_output.dtype != image_embeds.dtype:
+            query_output = query_output.to(image_embeds.dtype)
+
+        # step 3: use the language model, conditioned on the query outputs and the prompt
+        language_model_inputs = self.language_projection(query_output)
+
+        inputs_embeds = self.language_model.get_input_embeddings()(input_ids)
+
+        if attention_mask is None:
+            attention_mask = torch.ones_like(input_ids)
+
+        language_model_inputs = language_model_inputs.to(inputs_embeds.device, inputs_embeds.dtype)
+        special_image_mask = self.get_placeholder_mask(input_ids, inputs_embeds=inputs_embeds)
+        inputs_embeds = inputs_embeds.to(language_model_inputs.device).masked_scatter(
+            special_image_mask, language_model_inputs
+        )
+
+        if self.config.use_decoder_only_language_model:
+            outputs = self.language_model(
+                inputs_embeds=inputs_embeds,
+                attention_mask=attention_mask,
+                **kwargs,
+            )
+            logits = outputs[0]
+            loss = None
+            # we compute the loss here since we need to take into account the sequence length of the query embeds
+            if labels is not None:
+                labels = labels.to(logits.device)
+                logits = logits[:, -labels.size(1) :, :]
+                # Shift so that tokens < n predict n
+                shift_logits = logits[..., :-1, :].contiguous()
+                shift_labels = labels[..., 1:].contiguous().to(logits.device)
+
+                # Flatten the tokens
+                loss_fct = CrossEntropyLoss(reduction="mean")
+
+                loss = loss_fct(shift_logits.view(-1, self.config.text_config.vocab_size), shift_labels.view(-1))
+        else:
+            outputs = self.language_model(
+                inputs_embeds=inputs_embeds,
+                attention_mask=attention_mask,
+                decoder_input_ids=decoder_input_ids,
+                decoder_attention_mask=decoder_attention_mask,
+                labels=labels,
+                return_dict=True,
+                **kwargs,
+            )
+            loss = outputs.loss
+            logits = outputs.logits
+
+        return Blip2ForConditionalGenerationModelOutput(
+            loss=loss,
+            logits=logits,
+            vision_outputs=vision_outputs,
+            qformer_outputs=query_outputs,
+            language_model_outputs=outputs,
+        )
+
+
+@auto_docstring
+class Blip2TextModelWithProjection(Blip2PreTrainedModel):
+    supports_gradient_checkpointing = False
+    _keep_in_fp32_modules = ["query_tokens", "qformer"]
+    _supports_flash_attn = False  # because self.qformer does not support FA2
+
+    def __init__(self, config: Blip2Config):
+        super().__init__(config)
+
+        self.query_tokens = nn.Parameter(torch.zeros(1, config.num_query_tokens, config.qformer_config.hidden_size))
+        self.embeddings = Blip2TextEmbeddings(config.qformer_config)
+        self.qformer = Blip2QFormerModel(config.qformer_config)
+
+        # text projection layer
+        self.text_projection = nn.Linear(config.qformer_config.hidden_size, config.image_text_hidden_size)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embeddings.word_embeddings
+
+    def set_input_embeddings(self, value):
+        self.embeddings.word_embeddings = value
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.Tensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.Tensor | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | Blip2TextModelOutput:
+        r"""
+        Examples:
+
+        ```python
+        >>> import torch
+        >>> from transformers import AutoProcessor, Blip2TextModelWithProjection
+
+        >>> device = "cuda" if torch.cuda.is_available() else "cpu"
+
+        >>> model = Blip2TextModelWithProjection.from_pretrained(
+        ...     "Salesforce/blip2-itm-vit-g", dtype=torch.float16
+        ... )
+
+        >>> model.to(device)  # doctest: +IGNORE_RESULT
+
+        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-itm-vit-g")
+
+        >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], return_tensors="pt").to(device)
+
+        >>> outputs = model(**inputs)
+        >>> text_embeds = outputs.text_embeds
+        >>> print(text_embeds.shape)
+        torch.Size([2, 7, 256])
+        ```"""
+
+        query_embeds = self.embeddings(
+            input_ids=input_ids,
+            position_ids=position_ids,
+        )
+
+        text_outputs = self.qformer(
+            query_embeds=query_embeds,
+            query_length=0,
+            attention_mask=attention_mask,
+            **kwargs,
+        )
+
+        pooled_output = text_outputs[0]
+        pooled_output = pooled_output.to(dtype=self.text_projection.weight.dtype)
+
+        text_embeds = self.text_projection(pooled_output)
+        text_embeds = nn.functional.normalize(text_embeds, dim=-1)
+
+        return Blip2TextModelOutput(
+            text_embeds=text_embeds,
+            last_hidden_state=text_outputs.last_hidden_state,
+            hidden_states=text_outputs.hidden_states,
+            attentions=text_outputs.attentions,
+        )
+
+
+@auto_docstring
+class Blip2VisionModelWithProjection(Blip2PreTrainedModel):
+    main_input_name = "pixel_values"
+    input_modalities = ("image",)
+    _keep_in_fp32_modules = ["query_tokens", "qformer"]
+    _supports_flash_attn = False  # because self.qformer does not support FA2
+
+    def __init__(self, config: Blip2Config):
+        super().__init__(config)
+
+        self.vision_model = Blip2VisionModel._from_config(config.vision_config)
+
+        self.query_tokens = nn.Parameter(torch.zeros(1, config.num_query_tokens, config.qformer_config.hidden_size))
+        self.qformer = Blip2QFormerModel._from_config(config.qformer_config)
+
+        # vision projection layer
+        self.vision_projection = nn.Linear(config.qformer_config.hidden_size, config.image_text_hidden_size)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self) -> nn.Module:
+        return self.vision_model.embeddings.patch_embedding
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        pixel_values: torch.FloatTensor | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | Blip2VisionModelOutput:
+        r"""
+        Examples:
+
+        ```python
+        >>> import torch
+        >>> from transformers import AutoProcessor, Blip2VisionModelWithProjection
+        >>> from transformers.image_utils import load_image
+
+        >>> device = "cuda" if torch.cuda.is_available() else "cpu"
+
+        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-itm-vit-g")
+        >>> model = Blip2VisionModelWithProjection.from_pretrained(
+        ...     "Salesforce/blip2-itm-vit-g", dtype=torch.float16
+        ... )
+        >>> model.to(device)  # doctest: +IGNORE_RESULT
+
+        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+        >>> image = load_image(url)
+
+        >>> inputs = processor(images=image, return_tensors="pt").to(device, torch.float16)
+
+        >>> with torch.inference_mode():
+        ...     outputs = model(**inputs)
+        >>> image_embeds = outputs.image_embeds
+        >>> print(image_embeds.shape)
+        torch.Size([1, 32, 256])
+        ```"""
+        vision_outputs = self.vision_model(
+            pixel_values=pixel_values,
+            **kwargs,
+        )
+
+        pooled_output = vision_outputs[0]
+        image_attention_mask = torch.ones(pooled_output.size()[:-1], dtype=torch.long, device=pooled_output.device)
+        query_tokens = self.query_tokens.expand(pooled_output.shape[0], -1, -1)
+
+        query_outputs = self.qformer(
+            query_embeds=query_tokens,
+            encoder_hidden_states=pooled_output,
+            encoder_attention_mask=image_attention_mask,
+            **kwargs,
+        )
+
+        embeds = query_outputs[0]
+        embeds = embeds.to(dtype=self.vision_projection.weight.dtype)
+        image_embeds = self.vision_projection(embeds)
+        image_embeds = nn.functional.normalize(image_embeds, dim=-1)
+
+        return Blip2VisionModelOutput(
+            image_embeds=image_embeds,
+            last_hidden_state=vision_outputs.last_hidden_state,
+            hidden_states=vision_outputs.hidden_states,
+            attentions=vision_outputs.attentions,
+        )
+
+
+@auto_docstring(
+    custom_intro="""
+    BLIP-2 Model for generating text given an image and an optional text prompt. The model consists of a vision
+    encoder, Querying Transformer (Q-Former) and a language model.
+
+    One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue
+    the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token.
+
+    <Tip>
+
+    Note that Flan-T5 checkpoints cannot be cast to float16. They are pre-trained using bfloat16.
+
+    </Tip>
+    """
+)
+class Blip2ForConditionalGeneration(Blip2PreTrainedModel, GenerationMixin):
+    config: Blip2Config
+    main_input_name = "pixel_values"
+
+    _can_compile_fullgraph = True
+    _keep_in_fp32_modules = ["query_tokens", "qformer"]
+    _supports_flash_attn = False  # because self.qformer does not support FA2
+
+    def __init__(self, config: Blip2Config):
+        super().__init__(config)
+
+        self.vision_model = Blip2VisionModel._from_config(config.vision_config)
+
+        self.query_tokens = nn.Parameter(torch.zeros(1, config.num_query_tokens, config.qformer_config.hidden_size))
+        self.qformer = Blip2QFormerModel._from_config(config.qformer_config)
+
+        self.language_projection = nn.Linear(config.qformer_config.hidden_size, config.text_config.hidden_size)
+        if config.use_decoder_only_language_model:
+            language_model = AutoModelForCausalLM.from_config(config.text_config)
+        else:
+            language_model = AutoModelForSeq2SeqLM.from_config(config.text_config)
+
+        self.language_model = language_model
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def set_output_embeddings(self, new_embeddings):
+        self.language_model.set_output_embeddings(new_embeddings)
+
+    def get_output_embeddings(self) -> nn.Module:
+        return self.language_model.get_output_embeddings()
+
+    def get_encoder(self, modality=None):
+        if modality is None:
+            return self.language_model.get_encoder()
+        else:
+            return super().get_encoder(modality=modality)
+
+    def _preprocess_accelerate(self):
+        r"""
+        Some pre-processing hacks to make the model `accelerate` compatible. Check
+        https://github.com/huggingface/transformers/pull/21707 for more details.
+        """
+        hf_device_map = self.hf_device_map
+
+        if len(hf_device_map) > 1 and "language_model" not in hf_device_map and torch.cuda.device_count() > 1:
+            # warn users about unexpected behavior when using multi-GPU + BLIP-2 + `accelerate`.
+            logger.warning(
+                "The `language_model` is not in the `hf_device_map` dictionary and you are running your script"
+                " in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`."
+                " Please pass a `device_map` that contains `language_model` to remove this warning."
+                " Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for"
+                " more details on creating a `device_map` for large models.",
+            )
+
+        if hasattr(self.language_model, "_hf_hook"):
+            self.language_model._hf_hook.io_same_device = True  # For `generate` compatibility
+
+    @can_return_tuple
+    @auto_docstring
+    def get_image_features(
+        self,
+        pixel_values: torch.FloatTensor,
+        interpolate_pos_encoding: bool | None = False,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutputWithVisionQformerOutputs:
+        r"""
+        pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`):
+            The tensors corresponding to the input images.
+        """
+        # step 1: forward the images through the vision encoder,
+        # to get image embeddings of shape (batch_size, seq_len, hidden_size)
+        vision_outputs: BaseModelOutputWithPooling = self.vision_model(
+            pixel_values=pixel_values,
+            interpolate_pos_encoding=interpolate_pos_encoding,
+            return_dict=True,
+            **kwargs,
+        )
+        vision_outputs = BaseModelOutputWithVisionQformerOutputs(
+            last_hidden_state=vision_outputs.last_hidden_state,
+            pooler_output=vision_outputs.pooler_output,
+            hidden_states=vision_outputs.hidden_states,
+            attentions=vision_outputs.attentions,
+            vision_outputs=vision_outputs,
+            qformer_outputs=None,
+        )
+        image_embeds = vision_outputs[0]
+
+        # step 2: forward the query tokens through the QFormer, using the image embeddings for cross-attention
+        image_attention_mask = torch.ones(image_embeds.size()[:-1], dtype=torch.long, device=image_embeds.device)
+
+        query_tokens = self.query_tokens.expand(image_embeds.shape[0], -1, -1)
+        qformer_outputs = self.qformer(
+            query_embeds=query_tokens,
+            encoder_hidden_states=image_embeds,
+            encoder_attention_mask=image_attention_mask,
+            return_dict=True,
+        )
+        vision_outputs.qformer_outputs = qformer_outputs
+        query_output = qformer_outputs[0]
+
+        # Qformer is kept in fp32, we downcast the output back if needed
+        if query_output.dtype != image_embeds.dtype:
+            query_output = query_output.to(image_embeds.dtype)
+
+        # step 3: use the language model, conditioned on the query outputs and the prompt
+        image_features = self.language_projection(query_output)
+        vision_outputs.pooler_output = image_features
+
+        return vision_outputs
+
+    def get_placeholder_mask(self, input_ids: torch.LongTensor, inputs_embeds: torch.FloatTensor):
+        """
+        Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`.
+        """
+        if input_ids is None:
+            special_image_mask = inputs_embeds == self.get_input_embeddings()(
+                torch.tensor(self.config.image_token_id, dtype=torch.long, device=inputs_embeds.device)
+            )
+            special_image_mask = special_image_mask.all(-1)
+        else:
+            special_image_mask = input_ids == self.config.image_token_id
+
+        special_image_mask = special_image_mask.unsqueeze(-1).expand_as(inputs_embeds).to(inputs_embeds.device)
+        return special_image_mask
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        pixel_values: torch.FloatTensor,
+        input_ids: torch.LongTensor,
+        attention_mask: torch.LongTensor | None = None,
+        decoder_input_ids: torch.LongTensor | None = None,
+        decoder_attention_mask: torch.LongTensor | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        labels: torch.LongTensor | None = None,
+        interpolate_pos_encoding: bool = False,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | Blip2ForConditionalGenerationModelOutput:
+        r"""
+        input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be
+            provided to serve as text prompt, which the language model can continue.
+
+            Indices can be obtained using [`Blip2Processor`]. See [`Blip2Processor.__call__`] for details.
+
+            [What are input IDs?](../glossary#input-ids)
+        decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*):
+            Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also
+            be used by default.
+
+            Only relevant in case an encoder-decoder language model (like T5) is used.
+
+        Examples:
+
+        Prepare processor, model and image input
+
+        ```python
+        >>> from PIL import Image
+        >>> import httpx
+        >>> from io import BytesIO
+        >>> from transformers import Blip2Processor, Blip2ForConditionalGeneration
+        >>> import torch
+
+        >>> device = "cuda" if torch.cuda.is_available() else "cpu"
+
+        >>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
+        >>> model = Blip2ForConditionalGeneration.from_pretrained(
+        ...     "Salesforce/blip2-opt-2.7b", load_in_8bit=True, device_map={"": 0}, dtype=torch.float16
+        ... )  # doctest: +IGNORE_RESULT
+
+        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+        >>> with httpx.stream("GET", url) as response:
+        ...     image = Image.open(BytesIO(response.read()))
+        ```
+
+        Image captioning (without providing a text prompt):
+
+        ```python
+        >>> inputs = processor(images=image, return_tensors="pt").to(device, torch.float16)
+
+        >>> generated_ids = model.generate(**inputs)
+        >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
+        >>> print(generated_text)
+        two cats laying on a couch
+        ```
+
+        Visual question answering (prompt = question):
+
+        ```python
+        >>> prompt = "Question: how many cats are there? Answer:"
+        >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device="cuda", dtype=torch.float16)
+
+        >>> generated_ids = model.generate(**inputs)
+        >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
+        >>> print(generated_text)
+        two
+        ```
+
+        Note that int8 inference is also supported through [bitsandbytes](https://github.com/TimDettmers/bitsandbytes).
+        This greatly reduces the amount of memory used by the model while maintaining the same performance.
+
+        ```python
+        >>> model = Blip2ForConditionalGeneration.from_pretrained(
+        ...     "Salesforce/blip2-opt-2.7b", load_in_8bit=True, device_map={"": 0}, dtype=torch.bfloat16
+        ... )  # doctest: +IGNORE_RESULT
+
+        >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device="cuda", dtype=torch.bfloat16)
+
+        >>> generated_ids = model.generate(**inputs)
+        >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
+        >>> print(generated_text)
+        two
+        ```"""
+
+        image_features: BaseModelOutputWithVisionQformerOutputs = self.get_image_features(
+            pixel_values, interpolate_pos_encoding=interpolate_pos_encoding, return_dict=True
+        )
+        language_model_inputs = image_features.pooler_output
+        qformer_outputs = image_features.qformer_outputs
+        vision_outputs = image_features.vision_outputs
+
+        if inputs_embeds is None:
+            inputs_embeds = self.get_input_embeddings()(input_ids)
+
+        if attention_mask is None:
+            attention_mask = torch.ones_like(input_ids)
+
+        language_model_inputs = language_model_inputs.to(inputs_embeds.device, inputs_embeds.dtype)
+        special_image_mask = self.get_placeholder_mask(input_ids, inputs_embeds=inputs_embeds)
+        inputs_embeds = inputs_embeds.to(language_model_inputs.device).masked_scatter(
+            special_image_mask, language_model_inputs
+        )
+
+        if self.config.use_decoder_only_language_model:
+            outputs = self.language_model(
+                inputs_embeds=inputs_embeds,
+                attention_mask=attention_mask,
+                **kwargs,
+            )
+            logits = outputs[0]
+            loss = None
+            # we compute the loss here since we need to take into account the sequence length of the query embeds
+            if labels is not None:
+                labels = labels.to(logits.device)
+                logits = logits[:, -labels.size(1) :, :]
+                # Shift so that tokens < n predict n
+                shift_logits = logits[..., :-1, :].contiguous()
+                shift_labels = labels[..., 1:].contiguous().to(logits.device)
+
+                # Flatten the tokens
+                loss_fct = CrossEntropyLoss(reduction="mean")
+
+                loss = loss_fct(shift_logits.view(-1, self.config.text_config.vocab_size), shift_labels.view(-1))
+        else:
+            kwargs["return_dict"] = True
+            outputs = self.language_model(
+                inputs_embeds=inputs_embeds,
+                attention_mask=attention_mask,
+                decoder_input_ids=decoder_input_ids,
+                decoder_attention_mask=decoder_attention_mask,
+                labels=labels,
+                **kwargs,
+            )
+            loss = outputs.loss
+            logits = outputs.logits
+
+        return Blip2ForConditionalGenerationModelOutput(
+            loss=loss,
+            logits=logits,
+            vision_outputs=vision_outputs,
+            qformer_outputs=qformer_outputs,
+            language_model_outputs=outputs,
+        )
+
+    @torch.no_grad()
+    def generate(
+        self,
+        pixel_values: torch.FloatTensor,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.LongTensor | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        interpolate_pos_encoding: bool = False,
+        **generate_kwargs,
+    ) -> torch.LongTensor:
+        """
+        Overrides `generate` function to be able to use the model as a conditional generator.
+
+        Args:
+            pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width)):
+                Input images to be processed.
+            input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*):
+                The sequence used as a prompt for the generation.
+            attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*):
+                Mask to avoid performing attention on padding token indices
+            inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):
+                Embedded representation of the inputs. Should be float, not int tokens.
+            interpolate_pos_encoding (`bool`, *optional*, defaults to `False`):
+                Whether to interpolate the positional encoding of the image embeddings.
+
+        Returns:
+            captions (list): A list of strings of length batch_size * num_captions.
+        """
+        if hasattr(self, "hf_device_map"):
+            # preprocess for `accelerate`
+            self._preprocess_accelerate()
+
+        batch_size = pixel_values.shape[0]
+        image_embeds = self.vision_model(
+            pixel_values,
+            return_dict=True,
+            interpolate_pos_encoding=interpolate_pos_encoding,
+        ).last_hidden_state
+        image_attention_mask = torch.ones(image_embeds.size()[:-1], dtype=torch.long, device=image_embeds.device)
+
+        query_tokens = self.query_tokens.expand(image_embeds.shape[0], -1, -1)
+        query_outputs = self.qformer(
+            query_embeds=query_tokens,
+            encoder_hidden_states=image_embeds,
+            encoder_attention_mask=image_attention_mask,
+            return_dict=True,
+        )
+        query_output = query_outputs.last_hidden_state
+
+        # Qformer is kept in fp32, we downcast the output back if needed
+        if query_output.dtype != image_embeds.dtype:
+            query_output = query_output.to(image_embeds.dtype)
+
+        language_model_inputs = self.language_projection(query_output)
+
+        if inputs_embeds is None:
+            if input_ids is None:
+                image_tokens = [self.config.image_token_index] * self.config.num_query_tokens
+                start_tokens = image_tokens + [self.config.text_config.bos_token_id]
+                input_ids = torch.tensor([start_tokens], dtype=torch.long, device=image_embeds.device)
+                input_ids = input_ids.repeat(batch_size, 1)
+            inputs_embeds = self.get_input_embeddings()(input_ids)
+
+        if attention_mask is None:
+            attention_mask = torch.ones_like(input_ids)
+
+        if input_ids is None:
+            special_image_mask = inputs_embeds == self.get_input_embeddings()(
+                torch.tensor(self.config.image_token_id, dtype=torch.long, device=inputs_embeds.device)
+            )
+            special_image_mask = special_image_mask.all(-1)
+        else:
+            special_image_mask = input_ids == self.config.image_token_id
+
+        special_image_mask = special_image_mask.unsqueeze(-1).expand_as(inputs_embeds).to(inputs_embeds.device)
+        language_model_inputs = language_model_inputs.to(inputs_embeds.device, inputs_embeds.dtype)
+        inputs_embeds = inputs_embeds.masked_scatter(special_image_mask, language_model_inputs)
+
+        inputs = {"inputs_embeds": inputs_embeds, "attention_mask": attention_mask}
+        if not self.language_model.config.is_encoder_decoder:
+            inputs["input_ids"] = input_ids
+
+        outputs = self.language_model.generate(**inputs, **generate_kwargs)
+
+        return outputs
+
+
+@auto_docstring(
+    custom_intro="""
+    BLIP-2 Model with a vision and text projector, and a classification head on top. The model is used in the context
+    of image-text retrieval. Given an image and a text, the model returns the probability of the text being relevant to
+    the image.
+    """
+)
+class Blip2ForImageTextRetrieval(Blip2PreTrainedModel):
+    main_input_name = "pixel_values"
+    input_modalities = ("image",)
+    _keep_in_fp32_modules = ["query_tokens", "qformer"]
+    _supports_flash_attn = False  # because self.qformer does not support FA2
+
+    def __init__(self, config: Blip2Config):
+        super().__init__(config)
+
+        self.vision_model = Blip2VisionModel._from_config(config.vision_config)
+
+        self.query_tokens = nn.Parameter(torch.zeros(1, config.num_query_tokens, config.qformer_config.hidden_size))
+
+        self.embeddings = Blip2TextEmbeddings(config.qformer_config)
+        self.qformer = Blip2QFormerModel._from_config(config.qformer_config)
+
+        # vision projection layer
+        self.vision_projection = nn.Linear(config.qformer_config.hidden_size, config.image_text_hidden_size)
+
+        # text projection layer
+        self.text_projection = nn.Linear(config.qformer_config.hidden_size, config.image_text_hidden_size)
+
+        # image text matching head
+        self.itm_head = nn.Linear(config.qformer_config.hidden_size, 2)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embeddings.word_embeddings
+
+    def set_input_embeddings(self, value):
+        self.embeddings.word_embeddings = value
+
+    @auto_docstring
+    def forward(
+        self,
+        pixel_values: torch.FloatTensor,
+        input_ids: torch.LongTensor,
+        attention_mask: torch.LongTensor | None = None,
+        use_image_text_matching_head: bool | None = False,
+        output_attentions: bool | None = None,
+        output_hidden_states: bool | None = None,
+        return_dict: bool | None = None,
+        **kwargs,
+    ) -> tuple | Blip2ImageTextMatchingModelOutput:
+        r"""
+        input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be
+            provided to serve as text prompt, which the language model can continue.
+
+            Indices can be obtained using [`Blip2Processor`]. See [`Blip2Processor.__call__`] for details.
+
+            [What are input IDs?](../glossary#input-ids)
+        use_image_text_matching_head (`bool`, *optional*):
+            Whether to return the Image-Text Matching or Contrastive scores.
+
+        Examples:
+
+        ```python
+        >>> import torch
+        >>> from PIL import Image
+        >>> import httpx
+        >>> from io import BytesIO
+        >>> from transformers import AutoProcessor, Blip2ForImageTextRetrieval
+
+        >>> device = "cuda" if torch.cuda.is_available() else "cpu"
+
+        >>> model = Blip2ForImageTextRetrieval.from_pretrained("Salesforce/blip2-itm-vit-g", dtype=torch.float16)
+        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-itm-vit-g")
+
+        >>> model.to(device)  # doctest: +IGNORE_RESULT
+
+        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+        >>> with httpx.stream("GET", url) as response:
+        ...     image = Image.open(BytesIO(response.read()))
+        >>> text = "two cats laying on a pink blanket"
+
+        >>> inputs = processor(images=image, text=text, return_tensors="pt").to(device, torch.float16)
+        >>> itm_out = model(**inputs, use_image_text_matching_head=True)
+        >>> logits_per_image = torch.nn.functional.softmax(itm_out.logits_per_image, dim=1)
+        >>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
+
+        >>> print(f"{probs[0][0]:.1%} that image 0 is not '{text}'")
+        26.9% that image 0 is not 'two cats laying on a pink blanket'
+
+        >>> print(f"{probs[0][1]:.1%} that image 0 is '{text}'")
+        73.0% that image 0 is 'two cats laying on a pink blanket'
+
+        >>> texts = ["a photo of a cat", "a photo of a dog"]
+
+        >>> inputs = processor(images=image, text=texts, return_tensors="pt").to(device, torch.float16)
+        >>> itc_out = model(**inputs, use_image_text_matching_head=False)
+        >>> logits_per_image = itc_out.logits_per_image  # this is the image-text similarity score
+        >>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
+
+        >>> print(f"{probs[0][0]:.1%} that image 0 is '{texts[0]}'")
+        55.3% that image 0 is 'a photo of a cat'
+
+        >>> print(f"{probs[0][1]:.1%} that image 0 is '{texts[1]}'")
+        44.7% that image 0 is 'a photo of a dog'
+        ```
+        """
+        return_dict = return_dict if return_dict is not None else self.config.return_dict
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+
+        vision_outputs = self.vision_model(
+            pixel_values=pixel_values,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        image_embeds = vision_outputs[0]
+        image_attention_mask = torch.ones(image_embeds.size()[:-1], dtype=torch.long, device=image_embeds.device)
+
+        if use_image_text_matching_head:
+            query_tokens = self.query_tokens.expand(image_embeds.shape[0], -1, -1)
+            if self.config.image_token_index is not None:
+                input_ids = input_ids[:, self.config.num_query_tokens :]
+            else:
+                query_attention_mask = torch.ones(
+                    query_tokens.size()[:-1], dtype=torch.long, device=query_tokens.device
+                )
+                attention_mask = torch.cat([query_attention_mask, attention_mask], dim=1)
+
+            query_embeds = self.embeddings(
+                input_ids=input_ids,
+                query_embeds=query_tokens,
+            )
+
+            text_outputs = self.qformer(
+                query_embeds=query_embeds,
+                query_length=query_tokens.shape[1],
+                attention_mask=attention_mask,
+                encoder_hidden_states=image_embeds,
+                encoder_attention_mask=image_attention_mask,
+                return_dict=return_dict,
+            )
+            text_embeds = text_outputs[0] if not return_dict else text_outputs.last_hidden_state
+            text_embeds = text_embeds.to(dtype=self.itm_head.weight.dtype)
+
+            output = self.itm_head(text_embeds[:, : query_tokens.size(1), :])
+            logits_per_image = output.mean(dim=1)
+            logits_per_text = logits_per_image.t()
+        else:
+            query_tokens = self.query_tokens.expand(image_embeds.shape[0], -1, -1)
+            query_outputs = self.qformer(
+                query_embeds=query_tokens,
+                encoder_hidden_states=image_embeds,
+                encoder_attention_mask=image_attention_mask,
+                return_dict=return_dict,
+            )
+            image_embeds = query_outputs[0] if not return_dict else query_outputs.last_hidden_state
+            image_embeds = image_embeds.to(dtype=self.vision_projection.weight.dtype)
+
+            if self.config.image_token_index is not None:
+                input_ids = input_ids[:, self.config.num_query_tokens :]
+                attention_mask = attention_mask[:, self.config.num_query_tokens :]
+
+            query_embeds = self.embeddings(
+                input_ids=input_ids,
+            )
+            text_outputs = self.qformer(
+                query_embeds=query_embeds,
+                query_length=0,
+                attention_mask=attention_mask,
+                return_dict=return_dict,
+            )
+            question_embeds = text_outputs[0] if not return_dict else text_outputs.last_hidden_state
+            question_embeds = question_embeds.to(dtype=self.text_projection.weight.dtype)
+
+            # normalized features
+            image_embeds = nn.functional.normalize(self.vision_projection(image_embeds), dim=-1)
+            text_embeds = nn.functional.normalize(self.text_projection(question_embeds[:, 0, :]), dim=-1)
+
+            # cosine similarity as logits
+            logits_per_image = torch.matmul(image_embeds, text_embeds.t())
+            logits_per_image, _ = logits_per_image.max(dim=1)
+
+            logits_per_text = logits_per_image.t()
+
+        if not return_dict:
+            output = (logits_per_image, logits_per_text, text_embeds, image_embeds, text_outputs, vision_outputs)
+            return output
+
+        return Blip2ImageTextMatchingModelOutput(
+            logits_per_image=logits_per_image,
+            logits_per_text=logits_per_text,
+            text_embeds=text_embeds,
+            image_embeds=image_embeds,
+            text_model_output=text_outputs,
+            vision_model_output=vision_outputs,
+        )
+
+
+__all__ = [
+    "Blip2Model",
+    "Blip2VisionModelWithProjection",
+    "Blip2QFormerModel",
+    "Blip2PreTrainedModel",
+    "Blip2ForConditionalGeneration",
+    "Blip2ForImageTextRetrieval",
+    "Blip2VisionModel",
+    "Blip2TextModelWithProjection",
+]

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/blip_2/processing_blip_2.py

@@ -0,0 +1,114 @@
+# Copyright 2023 The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Processor class for BLIP-2.
+"""
+
+from ...image_processing_utils import BatchFeature
+from ...image_utils import ImageInput
+from ...processing_utils import ProcessingKwargs, ProcessorMixin, Unpack
+from ...tokenization_utils_base import AddedToken, BatchEncoding, PreTokenizedInput, TextInput
+from ...utils import auto_docstring, logging
+
+
+logger = logging.get_logger(__name__)
+
+
+class Blip2ProcessorKwargs(ProcessingKwargs, total=False):
+    _defaults = {
+        "text_kwargs": {
+            "add_special_tokens": True,
+            "padding": False,
+            "stride": 0,
+            "return_overflowing_tokens": False,
+            "return_special_tokens_mask": False,
+            "return_offsets_mapping": False,
+            "return_token_type_ids": False,
+            "return_length": False,
+            "verbose": True,
+        },
+    }
+
+
+@auto_docstring
+class Blip2Processor(ProcessorMixin):
+    def __init__(self, image_processor, tokenizer, num_query_tokens=None, **kwargs):
+        r"""
+        num_query_tokens (`int`, *optional*):
+            Number of tokens used by the Qformer as queries, should be same as in model's config.
+        """
+        tokenizer.return_token_type_ids = False
+        if not hasattr(tokenizer, "image_token"):
+            self.image_token = AddedToken("<image>", normalized=False, special=True)
+            tokenizer.add_tokens([self.image_token], special_tokens=True)
+        else:
+            self.image_token = tokenizer.image_token
+        self.num_query_tokens = num_query_tokens
+
+        super().__init__(image_processor, tokenizer)
+
+    @auto_docstring
+    def __call__(
+        self,
+        images: ImageInput | None = None,
+        text: str | list[str] | TextInput | PreTokenizedInput | None = None,
+        **kwargs: Unpack[Blip2ProcessorKwargs],
+    ) -> BatchEncoding:
+        if images is None and text is None:
+            raise ValueError("You have to specify either images or text.")
+        output_kwargs = self._merge_kwargs(
+            Blip2ProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+
+        # BC for explicit return_tensors
+        return_tensors = output_kwargs["text_kwargs"].pop("return_tensors", None)
+        max_length = output_kwargs["text_kwargs"].pop("max_length", None)
+        if max_length is not None:
+            output_kwargs["text_kwargs"]["max_length"] = max_length - self.num_query_tokens
+
+        encoding = BatchFeature(tensor_type=return_tensors)
+        if text is not None:
+            if isinstance(text, str):
+                text = [text]
+            elif not isinstance(text, list) and not isinstance(text[0], str):
+                raise ValueError("Invalid input text. Please provide a string, or a list of strings")
+
+            # We need this hacky manipulation because BLIP expects image tokens to be at the beginning even before BOS token
+            text_encoding = self.tokenizer(text, **output_kwargs["text_kwargs"])
+
+            if images is not None and self.num_query_tokens is not None:
+                # Image tokens should not be padded/truncated or prepended with special BOS token
+                image_tokens = self.image_token.content * self.num_query_tokens
+                output_kwargs["text_kwargs"]["add_special_tokens"] = False
+                output_kwargs["text_kwargs"]["padding"] = False
+                output_kwargs["text_kwargs"]["truncation"] = False
+                image_text_encoding = self.tokenizer(image_tokens, **output_kwargs["text_kwargs"])
+                for k in text_encoding:
+                    text_encoding[k] = [image_text_encoding[k] + sample for sample in text_encoding[k]]
+            encoding.update(text_encoding)
+
+        # Now add pixel_values encoding. If we also have text_encoding, update image encoding and return it.
+        # else, return the text encoding.
+        if images is not None:
+            image_encoding = self.image_processor(images, **output_kwargs["images_kwargs"])
+            encoding.update(image_encoding)
+
+        # Cast to desired return tensors type
+        encoding = BatchFeature(encoding, tensor_type=return_tensors)
+        return encoding
+
+
+__all__ = ["Blip2Processor"]

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/deberta_v2/__init__.py

@@ -0,0 +1,28 @@
+# Copyright 2024 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import TYPE_CHECKING
+
+from ...utils import _LazyModule
+from ...utils.import_utils import define_import_structure
+
+
+if TYPE_CHECKING:
+    from .configuration_deberta_v2 import *
+    from .modeling_deberta_v2 import *
+    from .tokenization_deberta_v2 import *
+else:
+    import sys
+
+    _file = globals()["__file__"]
+    sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/deberta_v2/configuration_deberta_v2.py

@@ -0,0 +1,94 @@
+# Copyright 2020, Microsoft and the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""DeBERTa-v2 model configuration"""
+
+from huggingface_hub.dataclasses import strict
+
+from ...configuration_utils import PreTrainedConfig
+from ...utils import auto_docstring
+
+
+@auto_docstring(checkpoint="microsoft/deberta-v2-xlarge")
+@strict
+class DebertaV2Config(PreTrainedConfig):
+    r"""
+    relative_attention (`bool`, *optional*, defaults to `True`):
+        Whether use relative position encoding.
+    max_relative_positions (`int`, *optional*, defaults to -1):
+        The range of relative positions `[-max_position_embeddings, max_position_embeddings]`. Use the same value
+        as `max_position_embeddings`.
+    position_biased_input (`bool`, *optional*, defaults to `True`):
+        Whether add absolute position embedding to content embedding.
+    pos_att_type (`list[str]`, *optional*):
+        The type of relative position attention, it can be a combination of `["p2c", "c2p"]`, e.g. `["p2c"]`,
+        `["p2c", "c2p"]`, `["p2c", "c2p"]`.
+    pooler_dropout (`float`, *optional*, defaults to `0`):
+        Dropout rate in the pooler module.
+    pooler_hidden_act (`str`, *optional*, defaults to `"gelu"`):
+        Activation function used in the dropout module.
+    legacy (`bool`, *optional*, defaults to `True`):
+        Whether or not the model should use the legacy `LegacyDebertaOnlyMLMHead`, which does not work properly
+        for mask infilling tasks.
+
+    Example:
+
+    ```python
+    >>> from transformers import DebertaV2Config, DebertaV2Model
+
+    >>> # Initializing a DeBERTa-v2 microsoft/deberta-v2-xlarge style configuration
+    >>> configuration = DebertaV2Config()
+
+    >>> # Initializing a model (with random weights) from the microsoft/deberta-v2-xlarge style configuration
+    >>> model = DebertaV2Model(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "deberta-v2"
+
+    vocab_size: int = 128100
+    hidden_size: int = 1536
+    num_hidden_layers: int = 24
+    num_attention_heads: int = 24
+    intermediate_size: int = 6144
+    hidden_act: str = "gelu"
+    hidden_dropout_prob: float | int = 0.1
+    attention_probs_dropout_prob: float | int = 0.1
+    max_position_embeddings: int = 512
+    type_vocab_size: int = 0
+    initializer_range: float = 0.02
+    layer_norm_eps: float = 1e-7
+    relative_attention: bool = False
+    max_relative_positions: int = -1
+    pad_token_id: int | None = 0
+    bos_token_id: int | None = None
+    eos_token_id: int | list[int] | None = None
+    position_biased_input: bool = True
+    pos_att_type: str | list[str] | None = None
+    pooler_dropout: float | int = 0.0
+    pooler_hidden_act: str = "gelu"
+    legacy: bool = True
+    tie_word_embeddings: bool = True
+
+    def __post_init__(self, **kwargs):
+        # Backwards compatibility
+        if isinstance(self.pos_att_type, str):
+            self.pos_att_type = [x.strip() for x in self.pos_att_type.lower().split("|")]
+
+        self.pooler_hidden_size = kwargs.get("pooler_hidden_size", self.hidden_size)
+        super().__post_init__(**kwargs)
+
+
+__all__ = ["DebertaV2Config"]

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py

@@ -0,0 +1,1361 @@
+# Copyright 2020 Microsoft and the Hugging Face Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch DeBERTa-v2 model."""
+
+from collections.abc import Sequence
+
+import torch
+from torch import nn
+from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, LayerNorm, MSELoss
+
+from ... import initialization as init
+from ...activations import ACT2FN
+from ...modeling_layers import GradientCheckpointingLayer
+from ...modeling_outputs import (
+    BaseModelOutput,
+    MaskedLMOutput,
+    MultipleChoiceModelOutput,
+    QuestionAnsweringModelOutput,
+    SequenceClassifierOutput,
+    TokenClassifierOutput,
+)
+from ...modeling_utils import PreTrainedModel
+from ...utils import auto_docstring, logging
+from .configuration_deberta_v2 import DebertaV2Config
+
+
+logger = logging.get_logger(__name__)
+
+
+# Copied from transformers.models.deberta.modeling_deberta.DebertaSelfOutput with DebertaLayerNorm->LayerNorm
+class DebertaV2SelfOutput(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+        self.LayerNorm = LayerNorm(config.hidden_size, config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+
+    def forward(self, hidden_states, input_tensor):
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.dropout(hidden_states)
+        hidden_states = self.LayerNorm(hidden_states + input_tensor)
+        return hidden_states
+
+
+@torch.jit.script
+def make_log_bucket_position(relative_pos, bucket_size: int, max_position: int):
+    sign = torch.sign(relative_pos)
+    mid = bucket_size // 2
+    abs_pos = torch.where(
+        (relative_pos < mid) & (relative_pos > -mid),
+        torch.tensor(mid - 1).type_as(relative_pos),
+        torch.abs(relative_pos),
+    )
+    log_pos = (
+        torch.ceil(torch.log(abs_pos / mid) / torch.log(torch.tensor((max_position - 1) / mid)) * (mid - 1)) + mid
+    )
+    bucket_pos = torch.where(abs_pos <= mid, relative_pos.type_as(log_pos), log_pos * sign)
+    return bucket_pos
+
+
+def build_relative_position(query_layer, key_layer, bucket_size: int = -1, max_position: int = -1):
+    """
+    Build relative position according to the query and key
+
+    We assume the absolute position of query \\(P_q\\) is range from (0, query_size) and the absolute position of key
+    \\(P_k\\) is range from (0, key_size), The relative positions from query to key is \\(R_{q \\rightarrow k} = P_q -
+    P_k\\)
+
+    Args:
+        query_size (int): the length of query
+        key_size (int): the length of key
+        bucket_size (int): the size of position bucket
+        max_position (int): the maximum allowed absolute position
+        device (`torch.device`): the device on which tensors will be created.
+
+    Return:
+        `torch.LongTensor`: A tensor with shape [1, query_size, key_size]
+    """
+    query_size = query_layer.size(-2)
+    key_size = key_layer.size(-2)
+
+    q_ids = torch.arange(query_size, dtype=torch.long, device=query_layer.device)
+    k_ids = torch.arange(key_size, dtype=torch.long, device=key_layer.device)
+    rel_pos_ids = q_ids[:, None] - k_ids[None, :]
+    if bucket_size > 0 and max_position > 0:
+        rel_pos_ids = make_log_bucket_position(rel_pos_ids, bucket_size, max_position)
+    rel_pos_ids = rel_pos_ids.to(torch.long)
+    rel_pos_ids = rel_pos_ids[:query_size, :]
+    rel_pos_ids = rel_pos_ids.unsqueeze(0)
+    return rel_pos_ids
+
+
+@torch.jit.script
+def c2p_dynamic_expand(c2p_pos, query_layer, relative_pos):
+    return c2p_pos.expand([query_layer.size(0), query_layer.size(1), query_layer.size(2), relative_pos.size(-1)])
+
+
+@torch.jit.script
+def p2c_dynamic_expand(c2p_pos, query_layer, key_layer):
+    return c2p_pos.expand([query_layer.size(0), query_layer.size(1), key_layer.size(-2), key_layer.size(-2)])
+
+
+@torch.jit.script
+def pos_dynamic_expand(pos_index, p2c_att, key_layer):
+    return pos_index.expand(p2c_att.size()[:2] + (pos_index.size(-2), key_layer.size(-2)))
+
+
+@torch.jit.script
+def scaled_size_sqrt(query_layer: torch.Tensor, scale_factor: int):
+    return torch.sqrt(torch.tensor(query_layer.size(-1), dtype=torch.float) * scale_factor)
+
+
+@torch.jit.script
+def build_rpos(query_layer, key_layer, relative_pos, position_buckets: int, max_relative_positions: int):
+    if key_layer.size(-2) != query_layer.size(-2):
+        return build_relative_position(
+            key_layer,
+            key_layer,
+            bucket_size=position_buckets,
+            max_position=max_relative_positions,
+        )
+    else:
+        return relative_pos
+
+
+class DisentangledSelfAttention(nn.Module):
+    """
+    Disentangled self-attention module
+
+    Parameters:
+        config (`DebertaV2Config`):
+            A model config class instance with the configuration to build a new model. The schema is similar to
+            *BertConfig*, for more details, please refer [`DebertaV2Config`]
+
+    """
+
+    def __init__(self, config):
+        super().__init__()
+        if config.hidden_size % config.num_attention_heads != 0:
+            raise ValueError(
+                f"The hidden size ({config.hidden_size}) is not a multiple of the number of attention "
+                f"heads ({config.num_attention_heads})"
+            )
+        self.num_attention_heads = config.num_attention_heads
+        _attention_head_size = config.hidden_size // config.num_attention_heads
+        self.attention_head_size = getattr(config, "attention_head_size", _attention_head_size)
+        self.all_head_size = self.num_attention_heads * self.attention_head_size
+        self.query_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=True)
+        self.key_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=True)
+        self.value_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=True)
+
+        self.share_att_key = getattr(config, "share_att_key", False)
+        self.pos_att_type = config.pos_att_type if config.pos_att_type is not None else []
+        self.relative_attention = getattr(config, "relative_attention", False)
+
+        if self.relative_attention:
+            self.position_buckets = getattr(config, "position_buckets", -1)
+            self.max_relative_positions = getattr(config, "max_relative_positions", -1)
+            if self.max_relative_positions < 1:
+                self.max_relative_positions = config.max_position_embeddings
+            self.pos_ebd_size = self.max_relative_positions
+            if self.position_buckets > 0:
+                self.pos_ebd_size = self.position_buckets
+
+            self.pos_dropout = nn.Dropout(config.hidden_dropout_prob)
+
+            if not self.share_att_key:
+                if "c2p" in self.pos_att_type:
+                    self.pos_key_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=True)
+                if "p2c" in self.pos_att_type:
+                    self.pos_query_proj = nn.Linear(config.hidden_size, self.all_head_size)
+
+        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
+
+    def transpose_for_scores(self, x, attention_heads) -> torch.Tensor:
+        new_x_shape = x.size()[:-1] + (attention_heads, -1)
+        x = x.view(new_x_shape)
+        return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1))
+
+    def forward(
+        self,
+        hidden_states,
+        attention_mask,
+        output_attentions=False,
+        query_states=None,
+        relative_pos=None,
+        rel_embeddings=None,
+    ):
+        """
+        Call the module
+
+        Args:
+            hidden_states (`torch.FloatTensor`):
+                Input states to the module usually the output from previous layer, it will be the Q,K and V in
+                *Attention(Q,K,V)*
+
+            attention_mask (`torch.BoolTensor`):
+                An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum
+                sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j*
+                th token.
+
+            output_attentions (`bool`, *optional*):
+                Whether return the attention matrix.
+
+            query_states (`torch.FloatTensor`, *optional*):
+                The *Q* state in *Attention(Q,K,V)*.
+
+            relative_pos (`torch.LongTensor`):
+                The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with
+                values ranging in [*-max_relative_positions*, *max_relative_positions*].
+
+            rel_embeddings (`torch.FloatTensor`):
+                The embedding of relative distances. It's a tensor of shape [\\(2 \\times
+                \\text{max_relative_positions}\\), *hidden_size*].
+
+
+        """
+        if query_states is None:
+            query_states = hidden_states
+        query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads)
+        key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads)
+        value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads)
+
+        rel_att = None
+        # Take the dot product between "query" and "key" to get the raw attention scores.
+        scale_factor = 1
+        if "c2p" in self.pos_att_type:
+            scale_factor += 1
+        if "p2c" in self.pos_att_type:
+            scale_factor += 1
+        scale = scaled_size_sqrt(query_layer, scale_factor)
+        attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype))
+        if self.relative_attention:
+            rel_embeddings = self.pos_dropout(rel_embeddings)
+            rel_att = self.disentangled_attention_bias(
+                query_layer, key_layer, relative_pos, rel_embeddings, scale_factor
+            )
+
+        if rel_att is not None:
+            attention_scores = attention_scores + rel_att
+        attention_scores = attention_scores.view(
+            -1, self.num_attention_heads, attention_scores.size(-2), attention_scores.size(-1)
+        )
+
+        attention_mask = attention_mask.bool()
+        attention_scores = attention_scores.masked_fill(~(attention_mask), torch.finfo(query_layer.dtype).min)
+        # bsz x height x length x dimension
+        attention_probs = nn.functional.softmax(attention_scores, dim=-1)
+
+        attention_probs = self.dropout(attention_probs)
+        context_layer = torch.bmm(
+            attention_probs.view(-1, attention_probs.size(-2), attention_probs.size(-1)), value_layer
+        )
+        context_layer = (
+            context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1))
+            .permute(0, 2, 1, 3)
+            .contiguous()
+        )
+        new_context_layer_shape = context_layer.size()[:-2] + (-1,)
+        context_layer = context_layer.view(new_context_layer_shape)
+        if not output_attentions:
+            return (context_layer, None)
+        return (context_layer, attention_probs)
+
+    def disentangled_attention_bias(self, query_layer, key_layer, relative_pos, rel_embeddings, scale_factor):
+        if relative_pos is None:
+            relative_pos = build_relative_position(
+                query_layer,
+                key_layer,
+                bucket_size=self.position_buckets,
+                max_position=self.max_relative_positions,
+            )
+        if relative_pos.dim() == 2:
+            relative_pos = relative_pos.unsqueeze(0).unsqueeze(0)
+        elif relative_pos.dim() == 3:
+            relative_pos = relative_pos.unsqueeze(1)
+        # bsz x height x query x key
+        elif relative_pos.dim() != 4:
+            raise ValueError(f"Relative position ids must be of dim 2 or 3 or 4. {relative_pos.dim()}")
+
+        att_span = self.pos_ebd_size
+        relative_pos = relative_pos.to(device=query_layer.device, dtype=torch.long)
+
+        rel_embeddings = rel_embeddings[0 : att_span * 2, :].unsqueeze(0)
+        if self.share_att_key:
+            pos_query_layer = self.transpose_for_scores(
+                self.query_proj(rel_embeddings), self.num_attention_heads
+            ).repeat(query_layer.size(0) // self.num_attention_heads, 1, 1)
+            pos_key_layer = self.transpose_for_scores(self.key_proj(rel_embeddings), self.num_attention_heads).repeat(
+                query_layer.size(0) // self.num_attention_heads, 1, 1
+            )
+        else:
+            if "c2p" in self.pos_att_type:
+                pos_key_layer = self.transpose_for_scores(
+                    self.pos_key_proj(rel_embeddings), self.num_attention_heads
+                ).repeat(query_layer.size(0) // self.num_attention_heads, 1, 1)  # .split(self.all_head_size, dim=-1)
+            if "p2c" in self.pos_att_type:
+                pos_query_layer = self.transpose_for_scores(
+                    self.pos_query_proj(rel_embeddings), self.num_attention_heads
+                ).repeat(query_layer.size(0) // self.num_attention_heads, 1, 1)  # .split(self.all_head_size, dim=-1)
+
+        score = 0
+        # content->position
+        if "c2p" in self.pos_att_type:
+            scale = scaled_size_sqrt(pos_key_layer, scale_factor)
+            c2p_att = torch.bmm(query_layer, pos_key_layer.transpose(-1, -2))
+            c2p_pos = torch.clamp(relative_pos + att_span, 0, att_span * 2 - 1)
+            c2p_att = torch.gather(
+                c2p_att,
+                dim=-1,
+                index=c2p_pos.squeeze(0).expand([query_layer.size(0), query_layer.size(1), relative_pos.size(-1)]),
+            )
+            score += c2p_att / scale.to(dtype=c2p_att.dtype)
+
+        # position->content
+        if "p2c" in self.pos_att_type:
+            scale = scaled_size_sqrt(pos_query_layer, scale_factor)
+            r_pos = build_rpos(
+                query_layer,
+                key_layer,
+                relative_pos,
+                self.max_relative_positions,
+                self.position_buckets,
+            )
+            p2c_pos = torch.clamp(-r_pos + att_span, 0, att_span * 2 - 1)
+            p2c_att = torch.bmm(key_layer, pos_query_layer.transpose(-1, -2))
+            p2c_att = torch.gather(
+                p2c_att,
+                dim=-1,
+                index=p2c_pos.squeeze(0).expand([query_layer.size(0), key_layer.size(-2), key_layer.size(-2)]),
+            ).transpose(-1, -2)
+            score += p2c_att / scale.to(dtype=p2c_att.dtype)
+
+        return score
+
+
+# Copied from transformers.models.deberta.modeling_deberta.DebertaAttention with Deberta->DebertaV2
+class DebertaV2Attention(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.self = DisentangledSelfAttention(config)
+        self.output = DebertaV2SelfOutput(config)
+        self.config = config
+
+    def forward(
+        self,
+        hidden_states,
+        attention_mask,
+        output_attentions: bool = False,
+        query_states=None,
+        relative_pos=None,
+        rel_embeddings=None,
+    ) -> tuple[torch.Tensor, torch.Tensor | None]:
+        self_output, att_matrix = self.self(
+            hidden_states,
+            attention_mask,
+            output_attentions,
+            query_states=query_states,
+            relative_pos=relative_pos,
+            rel_embeddings=rel_embeddings,
+        )
+        if query_states is None:
+            query_states = hidden_states
+        attention_output = self.output(self_output, query_states)
+
+        if output_attentions:
+            return (attention_output, att_matrix)
+        else:
+            return (attention_output, None)
+
+
+# Copied from transformers.models.bert.modeling_bert.BertIntermediate with Bert->DebertaV2
+class DebertaV2Intermediate(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
+        if isinstance(config.hidden_act, str):
+            self.intermediate_act_fn = ACT2FN[config.hidden_act]
+        else:
+            self.intermediate_act_fn = config.hidden_act
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.intermediate_act_fn(hidden_states)
+        return hidden_states
+
+
+# Copied from transformers.models.deberta.modeling_deberta.DebertaOutput with DebertaLayerNorm->LayerNorm
+class DebertaV2Output(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.intermediate_size, config.hidden_size)
+        self.LayerNorm = LayerNorm(config.hidden_size, config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        self.config = config
+
+    def forward(self, hidden_states, input_tensor):
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.dropout(hidden_states)
+        hidden_states = self.LayerNorm(hidden_states + input_tensor)
+        return hidden_states
+
+
+# Copied from transformers.models.deberta.modeling_deberta.DebertaLayer with Deberta->DebertaV2
+class DebertaV2Layer(GradientCheckpointingLayer):
+    def __init__(self, config):
+        super().__init__()
+        self.attention = DebertaV2Attention(config)
+        self.intermediate = DebertaV2Intermediate(config)
+        self.output = DebertaV2Output(config)
+
+    def forward(
+        self,
+        hidden_states,
+        attention_mask,
+        query_states=None,
+        relative_pos=None,
+        rel_embeddings=None,
+        output_attentions: bool = False,
+    ) -> tuple[torch.Tensor, torch.Tensor | None]:
+        attention_output, att_matrix = self.attention(
+            hidden_states,
+            attention_mask,
+            output_attentions=output_attentions,
+            query_states=query_states,
+            relative_pos=relative_pos,
+            rel_embeddings=rel_embeddings,
+        )
+        intermediate_output = self.intermediate(attention_output)
+        layer_output = self.output(intermediate_output, attention_output)
+
+        if output_attentions:
+            return (layer_output, att_matrix)
+        else:
+            return (layer_output, None)
+
+
+class ConvLayer(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        kernel_size = getattr(config, "conv_kernel_size", 3)
+        groups = getattr(config, "conv_groups", 1)
+        self.conv_act = getattr(config, "conv_act", "tanh")
+        self.conv = nn.Conv1d(
+            config.hidden_size, config.hidden_size, kernel_size, padding=(kernel_size - 1) // 2, groups=groups
+        )
+        self.LayerNorm = LayerNorm(config.hidden_size, config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        self.config = config
+
+    def forward(self, hidden_states, residual_states, input_mask):
+        out = self.conv(hidden_states.permute(0, 2, 1).contiguous()).permute(0, 2, 1).contiguous()
+        rmask = (1 - input_mask).bool()
+        out.masked_fill_(rmask.unsqueeze(-1).expand(out.size()), 0)
+        out = ACT2FN[self.conv_act](self.dropout(out))
+
+        layer_norm_input = residual_states + out
+        output = self.LayerNorm(layer_norm_input).to(layer_norm_input)
+
+        if input_mask is None:
+            output_states = output
+        else:
+            if input_mask.dim() != layer_norm_input.dim():
+                if input_mask.dim() == 4:
+                    input_mask = input_mask.squeeze(1).squeeze(1)
+                input_mask = input_mask.unsqueeze(2)
+
+            input_mask = input_mask.to(output.dtype)
+            output_states = output * input_mask
+
+        return output_states
+
+
+# Copied from transformers.models.deberta.modeling_deberta.DebertaEmbeddings with DebertaLayerNorm->LayerNorm,Deberta->DebertaV2
+class DebertaV2Embeddings(nn.Module):
+    """Construct the embeddings from word, position and token_type embeddings."""
+
+    def __init__(self, config):
+        super().__init__()
+        pad_token_id = getattr(config, "pad_token_id", 0)
+        self.embedding_size = getattr(config, "embedding_size", config.hidden_size)
+        self.word_embeddings = nn.Embedding(config.vocab_size, self.embedding_size, padding_idx=pad_token_id)
+
+        self.position_biased_input = getattr(config, "position_biased_input", True)
+        if not self.position_biased_input:
+            self.position_embeddings = None
+        else:
+            self.position_embeddings = nn.Embedding(config.max_position_embeddings, self.embedding_size)
+
+        if config.type_vocab_size > 0:
+            self.token_type_embeddings = nn.Embedding(config.type_vocab_size, self.embedding_size)
+        else:
+            self.token_type_embeddings = None
+
+        if self.embedding_size != config.hidden_size:
+            self.embed_proj = nn.Linear(self.embedding_size, config.hidden_size, bias=False)
+        else:
+            self.embed_proj = None
+
+        self.LayerNorm = LayerNorm(config.hidden_size, config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        self.config = config
+
+        # position_ids (1, len position emb) is contiguous in memory and exported when serialized
+        self.register_buffer(
+            "position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)), persistent=False
+        )
+
+    def forward(self, input_ids=None, token_type_ids=None, position_ids=None, mask=None, inputs_embeds=None):
+        if input_ids is not None:
+            input_shape = input_ids.size()
+        else:
+            input_shape = inputs_embeds.size()[:-1]
+
+        seq_length = input_shape[1]
+
+        if position_ids is None:
+            position_ids = self.position_ids[:, :seq_length]
+
+        if token_type_ids is None:
+            token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)
+
+        if inputs_embeds is None:
+            inputs_embeds = self.word_embeddings(input_ids)
+
+        if self.position_embeddings is not None:
+            position_embeddings = self.position_embeddings(position_ids.long())
+        else:
+            position_embeddings = torch.zeros_like(inputs_embeds)
+
+        embeddings = inputs_embeds
+        if self.position_biased_input:
+            embeddings = embeddings + position_embeddings
+        if self.token_type_embeddings is not None:
+            token_type_embeddings = self.token_type_embeddings(token_type_ids)
+            embeddings = embeddings + token_type_embeddings
+
+        if self.embed_proj is not None:
+            embeddings = self.embed_proj(embeddings)
+
+        embeddings = self.LayerNorm(embeddings)
+
+        if mask is not None:
+            if mask.dim() != embeddings.dim():
+                if mask.dim() == 4:
+                    mask = mask.squeeze(1).squeeze(1)
+                mask = mask.unsqueeze(2)
+            mask = mask.to(embeddings.dtype)
+
+            embeddings = embeddings * mask
+
+        embeddings = self.dropout(embeddings)
+        return embeddings
+
+
+class DebertaV2Encoder(nn.Module):
+    """Modified BertEncoder with relative position bias support"""
+
+    def __init__(self, config):
+        super().__init__()
+
+        self.layer = nn.ModuleList([DebertaV2Layer(config) for _ in range(config.num_hidden_layers)])
+        self.relative_attention = getattr(config, "relative_attention", False)
+
+        if self.relative_attention:
+            self.max_relative_positions = getattr(config, "max_relative_positions", -1)
+            if self.max_relative_positions < 1:
+                self.max_relative_positions = config.max_position_embeddings
+
+            self.position_buckets = getattr(config, "position_buckets", -1)
+            pos_ebd_size = self.max_relative_positions * 2
+
+            if self.position_buckets > 0:
+                pos_ebd_size = self.position_buckets * 2
+
+            self.rel_embeddings = nn.Embedding(pos_ebd_size, config.hidden_size)
+
+        self.norm_rel_ebd = [x.strip() for x in getattr(config, "norm_rel_ebd", "none").lower().split("|")]
+
+        if "layer_norm" in self.norm_rel_ebd:
+            self.LayerNorm = LayerNorm(config.hidden_size, config.layer_norm_eps, elementwise_affine=True)
+
+        self.conv = ConvLayer(config) if getattr(config, "conv_kernel_size", 0) > 0 else None
+        self.gradient_checkpointing = False
+
+    def get_rel_embedding(self):
+        rel_embeddings = self.rel_embeddings.weight if self.relative_attention else None
+        if rel_embeddings is not None and ("layer_norm" in self.norm_rel_ebd):
+            rel_embeddings = self.LayerNorm(rel_embeddings)
+        return rel_embeddings
+
+    def get_attention_mask(self, attention_mask):
+        if attention_mask.dim() <= 2:
+            extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)
+            attention_mask = extended_attention_mask * extended_attention_mask.squeeze(-2).unsqueeze(-1)
+        elif attention_mask.dim() == 3:
+            attention_mask = attention_mask.unsqueeze(1)
+
+        return attention_mask
+
+    def get_rel_pos(self, hidden_states, query_states=None, relative_pos=None):
+        if self.relative_attention and relative_pos is None:
+            if query_states is not None:
+                relative_pos = build_relative_position(
+                    query_states,
+                    hidden_states,
+                    bucket_size=self.position_buckets,
+                    max_position=self.max_relative_positions,
+                )
+            else:
+                relative_pos = build_relative_position(
+                    hidden_states,
+                    hidden_states,
+                    bucket_size=self.position_buckets,
+                    max_position=self.max_relative_positions,
+                )
+        return relative_pos
+
+    def forward(
+        self,
+        hidden_states,
+        attention_mask,
+        output_hidden_states=True,
+        output_attentions=False,
+        query_states=None,
+        relative_pos=None,
+        return_dict=True,
+    ):
+        if attention_mask.dim() <= 2:
+            input_mask = attention_mask
+        else:
+            input_mask = attention_mask.sum(-2) > 0
+        attention_mask = self.get_attention_mask(attention_mask)
+        relative_pos = self.get_rel_pos(hidden_states, query_states, relative_pos)
+
+        all_hidden_states: tuple[torch.Tensor] | None = (hidden_states,) if output_hidden_states else None
+        all_attentions = () if output_attentions else None
+
+        next_kv = hidden_states
+        rel_embeddings = self.get_rel_embedding()
+        for i, layer_module in enumerate(self.layer):
+            output_states, attn_weights = layer_module(
+                next_kv,
+                attention_mask,
+                query_states=query_states,
+                relative_pos=relative_pos,
+                rel_embeddings=rel_embeddings,
+                output_attentions=output_attentions,
+            )
+
+            if output_attentions:
+                all_attentions = all_attentions + (attn_weights,)
+
+            if i == 0 and self.conv is not None:
+                output_states = self.conv(hidden_states, output_states, input_mask)
+
+            if output_hidden_states:
+                all_hidden_states = all_hidden_states + (output_states,)
+
+            if query_states is not None:
+                query_states = output_states
+                if isinstance(hidden_states, Sequence):
+                    next_kv = hidden_states[i + 1] if i + 1 < len(self.layer) else None
+            else:
+                next_kv = output_states
+
+        if not return_dict:
+            return tuple(v for v in [output_states, all_hidden_states, all_attentions] if v is not None)
+        return BaseModelOutput(
+            last_hidden_state=output_states, hidden_states=all_hidden_states, attentions=all_attentions
+        )
+
+
+@auto_docstring
+class DebertaV2PreTrainedModel(PreTrainedModel):
+    config: DebertaV2Config
+    base_model_prefix = "deberta"
+    _keys_to_ignore_on_load_unexpected = ["position_embeddings"]
+    supports_gradient_checkpointing = True
+
+    @torch.no_grad()
+    def _init_weights(self, module):
+        """Initialize the weights."""
+        super()._init_weights(module)
+        if isinstance(module, (LegacyDebertaV2LMPredictionHead, DebertaV2LMPredictionHead)):
+            init.zeros_(module.bias)
+        elif isinstance(module, DebertaV2Embeddings):
+            init.copy_(module.position_ids, torch.arange(module.position_ids.shape[-1]).expand((1, -1)))
+
+
+@auto_docstring
+# Copied from transformers.models.deberta.modeling_deberta.DebertaModel with Deberta->DebertaV2
+class DebertaV2Model(DebertaV2PreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+
+        self.embeddings = DebertaV2Embeddings(config)
+        self.encoder = DebertaV2Encoder(config)
+        self.z_steps = 0
+        self.config = config
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embeddings.word_embeddings
+
+    def set_input_embeddings(self, new_embeddings):
+        self.embeddings.word_embeddings = new_embeddings
+
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.Tensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        token_type_ids: torch.Tensor | None = None,
+        position_ids: torch.Tensor | None = None,
+        inputs_embeds: torch.Tensor | None = None,
+        output_attentions: bool | None = None,
+        output_hidden_states: bool | None = None,
+        return_dict: bool | None = None,
+        **kwargs,
+    ) -> tuple | BaseModelOutput:
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.return_dict
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
+        elif input_ids is not None:
+            self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
+            input_shape = input_ids.size()
+        elif inputs_embeds is not None:
+            input_shape = inputs_embeds.size()[:-1]
+        else:
+            raise ValueError("You have to specify either input_ids or inputs_embeds")
+
+        device = input_ids.device if input_ids is not None else inputs_embeds.device
+
+        if attention_mask is None:
+            attention_mask = torch.ones(input_shape, device=device)
+        if token_type_ids is None:
+            token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=device)
+
+        embedding_output = self.embeddings(
+            input_ids=input_ids,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            mask=attention_mask,
+            inputs_embeds=inputs_embeds,
+        )
+
+        encoder_outputs = self.encoder(
+            embedding_output,
+            attention_mask,
+            output_hidden_states=True,
+            output_attentions=output_attentions,
+            return_dict=return_dict,
+        )
+        encoded_layers = encoder_outputs[1]
+
+        if self.z_steps > 1:
+            hidden_states = encoded_layers[-2]
+            layers = [self.encoder.layer[-1] for _ in range(self.z_steps)]
+            query_states = encoded_layers[-1]
+            rel_embeddings = self.encoder.get_rel_embedding()
+            attention_mask = self.encoder.get_attention_mask(attention_mask)
+            rel_pos = self.encoder.get_rel_pos(embedding_output)
+            for layer in layers[1:]:
+                query_states = layer(
+                    hidden_states,
+                    attention_mask,
+                    output_attentions=False,
+                    query_states=query_states,
+                    relative_pos=rel_pos,
+                    rel_embeddings=rel_embeddings,
+                )
+                encoded_layers.append(query_states)
+
+        sequence_output = encoded_layers[-1]
+
+        if not return_dict:
+            return (sequence_output,) + encoder_outputs[(1 if output_hidden_states else 2) :]
+
+        return BaseModelOutput(
+            last_hidden_state=sequence_output,
+            hidden_states=encoder_outputs.hidden_states if output_hidden_states else None,
+            attentions=encoder_outputs.attentions,
+        )
+
+
+# Copied from transformers.models.deberta.modeling_deberta.LegacyDebertaPredictionHeadTransform with Deberta->DebertaV2
+class LegacyDebertaV2PredictionHeadTransform(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.embedding_size = getattr(config, "embedding_size", config.hidden_size)
+
+        self.dense = nn.Linear(config.hidden_size, self.embedding_size)
+        if isinstance(config.hidden_act, str):
+            self.transform_act_fn = ACT2FN[config.hidden_act]
+        else:
+            self.transform_act_fn = config.hidden_act
+        self.LayerNorm = nn.LayerNorm(self.embedding_size, eps=config.layer_norm_eps)
+
+    def forward(self, hidden_states):
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.transform_act_fn(hidden_states)
+        hidden_states = self.LayerNorm(hidden_states)
+        return hidden_states
+
+
+class LegacyDebertaV2LMPredictionHead(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.transform = LegacyDebertaV2PredictionHeadTransform(config)
+
+        self.embedding_size = getattr(config, "embedding_size", config.hidden_size)
+        # The output weights are the same as the input embeddings, but there is
+        # an output-only bias for each token.
+        self.decoder = nn.Linear(self.embedding_size, config.vocab_size)
+
+        self.bias = nn.Parameter(torch.zeros(config.vocab_size))
+
+    def forward(self, hidden_states):
+        hidden_states = self.transform(hidden_states)
+        hidden_states = self.decoder(hidden_states)
+        return hidden_states
+
+
+class LegacyDebertaV2OnlyMLMHead(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.predictions = LegacyDebertaV2LMPredictionHead(config)
+
+    def forward(self, sequence_output):
+        prediction_scores = self.predictions(sequence_output)
+        return prediction_scores
+
+
+class DebertaV2LMPredictionHead(nn.Module):
+    """https://github.com/microsoft/DeBERTa/blob/master/DeBERTa/deberta/bert.py#L270"""
+
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+
+        if isinstance(config.hidden_act, str):
+            self.transform_act_fn = ACT2FN[config.hidden_act]
+        else:
+            self.transform_act_fn = config.hidden_act
+
+        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps, elementwise_affine=True)
+
+        self.bias = nn.Parameter(torch.zeros(config.vocab_size))
+
+    # note that the input embeddings must be passed as an argument
+    def forward(self, hidden_states, word_embeddings):
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.transform_act_fn(hidden_states)
+        hidden_states = self.LayerNorm(hidden_states)
+        hidden_states = torch.matmul(hidden_states, word_embeddings.weight.t()) + self.bias
+        return hidden_states
+
+
+class DebertaV2OnlyMLMHead(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.lm_head = DebertaV2LMPredictionHead(config)
+
+    # note that the input embeddings must be passed as an argument
+    def forward(self, sequence_output, word_embeddings):
+        prediction_scores = self.lm_head(sequence_output, word_embeddings)
+        return prediction_scores
+
+
+@auto_docstring
+class DebertaV2ForMaskedLM(DebertaV2PreTrainedModel):
+    _tied_weights_keys = {
+        "cls.predictions.decoder.bias": "cls.predictions.bias",
+        "cls.predictions.decoder.weight": "deberta.embeddings.word_embeddings.weight",
+    }
+    _keys_to_ignore_on_load_unexpected = [r"mask_predictions.*"]
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.legacy = config.legacy
+        self.deberta = DebertaV2Model(config)
+        if self.legacy:
+            self.cls = LegacyDebertaV2OnlyMLMHead(config)
+        else:
+            self._tied_weights_keys = {
+                "lm_predictions.lm_head.weight": "deberta.embeddings.word_embeddings.weight",
+            }
+            self.lm_predictions = DebertaV2OnlyMLMHead(config)
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_output_embeddings(self):
+        if self.legacy:
+            return self.cls.predictions.decoder
+        else:
+            return self.lm_predictions.lm_head.dense
+
+    def set_output_embeddings(self, new_embeddings):
+        if self.legacy:
+            self.cls.predictions.decoder = new_embeddings
+            self.cls.predictions.bias = new_embeddings.bias
+        else:
+            self.lm_predictions.lm_head.dense = new_embeddings
+            self.lm_predictions.lm_head.bias = new_embeddings.bias
+
+    @auto_docstring
+    # Copied from transformers.models.deberta.modeling_deberta.DebertaForMaskedLM.forward with Deberta->DebertaV2
+    def forward(
+        self,
+        input_ids: torch.Tensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        token_type_ids: torch.Tensor | None = None,
+        position_ids: torch.Tensor | None = None,
+        inputs_embeds: torch.Tensor | None = None,
+        labels: torch.Tensor | None = None,
+        output_attentions: bool | None = None,
+        output_hidden_states: bool | None = None,
+        return_dict: bool | None = None,
+        **kwargs,
+    ) -> tuple | MaskedLMOutput:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
+            config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
+            loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
+        """
+
+        return_dict = return_dict if return_dict is not None else self.config.return_dict
+
+        outputs = self.deberta(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        sequence_output = outputs[0]
+        if self.legacy:
+            prediction_scores = self.cls(sequence_output)
+        else:
+            prediction_scores = self.lm_predictions(sequence_output, self.deberta.embeddings.word_embeddings)
+
+        masked_lm_loss = None
+        if labels is not None:
+            loss_fct = CrossEntropyLoss()  # -100 index = padding token
+            masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
+
+        if not return_dict:
+            output = (prediction_scores,) + outputs[1:]
+            return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output
+
+        return MaskedLMOutput(
+            loss=masked_lm_loss,
+            logits=prediction_scores,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+# Copied from transformers.models.deberta.modeling_deberta.ContextPooler
+class ContextPooler(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.pooler_hidden_size, config.pooler_hidden_size)
+        self.dropout = nn.Dropout(config.pooler_dropout)
+        self.config = config
+
+    def forward(self, hidden_states):
+        # We "pool" the model by simply taking the hidden state corresponding
+        # to the first token.
+
+        context_token = hidden_states[:, 0]
+        context_token = self.dropout(context_token)
+        pooled_output = self.dense(context_token)
+        pooled_output = ACT2FN[self.config.pooler_hidden_act](pooled_output)
+        return pooled_output
+
+    @property
+    def output_dim(self):
+        return self.config.hidden_size
+
+
+@auto_docstring(
+    custom_intro="""
+    DeBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the
+    pooled output) e.g. for GLUE tasks.
+    """
+)
+class DebertaV2ForSequenceClassification(DebertaV2PreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+
+        num_labels = getattr(config, "num_labels", 2)
+        self.num_labels = num_labels
+
+        self.deberta = DebertaV2Model(config)
+        self.pooler = ContextPooler(config)
+        output_dim = self.pooler.output_dim
+
+        self.classifier = nn.Linear(output_dim, num_labels)
+        drop_out = getattr(config, "cls_dropout", None)
+        drop_out = self.config.hidden_dropout_prob if drop_out is None else drop_out
+        self.dropout = nn.Dropout(drop_out)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.deberta.get_input_embeddings()
+
+    def set_input_embeddings(self, new_embeddings):
+        self.deberta.set_input_embeddings(new_embeddings)
+
+    @auto_docstring
+    # Copied from transformers.models.deberta.modeling_deberta.DebertaForSequenceClassification.forward with Deberta->DebertaV2
+    def forward(
+        self,
+        input_ids: torch.Tensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        token_type_ids: torch.Tensor | None = None,
+        position_ids: torch.Tensor | None = None,
+        inputs_embeds: torch.Tensor | None = None,
+        labels: torch.Tensor | None = None,
+        output_attentions: bool | None = None,
+        output_hidden_states: bool | None = None,
+        return_dict: bool | None = None,
+        **kwargs,
+    ) -> tuple | SequenceClassifierOutput:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
+            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
+            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
+        """
+        return_dict = return_dict if return_dict is not None else self.config.return_dict
+
+        outputs = self.deberta(
+            input_ids,
+            token_type_ids=token_type_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        encoder_layer = outputs[0]
+        pooled_output = self.pooler(encoder_layer)
+        pooled_output = self.dropout(pooled_output)
+        logits = self.classifier(pooled_output)
+
+        loss = None
+        if labels is not None:
+            if self.config.problem_type is None:
+                if self.num_labels == 1:
+                    # regression task
+                    loss_fn = nn.MSELoss()
+                    logits = logits.view(-1).to(labels.dtype)
+                    loss = loss_fn(logits, labels.view(-1))
+                elif labels.dim() == 1 or labels.size(-1) == 1:
+                    label_index = (labels >= 0).nonzero()
+                    labels = labels.long()
+                    if label_index.size(0) > 0:
+                        labeled_logits = torch.gather(
+                            logits, 0, label_index.expand(label_index.size(0), logits.size(1))
+                        )
+                        labels = torch.gather(labels, 0, label_index.view(-1))
+                        loss_fct = CrossEntropyLoss()
+                        loss = loss_fct(labeled_logits.view(-1, self.num_labels).float(), labels.view(-1))
+                    else:
+                        loss = torch.tensor(0).to(logits)
+                else:
+                    log_softmax = nn.LogSoftmax(-1)
+                    loss = -((log_softmax(logits) * labels).sum(-1)).mean()
+            elif self.config.problem_type == "regression":
+                loss_fct = MSELoss()
+                if self.num_labels == 1:
+                    loss = loss_fct(logits.squeeze(), labels.squeeze())
+                else:
+                    loss = loss_fct(logits, labels)
+            elif self.config.problem_type == "single_label_classification":
+                loss_fct = CrossEntropyLoss()
+                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+            elif self.config.problem_type == "multi_label_classification":
+                loss_fct = BCEWithLogitsLoss()
+                loss = loss_fct(logits, labels)
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return SequenceClassifierOutput(
+            loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
+        )
+
+
+@auto_docstring
+# Copied from transformers.models.deberta.modeling_deberta.DebertaForTokenClassification with Deberta->DebertaV2
+class DebertaV2ForTokenClassification(DebertaV2PreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+
+        self.deberta = DebertaV2Model(config)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.Tensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        token_type_ids: torch.Tensor | None = None,
+        position_ids: torch.Tensor | None = None,
+        inputs_embeds: torch.Tensor | None = None,
+        labels: torch.Tensor | None = None,
+        output_attentions: bool | None = None,
+        output_hidden_states: bool | None = None,
+        return_dict: bool | None = None,
+        **kwargs,
+    ) -> tuple | TokenClassifierOutput:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
+        """
+        return_dict = return_dict if return_dict is not None else self.config.return_dict
+
+        outputs = self.deberta(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        sequence_output = outputs[0]
+
+        sequence_output = self.dropout(sequence_output)
+        logits = self.classifier(sequence_output)
+
+        loss = None
+        if labels is not None:
+            loss_fct = CrossEntropyLoss()
+            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return TokenClassifierOutput(
+            loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
+        )
+
+
+@auto_docstring
+class DebertaV2ForQuestionAnswering(DebertaV2PreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+
+        self.deberta = DebertaV2Model(config)
+        self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @auto_docstring
+    # Copied from transformers.models.deberta.modeling_deberta.DebertaForQuestionAnswering.forward with Deberta->DebertaV2
+    def forward(
+        self,
+        input_ids: torch.Tensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        token_type_ids: torch.Tensor | None = None,
+        position_ids: torch.Tensor | None = None,
+        inputs_embeds: torch.Tensor | None = None,
+        start_positions: torch.Tensor | None = None,
+        end_positions: torch.Tensor | None = None,
+        output_attentions: bool | None = None,
+        output_hidden_states: bool | None = None,
+        return_dict: bool | None = None,
+        **kwargs,
+    ) -> tuple | QuestionAnsweringModelOutput:
+        return_dict = return_dict if return_dict is not None else self.config.return_dict
+
+        outputs = self.deberta(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        sequence_output = outputs[0]
+
+        logits = self.qa_outputs(sequence_output)
+        start_logits, end_logits = logits.split(1, dim=-1)
+        start_logits = start_logits.squeeze(-1).contiguous()
+        end_logits = end_logits.squeeze(-1).contiguous()
+
+        total_loss = None
+        if start_positions is not None and end_positions is not None:
+            # If we are on multi-GPU, split add a dimension
+            if len(start_positions.size()) > 1:
+                start_positions = start_positions.squeeze(-1)
+            if len(end_positions.size()) > 1:
+                end_positions = end_positions.squeeze(-1)
+            # sometimes the start/end positions are outside our model inputs, we ignore these terms
+            ignored_index = start_logits.size(1)
+            start_positions = start_positions.clamp(0, ignored_index)
+            end_positions = end_positions.clamp(0, ignored_index)
+
+            loss_fct = CrossEntropyLoss(ignore_index=ignored_index)
+            start_loss = loss_fct(start_logits, start_positions)
+            end_loss = loss_fct(end_logits, end_positions)
+            total_loss = (start_loss + end_loss) / 2
+
+        if not return_dict:
+            output = (start_logits, end_logits) + outputs[1:]
+            return ((total_loss,) + output) if total_loss is not None else output
+
+        return QuestionAnsweringModelOutput(
+            loss=total_loss,
+            start_logits=start_logits,
+            end_logits=end_logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+@auto_docstring
+class DebertaV2ForMultipleChoice(DebertaV2PreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+
+        num_labels = getattr(config, "num_labels", 2)
+        self.num_labels = num_labels
+
+        self.deberta = DebertaV2Model(config)
+        self.pooler = ContextPooler(config)
+        output_dim = self.pooler.output_dim
+
+        self.classifier = nn.Linear(output_dim, 1)
+        drop_out = getattr(config, "cls_dropout", None)
+        drop_out = self.config.hidden_dropout_prob if drop_out is None else drop_out
+        self.dropout = nn.Dropout(drop_out)
+
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.deberta.get_input_embeddings()
+
+    def set_input_embeddings(self, new_embeddings):
+        self.deberta.set_input_embeddings(new_embeddings)
+
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.Tensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        token_type_ids: torch.Tensor | None = None,
+        position_ids: torch.Tensor | None = None,
+        inputs_embeds: torch.Tensor | None = None,
+        labels: torch.Tensor | None = None,
+        output_attentions: bool | None = None,
+        output_hidden_states: bool | None = None,
+        return_dict: bool | None = None,
+        **kwargs,
+    ) -> tuple | MultipleChoiceModelOutput:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the multiple choice classification loss. Indices should be in `[0, ...,
+            num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See
+            `input_ids` above)
+        """
+        return_dict = return_dict if return_dict is not None else self.config.return_dict
+        num_choices = input_ids.shape[1] if input_ids is not None else inputs_embeds.shape[1]
+
+        flat_input_ids = input_ids.view(-1, input_ids.size(-1)) if input_ids is not None else None
+        flat_position_ids = position_ids.view(-1, position_ids.size(-1)) if position_ids is not None else None
+        flat_token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1)) if token_type_ids is not None else None
+        flat_attention_mask = attention_mask.view(-1, attention_mask.size(-1)) if attention_mask is not None else None
+        flat_inputs_embeds = (
+            inputs_embeds.view(-1, inputs_embeds.size(-2), inputs_embeds.size(-1))
+            if inputs_embeds is not None
+            else None
+        )
+
+        outputs = self.deberta(
+            flat_input_ids,
+            position_ids=flat_position_ids,
+            token_type_ids=flat_token_type_ids,
+            attention_mask=flat_attention_mask,
+            inputs_embeds=flat_inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        encoder_layer = outputs[0]
+        pooled_output = self.pooler(encoder_layer)
+        pooled_output = self.dropout(pooled_output)
+        logits = self.classifier(pooled_output)
+        reshaped_logits = logits.view(-1, num_choices)
+
+        loss = None
+        if labels is not None:
+            loss_fct = CrossEntropyLoss()
+            loss = loss_fct(reshaped_logits, labels)
+
+        if not return_dict:
+            output = (reshaped_logits,) + outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return MultipleChoiceModelOutput(
+            loss=loss,
+            logits=reshaped_logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+__all__ = [
+    "DebertaV2ForMaskedLM",
+    "DebertaV2ForMultipleChoice",
+    "DebertaV2ForQuestionAnswering",
+    "DebertaV2ForSequenceClassification",
+    "DebertaV2ForTokenClassification",
+    "DebertaV2Model",
+    "DebertaV2PreTrainedModel",
+]

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/deberta_v2/tokenization_deberta_v2.py

@@ -0,0 +1,169 @@
+# Copyright 2020 Microsoft and the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tokenization class for model DeBERTa-v2."""
+
+from tokenizers import Regex, Tokenizer, decoders, normalizers, pre_tokenizers, processors
+from tokenizers.models import Unigram
+
+from ...tokenization_utils_tokenizers import TokenizersBackend
+from ...utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+VOCAB_FILES_NAMES = {"vocab_file": "spm.model", "tokenizer_file": "tokenizer.json"}
+
+
+class DebertaV2Tokenizer(TokenizersBackend):
+    """
+    Construct a DeBERTa-v2 tokenizer (backed by HuggingFace's *tokenizers* library). Based on Unigram tokenization.
+
+    This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
+    refer to this superclass for more information regarding those methods.
+
+    Args:
+        vocab_file (`str`, *optional*):
+            Path to the vocabulary file (SentencePiece model file). Not used directly but kept for compatibility.
+        vocab (`str`, `dict` or `list`, *optional*):
+            List of tuples (piece, score) for the vocabulary.
+        precompiled_charsmap (`bytes`, *optional*):
+            Precompiled character map for normalization.
+        do_lower_case (`bool`, *optional*, defaults to `False`):
+            Whether or not to lowercase the input when tokenizing.
+        split_by_punct (`bool`, *optional*, defaults to `False`):
+            Whether to split by punctuation.
+        bos_token (`str`, *optional*, defaults to `"[CLS]"`):
+            The beginning of sequence token.
+        eos_token (`str`, *optional*, defaults to `"[SEP]"`):
+            The end of sequence token.
+        unk_token (`str`, *optional*, defaults to `"[UNK]"`):
+            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
+            token instead.
+        sep_token (`str`, *optional*, defaults to `"[SEP]"`):
+            The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
+            sequence classification or for a text and a question for question answering. It is also used as the last
+            token of a sequence built with special tokens.
+        pad_token (`str`, *optional*, defaults to `"[PAD]"`):
+            The token used for padding, for example when batching sequences of different lengths.
+        cls_token (`str`, *optional*, defaults to `"[CLS]"`):
+            The classifier token which is used when doing sequence classification (classification of the whole sequence
+            instead of per-token classification). It is the first token of the sequence when built with special tokens.
+        mask_token (`str`, *optional*, defaults to `"[MASK]"`):
+            The token used for masking values. This is the token used when training this model with masked language
+            modeling. This is the token which the model will try to predict.
+        add_prefix_space (`bool`, *optional*, defaults to `True`):
+            Whether or not to add an initial space to the input. This allows to treat the leading word just as any
+            other word.
+        unk_id (`int`, *optional*, defaults to index of `unk_token` in vocab):
+            The ID of the unknown token in the vocabulary.
+    """
+
+    vocab_files_names = VOCAB_FILES_NAMES
+    model_input_names = ["input_ids", "attention_mask", "token_type_ids"]
+    model = Unigram
+
+    def __init__(
+        self,
+        vocab: str | dict | list | None = None,
+        do_lower_case=False,
+        split_by_punct=False,
+        bos_token="[CLS]",
+        eos_token="[SEP]",
+        unk_token="[UNK]",
+        sep_token="[SEP]",
+        pad_token="[PAD]",
+        cls_token="[CLS]",
+        mask_token="[MASK]",
+        add_prefix_space=True,
+        unk_id=1,
+        **kwargs,
+    ):
+        self.do_lower_case = do_lower_case
+        self.split_by_punct = split_by_punct
+        self.add_prefix_space = add_prefix_space
+
+        if vocab is None:
+            vocab = [
+                (str(pad_token), 0.0),
+                (str(unk_token), 0.0),
+                (str(bos_token), 0.0),
+                (str(eos_token), 0.0),
+                (str(sep_token), 0.0),
+                (str(cls_token), 0.0),
+                (str(mask_token), 0.0),
+            ]
+            unk_id = 1
+        elif isinstance(vocab, list):
+            unk_id = vocab.index((str(unk_token), 0.0)) if (str(unk_token), 0.0) in vocab else unk_id
+
+        self._vocab = vocab
+        self._tokenizer = Tokenizer(
+            Unigram(
+                self._vocab,
+                unk_id=unk_id,
+                byte_fallback=False,
+            )
+        )
+
+        list_normalizers = []
+        if do_lower_case:
+            list_normalizers.append(normalizers.Lowercase())
+
+        list_normalizers.extend(
+            [
+                normalizers.Replace(Regex(r"\s{2,}|[\n\r\t]"), " "),
+                normalizers.NFC(),
+                normalizers.Strip(left=False, right=True),
+            ]
+        )
+        self._tokenizer.normalizer = normalizers.Sequence(list_normalizers)
+
+        list_pretokenizers = []
+        if split_by_punct:
+            list_pretokenizers.append(pre_tokenizers.Punctuation(behavior="isolated"))
+
+        prepend_scheme = "always" if add_prefix_space else "first"
+        list_pretokenizers.append(pre_tokenizers.Metaspace(replacement="▁", prepend_scheme=prepend_scheme))
+
+        self._tokenizer.pre_tokenizer = pre_tokenizers.Sequence(list_pretokenizers)
+        self._tokenizer.decoder = decoders.Metaspace(replacement="▁", prepend_scheme=prepend_scheme)
+        super().__init__(
+            bos_token=bos_token,
+            eos_token=eos_token,
+            unk_token=unk_token,
+            sep_token=sep_token,
+            cls_token=cls_token,
+            pad_token=pad_token,
+            mask_token=mask_token,
+            unk_id=unk_id,
+            do_lower_case=do_lower_case,
+            split_by_punct=split_by_punct,
+            add_prefix_space=add_prefix_space,
+            **kwargs,
+        )
+
+        cls_token_id = self.cls_token_id if self.cls_token_id is not None else 0
+        sep_token_id = self.sep_token_id if self.sep_token_id is not None else 0
+
+        self._tokenizer.post_processor = processors.TemplateProcessing(
+            single=f"{str(self.cls_token)}:0 $A:0 {str(self.sep_token)}:0",
+            pair=f"{str(self.cls_token)}:0 $A:0 {str(self.sep_token)}:0 $B:1 {str(self.sep_token)}:1",
+            special_tokens=[
+                (str(self.cls_token), cls_token_id),
+                (str(self.sep_token), sep_token_id),
+            ],
+        )
+
+
+__all__ = ["DebertaV2Tokenizer"]

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/falcon_h1/__init__.py

@@ -0,0 +1,27 @@
+# Copyright 2025 TII and the HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import TYPE_CHECKING
+
+from ...utils import _LazyModule
+from ...utils.import_utils import define_import_structure
+
+
+if TYPE_CHECKING:
+    from .configuration_falcon_h1 import *
+    from .modeling_falcon_h1 import *
+else:
+    import sys
+
+    _file = globals()["__file__"]
+    sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/falcon_h1/configuration_falcon_h1.py

@@ -0,0 +1,139 @@
+# Copyright 2025 TII and the HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""FalconH1 model configuration"""
+
+from huggingface_hub.dataclasses import strict
+
+from ...configuration_utils import PreTrainedConfig
+from ...modeling_rope_utils import RopeParameters
+from ...utils import auto_docstring
+
+
+@auto_docstring(checkpoint="tiiuae/Falcon-H1-1.5B-Deep-Instruct")
+@strict
+class FalconH1Config(PreTrainedConfig):
+    r"""
+    num_logits_to_keep (`int` or `None`, *optional*, defaults to 1):
+        Number of prompt logits to calculate during generation. If `None`, all logits will be calculated. If an
+        integer value, only last `num_logits_to_keep` logits will be calculated. Default is 1 because only the
+        logits of the last prompt token are needed for generation. For long sequences, the logits for the entire
+        sequence may use a lot of memory so, setting `num_logits_to_keep=1` will reduce memory footprint
+        significantly.
+    projectors_bias (`bool`, *optional*, defaults to `False`):
+        Flag indicating whether or not to use bias in the input and output projections (["in_proj", "out_proj"]) of the attention block
+    lm_head_multiplier (`float`, *optional*, defaults to 1.0):
+        The multiplier for the LM head. This is used to scale the output of the LM head.
+    embedding_multiplier (`float`, *optional*, defaults to 1.0):
+        The multiplier for the embedding layer. This is used to scale the output of the embedding layer.
+    mlp_multipliers (`list[float]`, *optional*):
+        The multipliers for the MLP layers. This is used to scale the output of the MLP layers. The first value is
+        the multiplier of gate layer, the second value is the multiplier of the down_proj layer.
+    key_multiplier (`float`, *optional*):
+        The multiplier for the key layer. This is used to scale the output of the key layer.
+    attention_out_multiplier (`float`, *optional*):
+        The multiplier for the attention output layer. This is used to scale the output of the attention output
+    attention_in_multiplier (`float`, *optional*):
+        The multiplier for the attention input layer. This is used to scale the output of the attention input layer.
+    ssm_multipliers (`list[float]`, *optional*):
+        The multipliers for the SSM layers. This is used to scale the output of the SSM layers.
+    ssm_in_multiplier (`float`, *optional*):
+        The multiplier for the SSM input layer. This is used to scale the output of the SSM input layer.
+    ssm_out_multiplier (`float`, *optional*):
+        The multiplier for the SSM output layer. This is used to scale the output of the SSM output layer.
+    """
+
+    model_type = "falcon_h1"
+    attribute_map = {"layer_types": "layers_block_type"}
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    vocab_size: int = 128000
+    tie_word_embeddings: bool = False
+    hidden_size: int = 4096
+    intermediate_size: int = 14336
+    num_hidden_layers: int = 32
+    num_attention_heads: int = 32
+    num_key_value_heads: int | None = 8
+    hidden_act: str = "silu"
+    initializer_range: float = 0.02
+    rms_norm_eps: float = 1e-5
+    use_cache: bool | None = True
+    num_logits_to_keep: int | None = 1
+    pad_token_id: int | None = 0
+    bos_token_id: int | None = 1
+    eos_token_id: int | list[int] | None = 2
+    max_position_embeddings: int = 8192
+    attention_dropout: float | int | None = 0.0
+    mamba_d_ssm: int | None = 1024
+    mamba_n_heads: int | None = 128
+    mamba_d_head: str | int | None = "auto"
+    mamba_n_groups: int | None = 1
+    mamba_d_state: int | None = 256
+    mamba_d_conv: int | None = 4
+    mamba_expand: int | None = 2
+    mamba_chunk_size: int | None = 256
+    mamba_conv_bias: bool | None = True
+    mamba_proj_bias: bool | None = False
+    mamba_norm_before_gate: bool | None = True
+    mamba_rms_norm: bool | None = False
+    time_step_min: float | None = 0.001
+    time_step_max: float | None = 0.1
+    time_step_limit: list[float, float] | tuple[float, float] | None = (0.0, float("inf"))
+    projectors_bias: bool | None = False
+    rope_parameters: RopeParameters | dict | None = None
+    lm_head_multiplier: float | None = 1.0
+    embedding_multiplier: float | None = 1.0
+    mlp_multipliers: list[float] | None = None
+    key_multiplier: float | None = 1.0
+    attention_out_multiplier: float | None = 1.0
+    attention_in_multiplier: float | None = 1.0
+    ssm_multipliers: list[float] | None = None
+    ssm_in_multiplier: float | None = 1.0
+    ssm_out_multiplier: float | None = 1.0
+    attention_bias: bool = False
+    mlp_bias: bool = False
+
+    def __post_init__(self, **kwargs):
+        if self.num_key_value_heads is None:
+            self.num_key_value_heads = self.num_attention_heads
+
+        # for the mamba_v2, must satisfy the following
+        mamba_intermediate = self.mamba_expand * self.hidden_size if self.mamba_d_ssm is None else self.mamba_d_ssm
+        if self.mamba_d_head == "auto":
+            self.mamba_d_head = mamba_intermediate // self.mamba_n_heads
+
+        self.time_step_limit = tuple(self.time_step_limit) if self.time_step_limit is not None else None
+        if self.mlp_multipliers is None:
+            self.mlp_multipliers = [1.0, 1.0]
+
+        if self.ssm_multipliers is None:
+            self.ssm_multipliers = [1.0, 1.0, 1.0, 1.0, 1.0]
+
+        super().__post_init__(**kwargs)
+
+    def validate_architecture(self):
+        """Part of `@strict`-powered validation. Validates the architecture of the config."""
+        mamba_intermediate = self.mamba_expand * self.hidden_size if self.mamba_d_ssm is None else self.mamba_d_ssm
+
+        if mamba_intermediate % self.mamba_n_heads != 0:
+            raise ValueError("mamba_n_heads must divide mamba_expand * hidden_size")
+
+        if self.mamba_d_head * self.mamba_n_heads != mamba_intermediate:
+            raise ValueError("The dimensions for the Mamba head state do not match the model intermediate_size")
+
+    @property
+    def layers_block_type(self):
+        return ["hybrid" for i in range(self.num_hidden_layers)]
+
+
+__all__ = ["FalconH1Config"]

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/falcon_h1/modeling_falcon_h1.py

@@ -0,0 +1,1265 @@
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from src/transformers/models/falcon_h1/modular_falcon_h1.py.
+#               Do NOT edit this file manually as any edits will be overwritten by the generation of
+#             the file from the modular. If any change should be done, please apply the change to the
+#                          modular_falcon_h1.py file directly. One of our CI enforces this.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2025 Technology Innovation Institute and the HuggingFace Inc. team. All rights reserved.
+#
+# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
+# and OPT implementations in this library. It has been modified from its
+# original forms to accommodate minor architectural differences compared
+# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from collections.abc import Callable
+from typing import Optional
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from ... import initialization as init
+from ...activations import ACT2FN
+from ...cache_utils import Cache, DynamicCache
+from ...generation import GenerationMixin
+from ...integrations import use_kernel_forward_from_hub, use_kernel_func_from_hub, use_kernelized_func
+from ...integrations.hub_kernels import lazy_load_kernel
+from ...masking_utils import create_causal_mask
+from ...modeling_flash_attention_utils import FlashAttentionKwargs
+from ...modeling_layers import GradientCheckpointingLayer
+from ...modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
+from ...modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
+from ...modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from ...processing_utils import Unpack
+from ...utils import TransformersKwargs, auto_docstring, can_return_tuple, is_torchdynamo_compiling, logging
+from ...utils.generic import maybe_autocast, merge_with_config_defaults
+from ...utils.import_utils import resolve_internal_import
+from ...utils.output_capturing import capture_outputs
+from .configuration_falcon_h1 import FalconH1Config
+
+
+logger = logging.get_logger(__name__)
+
+
+class FalconH1RotaryEmbedding(nn.Module):
+    inv_freq: torch.Tensor  # fix linting for `register_buffer`
+
+    def __init__(self, config: FalconH1Config, device=None):
+        super().__init__()
+        self.max_seq_len_cached = config.max_position_embeddings
+        self.original_max_seq_len = config.max_position_embeddings
+
+        self.config = config
+
+        self.rope_type = self.config.rope_parameters["rope_type"]
+        rope_init_fn: Callable = self.compute_default_rope_parameters
+        if self.rope_type != "default":
+            rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
+        inv_freq, self.attention_scaling = rope_init_fn(self.config, device)
+
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self.register_buffer("original_inv_freq", inv_freq.clone(), persistent=False)
+
+    @staticmethod
+    def compute_default_rope_parameters(
+        config: FalconH1Config | None = None,
+        device: Optional["torch.device"] = None,
+        seq_len: int | None = None,
+    ) -> tuple["torch.Tensor", float]:
+        """
+        Computes the inverse frequencies according to the original RoPE implementation
+        Args:
+            config ([`~transformers.PreTrainedConfig`]):
+                The model configuration.
+            device (`torch.device`):
+                The device to use for initialization of the inverse frequencies.
+            seq_len (`int`, *optional*):
+                The current sequence length. Unused for this type of RoPE.
+        Returns:
+            Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the
+            post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE).
+        """
+        base = config.rope_parameters["rope_theta"]
+        dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
+
+        attention_factor = 1.0  # Unused in this type of RoPE
+
+        # Compute the inverse frequencies
+        inv_freq = 1.0 / (
+            base ** (torch.arange(0, dim, 2, dtype=torch.int64).to(device=device, dtype=torch.float) / dim)
+        )
+        return inv_freq, attention_factor
+
+    @torch.no_grad()
+    @dynamic_rope_update  # power user: used with advanced RoPE types (e.g. dynamic rope)
+    def forward(self, x, position_ids):
+        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1).to(x.device)
+        position_ids_expanded = position_ids[:, None, :].float()
+
+        device_type = x.device.type if isinstance(x.device.type, str) and x.device.type != "mps" else "cpu"
+        with maybe_autocast(device_type=device_type, enabled=False):  # Force float32
+            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
+            emb = torch.cat((freqs, freqs), dim=-1)
+            cos = emb.cos() * self.attention_scaling
+            sin = emb.sin() * self.attention_scaling
+
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+@use_kernel_func_from_hub("rotary_pos_emb")
+def apply_rotary_pos_emb(q, k, cos, sin, unsqueeze_dim=1):
+    """Applies Rotary Position Embedding to the query and key tensors.
+
+    Args:
+        q (`torch.Tensor`): The query tensor.
+        k (`torch.Tensor`): The key tensor.
+        cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        sin (`torch.Tensor`): The sine part of the rotary embedding.
+        unsqueeze_dim (`int`, *optional*, defaults to 1):
+            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
+            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
+            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
+            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
+            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
+            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
+    Returns:
+        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
+    """
+    cos = cos.unsqueeze(unsqueeze_dim)
+    sin = sin.unsqueeze(unsqueeze_dim)
+    q_embed = (q * cos) + (rotate_half(q) * sin)
+    k_embed = (k * cos) + (rotate_half(k) * sin)
+    return q_embed, k_embed
+
+
+def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
+    """
+    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
+    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
+    """
+    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
+    if n_rep == 1:
+        return hidden_states
+    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
+    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
+
+
+def eager_attention_forward(
+    module: nn.Module,
+    query: torch.Tensor,
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: torch.Tensor | None,
+    scaling: float,
+    dropout: float = 0.0,
+    **kwargs: Unpack[TransformersKwargs],
+):
+    key_states = repeat_kv(key, module.num_key_value_groups)
+    value_states = repeat_kv(value, module.num_key_value_groups)
+
+    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
+    if attention_mask is not None:
+        attn_weights = attn_weights + attention_mask
+
+    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
+    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
+    attn_output = torch.matmul(attn_weights, value_states)
+    attn_output = attn_output.transpose(1, 2).contiguous()
+
+    return attn_output, attn_weights
+
+
+@use_kernelized_func(apply_rotary_pos_emb)
+class FalconH1Attention(nn.Module):
+    """Multi-headed attention from 'Attention Is All You Need' paper"""
+
+    def __init__(self, config: FalconH1Config, layer_idx: int):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
+        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
+        self.scaling = self.head_dim**-0.5
+        self.attention_dropout = config.attention_dropout
+        self.is_causal = True
+
+        self.q_proj = nn.Linear(
+            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.k_proj = nn.Linear(
+            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.v_proj = nn.Linear(
+            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.o_proj = nn.Linear(
+            config.num_attention_heads * self.head_dim, config.hidden_size, bias=config.attention_bias
+        )
+        self.key_multiplier = config.key_multiplier
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor],
+        attention_mask: torch.Tensor | None,
+        past_key_values: Cache | None = None,
+        **kwargs: Unpack[FlashAttentionKwargs],
+    ) -> tuple[torch.Tensor, torch.Tensor | None]:
+        input_shape = hidden_states.shape[:-1]
+        hidden_shape = (*input_shape, -1, self.head_dim)
+
+        query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
+        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2) * self.key_multiplier
+        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
+
+        cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_values is not None:
+            key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
+
+        attention_interface: Callable = ALL_ATTENTION_FUNCTIONS.get_interface(
+            self.config._attn_implementation, eager_attention_forward
+        )
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            dropout=0.0 if not self.training else self.attention_dropout,
+            scaling=self.scaling,
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
+        attn_output = self.o_proj(attn_output)
+        return attn_output, attn_weights
+
+
+class FalconH1RMSNormGated(torch.nn.Module):
+    def __init__(self, hidden_size, eps=1e-6, n_groups=1, norm_before_gate=True):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+        self.n_groups = n_groups
+        self.norm_before_gate = norm_before_gate
+
+    def forward(self, hidden_states, gate=None):
+        input_dtype = hidden_states.dtype
+
+        if not self.norm_before_gate and gate is not None:
+            hidden_states = hidden_states * F.silu(gate.to(torch.float32))
+
+        if len(hidden_states.shape) == 3:
+            batch_size, seq_len, dim = hidden_states.shape
+        else:
+            batch_size, dim = hidden_states.shape
+            seq_len = 1
+        hidden_states = hidden_states.to(torch.float32)
+
+        hidden_states = hidden_states.view(batch_size, seq_len, self.n_groups, int(dim // self.n_groups))
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+
+        hidden_states = self.weight.view(self.n_groups, int(dim // self.n_groups)) * hidden_states
+        hidden_states = hidden_states.view(batch_size, seq_len, dim)
+
+        if seq_len == 1:
+            hidden_states = hidden_states.squeeze(1)
+
+        if self.norm_before_gate and gate is not None:
+            hidden_states = hidden_states * F.silu(gate.to(torch.float32))
+        return hidden_states.to(input_dtype)
+
+
+# Helper methods for segment sum computation
+
+
+def pad_tensor_by_size(input_tensor: torch.Tensor, pad_size: int):
+    """
+    Padding x tensor with `pad_size` on the seq_len dim (dim=1)
+
+    Assumes that we only have tensors of either size 4 or 3
+    """
+    pad_shape = (0, 0, 0, 0, 0, pad_size, 0, 0) if len(input_tensor.shape) == 4 else (0, 0, 0, pad_size, 0, 0)
+
+    return torch.nn.functional.pad(input_tensor, pad_shape, mode="constant", value=0)
+
+
+def reshape_into_chunks(input_tensor, pad_size, chunk_size):
+    """
+    Padding input_tensor with `pad_size` on the seq_len dim (dim=1) and
+    simultaneously splitting it into chunk sequences.
+
+    Assumes that we only have tensors of either size 4 or 3
+    """
+    # [bsz, seq_len, ...] -> [bsz, seq_len multiple of chunk_size, ...]
+    input_tensor = pad_tensor_by_size(input_tensor, pad_size)
+
+    if len(input_tensor.shape) == 3:
+        # [bsz, seq_len multiple of chunk_size, num_heads] -> [bsz, -1, chunk_size, num_heads]
+        return input_tensor.reshape(input_tensor.shape[0], -1, chunk_size, input_tensor.shape[2])
+    else:
+        # [bsz, seq_len multiple of chunk_size, num_heads, head_dim or state_size] -> [bsz, -1, chunk_size, num_heads, head_dim or state_size]
+        return input_tensor.reshape(
+            input_tensor.shape[0], -1, chunk_size, input_tensor.shape[2], input_tensor.shape[3]
+        )
+
+
+def segment_sum(input_tensor):
+    """
+    More stable segment sum calculation. Uses cumulative sums and masking instead of direct subtractions.
+    """
+    chunk_size = input_tensor.size(-1)
+    # 1. expand input tensor to have an additional dimension and repeat along that dimension
+    # [..., chunk_size] -> [..., chunk_size, chunk_size]
+    input_tensor = input_tensor[..., None].expand(*input_tensor.size(), chunk_size)
+    # 2. create a lower triangular mask with the diagonal set to 0 to 0 out elements above diag
+    mask = torch.tril(torch.ones(chunk_size, chunk_size, device=input_tensor.device, dtype=torch.bool), diagonal=-1)
+    input_tensor = input_tensor.masked_fill(~mask, 0)
+    # 3. compute actual cumsum
+    tensor_segsum = torch.cumsum(input_tensor, dim=-2)
+
+    # 4. apply mask to keep only the lower triangular part of the cumulative sum result (incl diagonal this time)
+    mask = torch.tril(torch.ones(chunk_size, chunk_size, device=input_tensor.device, dtype=torch.bool), diagonal=0)
+    tensor_segsum = tensor_segsum.masked_fill(~mask, -torch.inf)
+    return tensor_segsum
+
+
+def apply_mask_to_padding_states(hidden_states, attention_mask):
+    """
+    Tunes out the hidden states for padding tokens, see https://github.com/state-spaces/mamba/issues/66
+    """
+    # NOTE: attention mask is a 2D boolean tensor
+    if attention_mask is not None and attention_mask.shape[1] > 1 and attention_mask.shape[0] > 1:
+        dtype = hidden_states.dtype
+        hidden_states = (hidden_states * attention_mask[:, :, None]).to(dtype)
+
+    return hidden_states
+
+
+# Adapted from transformers.models.mamba2.modeling_mamba2.Mamba2Mixer
+class FalconH1Mixer(nn.Module):
+    """
+    FalconH1Mixer is identical to classic Mamba2 mixer classes but differs on two different things
+    - Users can pass custom intermediate_size through `config.mamba_d_ssm`
+    - The use of gated RMS normalization layer is optional
+    """
+
+    def __init__(self, config: FalconH1Config, layer_idx: int):
+        super().__init__()
+        self.num_heads = config.mamba_n_heads
+        self.hidden_size = config.hidden_size
+        self.ssm_state_size = config.mamba_d_state
+        self.conv_kernel_size = config.mamba_d_conv
+        self.intermediate_size = (
+            int(config.mamba_expand * self.hidden_size) if config.mamba_d_ssm is None else config.mamba_d_ssm
+        )
+        self.layer_idx = layer_idx
+        self.use_conv_bias = config.mamba_conv_bias
+        self.activation = config.hidden_act
+        self.act = ACT2FN[config.hidden_act]
+        self.use_bias = config.mamba_proj_bias
+
+        self.layer_norm_epsilon = config.rms_norm_eps
+        self.groups_time_state_size = config.mamba_n_groups * self.ssm_state_size
+
+        self.n_groups = config.mamba_n_groups
+        self.head_dim = config.mamba_d_head
+        self.chunk_size = config.mamba_chunk_size
+
+        self.time_step_limit = config.time_step_limit
+        self.time_step_min = config.time_step_min
+        self.time_step_max = config.time_step_max
+
+        self.conv_dim = self.intermediate_size + 2 * self.n_groups * self.ssm_state_size
+        self.conv1d = nn.Conv1d(
+            in_channels=self.conv_dim,
+            out_channels=self.conv_dim,
+            bias=config.mamba_conv_bias,
+            kernel_size=self.conv_kernel_size,
+            groups=self.conv_dim,
+            padding=self.conv_kernel_size - 1,
+        )
+
+        # projection of the input hidden states
+        projection_size = self.intermediate_size + self.conv_dim + self.num_heads
+        self.in_proj = nn.Linear(
+            self.hidden_size,
+            projection_size,
+            bias=self.use_bias,
+        )
+        # selective projection used to make dt, B and C input dependant
+
+        # time step projection (discretization)
+        # instantiate once and copy inv_dt in init_weights of PretrainedModel
+        self.dt_bias = nn.Parameter(torch.ones(self.num_heads))
+
+        # S4D real initialization. These are not discretized!
+        # The core is to load them, compute the discrete states, then write the updated state. Keeps the memory bounded
+        A = torch.arange(1, self.num_heads + 1)
+        self.A_log = nn.Parameter(torch.log(A))
+        self.mamba_rms_norm = config.mamba_rms_norm
+
+        if self.mamba_rms_norm:
+            self.norm = FalconH1RMSNormGated(
+                self.intermediate_size,
+                eps=self.layer_norm_epsilon,
+                n_groups=self.n_groups,
+                norm_before_gate=config.mamba_norm_before_gate,
+            )
+        self.D = nn.Parameter(torch.ones(self.num_heads))
+
+        self.out_proj = nn.Linear(self.intermediate_size, config.hidden_size, bias=config.projectors_bias)
+
+        global causal_conv1d_update, causal_conv1d_fn
+        causal_conv1d = lazy_load_kernel("causal-conv1d")
+        causal_conv1d_update = getattr(causal_conv1d, "causal_conv1d_update", None)
+        causal_conv1d_fn = getattr(causal_conv1d, "causal_conv1d_fn", None)
+
+        global selective_state_update, mamba_chunk_scan_combined, mamba_split_conv1d_scan_combined
+        mamba_ssm = lazy_load_kernel("mamba-ssm")
+        selective_state_update = resolve_internal_import(
+            mamba_ssm, chained_path="ops.triton.selective_state_update.selective_state_update"
+        )
+        mamba_chunk_scan_combined = resolve_internal_import(
+            mamba_ssm, chained_path="ops.triton.ssd_combined.mamba_chunk_scan_combined"
+        )
+        mamba_split_conv1d_scan_combined = resolve_internal_import(
+            mamba_ssm, chained_path="ops.triton.ssd_combined.mamba_split_conv1d_scan_combined"
+        )
+
+        global is_fast_path_available
+        is_fast_path_available = all(
+            (
+                selective_state_update,
+                mamba_chunk_scan_combined,
+                mamba_split_conv1d_scan_combined,
+                causal_conv1d_fn,
+                causal_conv1d_update,
+            )
+        )
+
+        if not is_fast_path_available:
+            logger.warning_once(
+                "The fast path is not available because one of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)`"
+                " is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and"
+                " https://github.com/Dao-AILab/causal-conv1d"
+            )
+        else:
+            logger.warning_once("The fast path for FalconH1 will be used when running the model on a GPU")
+
+        self.zxbcdt_multipliers = config.ssm_multipliers
+        self.ssm_in_multiplier = config.ssm_in_multiplier
+
+    def cuda_kernels_forward(
+        self,
+        hidden_states: torch.Tensor,
+        cache_params: Cache | None = None,
+        attention_mask: torch.Tensor | None = None,
+    ):
+        # 1. Gated MLP's linear projection
+        hidden_states = apply_mask_to_padding_states(hidden_states, attention_mask)
+        # Add Multipliers
+        hidden_states = hidden_states * self.ssm_in_multiplier
+        projected_states = self.in_proj(hidden_states)
+        projected_states = projected_states * self.mup_vector  # ADD Mup Multipliers
+        d_to_remove = 2 * self.intermediate_size + 2 * self.n_groups * self.ssm_state_size + self.num_heads
+
+        # Set up dimensions for reshapes later
+        batch_size, seq_len, _ = hidden_states.shape
+        groups_time_state_size = self.n_groups * self.ssm_state_size
+
+        use_precomputed_states = (
+            cache_params is not None and cache_params.has_previous_state(self.layer_idx) and seq_len == 1
+        )
+
+        # getting projected states from cache if it exists
+        if use_precomputed_states:
+            d_mlp = (projected_states.squeeze(1).shape[-1] - d_to_remove) // 2
+
+            z0, x0, gate, hidden_states_B_C, dt = projected_states.squeeze(1).split(
+                [d_mlp, d_mlp, self.intermediate_size, self.conv_dim, self.num_heads], dim=-1
+            )
+
+            # 2. Convolution sequence transformation
+            hidden_states_B_C = causal_conv1d_update(
+                hidden_states_B_C,
+                cache_params.layers[self.layer_idx].conv_states,
+                self.conv1d.weight.squeeze(1),
+                self.conv1d.bias,
+                self.activation,
+            )
+
+            hidden_states, B, C = torch.split(
+                hidden_states_B_C,
+                [self.intermediate_size, groups_time_state_size, groups_time_state_size],
+                dim=-1,
+            )
+
+            # 3. SSM transformation
+            A = -torch.exp(self.A_log.float())  # (nheads,)
+            A = A[:, None, ...][:, :, None].expand(-1, self.head_dim, self.ssm_state_size).to(dtype=torch.float32)
+            dt = dt[:, :, None].expand(-1, -1, self.head_dim)
+            dt_bias = self.dt_bias[:, None, ...].expand(-1, self.head_dim)
+            D = self.D[:, None, ...].expand(-1, self.head_dim)
+            B = B.view(batch_size, self.n_groups, B.shape[1] // self.n_groups)
+            C = C.view(batch_size, self.n_groups, C.shape[1] // self.n_groups)
+            hidden_states_reshaped = hidden_states.view(batch_size, self.num_heads, self.head_dim)
+            hidden_states = selective_state_update(
+                cache_params.layers[self.layer_idx].recurrent_states,
+                hidden_states_reshaped,
+                dt,
+                A,
+                B,
+                C,
+                D,
+                z=gate.view(batch_size, self.num_heads, self.head_dim) if not self.mamba_rms_norm else None,
+                dt_bias=dt_bias,
+                dt_softplus=True,
+            )
+            hidden_states = hidden_states.view(batch_size, self.num_heads * self.head_dim)
+
+            if self.mamba_rms_norm:
+                hidden_states = self.norm(hidden_states, gate)
+
+            if d_mlp > 0:
+                hidden_states = torch.cat([F.silu(z0) * x0, hidden_states], dim=-1)
+
+            # 4. Final linear projection
+            out = self.out_proj(hidden_states[:, None, ...])
+        # Fused calculations or step by step if no initialized cache is found
+        else:
+            A = -torch.exp(self.A_log.float())  # (num_heads) or (intermediate_size, state_size)
+            dt_limit_kwargs = {} if self.time_step_limit == (0.0, float("inf")) else {"dt_limit": self.time_step_limit}
+
+            # 2-4. Fused kernel for conv1d, SSM, and the final projection
+            if self.training and cache_params is None:
+                out = mamba_split_conv1d_scan_combined(
+                    projected_states,
+                    self.conv1d.weight.squeeze(1),
+                    self.conv1d.bias,
+                    self.dt_bias,
+                    A,
+                    D=self.D,
+                    chunk_size=self.chunk_size,
+                    seq_idx=None,  # was seq_idx
+                    activation=self.activation,
+                    rmsnorm_weight=self.norm.weight if self.mamba_rms_norm else None,
+                    rmsnorm_eps=self.norm.variance_epsilon if self.mamba_rms_norm else None,
+                    outproj_weight=self.out_proj.weight,
+                    outproj_bias=self.out_proj.bias,
+                    headdim=self.head_dim,
+                    ngroups=self.n_groups,
+                    norm_before_gate=False,
+                    return_final_states=False,
+                    **dt_limit_kwargs,
+                )
+
+            else:
+                d_mlp = (
+                    projected_states.shape[-1]
+                    - 2 * self.intermediate_size
+                    - 2 * self.n_groups * self.ssm_state_size
+                    - self.num_heads
+                ) // 2
+                if attention_mask is not None:
+                    projected_states = projected_states * attention_mask[..., None]
+                _, gate, hidden_states_B_C, dt = projected_states.split(
+                    [
+                        2 * d_mlp,
+                        self.intermediate_size,
+                        self.conv_dim,
+                        self.num_heads,
+                    ],
+                    dim=-1,
+                )
+
+                if cache_params is not None:
+                    conv_states = F.pad(
+                        hidden_states_B_C.permute(0, 2, 1),
+                        (self.conv_kernel_size - hidden_states_B_C.shape[-2], 0),
+                    )
+                    conv_states = cache_params.update_conv_state(conv_states, self.layer_idx)
+
+                time_step = nn.functional.softplus(dt + self.dt_bias)
+                # 1D Convolution
+                if causal_conv1d_fn is None or self.activation not in ["silu", "swish"]:
+                    hidden_states_B_C = self.act(
+                        self.conv1d(hidden_states_B_C.transpose(1, 2)).transpose(1, 2)[:, :seq_len]
+                    )  # (B, L, self.d_inner + 2 * ngroups * d_state)
+                else:
+                    hidden_states_B_C = causal_conv1d_fn(
+                        x=hidden_states_B_C.transpose(1, 2),
+                        weight=self.conv1d.weight.squeeze(1),
+                        bias=self.conv1d.bias,
+                        activation=self.activation,
+                    ).transpose(1, 2)[:, :seq_len]
+
+                hidden_states, B, C = torch.split(
+                    hidden_states_B_C,
+                    [
+                        self.intermediate_size,
+                        groups_time_state_size,
+                        groups_time_state_size,
+                    ],
+                    dim=-1,
+                )
+
+                if attention_mask is not None and attention_mask.shape[1] > 1 and attention_mask.shape[0] > 1:
+                    # tune out hidden states for pad tokens, see https://github.com/state-spaces/mamba/issues/66
+                    dtype = hidden_states.dtype
+                    hidden_states = (hidden_states * attention_mask[:, :, None]).to(dtype)
+                # This is a hack to make sure multi-GPU inference works with HF accelerate
+                # see: https://github.com/Dao-AILab/flash-attention/issues/523 for more details
+                with torch.cuda.device(hidden_states.device):
+                    scan_output, ssm_state = mamba_chunk_scan_combined(
+                        hidden_states.view(batch_size, seq_len, -1, self.head_dim),
+                        time_step,
+                        A,
+                        B.view(batch_size, seq_len, self.n_groups, -1),
+                        C.view(batch_size, seq_len, self.n_groups, -1),
+                        chunk_size=self.chunk_size,
+                        D=self.D,
+                        z=None,
+                        seq_idx=None,
+                        return_final_states=True,
+                        **dt_limit_kwargs,
+                    )
+                if ssm_state is not None and cache_params is not None:
+                    ssm_state = cache_params.update_recurrent_state(ssm_state, self.layer_idx)
+                scan_output = scan_output.view(batch_size, seq_len, -1)
+                # Multiply "gate" branch and apply extra normalization layer
+                if self.mamba_rms_norm:
+                    out = self.norm(scan_output, gate)
+                else:
+                    out = scan_output * torch.nn.functional.silu(gate)
+                out = self.out_proj(out)
+        return out
+
+    # fmt: off
+    def torch_forward(
+        self,
+        input_states,
+        cache_params: Cache | None = None,
+        attention_mask: torch.Tensor | None = None,
+    ):
+        batch_size, seq_len, _ = input_states.shape
+        dtype = input_states.dtype
+
+        # 1. Gated MLP's linear projection
+        input_states = apply_mask_to_padding_states(input_states, attention_mask)
+        # Add Multipliers
+        input_states = input_states * self.ssm_in_multiplier
+        projected_states = self.in_proj(input_states)
+        projected_states = projected_states * self.mup_vector  # ADD Mup Multipliers
+        gate, hidden_states_B_C, dt = projected_states.split([
+                self.intermediate_size, self.conv_dim, self.num_heads
+            ], dim=-1)
+        hidden_states_B_C = hidden_states_B_C.transpose(1,2)
+
+        use_precomputed_states = cache_params is not None and cache_params.has_previous_state(self.layer_idx) and seq_len == 1
+
+        # 2. Convolution sequence transformation
+        if use_precomputed_states:
+            conv_states = cache_params.update_conv_state(hidden_states_B_C, self.layer_idx)
+            # We need to guarantee that anything regarding the cache is on the same device
+            conv_states = conv_states.to(device=self.conv1d.weight.device)
+
+            hidden_states_B_C = torch.sum(
+                conv_states * self.conv1d.weight.squeeze(1), dim=-1
+            )
+            if self.use_conv_bias:
+                hidden_states_B_C = hidden_states_B_C + self.conv1d.bias
+            hidden_states_B_C = self.act(hidden_states_B_C)
+        else:
+            # Init cache
+            if cache_params is not None:
+                conv_states = nn.functional.pad(
+                    hidden_states_B_C, (self.conv_kernel_size - hidden_states_B_C.shape[-1], 0)
+                )
+                conv_states = cache_params.update_conv_state(conv_states, self.layer_idx)
+
+            hidden_states_B_C = self.act(self.conv1d(hidden_states_B_C)[..., :seq_len].transpose(1, 2))
+
+        hidden_states_B_C = apply_mask_to_padding_states(hidden_states_B_C, attention_mask)
+        hidden_states, B, C = torch.split(
+            hidden_states_B_C,
+            [self.intermediate_size, self.n_groups * self.ssm_state_size, self.n_groups * self.ssm_state_size],
+            dim=-1
+        )
+
+        # 3. SSM transformation
+        A = -torch.exp(self.A_log.float())                            # [num_heads]
+        if use_precomputed_states:
+            # We need to guarantee that anything regarding the cache is on the same device
+            cache_device = cache_params.layers[self.layer_idx].recurrent_states.device
+
+            # Note: there is no need to pad parameter matrices here, as there is just one new token
+            # for batched generation
+            dt = dt[:, 0, :][:, None, ...]
+            dt = dt.transpose(1, 2).expand(batch_size, dt.shape[-1], self.head_dim)
+            # [num_heads] -> [num_heads, head_dim]
+            dt_bias = self.dt_bias[..., None].expand(self.dt_bias.shape[0], self.head_dim)
+
+            dt = torch.nn.functional.softplus(dt + dt_bias.to(dt.dtype))
+            dt = torch.clamp(dt, self.time_step_limit[0], self.time_step_limit[1])
+            A = A[..., None, None].expand(self.num_heads, self.head_dim, self.ssm_state_size).to(dtype=torch.float32)
+            # [bsz, num_heads, head_dim, state_size]
+            dA = (torch.exp(dt[..., None] * A)).to(device=cache_device)
+
+            # Discretize B
+            # [bsz, n_groups * state_size] -> [bsz, n_groups, 1, state_size] ->
+            # -> [bsz, n_groups, group to head repetition factor, state_size] -> [bsz, num_heads, state_size]
+            B = B.reshape(batch_size, self.n_groups, -1)[..., None, :]
+            B = B.expand(batch_size, self.n_groups, self.num_heads // self.n_groups, B.shape[-1]).contiguous()
+            B = B.reshape(batch_size, -1, B.shape[-1])
+            # [bsz, num_heads, head_dim, state_size]
+            dB = dt[..., None] * B[..., None, :]
+
+            # Discretize x into dB
+            # [bsz, intermediate_size] -> [bsz, num_heads, head_dim]
+            hidden_states = hidden_states.reshape(batch_size, -1, self.head_dim)
+            dBx = (dB * hidden_states[..., None]).to(device=cache_device)
+
+            # State calculation
+            ssm_states = cache_params.layers[self.layer_idx].recurrent_states * dA + dBx
+            ssm_states = cache_params.update_recurrent_state(ssm_states, self.layer_idx)
+
+            # Subsequent output
+            # [bsz, n_groups * state_size] -> [bsz, num_heads, state_size]
+            C = C.reshape(batch_size, self.n_groups, -1)[..., None, :]
+            C = C.expand(batch_size, self.n_groups, self.num_heads // self.n_groups, C.shape[-1]).contiguous()
+            C = C.reshape(batch_size, -1, C.shape[-1])
+            # [bsz, num_heads, head_dim]
+
+            ssm_states = ssm_states.to(device=C.device, dtype=C.dtype)  # Shape: [b, h, d, n]
+            # Reshape ssm_states to merge the first two dimensions
+            ssm_states_reshaped = ssm_states.view(batch_size * self.num_heads, self.head_dim, self.ssm_state_size)  # Shape: [b*h, d, n]
+            C_reshaped = C.view(batch_size * self.num_heads, self.ssm_state_size, 1)  # Shape: [b*h, n, 1]
+            y = torch.bmm(ssm_states_reshaped, C_reshaped)
+            y = y.view(batch_size, self.num_heads, self.head_dim)
+
+            # D skip connection
+            # [num_heads] -> [num_heads, head_dim]
+            D = self.D[..., None].expand(self.D.shape[0], self.head_dim)
+            y = (y + hidden_states * D).to(y.dtype)
+
+            # [bsz, num_heads, head_dim] -> [bsz, 1, intermediate_size]
+            y = y.reshape(batch_size, -1)[:, None, ...]
+        else:
+            # begin ssd naive implementation without einsums
+            dt = nn.functional.softplus(dt + self.dt_bias)
+            dt = torch.clamp(dt, self.time_step_limit[0], self.time_step_limit[1])
+            hidden_states = hidden_states.reshape(batch_size, seq_len, -1, self.head_dim).float()
+            B = B.reshape(batch_size, seq_len, -1, self.ssm_state_size).float()
+            C = C.reshape(batch_size, seq_len, -1, self.ssm_state_size).float()
+            B = B.repeat_interleave(self.num_heads // self.n_groups, dim=2, output_size=self.num_heads)
+            C = C.repeat_interleave(self.num_heads // self.n_groups, dim=2, output_size=self.num_heads)
+            pad_size = (self.chunk_size - seq_len % self.chunk_size) % self.chunk_size
+
+            D_residual = self.D[..., None] * pad_tensor_by_size(hidden_states, pad_size)
+
+            # Discretize x and A
+            hidden_states = hidden_states * dt[..., None]
+            A = A.to(hidden_states.dtype) * dt
+
+            # Rearrange into blocks/chunks
+            hidden_states, A, B, C = [reshape_into_chunks(t, pad_size, self.chunk_size) for t in (hidden_states, A, B, C)]
+
+            # [bsz, -1, chunk_size, num_heads] -> [bsz, num_heads, -1, chunk_size]
+            A = A.permute(0, 3, 1, 2)
+            A_cumsum = torch.cumsum(A, dim=-1)
+
+            # 1. Compute the output for each intra-chunk (diagonal blocks)
+            # This is the analog of a causal mask
+            L = torch.exp(segment_sum(A))
+
+            # Contraction of C and B to get G (attention-weights like)
+            G_intermediate = C[:, :, :, None, :, :] * B[:, :, None, :, :, :]  # shape: (b, c, l, s, h, n)
+            G = G_intermediate.sum(dim=-1)  # shape: (b, c, l, s, h)
+
+            # Compute M, equivalent to applying attention mask to weights
+            M_intermediate = G[..., None] * L.permute(0, 2, 3, 4, 1)[..., None]
+            M = M_intermediate.sum(dim=-1)
+
+            # Compute Y_diag (apply to values)
+            Y_diag = (M[..., None] * hidden_states[:, :, None]).sum(dim=3)
+
+            # 2. Compute the state for each intra-chunk
+            # (right term of low-rank factorization of off-diagonal blocks; B terms)
+            decay_states = torch.exp(A_cumsum[:, :, :, -1:] - A_cumsum)
+            B_decay = B * decay_states.permute(0, -2, -1, 1)[..., None]
+            states = (B_decay[..., None, :] * hidden_states[..., None]).sum(dim=2)
+
+            # 3. Compute the inter-chunk SSM recurrence; produces correct SSM states at chunk boundaries
+            # (middle term of factorization of off-diag blocks; A terms)
+            previous_states = torch.zeros_like(states[:, :1])
+            states = torch.cat([previous_states, states], dim=1)
+            decay_chunk = torch.exp(segment_sum(nn.functional.pad(A_cumsum[:, :, :, -1], (1, 0))))
+            decay_chunk = decay_chunk.transpose(1, 3)
+            new_states = (decay_chunk[..., None, None] * states[:, :, None, ...]).sum(dim=1)
+            states, ssm_state = new_states[:, :-1], new_states[:, -1]
+
+            # 4. Compute state -> output conversion per chunk
+            # (left term of low-rank factorization of off-diagonal blocks; C terms)
+            state_decay_out = torch.exp(A_cumsum)
+            C_times_states = (C[..., None, :] * states[:, :, None, ...])
+            state_decay_out_permuted = state_decay_out.permute(0, 2, 3, 1)
+            Y_off = (C_times_states.sum(-1) * state_decay_out_permuted[..., None])
+
+            # Add output of intra-chunk and inter-chunk terms (diagonal and off-diagonal blocks)
+            y = Y_diag + Y_off
+            # [bsz, -1, self.chunk_size, num_heads, head_dim] -> [bsz, (padded) seq_len, num_heads, head_dim]
+            y = y.reshape(batch_size, -1, self.num_heads, self.head_dim)
+
+            y = y + D_residual
+            # Cutting off padded chunks
+            if pad_size > 0:
+                y = y[:, :seq_len, :, :]
+            y = y.reshape(batch_size, seq_len, -1)
+
+            # Init cache
+            if ssm_state is not None and cache_params is not None:
+                ssm_state = cache_params.update_recurrent_state(ssm_state, self.layer_idx)
+
+        if self.mamba_rms_norm:
+            scan_output = self.norm(y, gate)
+        else:
+            scan_output = y * torch.nn.functional.silu(gate)
+
+        # end ssd naive
+
+        # 4. Final linear projection
+        contextualized_states = self.out_proj(scan_output.to(dtype))  # [batch, seq_len, hidden_size]
+        return contextualized_states
+    # fmt: on
+
+    def forward(
+        self,
+        hidden_states,
+        cache_params: Cache | None = None,
+        attention_mask: torch.Tensor | None = None,
+        **kwargs,
+    ):
+        if is_fast_path_available and "cuda" in self.in_proj.weight.device.type and not is_torchdynamo_compiling():
+            return self.cuda_kernels_forward(hidden_states, cache_params, attention_mask)
+        dtype = hidden_states.dtype
+        if attention_mask is not None and attention_mask.shape[1] > 1 and attention_mask.shape[0] > 1:
+            # tune out hidden states for pad tokens, see https://github.com/state-spaces/mamba/issues/66
+            hidden_states = (hidden_states * attention_mask[:, :, None]).to(dtype)
+
+        return self.torch_forward(hidden_states, cache_params, attention_mask)
+
+
+class FalconH1MLP(nn.Module):
+    def __init__(self, config: FalconH1Config):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size
+        self.intermediate_size = config.intermediate_size
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
+        self.act_fn = ACT2FN[config.hidden_act]
+        self.gate_multiplier, self.down_multiplier = config.mlp_multipliers
+
+    def forward(self, x):
+        y = self.up_proj(x) * self.act_fn(self.gate_proj(x) * self.gate_multiplier)
+        y = self.down_proj(y) * self.down_multiplier
+        return y
+
+
+@use_kernel_forward_from_hub("RMSNorm")
+class FalconH1RMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps: float = 1e-6) -> None:
+        """
+        FalconH1RMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+    def extra_repr(self):
+        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
+
+
+class FalconH1DecoderLayer(GradientCheckpointingLayer):
+    def __init__(self, config: FalconH1Config, layer_idx: int):
+        super().__init__()
+        self.feed_forward = FalconH1MLP(config)
+
+        head_dim = config.hidden_size // config.num_attention_heads
+        self.channels_attn = config.num_attention_heads * head_dim + 2 * config.num_key_value_heads * head_dim
+
+        self.mamba = FalconH1Mixer(config=config, layer_idx=layer_idx)
+
+        self.self_attn = FalconH1Attention(config, layer_idx)
+
+        self.attention_in_multiplier = config.attention_in_multiplier
+        self.ssm_out_multiplier = config.ssm_out_multiplier
+        self.attn_out_multiplier = config.attention_out_multiplier
+
+        self.input_layernorm = FalconH1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.pre_ff_layernorm = FalconH1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: torch.Tensor | None = None,
+        mamba_attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        use_cache: bool | None = False,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
+        **kwargs,
+    ) -> tuple[torch.FloatTensor]:
+        """
+        Args:
+            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
+            attention_mask (`torch.FloatTensor`, *optional*): attention mask of size
+                `(batch, sequence_length)` where padding elements are indicated by 0.
+            past_key_values (`Cache`, *optional*): cached past key and value projection states
+            use_cache (`bool`, *optional*):
+                If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
+                (see `past_key_values`).
+            position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*):
+                Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`,
+                with `head_dim` being the embedding dimension of each attention head.
+            kwargs (`dict`, *optional*):
+                Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code
+                into the model
+        """
+
+        residual = hidden_states
+        hidden_states = self.input_layernorm(hidden_states)
+
+        mamba_hidden_states = self.mamba(
+            hidden_states=hidden_states,
+            cache_params=past_key_values,
+            attention_mask=mamba_attention_mask,
+        )
+        mamba_hidden_states = mamba_hidden_states * self.ssm_out_multiplier
+
+        attention_hidden_states, _ = self.self_attn(
+            hidden_states=hidden_states * self.attention_in_multiplier,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            position_embeddings=position_embeddings,
+            **kwargs,
+        )
+        attention_hidden_states = attention_hidden_states * self.attn_out_multiplier
+
+        hidden_states = mamba_hidden_states + attention_hidden_states
+
+        # residual connection after attention
+        hidden_states = residual + hidden_states
+
+        # feed-forward
+        residual = hidden_states
+        hidden_states = self.pre_ff_layernorm(hidden_states)
+        hidden_states = self.feed_forward(hidden_states)
+        hidden_states = residual + hidden_states
+
+        return (hidden_states,)
+
+
+def compute_mup_vector(config):
+    """
+    Computes the MuP vector based on model configuration.
+
+    FalconH1 applies different MuP multiplier for each dimension of the hidden states.
+    The MuP vector is partitioned into chunks, and each chunk is multiplied with its
+    corresponding projected dimension.
+
+    Args:
+        config: FalconH1Config object
+
+    Returns:
+        torch.Tensor: The computed MuP vector
+    """
+    # We'll need some values from the config to compute the vector dimensions
+    intermediate_size = (
+        config.mamba_d_ssm if config.mamba_d_ssm is not None else int(config.mamba_expand * config.hidden_size)
+    )
+    groups_time_state_size = config.mamba_n_groups * config.mamba_d_state
+    num_heads = config.mamba_n_heads
+    zxbcdt_multipliers = config.ssm_multipliers
+
+    vector_shape = 2 * intermediate_size + 2 * groups_time_state_size + num_heads
+    mup_vector = torch.ones(1, 1, vector_shape)
+
+    # Apply multipliers to different sections of the vector
+    mup_vector[:, :, :intermediate_size] *= zxbcdt_multipliers[0]
+    mup_vector[:, :, intermediate_size : 2 * intermediate_size] *= zxbcdt_multipliers[1]
+    mup_vector[:, :, 2 * intermediate_size : 2 * intermediate_size + groups_time_state_size] *= zxbcdt_multipliers[2]
+    mup_vector[
+        :, :, 2 * intermediate_size + groups_time_state_size : 2 * intermediate_size + 2 * groups_time_state_size
+    ] *= zxbcdt_multipliers[3]
+    mup_vector[:, :, 2 * intermediate_size + 2 * groups_time_state_size :] *= zxbcdt_multipliers[4]
+
+    return mup_vector
+
+
+@auto_docstring
+class FalconH1PreTrainedModel(PreTrainedModel):
+    config: FalconH1Config
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["FalconH1DecoderLayer"]
+    _skip_keys_device_placement = ["past_key_values"]
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    _is_stateful = True
+
+    _can_record_outputs = {
+        "hidden_states": FalconH1DecoderLayer,
+        "attentions": FalconH1Attention,
+    }
+
+    @torch.no_grad()
+    def _init_weights(self, module):
+        super()._init_weights(module)
+        if isinstance(module, FalconH1Mixer):
+            init.ones_(module.dt_bias)
+            init.copy_(module.A_log, torch.log(torch.arange(1, module.num_heads + 1)))
+            init.ones_(module.D)
+        elif isinstance(module, FalconH1Model):
+            mup_vector = compute_mup_vector(module.config)
+            for layer in module.layers:
+                init.copy_(layer.mamba.mup_vector, mup_vector)
+
+
+@auto_docstring
+# Adapted from transformers.models.jamba.modeling_jamba.JambaModel
+class FalconH1Model(FalconH1PreTrainedModel):
+    def __init__(self, config: FalconH1Config):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        decoder_layers = []
+        for i in range(config.num_hidden_layers):
+            decoder_layers.append(FalconH1DecoderLayer(config, layer_idx=i))
+        self.layers = nn.ModuleList(decoder_layers)
+
+        self._attn_implementation = config._attn_implementation
+        self.final_layernorm = FalconH1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.rotary_emb = FalconH1RotaryEmbedding(config=config)
+
+        self.embedding_multiplier = config.embedding_multiplier
+        self.lm_head_multiplier = config.lm_head_multiplier
+
+        self.gradient_checkpointing = False
+        # Compute the MuP vector once and register it for all layers
+        mup_vector = compute_mup_vector(config)
+        for layer in self.layers:
+            layer.mamba.register_buffer("mup_vector", mup_vector.clone(), persistent=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @merge_with_config_defaults
+    @capture_outputs
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        use_cache: bool | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutputWithPast:
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embed_tokens(input_ids) * self.embedding_multiplier
+        hidden_states = inputs_embeds
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache(config=self.config)
+
+        if position_ids is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            position_ids = torch.arange(hidden_states.shape[1], device=hidden_states.device) + past_seen_tokens
+            position_ids = position_ids.unsqueeze(0)
+
+        causal_mask = create_causal_mask(
+            config=self.config,
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            position_ids=position_ids,
+        )
+        mamba_mask = self._update_mamba_mask(attention_mask, past_key_values)
+        position_embeddings = self.rotary_emb(hidden_states, position_ids=position_ids)
+
+        for decoder_layer in self.layers:
+            layer_outputs = decoder_layer(
+                hidden_states,
+                attention_mask=causal_mask,
+                mamba_attention_mask=mamba_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                use_cache=use_cache,
+                position_embeddings=position_embeddings,
+            )
+
+            hidden_states = layer_outputs[0]
+
+        hidden_states = self.final_layernorm(hidden_states)
+
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+        )
+
+    def _update_mamba_mask(self, attention_mask, past_key_values):
+        """
+        No need for zeroing states when
+            1. Cached forward
+            2. Attending to all inputs
+        """
+        mamba_mask = attention_mask
+        if (past_key_values is not None and past_key_values.has_previous_state()) or (
+            attention_mask is not None and torch.all(attention_mask == 1)
+        ):
+            mamba_mask = None
+        return mamba_mask
+
+
+@auto_docstring
+class FalconH1ForCausalLM(FalconH1PreTrainedModel, GenerationMixin):
+    _tied_weights_keys = {"lm_head.weight": "model.embed_tokens.weight"}
+    _tp_plan = {"lm_head": "colwise_gather_output"}
+    _pp_plan = {"lm_head": (["hidden_states"], ["logits"])}
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = FalconH1Model(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        labels: torch.LongTensor | None = None,
+        use_cache: bool | None = None,
+        logits_to_keep: int | torch.Tensor = 0,
+        **kwargs,
+    ) -> tuple | CausalLMOutputWithPast:
+        r"""
+        Example:
+
+        ```python
+        >>> from transformers import AutoTokenizer, FalconH1ForCausalLM
+
+        >>> model = FalconH1ForCausalLM.from_pretrained("...")
+        >>> tokenizer = AutoTokenizer.from_pretrained("...")
+
+        >>> prompt = "Hey, are you conscious? Can you talk to me?"
+        >>> inputs = tokenizer(prompt, return_tensors="pt")
+
+        >>> # Generate
+        >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
+        >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
+        "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
+        ```"""
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            **kwargs,
+        )
+
+        hidden_states = outputs[0]
+        # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
+        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
+        logits = self.lm_head(hidden_states[:, slice_indices, :]) * self.model.lm_head_multiplier
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def prepare_inputs_for_generation(
+        self,
+        input_ids,
+        past_key_values=None,
+        attention_mask=None,
+        inputs_embeds=None,
+        position_ids=None,
+        use_cache=True,
+        is_first_iteration=False,
+        **kwargs,
+    ):
+        kwargs["logits_to_keep"] = self.config.num_logits_to_keep
+        model_inputs = super().prepare_inputs_for_generation(
+            input_ids,
+            past_key_values=past_key_values,
+            attention_mask=attention_mask,
+            inputs_embeds=inputs_embeds,
+            position_ids=position_ids,
+            use_cache=use_cache,
+            is_first_iteration=is_first_iteration,
+            **kwargs,
+        )
+
+        return model_inputs
+
+
+__all__ = ["FalconH1Model", "FalconH1ForCausalLM", "FalconH1PreTrainedModel"]

LTA_openwebtext_dualt/mini_owt_logdirichlet/.venv_qwen35_uv/lib/python3.12/site-packages/transformers/models/falcon_h1/modular_falcon_h1.py

@@ -0,0 +1,1014 @@
+# Copyright 2025 Technology Innovation Institute and the HuggingFace Inc. team. All rights reserved.
+#
+# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
+# and OPT implementations in this library. It has been modified from its
+# original forms to accommodate minor architectural differences compared
+# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch FalconH1 model."""
+
+from collections.abc import Callable
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from ... import initialization as init
+from ...activations import ACT2FN
+from ...cache_utils import Cache, DynamicCache
+from ...integrations.hub_kernels import lazy_load_kernel
+from ...masking_utils import create_causal_mask
+from ...modeling_flash_attention_utils import FlashAttentionKwargs
+from ...modeling_layers import GradientCheckpointingLayer
+from ...modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
+from ...modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from ...processing_utils import Unpack
+from ...utils import TransformersKwargs, auto_docstring, can_return_tuple, is_torchdynamo_compiling, logging
+from ...utils.generic import merge_with_config_defaults
+from ...utils.import_utils import resolve_internal_import
+from ...utils.output_capturing import capture_outputs
+from ..llama.modeling_llama import (
+    LlamaAttention,
+    LlamaForCausalLM,
+    LlamaMLP,
+    LlamaRMSNorm,
+    LlamaRotaryEmbedding,
+    apply_rotary_pos_emb,
+    eager_attention_forward,
+)
+from ..mamba2.modeling_mamba2 import (
+    MambaRMSNormGated,
+    apply_mask_to_padding_states,
+    pad_tensor_by_size,
+    reshape_into_chunks,
+    segment_sum,
+)
+from .configuration_falcon_h1 import FalconH1Config
+
+
+logger = logging.get_logger(__name__)
+
+
+class FalconH1RotaryEmbedding(LlamaRotaryEmbedding):
+    pass
+
+
+class FalconH1Attention(LlamaAttention):
+    def __init__(self, config: FalconH1Config, layer_idx: int):
+        super().__init__(config, layer_idx)
+        self.key_multiplier = config.key_multiplier
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor],
+        attention_mask: torch.Tensor | None,
+        past_key_values: Cache | None = None,
+        **kwargs: Unpack[FlashAttentionKwargs],
+    ) -> tuple[torch.Tensor, torch.Tensor | None]:
+        input_shape = hidden_states.shape[:-1]
+        hidden_shape = (*input_shape, -1, self.head_dim)
+
+        query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
+        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2) * self.key_multiplier
+        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
+
+        cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_values is not None:
+            key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
+
+        attention_interface: Callable = ALL_ATTENTION_FUNCTIONS.get_interface(
+            self.config._attn_implementation, eager_attention_forward
+        )
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            dropout=0.0 if not self.training else self.attention_dropout,
+            scaling=self.scaling,
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
+        attn_output = self.o_proj(attn_output)
+        return attn_output, attn_weights
+
+
+class FalconH1RMSNormGated(MambaRMSNormGated):
+    def __init__(self, hidden_size, eps=1e-6, n_groups=1, norm_before_gate=True):
+        super().__init__(hidden_size=hidden_size, eps=eps)
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+        self.n_groups = n_groups
+        self.norm_before_gate = norm_before_gate
+
+    def forward(self, hidden_states, gate=None):
+        input_dtype = hidden_states.dtype
+
+        if not self.norm_before_gate and gate is not None:
+            hidden_states = hidden_states * F.silu(gate.to(torch.float32))
+
+        if len(hidden_states.shape) == 3:
+            batch_size, seq_len, dim = hidden_states.shape
+        else:
+            batch_size, dim = hidden_states.shape
+            seq_len = 1
+        hidden_states = hidden_states.to(torch.float32)
+
+        hidden_states = hidden_states.view(batch_size, seq_len, self.n_groups, int(dim // self.n_groups))
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+
+        hidden_states = self.weight.view(self.n_groups, int(dim // self.n_groups)) * hidden_states
+        hidden_states = hidden_states.view(batch_size, seq_len, dim)
+
+        if seq_len == 1:
+            hidden_states = hidden_states.squeeze(1)
+
+        if self.norm_before_gate and gate is not None:
+            hidden_states = hidden_states * F.silu(gate.to(torch.float32))
+        return hidden_states.to(input_dtype)
+
+
+# Adapted from transformers.models.mamba2.modeling_mamba2.Mamba2Mixer
+class FalconH1Mixer(nn.Module):
+    """
+    FalconH1Mixer is identical to classic Mamba2 mixer classes but differs on two different things
+    - Users can pass custom intermediate_size through `config.mamba_d_ssm`
+    - The use of gated RMS normalization layer is optional
+    """
+
+    def __init__(self, config: FalconH1Config, layer_idx: int):
+        super().__init__()
+        self.num_heads = config.mamba_n_heads
+        self.hidden_size = config.hidden_size
+        self.ssm_state_size = config.mamba_d_state
+        self.conv_kernel_size = config.mamba_d_conv
+        self.intermediate_size = (
+            int(config.mamba_expand * self.hidden_size) if config.mamba_d_ssm is None else config.mamba_d_ssm
+        )
+        self.layer_idx = layer_idx
+        self.use_conv_bias = config.mamba_conv_bias
+        self.activation = config.hidden_act
+        self.act = ACT2FN[config.hidden_act]
+        self.use_bias = config.mamba_proj_bias
+
+        self.layer_norm_epsilon = config.rms_norm_eps
+        self.groups_time_state_size = config.mamba_n_groups * self.ssm_state_size
+
+        self.n_groups = config.mamba_n_groups
+        self.head_dim = config.mamba_d_head
+        self.chunk_size = config.mamba_chunk_size
+
+        self.time_step_limit = config.time_step_limit
+        self.time_step_min = config.time_step_min
+        self.time_step_max = config.time_step_max
+
+        self.conv_dim = self.intermediate_size + 2 * self.n_groups * self.ssm_state_size
+        self.conv1d = nn.Conv1d(
+            in_channels=self.conv_dim,
+            out_channels=self.conv_dim,
+            bias=config.mamba_conv_bias,
+            kernel_size=self.conv_kernel_size,
+            groups=self.conv_dim,
+            padding=self.conv_kernel_size - 1,
+        )
+
+        # projection of the input hidden states
+        projection_size = self.intermediate_size + self.conv_dim + self.num_heads
+        self.in_proj = nn.Linear(
+            self.hidden_size,
+            projection_size,
+            bias=self.use_bias,
+        )
+        # selective projection used to make dt, B and C input dependant
+
+        # time step projection (discretization)
+        # instantiate once and copy inv_dt in init_weights of PretrainedModel
+        self.dt_bias = nn.Parameter(torch.ones(self.num_heads))
+
+        # S4D real initialization. These are not discretized!
+        # The core is to load them, compute the discrete states, then write the updated state. Keeps the memory bounded
+        A = torch.arange(1, self.num_heads + 1)
+        self.A_log = nn.Parameter(torch.log(A))
+        self.mamba_rms_norm = config.mamba_rms_norm
+
+        if self.mamba_rms_norm:
+            self.norm = FalconH1RMSNormGated(
+                self.intermediate_size,
+                eps=self.layer_norm_epsilon,
+                n_groups=self.n_groups,
+                norm_before_gate=config.mamba_norm_before_gate,
+            )
+        self.D = nn.Parameter(torch.ones(self.num_heads))
+
+        self.out_proj = nn.Linear(self.intermediate_size, config.hidden_size, bias=config.projectors_bias)
+
+        global causal_conv1d_update, causal_conv1d_fn
+        causal_conv1d = lazy_load_kernel("causal-conv1d")
+        causal_conv1d_update = getattr(causal_conv1d, "causal_conv1d_update", None)
+        causal_conv1d_fn = getattr(causal_conv1d, "causal_conv1d_fn", None)
+
+        global selective_state_update, mamba_chunk_scan_combined, mamba_split_conv1d_scan_combined
+        mamba_ssm = lazy_load_kernel("mamba-ssm")
+        selective_state_update = resolve_internal_import(
+            mamba_ssm, chained_path="ops.triton.selective_state_update.selective_state_update"
+        )
+        mamba_chunk_scan_combined = resolve_internal_import(
+            mamba_ssm, chained_path="ops.triton.ssd_combined.mamba_chunk_scan_combined"
+        )
+        mamba_split_conv1d_scan_combined = resolve_internal_import(
+            mamba_ssm, chained_path="ops.triton.ssd_combined.mamba_split_conv1d_scan_combined"
+        )
+
+        global is_fast_path_available
+        is_fast_path_available = all(
+            (
+                selective_state_update,
+                mamba_chunk_scan_combined,
+                mamba_split_conv1d_scan_combined,
+                causal_conv1d_fn,
+                causal_conv1d_update,
+            )
+        )
+
+        if not is_fast_path_available:
+            logger.warning_once(
+                "The fast path is not available because one of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)`"
+                " is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and"
+                " https://github.com/Dao-AILab/causal-conv1d"
+            )
+        else:
+            logger.warning_once("The fast path for FalconH1 will be used when running the model on a GPU")
+
+        self.zxbcdt_multipliers = config.ssm_multipliers
+        self.ssm_in_multiplier = config.ssm_in_multiplier
+
+    def cuda_kernels_forward(
+        self,
+        hidden_states: torch.Tensor,
+        cache_params: Cache | None = None,
+        attention_mask: torch.Tensor | None = None,
+    ):
+        # 1. Gated MLP's linear projection
+        hidden_states = apply_mask_to_padding_states(hidden_states, attention_mask)
+        # Add Multipliers
+        hidden_states = hidden_states * self.ssm_in_multiplier
+        projected_states = self.in_proj(hidden_states)
+        projected_states = projected_states * self.mup_vector  # ADD Mup Multipliers
+        d_to_remove = 2 * self.intermediate_size + 2 * self.n_groups * self.ssm_state_size + self.num_heads
+
+        # Set up dimensions for reshapes later
+        batch_size, seq_len, _ = hidden_states.shape
+        groups_time_state_size = self.n_groups * self.ssm_state_size
+
+        use_precomputed_states = (
+            cache_params is not None and cache_params.has_previous_state(self.layer_idx) and seq_len == 1
+        )
+
+        # getting projected states from cache if it exists
+        if use_precomputed_states:
+            d_mlp = (projected_states.squeeze(1).shape[-1] - d_to_remove) // 2
+
+            z0, x0, gate, hidden_states_B_C, dt = projected_states.squeeze(1).split(
+                [d_mlp, d_mlp, self.intermediate_size, self.conv_dim, self.num_heads], dim=-1
+            )
+
+            # 2. Convolution sequence transformation
+            hidden_states_B_C = causal_conv1d_update(
+                hidden_states_B_C,
+                cache_params.layers[self.layer_idx].conv_states,
+                self.conv1d.weight.squeeze(1),
+                self.conv1d.bias,
+                self.activation,
+            )
+
+            hidden_states, B, C = torch.split(
+                hidden_states_B_C,
+                [self.intermediate_size, groups_time_state_size, groups_time_state_size],
+                dim=-1,
+            )
+
+            # 3. SSM transformation
+            A = -torch.exp(self.A_log.float())  # (nheads,)
+            A = A[:, None, ...][:, :, None].expand(-1, self.head_dim, self.ssm_state_size).to(dtype=torch.float32)
+            dt = dt[:, :, None].expand(-1, -1, self.head_dim)
+            dt_bias = self.dt_bias[:, None, ...].expand(-1, self.head_dim)
+            D = self.D[:, None, ...].expand(-1, self.head_dim)
+            B = B.view(batch_size, self.n_groups, B.shape[1] // self.n_groups)
+            C = C.view(batch_size, self.n_groups, C.shape[1] // self.n_groups)
+            hidden_states_reshaped = hidden_states.view(batch_size, self.num_heads, self.head_dim)
+            hidden_states = selective_state_update(
+                cache_params.layers[self.layer_idx].recurrent_states,
+                hidden_states_reshaped,
+                dt,
+                A,
+                B,
+                C,
+                D,
+                z=gate.view(batch_size, self.num_heads, self.head_dim) if not self.mamba_rms_norm else None,
+                dt_bias=dt_bias,
+                dt_softplus=True,
+            )
+            hidden_states = hidden_states.view(batch_size, self.num_heads * self.head_dim)
+
+            if self.mamba_rms_norm:
+                hidden_states = self.norm(hidden_states, gate)
+
+            if d_mlp > 0:
+                hidden_states = torch.cat([F.silu(z0) * x0, hidden_states], dim=-1)
+
+            # 4. Final linear projection
+            out = self.out_proj(hidden_states[:, None, ...])
+        # Fused calculations or step by step if no initialized cache is found
+        else:
+            A = -torch.exp(self.A_log.float())  # (num_heads) or (intermediate_size, state_size)
+            dt_limit_kwargs = {} if self.time_step_limit == (0.0, float("inf")) else {"dt_limit": self.time_step_limit}
+
+            # 2-4. Fused kernel for conv1d, SSM, and the final projection
+            if self.training and cache_params is None:
+                out = mamba_split_conv1d_scan_combined(
+                    projected_states,
+                    self.conv1d.weight.squeeze(1),
+                    self.conv1d.bias,
+                    self.dt_bias,
+                    A,
+                    D=self.D,
+                    chunk_size=self.chunk_size,
+                    seq_idx=None,  # was seq_idx
+                    activation=self.activation,
+                    rmsnorm_weight=self.norm.weight if self.mamba_rms_norm else None,
+                    rmsnorm_eps=self.norm.variance_epsilon if self.mamba_rms_norm else None,
+                    outproj_weight=self.out_proj.weight,
+                    outproj_bias=self.out_proj.bias,
+                    headdim=self.head_dim,
+                    ngroups=self.n_groups,
+                    norm_before_gate=False,
+                    return_final_states=False,
+                    **dt_limit_kwargs,
+                )
+
+            else:
+                d_mlp = (
+                    projected_states.shape[-1]
+                    - 2 * self.intermediate_size
+                    - 2 * self.n_groups * self.ssm_state_size
+                    - self.num_heads
+                ) // 2
+                if attention_mask is not None:
+                    projected_states = projected_states * attention_mask[..., None]
+                _, gate, hidden_states_B_C, dt = projected_states.split(
+                    [
+                        2 * d_mlp,
+                        self.intermediate_size,
+                        self.conv_dim,
+                        self.num_heads,
+                    ],
+                    dim=-1,
+                )
+
+                if cache_params is not None:
+                    conv_states = F.pad(
+                        hidden_states_B_C.permute(0, 2, 1),
+                        (self.conv_kernel_size - hidden_states_B_C.shape[-2], 0),
+                    )
+                    conv_states = cache_params.update_conv_state(conv_states, self.layer_idx)
+
+                time_step = nn.functional.softplus(dt + self.dt_bias)
+                # 1D Convolution
+                if causal_conv1d_fn is None or self.activation not in ["silu", "swish"]:
+                    hidden_states_B_C = self.act(
+                        self.conv1d(hidden_states_B_C.transpose(1, 2)).transpose(1, 2)[:, :seq_len]
+                    )  # (B, L, self.d_inner + 2 * ngroups * d_state)
+                else:
+                    hidden_states_B_C = causal_conv1d_fn(
+                        x=hidden_states_B_C.transpose(1, 2),
+                        weight=self.conv1d.weight.squeeze(1),
+                        bias=self.conv1d.bias,
+                        activation=self.activation,
+                    ).transpose(1, 2)[:, :seq_len]
+
+                hidden_states, B, C = torch.split(
+                    hidden_states_B_C,
+                    [
+                        self.intermediate_size,
+                        groups_time_state_size,
+                        groups_time_state_size,
+                    ],
+                    dim=-1,
+                )
+
+                if attention_mask is not None and attention_mask.shape[1] > 1 and attention_mask.shape[0] > 1:
+                    # tune out hidden states for pad tokens, see https://github.com/state-spaces/mamba/issues/66
+                    dtype = hidden_states.dtype
+                    hidden_states = (hidden_states * attention_mask[:, :, None]).to(dtype)
+                # This is a hack to make sure multi-GPU inference works with HF accelerate
+                # see: https://github.com/Dao-AILab/flash-attention/issues/523 for more details
+                with torch.cuda.device(hidden_states.device):
+                    scan_output, ssm_state = mamba_chunk_scan_combined(
+                        hidden_states.view(batch_size, seq_len, -1, self.head_dim),
+                        time_step,
+                        A,
+                        B.view(batch_size, seq_len, self.n_groups, -1),
+                        C.view(batch_size, seq_len, self.n_groups, -1),
+                        chunk_size=self.chunk_size,
+                        D=self.D,
+                        z=None,
+                        seq_idx=None,
+                        return_final_states=True,
+                        **dt_limit_kwargs,
+                    )
+                if ssm_state is not None and cache_params is not None:
+                    ssm_state = cache_params.update_recurrent_state(ssm_state, self.layer_idx)
+                scan_output = scan_output.view(batch_size, seq_len, -1)
+                # Multiply "gate" branch and apply extra normalization layer
+                if self.mamba_rms_norm:
+                    out = self.norm(scan_output, gate)
+                else:
+                    out = scan_output * torch.nn.functional.silu(gate)
+                out = self.out_proj(out)
+        return out
+
+    # fmt: off
+    def torch_forward(
+        self,
+        input_states,
+        cache_params: Cache | None = None,
+        attention_mask: torch.Tensor | None = None,
+    ):
+        batch_size, seq_len, _ = input_states.shape
+        dtype = input_states.dtype
+
+        # 1. Gated MLP's linear projection
+        input_states = apply_mask_to_padding_states(input_states, attention_mask)
+        # Add Multipliers
+        input_states = input_states * self.ssm_in_multiplier
+        projected_states = self.in_proj(input_states)
+        projected_states = projected_states * self.mup_vector  # ADD Mup Multipliers
+        gate, hidden_states_B_C, dt = projected_states.split([
+                self.intermediate_size, self.conv_dim, self.num_heads
+            ], dim=-1)
+        hidden_states_B_C = hidden_states_B_C.transpose(1,2)
+
+        use_precomputed_states = cache_params is not None and cache_params.has_previous_state(self.layer_idx) and seq_len == 1
+
+        # 2. Convolution sequence transformation
+        if use_precomputed_states:
+            conv_states = cache_params.update_conv_state(hidden_states_B_C, self.layer_idx)
+            # We need to guarantee that anything regarding the cache is on the same device
+            conv_states = conv_states.to(device=self.conv1d.weight.device)
+
+            hidden_states_B_C = torch.sum(
+                conv_states * self.conv1d.weight.squeeze(1), dim=-1
+            )
+            if self.use_conv_bias:
+                hidden_states_B_C = hidden_states_B_C + self.conv1d.bias
+            hidden_states_B_C = self.act(hidden_states_B_C)
+        else:
+            # Init cache
+            if cache_params is not None:
+                conv_states = nn.functional.pad(
+                    hidden_states_B_C, (self.conv_kernel_size - hidden_states_B_C.shape[-1], 0)
+                )
+                conv_states = cache_params.update_conv_state(conv_states, self.layer_idx)
+
+            hidden_states_B_C = self.act(self.conv1d(hidden_states_B_C)[..., :seq_len].transpose(1, 2))
+
+        hidden_states_B_C = apply_mask_to_padding_states(hidden_states_B_C, attention_mask)
+        hidden_states, B, C = torch.split(
+            hidden_states_B_C,
+            [self.intermediate_size, self.n_groups * self.ssm_state_size, self.n_groups * self.ssm_state_size],
+            dim=-1
+        )
+
+        # 3. SSM transformation
+        A = -torch.exp(self.A_log.float())                            # [num_heads]
+        if use_precomputed_states:
+            # We need to guarantee that anything regarding the cache is on the same device
+            cache_device = cache_params.layers[self.layer_idx].recurrent_states.device
+
+            # Note: there is no need to pad parameter matrices here, as there is just one new token
+            # for batched generation
+            dt = dt[:, 0, :][:, None, ...]
+            dt = dt.transpose(1, 2).expand(batch_size, dt.shape[-1], self.head_dim)
+            # [num_heads] -> [num_heads, head_dim]
+            dt_bias = self.dt_bias[..., None].expand(self.dt_bias.shape[0], self.head_dim)
+
+            dt = torch.nn.functional.softplus(dt + dt_bias.to(dt.dtype))
+            dt = torch.clamp(dt, self.time_step_limit[0], self.time_step_limit[1])
+            A = A[..., None, None].expand(self.num_heads, self.head_dim, self.ssm_state_size).to(dtype=torch.float32)
+            # [bsz, num_heads, head_dim, state_size]
+            dA = (torch.exp(dt[..., None] * A)).to(device=cache_device)
+
+            # Discretize B
+            # [bsz, n_groups * state_size] -> [bsz, n_groups, 1, state_size] ->
+            # -> [bsz, n_groups, group to head repetition factor, state_size] -> [bsz, num_heads, state_size]
+            B = B.reshape(batch_size, self.n_groups, -1)[..., None, :]
+            B = B.expand(batch_size, self.n_groups, self.num_heads // self.n_groups, B.shape[-1]).contiguous()
+            B = B.reshape(batch_size, -1, B.shape[-1])
+            # [bsz, num_heads, head_dim, state_size]
+            dB = dt[..., None] * B[..., None, :]
+
+            # Discretize x into dB
+            # [bsz, intermediate_size] -> [bsz, num_heads, head_dim]
+            hidden_states = hidden_states.reshape(batch_size, -1, self.head_dim)
+            dBx = (dB * hidden_states[..., None]).to(device=cache_device)
+
+            # State calculation
+            ssm_states = cache_params.layers[self.layer_idx].recurrent_states * dA + dBx
+            ssm_states = cache_params.update_recurrent_state(ssm_states, self.layer_idx)
+
+            # Subsequent output
+            # [bsz, n_groups * state_size] -> [bsz, num_heads, state_size]
+            C = C.reshape(batch_size, self.n_groups, -1)[..., None, :]
+            C = C.expand(batch_size, self.n_groups, self.num_heads // self.n_groups, C.shape[-1]).contiguous()
+            C = C.reshape(batch_size, -1, C.shape[-1])
+            # [bsz, num_heads, head_dim]
+
+            ssm_states = ssm_states.to(device=C.device, dtype=C.dtype)  # Shape: [b, h, d, n]
+            # Reshape ssm_states to merge the first two dimensions
+            ssm_states_reshaped = ssm_states.view(batch_size * self.num_heads, self.head_dim, self.ssm_state_size)  # Shape: [b*h, d, n]
+            C_reshaped = C.view(batch_size * self.num_heads, self.ssm_state_size, 1)  # Shape: [b*h, n, 1]
+            y = torch.bmm(ssm_states_reshaped, C_reshaped)
+            y = y.view(batch_size, self.num_heads, self.head_dim)
+
+            # D skip connection
+            # [num_heads] -> [num_heads, head_dim]
+            D = self.D[..., None].expand(self.D.shape[0], self.head_dim)
+            y = (y + hidden_states * D).to(y.dtype)
+
+            # [bsz, num_heads, head_dim] -> [bsz, 1, intermediate_size]
+            y = y.reshape(batch_size, -1)[:, None, ...]
+        else:
+            # begin ssd naive implementation without einsums
+            dt = nn.functional.softplus(dt + self.dt_bias)
+            dt = torch.clamp(dt, self.time_step_limit[0], self.time_step_limit[1])
+            hidden_states = hidden_states.reshape(batch_size, seq_len, -1, self.head_dim).float()
+            B = B.reshape(batch_size, seq_len, -1, self.ssm_state_size).float()
+            C = C.reshape(batch_size, seq_len, -1, self.ssm_state_size).float()
+            B = B.repeat_interleave(self.num_heads // self.n_groups, dim=2, output_size=self.num_heads)
+            C = C.repeat_interleave(self.num_heads // self.n_groups, dim=2, output_size=self.num_heads)
+            pad_size = (self.chunk_size - seq_len % self.chunk_size) % self.chunk_size
+
+            D_residual = self.D[..., None] * pad_tensor_by_size(hidden_states, pad_size)
+
+            # Discretize x and A
+            hidden_states = hidden_states * dt[..., None]
+            A = A.to(hidden_states.dtype) * dt
+
+            # Rearrange into blocks/chunks
+            hidden_states, A, B, C = [reshape_into_chunks(t, pad_size, self.chunk_size) for t in (hidden_states, A, B, C)]
+
+            # [bsz, -1, chunk_size, num_heads] -> [bsz, num_heads, -1, chunk_size]
+            A = A.permute(0, 3, 1, 2)
+            A_cumsum = torch.cumsum(A, dim=-1)
+
+            # 1. Compute the output for each intra-chunk (diagonal blocks)
+            # This is the analog of a causal mask
+            L = torch.exp(segment_sum(A))
+
+            # Contraction of C and B to get G (attention-weights like)
+            G_intermediate = C[:, :, :, None, :, :] * B[:, :, None, :, :, :]  # shape: (b, c, l, s, h, n)
+            G = G_intermediate.sum(dim=-1)  # shape: (b, c, l, s, h)
+
+            # Compute M, equivalent to applying attention mask to weights
+            M_intermediate = G[..., None] * L.permute(0, 2, 3, 4, 1)[..., None]
+            M = M_intermediate.sum(dim=-1)
+
+            # Compute Y_diag (apply to values)
+            Y_diag = (M[..., None] * hidden_states[:, :, None]).sum(dim=3)
+
+            # 2. Compute the state for each intra-chunk
+            # (right term of low-rank factorization of off-diagonal blocks; B terms)
+            decay_states = torch.exp(A_cumsum[:, :, :, -1:] - A_cumsum)
+            B_decay = B * decay_states.permute(0, -2, -1, 1)[..., None]
+            states = (B_decay[..., None, :] * hidden_states[..., None]).sum(dim=2)
+
+            # 3. Compute the inter-chunk SSM recurrence; produces correct SSM states at chunk boundaries
+            # (middle term of factorization of off-diag blocks; A terms)
+            previous_states = torch.zeros_like(states[:, :1])
+            states = torch.cat([previous_states, states], dim=1)
+            decay_chunk = torch.exp(segment_sum(nn.functional.pad(A_cumsum[:, :, :, -1], (1, 0))))
+            decay_chunk = decay_chunk.transpose(1, 3)
+            new_states = (decay_chunk[..., None, None] * states[:, :, None, ...]).sum(dim=1)
+            states, ssm_state = new_states[:, :-1], new_states[:, -1]
+
+            # 4. Compute state -> output conversion per chunk
+            # (left term of low-rank factorization of off-diagonal blocks; C terms)
+            state_decay_out = torch.exp(A_cumsum)
+            C_times_states = (C[..., None, :] * states[:, :, None, ...])
+            state_decay_out_permuted = state_decay_out.permute(0, 2, 3, 1)
+            Y_off = (C_times_states.sum(-1) * state_decay_out_permuted[..., None])
+
+            # Add output of intra-chunk and inter-chunk terms (diagonal and off-diagonal blocks)
+            y = Y_diag + Y_off
+            # [bsz, -1, self.chunk_size, num_heads, head_dim] -> [bsz, (padded) seq_len, num_heads, head_dim]
+            y = y.reshape(batch_size, -1, self.num_heads, self.head_dim)
+
+            y = y + D_residual
+            # Cutting off padded chunks
+            if pad_size > 0:
+                y = y[:, :seq_len, :, :]
+            y = y.reshape(batch_size, seq_len, -1)
+
+            # Init cache
+            if ssm_state is not None and cache_params is not None:
+                ssm_state = cache_params.update_recurrent_state(ssm_state, self.layer_idx)
+
+        if self.mamba_rms_norm:
+            scan_output = self.norm(y, gate)
+        else:
+            scan_output = y * torch.nn.functional.silu(gate)
+
+        # end ssd naive
+
+        # 4. Final linear projection
+        contextualized_states = self.out_proj(scan_output.to(dtype))  # [batch, seq_len, hidden_size]
+        return contextualized_states
+    # fmt: on
+
+    def forward(
+        self,
+        hidden_states,
+        cache_params: Cache | None = None,
+        attention_mask: torch.Tensor | None = None,
+        **kwargs,
+    ):
+        if is_fast_path_available and "cuda" in self.in_proj.weight.device.type and not is_torchdynamo_compiling():
+            return self.cuda_kernels_forward(hidden_states, cache_params, attention_mask)
+        dtype = hidden_states.dtype
+        if attention_mask is not None and attention_mask.shape[1] > 1 and attention_mask.shape[0] > 1:
+            # tune out hidden states for pad tokens, see https://github.com/state-spaces/mamba/issues/66
+            hidden_states = (hidden_states * attention_mask[:, :, None]).to(dtype)
+
+        return self.torch_forward(hidden_states, cache_params, attention_mask)
+
+
+class FalconH1MLP(LlamaMLP):
+    def __init__(self, config: FalconH1Config):
+        super().__init__(config)
+        self.gate_multiplier, self.down_multiplier = config.mlp_multipliers
+
+    def forward(self, x):
+        y = self.up_proj(x) * self.act_fn(self.gate_proj(x) * self.gate_multiplier)
+        y = self.down_proj(y) * self.down_multiplier
+        return y
+
+
+class FalconH1RMSNorm(LlamaRMSNorm):
+    pass
+
+
+class FalconH1DecoderLayer(GradientCheckpointingLayer):
+    def __init__(self, config: FalconH1Config, layer_idx: int):
+        super().__init__()
+        self.feed_forward = FalconH1MLP(config)
+
+        head_dim = config.hidden_size // config.num_attention_heads
+        self.channels_attn = config.num_attention_heads * head_dim + 2 * config.num_key_value_heads * head_dim
+
+        self.mamba = FalconH1Mixer(config=config, layer_idx=layer_idx)
+
+        self.self_attn = FalconH1Attention(config, layer_idx)
+
+        self.attention_in_multiplier = config.attention_in_multiplier
+        self.ssm_out_multiplier = config.ssm_out_multiplier
+        self.attn_out_multiplier = config.attention_out_multiplier
+
+        self.input_layernorm = FalconH1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.pre_ff_layernorm = FalconH1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: torch.Tensor | None = None,
+        mamba_attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        use_cache: bool | None = False,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
+        **kwargs,
+    ) -> tuple[torch.FloatTensor]:
+        """
+        Args:
+            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
+            attention_mask (`torch.FloatTensor`, *optional*): attention mask of size
+                `(batch, sequence_length)` where padding elements are indicated by 0.
+            past_key_values (`Cache`, *optional*): cached past key and value projection states
+            use_cache (`bool`, *optional*):
+                If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
+                (see `past_key_values`).
+            position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*):
+                Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`,
+                with `head_dim` being the embedding dimension of each attention head.
+            kwargs (`dict`, *optional*):
+                Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code
+                into the model
+        """
+
+        residual = hidden_states
+        hidden_states = self.input_layernorm(hidden_states)
+
+        mamba_hidden_states = self.mamba(
+            hidden_states=hidden_states,
+            cache_params=past_key_values,
+            attention_mask=mamba_attention_mask,
+        )
+        mamba_hidden_states = mamba_hidden_states * self.ssm_out_multiplier
+
+        attention_hidden_states, _ = self.self_attn(
+            hidden_states=hidden_states * self.attention_in_multiplier,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            position_embeddings=position_embeddings,
+            **kwargs,
+        )
+        attention_hidden_states = attention_hidden_states * self.attn_out_multiplier
+
+        hidden_states = mamba_hidden_states + attention_hidden_states
+
+        # residual connection after attention
+        hidden_states = residual + hidden_states
+
+        # feed-forward
+        residual = hidden_states
+        hidden_states = self.pre_ff_layernorm(hidden_states)
+        hidden_states = self.feed_forward(hidden_states)
+        hidden_states = residual + hidden_states
+
+        return (hidden_states,)
+
+
+@auto_docstring
+class FalconH1PreTrainedModel(PreTrainedModel):
+    config: FalconH1Config
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["FalconH1DecoderLayer"]
+    _skip_keys_device_placement = ["past_key_values"]
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    _is_stateful = True
+
+    _can_record_outputs = {
+        "hidden_states": FalconH1DecoderLayer,
+        "attentions": FalconH1Attention,
+    }
+
+    @torch.no_grad()
+    def _init_weights(self, module):
+        super()._init_weights(module)
+        if isinstance(module, FalconH1Mixer):
+            init.ones_(module.dt_bias)
+            init.copy_(module.A_log, torch.log(torch.arange(1, module.num_heads + 1)))
+            init.ones_(module.D)
+        elif isinstance(module, FalconH1Model):
+            mup_vector = compute_mup_vector(module.config)
+            for layer in module.layers:
+                init.copy_(layer.mamba.mup_vector, mup_vector)
+
+
+def compute_mup_vector(config):
+    """
+    Computes the MuP vector based on model configuration.
+
+    FalconH1 applies different MuP multiplier for each dimension of the hidden states.
+    The MuP vector is partitioned into chunks, and each chunk is multiplied with its
+    corresponding projected dimension.
+
+    Args:
+        config: FalconH1Config object
+
+    Returns:
+        torch.Tensor: The computed MuP vector
+    """
+    # We'll need some values from the config to compute the vector dimensions
+    intermediate_size = (
+        config.mamba_d_ssm if config.mamba_d_ssm is not None else int(config.mamba_expand * config.hidden_size)
+    )
+    groups_time_state_size = config.mamba_n_groups * config.mamba_d_state
+    num_heads = config.mamba_n_heads
+    zxbcdt_multipliers = config.ssm_multipliers
+
+    vector_shape = 2 * intermediate_size + 2 * groups_time_state_size + num_heads
+    mup_vector = torch.ones(1, 1, vector_shape)
+
+    # Apply multipliers to different sections of the vector
+    mup_vector[:, :, :intermediate_size] *= zxbcdt_multipliers[0]
+    mup_vector[:, :, intermediate_size : 2 * intermediate_size] *= zxbcdt_multipliers[1]
+    mup_vector[:, :, 2 * intermediate_size : 2 * intermediate_size + groups_time_state_size] *= zxbcdt_multipliers[2]
+    mup_vector[
+        :, :, 2 * intermediate_size + groups_time_state_size : 2 * intermediate_size + 2 * groups_time_state_size
+    ] *= zxbcdt_multipliers[3]
+    mup_vector[:, :, 2 * intermediate_size + 2 * groups_time_state_size :] *= zxbcdt_multipliers[4]
+
+    return mup_vector
+
+
+@auto_docstring
+# Adapted from transformers.models.jamba.modeling_jamba.JambaModel
+class FalconH1Model(FalconH1PreTrainedModel):
+    def __init__(self, config: FalconH1Config):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        decoder_layers = []
+        for i in range(config.num_hidden_layers):
+            decoder_layers.append(FalconH1DecoderLayer(config, layer_idx=i))
+        self.layers = nn.ModuleList(decoder_layers)
+
+        self._attn_implementation = config._attn_implementation
+        self.final_layernorm = FalconH1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.rotary_emb = FalconH1RotaryEmbedding(config=config)
+
+        self.embedding_multiplier = config.embedding_multiplier
+        self.lm_head_multiplier = config.lm_head_multiplier
+
+        self.gradient_checkpointing = False
+        # Compute the MuP vector once and register it for all layers
+        mup_vector = compute_mup_vector(config)
+        for layer in self.layers:
+            layer.mamba.register_buffer("mup_vector", mup_vector.clone(), persistent=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @merge_with_config_defaults
+    @capture_outputs
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        use_cache: bool | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutputWithPast:
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embed_tokens(input_ids) * self.embedding_multiplier
+        hidden_states = inputs_embeds
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache(config=self.config)
+
+        if position_ids is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            position_ids = torch.arange(hidden_states.shape[1], device=hidden_states.device) + past_seen_tokens
+            position_ids = position_ids.unsqueeze(0)
+
+        causal_mask = create_causal_mask(
+            config=self.config,
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            position_ids=position_ids,
+        )
+        mamba_mask = self._update_mamba_mask(attention_mask, past_key_values)
+        position_embeddings = self.rotary_emb(hidden_states, position_ids=position_ids)
+
+        for decoder_layer in self.layers:
+            layer_outputs = decoder_layer(
+                hidden_states,
+                attention_mask=causal_mask,
+                mamba_attention_mask=mamba_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                use_cache=use_cache,
+                position_embeddings=position_embeddings,
+            )
+
+            hidden_states = layer_outputs[0]
+
+        hidden_states = self.final_layernorm(hidden_states)
+
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+        )
+
+    def _update_mamba_mask(self, attention_mask, past_key_values):
+        """
+        No need for zeroing states when
+            1. Cached forward
+            2. Attending to all inputs
+        """
+        mamba_mask = attention_mask
+        if (past_key_values is not None and past_key_values.has_previous_state()) or (
+            attention_mask is not None and torch.all(attention_mask == 1)
+        ):
+            mamba_mask = None
+        return mamba_mask
+
+
+class FalconH1ForCausalLM(LlamaForCausalLM):
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        labels: torch.LongTensor | None = None,
+        use_cache: bool | None = None,
+        logits_to_keep: int | torch.Tensor = 0,
+        **kwargs,
+    ) -> tuple | CausalLMOutputWithPast:
+        r"""
+        Example:
+
+        ```python
+        >>> from transformers import AutoTokenizer, FalconH1ForCausalLM
+
+        >>> model = FalconH1ForCausalLM.from_pretrained("...")
+        >>> tokenizer = AutoTokenizer.from_pretrained("...")
+
+        >>> prompt = "Hey, are you conscious? Can you talk to me?"
+        >>> inputs = tokenizer(prompt, return_tensors="pt")
+
+        >>> # Generate
+        >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
+        >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
+        "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
+        ```"""
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            **kwargs,
+        )
+
+        hidden_states = outputs[0]
+        # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
+        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
+        logits = self.lm_head(hidden_states[:, slice_indices, :]) * self.model.lm_head_multiplier
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def prepare_inputs_for_generation(
+        self,
+        input_ids,
+        past_key_values=None,
+        attention_mask=None,
+        inputs_embeds=None,
+        position_ids=None,
+        use_cache=True,
+        is_first_iteration=False,
+        **kwargs,
+    ):
+        kwargs["logits_to_keep"] = self.config.num_logits_to_keep
+        model_inputs = super().prepare_inputs_for_generation(
+            input_ids,
+            past_key_values=past_key_values,
+            attention_mask=attention_mask,
+            inputs_embeds=inputs_embeds,
+            position_ids=position_ids,
+            use_cache=use_cache,
+            is_first_iteration=is_first_iteration,
+            **kwargs,
+        )
+
+        return model_inputs
+
+
+__all__ = ["FalconH1Model", "FalconH1ForCausalLM", "FalconH1PreTrainedModel"]

LTA_openwebtext_dualt/mini_owt_logdirichlet/runs/owt_t5_elftokenized_full_len1024_C1_to_1024_pow1_d768_l12_h12_gbs512_2x8gpu_50ep_lr3e3_ema0p9999_elfopt_not5_bottleneck128_unfixed_norm_stateprobadd_selfcond_ce_fast_20260609_230847/step_114000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55dbd66d275adb0aa5d4d1f0d2473f926e08c715c00d20992197d46d60232509
+size 927700322

LTA_openwebtext_dualt/mini_owt_logdirichlet/runs/owt_t5_elftokenized_full_len1024_C1_to_1024_pow1_d768_l12_h12_gbs512_2x8gpu_50ep_lr3e3_ema0p9999_elfopt_not5_bottleneck128_unfixed_norm_stateprobadd_selfcond_ce_fast_20260609_230847/step_266000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:705084e921433bd981a7322943deef3c892aa818f1f9182fdf67d23af3d0c259
+size 927700322

LTA_openwebtext_dualt/mini_owt_logdirichlet/samples/tinystories_t5_len1024_d768_8gpu_step1000_decode128_quick_n8/first8.txt

@@ -0,0 +1,38 @@
+checkpoint=runs/tinystories_t5_logdirichlet_len1024_C1_to_64_d768_l12_h12_gbs512_8gpu_40k_20260527_121803/step_001000.pt
+step=1000
+decode=dualline_time_aligned_dirichlet_final_state
+c_min=1.0 c_max=64.0
+steps=128 temp=1.45 bridge_power=1.0 temp0=0.0
+bos=1:</s> eos=1:</s>
+===== sample 0 =====
+head_tokens: ['</s>', '▁no', 'dded', '.', '▁She', '▁said', ',', '▁"', 'I', 't', "'", 's', '▁noble', '▁mine', '!"', '▁Li']
+tail_tokens: ['▁hug', '.', '▁He', '▁', 's', 'cream', 'ink', '▁', 'a', '▁', 'pond', '▁and', '▁saw', '▁', 'a', '</s>']
+</s> nodded. She said, "It's noble mine!" Lily and said, sing. She ran to the dirt and waited. She was surprised and ignored him. She smiled and said she was sorry. She looked at her mom and hugged her kindly. She had to be harbor. She gave him a punishment and put the Zeitraum on the table. They felt sad and said sorry. They had aips. They loved their names. They looked east and Car and laughed. They were happy and aminte the prince. They ran to the valley. They tested the sap and tapped and their mad. They promised to play with the insects and the explored.</s> Lily was a famous lamp. They liked to explore the crowd. They liked to create stuff. They used their fans and dive to the airport. They made a big boom and the crowd. They had many beds and headss and potatoes and smell. They would to the pages. They had asti a cliff. Then they saw a big boom and saw a web. She did not know thevoller to bother it. She sounded a zigQupop, but Lily said. She had a big backpack and planted him. She was a escape. The wizard was small. She jumped and searched and spoke. They rescue the sketch. They had a bad cliff. They screamed and revealed. They said they had made a boom and a valuable EP. They were bored and happy. But they had sixweighed. They stayed home and simpleted. They wanted to escape. They had a taxi. Lily felt a very lizici squirrel. They was happy that that they wanted to create aress. They wished theing was not ruined.</s> Lily wanted to build a big pond. She had had many incredibles and videos. She liked the crosters and buses. She had notresident on the swing. She had a big rad pond with the straw. She swung a big radze. It was an otte and a lizard. "It's, Lily. It's so modern shade!" Lily. They searched and liked to destroy themerkt. They lizlinked and vanilla and had a feast. They had a explorer crober of helicopter. They opened the tracks and roar and saw a pond. They were castle. They walked to the university and the lionolli and a big lion. "Let'sKI, spannend gloves!" Lily and said. "That was a modern oasis. They d and a lion. They was surprised. They ate laughed. They smiled and smiled. They said, "Look, it's a Finance. We can protect liking it." They clapped. They were happy and smart and become ending Grecia.</s> Lily and Tom were friends. They liked to play outside and solve mattresss. They liked the lizgars and bluetang. Lily and Tom were very happy. They had a lot and thepig and started to gesture. They squaung the oasis and lizncy. They disturbed their runs. They did not know the king. They caught the melon and breathed. They liked to each other and drinks. They crossed the bucket and Fence and shorter. They aimed away and the coins. They had a blast. They was frog lion and laughed. They sounded lizmer and wanted to play together. They had a shock. They qud and happy. The end.</s> Tom was a big lizaccoon in the island. It was a big pond and a rat. They likedassembling to the cabin and a pond. It was a lizber. Tom and Tom went to the cros. They had aaxe. The lizco and shouted. They did not like the monkey. They waited and came to the valley. They lizaked and a lion. They loved to Bis and croleton, and crashed into the fox. They ran to the pond and the pond and the lizcoon. They revealed their den and wing and escaped. They shouted and the pig and sang. They made the lizstrich and hug incalzireers and  Binffes. They was happy and enjoyed their banque home.</s> lion was a lion and wreck to Reduce him. He wanted to go and make pond. His he walked to the pond with him. He was a cough and a scarf and made a hug. He screamink a pond and saw a</s>
+===== sample 1 =====
+head_tokens: ['</s>', '▁when', '▁', 'he', '▁saw', '▁', 'a', '▁silver', '▁lawn', '.', '▁The', '▁dinosaur', '▁was', '▁so', '▁frustrated', '▁and']
+tail_tokens: ['f', 'rog', '▁mondiale', '▁inside', '.', '▁He', '▁was', '▁', 'a', '▁shock', '▁and', '▁', 'roar', '▁and', '▁', '</s>']
+</s> when he saw a silver lawn. The dinosaur was so frustrated and Jerry to thegust, but when he didn't realize his property. He said that he wanted to discuss what to do. He was embarrassed because he didn't want to do anything. He had a challenge where he was too late. He dl onto his wrist, that he was a peculiar, a valuable fruit echo as he ate thepainters. The sponge was thrilled and he wanted to dive again. He was so everyone child, but he felt embarrassed that he couldn't wait.</s> Once upon a time, there was a powerful lizco. The motion had a big lizard. One day, a fox went to a pond with a Tun. He wanted to decide himself, he wanted to make a sensation. Suddenly he heard a lizco. Thecorn Israeli a Fear and a crocohut off prove a chat. The  ratco repaycan of himself. The lizhave was a little lizco. The lizgarleton  reliability a fox. He asked the Mother s3-2 it crocooence. The crocod and sent it left from the harbor. The lizuddle a magical lizmaion. The little mouse and affe cro lizgarbling was a lion. The rat550 the lizcoity in the forest. It was very quick and he  Body mean to the experience.</s> Once upon a time there was a poor mighty lizard. He had an radrog and loved everything. One day he was aright flower that explorer on the villages. He felt so polite. He ran to the pond and said, "That's too family for a croleton helping when when he was a simpleer and grabbed a rhythm. He had a short tools, José thecorns and smiled. He hopped down and how he was a valuable Anwendung. He ran to the tool and he would be to receive the mouse. Theaded was transactions and passing investments, but it was impossible. He put therichten in the attacking and he was bouSeveral. He wished it was impossible. He felt so sad he liked An. He didn't want to introduce anything, when he made sure. Then, Charlie had a valuable tool long movements. He was so happy to him. He couldn't resist his shoulder.</s> Once upon a time there was a boy named Riley. He had a reallycontinent and he had completed tools. One day, a parade had had a invitation in thenom Michael. He wanted to ignore it, but he managed to the beat, and beautifully cousin. Suddenly, the little boy came to the hotel and found lots of cute paintings. He felt very happy when he saw a dead clearing. He had grow shaking and asked it. He stayed towards the little boy. He tried to ignore the comment, but Name that he started to spread the anger and following the brake. It was clouds, the mouse was healed! He thanked the sur cadouri and theeast and the highest battery often succeeded returning forehead. He smiled back and said, "You are lessDB Bob!" His brother smiled and smiled as he said. He asked the recueil that he had received a magical involvedul of indeed.</s> Once upon a time, there was a ce. He loved to play with his friends. One day, unilateral saw a mechanic and he wanted to go to the market. He was very scared as a jutter loft as he had an idea. He knew that he was a nice quicklyened. Suddenly, a little bird fox was flutter complaining down on the ground. Then he realized, it was a smile. The lion smiled and smiled a lot of shelter. The fox bowed down and couldn't manage to warmth he felt miserable again and he liked to risk himself. He was what to belonged.</s> Once there was a lion. He wanted to protecteptic excited and rules, but he was so carefulter. One day, he saw a big chimney and had a roar. He was excited and walking, he spotted a tough walking in the park. As he was a, he felt a nibmeter on the ground. He shooktered grey, and he knew the random why he wasige andious a big chic frog mondiale inside. He was a shock and roar and </s>
+===== sample 2 =====
+head_tokens: ['</s>', '▁repair', '▁the', '4,', '.', '▁But', '▁the', 'n', ',', '▁it', '▁was', '▁gone', '▁', 'a', '▁', 'wolf']
+tail_tokens: ['▁stirring', '▁in', '▁the', '▁Buddha', '.', '▁They', '▁broke', '▁the', '▁taxi', '▁and', '▁', 'escaped', '.', '▁The', '▁tub', '</s>']
+</s> repair the4,. But then, it was gone a wolf. It was dull and mysterious. "It's go to the stream. It's Therapeutic the rat!" They said. They nodded and thanked the fish. They smiled and held down the pond and down. It was a bacco swimmer on the nostalgic. They warned and a tour. They liked cheese, he saw a big cro. He was a big landscape. He started to realize the seara was a zig. It was a wing. Suddenly, it was a big bcod it. They sounded a wing. It looked deliberately to behind. "No, I do you want to be afraid. It was too Shoes and turning Amy. It climatique the spreading and sparkly and caught a ha. "Shh, crown, Pipe. It was a VR. "It tehnici!" she shouted. "I'm here!" she said. They were happy and wanted to serve the House. They went to the grass and saw the arop. They was running and hopeful. They wanted to see the workers and a Gardenrland. They had Auf the departure. They lived in a big sign. She was neuen in a priber. She liked to strip the lizber and make them. It was snacks. She tfini and listened to him. She smiled and said, "yesams. I recommend you live the basement.”. So, she went to the lizster. She held the lime captive to the oasis. She had a lot ofplace. She gave was a basket of theSet. She went to the pig bank and photos. She had a lot of fun. She had the ornament in the Miami and loved the husband. They had a yummy suspension and Back the mitten. She was very happy.</s> Tom was a modest thoughtful girl. She liked to snugglops and a monkey in the forest. She would build a pig and the mud. One day, Tom was a big neck. They wanted to play with it. They had a fat mission. So he wanted to solve the pond. They spread some water. They made a click. It was red and juicy and had po velvet. They raised the pond and a justrich. They waited for him. They saw a oasis and a pond. They made a sides and the mouse and a boom. They liked to pass it. They ran to the pond and swept to the3.2. They laughed and started to collect more. They argued away. They was a nice urma. They had the art. They bought the palace and sent to the area. They were very sad. They had a simple barbecue. They wanted to bother it. They had a a bweb and acorn. It was not ugly. The kanostrich das Tom and decided to go home. They liked the sand. They had the lizber. They was Elle. "It's  degrees!" The team and shouted. They started to argue and argue. They had to the bride. They was a sword and escape to the holiday. They had coloan Apprentice. They was explorer and tiger. They had a slices, dessert and pinetended to the hotel. It was very sad and less more. They was a lizmer.</s> One day, she went to the park and saw the girl. She liked that it was ak. She liked to play with the strawberry. She knew that it was cold and a erreicht. She wanted to disturb the lizncy. She had  fiberglass a cro pond. She had a cliff and barre and cracked the rat. They wanted to marry the pond. They wanted the tent and answers. "Ow, mom! It's Camp!" she said. She ran to the line of the water. They tangled the anchor. Lily's mom was surprised and happy. She was sad. She rlinked and held the oasis and crash. She liked the rat and the grill. She gave them a hug.</s> Lily and her mom. They liked to play with their mom and dad. They were lol and stuck. They wanted to examine it inside. They saw a hurricane. It was a big cow. Lily and the rush. They wanted to serve the curtain. They had fourteen and amplasat. They ran to the third. The performers was faster and covered the lightning. But it was a big retire with a loudpaw and sent it burned to jeunes. The dictionary was mild and stirring in the Buddha. They broke the taxi and escaped. The tub</s>
+===== sample 3 =====
+head_tokens: ['</s>', 'Gra', '▁Administrator', '▁worry', '▁that', '▁', 'he', '▁had', '▁managed', '▁to', '▁be', '▁punished', '.', '▁He', '▁had', '▁fake']
+tail_tokens: ['▁there', '▁was', '▁', 'a', '▁little', '▁girl', '▁named', '▁Mi', 'a', '.', '▁Her', '▁extensions', '▁was', '▁3', '▁years', '</s>']
+</s>Gra Administrator worry that he had managed to be punished. He had fake discomfort that he didn't recognize the kenn steady. He knew he had planned he eat the lizber. It was a castle. The Ei was very embarrassed and protecting the rat. The fox had a great idea. He hopped over his describe, the rhythm he was glad that he had completed the  medicine.</s> Riley was a croco the pig. He never wanted a further, but he wanted to explore a wish. He was so delighted that he could travelling when he heard a roar. It was a thion. The fairy was curse because it was so slippery and *. They had found a big lizmeron too. It was there and modest distinct. With a pig, it was a rat. But it was a fox. The jcob bank and offered thelion to treated becoming in the garden. It was a lizumble. The crocoed the radco who wanted to dive thescreen. They wrapped the hippounk in the garden.</s> Ed had a stuffed ju. She liked to point on the journey, he remembered the frog provided Elephant. She had stiff the pond onto the crowd. It was a kindlyloom the pipe. She walked closer to the harbor, she saw a thief pond. It was a very frightening gift. She had a friendly cliff. It was so nice and d the villagers. She had stolenexploded and Georg. It was a whole fox to be it. She smiled and started to reverse as the pond acted as the pig, he felt di relaxed.</s> Once upon a time there was a little girl who loved to prevent therobo. One day, she went to a house. She wanted to encourage the servant to be gre her undergraduate and attention to the core. One Suddenly, the girl was determined to receive. Her elderlyttle's luggage was fragilemonetiz challenging and wanted it. When the vendor, she saw a little bit containers. She examined thelandais and made Rover traffic Zimmer. She felt ready to explore the toilet. She skipped down and fixed the sea and shape. She conquered and removed the temptation. She had a great time and flupolitik honour. Still, she started to checks the ight Birthday and it was organised. She said pray as the wholeexplorer went inside and enjoyed them again.</s> Once upon a time there was a little girl called Nora. She had a lot of companion and unusuald. It was a po ROedge and the elderly bell. One day,, a little girl named aaccelerated to played. It was a donors and a collection of sincerely. She was so happy and when she went to university and examined thelikewise. She ran to the pond. She looked at it and saw a hugeм. It was a mur changes that he couldn't believe what a  contained. When he was a bench, she looked right and harmless underneath them anyway. Once upon a time, there was a tricwane. One of the lizleton. It was a big thiel. He had changingrely invitation that he had never scent, a nearest and protected sensation. She wanted to cross the cliff, so she mixing the destination and she couldn't get down. The lion was so nice, but a little girl was worried. Suddenly, the girl was complete a lion. She smiled and said, "Hello, I'm so lucky to create a question." The girl was so sad and he asked the now to celebrate. Suddenly, he s Constantalessness towards the again. The frog was so sad. The pig smiled. When he went home. He smiled and followed theApotheke and found a massage. The girl was sad. The girl smiled and said he had made a valuable outdoor giant.</s> Once there was a musician. He was out to the Too hills. He had an idea that he was very opened! He was so excited to explore the world around it. He had warm before he had managed. He opened the key and walked down to the cash properly. He peeked and recorded it. He watched as he searched as he was examining, andvous that he had managed to cross the temperature. He smiled and he had a hug and gave him to the grandmother. His he was so proud of himself.</s> Once there was a little girl named Mia. Her extensions was 3 years</s>
+===== sample 4 =====
+head_tokens: ['</s>', '▁', 'hopped', '▁in', '▁the', '▁garden', '.', '▁She', '▁saw', '▁', 'a', '▁mysterious', '▁', 'lion', '▁', 'b']
+tail_tokens: ['▁with', '▁his', '▁', 'h', 'gator', '▁and', '▁his', '▁body', '.', '▁One', '▁day', ',', '▁', 'a', '▁messy', '</s>']
+</s> hopped in the garden. She saw a mysterious lion blutter dedicated as she could bend it, as she was practicing to move the lion. They knew that the unusual chase the truth. The attendant wanted to restore the lion's die. It was a valuable leap in the lizcoleton. The croaked down the island. Dr swaneptilor a thoughtful 2008.. The mul as he balanced in the pig.</s> Once upon a time there was a little lizc ile. He had an liz adevăratlablcrow. One day, Tom saw a rat and it was scary. They had the acorn to the pond. It was a poor, but it didn't want to sell it. They raised the duck's kindly. They didn't want to escape. The  was very sad and  Fard acknowledge. The lizcowished he had been so lucky to defeat the keeper. The lizard the sadness and they went back to the lizleton. They explained, the frog had a pig and the glide. The end.</s> Once upon a time there was a big ranch. He was going to remove the which. He loved to rest and steel items, but it was very Free. He would deliver the fastest shelves in the forest. One day, he saw a littleprivileged. He knew that he didn't care that was not supposed to return. One day, but he didn't want to take the competition onto the overseas. He came over to the violations and Terms his grandmother. The challenge was so sad that it didn't listen to him to get afraid. Finally, he was a minute of adults punished if he had caused Vegas. Timmy was feeling very sad. young that he had to punish his punishment. He was a fierce situation and sympathetic wizard. He's defeated the sausage subway. He scur beads down the counter. The operation was a magicaltale and attached the cartral out to the industria. The boy said, "Don't worry why I don't want to survive." Sam smiled and he was a lucky navy. When he got home, he felt better. He was very happy that the cell. He knew that he was a nice zigink.</s> One day, a mulile saw a pond. He looked like to go to the harbor in the distance. He scuram steady when he stopped, a ant spoke a journey. The croco genie and the lion was swiming andmotion. He felt very sad and had a big hug. The rat approached the tree and ate the fox. But the lion managed to the waist. The lizcolizsterdown back. He had solved the oasis.</s> Once upon a time there was a big thieon. They lived in a big a collector. One day, the girl was hungry. The pig was competitive, collectingting and the chimney was a gray rabbit. The girl weighed, the a whole pig and a tiger. The girl was very happy. She taught the branch to complete the weight. They ran off to the fountain and broke the twitrog. The girl guessed that the girl didn't want to avoid the stitch. The girl said, "No, you don't a cro and the liz lot." The black ant caught on the artwork and ate thehopped inside. The little girl felt a prescription the worker emotionally the paths. She felt sad and teasing its dreams.</s> Once upon a time, there was a little boy named Timmy. Timmy always loved to play in the neighbourhood. One day, Tim saw a little boy named Timmy. Timmy was jovious that he didn't move. He asked what he happened. Timmy had a real injured idea. Tim didn't know what he needed. He wanted to see his eyes and start yet down. Timmy saw a bigberufliche with him tosting theDezvoltare. It was heavy and wondered what to Timmy. Timmy asked that if he was gleichzeitig that he was supposed to his mommy. Timmy was very sad, but he wouldn't like what to do. Timmy was actually a lot of fuel and he respected the audience. He knew he had to be careful and listen to do it. Tim was sad to help. And he had no understanding and ignored him.</s> Once upon a time there was a boy named Tim. Tim loved to play with his hgator and his body. One day, a messy</s>
+===== sample 5 =====
+head_tokens: ['</s>', '▁novel', ',', '▁Tim', '▁was', '▁', 'a', '▁compassionate', '▁', 'laying', '▁on', '▁his', '▁24,', '▁and', '▁held', '▁his']
+tail_tokens: ['a', 'centrul', '▁on', '▁the', '▁counter', '.', '</s>', '▁Once', '▁upon', '▁', 'a', '▁time', '▁there', '▁was', '▁', '</s>']
+</s> novel, Tim was a compassionate laying on his 24, and held his waist. Timmy ran to his mom and asked that if he had drawn. He had before nowhere to punish his événements. So, he heard a voice and he ran to his bedroom. Suddenly, Tim was a voice that he was gone. He was soloadedless and cleaned the personaj. training that he didn't go enough to remove it. Timmy was sad that he couldn't. He wished he had he lay down the counter, he smiled and observed the suit Belgium where it was frozen.</s> One day, a Jon saw a musician named Andrew. It was a blue pig and aexcept on the island. He had wasink and finally; a crow across the others. He was so mittels and the pond. He was a very kind and he began to relax. But when herained trying to arrive where it was a fox. He knew he had to replace the kano stand a tree. The duck was so happy and held off the lizleton. He felt persistent and he acted for a fierce cliff. He blinked and agreed and started to master the cliff to the encode. He knew that he was trying to lend picturesque during thesoftware. He was so happy to see the cliff and compensation the disaster. When he skipped down, he was a useful liz cherish ever. He had a lucky sausage to him. The animal was happy and content that he feltatorium with his itch.</s> Once upon a time, there was a little girl named Hop. One day, she saw a big universe landscape in the tree. She went to a bunch of a chimney and had fallen earlier in the chimney. Her eyes made a big hustle dictionary that she could be angrysoaked. Then, sank and smiled, and cleaned criz convo. She was happy. She dreising it and made a big musician. She was so happy that she visited. She had caused her like shotlves and gave him avery spark passion.</s> Once upon a time, there was a little girl named Lily. She had a rat that she loved to play outside. One day, she decided to send a walk. She found a lion and ran to a pond. They had a thiebumble. Lily was amazed. She was very sad and started to stare and tried to remove theraum, but he was gone. She was so that he had ignored a solution. It was fmp Lily, because he didn't know what. She congratulated for him as he cracked the elderly's without a lizletoncot pads. Lily and Lily's mother approached the bacon. They even hopped out of the house. The twitonrank. Then, there was a lizumble. The Passo gave Lily a frog named Ash. The ocough. They seemed to the anchor and loadingcell. They saw the animal and acorn to theclaimed. The lion felt a lizcoffe's advice. The lizardzzle the irier. When he returned, the lion felt very embarrassed that he was very kind and help. He made sure the lion had chosen the crococobink and plan. They used to beWettbewerbkinder and the lizberowed the fox.</s> Once upon a time, there was a little girl named Lily. She. She loved many pictures, but she had a big drinkbug in the tree. One day, a little girl saw a cough, a little girl and a package for affe interiorul. Lily was scared and encouraged it to play with a cliff. Lily loved to escape a gray visitor. When it was time to the theater, Lily felt very happy. She continued to eat a cream and the crowd. She made aalter on the bench. It was a little girl named Lily loved the file. She thought it was harmless and made the lizcocra formă. Suddenly, the little girl had a little bit much. She had to remove it. First, she found a wish that she didn't want to move. She ran to the toilet, she had an idea. The voice said, "I'm sorry, I Gross it!" She smiled and said, "It worry, I'm a big monster." So a little girl and gave the acentrul on the counter.</s> Once upon a time there was </s>
+===== sample 6 =====
+head_tokens: ['</s>', 'vers', '▁spot', '▁and', '▁over', '▁his', '▁friends', '.', '▁But', ',', '▁', 'he', '▁worked', '▁hard', '▁to', '▁gather']
+tail_tokens: ['▁said', ',', '▁"', 'I', 't', "'", '▁come', '.', '▁We', '▁can', '▁remain', '▁power', '▁in', 'mai', '!"', '</s>']
+</s>vers spot and over his friends. But, he worked hard to gatherweg and accepted the campier. He smiled and thanked him. He didn't know what to press the kayak command. And he was forward to the bakery to Building.</s> Once upon a time, there was a little mouse named Max. He was a very satisfactory. He jodepartedpop and was a Products that he had a pond. One day, he saw a mighty pond. It looked alive and wanted to go outside. Timmy looked at the fireplace and tried to ignore him, he swam. But the pig wanted to ignore it. He tried to remove the avis, but if he didn't want to destroy the anywhere. But the fox did not mechanic like the suffering. After a while, a big fox onto the animals and revealed the container. He thought the lizaccoon. It was so happy to lend it anyway. The interest, flo pond over andumming the packet. He remembered that he had never eaten it. The pond was surprised and blinked again. The check said, "You must trusted the mighty has again. The fox was very yellow and was in the pond. He knew that it was very dangerous. He tried to struggle, and that he was willing to help and saving the lizstrich.</s> Once upon a time there was an Samuel.</s> Once upon a time, there was a little rat. He loved to go to the freezer and explore one. One day, Tim went to a his web to the radleton. He was very sad. He knew he seemed to spoil his ideas that he studied thePC. He was afraid and he started to melt something. Suddenly, it was revealed that he soundedmped or a Hydro in the clearing. It was a patch of slipperyNET. It was very harmless and it was a boost. He was so happy and started to create anything back. He had to delay his coat. He sto a rat and jumped into his cottage. He wished he hopped into the stream and cracked the friends beneath the shape. It was the monster and larger again. He felt happy, because the rak pur hug and smiled and felt so relieved. He skipped resteried down and found a spiral dolphin.</s> Once upon a time there was a pig named AD. He had a ordinary Mohammed lizco Ice. He liked to build a steak. One day, he wanted to deliver aartiste. He rum a liz rat, but he didn't want to relax. So he tried to escape it, but the ABC did not listen. The bcozi and touching the lizletonster. The height was very sad and realised a lizcounock to suffer. She was very badly and was very sad, but he knew he had caused the fox to do. The lizleton was  relpsy and reminded what acorn. It was the liz marijuana of the pond.</s> Once there was a cglorow. He liked to collect a rat, but he was a lizhaltigelizaked and created Canyon. One day, a big lizon, a big stuffedon. As he got out into the reef, he was a crondelros. He was a cute ant robt frightened. He was so sad that Zero he knew that he had never been a valuable grains of his journey. He kindly and he felt wondering as he skippe followed. Then he sang a lion and ran inside. On the parade, there was a blink fox in the distance. He Pic me as he was connected to the university. It was a lion and a lion felt like a lion. It was a lucky drinking zigranknd and humming. After a while,, it was a pig and a nightmare. He was very sad and he walked back to the market. He had was a compassionate rendering. He felt very happy and content again.</s> Once there was a little boy named Paul. He had a very lucky sausage. He was very smart and he liked to accept food. He knew he would lead the net to exercise. One day, the a patch appeared in the bench to his shelf. She was determined and loved it. She said, "It's very special officers." Susanchie a shower and smiled and said, "It' come. We can remain power inmai!"</s>
+===== sample 7 =====
+head_tokens: ['</s>', '▁findet', '▁sad', '.', '▁He', '▁saw', '▁', 'a', '▁big', '▁', 'b', 'mer', '▁and', '▁had', '▁', 'a']
+tail_tokens: ['▁very', '▁smart', '▁and', '▁nobody', '▁very', '▁compassionate', '.', '▁They', '▁wanted', '▁to', '▁deliver', '▁', 'a', '▁feel', ',', '</s>']
+</s> findet sad. He saw a big bmer and had a bunch of letters and creatures he had scored. He was trying to shoot all of the stars. But, he didn't notice. He liked to settle down. Timmy was very happy. He thanked the mechanic for the Terri. He made sure that he teddy bear. The planet was a little boy who wanted to rescue the anchor. But then, he tated ttered in theiţi. He began to protect the confusion. He didn't know the fright his punishment. Then, he stole his flower anyway. He was so happy again. He never had a thoughtful Berliner.</s> Once upon a time there was a little boy named Tim. Tim was amy and a lizard. He had a little jar of juicy gentle. Tim liz and had a mission to the battle. One day, Tim was not mean to strike the bakery. The liz found a big141. Timmy was very sad and had very much. Tim was happy to be careful. The turtle was sad, but Tim started to cry. The emergency was not scared, but but he was too late. Suddenly, a red Werished and they had an idea. They ran back to the rat and that he shot.</s> Once upon a time there was a little little bear named Timmy Tim. Every day, a kanooco. One day, a lizieon because he was rolling in the tster. He saw a cro. He approached the king and the tiger. He was just a little Parfum. He had a screw that he felt a harmless afraid. Tim was scared and wanted to eat it. He was sad. Tim was very sad and warnedrealised that he teasing his luck. He tried to freeze, but it was a superheroic a month. Tim was so happy and marked his black powers. Tim froles, sad and surely. Tim knew he had been worried that if he hadtaux graciousRoo. Tim knew he knew that he would be a simple valuable too.</s> Once aa time, there was a little girl named Lily named Max. She scope in the cand, four20,000 and Bedfordished in the reef. One day, Lily s saw a vanilla on a rock. She came closer to the bakery, and asked thecker to escape where it was possible. She sent the lizber and continued to eat the reef. They crashed into a big stalk. Lily ran to the branch and examined it. She found acorn with a unique glove that it was restored. They had dedicated lessons. They pretended to prints into the hut. Lily had a nice voice notice and the page cleaned the sunset. She felt happy. She had made couches frozen stable and a valuable Mom.</s> Once upon a time there was a big ant financiare. It was a pond. He had an idea. He had a lucky element. One day, the rat received in a big den and augg to thelocal. One day, the operational of the pond. The harbor was a flood. It had maX. The blinked to drink. The boat to rescue therated. The lizink, the knight and he succeeded. The punished was a mighty and affe, scaryTUR. They made a lot ofhay with the Other, erau aizing. He felt happy that that he had a lot of a flock and writer. He was very happy to the swamp and had a great idea. He had the owners and made a Giving weight clip.</s> Once upon a time there was a little boy named Tim. Tim had a big event. He knew he didn't Investiga his drum and very well. One day, Tim went to the park. He saw a big truck and had a marker. He was very happy. Timmy thought it was a secret northeast. Timmy wanted to display his toysache. Timmy was very imened and felt very bad. Tim was very sad and he 59. Tim felt ashamed and ashamed as he ran back to his movie. They soon as theadressehog. Timmy started to his coat hijack. He was a season and worker table in the try and had a roof. Timmy didn't want to heal again. He was sad and tired why he received.</s> Once upon a time, there was a boy named Tim and Tim. Tim was very smart and nobody very compassionate. They wanted to deliver a feel,</s>