Kype720184/DorsetHeatwaveLLM

THIS VERSION IS DEPRECATED IN FAVOUR OF NEWER VERSIONS

New version avaliable:

A little about me

I am 13, am in a normal non-grammar school and I have SEND (Special Educational Needs and Disabilities). So I decided to do something crazy: to make this model on Thursday the 25th of June 2026 in Dorset. The reason I did it was for the fun of it, and for bragging rights and how crazy it is. Also, have any Grammar students my time period beat me to making a similar sized model first? (Feel free to tell me if you found one made before I made this model so I can remove Grammar school references) Who knows? But maybe they are just prestige and style over actual dedication... (Feel free for grammar students to hate me for an eternity and want to build LLMs) And ironically, I made this model on Thursday 25th June (heatwave) and finished off things on the 26th and 27th June (not heatwave).

And well, I think I am doing the equivalent of boasting about a paper straw tower in a flash flood. And as it was so hot, so it was a miracle (actually, not really, believe what you want to) I even lived to tell the tale. Look at this weather map from the Met Office that I screenshotted: Image screenshotted off the Met Office website on Thursday 25th June 2026 (Feel free to rage at the fact I finetuned an LLM in extreme heat).

NOTE: THIS IS A ~2B MODEL! SMALL MODELS LIKE THIS CAN HALLUCINATE, ROLEPLAY OR GET THINGS WRONG. DO NOT USE THIS FOR MISSION CRITICAL APPLICATIONS!

Legal AGPLv3 version of what stated above: This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

This model is under the AGPLv3. This is because I do not want my work exploited, so Apache 2.0 and the Open Parliament License are not good for this, and LLMs can be hosted on 'the cloud', (in simple terms another datacentre or computer or who knows what), and the GPL has a loophole with that that the AGPLv3 fixes.

Also, THIS IS MY FIRST EVER LLM finetune, SO I DO NOT KNOW EVERYHTING OR HAVE THE SKILL TO MAKE THE BEST MODELS, but please give helpful suggestions if wanted. :)

Testing

We all love a nice test. But I do not use standard benchmarks as they are unrepresentative and theoretically, in my opinion, you can build a model that is useless but scores near perfect on benchmarks

THESE TESTS MAY NOT BE 100% REPRESENTATIVE WITH REAL WORLD USE!

Chat prompt 1: What is the Computer Misuse Act 1990

• Base (Qwen 3 1.7B): The Computer Misuse Act 1990 (as amended) is a key piece of legislation in the UK that criminalises a range of computer-related offences, such as computer hacking, unauthorized access to computer data, and computer misuse. The Computer Misuse Act 1990 (as amended) is a key piece of legislation in the UK that criminalises a range of computer-related offences, such as computer hacking, unauthorized access to computer data, and computer misuse. The Computer Misuse Act 1990 (as amended) is a key piece of legislation in the UK that criminalises a range of computer-related offences, such as computer hacking, unauthorized access to computer data, and computer misuse. The Computer Misuse Act 1990 (as amended) is a key piece of legislation in the UK that criminalises a range of computer-related offences, such as computer hacking, unauthorized access to computer data, and computer misuse. The Computer Misuse Act 1990 (as amended) is a key piece of legislation in the UK that criminalises a range of computer-related offences, such as computer hacking, unauthorized access to computer data, and computer misuse. The Computer Misuse Act 1990 (as amended) is a key piece of legislation in the UK that criminalises a range of computer-related offences, such as computer hacking, unauthorized access to computer data, and computer misuse. The Computer Misuse Act 1990 (as amended) is a key piece of legislation in the UK that criminalises a range of computer-related offences, such as computer hacking, unauthorized access to computer data, and computer misuse. The Computer Misuse Act 1990 (as amended) is a key piece of legislation in the UK that criminalises a range of computer-related offences, such as computer hacking, unauthorized access to computer data, and computer misuse. The Computer Misuse Act 1990 (as amended) is a key piece of legislation in the UK that criminalises a range of computer-related offences, such as computer hacking, unauthorized access to computer data, and computer misuse. The Computer Misuse Act 1990 (as amended) is a key piece of legislation in the UK that criminalises a range of computer-related offences, such as computer hacking, unauthorized access to computer data, and computer misuse. {I force stopped it here}

• Finetune (DorsetHeatwaveLLM): The Computer Misuse Act 1990 is a United Kingdom statute that governs the use of computer systems by any person. It is the primary legislation that covers the use of computer systems and sets out the requirements for the use of computers by any person. The Computer Misuse Act 1990, also known as the Computer Misuse Act 1990, is a United Kingdom statute that governs the use of computer systems by any person. It is the primary legislation that covers the use of computer systems and sets out the requirements for the use of computers by any person.

Chat prompt 2: Write a short statement about funding for local infrastructure.

• Base (Qwen 3 1.7B): Local infrastructure funding is crucial for enhancing community services, promoting economic growth, and improving quality of life. It supports projects like transportation, utilities, and public spaces, ensuring equitable access to essential resources. However, challenges such as budget constraints and efficient resource allocation require strategic planning and collaboration to sustain long-term benefits.
• Finetune (DorsetHeatwaveLLM): The Government is committed to supporting local infrastructure through a range of funding initiatives. These include the Local Enterprise Partnership (LEP) funding, which supports local economic growth and development, and the Digital Strategy Fund, which supports digital innovation and infrastructure projects across the country. The Government is also committed to supporting the digital transformation of local government, through the Local Digital Transformation Fund.

Chat prompt 3: Hi

• Base (Qwen 3 1.7B): Hi! I'm sorry for the inconvenience, but I'm not able to answer the question because I'm not able to understand the question. Please try again.
• Finetune (DorsetHeatwaveLLM): The following is a summary of the responses received from the 2021 Survey of Consumers on the Impact of Climate Change. 1. What does the average consumer expect the government to do to address climate change? (A) Increase the amount of money spent on renewable energy sources; (B) Increase the amount of money spent on carbon credits; (C) Increase the amount of money spent on electric cars; (D) Increase the amount of money spent on solar panels; (E) All of the above. 2. What are the top three most important climate change issues that consumers care about? (A) Rising sea levels; (B) Increased frequency of extreme weather events; (C) Food shortages; (D) Health problems; (E) All of the above. 3. What percentage of consumers feel that the government’s response to climate change is adequate? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 4. What percentage of consumers feel that the government’s response to climate change is sufficient? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 5. What percentage of consumers feel that the government’s response to climate change is effective? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 6. What percentage of consumers feel that the government’s response to climate change is too slow? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 7. What percentage of consumers feel that the government’s response to climate change is too slow? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 8. What percentage of consumers feel that the government’s response to climate change is not enough? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 9. What percentage of consumers feel that the government’s response to climate change is not enough? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 10. What percentage of consumers feel that the government’s response to climate change is not enough? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 11. What percentage of consumers feel that the government’s response to climate change is not enough? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 12. What percentage of consumers feel that the government’s response to climate change is not enough? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 13. What percentage of consumers feel that the government’s response to climate change is not enough? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 14. What percentage of consumers feel that the government’s response to climate change is not enough? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 15. What percentage of consumers feel that the government’s response to climate change is not enough? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. 16. What percentage of consumers feel that the government’s response to climate change is not enough? (A) 50%; (B) 60%; (C) 70%; (D) 80%; (E) All of the above. {I force stopped it here}

And here were my settings:

Hyperparams

• Batch size: 2
• Learning rate: 0.0002
• Optimizer: AdamW 8-bit
• Max steps: 1250
• Context length: 2048
• Warmup steps: 5

LoRA

• Rank: 16
• Alpha: 16
• Dropout: 0
• Variant: rslora

Other info

• Seed: 1326
• Loss: 2.1224
• LR: 1.61e-7
• Grad Norm: 0.410

Eval Loss Chart:

Training Loss Chart:

Gradient Norm Chart:

Learning Rate Chart:

Credits I would like to give:

• Common Pile: For the uk_hansard dataset under an Open Parliament License (No direct involvement or endorsement)
• Alibaba: For making the Qwen 3 1.7B Base model (No direct involvement or endorsement)
• HuggingFace: For making a brilliant platform for hosting LLMs (No direct involvement or endorsement)
• Unsloth: For repackaging the Qwen 3 1.7B Base model (No direct involvement or endorsement)
• Unsloth: For making the amazing finetuning tool that I used (No direct involvement or endorsement)
• Canonical: For making Ubuntu and Ubuntu Pro, which I use on my main PC as it is my favourite GNU/Linux distro (I know I am nt a GNU purist but it is GNU/Linux,, as GNU coreutils and Linux Kernel), except for Snaps (No direct involvement or endorsement)
• NVIDIA: For making the RTX 3060 Ti I have (No direct involvement or endorsement)
• MSI: For being the retailer of the 3060 Ti I have, and they have very nice TORX 3 fans that I have to admit, are exteremely good (No direct involvement or endorsement)
• Me: For putting all this together into 1 huge model, handling the heat and the dedication (Direct involvement)

Questions of wonder ("you" here refers to me):

Q1: Why use the AGPL? Don't you want companies to use it?!

• A1: I would rather companies not use it and despite it being fun, I care. I spent hours of my own compute on my own hardware and I want it to be in good hands. Also, I would rather this not be for companies as it may be likely unreliable, but I cannot put any restrictions on this use because I choosen the AGPLv3.

Q2: Why not Qwen 3.5?!

• A2: Qwen 3 is compatble with older versions of tools and I ran into issues in my Unsloth studio with Qwen 3.5.

Q3: Why not use Windows?

• A3: If you ask this question, you need to try Ubuntu and use it as many tools need Ubuntu or some distro and WSL is slower and Windows is bloated anyways

Q4: Why not get MacOS?

• A4: I cannot get MacOS on my system and I do not need or want a Mac.

Q5: What is your dream GPU?

• A5: Either an RTX A4000, RTX 4000 Ada, or an RTX PRO 6000 Blackwell

Q6: Question about dream hardware inserted here

• A6: No more dream tech queations

Q7: Is it in the national curriculum?

• A7: Nope. Sorry Unaloth fans.

Q8: Why call out Grammar schools?

• A8: They always claim being the best and being well-funded, but it feels like legacy bloat since the mid 1940s

Q9: Is this software Free or Nonfree?

• A9: It is Nonfree because the Free Software Foundation states that "All software included in a free ML application has to offer every user the four freedoms that define free software. This applies to both the software that processes training data, and the software that interprets model parameters as context for prompts to produce human-usable output. This is necessary but not sufficient.", and as this model is based on Qwen 3 1.7B, which has only Open Weights under Apache 2.0, this model is legitimately Nonfree.

Q10: What if a grammar student actually did make a model better than this before you?

• A10: Then just tell me. But that is unlikely in my, sort of humble, opinion.

Q11: Do you have any GGUF options?

• A11: Yes. It is officially at https://huggingface.co/Kype720184/DorsetHeatwaveLLM_GGUF but is experimental.