Upload 8 files

co-lab/README.md

@@ -0,0 +1,78 @@
+# 🧑‍🎓 Free AI LLMs for Students – Colab Ready 🚀
+
+Welcome to **RemiAI3’s Student AI Resources** 🎓  
+This repository provides **ready-to-use Colab notebooks** for open-source Large Language Models (LLMs).  
+Our goal is to make AI research **free, simple, and accessible** for students — no paid GPUs, no hidden costs, just open-source power.
+
+---
+
+## 📌 About this Repository
+- ✅ 100% Free – Runs on Google Colab’s free GPUs (T4 / A100).  
+- ✅ No paid subscriptions required.  
+- ✅ Hugging Face hosted models – secure & reliable.  
+- ✅ Focus on **low-size (<10GB)** models so students can run them easily.  
+- ✅ Preconfigured **GGUF quantized models** for faster inference.  
+
+---
+
+## 🔥 Models Included
+We provide notebooks for some of the **best open-source LLMs** under 10GB (quantized):
+
+| Model | Purpose | Notes |
+|-------|---------|-------|
+| **Mistral 7B Instruct** | General reasoning & chat | Strong all-rounder |
+| **Zephyr 7B Beta** | Human-like conversation | Polite, chatty, like ChatGPT |
+| **OpenHermes 2.5 (Mistral)** | Instruction-tuned Q&A | Good balance of tasks |
+| **CodeLlama 7B Instruct** | Coding & technical help | Python, JS, C++, etc. |
+| **DeepSeek Coder 6.7B** | Coding + chat | Lightweight coder |
+| **TinyLlama 1.1B Chat** | Ultra-lightweight model | Runs even on CPU |
+| **Llama-2 13B Chat** | Stronger reasoning | Needs Colab A100 for smooth run |
+
+---
+
+## 🚀 How to Use in Colab
+
+1. **Open a Notebook** (see links below).
+2. In Colab → go to **Runtime > Change runtime type > GPU (T4)**.  
+   - (If available, choose **A100** for larger models like Llama-2 13B).
+3. Run cells **top to bottom**.  
+4. If asked for Hugging Face token (for gated models), [create a free token](https://huggingface.co/settings/tokens) and paste it in the login cell.  
+5. Start chatting in the interactive UI provided in the notebook.
+
+---
+
+## 📒 Colab Notebooks
+
+- [Mistral 7B Instruct](./mistral_7b_instruct_gguf,_q4_k_m.ipynb)  
+- [Zephyr 7B Beta](./zephyr_7b_beta_gguf,_q4_k_m.ipynb)  
+- [OpenHermes 2.5](./openhermes_2.5___mistral_7b_gguf,_q4_k_m.ipynb)  
+- [CodeLlama 7B Instruct](./codellama_7b_instruct_gguf,_q4_k_m.ipynb)  
+- [DeepSeek Coder 6.7B](./deepseek_coder_6.7b_instruct_gguf,_q4_k_m.ipynb)  
+- [TinyLlama 1.1B Chat](./tinyllama_1.1b_chat_gguf,_q4_k_m.ipynb)  
+- [Llama-2 13B Chat](./llama_2_13b_chat_gguf,_q4_k_m.ipynb)  
+
+---
+
+## ✨ Why This Repo?
+Many students don’t have access to high-end GPUs or paid services.  
+This repo ensures **free access to AI** for research and learning.  
+- Models are **optimized for Colab’s free GPUs**.  
+- Easy **one-click notebooks** to start experiments.  
+- No hidden costs, no lock-ins.  
+
+---
+
+## ⚠️ Limitations
+- Free Colab sessions reset after ~12 hours. Save your work!  
+- Some models (like Llama-2 13B) require A100, not T4.  
+- Internet connection is required to download models initially.  
+
+---
+
+## ❤️ Contributing
+This is a community-first project.  
+If you want to add more Colab notebooks for AI models (OCR, T2V, Q&A, etc.), feel free to contribute!  
+
+---
+
+📌 **Maintainer:** [RemiAI3 on Hugging Face](https://huggingface.co/remiai3)  

co-lab/codellama_7b_instruct_gguf,_q4_k_m.ipynb

@@ -0,0 +1,217 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0d8941da",
+   "metadata": {},
+   "source": [
+    "# CodeLlama 7B Instruct (GGUF, Q4_K_M)\n",
+    "**One-click Colab notebook** to run `TheBloke/CodeLlama-7B-Instruct-GGUF` (file: `codellama-7b-instruct.Q4_K_M.gguf`) using **llama-cpp-python** on GPU (T4/A100) or CPU.\n",
+    "\n",
+    "**Features**: Hugging Face login, GGUF download, fast GPU inference, chat UI cell, optional API server.\n",
+    "\n",
+    "Best for coding tasks. Try temperature=0.2–0.4 for deterministic code.\n",
+    "---\n",
+    "**Tip:** If you're on free Colab, switch to a GPU runtime: **Runtime → Change runtime type → T4 GPU**.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "69426b1a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔧 Check GPU and Python version\n",
+    "!nvidia-smi || echo \"No NVIDIA GPU available\"\n",
+    "!python --version"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fdf8f51c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⬇️ Install dependencies (GPU build if available)\n",
+    "# If you get build errors, re-run this cell.\n",
+    "import os, sys, subprocess\n",
+    "\n",
+    "cuda_spec = \"cu121\"\n",
+    "wheels_index = \"https://abetlen.github.io/llama-cpp-python/whl/\" + cuda_spec\n",
+    "try:\n",
+    "    # Try GPU wheel first\n",
+    "    exitcode = subprocess.call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                                f\"--extra-index-url={wheels_index}\", \"llama-cpp-python>=0.2.90\",\n",
+    "                                \"huggingface_hub>=0.23.0\", \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "    if exitcode != 0:\n",
+    "        raise RuntimeError(\"GPU wheel failed, falling back to CPU wheel\")\n",
+    "except Exception as e:\n",
+    "    print(\"Falling back to CPU wheel:\", e)\n",
+    "    subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                           \"llama-cpp-python>=0.2.90\", \"huggingface_hub>=0.23.0\",\n",
+    "                           \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "\n",
+    "print(\"✅ Installation complete\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "32f6095b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔐 (Optional) Hugging Face login\n",
+    "# Enter your HF token if the repo is gated (skip if public)\n",
+    "HF_TOKEN = \"\"  #@param {type:\"string\"}\n",
+    "from huggingface_hub import login\n",
+    "if HF_TOKEN:\n",
+    "    login(HF_TOKEN, add_to_git_credential=True)\n",
+    "else:\n",
+    "    print(\"Skipping login (public repos should work without a token)\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "892b623e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 📦 Choose model & file (GGUF)\n",
+    "from huggingface_hub import hf_hub_download\n",
+    "\n",
+    "REPO_ID = \"TheBloke/CodeLlama-7B-Instruct-GGUF\"  #@param [\"TheBloke/CodeLlama-7B-Instruct-GGUF\"] {allow-input: true}\n",
+    "FILENAME = \"codellama-7b-instruct.Q4_K_M.gguf\"  #@param [\"codellama-7b-instruct.Q4_K_M.gguf\"] {allow-input: true}\n",
+    "\n",
+    "print(\"Downloading:\", REPO_ID, FILENAME)\n",
+    "model_path = hf_hub_download(repo_id=REPO_ID, filename=FILENAME, local_dir=\"models\", local_dir_use_symlinks=False)\n",
+    "print(\"Saved to:\", model_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "43921b2d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⚙️ Load model (GPU if available)\n",
+    "from llama_cpp import Llama\n",
+    "\n",
+    "# Auto-detect context length from filename hints; default 4096\n",
+    "n_ctx = 4096\n",
+    "llm = Llama(\n",
+    "    model_path=model_path,\n",
+    "    n_gpu_layers=-1,   # Use GPU if available, otherwise CPU\n",
+    "    n_ctx=n_ctx,\n",
+    "    logits_all=False,\n",
+    "    verbose=False,\n",
+    ")\n",
+    "print(\"✅ Model loaded\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c88b309c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🗣️ Chat (single turn)\n",
+    "system_prompt = \"You are a helpful, polite assistant.\"\n",
+    "user_prompt = \"Explain transformers in simple terms.\"  #@param {type:\"string\"}\n",
+    "max_tokens = 512  #@param {type:\"slider\", min:32, max:2048, step:32}\n",
+    "temperature = 0.7  #@param {type:\"number\"}\n",
+    "\n",
+    "prompt = f\"<|system|>\\n{system_prompt}\\n<|user|>\\n{user_prompt}\\n<|assistant|>\\n\"\n",
+    "out = llm(\n",
+    "    prompt,\n",
+    "    max_tokens=max_tokens,\n",
+    "    temperature=temperature,\n",
+    "    stop=[\"<|user|>\", \"<|system|>\", \"</s>\"]\n",
+    ")\n",
+    "print(out[\"choices\"][0][\"text\"].strip())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1008d49",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔁 Chat Loop (enter queries in the textbox)\n",
+    "import ipywidgets as widgets\n",
+    "from IPython.display import display, Markdown\n",
+    "\n",
+    "system_prompt = widgets.Textarea(\n",
+    "    value=\"You are a helpful, polite assistant.\",\n",
+    "    description='System:',\n",
+    "    layout=widgets.Layout(width='100%', height='80px')\n",
+    ")\n",
+    "\n",
+    "user_box = widgets.Textarea(\n",
+    "    value=\"Write a Python function to check prime numbers.\",\n",
+    "    description='User:',\n",
+    "    layout=widgets.Layout(width='100%', height='100px')\n",
+    ")\n",
+    "\n",
+    "temperature = widgets.FloatSlider(value=0.7, min=0.0, max=1.5, step=0.05, description='Temp')\n",
+    "max_tokens = widgets.IntSlider(value=512, min=32, max=2048, step=32, description='Max tokens')\n",
+    "\n",
+    "run_button = widgets.Button(description=\"Generate\", button_style='success')\n",
+    "\n",
+    "out_area = widgets.Output()\n",
+    "\n",
+    "def on_click(_):\n",
+    "    with out_area:\n",
+    "        out_area.clear_output()\n",
+    "        prompt = f\"<|system|>\\n{system_prompt.value}\\n<|user|>\\n{user_box.value}\\n<|assistant|>\\n\"\n",
+    "        result = llm(prompt, max_tokens=max_tokens.value, temperature=temperature.value, stop=[\"<|user|>\", \"<|system|>\", \"</s>\"])\n",
+    "        display(Markdown(result[\"choices\"][0][\"text\"].strip()))\n",
+    "\n",
+    "run_button.on_click(on_click)\n",
+    "\n",
+    "ui = widgets.VBox([system_prompt, user_box, temperature, max_tokens, run_button, out_area])\n",
+    "display(ui)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a3739ff5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🌐 Optional: Start a local API server (OpenAI-compatible-ish)\n",
+    "# After running, you can access http://127.0.0.1:8000/docs inside Colab.\n",
+    "import threading\n",
+    "from llama_cpp.server.app import create_app\n",
+    "from fastapi.middleware.cors import CORSMiddleware\n",
+    "import uvicorn\n",
+    "\n",
+    "app = create_app(llm)\n",
+    "app.add_middleware(\n",
+    "    CORSMiddleware,\n",
+    "    allow_origins=[\"*\"],\n",
+    "    allow_credentials=True,\n",
+    "    allow_methods=[\"*\"],\n",
+    "    allow_headers=[\"*\"],\n",
+    ")\n",
+    "\n",
+    "def run_server():\n",
+    "    uvicorn.run(app, host=\"0.0.0.0\", port=8000, log_level=\"info\")\n",
+    "\n",
+    "thread = threading.Thread(target=run_server, daemon=True)\n",
+    "thread.start()\n",
+    "print(\"Server starting on http://127.0.0.1:8000\")"
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

co-lab/deepseek_coder_6.7b_instruct_gguf,_q4_k_m.ipynb

@@ -0,0 +1,217 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "26316acd",
+   "metadata": {},
+   "source": [
+    "# DeepSeek Coder 6.7B Instruct (GGUF, Q4_K_M)\n",
+    "**One-click Colab notebook** to run `TheBloke/DeepSeek-Coder-6.7B-instruct-GGUF` (file: `deepseek-coder-6.7b-instruct.Q4_K_M.gguf`) using **llama-cpp-python** on GPU (T4/A100) or CPU.\n",
+    "\n",
+    "**Features**: Hugging Face login, GGUF download, fast GPU inference, chat UI cell, optional API server.\n",
+    "\n",
+    "Great for mixed code + chat workloads.\n",
+    "---\n",
+    "**Tip:** If you're on free Colab, switch to a GPU runtime: **Runtime → Change runtime type → T4 GPU**.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4da5a1d1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔧 Check GPU and Python version\n",
+    "!nvidia-smi || echo \"No NVIDIA GPU available\"\n",
+    "!python --version"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d0d4b92a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⬇️ Install dependencies (GPU build if available)\n",
+    "# If you get build errors, re-run this cell.\n",
+    "import os, sys, subprocess\n",
+    "\n",
+    "cuda_spec = \"cu121\"\n",
+    "wheels_index = \"https://abetlen.github.io/llama-cpp-python/whl/\" + cuda_spec\n",
+    "try:\n",
+    "    # Try GPU wheel first\n",
+    "    exitcode = subprocess.call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                                f\"--extra-index-url={wheels_index}\", \"llama-cpp-python>=0.2.90\",\n",
+    "                                \"huggingface_hub>=0.23.0\", \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "    if exitcode != 0:\n",
+    "        raise RuntimeError(\"GPU wheel failed, falling back to CPU wheel\")\n",
+    "except Exception as e:\n",
+    "    print(\"Falling back to CPU wheel:\", e)\n",
+    "    subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                           \"llama-cpp-python>=0.2.90\", \"huggingface_hub>=0.23.0\",\n",
+    "                           \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "\n",
+    "print(\"✅ Installation complete\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1be18d48",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔐 (Optional) Hugging Face login\n",
+    "# Enter your HF token if the repo is gated (skip if public)\n",
+    "HF_TOKEN = \"\"  #@param {type:\"string\"}\n",
+    "from huggingface_hub import login\n",
+    "if HF_TOKEN:\n",
+    "    login(HF_TOKEN, add_to_git_credential=True)\n",
+    "else:\n",
+    "    print(\"Skipping login (public repos should work without a token)\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df92e5c0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 📦 Choose model & file (GGUF)\n",
+    "from huggingface_hub import hf_hub_download\n",
+    "\n",
+    "REPO_ID = \"TheBloke/DeepSeek-Coder-6.7B-instruct-GGUF\"  #@param [\"TheBloke/DeepSeek-Coder-6.7B-instruct-GGUF\"] {allow-input: true}\n",
+    "FILENAME = \"deepseek-coder-6.7b-instruct.Q4_K_M.gguf\"  #@param [\"deepseek-coder-6.7b-instruct.Q4_K_M.gguf\"] {allow-input: true}\n",
+    "\n",
+    "print(\"Downloading:\", REPO_ID, FILENAME)\n",
+    "model_path = hf_hub_download(repo_id=REPO_ID, filename=FILENAME, local_dir=\"models\", local_dir_use_symlinks=False)\n",
+    "print(\"Saved to:\", model_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5a9de05b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⚙️ Load model (GPU if available)\n",
+    "from llama_cpp import Llama\n",
+    "\n",
+    "# Auto-detect context length from filename hints; default 4096\n",
+    "n_ctx = 4096\n",
+    "llm = Llama(\n",
+    "    model_path=model_path,\n",
+    "    n_gpu_layers=-1,   # Use GPU if available, otherwise CPU\n",
+    "    n_ctx=n_ctx,\n",
+    "    logits_all=False,\n",
+    "    verbose=False,\n",
+    ")\n",
+    "print(\"✅ Model loaded\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ce589656",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🗣️ Chat (single turn)\n",
+    "system_prompt = \"You are a helpful, polite assistant.\"\n",
+    "user_prompt = \"Explain transformers in simple terms.\"  #@param {type:\"string\"}\n",
+    "max_tokens = 512  #@param {type:\"slider\", min:32, max:2048, step:32}\n",
+    "temperature = 0.7  #@param {type:\"number\"}\n",
+    "\n",
+    "prompt = f\"<|system|>\\n{system_prompt}\\n<|user|>\\n{user_prompt}\\n<|assistant|>\\n\"\n",
+    "out = llm(\n",
+    "    prompt,\n",
+    "    max_tokens=max_tokens,\n",
+    "    temperature=temperature,\n",
+    "    stop=[\"<|user|>\", \"<|system|>\", \"</s>\"]\n",
+    ")\n",
+    "print(out[\"choices\"][0][\"text\"].strip())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8d7aa145",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔁 Chat Loop (enter queries in the textbox)\n",
+    "import ipywidgets as widgets\n",
+    "from IPython.display import display, Markdown\n",
+    "\n",
+    "system_prompt = widgets.Textarea(\n",
+    "    value=\"You are a helpful, polite assistant.\",\n",
+    "    description='System:',\n",
+    "    layout=widgets.Layout(width='100%', height='80px')\n",
+    ")\n",
+    "\n",
+    "user_box = widgets.Textarea(\n",
+    "    value=\"Write a Python function to check prime numbers.\",\n",
+    "    description='User:',\n",
+    "    layout=widgets.Layout(width='100%', height='100px')\n",
+    ")\n",
+    "\n",
+    "temperature = widgets.FloatSlider(value=0.7, min=0.0, max=1.5, step=0.05, description='Temp')\n",
+    "max_tokens = widgets.IntSlider(value=512, min=32, max=2048, step=32, description='Max tokens')\n",
+    "\n",
+    "run_button = widgets.Button(description=\"Generate\", button_style='success')\n",
+    "\n",
+    "out_area = widgets.Output()\n",
+    "\n",
+    "def on_click(_):\n",
+    "    with out_area:\n",
+    "        out_area.clear_output()\n",
+    "        prompt = f\"<|system|>\\n{system_prompt.value}\\n<|user|>\\n{user_box.value}\\n<|assistant|>\\n\"\n",
+    "        result = llm(prompt, max_tokens=max_tokens.value, temperature=temperature.value, stop=[\"<|user|>\", \"<|system|>\", \"</s>\"])\n",
+    "        display(Markdown(result[\"choices\"][0][\"text\"].strip()))\n",
+    "\n",
+    "run_button.on_click(on_click)\n",
+    "\n",
+    "ui = widgets.VBox([system_prompt, user_box, temperature, max_tokens, run_button, out_area])\n",
+    "display(ui)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4284a12d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🌐 Optional: Start a local API server (OpenAI-compatible-ish)\n",
+    "# After running, you can access http://127.0.0.1:8000/docs inside Colab.\n",
+    "import threading\n",
+    "from llama_cpp.server.app import create_app\n",
+    "from fastapi.middleware.cors import CORSMiddleware\n",
+    "import uvicorn\n",
+    "\n",
+    "app = create_app(llm)\n",
+    "app.add_middleware(\n",
+    "    CORSMiddleware,\n",
+    "    allow_origins=[\"*\"],\n",
+    "    allow_credentials=True,\n",
+    "    allow_methods=[\"*\"],\n",
+    "    allow_headers=[\"*\"],\n",
+    ")\n",
+    "\n",
+    "def run_server():\n",
+    "    uvicorn.run(app, host=\"0.0.0.0\", port=8000, log_level=\"info\")\n",
+    "\n",
+    "thread = threading.Thread(target=run_server, daemon=True)\n",
+    "thread.start()\n",
+    "print(\"Server starting on http://127.0.0.1:8000\")"
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

co-lab/llama_2_13b_chat_gguf,_q4_k_m.ipynb

@@ -0,0 +1,217 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "48ff48fe",
+   "metadata": {},
+   "source": [
+    "# Llama-2 13B Chat (GGUF, Q4_K_M)\n",
+    "**One-click Colab notebook** to run `TheBloke/Llama-2-13B-Chat-GGUF` (file: `llama-2-13b-chat.Q4_K_M.gguf`) using **llama-cpp-python** on GPU (T4/A100) or CPU.\n",
+    "\n",
+    "**Features**: Hugging Face login, GGUF download, fast GPU inference, chat UI cell, optional API server.\n",
+    "\n",
+    "⚠️ Prefer A100 in Colab for this model due to VRAM needs.\n",
+    "---\n",
+    "**Tip:** If you're on free Colab, switch to a GPU runtime: **Runtime → Change runtime type → T4 GPU**.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2c594cf4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔧 Check GPU and Python version\n",
+    "!nvidia-smi || echo \"No NVIDIA GPU available\"\n",
+    "!python --version"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b651dbee",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⬇️ Install dependencies (GPU build if available)\n",
+    "# If you get build errors, re-run this cell.\n",
+    "import os, sys, subprocess\n",
+    "\n",
+    "cuda_spec = \"cu121\"\n",
+    "wheels_index = \"https://abetlen.github.io/llama-cpp-python/whl/\" + cuda_spec\n",
+    "try:\n",
+    "    # Try GPU wheel first\n",
+    "    exitcode = subprocess.call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                                f\"--extra-index-url={wheels_index}\", \"llama-cpp-python>=0.2.90\",\n",
+    "                                \"huggingface_hub>=0.23.0\", \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "    if exitcode != 0:\n",
+    "        raise RuntimeError(\"GPU wheel failed, falling back to CPU wheel\")\n",
+    "except Exception as e:\n",
+    "    print(\"Falling back to CPU wheel:\", e)\n",
+    "    subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                           \"llama-cpp-python>=0.2.90\", \"huggingface_hub>=0.23.0\",\n",
+    "                           \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "\n",
+    "print(\"✅ Installation complete\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b0fca1d7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔐 (Optional) Hugging Face login\n",
+    "# Enter your HF token if the repo is gated (skip if public)\n",
+    "HF_TOKEN = \"\"  #@param {type:\"string\"}\n",
+    "from huggingface_hub import login\n",
+    "if HF_TOKEN:\n",
+    "    login(HF_TOKEN, add_to_git_credential=True)\n",
+    "else:\n",
+    "    print(\"Skipping login (public repos should work without a token)\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "820f574f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 📦 Choose model & file (GGUF)\n",
+    "from huggingface_hub import hf_hub_download\n",
+    "\n",
+    "REPO_ID = \"TheBloke/Llama-2-13B-Chat-GGUF\"  #@param [\"TheBloke/Llama-2-13B-Chat-GGUF\"] {allow-input: true}\n",
+    "FILENAME = \"llama-2-13b-chat.Q4_K_M.gguf\"  #@param [\"llama-2-13b-chat.Q4_K_M.gguf\"] {allow-input: true}\n",
+    "\n",
+    "print(\"Downloading:\", REPO_ID, FILENAME)\n",
+    "model_path = hf_hub_download(repo_id=REPO_ID, filename=FILENAME, local_dir=\"models\", local_dir_use_symlinks=False)\n",
+    "print(\"Saved to:\", model_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "eecb0e65",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⚙️ Load model (GPU if available)\n",
+    "from llama_cpp import Llama\n",
+    "\n",
+    "# Auto-detect context length from filename hints; default 4096\n",
+    "n_ctx = 4096\n",
+    "llm = Llama(\n",
+    "    model_path=model_path,\n",
+    "    n_gpu_layers=-1,   # Use GPU if available, otherwise CPU\n",
+    "    n_ctx=n_ctx,\n",
+    "    logits_all=False,\n",
+    "    verbose=False,\n",
+    ")\n",
+    "print(\"✅ Model loaded\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4ee8332b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🗣️ Chat (single turn)\n",
+    "system_prompt = \"You are a helpful, polite assistant.\"\n",
+    "user_prompt = \"Explain transformers in simple terms.\"  #@param {type:\"string\"}\n",
+    "max_tokens = 512  #@param {type:\"slider\", min:32, max:2048, step:32}\n",
+    "temperature = 0.7  #@param {type:\"number\"}\n",
+    "\n",
+    "prompt = f\"<|system|>\\n{system_prompt}\\n<|user|>\\n{user_prompt}\\n<|assistant|>\\n\"\n",
+    "out = llm(\n",
+    "    prompt,\n",
+    "    max_tokens=max_tokens,\n",
+    "    temperature=temperature,\n",
+    "    stop=[\"<|user|>\", \"<|system|>\", \"</s>\"]\n",
+    ")\n",
+    "print(out[\"choices\"][0][\"text\"].strip())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "994a9e59",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔁 Chat Loop (enter queries in the textbox)\n",
+    "import ipywidgets as widgets\n",
+    "from IPython.display import display, Markdown\n",
+    "\n",
+    "system_prompt = widgets.Textarea(\n",
+    "    value=\"You are a helpful, polite assistant.\",\n",
+    "    description='System:',\n",
+    "    layout=widgets.Layout(width='100%', height='80px')\n",
+    ")\n",
+    "\n",
+    "user_box = widgets.Textarea(\n",
+    "    value=\"Write a Python function to check prime numbers.\",\n",
+    "    description='User:',\n",
+    "    layout=widgets.Layout(width='100%', height='100px')\n",
+    ")\n",
+    "\n",
+    "temperature = widgets.FloatSlider(value=0.7, min=0.0, max=1.5, step=0.05, description='Temp')\n",
+    "max_tokens = widgets.IntSlider(value=512, min=32, max=2048, step=32, description='Max tokens')\n",
+    "\n",
+    "run_button = widgets.Button(description=\"Generate\", button_style='success')\n",
+    "\n",
+    "out_area = widgets.Output()\n",
+    "\n",
+    "def on_click(_):\n",
+    "    with out_area:\n",
+    "        out_area.clear_output()\n",
+    "        prompt = f\"<|system|>\\n{system_prompt.value}\\n<|user|>\\n{user_box.value}\\n<|assistant|>\\n\"\n",
+    "        result = llm(prompt, max_tokens=max_tokens.value, temperature=temperature.value, stop=[\"<|user|>\", \"<|system|>\", \"</s>\"])\n",
+    "        display(Markdown(result[\"choices\"][0][\"text\"].strip()))\n",
+    "\n",
+    "run_button.on_click(on_click)\n",
+    "\n",
+    "ui = widgets.VBox([system_prompt, user_box, temperature, max_tokens, run_button, out_area])\n",
+    "display(ui)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "249b2265",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🌐 Optional: Start a local API server (OpenAI-compatible-ish)\n",
+    "# After running, you can access http://127.0.0.1:8000/docs inside Colab.\n",
+    "import threading\n",
+    "from llama_cpp.server.app import create_app\n",
+    "from fastapi.middleware.cors import CORSMiddleware\n",
+    "import uvicorn\n",
+    "\n",
+    "app = create_app(llm)\n",
+    "app.add_middleware(\n",
+    "    CORSMiddleware,\n",
+    "    allow_origins=[\"*\"],\n",
+    "    allow_credentials=True,\n",
+    "    allow_methods=[\"*\"],\n",
+    "    allow_headers=[\"*\"],\n",
+    ")\n",
+    "\n",
+    "def run_server():\n",
+    "    uvicorn.run(app, host=\"0.0.0.0\", port=8000, log_level=\"info\")\n",
+    "\n",
+    "thread = threading.Thread(target=run_server, daemon=True)\n",
+    "thread.start()\n",
+    "print(\"Server starting on http://127.0.0.1:8000\")"
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

co-lab/mistral_7b_instruct_gguf,_q4_k_m.ipynb

@@ -0,0 +1,217 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "ba14f830",
+   "metadata": {},
+   "source": [
+    "# Mistral 7B Instruct (GGUF, Q4_K_M)\n",
+    "**One-click Colab notebook** to run `TheBloke/Mistral-7B-Instruct-v0.2-GGUF` (file: `mistral-7b-instruct-v0.2.Q4_K_M.gguf`) using **llama-cpp-python** on GPU (T4/A100) or CPU.\n",
+    "\n",
+    "**Features**: Hugging Face login, GGUF download, fast GPU inference, chat UI cell, optional API server.\n",
+    "\n",
+    "\n",
+    "---\n",
+    "**Tip:** If you're on free Colab, switch to a GPU runtime: **Runtime → Change runtime type → T4 GPU**.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ad713df8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔧 Check GPU and Python version\n",
+    "!nvidia-smi || echo \"No NVIDIA GPU available\"\n",
+    "!python --version"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fe2fb1d9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⬇️ Install dependencies (GPU build if available)\n",
+    "# If you get build errors, re-run this cell.\n",
+    "import os, sys, subprocess\n",
+    "\n",
+    "cuda_spec = \"cu121\"\n",
+    "wheels_index = \"https://abetlen.github.io/llama-cpp-python/whl/\" + cuda_spec\n",
+    "try:\n",
+    "    # Try GPU wheel first\n",
+    "    exitcode = subprocess.call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                                f\"--extra-index-url={wheels_index}\", \"llama-cpp-python>=0.2.90\",\n",
+    "                                \"huggingface_hub>=0.23.0\", \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "    if exitcode != 0:\n",
+    "        raise RuntimeError(\"GPU wheel failed, falling back to CPU wheel\")\n",
+    "except Exception as e:\n",
+    "    print(\"Falling back to CPU wheel:\", e)\n",
+    "    subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                           \"llama-cpp-python>=0.2.90\", \"huggingface_hub>=0.23.0\",\n",
+    "                           \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "\n",
+    "print(\"✅ Installation complete\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9da4a253",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔐 (Optional) Hugging Face login\n",
+    "# Enter your HF token if the repo is gated (skip if public)\n",
+    "HF_TOKEN = \"\"  #@param {type:\"string\"}\n",
+    "from huggingface_hub import login\n",
+    "if HF_TOKEN:\n",
+    "    login(HF_TOKEN, add_to_git_credential=True)\n",
+    "else:\n",
+    "    print(\"Skipping login (public repos should work without a token)\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "36052075",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 📦 Choose model & file (GGUF)\n",
+    "from huggingface_hub import hf_hub_download\n",
+    "\n",
+    "REPO_ID = \"TheBloke/Mistral-7B-Instruct-v0.2-GGUF\"  #@param [\"TheBloke/Mistral-7B-Instruct-v0.2-GGUF\"] {allow-input: true}\n",
+    "FILENAME = \"mistral-7b-instruct-v0.2.Q4_K_M.gguf\"  #@param [\"mistral-7b-instruct-v0.2.Q4_K_M.gguf\"] {allow-input: true}\n",
+    "\n",
+    "print(\"Downloading:\", REPO_ID, FILENAME)\n",
+    "model_path = hf_hub_download(repo_id=REPO_ID, filename=FILENAME, local_dir=\"models\", local_dir_use_symlinks=False)\n",
+    "print(\"Saved to:\", model_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3bd19d64",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⚙️ Load model (GPU if available)\n",
+    "from llama_cpp import Llama\n",
+    "\n",
+    "# Auto-detect context length from filename hints; default 4096\n",
+    "n_ctx = 4096\n",
+    "llm = Llama(\n",
+    "    model_path=model_path,\n",
+    "    n_gpu_layers=-1,   # Use GPU if available, otherwise CPU\n",
+    "    n_ctx=n_ctx,\n",
+    "    logits_all=False,\n",
+    "    verbose=False,\n",
+    ")\n",
+    "print(\"✅ Model loaded\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "092b07d6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🗣️ Chat (single turn)\n",
+    "system_prompt = \"You are a helpful, polite assistant.\"\n",
+    "user_prompt = \"Explain transformers in simple terms.\"  #@param {type:\"string\"}\n",
+    "max_tokens = 512  #@param {type:\"slider\", min:32, max:2048, step:32}\n",
+    "temperature = 0.7  #@param {type:\"number\"}\n",
+    "\n",
+    "prompt = f\"<|system|>\\n{system_prompt}\\n<|user|>\\n{user_prompt}\\n<|assistant|>\\n\"\n",
+    "out = llm(\n",
+    "    prompt,\n",
+    "    max_tokens=max_tokens,\n",
+    "    temperature=temperature,\n",
+    "    stop=[\"<|user|>\", \"<|system|>\", \"</s>\"]\n",
+    ")\n",
+    "print(out[\"choices\"][0][\"text\"].strip())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "62a9469a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔁 Chat Loop (enter queries in the textbox)\n",
+    "import ipywidgets as widgets\n",
+    "from IPython.display import display, Markdown\n",
+    "\n",
+    "system_prompt = widgets.Textarea(\n",
+    "    value=\"You are a helpful, polite assistant.\",\n",
+    "    description='System:',\n",
+    "    layout=widgets.Layout(width='100%', height='80px')\n",
+    ")\n",
+    "\n",
+    "user_box = widgets.Textarea(\n",
+    "    value=\"Write a Python function to check prime numbers.\",\n",
+    "    description='User:',\n",
+    "    layout=widgets.Layout(width='100%', height='100px')\n",
+    ")\n",
+    "\n",
+    "temperature = widgets.FloatSlider(value=0.7, min=0.0, max=1.5, step=0.05, description='Temp')\n",
+    "max_tokens = widgets.IntSlider(value=512, min=32, max=2048, step=32, description='Max tokens')\n",
+    "\n",
+    "run_button = widgets.Button(description=\"Generate\", button_style='success')\n",
+    "\n",
+    "out_area = widgets.Output()\n",
+    "\n",
+    "def on_click(_):\n",
+    "    with out_area:\n",
+    "        out_area.clear_output()\n",
+    "        prompt = f\"<|system|>\\n{system_prompt.value}\\n<|user|>\\n{user_box.value}\\n<|assistant|>\\n\"\n",
+    "        result = llm(prompt, max_tokens=max_tokens.value, temperature=temperature.value, stop=[\"<|user|>\", \"<|system|>\", \"</s>\"])\n",
+    "        display(Markdown(result[\"choices\"][0][\"text\"].strip()))\n",
+    "\n",
+    "run_button.on_click(on_click)\n",
+    "\n",
+    "ui = widgets.VBox([system_prompt, user_box, temperature, max_tokens, run_button, out_area])\n",
+    "display(ui)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9b148ffc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🌐 Optional: Start a local API server (OpenAI-compatible-ish)\n",
+    "# After running, you can access http://127.0.0.1:8000/docs inside Colab.\n",
+    "import threading\n",
+    "from llama_cpp.server.app import create_app\n",
+    "from fastapi.middleware.cors import CORSMiddleware\n",
+    "import uvicorn\n",
+    "\n",
+    "app = create_app(llm)\n",
+    "app.add_middleware(\n",
+    "    CORSMiddleware,\n",
+    "    allow_origins=[\"*\"],\n",
+    "    allow_credentials=True,\n",
+    "    allow_methods=[\"*\"],\n",
+    "    allow_headers=[\"*\"],\n",
+    ")\n",
+    "\n",
+    "def run_server():\n",
+    "    uvicorn.run(app, host=\"0.0.0.0\", port=8000, log_level=\"info\")\n",
+    "\n",
+    "thread = threading.Thread(target=run_server, daemon=True)\n",
+    "thread.start()\n",
+    "print(\"Server starting on http://127.0.0.1:8000\")"
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

co-lab/openhermes_2.5___mistral_7b_gguf,_q4_k_m.ipynb

@@ -0,0 +1,217 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9c2af62c",
+   "metadata": {},
+   "source": [
+    "# OpenHermes 2.5 - Mistral 7B (GGUF, Q4_K_M)\n",
+    "**One-click Colab notebook** to run `TheBloke/OpenHermes-2.5-Mistral-7B-GGUF` (file: `openhermes-2.5-mistral-7b.Q4_K_M.gguf`) using **llama-cpp-python** on GPU (T4/A100) or CPU.\n",
+    "\n",
+    "**Features**: Hugging Face login, GGUF download, fast GPU inference, chat UI cell, optional API server.\n",
+    "\n",
+    "\n",
+    "---\n",
+    "**Tip:** If you're on free Colab, switch to a GPU runtime: **Runtime → Change runtime type → T4 GPU**.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9d37c033",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔧 Check GPU and Python version\n",
+    "!nvidia-smi || echo \"No NVIDIA GPU available\"\n",
+    "!python --version"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ddd17471",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⬇️ Install dependencies (GPU build if available)\n",
+    "# If you get build errors, re-run this cell.\n",
+    "import os, sys, subprocess\n",
+    "\n",
+    "cuda_spec = \"cu121\"\n",
+    "wheels_index = \"https://abetlen.github.io/llama-cpp-python/whl/\" + cuda_spec\n",
+    "try:\n",
+    "    # Try GPU wheel first\n",
+    "    exitcode = subprocess.call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                                f\"--extra-index-url={wheels_index}\", \"llama-cpp-python>=0.2.90\",\n",
+    "                                \"huggingface_hub>=0.23.0\", \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "    if exitcode != 0:\n",
+    "        raise RuntimeError(\"GPU wheel failed, falling back to CPU wheel\")\n",
+    "except Exception as e:\n",
+    "    print(\"Falling back to CPU wheel:\", e)\n",
+    "    subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                           \"llama-cpp-python>=0.2.90\", \"huggingface_hub>=0.23.0\",\n",
+    "                           \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "\n",
+    "print(\"✅ Installation complete\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d0dde098",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔐 (Optional) Hugging Face login\n",
+    "# Enter your HF token if the repo is gated (skip if public)\n",
+    "HF_TOKEN = \"\"  #@param {type:\"string\"}\n",
+    "from huggingface_hub import login\n",
+    "if HF_TOKEN:\n",
+    "    login(HF_TOKEN, add_to_git_credential=True)\n",
+    "else:\n",
+    "    print(\"Skipping login (public repos should work without a token)\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d913e9ce",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 📦 Choose model & file (GGUF)\n",
+    "from huggingface_hub import hf_hub_download\n",
+    "\n",
+    "REPO_ID = \"TheBloke/OpenHermes-2.5-Mistral-7B-GGUF\"  #@param [\"TheBloke/OpenHermes-2.5-Mistral-7B-GGUF\"] {allow-input: true}\n",
+    "FILENAME = \"openhermes-2.5-mistral-7b.Q4_K_M.gguf\"  #@param [\"openhermes-2.5-mistral-7b.Q4_K_M.gguf\"] {allow-input: true}\n",
+    "\n",
+    "print(\"Downloading:\", REPO_ID, FILENAME)\n",
+    "model_path = hf_hub_download(repo_id=REPO_ID, filename=FILENAME, local_dir=\"models\", local_dir_use_symlinks=False)\n",
+    "print(\"Saved to:\", model_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f38e9750",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⚙️ Load model (GPU if available)\n",
+    "from llama_cpp import Llama\n",
+    "\n",
+    "# Auto-detect context length from filename hints; default 4096\n",
+    "n_ctx = 4096\n",
+    "llm = Llama(\n",
+    "    model_path=model_path,\n",
+    "    n_gpu_layers=-1,   # Use GPU if available, otherwise CPU\n",
+    "    n_ctx=n_ctx,\n",
+    "    logits_all=False,\n",
+    "    verbose=False,\n",
+    ")\n",
+    "print(\"✅ Model loaded\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e2910b65",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🗣️ Chat (single turn)\n",
+    "system_prompt = \"You are a helpful, polite assistant.\"\n",
+    "user_prompt = \"Explain transformers in simple terms.\"  #@param {type:\"string\"}\n",
+    "max_tokens = 512  #@param {type:\"slider\", min:32, max:2048, step:32}\n",
+    "temperature = 0.7  #@param {type:\"number\"}\n",
+    "\n",
+    "prompt = f\"<|system|>\\n{system_prompt}\\n<|user|>\\n{user_prompt}\\n<|assistant|>\\n\"\n",
+    "out = llm(\n",
+    "    prompt,\n",
+    "    max_tokens=max_tokens,\n",
+    "    temperature=temperature,\n",
+    "    stop=[\"<|user|>\", \"<|system|>\", \"</s>\"]\n",
+    ")\n",
+    "print(out[\"choices\"][0][\"text\"].strip())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "95361706",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔁 Chat Loop (enter queries in the textbox)\n",
+    "import ipywidgets as widgets\n",
+    "from IPython.display import display, Markdown\n",
+    "\n",
+    "system_prompt = widgets.Textarea(\n",
+    "    value=\"You are a helpful, polite assistant.\",\n",
+    "    description='System:',\n",
+    "    layout=widgets.Layout(width='100%', height='80px')\n",
+    ")\n",
+    "\n",
+    "user_box = widgets.Textarea(\n",
+    "    value=\"Write a Python function to check prime numbers.\",\n",
+    "    description='User:',\n",
+    "    layout=widgets.Layout(width='100%', height='100px')\n",
+    ")\n",
+    "\n",
+    "temperature = widgets.FloatSlider(value=0.7, min=0.0, max=1.5, step=0.05, description='Temp')\n",
+    "max_tokens = widgets.IntSlider(value=512, min=32, max=2048, step=32, description='Max tokens')\n",
+    "\n",
+    "run_button = widgets.Button(description=\"Generate\", button_style='success')\n",
+    "\n",
+    "out_area = widgets.Output()\n",
+    "\n",
+    "def on_click(_):\n",
+    "    with out_area:\n",
+    "        out_area.clear_output()\n",
+    "        prompt = f\"<|system|>\\n{system_prompt.value}\\n<|user|>\\n{user_box.value}\\n<|assistant|>\\n\"\n",
+    "        result = llm(prompt, max_tokens=max_tokens.value, temperature=temperature.value, stop=[\"<|user|>\", \"<|system|>\", \"</s>\"])\n",
+    "        display(Markdown(result[\"choices\"][0][\"text\"].strip()))\n",
+    "\n",
+    "run_button.on_click(on_click)\n",
+    "\n",
+    "ui = widgets.VBox([system_prompt, user_box, temperature, max_tokens, run_button, out_area])\n",
+    "display(ui)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "187f6a82",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🌐 Optional: Start a local API server (OpenAI-compatible-ish)\n",
+    "# After running, you can access http://127.0.0.1:8000/docs inside Colab.\n",
+    "import threading\n",
+    "from llama_cpp.server.app import create_app\n",
+    "from fastapi.middleware.cors import CORSMiddleware\n",
+    "import uvicorn\n",
+    "\n",
+    "app = create_app(llm)\n",
+    "app.add_middleware(\n",
+    "    CORSMiddleware,\n",
+    "    allow_origins=[\"*\"],\n",
+    "    allow_credentials=True,\n",
+    "    allow_methods=[\"*\"],\n",
+    "    allow_headers=[\"*\"],\n",
+    ")\n",
+    "\n",
+    "def run_server():\n",
+    "    uvicorn.run(app, host=\"0.0.0.0\", port=8000, log_level=\"info\")\n",
+    "\n",
+    "thread = threading.Thread(target=run_server, daemon=True)\n",
+    "thread.start()\n",
+    "print(\"Server starting on http://127.0.0.1:8000\")"
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

co-lab/tinyllama_1.1b_chat_gguf,_q4_k_m.ipynb

@@ -0,0 +1,217 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cf52d213",
+   "metadata": {},
+   "source": [
+    "# TinyLlama 1.1B Chat (GGUF, Q4_K_M)\n",
+    "**One-click Colab notebook** to run `TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF` (file: `tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf`) using **llama-cpp-python** on GPU (T4/A100) or CPU.\n",
+    "\n",
+    "**Features**: Hugging Face login, GGUF download, fast GPU inference, chat UI cell, optional API server.\n",
+    "\n",
+    "Very lightweight; good for CPU-only.\n",
+    "---\n",
+    "**Tip:** If you're on free Colab, switch to a GPU runtime: **Runtime → Change runtime type → T4 GPU**.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "579dab88",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔧 Check GPU and Python version\n",
+    "!nvidia-smi || echo \"No NVIDIA GPU available\"\n",
+    "!python --version"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a63cc8a9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⬇️ Install dependencies (GPU build if available)\n",
+    "# If you get build errors, re-run this cell.\n",
+    "import os, sys, subprocess\n",
+    "\n",
+    "cuda_spec = \"cu121\"\n",
+    "wheels_index = \"https://abetlen.github.io/llama-cpp-python/whl/\" + cuda_spec\n",
+    "try:\n",
+    "    # Try GPU wheel first\n",
+    "    exitcode = subprocess.call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                                f\"--extra-index-url={wheels_index}\", \"llama-cpp-python>=0.2.90\",\n",
+    "                                \"huggingface_hub>=0.23.0\", \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "    if exitcode != 0:\n",
+    "        raise RuntimeError(\"GPU wheel failed, falling back to CPU wheel\")\n",
+    "except Exception as e:\n",
+    "    print(\"Falling back to CPU wheel:\", e)\n",
+    "    subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                           \"llama-cpp-python>=0.2.90\", \"huggingface_hub>=0.23.0\",\n",
+    "                           \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "\n",
+    "print(\"✅ Installation complete\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f44766ae",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔐 (Optional) Hugging Face login\n",
+    "# Enter your HF token if the repo is gated (skip if public)\n",
+    "HF_TOKEN = \"\"  #@param {type:\"string\"}\n",
+    "from huggingface_hub import login\n",
+    "if HF_TOKEN:\n",
+    "    login(HF_TOKEN, add_to_git_credential=True)\n",
+    "else:\n",
+    "    print(\"Skipping login (public repos should work without a token)\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f7008eb8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 📦 Choose model & file (GGUF)\n",
+    "from huggingface_hub import hf_hub_download\n",
+    "\n",
+    "REPO_ID = \"TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF\"  #@param [\"TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF\"] {allow-input: true}\n",
+    "FILENAME = \"tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf\"  #@param [\"tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf\"] {allow-input: true}\n",
+    "\n",
+    "print(\"Downloading:\", REPO_ID, FILENAME)\n",
+    "model_path = hf_hub_download(repo_id=REPO_ID, filename=FILENAME, local_dir=\"models\", local_dir_use_symlinks=False)\n",
+    "print(\"Saved to:\", model_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b49d39a5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⚙️ Load model (GPU if available)\n",
+    "from llama_cpp import Llama\n",
+    "\n",
+    "# Auto-detect context length from filename hints; default 4096\n",
+    "n_ctx = 4096\n",
+    "llm = Llama(\n",
+    "    model_path=model_path,\n",
+    "    n_gpu_layers=-1,   # Use GPU if available, otherwise CPU\n",
+    "    n_ctx=n_ctx,\n",
+    "    logits_all=False,\n",
+    "    verbose=False,\n",
+    ")\n",
+    "print(\"✅ Model loaded\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2aac08c3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🗣️ Chat (single turn)\n",
+    "system_prompt = \"You are a helpful, polite assistant.\"\n",
+    "user_prompt = \"Explain transformers in simple terms.\"  #@param {type:\"string\"}\n",
+    "max_tokens = 512  #@param {type:\"slider\", min:32, max:2048, step:32}\n",
+    "temperature = 0.7  #@param {type:\"number\"}\n",
+    "\n",
+    "prompt = f\"<|system|>\\n{system_prompt}\\n<|user|>\\n{user_prompt}\\n<|assistant|>\\n\"\n",
+    "out = llm(\n",
+    "    prompt,\n",
+    "    max_tokens=max_tokens,\n",
+    "    temperature=temperature,\n",
+    "    stop=[\"<|user|>\", \"<|system|>\", \"</s>\"]\n",
+    ")\n",
+    "print(out[\"choices\"][0][\"text\"].strip())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7494f779",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔁 Chat Loop (enter queries in the textbox)\n",
+    "import ipywidgets as widgets\n",
+    "from IPython.display import display, Markdown\n",
+    "\n",
+    "system_prompt = widgets.Textarea(\n",
+    "    value=\"You are a helpful, polite assistant.\",\n",
+    "    description='System:',\n",
+    "    layout=widgets.Layout(width='100%', height='80px')\n",
+    ")\n",
+    "\n",
+    "user_box = widgets.Textarea(\n",
+    "    value=\"Write a Python function to check prime numbers.\",\n",
+    "    description='User:',\n",
+    "    layout=widgets.Layout(width='100%', height='100px')\n",
+    ")\n",
+    "\n",
+    "temperature = widgets.FloatSlider(value=0.7, min=0.0, max=1.5, step=0.05, description='Temp')\n",
+    "max_tokens = widgets.IntSlider(value=512, min=32, max=2048, step=32, description='Max tokens')\n",
+    "\n",
+    "run_button = widgets.Button(description=\"Generate\", button_style='success')\n",
+    "\n",
+    "out_area = widgets.Output()\n",
+    "\n",
+    "def on_click(_):\n",
+    "    with out_area:\n",
+    "        out_area.clear_output()\n",
+    "        prompt = f\"<|system|>\\n{system_prompt.value}\\n<|user|>\\n{user_box.value}\\n<|assistant|>\\n\"\n",
+    "        result = llm(prompt, max_tokens=max_tokens.value, temperature=temperature.value, stop=[\"<|user|>\", \"<|system|>\", \"</s>\"])\n",
+    "        display(Markdown(result[\"choices\"][0][\"text\"].strip()))\n",
+    "\n",
+    "run_button.on_click(on_click)\n",
+    "\n",
+    "ui = widgets.VBox([system_prompt, user_box, temperature, max_tokens, run_button, out_area])\n",
+    "display(ui)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dda17cdf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🌐 Optional: Start a local API server (OpenAI-compatible-ish)\n",
+    "# After running, you can access http://127.0.0.1:8000/docs inside Colab.\n",
+    "import threading\n",
+    "from llama_cpp.server.app import create_app\n",
+    "from fastapi.middleware.cors import CORSMiddleware\n",
+    "import uvicorn\n",
+    "\n",
+    "app = create_app(llm)\n",
+    "app.add_middleware(\n",
+    "    CORSMiddleware,\n",
+    "    allow_origins=[\"*\"],\n",
+    "    allow_credentials=True,\n",
+    "    allow_methods=[\"*\"],\n",
+    "    allow_headers=[\"*\"],\n",
+    ")\n",
+    "\n",
+    "def run_server():\n",
+    "    uvicorn.run(app, host=\"0.0.0.0\", port=8000, log_level=\"info\")\n",
+    "\n",
+    "thread = threading.Thread(target=run_server, daemon=True)\n",
+    "thread.start()\n",
+    "print(\"Server starting on http://127.0.0.1:8000\")"
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

co-lab/zephyr_7b_beta_gguf,_q4_k_m.ipynb

@@ -0,0 +1,217 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "aec8642b",
+   "metadata": {},
+   "source": [
+    "# Zephyr 7B Beta (GGUF, Q4_K_M)\n",
+    "**One-click Colab notebook** to run `TheBloke/zephyr-7B-beta-GGUF` (file: `zephyr-7b-beta.Q4_K_M.gguf`) using **llama-cpp-python** on GPU (T4/A100) or CPU.\n",
+    "\n",
+    "**Features**: Hugging Face login, GGUF download, fast GPU inference, chat UI cell, optional API server.\n",
+    "\n",
+    "\n",
+    "---\n",
+    "**Tip:** If you're on free Colab, switch to a GPU runtime: **Runtime → Change runtime type → T4 GPU**.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0ecdac0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔧 Check GPU and Python version\n",
+    "!nvidia-smi || echo \"No NVIDIA GPU available\"\n",
+    "!python --version"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1f7b92f8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⬇️ Install dependencies (GPU build if available)\n",
+    "# If you get build errors, re-run this cell.\n",
+    "import os, sys, subprocess\n",
+    "\n",
+    "cuda_spec = \"cu121\"\n",
+    "wheels_index = \"https://abetlen.github.io/llama-cpp-python/whl/\" + cuda_spec\n",
+    "try:\n",
+    "    # Try GPU wheel first\n",
+    "    exitcode = subprocess.call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                                f\"--extra-index-url={wheels_index}\", \"llama-cpp-python>=0.2.90\",\n",
+    "                                \"huggingface_hub>=0.23.0\", \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "    if exitcode != 0:\n",
+    "        raise RuntimeError(\"GPU wheel failed, falling back to CPU wheel\")\n",
+    "except Exception as e:\n",
+    "    print(\"Falling back to CPU wheel:\", e)\n",
+    "    subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-qU\",\n",
+    "                           \"llama-cpp-python>=0.2.90\", \"huggingface_hub>=0.23.0\",\n",
+    "                           \"ipywidgets\", \"pydantic<3\", \"uvicorn\", \"fastapi\"])\n",
+    "\n",
+    "print(\"✅ Installation complete\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "141768d1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔐 (Optional) Hugging Face login\n",
+    "# Enter your HF token if the repo is gated (skip if public)\n",
+    "HF_TOKEN = \"\"  #@param {type:\"string\"}\n",
+    "from huggingface_hub import login\n",
+    "if HF_TOKEN:\n",
+    "    login(HF_TOKEN, add_to_git_credential=True)\n",
+    "else:\n",
+    "    print(\"Skipping login (public repos should work without a token)\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "362ebcb2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 📦 Choose model & file (GGUF)\n",
+    "from huggingface_hub import hf_hub_download\n",
+    "\n",
+    "REPO_ID = \"TheBloke/zephyr-7B-beta-GGUF\"  #@param [\"TheBloke/zephyr-7B-beta-GGUF\"] {allow-input: true}\n",
+    "FILENAME = \"zephyr-7b-beta.Q4_K_M.gguf\"  #@param [\"zephyr-7b-beta.Q4_K_M.gguf\"] {allow-input: true}\n",
+    "\n",
+    "print(\"Downloading:\", REPO_ID, FILENAME)\n",
+    "model_path = hf_hub_download(repo_id=REPO_ID, filename=FILENAME, local_dir=\"models\", local_dir_use_symlinks=False)\n",
+    "print(\"Saved to:\", model_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "71308945",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title ⚙️ Load model (GPU if available)\n",
+    "from llama_cpp import Llama\n",
+    "\n",
+    "# Auto-detect context length from filename hints; default 4096\n",
+    "n_ctx = 4096\n",
+    "llm = Llama(\n",
+    "    model_path=model_path,\n",
+    "    n_gpu_layers=-1,   # Use GPU if available, otherwise CPU\n",
+    "    n_ctx=n_ctx,\n",
+    "    logits_all=False,\n",
+    "    verbose=False,\n",
+    ")\n",
+    "print(\"✅ Model loaded\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "887f05da",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🗣️ Chat (single turn)\n",
+    "system_prompt = \"You are a helpful, polite assistant.\"\n",
+    "user_prompt = \"Explain transformers in simple terms.\"  #@param {type:\"string\"}\n",
+    "max_tokens = 512  #@param {type:\"slider\", min:32, max:2048, step:32}\n",
+    "temperature = 0.7  #@param {type:\"number\"}\n",
+    "\n",
+    "prompt = f\"<|system|>\\n{system_prompt}\\n<|user|>\\n{user_prompt}\\n<|assistant|>\\n\"\n",
+    "out = llm(\n",
+    "    prompt,\n",
+    "    max_tokens=max_tokens,\n",
+    "    temperature=temperature,\n",
+    "    stop=[\"<|user|>\", \"<|system|>\", \"</s>\"]\n",
+    ")\n",
+    "print(out[\"choices\"][0][\"text\"].strip())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ca60e43d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🔁 Chat Loop (enter queries in the textbox)\n",
+    "import ipywidgets as widgets\n",
+    "from IPython.display import display, Markdown\n",
+    "\n",
+    "system_prompt = widgets.Textarea(\n",
+    "    value=\"You are a helpful, polite assistant.\",\n",
+    "    description='System:',\n",
+    "    layout=widgets.Layout(width='100%', height='80px')\n",
+    ")\n",
+    "\n",
+    "user_box = widgets.Textarea(\n",
+    "    value=\"Write a Python function to check prime numbers.\",\n",
+    "    description='User:',\n",
+    "    layout=widgets.Layout(width='100%', height='100px')\n",
+    ")\n",
+    "\n",
+    "temperature = widgets.FloatSlider(value=0.7, min=0.0, max=1.5, step=0.05, description='Temp')\n",
+    "max_tokens = widgets.IntSlider(value=512, min=32, max=2048, step=32, description='Max tokens')\n",
+    "\n",
+    "run_button = widgets.Button(description=\"Generate\", button_style='success')\n",
+    "\n",
+    "out_area = widgets.Output()\n",
+    "\n",
+    "def on_click(_):\n",
+    "    with out_area:\n",
+    "        out_area.clear_output()\n",
+    "        prompt = f\"<|system|>\\n{system_prompt.value}\\n<|user|>\\n{user_box.value}\\n<|assistant|>\\n\"\n",
+    "        result = llm(prompt, max_tokens=max_tokens.value, temperature=temperature.value, stop=[\"<|user|>\", \"<|system|>\", \"</s>\"])\n",
+    "        display(Markdown(result[\"choices\"][0][\"text\"].strip()))\n",
+    "\n",
+    "run_button.on_click(on_click)\n",
+    "\n",
+    "ui = widgets.VBox([system_prompt, user_box, temperature, max_tokens, run_button, out_area])\n",
+    "display(ui)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f832998a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#@title 🌐 Optional: Start a local API server (OpenAI-compatible-ish)\n",
+    "# After running, you can access http://127.0.0.1:8000/docs inside Colab.\n",
+    "import threading\n",
+    "from llama_cpp.server.app import create_app\n",
+    "from fastapi.middleware.cors import CORSMiddleware\n",
+    "import uvicorn\n",
+    "\n",
+    "app = create_app(llm)\n",
+    "app.add_middleware(\n",
+    "    CORSMiddleware,\n",
+    "    allow_origins=[\"*\"],\n",
+    "    allow_credentials=True,\n",
+    "    allow_methods=[\"*\"],\n",
+    "    allow_headers=[\"*\"],\n",
+    ")\n",
+    "\n",
+    "def run_server():\n",
+    "    uvicorn.run(app, host=\"0.0.0.0\", port=8000, log_level=\"info\")\n",
+    "\n",
+    "thread = threading.Thread(target=run_server, daemon=True)\n",
+    "thread.start()\n",
+    "print(\"Server starting on http://127.0.0.1:8000\")"
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}