leeroy-jankins/buddy

buddy is a quantized GGUF deployment artifact used by the buddy application. The application source indicates that this model is loaded locally through llama-cpp-python and is based on google/gemma-3-270m-it.

Please use the correct settings: temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0

This repository is intended to host the GGUF model file used by the application, while the main application code is hosted separately on GitHub:

• Main application repository: https://github.com/is-leeroy-jenkins/buddy.git
• Model file: gemma-3-270m-it-Q4_K_M.gguf
• Base model: google/gemma-3-270m-it
• Primary runtime for this artifact: llama.cpp / llama-cpp-python
• Role in the application: local fallback model for text generation when the primary provider is unavailable or when a local path is preferred

⚙️ Code Respository

🧰 Streamlit UI

Within that architecture, buddy serves as a lightweight local option for text generation when a local model is preferred or when the primary remote provider path is not being used.

The application configuration points to the following default local model path:

buddy/gemma-3-270m-it-Q4_K_M.gguf

The runtime loads the model through Llama(...) from llama_cpp, and the application defaults to a 4096-token context window for local inference.

Base Model Description

Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. Gemma 3 models are multimodal, handling text and image input and generating text output, with open weights for both pre-trained variants and instruction-tuned variants. Gemma 3 has a large, 128K context window, multilingual support in over 140 languages, and is available in more sizes than previous versions. Gemma 3 models are well-suited for a variety of text generation and image understanding tasks, including question answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as laptops, desktops or your own cloud infrastructure, democratizing access to state of the art AI models and helping foster innovation for everyone.

Inputs and outputs

• Input:

  • Text string, such as a question, a prompt, or a document to be summarized
  • Images, normalized to 896 x 896 resolution and encoded to 256 tokens each
  • Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and 32K tokens for the 1B and 270M sizes.

• Output:

  • Generated text in response to the input, such as an answer to a question, analysis of image content, or a summary of a document
  • Total output context up to 128K tokens for the 4B, 12B, and 27B sizes, and 32K tokens for the 1B and 270M sizes per request, subtracting the request input tokens

Intended Usage

This model is intended for local inference in GGUF-compatible runtimes, especially:

• llama.cpp
• llama-cpp-python
• desktop applications or Streamlit applications that load GGUF models directly

In the context of buddy, this repository should be understood as the model-hosting companion to the main GitHub application repository rather than as the complete application itself.

Typical usage scenarios include:

• lightweight local assistants
• document-grounded Q&A
• prompt-based drafting and summarization
• local fallback inference for a larger multimodal GPT-5.x application
• experimentation with small-footprint local Gemma-family deployments

Example: llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="buddy/gemma-3-270m-it-Q4_K_M.gguf",
    n_ctx=4096,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain what this model is for."},
    ]
)

print(response["choices"][0]["message"]["content"])

Example: llama.cpp

./llama-cli \
  -m buddy\gemma-3-270m-it-Q4_K_M.gguf\
  -c 4096 \
  -p "Write a short description of buddy."

Prompting Notes

Because this file is based on an instruction-tuned Gemma-family model, best results generally come from:

• clear task-oriented prompts
• concise system instructions
• grounded context when using RAG
• short to moderate generations for factual tasks

For document Q&A workflows, pair the model with retrieved context rather than relying on parametric memory alone.

Quantization Notes

This repository hosts a GGUF quantized artifact rather than an original full-precision checkpoint. The file name indicates a Q4_K_M quantization variant.

Quantization typically reduces model size and memory requirements, making local inference easier on consumer hardware, but it may also reduce generation quality relative to higher-precision variants.

Provenance and What Is Known

The following points are supported by the buddy source files:

• the application expects a model file named llm/gemma-3-270m-it-Q4_K_M.gguf
• the configured model path is buddy/gemma-3-270m-it-Q4_K_M.gguf
• inference is performed through llama-cpp-python
• the application uses a 4096-token context window by default
• the application combines local text generation with embedding-based retrieval components

⚙️ Fine-Tuning & Training Datasets

File Name	Description
Balanced Budget and Emergency Deficit Control Act of 1985	Establishes statutory limits on federal spending and deficit control mechanisms, including sequestration procedures.
Budget Control Act of 2011	Sets discretionary spending caps and establishes enforcement mechanisms to control federal deficits.
Digital Accountability And Transparency Act of 2014	Requires standardized federal spending data and improved transparency through government-wide financial reporting.
Federal Account Symbols And Titles Book	Defines Treasury account symbols and official titles used for federal budgetary and accounting purposes.
Federal Acquisition Regulation	Establishes uniform policies and procedures governing the acquisition of goods and services by federal agencies.
Federal Government Standards For Internal Controls	Defines the internal control framework for federal agencies to ensure accountability, integrity, and compliance.
Federal Managers Financial Integrity Act of 1982	Requires agencies to establish internal controls and report annually on their effectiveness.
Federal Trust Fund Accounting Guide	Provides accounting guidance for the management and reporting of federal trust funds.
Financial Management Regulations DOD 7000-14-R	Establishes DoD-specific financial management policies, procedures, and accounting requirements.
Fiscal Responsibility Act	Establishes statutory measures intended to improve fiscal discipline and control federal spending.
Government Auditing Standards	Sets professional standards for audits of government organizations, programs, activities, and functions.
Government Invoicing User Guide	Provides guidance on federal invoicing standards and processes for government transactions.
Government Performance and Results Act of 1993	Requires agencies to engage in strategic planning and performance measurement to improve program effectiveness.
GPRA Modernization Act of 2010	Updates GPRA by strengthening performance management, cross-agency goals, and accountability.
OMB Circular A-11 Preparation Submission And Execution Of The Budget	Provides comprehensive guidance for preparing, submitting, and executing the President’s Budget.
OMB Circular A-11 Section 120 Apportionment Process	Defines the apportionment process used to control the rate of obligation of budgetary resources.
OMB Circular A-123 Managements Responsibility for Enterprise Risk Management and Internal Control	Defines management responsibilities for internal control and enterprise risk management across federal agencies.
Federal Trust Fund Accounting Guide	Establishes requirements for federal agency financial statements and reporting.
Principles Of Federal Appropriations Law Volume One	Authoritative GAO guidance on foundational principles governing the use of federal appropriations.
Statements of Federal Federal Financial Accounting Concepts and Standards	Establishes accounting concepts and standards for federal financial reporting.
The Anti-Deficiency Act PL 97-258	Prohibits federal agencies from obligating or expending funds in excess of appropriations or before enactment.
The Anti-Deficiency Reform and Enforcement Act of 2018	Strengthens Anti-Deficiency Act enforcement and reporting requirements to improve fiscal accountability.
The Chief Financial Officers Act of 1990	Establishes agency Chief Financial Officers and modernizes federal financial management practices.
The Congressional Budget and Impoundment Control Act of 1974	Establishes the congressional budget process and restricts executive impoundment of appropriated funds.
Statutory Pay As You Go Act of 2010	Authorizes interagency agreements for the provision of goods and services on a reimbursable basis.
The Stafford Act	Provides the statutory framework for federal disaster response and emergency assistance.
Federal Trust Fund Accounting Guide	Provides additional appropriations authority beyond regular annual funding acts.
Title 2 Code of Federal Regulations – Uniform Administrative Requirements, Cost Principles, and Audit	Establishes uniform administrative, cost, and audit requirements for federal financial assistance.
Title 31 Code of Federal Regulations – Money and Finance	Codifies Treasury and federal financial management regulations governing money and finance.
US Standard General Ledger Account Definitions	Defines standardized account structures used for federal accounting and financial reporting.

If you have those details, they should be added explicitly in a later revision.

Limitations

As a small quantized instruction model, this artifact may:

• hallucinate facts
• struggle with long multi-step reasoning
• lose fidelity on highly technical or domain-dense tasks
• perform worse than larger or less aggressively quantized models
• require careful retrieval support for document-heavy workflows

It should be treated as an assistive generation component, not as an authoritative source.

Safety and Responsible Use

Users should review outputs before acting on them, especially for:

• legal matters
• financial decisions
• medical or health-related questions
• employment or compliance workflows
• any task requiring high factual precision

Do not rely on model output as a substitute for professional judgment or verified source material.

Hardware Considerations

Because this is a small GGUF quantized model, it is suitable for lightweight local inference relative to larger checkpoints. Actual performance will depend on:

• runtime configuration
• CPU versus GPU offloading
• available RAM / VRAM
• context length
• batch size and thread settings

License and Upstream Terms

This artifact is based on google/gemma-3-270m-it. Use of this repository and any redistributed artifacts should comply with:

• the license and usage terms attached to the upstream Gemma model
• any additional redistribution requirements that apply to converted or quantized derivatives

Before publishing, confirm that your intended distribution of the GGUF file is consistent with the applicable upstream license terms.

Files

This repository is expected to contain:

README.md
buddy/gemma-3-270m-it-Q4_K_M.gguf

Relationship to the Main buddy Repository

This repository is best used alongside the main buddy codebase hosted on GitHub:

https://github.com/is-leeroy-jenkins/buddy.git

The GitHub repository contains the application logic and user-facing features, while this repository is intended to host the GGUF model artifact used for local fallback inference.

Recommended Repository Description

A GGUF-hosted quantized local fallback model for the buddy application, based on google/gemma-3-270m-it, intended for local inference with llama.cpp / llama-cpp-python, and used alongside the main buddy application hosted on GitHub.

Acknowledgments

• Google for the upstream Gemma model family
• The llama.cpp and llama-cpp-python communities for GGUF-compatible local inference tooling
• The buddy application source, which documents how this model is loaded and used in practice}