docs/Explanations.md

<!-- ======================================================== -->
## Table Of Contents
<!-- ======================================================== -->

1. [Explanations](#1-explanations)
2. [System Architecture](#2-system-architecture)
   1. [System Model](#system-model)
   2. [Package Layout](#package-layout)
   3. [Prompt And Resource Ownership](#prompt-and-resource-ownership)
3. [Configuration And Dependency Model](#3-configuration-and-dependency-model)
   1. [Configuration Model](#configuration-model)
   2. [Dependency Model](#dependency-model)
4. [Workflow Models](#4-workflow-models)
   1. [Manifest-To-Export Model](#manifest-to-export-model)
   2. [Caption Sidecar Model](#caption-sidecar-model)
   3. [LoRA Workflow Model](#lora-workflow-model)
   4. [Staged T2I Pipeline Model](#staged-t2i-pipeline-model)
   5. [Image Utility Model](#image-utility-model)
5. [Failure Model](#5-failure-model)

<br>

# 1. Explanations

Use this file when you need to understand why `kneifftools` is shaped the way it
is. Use [How-To User Guides](How-To-User-Guides.md) for commands and
[References](References.md) for exact names.

<br>

# 2. System Architecture

<!-- ======================================================== -->
## System Model
<!-- ======================================================== -->

`kneifftools` is a src-layout Python project. Runtime code lives under
[src/kneiff](../src/kneiff), maintainer-only helpers live under
[src/kneiff_dev](../src/kneiff_dev), tests live under [tests](../tests), and
long-form documentation lives under [docs](.).

The main idea is simple:

1. `pyproject.toml` describes the package, dependencies, tools, and entry
   points.
2. `src/kneiff` owns importable runtime behavior.
3. `tests` verifies behavior from the outside where possible.
4. `docs` explains setup, workflows, exact names, and design context.
5. `justfile` provides repeatable maintainer commands.

> [!NOTE]
> Related links:
> - Use [install the package](How-To-User-Guides.md#install-the-package) for the first setup path.
> - Use [project paths](References.md#project-paths) for exact source owners.
> - Use [repository routing](Development.md#repository-routing) before moving behavior.

<br>

<!-- ======================================================== -->
## Package Layout
<!-- ======================================================== -->

The project uses a `src` layout so imports come from the installed package
rather than accidentally from the repository root.

Core boundaries:

| Area | Responsibility |
|---|---|
| `src/kneiff` | Runtime behavior and package-owned helpers. |
| `src/kneiff_dev` | Maintainer-only helpers. |
| `tests` | Behavior checks, fixtures, and regression tests. |
| `docs` | Longer usage, reference, and architecture material. |
| `docs/assets` | Tracked generated documentation figures. |
| `assets/*.IGNORE*` | Ignored local diagram scratch files. |
| `experiments/` | Ignored local research and one-off probes. |

Keep broadly reusable helpers near the package area that owns the domain. Keep
one-off diagnostics in the nearest existing tooling, test, or experiment area.

> [!NOTE]
> Related: use [Development: source editing rules](Development.md#source-editing-rules)
> for implementation rules that preserve these boundaries.

<br>

<!-- ======================================================== -->
## Prompt And Resource Ownership
<!-- ======================================================== -->

Prompt catalog parsing does not select a project or import an inference or
training backend. The dependency direction is deliberately one-way:

```text
project_resources ──> prompts.catalog
        │
        └───────────> comfy.workflow_presets ──> prompts.catalog

training.lora ──────> project_resources + prompts.catalog
comfy.lora ─────────> no prompt or showcase modules
```

`kneiff.prompts.catalog.load_prompt_catalog()` receives a vocabulary and an
optional overlay path. `kneiff.project_resources.load_project_prompt_catalog()`
owns project path validation and vocabulary loading. SimpleTuner artifact
builders receive a resolved catalog instead of reaching back into project
selection. Generic ComfyUI LoRA resolution remains independent of showcase
request types so other pipelines can reuse it without importing prompt code.

<br>

# 3. Configuration And Dependency Model

<!-- ======================================================== -->
## Configuration Model
<!-- ======================================================== -->

Kneiff uses explicit files for workflow state:

- The AppRC application name is `knf`. `~/.config/knf/knf.apprc.toml` owns its
  registry of persistent storage roots; `KNF_APPRC_TOML` can point Kneiff at a
  different registry path.
- `~/.config/knf/.env.apprc-app` owns app-wide Kneiff overrides.
- `vocabulary.knf.yaml` owns project identity tokens, controlled caption axes,
  and project-specific vocabulary layered over Kneiff's reusable JTP axes.
- Kneiff's packaged core prompt catalog owns reusable prompt defaults,
  `character_reference`, and fixed validation controls. `prompts.knf.yaml` is
  the optional project overlay for scenes and project-specific prompt choices.
  The resolved catalog routes rows through explicit `uses`; Kneiff does not
  infer showcase eligibility.
- `workflows/showcase_<preset>.workflow.json` optionally overrides a packaged
  showcase workflow for one project.
 - `configs/*.knf.yaml` owns export and optional SimpleTuner training settings.
  `knf project init` always creates `configs/ANIMA.knf.yaml`, the Sygred Anima
  training template with local component paths, and `configs/F2K_9B.knf.yaml`,
  the Flux2 Klein 9B template using the official base repository and unset
  component paths. The Flux2 config uses Sygred's 320-step profile with a
  fullbody/genitals warm-up; the Anima config retains its small-run settings.
- `SOURCE/` owns source images scanned by dataset sync. A new project includes
  `0-FULLBODY/` and `2-HEAD/`; add an anatomy-focused folder only when a
  project config maps it.
- `MANIFEST.knf.xlsx` owns the editable dataset index with paths relative to
  `SOURCE/`; generated `MANIFEST.yaml` sits beside it for Git review.
- `HF/<config-id>/` owns public Hugging Face dataset exports. The root `HF/`
  directory is created empty by the scaffold and ignored by Git.
- `TRAINING/<config-id>_<run>/` owns SimpleTuner run state. The root
  `TRAINING/` directory is created empty by the scaffold and ignored by Git.
- `.old_manifests/` owns archived manifest workbooks.
- `.env.apprc-storage` owns machine-local AppRC overrides such as
  `KNF_WORKERS`, `COMFY_MODELS_DIR` for ComfyUI showcase LoRA discovery,
  `COMFY_LORAS_DIR_1` for the interactive picker start directory, and
  `COMFY_UPSCALE_MODEL` for server-side ComfyUI upscaling.
- `.git/`, `.gitignore`, and `.gitattributes` own the new project's local Git
  repository and generic ignore/LFS tracking policy. The scaffold initializes
  a `main` branch and local Git user name only for a new repository; it never
  creates a commit, configures a remote or email, alters global Git settings,
  or installs Git LFS.
- `default_tags.txt` is a project-owned seed containing `kneiff` plus the
  project's activation and species tokens. It is intentionally not runtime
  input to Kneiff.
- `.env` owns local OpenAI-compatible caption server settings.

Resource ownership follows semantic scope, not which project first needed a
resource. A caption tag or model-family workflow stays packaged when unrelated
projects can reuse its meaning unchanged. Identity tokens, project-only axes,
curated prompts, and workflows containing project-specific choices belong in
the selected storage root. The default Anima config follows the same boundary:
it uses a generic model/training recipe and packaged validation controls, while
the root vocabulary and optional prompt overlay supply project identity and
scenes. No Rook prompt, identity, local model path, tracker, or publishing
setting is copied into a new project.

AppRC constructs one typed `KneiffConfig` bundle from native `StorageConfig`,
`ComfyConfig`, `LmStudioConfig`, and `PromptgenConfig` sections. Runtimeful
commands require an explicit storage selector through `KNF_STORAGE` or the root
`--storage` option. A lone registry entry is not selected implicitly. The
selected AppRC storage root is the project boundary, and Kneiff derives every
project path from the fixed scaffold above. `.env.apprc-storage` cannot select
that root because AppRC must resolve the selector before it knows which local
file to load. `knf project use` writes an app-wide fallback; shell `KNF_STORAGE`
and root `--storage` override it.

Kneiff does not use a second project descriptor such as
`kneiff.project.yaml`. Such a file would duplicate the storage selector and
fixed path convention without adding a separate source of truth.

Explicit config paths must remain inside the selected project root. Rejecting
cross-project paths prevents one project's config from silently using another
project's vocabulary or prompts.

When configuration affects a user-visible workflow, update
[How-To User Guides](How-To-User-Guides.md) and [References](References.md)
together.

> [!NOTE]
> Related: use [environment variables](References.md#environment-variables) for
> exact variable names and [configuration files](References.md#configuration-files)
> for file owners.

<br>

<!-- ======================================================== -->
## Dependency Model
<!-- ======================================================== -->

The project separates dependency types by audience:

| Dependency Type | Owner | Audience |
|---|---|---|
| Runtime dependency | `[project].dependencies` | Users who install the package. |
| Dependency group | `[dependency-groups]` | Maintainers who run tests, typing, linting, docs, or profiling. |
| Lock file | `uv.lock` | Maintainers or CI jobs that want reproducible `uv` syncs. |

This split keeps normal installs usable without requiring `uv`, while leaving
maintainer workflows repeatable.

> [!NOTE]
> Related links:
> - Use [install the package](How-To-User-Guides.md#install-the-package) for install commands.
> - Use [dependency surfaces](References.md#dependency-surfaces) for exact `pyproject.toml` sections.

<br>

# 4. Workflow Models

<!-- ======================================================== -->
## Manifest-To-Export Model
<!-- ======================================================== -->

Kneiff treats the source image folder as the source of truth and the export
folder as disposable output.

After AppRC selects a storage root, config discovery starts from that root. A
config selector such as `chroma` matches files like
`configs/CHROMA-2.knf.yaml` case-insensitively and only auto-selects when the
result is unambiguous. Absolute config paths are valid selection inputs, but do
not bypass the required AppRC storage root and must stay inside it.

The dataset path is:

1. `knf dataset sync configs/example.knf.yaml` or `knf dataset sync example`
   scans image files under `SOURCE/`.
2. `MANIFEST.knf.xlsx` stores one fixed 13-row block per image. Column C owns
   user input; the merged columns D-H show `tag`, `json`, `nlg`, `chroma`, and
   `prose` renderings from one shared Kneifftags analysis.
3. Sync rewrites schema-version-2 `MANIFEST.yaml` with logical identity and
   user input but no derived captions.
4. Manual annotation fills the fixed semantic fields and the `SFW` control.
5. `configs/*.knf.yaml` maps `SOURCE`-relative folders to export subsets.
6. Export planning selects rows, reuses their analysis, optionally resizes images,
   and writes a public dataset under `HF/<config-id>/`.

The config filename selects the dataset id; config content does not own paths.
Kneiff rejects `source_root`, `manifest_path`, `export_root`, and
`allow_export_inside_source` so a config cannot redirect reads or writes outside
the selected AppRC storage.

Dataset sync is dataset-only. LoRA training state is generated later by
`knf train prepare` or fresh `knf train start` under numbered
`TRAINING/<config-id>_<run>/` workspaces, so manifest sync does not need
underscore-prefixed output guards.

> [!NOTE]
> Related links:
> - Use [sync a manifest workbook](How-To-User-Guides.md#sync-a-manifest-workbook) for the command sequence.
> - Use [configuration files](References.md#configuration-files) for exact file names.

<br>

<!-- ======================================================== -->
## Caption Sidecar Model
<!-- ======================================================== -->

Captions are rendered from the structured values in each fixed image block.
This gives workbook previews, validation prompts, ComfyUI prompts, and export
sidecars one semantic source.

The main pieces are:

| Piece | Role |
|---|---|
| Kneifftags built-in vocabulary | Owns reusable categories, groups, aliases, conflicts, output tags, and profile prose. |
| `vocabulary.knf.yaml` | Strict schema-2 extension for project identities and project-only vocabulary. |
| `caption_outputs` | Sets global sidecar mode, formats, and tag scope. |
| `caption_outputs_overrides` | Changes sidecar output behavior for one subset. |
| `caption` | Supplies an optional project-wide `subject_sex` fallback. |

Kneifftools gathers the fields in block order, omits `SFW` and `Notes`, and
calls the selected project engine once. The immutable analysis contains
resolved definitions, category membership, unknown inputs, ambiguities,
conflicts, and diagnostics. All output profiles reuse that result; no consumer
parses the row a second time.

Unknown tags are preserved through Kneifftags fallback rendering. Warnings do
not discard input, while active engine errors block activation or export. A row
marked `SFW` receives an additional application-level safety check because
`SFW` is workflow control rather than caption content.

The project extension is composed with the packaged Kneifftags vocabulary in an
isolated engine. Character and species defaults use namespace-qualified inputs
to prevent collisions with built-in spellings. Kneifftools reads vocabulary
definitions through the public root API for prompt and field catalogs; it does
not inspect Kneifftags internals.

`hybrid_txt` writes one `.txt` sidecar for each image variant. `separate_txt`
writes separate sidecars for selected formats. The canonical format names are
`tags`, `natural`, `json`, `nlg`, `chroma`, and `hybrid`. Removed `tag` and
`prose` aliases are rejected at the config and prompt-catalog boundaries.

`tag_scope: supplemental` omits tag facts already expressed by natural text
inside a joined caption; `tag_scope: all` keeps the complete tag tail.
Standalone `tags` and `chroma` files use all tags. Mirrored Chroma sidecars
receive the application-owned `variant.` prefix. Caption-text compatibility
does not make LoRA weights cross-model-compatible.

> [!NOTE]
> Related: use [configure caption outputs](How-To-User-Guides.md#configure-caption-outputs)
> for the config shape.

<br>

<!-- ======================================================== -->
## LoRA Workflow Model
<!-- ======================================================== -->

LoRA training is layered on top of dataset sync. The same `configs/*.knf.yaml` file
owns both the export plan and the optional `training.simpletuner` block.

The LoRA path is:

1. Resolve the exported dataset root from `HF/<config-id>/`. This root must
   already be populated by `knf dataset sync`; training commands validate it
   rather than creating or refreshing it.
2. Resolve the run workspace as `TRAINING/<config-id>_<run>/`; fresh prepare or
   start commands choose the next free run number.
3. Copy configured training subsets into the run workspace's `dataset/` so
   SimpleTuner can write bucket metadata without touching public subset folders.
4. Translate `training.simpletuner` into SimpleTuner JSON artifacts under the
   run workspace and write `kneiff-training-run.json` as `not_started`. Kneiff
   owns fixed names for the trainer JSON, backend JSON, optional validation
   prompt JSON, cache directory, and output directory.
5. Review the generated artifacts before launch, or resume an existing
   `not_started` or `incomplete` run with `knf train start --resume`.
6. Resolve the interpreter declared by the installed `simpletuner` tool, then
   launch Kneiff's copied patch runner in an isolated child process with
   environment variables prepared for the selected CUDA device. Update the run
   state to `running`, `failed`, `incomplete`, or `complete` without importing
   SimpleTuner or mutating process-global state in Kneiff.
7. After training, convert Chroma and Anima LoRA checkpoints to
   ComfyUI-native module names when the trainer asks for ComfyUI format and
   write the final validation contact sheet as
   `TRAINING/<config-id>_<run>-kneiff-validation-progress-grid.jpg`.

A numbered workspace is valid only with its generated
`kneiff-training-run.json` at schema version 1. Persisted paths must be
normalized absolute paths that match the current project and remain below that
workspace. Persisted subset names must exactly match the direct dataset child
directories referenced by active image backends. Before launch, Kneiff verifies
the stored hashes and validates paths and optional prompt content from the
generated JSON itself. Run state retains only metadata deliberately excluded
from the parser JSON; it cannot override disk-owned fields. Kneiff rejects old,
edited, symlinked, or external state instead of guessing paths and rewriting
generated files.

The conversion code fuses split Diffusers-style attention LoRA tensors into
ComfyUI's fused Chroma layer names and renames Diffusers-style Anima attention
adapters to ComfyUI's native Anima blocks while preserving tensor data.

> [!NOTE]
> Related: use [LoRA workflows](How-To-User-Guides.md#4-lora-workflows) for the
> commands and [public interfaces](References.md#public-interfaces) for config names.

<br>

<!-- ======================================================== -->
## Staged T2I Pipeline Model
<!-- ======================================================== -->

Solo and duo are presets over one staged runner. The runner owns job planning,
coordinate-derived seeds, queue ordering, downloads, cancellation, and the run
manifest. The public `t2i_solo.run()` and `t2i_duo.run()` functions only select
the allowed preset shape.

Prompt generation and image generation overlap deliberately. After LM Studio
reviews one variation, all baseline branches for that variation are queued
before the next LM request starts. Duo cleanup is a barrier stage: it begins
only after every Anima baseline has completed. That ordering lets ComfyUI use
one generation model stage followed by one Flux2 edit stage instead of
alternating large models.

The root seed identifies an image experiment, but it does not seed LM Studio.
Each image branch hashes the root seed with its stage and prompt, image, and
cleanup coordinates. Adding later branches therefore cannot renumber earlier
seeds. The manifest preserves sampled raw LM responses and the effective
prompts after exact activation-token repair.

Participant identity remains project-owned. Slot 1 is the canonical
`custom_tokens.character` value; `additional_activation_tokens` supplies later
slots in order. CLI activation-token values affect only one run. They can
override a vocabulary slot or fill a missing slot with a warning, but the
runner never persists them.

Cancellation is based on positive ownership. Each run has a unique ComfyUI
`client_id` and retains every prompt id returned for its jobs. Ctrl-C deletes
only those known ids from the pending queue. It interrupts a running prompt
only after the current queue state confirms that the active id belongs to the
same set. The servers and unrelated OS processes are never stopped.

`run.knf.yaml` is a provenance and partial-failure record, not a checkpoint.
Atomic replacement keeps it readable after process failure, while the absence
of resume support avoids treating server queue state and local partial files as
a recoverable transaction.

<br>

<!-- ======================================================== -->
## Image Utility Model
<!-- ======================================================== -->

Image utilities are CLI wrappers around focused package modules:

| Command Group | Model |
|---|---|
| `png2jpg` | Batch convert PNG files into JPEG files under an output subdirectory. |
| `concat` | Build quick horizontal contact sheets. |
| `rename` | Apply predictable file names before dataset sync. |
| `upscale` | Run image upscaling with selectable model weights and output settings. |
| `tag` | Run RedRocket/JTP-3 tagging over files or directories. |
| `caption` | Generate captions through an OpenAI-compatible server or BLIP/Qwen path. |

These utilities are deliberately narrow. Dataset structure and training sidecar
policy still belong to `knf dataset`, not the image commands.

> [!NOTE]
> Related: use [image workflows](How-To-User-Guides.md#5-image-workflows) for
> command examples.

<br>

# 5. Failure Model

Most failures become easier to debug when checked in this order:

1. Confirm whether the command expects the Kneiff repository root or the
   AppRC-selected project root.
2. Confirm the active Python executable.
3. Confirm `kneiff` imports from the editable install.
4. Confirm dependencies are installed for the workflow.
5. Confirm the command is documented in [References](References.md).
6. Confirm the expected config files and environment variables exist.
7. Re-run the smallest command that reproduces the problem.

```bash
pwd
python -c "import sys; print(sys.executable)"
python -c "import kneiff; print(kneiff.__file__)"
knf --help
```

Dataset-specific failures usually narrow down to one of these:

| Symptom | First Check |
|---|---|
| Missing rows | Confirm `Relative_path` and workbook sheets after `knf dataset sync configs/example.knf.yaml --manifest-only`. |
| Export validation error | Run `knf dataset sync configs/example.knf.yaml --dry-run`. |
| Missing captions | Check populated manual columns and the active `caption_outputs` config. |
| Caption server error | Check `.env`, `BASE_URL`, and `OPENAI_API_KEY`. |
| LoRA export or launch error | Run `knf dataset sync configs/example.knf.yaml`, prepare a new run, then check generated SimpleTuner artifacts and CUDA device options. |

> [!NOTE]
> Related links:
> - Use [environment problems](How-To-User-Guides.md#environment-problems) for import and interpreter checks.
> - Use [command problems](How-To-User-Guides.md#command-problems) when a recipe fails.
> - Use [command reference](References.md#command-reference) for the expected command names.