Spaces:
Build error
Build error
docs: archive current README before Docker deployment rewrite
Browse files- docs/europalex-readme-old.md +222 -0
docs/europalex-readme-old.md
ADDED
|
@@ -0,0 +1,222 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Europa Lex
|
| 2 |
+
|
| 3 |
+
AI-powered flashcard generator for European languages. Generates target-language translations, text-to-speech audio, and illustrative images β exports as a proper `.apkg` file via genanki or a zipped CSV folder with flat media files.
|
| 4 |
+
|
| 5 |
+
> **Note:** All commands and paths in this document are relative to the `EuropaLex/` project root. Assume you are already inside this directory.
|
| 6 |
+
|
| 7 |
+
## Hackathon Criteria
|
| 8 |
+
|
| 9 |
+
- **Off-Brand** β Custom CSS pushes past the default Gradio look; styled card widgets resembling physical flashcards
|
| 10 |
+
- **Llama Champion** β Model runs through llama.cpp runtime locally
|
| 11 |
+
- **Off the Grid** β No cloud APIs, all inference on local models or Modal-hosted endpoints
|
| 12 |
+
- **Sharing is Caring** β Cards exportable as `.apkg`/`.csv`; card datasets shareable via Hugging Face Hub
|
| 13 |
+
|
| 14 |
+
## Setup
|
| 15 |
+
|
| 16 |
+
### Local Development
|
| 17 |
+
|
| 18 |
+
Requires [uv](https://github.com/astral-sh/uv):
|
| 19 |
+
|
| 20 |
+
```bash
|
| 21 |
+
uv sync
|
| 22 |
+
# or install from requirements.txt:
|
| 23 |
+
pip install -r requirements.txt
|
| 24 |
+
```
|
| 25 |
+
|
| 26 |
+
Run the app:
|
| 27 |
+
|
| 28 |
+
```bash
|
| 29 |
+
uv run app.py
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
> **Dependencies:** This project requires PyTorch, diffusers, omnivoice, pydantic, and soundfile in addition to Gradio. These are installed automatically by `uv sync`.
|
| 33 |
+
|
| 34 |
+
### Running Tests
|
| 35 |
+
|
| 36 |
+
All tests use pytest. Run the full suite:
|
| 37 |
+
|
| 38 |
+
```bash
|
| 39 |
+
# Run all tests
|
| 40 |
+
uv run pytest tests/ -v
|
| 41 |
+
|
| 42 |
+
# Run specific test file
|
| 43 |
+
uv run pytest tests/cards_test.py -v
|
| 44 |
+
|
| 45 |
+
# Run with coverage
|
| 46 |
+
uv run pytest tests/ -v --cov=core --cov=frontend --cov=app.py
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
The test suite mocks all GPU/model code β no model weights or GPU required to run tests.
|
| 50 |
+
|
| 51 |
+
### Quick Smoke Check
|
| 52 |
+
|
| 53 |
+
For a quick sanity check before committing:
|
| 54 |
+
|
| 55 |
+
```bash
|
| 56 |
+
uv run pytest tests/smoke_test.py -v
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
This checks imports for core types, engine classes, frontend UI, and the app module. The Gradio app must construct without errors β all widgets are created inside a `gr.Blocks()` context and the context variable is returned (not a fresh empty `Blocks` instance). Generator event handlers use `yield (val1, val2)` not `yield from` to match output component counts.
|
| 60 |
+
|
| 61 |
+
### Model Weights
|
| 62 |
+
|
| 63 |
+
All models are GGUF format, downloaded from Hugging Face Hub at runtime (no git submodules). Each model uses a different runtime:
|
| 64 |
+
|
| 65 |
+
```bash
|
| 66 |
+
# Download all models
|
| 67 |
+
uv run python -m models.download_models
|
| 68 |
+
|
| 69 |
+
# Or download specific models
|
| 70 |
+
uv run python -m models.download_models minicpm tiny_aya # Text generation + translation (~3.2 GB)
|
| 71 |
+
uv run python -m models.download_models omnivoice # TTS only (~945 MB)
|
| 72 |
+
uv run python -m models.download_models flux # Image gen only (~2.6 GB)
|
| 73 |
+
|
| 74 |
+
# Custom output directory
|
| 75 |
+
uv run python -m models.download_models --output-dir ./my-models
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
| Model | HF Hub Repo | GGUF File | Runtime | Params | Size | Role |
|
| 79 |
+
|---|---|---|---|---|---|---|
|
| 80 |
+
| MiniCPM5-1B Q8_0 | [Abiray/MiniCPM5-1B-GGUF](https://huggingface.co/Abiray/MiniCPM5-1B-GGUF) | `minicpm5-1b-Q8_0.gguf` | llama-cpp-python | 1.08 B | ~1.1 GB | English text generation (Phase 1) |
|
| 81 |
+
| tiny-aya-water Q4_K_M | [CohereLabs/tiny-aya-water-GGUF](https://huggingface.co/CohereLabs/tiny-aya-water-GGUF) | `tiny-aya-water-q4_k_m.gguf` | llama-cpp-python | 3.35 B | ~2.1 GB | Translation (active) |
|
| 82 |
+
| OmniVoice Q8_0 (base + tokenizer) | [Serveurperso/OmniVoice-GGUF](https://huggingface.co/Serveurperso/OmniVoice-GGUF) | `omnivoice-base-Q8_0.gguf` + `omnivoice-tokenizer-Q8_0.gguf` | omnivoice.cpp | 0.6 B | ~950 MB | Text-to-speech |
|
| 83 |
+
| FLUX.2-klein 4B Q4_K_M | [unsloth/FLUX.2-klein-4B-GGUF](https://huggingface.co/unsloth/FLUX.2-klein-4B-GGUF) | `flux-2-klein-4b-Q4_K_M.gguf` | ComfyUI-GGUF / diffusers | 4 B | ~2.6 GB | Image generation |
|
| 84 |
+
|
| 85 |
+
> **Note:** Models use different runtimes:
|
| 86 |
+
> - **llama-cpp-python** for MiniCPM5-1B (English text generation) β lazy-load/unload via Python bindings (~1.1 GB RAM)
|
| 87 |
+
> - **llama-cpp-python** for tiny-aya-water (translation) β lazy-load/unload via Python bindings (~2 GB VRAM)
|
| 88 |
+
> - **omnivoice.cpp** for OmniVoice β text-to-speech (C++/GGML port)
|
| 89 |
+
> - **ComfyUI-GGUF / diffusers** for FLUX.2 β image generation (diffusion model)
|
| 90 |
+
>
|
| 91 |
+
### Anki Integration
|
| 92 |
+
|
| 93 |
+
**CSV export:** Click **Download CSV + Media** after Phase 2 completes. The app creates a `.zip` archive containing:
|
| 94 |
+
- `cards.csv` β columns: scenario, cefr_level, target_language, english_text, translated_text, audio_filename, image_filename
|
| 95 |
+
- Media files in a flat folder alongside the CSV (no subfolders)
|
| 96 |
+
- Folder naming: `{scenario_slug}_{CEFR}_{LANG_ABBREV}` (e.g., `ordering_coffee_A2_LV`)
|
| 97 |
+
- Media file naming: `{scenario_slug}_{CEFR}_{LANG_ABBREV}_{card_index}.{ext}` (e.g., `ordering_coffee_A2_LV_0.wav`, `ordering_coffee_A2_LV_1.png`)
|
| 98 |
+
|
| 99 |
+
**Anki `.apkg` export:** Click **Export Anki Cards** after Phase 2 completes. The app creates a proper `.apkg` file using genanki, containing:
|
| 100 |
+
- `collection.anki2` β SQLite database with deck, model, and note definitions
|
| 101 |
+
- `media/` β bundled media files (`.wav`, `.png`) referenced by the notes
|
| 102 |
+
- Deck name: "EuropaLex Flashcards" with custom card styling (rounded images, centered layout)
|
| 103 |
+
- Anki imports this directly via File β Import.
|
| 104 |
+
|
| 105 |
+
## Workflow
|
| 106 |
+
|
| 107 |
+
EuropaLex generates flashcards in two phases: English text first (Phase 1), then translation + media (Phase 2).
|
| 108 |
+
|
| 109 |
+
### Phase 1 β Generate English Text
|
| 110 |
+
|
| 111 |
+
1. Enter a scenario or paste text in the input box
|
| 112 |
+
2. Select a CEFR level (`A0`β`C2`) from the dropdown
|
| 113 |
+
3. Set the batch size with the slider (number of cards to generate)
|
| 114 |
+
4. Click **Generate Text**
|
| 115 |
+
5. The app generates English sentences via MiniCPM5-1B (`MiniCPMTextEngine`, llama-cpp-python, lazy-load/unload)
|
| 116 |
+
6. Cards appear in the gallery with English text on the front and a placeholder on the back
|
| 117 |
+
|
| 118 |
+
> **Note:** Translation is deferred to Phase 2. Phase 1 produces English-only cards.
|
| 119 |
+
|
| 120 |
+
### Phase 2 β Generate Translation + Media
|
| 121 |
+
|
| 122 |
+
1. Select a target language from the **Target Language** dropdown (23 EU languages: Bulgarian, Croatian, Czech, Danish, Dutch, Estonian, Finnish, French, German, Greek, Hungarian, Irish, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish)
|
| 123 |
+
2. After Phase 1 completes, the **Images** and **Audio** toggles become active (unchecked by default)
|
| 124 |
+
3. Toggle on whichever media types you want (images, audio, or both)
|
| 125 |
+
4. Click **Generate Cards**
|
| 126 |
+
5. The app translates via tiny-aya-water (`LlamaCppTextEngine`) with retry validation
|
| 127 |
+
6. If Audio is toggled ON, TTS audio is generated via OmniVoice (`TTSEngine`) with voice design mode
|
| 128 |
+
7. If Images is toggled ON, images are generated via `ImageGenEngine` (diffusers Flux2KleinPipeline)
|
| 129 |
+
8. Cards update: translation moves to the front, English stays on the back; image and audio controls appear alongside translations
|
| 130 |
+
|
| 131 |
+
> **Regenerating Cards:** After Phase 2 completes, changing any parameter (target language, audio/image toggles, or voice) automatically restores the **Generate Cards** button so you can regenerate with new settings without re-running Phase 1.
|
| 132 |
+
|
| 133 |
+
### Export
|
| 134 |
+
|
| 135 |
+
1. Once Phase 2 completes, click **Export CSV + Media** to download a `.zip` file containing the CSV and all media files (flat folder structure)
|
| 136 |
+
2. Click **Export Anki Cards** to download a proper `.apkg` file with bundled media and custom card styling
|
| 137 |
+
3. Import into Anki via File β Import
|
| 138 |
+
|
| 139 |
+
## Architecture
|
| 140 |
+
|
| 141 |
+
EuropaLex is organized into five main modules:
|
| 142 |
+
|
| 143 |
+
| Module | Purpose |
|
| 144 |
+
|---|---|
|
| 145 |
+
| `core/` | Data types (`types.py`), text engines + EnginePool (`engine.py`), TTS (`audio_gen.py`), image gen (`image_gen.py`), sentence extraction & generation helpers (`text_gen.py`), Phase 2 translation orchestration (`pipeline.py`) |
|
| 146 |
+
| `frontend/` | Gradio 6 UI: styled toggles (`widgets.py`), card rendering with two-phase layout (`cards.py`), custom CSS (`css/custom.css`) |
|
| 147 |
+
| `models/` | Hugging Face Hub model downloader β fetches models at runtime, no git submodules |
|
| 148 |
+
| `export/` | Anki `.apkg` export via genanki (`apkg_export.py`), standard CSV zip export with flat media files (`csv_export.py`) |
|
| 149 |
+
| `app.py` | Entry point β wires inputs to two-phase click handlers with progress tracking |
|
| 150 |
+
|
| 151 |
+
### Data Flow
|
| 152 |
+
|
| 153 |
+
```
|
| 154 |
+
User Input β [Gradio UI] β EnginePool (singleton) β MiniCPMTextEngine (Phase 1) β pipeline.generate_phase2() β LlamaCppTextEngine (translation, Phase 2) β TTSEngine (`core/audio_gen.py`, TTS audio, Phase 2) β Card Gallery β Export (.apkg / .csv)
|
| 155 |
+
```
|
| 156 |
+
|
| 157 |
+
- **Inference:** `core/engine.py` defines five engine classes:
|
| 158 |
+
- `MiniCPMTextEngine` β llama-cpp-python wrapper for MiniCPM5-1B Q8_0 (lazy-load/unload, ~1.1 GB RAM, uses apply_chat_template). Uses `TextResult.validate_and_parse()` to strip `<thinking>` tags and enforce exact sentence count; retries with stricter prompts on mismatch (max 3 attempts). Used in Phase 1 for English text generation only.
|
| 159 |
+
- `LlamaCppTextEngine` β llama-cpp-python wrapper for tiny-aya-water translation (lazy-load/unload, ~2 GB VRAM). Validates output line count against `batch_size`; retries with stricter prompts on mismatch (max 3 attempts). Used in Phase 2 for translation.
|
| 160 |
+
- `TTSEngine` (`core/audio_gen.py`) β OmniVoice Python package with lazy-load/unload cycle. Supports voice design mode via `instruct` parameter (e.g., "female, young adult"). Used in Phase 2 for TTS audio.
|
| 161 |
+
- `ImageGenEngine` (`core/image_gen.py`) β diffusers Flux2KleinPipeline with lazy-load/unload cycle (GPU memory managed by EnginePool). Image generation toggle is available but not yet wired into the pipeline.
|
| 162 |
+
- `EnginePool` β singleton orchestrator enforcing mutual exclusion between all GPU engines. Phase 1 uses only `MiniCPMTextEngine` (~1.1 GB RAM). Phase 2 loads GPU engines sequentially: translation β TTS/images.
|
| 163 |
+
- **Types:** `core/types.py` provides Pydantic models (`CardData`, `CEFRLevel`, `ValidationError`, `TextResult`, `AudioResult`, `ImageResult`, `EngineConfig`) for type-safe boundaries. `TextResult.generated_texts` replaces the legacy `.translations`; `AudioResult.audio_paths` and `ImageResult.image_paths` are `list[str | None]` (never None at top level).
|
| 164 |
+
- **Pipeline:** `core/pipeline.py` provides `generate_phase2()` β a generator function that yields `(progress_percent, phase_label, cards)` tuples for real-time UI updates. Extends this when adding new media types (TTS, images).
|
| 165 |
+
- **Frontend:** `frontend/ui/cards.py` renders individual cards as HTML with conditional media elements; `generate_cards_html()` layouts them in a flex gallery with natural rotation offsets.
|
| 166 |
+
- **Export:** `export/apkg_export.py` builds proper `.apkg` files via genanki (SQLite-based collection.anki2, bundled media); `export/csv_export.py` creates zipped folders containing CSV + flat media files.
|
| 167 |
+
|
| 168 |
+
## Repository Structure
|
| 169 |
+
|
| 170 |
+
```
|
| 171 |
+
EuropaLex/
|
| 172 |
+
βββ app.py # Entry point β Gradio UI wiring, two-phase generation handlers
|
| 173 |
+
βββ pyproject.toml # Project config (uv)
|
| 174 |
+
βββ requirements.txt # pip install dependencies
|
| 175 |
+
βββ uv.lock # uv lock file
|
| 176 |
+
βββ .gitignore
|
| 177 |
+
βββ README.md # This file
|
| 178 |
+
βββ AGENTS.md # AI agent conventions guide
|
| 179 |
+
βββ core/ # Shared business logic
|
| 180 |
+
β βββ __init__.py
|
| 181 |
+
β βββ types.py # Pydantic models: CardData, CEFRLevel, TextResult, AudioResult, ImageResult, EngineConfig
|
| 182 |
+
β βββ engine.py # MiniCPMTextEngine, LlamaCppTextEngine, EnginePool
|
| 183 |
+
β βββ audio_gen.py # TTSEngine (OmniVoice)
|
| 184 |
+
β βββ image_gen.py # ImageGenEngine (diffusers Flux2KleinPipeline)
|
| 185 |
+
β βββ text_gen.py # Sentence extraction (extract_sentences) and generation with retry loop (generate_sentences)
|
| 186 |
+
β βββ pipeline.py # Phase 2 translation orchestration β generate_phase2() generator with progress tracking
|
| 187 |
+
βββ frontend/ # Gradio 6 UI
|
| 188 |
+
β βββ __init__.py
|
| 189 |
+
β βββ ui/
|
| 190 |
+
β β βββ __init__.py
|
| 191 |
+
β β βββ widgets.py # Styled toggle checkbox wrappers
|
| 192 |
+
β β βββ cards.py # Card rendering, gallery layout, progress bar
|
| 193 |
+
β βββ css/
|
| 194 |
+
β βββ custom.css # Plain-white theme, card styling, disabled states
|
| 195 |
+
βββ models/ # Model management
|
| 196 |
+
β βββ __init__.py
|
| 197 |
+
β βββ download_models.py # HF Hub model downloader (runtime)
|
| 198 |
+
βββ configs/ # Configuration
|
| 199 |
+
β βββ settings.yaml # App settings, word lists
|
| 200 |
+
βββ export/ # Export formats
|
| 201 |
+
β βββ __init__.py
|
| 202 |
+
β βββ apkg_export.py # Anki `.apkg` export via genanki (SQLite collection.anki2, bundled media)
|
| 203 |
+
β βββ csv_export.py # Standard CSV export utility (flat folder structure)
|
| 204 |
+
β βββ anki_tunnel.py # MCP tunnel sync for live Anki import
|
| 205 |
+
βββ docs/ # Design specs and implementation plans
|
| 206 |
+
β βββ superpowers/
|
| 207 |
+
β βββ specs/ # Design specification documents
|
| 208 |
+
β βββ plans/ # Implementation plans
|
| 209 |
+
βββ tests/ # Test suite (pytest-discoverable)
|
| 210 |
+
β βββ smoke_test.py # Integration test β module imports, app construction
|
| 211 |
+
β βββ count_enforcement_test.py # TextResult.validate_and_parse() testing
|
| 212 |
+
β βββ extract_sentences_test.py # core.text_gen.extract_sentences() testing
|
| 213 |
+
β βββ progression_test.py # _progress_pct() helper testing
|
| 214 |
+
β βββ translation_retry_test.py# LlamaCppTextEngine retry loop testing
|
| 215 |
+
```
|
| 216 |
+
|
| 217 |
+
## CEFR Levels
|
| 218 |
+
|
| 219 |
+
`[A0, A1, A2, B1, B2, C1, C2]`
|
| 220 |
+
|
| 221 |
+
- **A0:** Uses curated common words list (no text generation model needed)
|
| 222 |
+
- **A1βC2:** MiniCPM5-1B generates English sentences at the selected level in Phase 1; tiny-aya-water translates them in Phase 2
|