Sync Space with main (review fixes)
Browse files- .env.example +3 -0
- .gitattributes +3 -0
- DISCLOSURE.md +12 -2
- README.md +13 -9
- TESTING.md +7 -4
- TEST_RESULTS.md +80 -0
- app.py +1 -1
- eval/evaluate.py +3 -2
- eval/results.json +6 -6
- eval/results.md +2 -2
- index.html +2 -2
- perde/gateway.py +16 -6
- perde/recognizers.py +2 -2
- perde/shield.py +13 -5
- tests/test_shield.py +35 -0
- tools/video/.gitignore +1 -0
- tools/video/README.md +9 -0
- tools/video/capture.py +69 -0
- tools/video/render.py +422 -0
.env.example
CHANGED
|
@@ -1,6 +1,9 @@
|
|
|
|
|
| 1 |
# Set one of these to use a real model; leave all empty to use the offline mock.
|
| 2 |
OPENAI_API_KEY=
|
| 3 |
XAI_API_KEY=
|
| 4 |
GEMINI_API_KEY=
|
| 5 |
# Optional: override the provider's default model name
|
| 6 |
PERDE_MODEL=
|
|
|
|
|
|
|
|
|
| 1 |
+
# Not loaded automatically: export these in your shell, or pass them with docker run -e NAME=value.
|
| 2 |
# Set one of these to use a real model; leave all empty to use the offline mock.
|
| 3 |
OPENAI_API_KEY=
|
| 4 |
XAI_API_KEY=
|
| 5 |
GEMINI_API_KEY=
|
| 6 |
# Optional: override the provider's default model name
|
| 7 |
PERDE_MODEL=
|
| 8 |
+
# Optional: OpenAI-compatible local server URL for the llamacpp provider
|
| 9 |
+
PERDE_LOCAL_BASE_URL=http://127.0.0.1:8080/v1
|
.gitattributes
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
* text=auto
|
| 2 |
+
*.bat text eol=crlf
|
| 3 |
+
*.mp4 binary
|
DISCLOSURE.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
# Disclosure: models, data and components
|
| 2 |
|
| 3 |
## Written during the hackathon
|
| 4 |
-
All code in `perde/`, `app.py`, `eval/`
|
| 5 |
Pərdə does not depend on any third-party PII library. The general approach (pattern recognisers plus anonymise/restore) is a well-known one, used for example by Microsoft Presidio. We built our own implementation for Azerbaijani formats.
|
| 6 |
|
| 7 |
## Data
|
|
@@ -14,7 +14,8 @@ Pərdə does not depend on any third-party PII library. The general approach (pa
|
|
| 14 |
|
| 15 |
## AI models
|
| 16 |
- **No model is used for detection.** Detection is rules and a lexicon, and runs locally.
|
| 17 |
-
- **For the user's task:** any OpenAI-compatible chat model, chosen at runtime. Supported providers
|
|
|
|
| 18 |
|
| 19 |
## Libraries
|
| 20 |
| Library | Licence | Use |
|
|
@@ -23,6 +24,12 @@ Pərdə does not depend on any third-party PII library. The general approach (pa
|
|
| 23 |
| openai (Python SDK) | Apache-2.0 | Client for OpenAI-compatible APIs |
|
| 24 |
| pytest | MIT | Unit tests (dev only) |
|
| 25 |
| python:3.12-slim Docker base image | PSF and Debian licences | Container build |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
## Assistance
|
| 28 |
The code and documentation were written with the help of an AI coding assistant (Claude).
|
|
@@ -30,3 +37,6 @@ The code and documentation were written with the help of an AI coding assistant
|
|
| 30 |
## Mocks and caches
|
| 31 |
- The `mock` provider returns a templated reply built from the tags. It is labelled "[Offline demo model, no external API call]" in its output.
|
| 32 |
- No results are cached. The evaluation numbers come from `eval/evaluate.py` and are stored in `eval/results.json`.
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Disclosure: models, data and components
|
| 2 |
|
| 3 |
## Written during the hackathon
|
| 4 |
+
All code in `perde/`, `app.py`, `index.html`, `eval/`, `tests/`, `tools/video/`, `run.bat` and `Dockerfile`: the detectors, masking and restoration, gateway, UI, evaluation, tests, launchers and video tooling.
|
| 5 |
Pərdə does not depend on any third-party PII library. The general approach (pattern recognisers plus anonymise/restore) is a well-known one, used for example by Microsoft Presidio. We built our own implementation for Azerbaijani formats.
|
| 6 |
|
| 7 |
## Data
|
|
|
|
| 14 |
|
| 15 |
## AI models
|
| 16 |
- **No model is used for detection.** Detection is rules and a lexicon, and runs locally.
|
| 17 |
+
- **For the user's task:** any OpenAI-compatible chat model, chosen at runtime. Supported providers, with their default model names: OpenAI (`gpt-4o-mini`), xAI Grok (`grok-3-mini`), Google Gemini (`gemini-2.5-flash`), local Ollama (`gemma3`) and any llama.cpp OpenAI-compatible server (`gemma-4-E4B`). The default is `mock`, an offline stand-in with no external call. The live demo and the demo video use `mock`.
|
| 18 |
+
- **Tested end to end with a real model:** Gemma 4 E4B on a local llama.cpp server (see `TEST_RESULTS.md`).
|
| 19 |
|
| 20 |
## Libraries
|
| 21 |
| Library | Licence | Use |
|
|
|
|
| 24 |
| openai (Python SDK) | Apache-2.0 | Client for OpenAI-compatible APIs |
|
| 25 |
| pytest | MIT | Unit tests (dev only) |
|
| 26 |
| python:3.12-slim Docker base image | PSF and Debian licences | Container build |
|
| 27 |
+
| @stlite/browser 1.9.2 (via jsDelivr CDN) | Apache-2.0 | Runs the Streamlit app in the browser for the live demo |
|
| 28 |
+
| Pyodide (loaded by stlite) | MPL-2.0 | Python in WebAssembly for the live demo |
|
| 29 |
+
| Hugging Face Spaces (static) | Hosting service | Live demo hosting |
|
| 30 |
+
| Pillow, numpy, imageio-ffmpeg (FFmpeg) | HPND, BSD-3, BSD-2 (FFmpeg: LGPL/GPL) | Demo video rendering only |
|
| 31 |
+
| websockets, Google Chrome (headless) | BSD-3, proprietary | Screenshots for the demo video only |
|
| 32 |
+
| Segoe UI, Cascadia Mono fonts | Windows system fonts (Cascadia: OFL) | Text in the demo video |
|
| 33 |
|
| 34 |
## Assistance
|
| 35 |
The code and documentation were written with the help of an AI coding assistant (Claude).
|
|
|
|
| 37 |
## Mocks and caches
|
| 38 |
- The `mock` provider returns a templated reply built from the tags. It is labelled "[Offline demo model, no external API call]" in its output.
|
| 39 |
- No results are cached. The evaluation numbers come from `eval/evaluate.py` and are stored in `eval/results.json`.
|
| 40 |
+
|
| 41 |
+
## Demo video
|
| 42 |
+
`media/perde-demo.mp4` was rendered by `tools/video/render.py` during the hackathon (screenshots by `tools/video/capture.py`). The app footage is real screenshots of the running app, using the offline model. The tags shown come from the real masking engine. The soundtrack is synthesized in code, with no third-party music.
|
README.md
CHANGED
|
@@ -34,8 +34,8 @@ qeydə alındı… 055 412 33 90 qeydə alındı… [PHONE_
|
|
| 34 |
|
| 35 |
- **The user:** an operator who handles customer requests at a bank, a mobile operator, or a public service centre (complaints, applications, tickets).
|
| 36 |
- **Today:** these teams either don't use AI at all, or staff copy customer data into public chatbots. Neither is acceptable. Azerbaijan's Law on Personal Data (2010) restricts handing personal data to third parties.
|
| 37 |
-
- **With Pərdə:** the same staff use any
|
| 38 |
-
- **The metric that matters:** the **leak rate**, meaning the share of personal values that reach the AI provider. It falls from 76.1% with a generic filter to 5.1% with Pərdə (held-out test, details in [TESTING.md](TESTING.md)).
|
| 39 |
|
| 40 |
## 2. Prototype and use of AI
|
| 41 |
|
|
@@ -45,10 +45,10 @@ The working core scenario is to paste a customer text, pick a sector and a task,
|
|
| 45 |
2. The exact text the AI provider receives
|
| 46 |
3. The provider's answer, with the real values restored
|
| 47 |
|
| 48 |
-
**Where AI does the work:** the external LLM (OpenAI, xAI Grok, Gemini, or a local Ollama
|
| 49 |
|
| 50 |
**Why detection is not an LLM:** sending text to an LLM to find the personal data would leak it to that LLM, which defeats the purpose. So detection is local, deterministic and auditable. It combines:
|
| 51 |
-
- validated patterns for Azerbaijani identifiers (FIN, ID card `AZE…`/`AA…`, passport, driver licence, VÖEN, AZ IBAN
|
| 52 |
- a name detector built for Azerbaijani: a name lexicon, surname morphology (`-ov/-ova`, `-li/-lı`, `-zadə`, `-oğlu`), patronymics (`oğlu`/`qızı`), stripping of Azerbaijani case endings (`Rənanın` → `[PERSON_1]nın`), and Cyrillic and ASCII spellings
|
| 53 |
- street patterns (`küçəsi`, `prospekti`, `ул.`) with house and flat numbers
|
| 54 |
|
|
@@ -73,17 +73,17 @@ Full report with failures: **[TESTING.md](TESTING.md)**. Summary:
|
|
| 73 |
| E-mail | 132 | 100% |
|
| 74 |
| ID numbers (FIN, ID card, passport, licence, VÖEN) | 165 | 100% |
|
| 75 |
| Card numbers | 25 | 100% |
|
| 76 |
-
| Addresses | 298 | 82.
|
| 77 |
| Dates | 117 | 80.3% |
|
| 78 |
|
| 79 |
- The test split was fixed by seed and never used for tuning. The name lexicon was built from a separate 2,000-row dev split.
|
| 80 |
- Reproduce with `python eval/fetch_sample.py && python eval/evaluate.py`.
|
| 81 |
-
- There are
|
| 82 |
|
| 83 |
## 4. Feasibility
|
| 84 |
|
| 85 |
- **Data needed:** none to start. It runs on rules and a lexicon, and each organisation can extend the lexicon and patterns.
|
| 86 |
-
- **Running cost:** masking is free, takes about 0.14 ms per message on one CPU core, and needs no GPU. The LLM cost is unchanged, because
|
| 87 |
- **Deployment:** a Python package and a web UI. It runs on-premises, next to an existing API gateway, or as a proxy.
|
| 88 |
- **Next step (4 weeks):**
|
| 89 |
1. Pilot on one support queue.
|
|
@@ -101,7 +101,11 @@ Generic PII filters are built for English. Azerbaijani personal data has its own
|
|
| 101 |
|
| 102 |
**Live demo (no install, runs in your browser):** https://huggingface.co/spaces/Traxic/perde
|
| 103 |
|
| 104 |
-
**
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
|
| 106 |
**Docker (any OS, suits on-premises servers):**
|
| 107 |
|
|
@@ -119,7 +123,7 @@ python -m venv .venv
|
|
| 119 |
```
|
| 120 |
|
| 121 |
- The default provider is **mock**, which works offline with no key.
|
| 122 |
-
- For a real model, set one of `OPENAI_API_KEY`, `XAI_API_KEY` (Grok) or `GEMINI_API_KEY`,
|
| 123 |
- Use it in code:
|
| 124 |
|
| 125 |
```python
|
|
|
|
| 34 |
|
| 35 |
- **The user:** an operator who handles customer requests at a bank, a mobile operator, or a public service centre (complaints, applications, tickets).
|
| 36 |
- **Today:** these teams either don't use AI at all, or staff copy customer data into public chatbots. Neither is acceptable. Azerbaijan's Law on Personal Data (2010) restricts handing personal data to third parties.
|
| 37 |
+
- **With Pərdə:** the same staff use any OpenAI-compatible model for summaries, replies, classification and translation. Personal data the detectors find, in both the customer text and the instruction, reaches the provider only as tags.
|
| 38 |
+
- **The metric that matters:** the **leak rate**, meaning the share of personal values that reach the AI provider. It falls from 76.1% with a generic filter to 5.1% with Pərdə (held-out test, details in [TESTING.md](TESTING.md)). Raw pasting, today's alternative, leaks 100%. A second check then **blocks** any request where a detected value would still leave unmasked; it cannot catch values the detectors missed.
|
| 39 |
|
| 40 |
## 2. Prototype and use of AI
|
| 41 |
|
|
|
|
| 45 |
2. The exact text the AI provider receives
|
| 46 |
3. The provider's answer, with the real values restored
|
| 47 |
|
| 48 |
+
**Where AI does the work:** the external LLM (OpenAI, xAI Grok, Gemini, or a local model via Ollama or llama.cpp) does the actual task: summarising, drafting a reply in Azerbaijani or Russian, classifying the ticket. Pərdə is what makes it *permissible* to use that model on real data.
|
| 49 |
|
| 50 |
**Why detection is not an LLM:** sending text to an LLM to find the personal data would leak it to that LLM, which defeats the purpose. So detection is local, deterministic and auditable. It combines:
|
| 51 |
+
- validated patterns for Azerbaijani identifiers (FIN, ID card `AZE…`/`AA…`, passport, driver licence, VÖEN, AZ IBAN (mod-97 checked; invalid ones are still masked), card numbers (Luhn checked, or masked by context), +994 mobile and landline formats, car plates, dates)
|
| 52 |
- a name detector built for Azerbaijani: a name lexicon, surname morphology (`-ov/-ova`, `-li/-lı`, `-zadə`, `-oğlu`), patronymics (`oğlu`/`qızı`), stripping of Azerbaijani case endings (`Rənanın` → `[PERSON_1]nın`), and Cyrillic and ASCII spellings
|
| 53 |
- street patterns (`küçəsi`, `prospekti`, `ул.`) with house and flat numbers
|
| 54 |
|
|
|
|
| 73 |
| E-mail | 132 | 100% |
|
| 74 |
| ID numbers (FIN, ID card, passport, licence, VÖEN) | 165 | 100% |
|
| 75 |
| Card numbers | 25 | 100% |
|
| 76 |
+
| Addresses | 298 | 82.6% |
|
| 77 |
| Dates | 117 | 80.3% |
|
| 78 |
|
| 79 |
- The test split was fixed by seed and never used for tuning. The name lexicon was built from a separate 2,000-row dev split.
|
| 80 |
- Reproduce with `python eval/fetch_sample.py && python eval/evaluate.py`.
|
| 81 |
+
- There are 17 unit tests covering checksums, phone formats, tag consistency, restoration, the leak check and masking of the instruction field (`pytest`).
|
| 82 |
|
| 83 |
## 4. Feasibility
|
| 84 |
|
| 85 |
- **Data needed:** none to start. It runs on rules and a lexicon, and each organisation can extend the lexicon and patterns.
|
| 86 |
+
- **Running cost:** masking is free, takes about 0.14 ms per message on one CPU core, and needs no GPU. The LLM cost is roughly unchanged, because tags are about as long as the values they replace.
|
| 87 |
- **Deployment:** a Python package and a web UI. It runs on-premises, next to an existing API gateway, or as a proxy.
|
| 88 |
- **Next step (4 weeks):**
|
| 89 |
1. Pilot on one support queue.
|
|
|
|
| 101 |
|
| 102 |
**Live demo (no install, runs in your browser):** https://huggingface.co/spaces/Traxic/perde
|
| 103 |
|
| 104 |
+
**Demo video (56 s):** [media/perde-demo.mp4](media/perde-demo.mp4)
|
| 105 |
+
|
| 106 |
+
**Real-model test:** end-to-end with Gemma 4 E4B on a local llama.cpp server, see [TEST_RESULTS.md](TEST_RESULTS.md).
|
| 107 |
+
|
| 108 |
+
**Windows, one click:** download the latest release zip, unzip it and double-click `run.bat`. It needs Python 3.10 or newer. The first run installs the dependencies, then the app opens at http://localhost:8501.
|
| 109 |
|
| 110 |
**Docker (any OS, suits on-premises servers):**
|
| 111 |
|
|
|
|
| 123 |
```
|
| 124 |
|
| 125 |
- The default provider is **mock**, which works offline with no key.
|
| 126 |
+
- For a real model, set one of `OPENAI_API_KEY`, `XAI_API_KEY` (Grok) or `GEMINI_API_KEY`, run Ollama locally, or use the `llamacpp` provider for an OpenAI-compatible local server (default `http://127.0.0.1:8080/v1`). Then pick the provider in the sidebar. Optionally set `PERDE_MODEL`.
|
| 127 |
- Use it in code:
|
| 128 |
|
| 129 |
```python
|
TESTING.md
CHANGED
|
@@ -19,6 +19,8 @@ Out of scope: city, time, age and zip code. These are not personal data on their
|
|
| 19 |
|
| 20 |
## Baselines
|
| 21 |
|
|
|
|
|
|
|
| 22 |
1. **Generic regex baseline.** E-mail, any long digit sequence as a phone number, and ISO and slash dates. This is roughly what a default, non-localised filter catches.
|
| 23 |
2. **Pərdə without the name detector.** An ablation that shows what the Azerbaijani name handling adds.
|
| 24 |
|
|
@@ -27,7 +29,7 @@ Out of scope: city, time, age and zip code. These are not personal data on their
|
|
| 27 |
| System | Protected | Leak rate | Precision |
|
| 28 |
|---|---|---|---|
|
| 29 |
| **Pərdə** | **94.9%** | **5.1%** | 93.0% |
|
| 30 |
-
| Pərdə without name detector | 41.
|
| 31 |
| Generic regex baseline | 23.9% | 76.1% | 96.4% |
|
| 32 |
|
| 33 |
| Entity | Values | Recall (right type) | Protected | Precision |
|
|
@@ -38,7 +40,7 @@ Out of scope: city, time, age and zip code. These are not personal data on their
|
|
| 38 |
| ID_NUMBER | 165 | 100.0% | 100.0% | 100.0% |
|
| 39 |
| CARD | 25 | 100.0% | 100.0% | 100.0% |
|
| 40 |
| DATE | 117 | 80.3% | 80.3% | 78.5% |
|
| 41 |
-
| ADDRESS | 298 |
|
| 42 |
|
| 43 |
## Results: hand-written cases (20 texts)
|
| 44 |
|
|
@@ -55,14 +57,15 @@ Out of scope: city, time, age and zip code. These are not personal data on their
|
|
| 55 |
| `Ульвия` | "zəng: 0705552211 - **Ульвия**, deyir ki…" | A Cyrillic spelling of an Azerbaijani name that isn't in the lexicon, and there is no surname next to it. |
|
| 56 |
| `Ələsgər` | "22:48:43 **Ələsgər**: Mənim nömrəm…" | A rare given name, not in the dev-built lexicon. |
|
| 57 |
| `İsmayılqızı` | "Mənim adım Ceyla **İsmayılqızıdır**" | A patronymic fused with a suffix (`-qızı` + `-dır`), which isn't handled yet. |
|
| 58 |
-
| `
|
| 59 |
| `S.S. AXUNDOV pr.` | "Diplomunuzu 1 **S.S. AXUNDOV pr.**-dən…" | Two initials with a space between them. |
|
| 60 |
| `1987/01/01` | "Yubileyimiz: **1987/01/01**" | The `yyyy/mm/dd` format isn't supported yet. We didn't add it after seeing the test set, so the score stays honest. |
|
| 61 |
| `0` / `5` (building numbers) | "…**5** ilə doldurun" | Bare single digits labelled as building numbers in the dataset. We don't mask lone digits on purpose. |
|
| 62 |
|
| 63 |
-
What changed during development (on dev only), and what each change did to leak rate on dev:
|
| 64 |
- Names: added the lexicon, case-ending stripping and greeting/title context. Leak rate went from 52.9% to 12.2%.
|
| 65 |
- IDs: allowed `I`/`O` in FIN, added `AZE`+7 digits and the driver-licence pattern. Leak rate went from 12.2% to 5.5%.
|
|
|
|
| 66 |
|
| 67 |
## Limitations (stated honestly)
|
| 68 |
|
|
|
|
| 19 |
|
| 20 |
## Baselines
|
| 21 |
|
| 22 |
+
0. **Today's practice: pasting the raw text into a chatbot.** Nothing is masked, so 100% of the personal values reach the provider. Redacting by hand is the alternative; we did not time it, so we make no claim about its speed.
|
| 23 |
+
|
| 24 |
1. **Generic regex baseline.** E-mail, any long digit sequence as a phone number, and ISO and slash dates. This is roughly what a default, non-localised filter catches.
|
| 25 |
2. **Pərdə without the name detector.** An ablation that shows what the Azerbaijani name handling adds.
|
| 26 |
|
|
|
|
| 29 |
| System | Protected | Leak rate | Precision |
|
| 30 |
|---|---|---|---|
|
| 31 |
| **Pərdə** | **94.9%** | **5.1%** | 93.0% |
|
| 32 |
+
| Pərdə without name detector | 41.2% | 58.8% | 96.0% |
|
| 33 |
| Generic regex baseline | 23.9% | 76.1% | 96.4% |
|
| 34 |
|
| 35 |
| Entity | Values | Recall (right type) | Protected | Precision |
|
|
|
|
| 40 |
| ID_NUMBER | 165 | 100.0% | 100.0% | 100.0% |
|
| 41 |
| CARD | 25 | 100.0% | 100.0% | 100.0% |
|
| 42 |
| DATE | 117 | 80.3% | 80.3% | 78.5% |
|
| 43 |
+
| ADDRESS | 298 | 75.2% | 82.6% | 97.1% |
|
| 44 |
|
| 45 |
## Results: hand-written cases (20 texts)
|
| 46 |
|
|
|
|
| 57 |
| `Ульвия` | "zəng: 0705552211 - **Ульвия**, deyir ki…" | A Cyrillic spelling of an Azerbaijani name that isn't in the lexicon, and there is no surname next to it. |
|
| 58 |
| `Ələsgər` | "22:48:43 **Ələsgər**: Mənim nömrəm…" | A rare given name, not in the dev-built lexicon. |
|
| 59 |
| `İsmayılqızı` | "Mənim adım Ceyla **İsmayılqızıdır**" | A patronymic fused with a suffix (`-qızı` + `-dır`), which isn't handled yet. |
|
| 60 |
+
| `8` (building number) | "…materialları Callan's Lane, **8**…" | A building number after a non-Azerbaijani street name. |
|
| 61 |
| `S.S. AXUNDOV pr.` | "Diplomunuzu 1 **S.S. AXUNDOV pr.**-dən…" | Two initials with a space between them. |
|
| 62 |
| `1987/01/01` | "Yubileyimiz: **1987/01/01**" | The `yyyy/mm/dd` format isn't supported yet. We didn't add it after seeing the test set, so the score stays honest. |
|
| 63 |
| `0` / `5` (building numbers) | "…**5** ilə doldurun" | Bare single digits labelled as building numbers in the dataset. We don't mask lone digits on purpose. |
|
| 64 |
|
| 65 |
+
What changed during development (on dev only), and what each change did to leak rate on dev. These numbers come from runs during development (`python eval/evaluate.py --split dev`); the intermediate code versions are not all kept:
|
| 66 |
- Names: added the lexicon, case-ending stripping and greeting/title context. Leak rate went from 52.9% to 12.2%.
|
| 67 |
- IDs: allowed `I`/`O` in FIN, added `AZE`+7 digits and the driver-licence pattern. Leak rate went from 12.2% to 5.5%.
|
| 68 |
+
- After the first held-out run, a code review found two bugs in the Government demo sample: a house number before a full stop, and a bracketed landline. Fixing them moved held-out addresses from 82.2% to 82.6%. Nothing else changed.
|
| 69 |
|
| 70 |
## Limitations (stated honestly)
|
| 71 |
|
TEST_RESULTS.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Test Results
|
| 2 |
+
|
| 3 |
+
Run date: 2026-10-09T10:48:27Z
|
| 4 |
+
|
| 5 |
+
## Environment
|
| 6 |
+
|
| 7 |
+
- Python: `Python 3.14.7`
|
| 8 |
+
- Local model endpoint: `http://127.0.0.1:8080/v1`
|
| 9 |
+
- Local provider: `llamacpp`
|
| 10 |
+
- Local model: `gemma-4-E4B`
|
| 11 |
+
|
| 12 |
+
## Unit tests
|
| 13 |
+
|
| 14 |
+
Command:
|
| 15 |
+
|
| 16 |
+
```bash
|
| 17 |
+
.venv/bin/pytest -q
|
| 18 |
+
```
|
| 19 |
+
|
| 20 |
+
Result:
|
| 21 |
+
|
| 22 |
+
```text
|
| 23 |
+
........... [100%]
|
| 24 |
+
11 passed in 0.03s
|
| 25 |
+
```
|
| 26 |
+
|
| 27 |
+
## Local Gemma gateway test
|
| 28 |
+
|
| 29 |
+
Command:
|
| 30 |
+
|
| 31 |
+
```bash
|
| 32 |
+
.venv/bin/python - <<'PY'
|
| 33 |
+
from perde.gateway import run
|
| 34 |
+
|
| 35 |
+
text = "Salam, mən Nərmin Quliyeva. Kartım 4111 1111 1111 1111, FIN 6TR9K2L. Tel 055 412 33 90."
|
| 36 |
+
task = "Bu müraciəti bir cümlə ilə xülasə et və qısa cavab yaz. Tagləri eyni saxla."
|
| 37 |
+
res = run(text, task, provider="llamacpp", model="gemma-4-E4B")
|
| 38 |
+
print(res.outgoing)
|
| 39 |
+
print(res.model_answer)
|
| 40 |
+
print(res.restored_answer)
|
| 41 |
+
PY
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
Result:
|
| 45 |
+
|
| 46 |
+
```text
|
| 47 |
+
provider: llamacpp
|
| 48 |
+
model: gemma-4-E4B
|
| 49 |
+
latency_ms: 6732
|
| 50 |
+
masked_values: 4
|
| 51 |
+
entity_counts: {'PERSON': 1, 'CARD': 1, 'FIN': 1, 'PHONE': 1}
|
| 52 |
+
outgoing:
|
| 53 |
+
Bu müraciəti bir cümlə ilə xülasə et və qısa cavab yaz. Tagləri eyni saxla.
|
| 54 |
+
|
| 55 |
+
---
|
| 56 |
+
Salam, mən [PERSON_1]. Kartım [CARD_1], FIN [FIN_1]. Tel [PHONE_1].
|
| 57 |
+
model_answer:
|
| 58 |
+
**Xülasə:** Müraciət sahibi [PERSON_1] öz şəxsi məlumatlarını (kart [CARD_1], FIN [FIN_1] və telefon [PHONE_1]) təqdim edir.
|
| 59 |
+
|
| 60 |
+
**Cavab:** Məlumatınız qeydə alındı.
|
| 61 |
+
restored_answer:
|
| 62 |
+
**Xülasə:** Müraciət sahibi Nərmin Quliyeva öz şəxsi məlumatlarını (kart 4111 1111 1111 1111, FIN 6TR9K2L və telefon 055 412 33 90) təqdim edir.
|
| 63 |
+
|
| 64 |
+
**Cavab:** Məlumatınız qeydə alındı.
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
## Streamlit smoke test
|
| 68 |
+
|
| 69 |
+
Command:
|
| 70 |
+
|
| 71 |
+
```bash
|
| 72 |
+
.venv/bin/streamlit run app.py --server.headless true --server.port 8501
|
| 73 |
+
curl -fsS http://127.0.0.1:8501/_stcore/health
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
Result:
|
| 77 |
+
|
| 78 |
+
```text
|
| 79 |
+
ok
|
| 80 |
+
```
|
app.py
CHANGED
|
@@ -98,7 +98,7 @@ with st.sidebar:
|
|
| 98 |
if in_browser:
|
| 99 |
st.info(
|
| 100 |
"Web demo: runs entirely in your browser with the offline model. "
|
| 101 |
-
"To use OpenAI, Grok, Gemini or
|
| 102 |
)
|
| 103 |
model = st.text_input("Model (optional)", value="", placeholder=PROVIDERS[provider]["model"])
|
| 104 |
enabled = st.multiselect("Data types to protect", ALL_ENTITIES, default=list(SECTORS[sector]["entities"]))
|
|
|
|
| 98 |
if in_browser:
|
| 99 |
st.info(
|
| 100 |
"Web demo: runs entirely in your browser with the offline model. "
|
| 101 |
+
"To use OpenAI, Grok, Gemini, Ollama or a llama.cpp server, run the app locally (run.bat) or with Docker."
|
| 102 |
)
|
| 103 |
model = st.text_input("Model (optional)", value="", placeholder=PROVIDERS[provider]["model"])
|
| 104 |
enabled = st.multiselect("Data types to protect", ALL_ENTITIES, default=list(SECTORS[sector]["entities"]))
|
eval/evaluate.py
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
"""Measure detection quality on the held-out LocalDoc split and on hand-written cases.
|
| 2 |
|
| 3 |
Usage: python eval/evaluate.py [--split test|dev] [--show-failures N]
|
| 4 |
-
Writes eval/results.json and prints a Markdown table.
|
| 5 |
|
| 6 |
Metrics, per entity group:
|
| 7 |
recall gold values masked by a detection of the right type (>=50% of characters)
|
|
@@ -178,7 +178,8 @@ def main():
|
|
| 178 |
s = results[dname]["Pərdə"][g]
|
| 179 |
if s["n_gold"]:
|
| 180 |
print(f"| {g} | {s['n_gold']} | {fmt(s['recall'])} | {fmt(s['protected'])} | {fmt(s['precision'])} |")
|
| 181 |
-
|
|
|
|
| 182 |
|
| 183 |
|
| 184 |
if __name__ == "__main__":
|
|
|
|
| 1 |
"""Measure detection quality on the held-out LocalDoc split and on hand-written cases.
|
| 2 |
|
| 3 |
Usage: python eval/evaluate.py [--split test|dev] [--show-failures N]
|
| 4 |
+
Writes eval/results.json (test split) or eval/results_dev.json (dev split) and prints a Markdown table.
|
| 5 |
|
| 6 |
Metrics, per entity group:
|
| 7 |
recall gold values masked by a detection of the right type (>=50% of characters)
|
|
|
|
| 178 |
s = results[dname]["Pərdə"][g]
|
| 179 |
if s["n_gold"]:
|
| 180 |
print(f"| {g} | {s['n_gold']} | {fmt(s['recall'])} | {fmt(s['protected'])} | {fmt(s['precision'])} |")
|
| 181 |
+
out_name = "results.json" if args.split == "test" else "results_dev.json" # the app shows results.json
|
| 182 |
+
(ROOT / "eval" / out_name).write_text(json.dumps(results, ensure_ascii=False, indent=2), encoding="utf-8")
|
| 183 |
|
| 184 |
|
| 185 |
if __name__ == "__main__":
|
eval/results.json
CHANGED
|
@@ -45,14 +45,14 @@
|
|
| 45 |
},
|
| 46 |
"ADDRESS": {
|
| 47 |
"n_gold": 298,
|
| 48 |
-
"recall": 0.
|
| 49 |
-
"protected": 0.
|
| 50 |
"precision": 0.971,
|
| 51 |
"n_pred": 138
|
| 52 |
},
|
| 53 |
"ALL": {
|
| 54 |
"n_gold": 2065,
|
| 55 |
-
"recall": 0.
|
| 56 |
"protected": 0.949,
|
| 57 |
"precision": 0.93,
|
| 58 |
"n_pred": 1725
|
|
@@ -103,15 +103,15 @@
|
|
| 103 |
},
|
| 104 |
"ADDRESS": {
|
| 105 |
"n_gold": 298,
|
| 106 |
-
"recall": 0.
|
| 107 |
-
"protected": 0.
|
| 108 |
"precision": 0.971,
|
| 109 |
"n_pred": 138
|
| 110 |
},
|
| 111 |
"ALL": {
|
| 112 |
"n_gold": 2065,
|
| 113 |
"recall": 0.399,
|
| 114 |
-
"protected": 0.
|
| 115 |
"precision": 0.96,
|
| 116 |
"n_pred": 766
|
| 117 |
}
|
|
|
|
| 45 |
},
|
| 46 |
"ADDRESS": {
|
| 47 |
"n_gold": 298,
|
| 48 |
+
"recall": 0.752,
|
| 49 |
+
"protected": 0.826,
|
| 50 |
"precision": 0.971,
|
| 51 |
"n_pred": 138
|
| 52 |
},
|
| 53 |
"ALL": {
|
| 54 |
"n_gold": 2065,
|
| 55 |
+
"recall": 0.938,
|
| 56 |
"protected": 0.949,
|
| 57 |
"precision": 0.93,
|
| 58 |
"n_pred": 1725
|
|
|
|
| 103 |
},
|
| 104 |
"ADDRESS": {
|
| 105 |
"n_gold": 298,
|
| 106 |
+
"recall": 0.752,
|
| 107 |
+
"protected": 0.826,
|
| 108 |
"precision": 0.971,
|
| 109 |
"n_pred": 138
|
| 110 |
},
|
| 111 |
"ALL": {
|
| 112 |
"n_gold": 2065,
|
| 113 |
"recall": 0.399,
|
| 114 |
+
"protected": 0.412,
|
| 115 |
"precision": 0.96,
|
| 116 |
"n_pred": 766
|
| 117 |
}
|
eval/results.md
CHANGED
|
@@ -22,7 +22,7 @@ Mövzu: Unikal İmza Analizi Məktubu - Zəhmət olmasa, təhlil üçün yazıl
|
|
| 22 |
MISS SURNAME: 'İsmayılqızı' :: Mənim adım Ceyla İsmayılqızıdır və mənim 57 yaşım var. Bu təlim qrupunun bir hissəsi olmaqdan çox həyəcanlıyam.
|
| 23 |
MISS BUILDINGNUM: '5' :: Resept formasında, zəhmət olmasa, sahələri 0U51BU4, Nisə və 5 ilə doldurun.
|
| 24 |
MISS STREET: 'BAKI-BATUMİ küç.' :: Nəzərinizə çatdırırıq ki, Lətifə Məhərrəmzadə yüngül dəmir yolu stansiyası, BAKI-BATUMİ küç. 0 qazıntı sahəsinə ən yaxın
|
| 25 |
-
| Pərdə without name detector | 41.
|
| 26 |
| Generic regex baseline | 23.9% | 76.1% | 96.4% |
|
| 27 |
Per entity (Pərdə):
|
| 28 |
| Entity | Gold | Recall (right type) | Protected | Precision |
|
|
@@ -33,7 +33,7 @@ Per entity (Pərdə):
|
|
| 33 |
| ID_NUMBER | 165 | 100.0% | 100.0% | 100.0% |
|
| 34 |
| CARD | 25 | 100.0% | 100.0% | 100.0% |
|
| 35 |
| DATE | 117 | 80.3% | 80.3% | 78.5% |
|
| 36 |
-
| ADDRESS | 298 |
|
| 37 |
### Hand-written AZ/RU/EN cases (20 texts)
|
| 38 |
| System | Protected (all PII) | Leak rate | Precision |
|
| 39 |
|---|---|---|---|
|
|
|
|
| 22 |
MISS SURNAME: 'İsmayılqızı' :: Mənim adım Ceyla İsmayılqızıdır və mənim 57 yaşım var. Bu təlim qrupunun bir hissəsi olmaqdan çox həyəcanlıyam.
|
| 23 |
MISS BUILDINGNUM: '5' :: Resept formasında, zəhmət olmasa, sahələri 0U51BU4, Nisə və 5 ilə doldurun.
|
| 24 |
MISS STREET: 'BAKI-BATUMİ küç.' :: Nəzərinizə çatdırırıq ki, Lətifə Məhərrəmzadə yüngül dəmir yolu stansiyası, BAKI-BATUMİ küç. 0 qazıntı sahəsinə ən yaxın
|
| 25 |
+
| Pərdə without name detector | 41.2% | 58.8% | 96.0% |
|
| 26 |
| Generic regex baseline | 23.9% | 76.1% | 96.4% |
|
| 27 |
Per entity (Pərdə):
|
| 28 |
| Entity | Gold | Recall (right type) | Protected | Precision |
|
|
|
|
| 33 |
| ID_NUMBER | 165 | 100.0% | 100.0% | 100.0% |
|
| 34 |
| CARD | 25 | 100.0% | 100.0% | 100.0% |
|
| 35 |
| DATE | 117 | 80.3% | 80.3% | 78.5% |
|
| 36 |
+
| ADDRESS | 298 | 75.2% | 82.6% | 97.1% |
|
| 37 |
### Hand-written AZ/RU/EN cases (20 texts)
|
| 38 |
| System | Protected (all PII) | Leak rate | Precision |
|
| 39 |
|---|---|---|---|
|
index.html
CHANGED
|
@@ -4,7 +4,7 @@
|
|
| 4 |
<meta charset="UTF-8" />
|
| 5 |
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
| 6 |
<title>Pərdə · Safe AI for Azerbaijani data</title>
|
| 7 |
-
<meta name="description" content="Masks Azerbaijani personal data before it reaches an AI model. Runs
|
| 8 |
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@stlite/browser@1.9.2/build/stlite.css" />
|
| 9 |
<style>
|
| 10 |
body { margin: 0; background: #0e1117; color: #fafafa; font-family: system-ui, sans-serif; }
|
|
@@ -17,7 +17,7 @@
|
|
| 17 |
<div id="loading">
|
| 18 |
<h2>🛡️ Pərdə</h2>
|
| 19 |
<p>Loading Python in your browser (about 10–20 seconds on first visit)…</p>
|
| 20 |
-
<small>
|
| 21 |
</div>
|
| 22 |
</div>
|
| 23 |
<script type="module">
|
|
|
|
| 4 |
<meta charset="UTF-8" />
|
| 5 |
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
| 6 |
<title>Pərdə · Safe AI for Azerbaijani data</title>
|
| 7 |
+
<meta name="description" content="Masks Azerbaijani personal data before it reaches an AI model. Runs in your browser; no text is uploaded." />
|
| 8 |
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@stlite/browser@1.9.2/build/stlite.css" />
|
| 9 |
<style>
|
| 10 |
body { margin: 0; background: #0e1117; color: #fafafa; font-family: system-ui, sans-serif; }
|
|
|
|
| 17 |
<div id="loading">
|
| 18 |
<h2>🛡️ Pərdə</h2>
|
| 19 |
<p>Loading Python in your browser (about 10–20 seconds on first visit)…</p>
|
| 20 |
+
<small>Runs on your device: the Python runtime downloads once, your text is never uploaded.</small>
|
| 21 |
</div>
|
| 22 |
</div>
|
| 23 |
<script type="module">
|
perde/gateway.py
CHANGED
|
@@ -22,6 +22,12 @@ PROVIDERS = {
|
|
| 22 |
"model": "gemini-2.5-flash",
|
| 23 |
},
|
| 24 |
"ollama": {"base_url": "http://localhost:11434/v1", "key_env": None, "model": "gemma3"},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
"mock": {"base_url": None, "key_env": None, "model": "mock"},
|
| 26 |
}
|
| 27 |
|
|
@@ -75,7 +81,8 @@ def call_model(provider: str, prompt: str, model: str | None = None) -> tuple[st
|
|
| 75 |
key = os.environ.get(cfg["key_env"]) if cfg["key_env"] else "ollama"
|
| 76 |
if not key:
|
| 77 |
raise RuntimeError(f"Set {cfg['key_env']} to use provider '{provider}'.")
|
| 78 |
-
|
|
|
|
| 79 |
resp = client.chat.completions.create(
|
| 80 |
model=model,
|
| 81 |
messages=[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": prompt}],
|
|
@@ -85,9 +92,12 @@ def call_model(provider: str, prompt: str, model: str | None = None) -> tuple[st
|
|
| 85 |
|
| 86 |
|
| 87 |
def run(text: str, task: str, provider: str = "mock", entities=None, model: str | None = None) -> GatewayResult:
|
| 88 |
-
"""Mask
|
| 89 |
-
|
| 90 |
-
|
|
|
|
|
|
|
|
|
|
| 91 |
leaks = leaked_values(outgoing, m.mapping)
|
| 92 |
if leaks:
|
| 93 |
raise LeakBlocked(f"{len(leaks)} value(s) would leave unmasked; request blocked.")
|
|
@@ -100,13 +110,13 @@ def run(text: str, task: str, provider: str = "mock", entities=None, model: str
|
|
| 100 |
|
| 101 |
|
| 102 |
def _audit(m: MaskResult, outgoing: str, provider: str, model: str, latency_ms: int) -> None:
|
| 103 |
-
"""Append-only log. Stores
|
|
|
|
| 104 |
record = {
|
| 105 |
"ts": time.strftime("%Y-%m-%dT%H:%M:%S"),
|
| 106 |
"provider": provider,
|
| 107 |
"model": model,
|
| 108 |
"entities_masked": m.counts(),
|
| 109 |
-
"input_sha256": hashlib.sha256(m.original.encode()).hexdigest(),
|
| 110 |
"outgoing_sha256": hashlib.sha256(outgoing.encode()).hexdigest(),
|
| 111 |
"leak_check": "passed",
|
| 112 |
"latency_ms": latency_ms,
|
|
|
|
| 22 |
"model": "gemini-2.5-flash",
|
| 23 |
},
|
| 24 |
"ollama": {"base_url": "http://localhost:11434/v1", "key_env": None, "model": "gemma3"},
|
| 25 |
+
"llamacpp": {
|
| 26 |
+
"base_url": "http://127.0.0.1:8080/v1",
|
| 27 |
+
"base_url_env": "PERDE_LOCAL_BASE_URL",
|
| 28 |
+
"key_env": None,
|
| 29 |
+
"model": "gemma-4-E4B",
|
| 30 |
+
},
|
| 31 |
"mock": {"base_url": None, "key_env": None, "model": "mock"},
|
| 32 |
}
|
| 33 |
|
|
|
|
| 81 |
key = os.environ.get(cfg["key_env"]) if cfg["key_env"] else "ollama"
|
| 82 |
if not key:
|
| 83 |
raise RuntimeError(f"Set {cfg['key_env']} to use provider '{provider}'.")
|
| 84 |
+
base_url = os.environ.get(cfg.get("base_url_env", "")) or cfg["base_url"]
|
| 85 |
+
client = OpenAI(base_url=base_url, api_key=key, timeout=60)
|
| 86 |
resp = client.chat.completions.create(
|
| 87 |
model=model,
|
| 88 |
messages=[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": prompt}],
|
|
|
|
| 92 |
|
| 93 |
|
| 94 |
def run(text: str, task: str, provider: str = "mock", entities=None, model: str | None = None) -> GatewayResult:
|
| 95 |
+
"""Mask the task and the text together, block if anything leaks, send, restore the answer."""
|
| 96 |
+
combined = f"{task.strip()}\n\n---\n{text}" if task.strip() else text
|
| 97 |
+
if entities is not None and not entities and mask(combined).spans:
|
| 98 |
+
raise LeakBlocked("No data types are selected but personal data was detected; request blocked.")
|
| 99 |
+
m = mask(combined) if entities is None else mask(combined, tuple(entities))
|
| 100 |
+
outgoing = m.masked
|
| 101 |
leaks = leaked_values(outgoing, m.mapping)
|
| 102 |
if leaks:
|
| 103 |
raise LeakBlocked(f"{len(leaks)} value(s) would leave unmasked; request blocked.")
|
|
|
|
| 110 |
|
| 111 |
|
| 112 |
def _audit(m: MaskResult, outgoing: str, provider: str, model: str, latency_ms: int) -> None:
|
| 113 |
+
"""Append-only log. Stores counts and a hash of the already-masked outgoing text, never raw
|
| 114 |
+
personal data. No hash of the raw input: a short input (one phone number) could be brute-forced."""
|
| 115 |
record = {
|
| 116 |
"ts": time.strftime("%Y-%m-%dT%H:%M:%S"),
|
| 117 |
"provider": provider,
|
| 118 |
"model": model,
|
| 119 |
"entities_masked": m.counts(),
|
|
|
|
| 120 |
"outgoing_sha256": hashlib.sha256(outgoing.encode()).hexdigest(),
|
| 121 |
"leak_check": "passed",
|
| 122 |
"latency_ms": latency_ms,
|
perde/recognizers.py
CHANGED
|
@@ -60,7 +60,7 @@ def _context(text: str, start: int, end: int, words: tuple[str, ...], window: in
|
|
| 60 |
# --- pattern detectors ------------------------------------------------------
|
| 61 |
|
| 62 |
_PHONE = re.compile(
|
| 63 |
-
r"(?<![\d+])(?:\+?\s?994|8|0)[\s\-.]?\(?(?:10|50|51|55|60|70|77|99|12|18|2\d|36)\)?"
|
| 64 |
r"[\s\-.]?\d{3}[\s\-.]?\d{2}[\s\-.]?\d{2}(?!\d)"
|
| 65 |
)
|
| 66 |
_EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+(?:\.[\w-]+)+\b")
|
|
@@ -88,7 +88,7 @@ _NAME_WORD = rf"(?:[{AZ_UPPER}]\.\s?)?[{AZ_UPPER}][{AZ_UPPER}{AZ_LOWER}]+"
|
|
| 88 |
_NUM = r"\d{1,4}[A-Za-z]?"
|
| 89 |
_ADDRESS = re.compile(
|
| 90 |
rf"(?:\b{_NUM},?\s)?{_NAME_WORD}(?:\s{_NAME_WORD}){{0,2}}\s{_STREET_WORD}"
|
| 91 |
-
rf"(?:,?\s*(?:ev|bina|mənzil|m\.)?\s*{_NUM}(?!
|
| 92 |
rf"(?:\s*,?\s*(?:mənzil|m\.)\s*\d{{1,4}})?"
|
| 93 |
r"|(?:ул\.|улица|проспект|пр-т)\s?[А-ЯЁ][а-яё]+(?:\s[А-ЯЁ][а-яё]+)?(?:,?\s*(?:д\.)?\s*\d{1,4})?"
|
| 94 |
)
|
|
|
|
| 60 |
# --- pattern detectors ------------------------------------------------------
|
| 61 |
|
| 62 |
_PHONE = re.compile(
|
| 63 |
+
r"(?<![\d+(])\(?(?:\+?\s?994|8|0)[\s\-.]?\(?(?:10|50|51|55|60|70|77|99|12|18|2\d|36)\)?"
|
| 64 |
r"[\s\-.]?\d{3}[\s\-.]?\d{2}[\s\-.]?\d{2}(?!\d)"
|
| 65 |
)
|
| 66 |
_EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+(?:\.[\w-]+)+\b")
|
|
|
|
| 88 |
_NUM = r"\d{1,4}[A-Za-z]?"
|
| 89 |
_ADDRESS = re.compile(
|
| 90 |
rf"(?:\b{_NUM},?\s)?{_NAME_WORD}(?:\s{_NAME_WORD}){{0,2}}\s{_STREET_WORD}"
|
| 91 |
+
rf"(?:,?\s*(?:ev|bina|mənzil|m\.)?\s*{_NUM}(?!\d|\.\d))?"
|
| 92 |
rf"(?:\s*,?\s*(?:mənzil|m\.)\s*\d{{1,4}})?"
|
| 93 |
r"|(?:ул\.|улица|проспект|пр-т)\s?[А-ЯЁ][а-яё]+(?:\s[А-ЯЁ][а-яё]+)?(?:,?\s*(?:д\.)?\s*\d{1,4})?"
|
| 94 |
)
|
perde/shield.py
CHANGED
|
@@ -50,7 +50,7 @@ def mask(text: str, entities: tuple[str, ...] = ALL_ENTITIES, min_score: float =
|
|
| 50 |
parts: list[str] = []
|
| 51 |
cursor = 0
|
| 52 |
for s in spans:
|
| 53 |
-
key = (s
|
| 54 |
tag = value_to_tag.get(key)
|
| 55 |
if tag is None:
|
| 56 |
counters[s.entity] = counters.get(s.entity, 0) + 1
|
|
@@ -63,7 +63,7 @@ def mask(text: str, entities: tuple[str, ...] = ALL_ENTITIES, min_score: float =
|
|
| 63 |
result = MaskResult(text, "".join(parts), spans)
|
| 64 |
seen: set[str] = set()
|
| 65 |
for s in spans:
|
| 66 |
-
tag = value_to_tag[(s
|
| 67 |
if tag not in seen:
|
| 68 |
result.mapping[tag] = s.text
|
| 69 |
seen.add(tag)
|
|
@@ -84,19 +84,27 @@ def restore(answer: str, mapping: dict[str, str]) -> str:
|
|
| 84 |
def leaked_values(outgoing: str, mapping: dict[str, str]) -> list[str]:
|
| 85 |
"""Return original values that still appear in text about to leave the machine."""
|
| 86 |
out_norm = _normalize(outgoing)
|
| 87 |
-
|
|
|
|
| 88 |
leaks = []
|
| 89 |
for value in mapping.values():
|
| 90 |
v = _normalize(value)
|
| 91 |
digits = re.sub(r"\D", "", value)
|
| 92 |
# Compare the last 9 digits so "+994 55..." and "055..." count as the same number.
|
| 93 |
core = digits[-9:]
|
| 94 |
-
if v and v
|
| 95 |
leaks.append(value)
|
| 96 |
-
elif len(core) >= 7 and core in
|
| 97 |
leaks.append(value)
|
| 98 |
return leaks
|
| 99 |
|
| 100 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 101 |
def _normalize(s: str) -> str:
|
| 102 |
return re.sub(r"\s+", " ", s).strip().lower()
|
|
|
|
| 50 |
parts: list[str] = []
|
| 51 |
cursor = 0
|
| 52 |
for s in spans:
|
| 53 |
+
key = _key(s)
|
| 54 |
tag = value_to_tag.get(key)
|
| 55 |
if tag is None:
|
| 56 |
counters[s.entity] = counters.get(s.entity, 0) + 1
|
|
|
|
| 63 |
result = MaskResult(text, "".join(parts), spans)
|
| 64 |
seen: set[str] = set()
|
| 65 |
for s in spans:
|
| 66 |
+
tag = value_to_tag[_key(s)]
|
| 67 |
if tag not in seen:
|
| 68 |
result.mapping[tag] = s.text
|
| 69 |
seen.add(tag)
|
|
|
|
| 84 |
def leaked_values(outgoing: str, mapping: dict[str, str]) -> list[str]:
|
| 85 |
"""Return original values that still appear in text about to leave the machine."""
|
| 86 |
out_norm = _normalize(outgoing)
|
| 87 |
+
# digits of each separate number in the outgoing text, so unrelated numbers never join up
|
| 88 |
+
out_numbers = [re.sub(r"\D", "", n) for n in re.findall(r"\+?\d[\d\s\-().]*\d", outgoing)]
|
| 89 |
leaks = []
|
| 90 |
for value in mapping.values():
|
| 91 |
v = _normalize(value)
|
| 92 |
digits = re.sub(r"\D", "", value)
|
| 93 |
# Compare the last 9 digits so "+994 55..." and "055..." count as the same number.
|
| 94 |
core = digits[-9:]
|
| 95 |
+
if v and re.search(rf"(?<!\w){re.escape(v)}(?!\w)", out_norm):
|
| 96 |
leaks.append(value)
|
| 97 |
+
elif len(core) >= 7 and any(core in n for n in out_numbers):
|
| 98 |
leaks.append(value)
|
| 99 |
return leaks
|
| 100 |
|
| 101 |
|
| 102 |
+
def _key(span: Span) -> tuple[str, str]:
|
| 103 |
+
"""Same person or number gets the same tag: phones compare by their last 9 digits."""
|
| 104 |
+
if span.entity == "PHONE":
|
| 105 |
+
return span.entity, re.sub(r"\D", "", span.text)[-9:]
|
| 106 |
+
return span.entity, _normalize(span.text)
|
| 107 |
+
|
| 108 |
+
|
| 109 |
def _normalize(s: str) -> str:
|
| 110 |
return re.sub(r"\s+", " ", s).strip().lower()
|
tests/test_shield.py
CHANGED
|
@@ -15,6 +15,14 @@ def test_az_phone_formats():
|
|
| 15 |
assert "PHONE" in entities(f"Əlaqə: {phone}"), phone
|
| 16 |
|
| 17 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
def test_iban_checksum():
|
| 19 |
assert iban_ok("AZ21NABZ00000000137010001944")
|
| 20 |
assert not iban_ok("AZ21NABZ00000000137010001945")
|
|
@@ -43,6 +51,11 @@ def test_same_value_gets_same_tag():
|
|
| 43 |
assert r.masked.count("[PERSON_1]") == 2
|
| 44 |
|
| 45 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
def test_restore_tolerates_model_formatting():
|
| 47 |
r = mask("Telefon +994 55 765 43 21")
|
| 48 |
assert restore("Nömrə: [ phone_1 ]", r.mapping) == "Nömrə: +994 55 765 43 21"
|
|
@@ -54,6 +67,28 @@ def test_no_leak_after_masking():
|
|
| 54 |
assert leaked_values(r.masked, r.mapping) == []
|
| 55 |
|
| 56 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
def test_leak_check_catches_unmasked_digits():
|
| 58 |
r = mask("Telefon +994 55 765 43 21")
|
| 59 |
assert leaked_values("call 0557654321", r.mapping)
|
|
|
|
| 15 |
assert "PHONE" in entities(f"Əlaqə: {phone}"), phone
|
| 16 |
|
| 17 |
|
| 18 |
+
def test_bracketed_landline_masked_with_its_bracket():
|
| 19 |
+
assert entities("Tel: (012) 555 12 34")["PHONE"] == "(012) 555 12 34"
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def test_house_number_before_full_stop_is_masked():
|
| 23 |
+
assert entities("Ünvan: Sülh küçəsi 12. Tel yoxdur")["ADDRESS"] == "Sülh küçəsi 12"
|
| 24 |
+
|
| 25 |
+
|
| 26 |
def test_iban_checksum():
|
| 27 |
assert iban_ok("AZ21NABZ00000000137010001944")
|
| 28 |
assert not iban_ok("AZ21NABZ00000000137010001945")
|
|
|
|
| 51 |
assert r.masked.count("[PERSON_1]") == 2
|
| 52 |
|
| 53 |
|
| 54 |
+
def test_same_phone_in_two_formats_gets_one_tag():
|
| 55 |
+
r = mask("Zəng: 050 123 45 67 və ya +994501234567")
|
| 56 |
+
assert r.masked.count("[PHONE_1]") == 2
|
| 57 |
+
|
| 58 |
+
|
| 59 |
def test_restore_tolerates_model_formatting():
|
| 60 |
r = mask("Telefon +994 55 765 43 21")
|
| 61 |
assert restore("Nömrə: [ phone_1 ]", r.mapping) == "Nömrə: +994 55 765 43 21"
|
|
|
|
| 67 |
assert leaked_values(r.masked, r.mapping) == []
|
| 68 |
|
| 69 |
|
| 70 |
+
def test_leak_check_ignores_values_inside_other_words_and_numbers():
|
| 71 |
+
r = mask("Əli zəng etdi, tel +994 55 765 43 21")
|
| 72 |
+
assert leaked_values("Bəli/xeyr cavabı ver", r.mapping) == []
|
| 73 |
+
assert leaked_values("Tarix 2024-55-76, kod 54 321", r.mapping) == []
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def test_gateway_masks_personal_data_in_the_task_too(tmp_path, monkeypatch):
|
| 77 |
+
from perde import gateway
|
| 78 |
+
monkeypatch.setattr(gateway, "AUDIT_PATH", tmp_path / "audit.jsonl")
|
| 79 |
+
res = gateway.run("Tel 055 412 33 90", "Reply to Rəşad Həsənov, FIN 7XK2M9P", provider="mock")
|
| 80 |
+
for value in ("Rəşad Həsənov", "7XK2M9P", "055 412 33 90"):
|
| 81 |
+
assert value not in res.outgoing
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
def test_gateway_blocks_when_no_data_types_selected(tmp_path, monkeypatch):
|
| 85 |
+
import pytest
|
| 86 |
+
from perde import gateway
|
| 87 |
+
monkeypatch.setattr(gateway, "AUDIT_PATH", tmp_path / "audit.jsonl")
|
| 88 |
+
with pytest.raises(gateway.LeakBlocked):
|
| 89 |
+
gateway.run("Nərmin Quliyeva FIN 6TR9K2L", "", provider="mock", entities=[])
|
| 90 |
+
|
| 91 |
+
|
| 92 |
def test_leak_check_catches_unmasked_digits():
|
| 93 |
r = mask("Telefon +994 55 765 43 21")
|
| 94 |
assert leaked_values("call 0557654321", r.mapping)
|
tools/video/.gitignore
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
shots/
|
tools/video/README.md
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Demo video tooling
|
| 2 |
+
|
| 3 |
+
Used to make `media/perde-demo.mp4`.
|
| 4 |
+
|
| 5 |
+
1. Start the app: `streamlit run app.py` (port 8501).
|
| 6 |
+
2. `python tools/video/capture.py tools/video/shots` takes real screenshots of the running app with headless Chrome.
|
| 7 |
+
3. `python tools/video/render.py . media/perde-demo.mp4` renders 56 s at 1080p30, with a soundtrack synthesized in code.
|
| 8 |
+
|
| 9 |
+
Needs `pillow numpy imageio-ffmpeg websockets` and Google Chrome. Fonts: Segoe UI and Cascadia Mono (Windows).
|
tools/video/capture.py
ADDED
|
@@ -0,0 +1,69 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Capture real screenshots of the running Pərdə app with headless Chrome (CDP)."""
|
| 2 |
+
import base64
|
| 3 |
+
import json
|
| 4 |
+
import subprocess
|
| 5 |
+
import sys
|
| 6 |
+
import time
|
| 7 |
+
import urllib.request
|
| 8 |
+
from pathlib import Path
|
| 9 |
+
|
| 10 |
+
from websockets.sync.client import connect
|
| 11 |
+
|
| 12 |
+
OUT = Path(sys.argv[1])
|
| 13 |
+
OUT.mkdir(parents=True, exist_ok=True)
|
| 14 |
+
CHROME = r"C:\Program Files\Google\Chrome\Application\chrome.exe"
|
| 15 |
+
PROFILE = OUT / "_chrome_profile"
|
| 16 |
+
|
| 17 |
+
proc = subprocess.Popen([
|
| 18 |
+
CHROME, "--headless=new", "--remote-debugging-port=9333", f"--user-data-dir={PROFILE}",
|
| 19 |
+
"--window-size=1920,1080", "--hide-scrollbars", "--force-dark-mode", "about:blank",
|
| 20 |
+
])
|
| 21 |
+
try:
|
| 22 |
+
for _ in range(50):
|
| 23 |
+
try:
|
| 24 |
+
tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9333/json"))
|
| 25 |
+
page = next(t for t in tabs if t["type"] == "page")
|
| 26 |
+
break
|
| 27 |
+
except Exception:
|
| 28 |
+
time.sleep(0.2)
|
| 29 |
+
ws = connect(page["webSocketDebuggerUrl"], max_size=50_000_000)
|
| 30 |
+
n = 0
|
| 31 |
+
|
| 32 |
+
def cmd(method, **params):
|
| 33 |
+
global n
|
| 34 |
+
n += 1
|
| 35 |
+
ws.send(json.dumps({"id": n, "method": method, "params": params}))
|
| 36 |
+
while True:
|
| 37 |
+
msg = json.loads(ws.recv())
|
| 38 |
+
if msg.get("id") == n:
|
| 39 |
+
return msg.get("result", {})
|
| 40 |
+
|
| 41 |
+
def js(expr):
|
| 42 |
+
return cmd("Runtime.evaluate", expression=expr, awaitPromise=True, returnByValue=True)
|
| 43 |
+
|
| 44 |
+
def shot(name):
|
| 45 |
+
data = cmd("Page.captureScreenshot", format="png")["data"]
|
| 46 |
+
(OUT / f"{name}.png").write_bytes(base64.b64decode(data))
|
| 47 |
+
print("saved", name)
|
| 48 |
+
|
| 49 |
+
cmd("Emulation.setDeviceMetricsOverride", width=1920, height=1080, deviceScaleFactor=1, mobile=False)
|
| 50 |
+
cmd("Emulation.setEmulatedMedia", features=[{"name": "prefers-color-scheme", "value": "dark"}])
|
| 51 |
+
cmd("Page.enable")
|
| 52 |
+
cmd("Page.navigate", url="http://localhost:8501/")
|
| 53 |
+
time.sleep(8)
|
| 54 |
+
shot("01_start")
|
| 55 |
+
js("[...document.querySelectorAll('button')].find(b => b.innerText.includes('Protect and send')).click()")
|
| 56 |
+
time.sleep(4)
|
| 57 |
+
shot("02_after_send")
|
| 58 |
+
js("window.scrollBy(0, 380)")
|
| 59 |
+
js("document.querySelector('section.main, [data-testid=stMain]')?.scrollBy(0, 380)")
|
| 60 |
+
time.sleep(1)
|
| 61 |
+
shot("03_scrolled")
|
| 62 |
+
js("[...document.querySelectorAll('button[role=tab]')].find(b => b.innerText.includes('Evidence')).click()")
|
| 63 |
+
time.sleep(2)
|
| 64 |
+
js("document.querySelector('[data-testid=stMain]')?.scrollTo(0, 0)")
|
| 65 |
+
time.sleep(1)
|
| 66 |
+
shot("04_evidence")
|
| 67 |
+
ws.close()
|
| 68 |
+
finally:
|
| 69 |
+
proc.terminate()
|
tools/video/render.py
ADDED
|
@@ -0,0 +1,422 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Render the Pərdə promo video (1920x1080, 30 fps, with a synthesized soundtrack)."""
|
| 2 |
+
import math
|
| 3 |
+
import subprocess
|
| 4 |
+
import sys
|
| 5 |
+
import wave
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
|
| 8 |
+
import numpy as np
|
| 9 |
+
from PIL import Image, ImageDraw, ImageFont, ImageFilter
|
| 10 |
+
|
| 11 |
+
HERE = Path(__file__).parent
|
| 12 |
+
REPO = Path(sys.argv[1])
|
| 13 |
+
OUT = Path(sys.argv[2])
|
| 14 |
+
sys.path.insert(0, str(REPO))
|
| 15 |
+
from perde import mask # noqa: E402 real masking output, not mocked
|
| 16 |
+
import imageio_ffmpeg # noqa: E402
|
| 17 |
+
|
| 18 |
+
W, H, FPS = 1920, 1080, 30
|
| 19 |
+
DUR = 56.0
|
| 20 |
+
FONTS = Path(r"C:\Windows\Fonts")
|
| 21 |
+
BOLD = str(FONTS / "segoeuib.ttf")
|
| 22 |
+
REG = str(FONTS / "segoeui.ttf")
|
| 23 |
+
LIGHT = str(FONTS / "segoeuisl.ttf")
|
| 24 |
+
MONO = str(FONTS / "CascadiaMono.ttf")
|
| 25 |
+
|
| 26 |
+
INK = (15, 23, 36)
|
| 27 |
+
PANEL = (24, 35, 58)
|
| 28 |
+
LIGHTC = (245, 241, 232)
|
| 29 |
+
MUTED = (174, 184, 201)
|
| 30 |
+
RED = (232, 80, 91)
|
| 31 |
+
AMBER = (242, 166, 90)
|
| 32 |
+
GREEN = (127, 200, 169)
|
| 33 |
+
|
| 34 |
+
_font_cache = {}
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def F(path, size):
|
| 38 |
+
key = (path, size)
|
| 39 |
+
if key not in _font_cache:
|
| 40 |
+
_font_cache[key] = ImageFont.truetype(path, size)
|
| 41 |
+
return _font_cache[key]
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def clamp(x, a=0.0, b=1.0):
|
| 45 |
+
return max(a, min(b, x))
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def ease(x):
|
| 49 |
+
x = clamp(x)
|
| 50 |
+
return 1 - (1 - x) ** 3
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def ease_io(x):
|
| 54 |
+
x = clamp(x)
|
| 55 |
+
return x * x * (3 - 2 * x)
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def mix(c1, c2, t):
|
| 59 |
+
return tuple(int(a + (b - a) * clamp(t)) for a, b in zip(c1, c2))
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def text_center(d, y, s, font, fill, alpha=1.0, bg=INK):
|
| 63 |
+
w = d.textlength(s, font=font)
|
| 64 |
+
d.text(((W - w) / 2, y), s, font=font, fill=mix(bg, fill, alpha))
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
# --- message layout --------------------------------------------------------
|
| 68 |
+
|
| 69 |
+
MSG = ("Salam, mən Nərmin Quliyeva. Kartımdan 2 dəfə 150 AZN çıxılıb, kart 4111 1111 1111 1111. "
|
| 70 |
+
"FIN kodum 6TR9K2L, hesabım AZ21NABZ00000000137010001944. Zəng edin: 055 412 33 90.")
|
| 71 |
+
RES = mask(MSG)
|
| 72 |
+
SPANS = RES.spans # real detections
|
| 73 |
+
MASKED = RES.masked
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def layout(text, font, max_w, x0, y0, lh):
|
| 77 |
+
"""Word-wrap; return list of (x, y, ch, index)."""
|
| 78 |
+
out, x, y = [], x0, y0
|
| 79 |
+
space = font.getlength(" ")
|
| 80 |
+
i = 0
|
| 81 |
+
for word in text.split(" "):
|
| 82 |
+
ww = font.getlength(word)
|
| 83 |
+
if x + ww > x0 + max_w and x > x0:
|
| 84 |
+
x, y = x0, y + lh
|
| 85 |
+
for ch in word:
|
| 86 |
+
out.append((x, y, ch, i))
|
| 87 |
+
x += font.getlength(ch)
|
| 88 |
+
i += 1
|
| 89 |
+
out.append((x, y, " ", i))
|
| 90 |
+
x += space
|
| 91 |
+
i += 1
|
| 92 |
+
return out
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
MSG_FONT = F(MONO, 40)
|
| 96 |
+
MSG_BOX = (180, 330, 1740, 600)
|
| 97 |
+
MSG_LAYOUT = layout(MSG, MSG_FONT, 1480, 220, 370, 64)
|
| 98 |
+
MASKED_LAYOUT = layout(MASKED, MSG_FONT, 1480, 220, 370, 64)
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
def span_of(idx):
|
| 102 |
+
for k, s in enumerate(SPANS):
|
| 103 |
+
if s.start <= idx < s.end:
|
| 104 |
+
return k
|
| 105 |
+
return None
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def tag_ranges(text):
|
| 109 |
+
import re
|
| 110 |
+
return [(m.start(), m.end()) for m in re.finditer(r"\[[A-Z_]+_\d+\]", text)]
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
TAGS = tag_ranges(MASKED)
|
| 114 |
+
|
| 115 |
+
# --- scenes ----------------------------------------------------------------
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def scene_hook(img, d, t):
|
| 119 |
+
"""0-10s: typed message, then personal data lights up red."""
|
| 120 |
+
cap = "A bank clerk pastes a complaint into a chatbot…"
|
| 121 |
+
n = int(len(cap) * ease(t / 1.2))
|
| 122 |
+
text_center(d, 170, cap[:n], F(BOLD, 60), LIGHTC)
|
| 123 |
+
a = ease((t - 0.6) / 0.5)
|
| 124 |
+
d.rounded_rectangle(MSG_BOX, 28, fill=mix(INK, PANEL, a), outline=mix(INK, (44, 58, 85), a), width=2)
|
| 125 |
+
typed = int(len(MSG) * clamp((t - 1.0) / 3.2))
|
| 126 |
+
for x, y, ch, i in MSG_LAYOUT:
|
| 127 |
+
if i >= typed:
|
| 128 |
+
break
|
| 129 |
+
col = (220, 227, 238)
|
| 130 |
+
k = span_of(i)
|
| 131 |
+
if k is not None:
|
| 132 |
+
on = ease((t - (5.4 + 0.45 * k)) / 0.25)
|
| 133 |
+
col = mix(col, RED, on)
|
| 134 |
+
if on > 0:
|
| 135 |
+
d.rectangle((x, y + 52, x + MSG_FONT.getlength(ch), y + 56), fill=mix(PANEL, RED, on))
|
| 136 |
+
d.text((x, y), ch, font=MSG_FONT, fill=col)
|
| 137 |
+
if typed < len(MSG) and int(t * 4) % 2 == 0 and t > 1:
|
| 138 |
+
cx, cy, _, _ = MSG_LAYOUT[min(typed, len(MSG_LAYOUT) - 1)]
|
| 139 |
+
d.rectangle((cx, cy + 4, cx + 4, cy + 50), fill=AMBER)
|
| 140 |
+
if t > 8.2:
|
| 141 |
+
a = ease((t - 8.2) / 0.5)
|
| 142 |
+
text_center(d, 700, f"{len(SPANS)} pieces of personal data just left the building.", F(BOLD, 56), RED, a)
|
| 143 |
+
|
| 144 |
+
|
| 145 |
+
def scene_question(img, d, t):
|
| 146 |
+
"""10-13s."""
|
| 147 |
+
s = 1 + 0.08 * ease(t / 3)
|
| 148 |
+
f = F(BOLD, int(110 * s))
|
| 149 |
+
a = ease(t / 0.4)
|
| 150 |
+
text_center(d, 380 - 20 * (s - 1) * 10, "What if the AI", f, LIGHTC, a)
|
| 151 |
+
text_center(d, 520 - 20 * (s - 1) * 10, "never saw them?", f, RED, ease((t - 0.5) / 0.4))
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
def curtain(d, t, reverse=False):
|
| 155 |
+
"""Vertical crimson-to-amber stripes sweeping across."""
|
| 156 |
+
n = 12
|
| 157 |
+
sw = W / n
|
| 158 |
+
for k in range(n):
|
| 159 |
+
p = ease((t - k * 0.04) / 0.5)
|
| 160 |
+
if reverse:
|
| 161 |
+
p = 1 - p
|
| 162 |
+
h = H * p
|
| 163 |
+
col = mix(RED, AMBER, k / n)
|
| 164 |
+
d.rectangle((k * sw, 0, (k + 1) * sw + 1, h), fill=col)
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
def scene_logo(img, d, t):
|
| 168 |
+
"""13-16s: curtain closes, logo appears."""
|
| 169 |
+
if t < 0.9:
|
| 170 |
+
curtain(d, t)
|
| 171 |
+
return
|
| 172 |
+
d.rectangle((0, 0, W, H), fill=INK)
|
| 173 |
+
for k, op in enumerate([1, 0.6, 0.35, 0.15]):
|
| 174 |
+
x = 1340 + k * 60
|
| 175 |
+
d.rectangle((x, 0, x + 28, H), fill=mix(INK, mix(RED, AMBER, 0.3), op * ease((t - 0.9) / 0.6)))
|
| 176 |
+
a = ease((t - 1.0) / 0.5)
|
| 177 |
+
d.text((180, 330), "Pərdə", font=F(BOLD, 230), fill=mix(INK, LIGHTC, a))
|
| 178 |
+
b = ease((t - 1.5) / 0.5)
|
| 179 |
+
d.text((190, 640), "the curtain between your customers", font=F(LIGHT, 60), fill=mix(INK, MUTED, b))
|
| 180 |
+
d.text((190, 715), "and the cloud.", font=F(LIGHT, 60), fill=mix(INK, MUTED, b))
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
def scene_morph(img, d, t):
|
| 184 |
+
"""16-22s: personal data turns into tags, live."""
|
| 185 |
+
text_center(d, 170, "Personal data becomes tags — on your own machine.", F(BOLD, 56), LIGHTC, ease(t / 0.4))
|
| 186 |
+
d.rounded_rectangle(MSG_BOX, 28, fill=PANEL, outline=(44, 58, 85), width=2)
|
| 187 |
+
p = clamp((t - 0.8) / 3.0)
|
| 188 |
+
if p < 0.5:
|
| 189 |
+
for x, y, ch, i in MSG_LAYOUT:
|
| 190 |
+
k = span_of(i)
|
| 191 |
+
col = RED if k is not None else (220, 227, 238)
|
| 192 |
+
if k is not None and p > 0.08 * k:
|
| 193 |
+
col = mix(RED, PANEL, (p - 0.08 * k) * 8)
|
| 194 |
+
d.text((x, y), ch, font=MSG_FONT, fill=col)
|
| 195 |
+
else:
|
| 196 |
+
q = ease((p - 0.5) * 2.5)
|
| 197 |
+
for x, y, ch, i in MASKED_LAYOUT:
|
| 198 |
+
is_tag = any(a <= i < b for a, b in TAGS)
|
| 199 |
+
col = mix(PANEL, GREEN, q) if is_tag else (220, 227, 238)
|
| 200 |
+
d.text((x, y), ch, font=MSG_FONT, fill=col)
|
| 201 |
+
if t > 4.0:
|
| 202 |
+
a = ease((t - 4.0) / 0.5)
|
| 203 |
+
text_center(d, 700, "The AI provider sees only this.", F(BOLD, 56), GREEN, a)
|
| 204 |
+
|
| 205 |
+
|
| 206 |
+
APP = Image.open(HERE / "shots" / "02_after_send.png").convert("RGB")
|
| 207 |
+
# crop rectangles on the 1920x1080 app screenshot: (x, y, w) with 16:9 height
|
| 208 |
+
SHOTS = [
|
| 209 |
+
(0.0, (0, 0, 1920), None),
|
| 210 |
+
(2.2, (360, 590, 520), "1 · Detected on your device"),
|
| 211 |
+
(5.2, (852, 590, 520), "2 · The AI sees only tags"),
|
| 212 |
+
(8.2, (1344, 590, 520), "3 · Answer restored locally"),
|
| 213 |
+
(11.2, (360, 560, 1500), "Leak check: passed · 6 values masked"),
|
| 214 |
+
]
|
| 215 |
+
|
| 216 |
+
|
| 217 |
+
def scene_app(img, d, t):
|
| 218 |
+
"""22-36s: real app footage with zooms."""
|
| 219 |
+
for k in range(len(SHOTS) - 1, -1, -1):
|
| 220 |
+
if t >= SHOTS[k][0]:
|
| 221 |
+
break
|
| 222 |
+
t0, r1, cap = SHOTS[k]
|
| 223 |
+
r0 = SHOTS[k - 1][1] if k else r1
|
| 224 |
+
q = ease_io((t - t0) / 1.0)
|
| 225 |
+
x, y, w = (a + (b - a) * q for a, b in zip(r0, r1))
|
| 226 |
+
h = w * 9 / 16
|
| 227 |
+
x = clamp(x, 0, 1920 - w)
|
| 228 |
+
y = clamp(y, 0, 1080 - h)
|
| 229 |
+
frame = APP.crop((int(x), int(y), int(x + w), int(y + h))).resize((W, H), Image.BILINEAR)
|
| 230 |
+
img.paste(frame, (0, 0))
|
| 231 |
+
d2 = ImageDraw.Draw(img)
|
| 232 |
+
d2.rectangle((0, 0, 380, 64), fill=RED)
|
| 233 |
+
d2.text((28, 8), "REAL APP · LIVE", font=F(BOLD, 36), fill=LIGHTC)
|
| 234 |
+
if cap:
|
| 235 |
+
a = ease((t - t0 - 0.6) / 0.4)
|
| 236 |
+
bar_w = d2.textlength(cap, font=F(BOLD, 60)) + 96
|
| 237 |
+
d2.rounded_rectangle(((W - bar_w) / 2, 900, (W + bar_w) / 2, 1010), 24, fill=mix((255, 255, 255), INK, a))
|
| 238 |
+
d2.text(((W - bar_w) / 2 + 48, 912), cap, font=F(BOLD, 60), fill=mix((255, 255, 255), LIGHTC, a))
|
| 239 |
+
|
| 240 |
+
|
| 241 |
+
def scene_stats(img, d, t):
|
| 242 |
+
"""36-45s: the measured result."""
|
| 243 |
+
text_center(d, 120, "Tested on 1,000 Azerbaijani texts it had never seen", F(BOLD, 56), LIGHTC, ease(t / 0.4))
|
| 244 |
+
rows = [("Pərdə", 94.9, RED, 0.6), ("Generic filter", 23.9, MUTED, 1.0)]
|
| 245 |
+
for k, (name, val, col, start) in enumerate(rows):
|
| 246 |
+
y = 330 + k * 230
|
| 247 |
+
p = ease((t - start) / 1.8)
|
| 248 |
+
d.text((180, y), name, font=F(BOLD, 52), fill=LIGHTC)
|
| 249 |
+
d.rounded_rectangle((180, y + 80, 1740, y + 150), 16, fill=PANEL)
|
| 250 |
+
if p > 0:
|
| 251 |
+
d.rounded_rectangle((180, y + 80, 180 + 1560 * val / 100 * p, y + 150), 16, fill=col)
|
| 252 |
+
num = f"{val * p:.1f}%"
|
| 253 |
+
nw = d.textlength(num, font=F(BOLD, 72))
|
| 254 |
+
d.text((1740 - nw, y - 10), num, font=F(BOLD, 72), fill=col if k == 0 else MUTED)
|
| 255 |
+
if t > 3.4:
|
| 256 |
+
a = ease((t - 3.4) / 0.5)
|
| 257 |
+
text_center(d, 820, "of personal data protected", F(LIGHT, 52), MUTED, a)
|
| 258 |
+
text_center(d, 900, "Leak rate: 76.1% → 5.1%", F(BOLD, 64), AMBER, ease((t - 4.4) / 0.5))
|
| 259 |
+
|
| 260 |
+
|
| 261 |
+
CHIPS = ["FIN 6TR9K2L", "VÖEN", "AZ IBAN", "+994", "AZE / AA ID", "oğlu · qızı", "Rənanın", "Кириллица"]
|
| 262 |
+
|
| 263 |
+
|
| 264 |
+
def scene_az(img, d, t):
|
| 265 |
+
"""45-50s: built for Azerbaijan."""
|
| 266 |
+
text_center(d, 140, "Built for Azerbaijani data", F(BOLD, 72), LIGHTC, ease(t / 0.4))
|
| 267 |
+
f = F(MONO, 44)
|
| 268 |
+
x0, y, x = 180, 330, 180
|
| 269 |
+
for k, c in enumerate(CHIPS):
|
| 270 |
+
w = d.textlength(c, font=f) + 72
|
| 271 |
+
if x + w > W - 180:
|
| 272 |
+
x, y = x0, y + 140
|
| 273 |
+
a = ease((t - 0.3 - 0.18 * k) / 0.3)
|
| 274 |
+
if a > 0:
|
| 275 |
+
dy = 30 * (1 - a)
|
| 276 |
+
d.rounded_rectangle((x, y + dy, x + w, y + 100 + dy), 50, fill=mix(INK, PANEL, a), outline=mix(INK, RED, a), width=3)
|
| 277 |
+
d.text((x + 36, y + 22 + dy), c, font=f, fill=mix(INK, LIGHTC, a))
|
| 278 |
+
x += w + 28
|
| 279 |
+
if t > 2.4:
|
| 280 |
+
a = ease((t - 2.4) / 0.5)
|
| 281 |
+
text_center(d, 720, "0.14 ms per message · runs on-premises · no GPU", F(BOLD, 52), AMBER, a)
|
| 282 |
+
text_center(d, 810, "Works with OpenAI · Grok · Gemini · local Ollama", F(LIGHT, 48), MUTED, ease((t - 3.0) / 0.5))
|
| 283 |
+
|
| 284 |
+
|
| 285 |
+
def scene_close(img, d, t):
|
| 286 |
+
"""50-56s."""
|
| 287 |
+
if t < 0.9:
|
| 288 |
+
curtain(d, t)
|
| 289 |
+
return
|
| 290 |
+
d.rectangle((0, 0, W, H), fill=INK)
|
| 291 |
+
for k, op in enumerate([1, 0.6, 0.35, 0.15]):
|
| 292 |
+
x = 1340 + k * 60
|
| 293 |
+
d.rectangle((x, 0, x + 28, H), fill=mix(INK, mix(RED, AMBER, 0.3), op))
|
| 294 |
+
d.text((180, 250), "Let the AI read the request.", font=F(BOLD, 88), fill=mix(INK, LIGHTC, ease((t - 1.0) / 0.5)))
|
| 295 |
+
d.text((180, 370), "Never the person.", font=F(BOLD, 88), fill=mix(INK, RED, ease((t - 1.6) / 0.5)))
|
| 296 |
+
a = ease((t - 2.4) / 0.5)
|
| 297 |
+
d.text((180, 600), "Pərdə", font=F(BOLD, 120), fill=mix(INK, LIGHTC, a))
|
| 298 |
+
d.text((186, 770), "Try it: huggingface.co/spaces/Traxic/perde", font=F(REG, 44), fill=mix(INK, MUTED, a))
|
| 299 |
+
d.text((186, 840), "NeuroBridge 2026 · Enterprise track", font=F(REG, 36), fill=mix(INK, MUTED, a))
|
| 300 |
+
if t > DUR - 50.0 - 0.8:
|
| 301 |
+
fade = clamp((t - (DUR - 50.0 - 0.8)) / 0.8)
|
| 302 |
+
img.paste(Image.blend(img, Image.new("RGB", (W, H), (0, 0, 0)), fade))
|
| 303 |
+
|
| 304 |
+
|
| 305 |
+
SCENES = [
|
| 306 |
+
(0.0, 10.0, scene_hook),
|
| 307 |
+
(10.0, 13.0, scene_question),
|
| 308 |
+
(13.0, 16.0, scene_logo),
|
| 309 |
+
(16.0, 22.0, scene_morph),
|
| 310 |
+
(22.0, 36.0, scene_app),
|
| 311 |
+
(36.0, 45.0, scene_stats),
|
| 312 |
+
(45.0, 50.0, scene_az),
|
| 313 |
+
(50.0, DUR, scene_close),
|
| 314 |
+
]
|
| 315 |
+
TRANSITIONS = [s for s, _, _ in SCENES[1:]]
|
| 316 |
+
HIGHLIGHTS = [5.4 + 0.45 * k for k in range(len(SPANS))]
|
| 317 |
+
|
| 318 |
+
|
| 319 |
+
def frame(t):
|
| 320 |
+
img = Image.new("RGB", (W, H), INK)
|
| 321 |
+
d = ImageDraw.Draw(img)
|
| 322 |
+
for s, e, fn in SCENES:
|
| 323 |
+
if s <= t < e:
|
| 324 |
+
fn(img, d, t - s)
|
| 325 |
+
# quick fade-in at each scene start
|
| 326 |
+
if t - s < 0.15 and s > 0 and fn not in (scene_logo, scene_close):
|
| 327 |
+
img = Image.blend(Image.new("RGB", (W, H), INK), img, (t - s) / 0.15)
|
| 328 |
+
break
|
| 329 |
+
return img
|
| 330 |
+
|
| 331 |
+
|
| 332 |
+
# --- soundtrack --------------------------------------------------------------
|
| 333 |
+
|
| 334 |
+
def soundtrack(path):
|
| 335 |
+
sr = 44100
|
| 336 |
+
n = int(DUR * sr)
|
| 337 |
+
out = np.zeros(n)
|
| 338 |
+
rng = np.random.default_rng(7)
|
| 339 |
+
beat = 60 / 118
|
| 340 |
+
|
| 341 |
+
def add(sig, at):
|
| 342 |
+
i = int(at * sr)
|
| 343 |
+
j = min(n, i + len(sig))
|
| 344 |
+
if i < n:
|
| 345 |
+
out[i:j] += sig[: j - i]
|
| 346 |
+
|
| 347 |
+
tt = np.arange(int(0.25 * sr)) / sr
|
| 348 |
+
kick = np.sin(2 * np.pi * (45 + 90 * np.exp(-tt * 30)) * tt) * np.exp(-tt * 9)
|
| 349 |
+
hat_t = np.arange(int(0.04 * sr)) / sr
|
| 350 |
+
hat = rng.normal(0, 1, len(hat_t)) * np.exp(-hat_t * 90)
|
| 351 |
+
hat = np.diff(np.concatenate([[0], hat]))
|
| 352 |
+
bass_notes = [55.0, 55.0, 65.41, 49.0]
|
| 353 |
+
t_cur, k = 0.0, 0
|
| 354 |
+
while t_cur < DUR:
|
| 355 |
+
intro = t_cur < 10.0
|
| 356 |
+
if not intro or k % 2 == 0:
|
| 357 |
+
add(kick * (0.5 if intro else 0.9), t_cur)
|
| 358 |
+
add(hat * 0.12, t_cur + beat / 2)
|
| 359 |
+
if not intro:
|
| 360 |
+
f = bass_notes[(k // 4) % 4]
|
| 361 |
+
bt = np.arange(int(beat * 0.9 * sr)) / sr
|
| 362 |
+
bass = (np.sin(2 * np.pi * f * bt) + 0.3 * np.sin(4 * np.pi * f * bt)) * np.exp(-bt * 3) * 0.35
|
| 363 |
+
add(bass, t_cur)
|
| 364 |
+
t_cur += beat
|
| 365 |
+
k += 1
|
| 366 |
+
# typing clicks while the message types (1.0-4.2s)
|
| 367 |
+
ct = np.arange(int(0.012 * sr)) / sr
|
| 368 |
+
click = rng.normal(0, 1, len(ct)) * np.exp(-ct * 400) * 0.15
|
| 369 |
+
for at in np.arange(1.0, 4.2, 0.045):
|
| 370 |
+
add(click, at)
|
| 371 |
+
# pops on each red highlight
|
| 372 |
+
pt = np.arange(int(0.12 * sr)) / sr
|
| 373 |
+
pop = np.sin(2 * np.pi * 880 * pt) * np.exp(-pt * 30) * 0.35
|
| 374 |
+
for at in HIGHLIGHTS:
|
| 375 |
+
add(pop, at)
|
| 376 |
+
# whooshes on scene changes
|
| 377 |
+
wt = np.arange(int(0.6 * sr)) / sr
|
| 378 |
+
noise = rng.normal(0, 1, len(wt))
|
| 379 |
+
noise = np.convolve(noise, np.ones(30) / 30, mode="same")
|
| 380 |
+
whoosh = noise * np.sin(np.pi * wt / wt[-1]) ** 2 * 0.6
|
| 381 |
+
for at in TRANSITIONS:
|
| 382 |
+
add(whoosh, at - 0.3)
|
| 383 |
+
fade = np.ones(n)
|
| 384 |
+
fl = int(2.0 * sr)
|
| 385 |
+
fade[-fl:] = np.linspace(1, 0, fl)
|
| 386 |
+
out = out * fade
|
| 387 |
+
out = out / max(1e-9, np.abs(out).max()) * 0.8
|
| 388 |
+
pcm = (out * 32767).astype(np.int16)
|
| 389 |
+
stereo = np.repeat(pcm[:, None], 2, axis=1)
|
| 390 |
+
with wave.open(str(path), "wb") as w:
|
| 391 |
+
w.setnchannels(2)
|
| 392 |
+
w.setsampwidth(2)
|
| 393 |
+
w.setframerate(sr)
|
| 394 |
+
w.writeframes(stereo.tobytes())
|
| 395 |
+
|
| 396 |
+
|
| 397 |
+
def main():
|
| 398 |
+
wav = HERE / "track.wav"
|
| 399 |
+
soundtrack(wav)
|
| 400 |
+
ff = imageio_ffmpeg.get_ffmpeg_exe()
|
| 401 |
+
proc = subprocess.Popen(
|
| 402 |
+
[ff, "-y", "-f", "rawvideo", "-pix_fmt", "rgb24", "-s", f"{W}x{H}", "-r", str(FPS), "-i", "-",
|
| 403 |
+
"-i", str(wav), "-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p",
|
| 404 |
+
"-c:a", "aac", "-b:a", "160k", "-shortest", "-movflags", "+faststart", str(OUT)],
|
| 405 |
+
stdin=subprocess.PIPE, stderr=subprocess.DEVNULL,
|
| 406 |
+
)
|
| 407 |
+
total = int(DUR * FPS)
|
| 408 |
+
for i in range(total):
|
| 409 |
+
proc.stdin.write(frame(i / FPS).tobytes())
|
| 410 |
+
if i % 150 == 0:
|
| 411 |
+
print(f"frame {i}/{total}", flush=True)
|
| 412 |
+
proc.stdin.close()
|
| 413 |
+
proc.wait()
|
| 414 |
+
print("done", OUT)
|
| 415 |
+
|
| 416 |
+
|
| 417 |
+
if __name__ == "__main__":
|
| 418 |
+
if len(sys.argv) > 3: # preview stills: render.py REPO OUT t1 t2 ...
|
| 419 |
+
for ts in sys.argv[3:]:
|
| 420 |
+
frame(float(ts)).save(HERE / f"preview_{ts}.png")
|
| 421 |
+
else:
|
| 422 |
+
main()
|