Traxic commited on
Commit
fb70710
·
1 Parent(s): 588e04e

Sync Space with main (review fixes)

Browse files
.env.example CHANGED
@@ -1,6 +1,9 @@
 
1
  # Set one of these to use a real model; leave all empty to use the offline mock.
2
  OPENAI_API_KEY=
3
  XAI_API_KEY=
4
  GEMINI_API_KEY=
5
  # Optional: override the provider's default model name
6
  PERDE_MODEL=
 
 
 
1
+ # Not loaded automatically: export these in your shell, or pass them with docker run -e NAME=value.
2
  # Set one of these to use a real model; leave all empty to use the offline mock.
3
  OPENAI_API_KEY=
4
  XAI_API_KEY=
5
  GEMINI_API_KEY=
6
  # Optional: override the provider's default model name
7
  PERDE_MODEL=
8
+ # Optional: OpenAI-compatible local server URL for the llamacpp provider
9
+ PERDE_LOCAL_BASE_URL=http://127.0.0.1:8080/v1
.gitattributes ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ * text=auto
2
+ *.bat text eol=crlf
3
+ *.mp4 binary
DISCLOSURE.md CHANGED
@@ -1,7 +1,7 @@
1
  # Disclosure: models, data and components
2
 
3
  ## Written during the hackathon
4
- All code in `perde/`, `app.py`, `eval/` and `tests/`: the detectors, masking and restoration, gateway, UI, evaluation and tests.
5
  Pərdə does not depend on any third-party PII library. The general approach (pattern recognisers plus anonymise/restore) is a well-known one, used for example by Microsoft Presidio. We built our own implementation for Azerbaijani formats.
6
 
7
  ## Data
@@ -14,7 +14,8 @@ Pərdə does not depend on any third-party PII library. The general approach (pa
14
 
15
  ## AI models
16
  - **No model is used for detection.** Detection is rules and a lexicon, and runs locally.
17
- - **For the user's task:** any OpenAI-compatible chat model, chosen at runtime. Supported providers are OpenAI, xAI (Grok), Google Gemini, and local Ollama (for example `gemma3`). The default is `mock`, an offline stand-in with no external call, which is used in the demo video unless stated otherwise.
 
18
 
19
  ## Libraries
20
  | Library | Licence | Use |
@@ -23,6 +24,12 @@ Pərdə does not depend on any third-party PII library. The general approach (pa
23
  | openai (Python SDK) | Apache-2.0 | Client for OpenAI-compatible APIs |
24
  | pytest | MIT | Unit tests (dev only) |
25
  | python:3.12-slim Docker base image | PSF and Debian licences | Container build |
 
 
 
 
 
 
26
 
27
  ## Assistance
28
  The code and documentation were written with the help of an AI coding assistant (Claude).
@@ -30,3 +37,6 @@ The code and documentation were written with the help of an AI coding assistant
30
  ## Mocks and caches
31
  - The `mock` provider returns a templated reply built from the tags. It is labelled "[Offline demo model, no external API call]" in its output.
32
  - No results are cached. The evaluation numbers come from `eval/evaluate.py` and are stored in `eval/results.json`.
 
 
 
 
1
  # Disclosure: models, data and components
2
 
3
  ## Written during the hackathon
4
+ All code in `perde/`, `app.py`, `index.html`, `eval/`, `tests/`, `tools/video/`, `run.bat` and `Dockerfile`: the detectors, masking and restoration, gateway, UI, evaluation, tests, launchers and video tooling.
5
  Pərdə does not depend on any third-party PII library. The general approach (pattern recognisers plus anonymise/restore) is a well-known one, used for example by Microsoft Presidio. We built our own implementation for Azerbaijani formats.
6
 
7
  ## Data
 
14
 
15
  ## AI models
16
  - **No model is used for detection.** Detection is rules and a lexicon, and runs locally.
17
+ - **For the user's task:** any OpenAI-compatible chat model, chosen at runtime. Supported providers, with their default model names: OpenAI (`gpt-4o-mini`), xAI Grok (`grok-3-mini`), Google Gemini (`gemini-2.5-flash`), local Ollama (`gemma3`) and any llama.cpp OpenAI-compatible server (`gemma-4-E4B`). The default is `mock`, an offline stand-in with no external call. The live demo and the demo video use `mock`.
18
+ - **Tested end to end with a real model:** Gemma 4 E4B on a local llama.cpp server (see `TEST_RESULTS.md`).
19
 
20
  ## Libraries
21
  | Library | Licence | Use |
 
24
  | openai (Python SDK) | Apache-2.0 | Client for OpenAI-compatible APIs |
25
  | pytest | MIT | Unit tests (dev only) |
26
  | python:3.12-slim Docker base image | PSF and Debian licences | Container build |
27
+ | @stlite/browser 1.9.2 (via jsDelivr CDN) | Apache-2.0 | Runs the Streamlit app in the browser for the live demo |
28
+ | Pyodide (loaded by stlite) | MPL-2.0 | Python in WebAssembly for the live demo |
29
+ | Hugging Face Spaces (static) | Hosting service | Live demo hosting |
30
+ | Pillow, numpy, imageio-ffmpeg (FFmpeg) | HPND, BSD-3, BSD-2 (FFmpeg: LGPL/GPL) | Demo video rendering only |
31
+ | websockets, Google Chrome (headless) | BSD-3, proprietary | Screenshots for the demo video only |
32
+ | Segoe UI, Cascadia Mono fonts | Windows system fonts (Cascadia: OFL) | Text in the demo video |
33
 
34
  ## Assistance
35
  The code and documentation were written with the help of an AI coding assistant (Claude).
 
37
  ## Mocks and caches
38
  - The `mock` provider returns a templated reply built from the tags. It is labelled "[Offline demo model, no external API call]" in its output.
39
  - No results are cached. The evaluation numbers come from `eval/evaluate.py` and are stored in `eval/results.json`.
40
+
41
+ ## Demo video
42
+ `media/perde-demo.mp4` was rendered by `tools/video/render.py` during the hackathon (screenshots by `tools/video/capture.py`). The app footage is real screenshots of the running app, using the offline model. The tags shown come from the real masking engine. The soundtrack is synthesized in code, with no third-party music.
README.md CHANGED
@@ -34,8 +34,8 @@ qeydə alındı… 055 412 33 90 qeydə alındı… [PHONE_
34
 
35
  - **The user:** an operator who handles customer requests at a bank, a mobile operator, or a public service centre (complaints, applications, tickets).
36
  - **Today:** these teams either don't use AI at all, or staff copy customer data into public chatbots. Neither is acceptable. Azerbaijan's Law on Personal Data (2010) restricts handing personal data to third parties.
37
- - **With Pərdə:** the same staff use any AI model for summaries, replies, classification and translation. The provider only ever sees tags.
38
- - **The metric that matters:** the **leak rate**, meaning the share of personal values that reach the AI provider. It falls from 76.1% with a generic filter to 5.1% with Pərdə (held-out test, details in [TESTING.md](TESTING.md)). A second, exact-match check then **blocks** any request where a detected value would still leave unmasked.
39
 
40
  ## 2. Prototype and use of AI
41
 
@@ -45,10 +45,10 @@ The working core scenario is to paste a customer text, pick a sector and a task,
45
  2. The exact text the AI provider receives
46
  3. The provider's answer, with the real values restored
47
 
48
- **Where AI does the work:** the external LLM (OpenAI, xAI Grok, Gemini, or a local Ollama model) does the actual task: summarising, drafting a reply in Azerbaijani or Russian, classifying the ticket. Pərdə is what makes it *permissible* to use that model on real data.
49
 
50
  **Why detection is not an LLM:** sending text to an LLM to find the personal data would leak it to that LLM, which defeats the purpose. So detection is local, deterministic and auditable. It combines:
51
- - validated patterns for Azerbaijani identifiers (FIN, ID card `AZE…`/`AA…`, passport, driver licence, VÖEN, AZ IBAN with a mod-97 checksum, card numbers with a Luhn check, +994 mobile and landline formats, car plates, dates)
52
  - a name detector built for Azerbaijani: a name lexicon, surname morphology (`-ov/-ova`, `-li/-lı`, `-zadə`, `-oğlu`), patronymics (`oğlu`/`qızı`), stripping of Azerbaijani case endings (`Rənanın` → `[PERSON_1]nın`), and Cyrillic and ASCII spellings
53
  - street patterns (`küçəsi`, `prospekti`, `ул.`) with house and flat numbers
54
 
@@ -73,17 +73,17 @@ Full report with failures: **[TESTING.md](TESTING.md)**. Summary:
73
  | E-mail | 132 | 100% |
74
  | ID numbers (FIN, ID card, passport, licence, VÖEN) | 165 | 100% |
75
  | Card numbers | 25 | 100% |
76
- | Addresses | 298 | 82.2% |
77
  | Dates | 117 | 80.3% |
78
 
79
  - The test split was fixed by seed and never used for tuning. The name lexicon was built from a separate 2,000-row dev split.
80
  - Reproduce with `python eval/fetch_sample.py && python eval/evaluate.py`.
81
- - There are 11 unit tests covering checksums, phone formats, tag consistency, restoration and the leak blocker (`pytest`).
82
 
83
  ## 4. Feasibility
84
 
85
  - **Data needed:** none to start. It runs on rules and a lexicon, and each organisation can extend the lexicon and patterns.
86
- - **Running cost:** masking is free, takes about 0.14 ms per message on one CPU core, and needs no GPU. The LLM cost is unchanged, because the masked text is the same length.
87
  - **Deployment:** a Python package and a web UI. It runs on-premises, next to an existing API gateway, or as a proxy.
88
  - **Next step (4 weeks):**
89
  1. Pilot on one support queue.
@@ -101,7 +101,11 @@ Generic PII filters are built for English. Azerbaijani personal data has its own
101
 
102
  **Live demo (no install, runs in your browser):** https://huggingface.co/spaces/Traxic/perde
103
 
104
- **Windows, one click:** download the v1.0 release zip, unzip it and double-click `run.bat`. It needs Python 3.10 or newer. The first run installs the dependencies, then the app opens at http://localhost:8501.
 
 
 
 
105
 
106
  **Docker (any OS, suits on-premises servers):**
107
 
@@ -119,7 +123,7 @@ python -m venv .venv
119
  ```
120
 
121
  - The default provider is **mock**, which works offline with no key.
122
- - For a real model, set one of `OPENAI_API_KEY`, `XAI_API_KEY` (Grok) or `GEMINI_API_KEY`, or run Ollama locally. Then pick the provider in the sidebar. Optionally set `PERDE_MODEL`.
123
  - Use it in code:
124
 
125
  ```python
 
34
 
35
  - **The user:** an operator who handles customer requests at a bank, a mobile operator, or a public service centre (complaints, applications, tickets).
36
  - **Today:** these teams either don't use AI at all, or staff copy customer data into public chatbots. Neither is acceptable. Azerbaijan's Law on Personal Data (2010) restricts handing personal data to third parties.
37
+ - **With Pərdə:** the same staff use any OpenAI-compatible model for summaries, replies, classification and translation. Personal data the detectors find, in both the customer text and the instruction, reaches the provider only as tags.
38
+ - **The metric that matters:** the **leak rate**, meaning the share of personal values that reach the AI provider. It falls from 76.1% with a generic filter to 5.1% with Pərdə (held-out test, details in [TESTING.md](TESTING.md)). Raw pasting, today's alternative, leaks 100%. A second check then **blocks** any request where a detected value would still leave unmasked; it cannot catch values the detectors missed.
39
 
40
  ## 2. Prototype and use of AI
41
 
 
45
  2. The exact text the AI provider receives
46
  3. The provider's answer, with the real values restored
47
 
48
+ **Where AI does the work:** the external LLM (OpenAI, xAI Grok, Gemini, or a local model via Ollama or llama.cpp) does the actual task: summarising, drafting a reply in Azerbaijani or Russian, classifying the ticket. Pərdə is what makes it *permissible* to use that model on real data.
49
 
50
  **Why detection is not an LLM:** sending text to an LLM to find the personal data would leak it to that LLM, which defeats the purpose. So detection is local, deterministic and auditable. It combines:
51
+ - validated patterns for Azerbaijani identifiers (FIN, ID card `AZE…`/`AA…`, passport, driver licence, VÖEN, AZ IBAN (mod-97 checked; invalid ones are still masked), card numbers (Luhn checked, or masked by context), +994 mobile and landline formats, car plates, dates)
52
  - a name detector built for Azerbaijani: a name lexicon, surname morphology (`-ov/-ova`, `-li/-lı`, `-zadə`, `-oğlu`), patronymics (`oğlu`/`qızı`), stripping of Azerbaijani case endings (`Rənanın` → `[PERSON_1]nın`), and Cyrillic and ASCII spellings
53
  - street patterns (`küçəsi`, `prospekti`, `ул.`) with house and flat numbers
54
 
 
73
  | E-mail | 132 | 100% |
74
  | ID numbers (FIN, ID card, passport, licence, VÖEN) | 165 | 100% |
75
  | Card numbers | 25 | 100% |
76
+ | Addresses | 298 | 82.6% |
77
  | Dates | 117 | 80.3% |
78
 
79
  - The test split was fixed by seed and never used for tuning. The name lexicon was built from a separate 2,000-row dev split.
80
  - Reproduce with `python eval/fetch_sample.py && python eval/evaluate.py`.
81
+ - There are 17 unit tests covering checksums, phone formats, tag consistency, restoration, the leak check and masking of the instruction field (`pytest`).
82
 
83
  ## 4. Feasibility
84
 
85
  - **Data needed:** none to start. It runs on rules and a lexicon, and each organisation can extend the lexicon and patterns.
86
+ - **Running cost:** masking is free, takes about 0.14 ms per message on one CPU core, and needs no GPU. The LLM cost is roughly unchanged, because tags are about as long as the values they replace.
87
  - **Deployment:** a Python package and a web UI. It runs on-premises, next to an existing API gateway, or as a proxy.
88
  - **Next step (4 weeks):**
89
  1. Pilot on one support queue.
 
101
 
102
  **Live demo (no install, runs in your browser):** https://huggingface.co/spaces/Traxic/perde
103
 
104
+ **Demo video (56 s):** [media/perde-demo.mp4](media/perde-demo.mp4)
105
+
106
+ **Real-model test:** end-to-end with Gemma 4 E4B on a local llama.cpp server, see [TEST_RESULTS.md](TEST_RESULTS.md).
107
+
108
+ **Windows, one click:** download the latest release zip, unzip it and double-click `run.bat`. It needs Python 3.10 or newer. The first run installs the dependencies, then the app opens at http://localhost:8501.
109
 
110
  **Docker (any OS, suits on-premises servers):**
111
 
 
123
  ```
124
 
125
  - The default provider is **mock**, which works offline with no key.
126
+ - For a real model, set one of `OPENAI_API_KEY`, `XAI_API_KEY` (Grok) or `GEMINI_API_KEY`, run Ollama locally, or use the `llamacpp` provider for an OpenAI-compatible local server (default `http://127.0.0.1:8080/v1`). Then pick the provider in the sidebar. Optionally set `PERDE_MODEL`.
127
  - Use it in code:
128
 
129
  ```python
TESTING.md CHANGED
@@ -19,6 +19,8 @@ Out of scope: city, time, age and zip code. These are not personal data on their
19
 
20
  ## Baselines
21
 
 
 
22
  1. **Generic regex baseline.** E-mail, any long digit sequence as a phone number, and ISO and slash dates. This is roughly what a default, non-localised filter catches.
23
  2. **Pərdə without the name detector.** An ablation that shows what the Azerbaijani name handling adds.
24
 
@@ -27,7 +29,7 @@ Out of scope: city, time, age and zip code. These are not personal data on their
27
  | System | Protected | Leak rate | Precision |
28
  |---|---|---|---|
29
  | **Pərdə** | **94.9%** | **5.1%** | 93.0% |
30
- | Pərdə without name detector | 41.1% | 58.9% | 96.0% |
31
  | Generic regex baseline | 23.9% | 76.1% | 96.4% |
32
 
33
  | Entity | Values | Recall (right type) | Protected | Precision |
@@ -38,7 +40,7 @@ Out of scope: city, time, age and zip code. These are not personal data on their
38
  | ID_NUMBER | 165 | 100.0% | 100.0% | 100.0% |
39
  | CARD | 25 | 100.0% | 100.0% | 100.0% |
40
  | DATE | 117 | 80.3% | 80.3% | 78.5% |
41
- | ADDRESS | 298 | 74.8% | 82.2% | 97.1% |
42
 
43
  ## Results: hand-written cases (20 texts)
44
 
@@ -55,14 +57,15 @@ Out of scope: city, time, age and zip code. These are not personal data on their
55
  | `Ульвия` | "zəng: 0705552211 - **Ульвия**, deyir ki…" | A Cyrillic spelling of an Azerbaijani name that isn't in the lexicon, and there is no surname next to it. |
56
  | `Ələsgər` | "22:48:43 **Ələsgər**: Mənim nömrəm…" | A rare given name, not in the dev-built lexicon. |
57
  | `İsmayılqızı` | "Mənim adım Ceyla **İsmayılqızıdır**" | A patronymic fused with a suffix (`-qızı` + `-dır`), which isn't handled yet. |
58
- | `Callan's Lane, 8` | "…materialları **Callan's Lane, 8**…" | A non-Azerbaijani street format. |
59
  | `S.S. AXUNDOV pr.` | "Diplomunuzu 1 **S.S. AXUNDOV pr.**-dən…" | Two initials with a space between them. |
60
  | `1987/01/01` | "Yubileyimiz: **1987/01/01**" | The `yyyy/mm/dd` format isn't supported yet. We didn't add it after seeing the test set, so the score stays honest. |
61
  | `0` / `5` (building numbers) | "…**5** ilə doldurun" | Bare single digits labelled as building numbers in the dataset. We don't mask lone digits on purpose. |
62
 
63
- What changed during development (on dev only), and what each change did to leak rate on dev:
64
  - Names: added the lexicon, case-ending stripping and greeting/title context. Leak rate went from 52.9% to 12.2%.
65
  - IDs: allowed `I`/`O` in FIN, added `AZE`+7 digits and the driver-licence pattern. Leak rate went from 12.2% to 5.5%.
 
66
 
67
  ## Limitations (stated honestly)
68
 
 
19
 
20
  ## Baselines
21
 
22
+ 0. **Today's practice: pasting the raw text into a chatbot.** Nothing is masked, so 100% of the personal values reach the provider. Redacting by hand is the alternative; we did not time it, so we make no claim about its speed.
23
+
24
  1. **Generic regex baseline.** E-mail, any long digit sequence as a phone number, and ISO and slash dates. This is roughly what a default, non-localised filter catches.
25
  2. **Pərdə without the name detector.** An ablation that shows what the Azerbaijani name handling adds.
26
 
 
29
  | System | Protected | Leak rate | Precision |
30
  |---|---|---|---|
31
  | **Pərdə** | **94.9%** | **5.1%** | 93.0% |
32
+ | Pərdə without name detector | 41.2% | 58.8% | 96.0% |
33
  | Generic regex baseline | 23.9% | 76.1% | 96.4% |
34
 
35
  | Entity | Values | Recall (right type) | Protected | Precision |
 
40
  | ID_NUMBER | 165 | 100.0% | 100.0% | 100.0% |
41
  | CARD | 25 | 100.0% | 100.0% | 100.0% |
42
  | DATE | 117 | 80.3% | 80.3% | 78.5% |
43
+ | ADDRESS | 298 | 75.2% | 82.6% | 97.1% |
44
 
45
  ## Results: hand-written cases (20 texts)
46
 
 
57
  | `Ульвия` | "zəng: 0705552211 - **Ульвия**, deyir ki…" | A Cyrillic spelling of an Azerbaijani name that isn't in the lexicon, and there is no surname next to it. |
58
  | `Ələsgər` | "22:48:43 **Ələsgər**: Mənim nömrəm…" | A rare given name, not in the dev-built lexicon. |
59
  | `İsmayılqızı` | "Mənim adım Ceyla **İsmayılqızıdır**" | A patronymic fused with a suffix (`-qızı` + `-dır`), which isn't handled yet. |
60
+ | `8` (building number) | "…materialları Callan's Lane, **8**…" | A building number after a non-Azerbaijani street name. |
61
  | `S.S. AXUNDOV pr.` | "Diplomunuzu 1 **S.S. AXUNDOV pr.**-dən…" | Two initials with a space between them. |
62
  | `1987/01/01` | "Yubileyimiz: **1987/01/01**" | The `yyyy/mm/dd` format isn't supported yet. We didn't add it after seeing the test set, so the score stays honest. |
63
  | `0` / `5` (building numbers) | "…**5** ilə doldurun" | Bare single digits labelled as building numbers in the dataset. We don't mask lone digits on purpose. |
64
 
65
+ What changed during development (on dev only), and what each change did to leak rate on dev. These numbers come from runs during development (`python eval/evaluate.py --split dev`); the intermediate code versions are not all kept:
66
  - Names: added the lexicon, case-ending stripping and greeting/title context. Leak rate went from 52.9% to 12.2%.
67
  - IDs: allowed `I`/`O` in FIN, added `AZE`+7 digits and the driver-licence pattern. Leak rate went from 12.2% to 5.5%.
68
+ - After the first held-out run, a code review found two bugs in the Government demo sample: a house number before a full stop, and a bracketed landline. Fixing them moved held-out addresses from 82.2% to 82.6%. Nothing else changed.
69
 
70
  ## Limitations (stated honestly)
71
 
TEST_RESULTS.md ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Test Results
2
+
3
+ Run date: 2026-10-09T10:48:27Z
4
+
5
+ ## Environment
6
+
7
+ - Python: `Python 3.14.7`
8
+ - Local model endpoint: `http://127.0.0.1:8080/v1`
9
+ - Local provider: `llamacpp`
10
+ - Local model: `gemma-4-E4B`
11
+
12
+ ## Unit tests
13
+
14
+ Command:
15
+
16
+ ```bash
17
+ .venv/bin/pytest -q
18
+ ```
19
+
20
+ Result:
21
+
22
+ ```text
23
+ ........... [100%]
24
+ 11 passed in 0.03s
25
+ ```
26
+
27
+ ## Local Gemma gateway test
28
+
29
+ Command:
30
+
31
+ ```bash
32
+ .venv/bin/python - <<'PY'
33
+ from perde.gateway import run
34
+
35
+ text = "Salam, mən Nərmin Quliyeva. Kartım 4111 1111 1111 1111, FIN 6TR9K2L. Tel 055 412 33 90."
36
+ task = "Bu müraciəti bir cümlə ilə xülasə et və qısa cavab yaz. Tagləri eyni saxla."
37
+ res = run(text, task, provider="llamacpp", model="gemma-4-E4B")
38
+ print(res.outgoing)
39
+ print(res.model_answer)
40
+ print(res.restored_answer)
41
+ PY
42
+ ```
43
+
44
+ Result:
45
+
46
+ ```text
47
+ provider: llamacpp
48
+ model: gemma-4-E4B
49
+ latency_ms: 6732
50
+ masked_values: 4
51
+ entity_counts: {'PERSON': 1, 'CARD': 1, 'FIN': 1, 'PHONE': 1}
52
+ outgoing:
53
+ Bu müraciəti bir cümlə ilə xülasə et və qısa cavab yaz. Tagləri eyni saxla.
54
+
55
+ ---
56
+ Salam, mən [PERSON_1]. Kartım [CARD_1], FIN [FIN_1]. Tel [PHONE_1].
57
+ model_answer:
58
+ **Xülasə:** Müraciət sahibi [PERSON_1] öz şəxsi məlumatlarını (kart [CARD_1], FIN [FIN_1] və telefon [PHONE_1]) təqdim edir.
59
+
60
+ **Cavab:** Məlumatınız qeydə alındı.
61
+ restored_answer:
62
+ **Xülasə:** Müraciət sahibi Nərmin Quliyeva öz şəxsi məlumatlarını (kart 4111 1111 1111 1111, FIN 6TR9K2L və telefon 055 412 33 90) təqdim edir.
63
+
64
+ **Cavab:** Məlumatınız qeydə alındı.
65
+ ```
66
+
67
+ ## Streamlit smoke test
68
+
69
+ Command:
70
+
71
+ ```bash
72
+ .venv/bin/streamlit run app.py --server.headless true --server.port 8501
73
+ curl -fsS http://127.0.0.1:8501/_stcore/health
74
+ ```
75
+
76
+ Result:
77
+
78
+ ```text
79
+ ok
80
+ ```
app.py CHANGED
@@ -98,7 +98,7 @@ with st.sidebar:
98
  if in_browser:
99
  st.info(
100
  "Web demo: runs entirely in your browser with the offline model. "
101
- "To use OpenAI, Grok, Gemini or Ollama, run the app locally (run.bat) or with Docker."
102
  )
103
  model = st.text_input("Model (optional)", value="", placeholder=PROVIDERS[provider]["model"])
104
  enabled = st.multiselect("Data types to protect", ALL_ENTITIES, default=list(SECTORS[sector]["entities"]))
 
98
  if in_browser:
99
  st.info(
100
  "Web demo: runs entirely in your browser with the offline model. "
101
+ "To use OpenAI, Grok, Gemini, Ollama or a llama.cpp server, run the app locally (run.bat) or with Docker."
102
  )
103
  model = st.text_input("Model (optional)", value="", placeholder=PROVIDERS[provider]["model"])
104
  enabled = st.multiselect("Data types to protect", ALL_ENTITIES, default=list(SECTORS[sector]["entities"]))
eval/evaluate.py CHANGED
@@ -1,7 +1,7 @@
1
  """Measure detection quality on the held-out LocalDoc split and on hand-written cases.
2
 
3
  Usage: python eval/evaluate.py [--split test|dev] [--show-failures N]
4
- Writes eval/results.json and prints a Markdown table.
5
 
6
  Metrics, per entity group:
7
  recall gold values masked by a detection of the right type (>=50% of characters)
@@ -178,7 +178,8 @@ def main():
178
  s = results[dname]["Pərdə"][g]
179
  if s["n_gold"]:
180
  print(f"| {g} | {s['n_gold']} | {fmt(s['recall'])} | {fmt(s['protected'])} | {fmt(s['precision'])} |")
181
- (ROOT / "eval" / "results.json").write_text(json.dumps(results, ensure_ascii=False, indent=2), encoding="utf-8")
 
182
 
183
 
184
  if __name__ == "__main__":
 
1
  """Measure detection quality on the held-out LocalDoc split and on hand-written cases.
2
 
3
  Usage: python eval/evaluate.py [--split test|dev] [--show-failures N]
4
+ Writes eval/results.json (test split) or eval/results_dev.json (dev split) and prints a Markdown table.
5
 
6
  Metrics, per entity group:
7
  recall gold values masked by a detection of the right type (>=50% of characters)
 
178
  s = results[dname]["Pərdə"][g]
179
  if s["n_gold"]:
180
  print(f"| {g} | {s['n_gold']} | {fmt(s['recall'])} | {fmt(s['protected'])} | {fmt(s['precision'])} |")
181
+ out_name = "results.json" if args.split == "test" else "results_dev.json" # the app shows results.json
182
+ (ROOT / "eval" / out_name).write_text(json.dumps(results, ensure_ascii=False, indent=2), encoding="utf-8")
183
 
184
 
185
  if __name__ == "__main__":
eval/results.json CHANGED
@@ -45,14 +45,14 @@
45
  },
46
  "ADDRESS": {
47
  "n_gold": 298,
48
- "recall": 0.748,
49
- "protected": 0.822,
50
  "precision": 0.971,
51
  "n_pred": 138
52
  },
53
  "ALL": {
54
  "n_gold": 2065,
55
- "recall": 0.937,
56
  "protected": 0.949,
57
  "precision": 0.93,
58
  "n_pred": 1725
@@ -103,15 +103,15 @@
103
  },
104
  "ADDRESS": {
105
  "n_gold": 298,
106
- "recall": 0.748,
107
- "protected": 0.822,
108
  "precision": 0.971,
109
  "n_pred": 138
110
  },
111
  "ALL": {
112
  "n_gold": 2065,
113
  "recall": 0.399,
114
- "protected": 0.411,
115
  "precision": 0.96,
116
  "n_pred": 766
117
  }
 
45
  },
46
  "ADDRESS": {
47
  "n_gold": 298,
48
+ "recall": 0.752,
49
+ "protected": 0.826,
50
  "precision": 0.971,
51
  "n_pred": 138
52
  },
53
  "ALL": {
54
  "n_gold": 2065,
55
+ "recall": 0.938,
56
  "protected": 0.949,
57
  "precision": 0.93,
58
  "n_pred": 1725
 
103
  },
104
  "ADDRESS": {
105
  "n_gold": 298,
106
+ "recall": 0.752,
107
+ "protected": 0.826,
108
  "precision": 0.971,
109
  "n_pred": 138
110
  },
111
  "ALL": {
112
  "n_gold": 2065,
113
  "recall": 0.399,
114
+ "protected": 0.412,
115
  "precision": 0.96,
116
  "n_pred": 766
117
  }
eval/results.md CHANGED
@@ -22,7 +22,7 @@ Mövzu: Unikal İmza Analizi Məktubu - Zəhmət olmasa, təhlil üçün yazıl
22
  MISS SURNAME: 'İsmayılqızı' :: Mənim adım Ceyla İsmayılqızıdır və mənim 57 yaşım var. Bu təlim qrupunun bir hissəsi olmaqdan çox həyəcanlıyam.
23
  MISS BUILDINGNUM: '5' :: Resept formasında, zəhmət olmasa, sahələri 0U51BU4, Nisə və 5 ilə doldurun.
24
  MISS STREET: 'BAKI-BATUMİ küç.' :: Nəzərinizə çatdırırıq ki, Lətifə Məhərrəmzadə yüngül dəmir yolu stansiyası, BAKI-BATUMİ küç. 0 qazıntı sahəsinə ən yaxın
25
- | Pərdə without name detector | 41.1% | 58.9% | 96.0% |
26
  | Generic regex baseline | 23.9% | 76.1% | 96.4% |
27
  Per entity (Pərdə):
28
  | Entity | Gold | Recall (right type) | Protected | Precision |
@@ -33,7 +33,7 @@ Per entity (Pərdə):
33
  | ID_NUMBER | 165 | 100.0% | 100.0% | 100.0% |
34
  | CARD | 25 | 100.0% | 100.0% | 100.0% |
35
  | DATE | 117 | 80.3% | 80.3% | 78.5% |
36
- | ADDRESS | 298 | 74.8% | 82.2% | 97.1% |
37
  ### Hand-written AZ/RU/EN cases (20 texts)
38
  | System | Protected (all PII) | Leak rate | Precision |
39
  |---|---|---|---|
 
22
  MISS SURNAME: 'İsmayılqızı' :: Mənim adım Ceyla İsmayılqızıdır və mənim 57 yaşım var. Bu təlim qrupunun bir hissəsi olmaqdan çox həyəcanlıyam.
23
  MISS BUILDINGNUM: '5' :: Resept formasında, zəhmət olmasa, sahələri 0U51BU4, Nisə və 5 ilə doldurun.
24
  MISS STREET: 'BAKI-BATUMİ küç.' :: Nəzərinizə çatdırırıq ki, Lətifə Məhərrəmzadə yüngül dəmir yolu stansiyası, BAKI-BATUMİ küç. 0 qazıntı sahəsinə ən yaxın
25
+ | Pərdə without name detector | 41.2% | 58.8% | 96.0% |
26
  | Generic regex baseline | 23.9% | 76.1% | 96.4% |
27
  Per entity (Pərdə):
28
  | Entity | Gold | Recall (right type) | Protected | Precision |
 
33
  | ID_NUMBER | 165 | 100.0% | 100.0% | 100.0% |
34
  | CARD | 25 | 100.0% | 100.0% | 100.0% |
35
  | DATE | 117 | 80.3% | 80.3% | 78.5% |
36
+ | ADDRESS | 298 | 75.2% | 82.6% | 97.1% |
37
  ### Hand-written AZ/RU/EN cases (20 texts)
38
  | System | Protected (all PII) | Leak rate | Precision |
39
  |---|---|---|---|
index.html CHANGED
@@ -4,7 +4,7 @@
4
  <meta charset="UTF-8" />
5
  <meta name="viewport" content="width=device-width, initial-scale=1" />
6
  <title>Pərdə · Safe AI for Azerbaijani data</title>
7
- <meta name="description" content="Masks Azerbaijani personal data before it reaches an AI model. Runs fully in your browser." />
8
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@stlite/browser@1.9.2/build/stlite.css" />
9
  <style>
10
  body { margin: 0; background: #0e1117; color: #fafafa; font-family: system-ui, sans-serif; }
@@ -17,7 +17,7 @@
17
  <div id="loading">
18
  <h2>🛡️ Pərdə</h2>
19
  <p>Loading Python in your browser (about 10–20 seconds on first visit)…</p>
20
- <small>Everything runs on your device. No personal data is uploaded.</small>
21
  </div>
22
  </div>
23
  <script type="module">
 
4
  <meta charset="UTF-8" />
5
  <meta name="viewport" content="width=device-width, initial-scale=1" />
6
  <title>Pərdə · Safe AI for Azerbaijani data</title>
7
+ <meta name="description" content="Masks Azerbaijani personal data before it reaches an AI model. Runs in your browser; no text is uploaded." />
8
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@stlite/browser@1.9.2/build/stlite.css" />
9
  <style>
10
  body { margin: 0; background: #0e1117; color: #fafafa; font-family: system-ui, sans-serif; }
 
17
  <div id="loading">
18
  <h2>🛡️ Pərdə</h2>
19
  <p>Loading Python in your browser (about 10–20 seconds on first visit)…</p>
20
+ <small>Runs on your device: the Python runtime downloads once, your text is never uploaded.</small>
21
  </div>
22
  </div>
23
  <script type="module">
perde/gateway.py CHANGED
@@ -22,6 +22,12 @@ PROVIDERS = {
22
  "model": "gemini-2.5-flash",
23
  },
24
  "ollama": {"base_url": "http://localhost:11434/v1", "key_env": None, "model": "gemma3"},
 
 
 
 
 
 
25
  "mock": {"base_url": None, "key_env": None, "model": "mock"},
26
  }
27
 
@@ -75,7 +81,8 @@ def call_model(provider: str, prompt: str, model: str | None = None) -> tuple[st
75
  key = os.environ.get(cfg["key_env"]) if cfg["key_env"] else "ollama"
76
  if not key:
77
  raise RuntimeError(f"Set {cfg['key_env']} to use provider '{provider}'.")
78
- client = OpenAI(base_url=cfg["base_url"], api_key=key, timeout=60)
 
79
  resp = client.chat.completions.create(
80
  model=model,
81
  messages=[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": prompt}],
@@ -85,9 +92,12 @@ def call_model(provider: str, prompt: str, model: str | None = None) -> tuple[st
85
 
86
 
87
  def run(text: str, task: str, provider: str = "mock", entities=None, model: str | None = None) -> GatewayResult:
88
- """Mask `text`, block if anything leaks, send task + masked text, restore the answer."""
89
- m = mask(text) if entities is None else mask(text, tuple(entities))
90
- outgoing = f"{task.strip()}\n\n---\n{m.masked}" if task.strip() else m.masked
 
 
 
91
  leaks = leaked_values(outgoing, m.mapping)
92
  if leaks:
93
  raise LeakBlocked(f"{len(leaks)} value(s) would leave unmasked; request blocked.")
@@ -100,13 +110,13 @@ def run(text: str, task: str, provider: str = "mock", entities=None, model: str
100
 
101
 
102
  def _audit(m: MaskResult, outgoing: str, provider: str, model: str, latency_ms: int) -> None:
103
- """Append-only log. Stores hashes and counts, never raw personal data."""
 
104
  record = {
105
  "ts": time.strftime("%Y-%m-%dT%H:%M:%S"),
106
  "provider": provider,
107
  "model": model,
108
  "entities_masked": m.counts(),
109
- "input_sha256": hashlib.sha256(m.original.encode()).hexdigest(),
110
  "outgoing_sha256": hashlib.sha256(outgoing.encode()).hexdigest(),
111
  "leak_check": "passed",
112
  "latency_ms": latency_ms,
 
22
  "model": "gemini-2.5-flash",
23
  },
24
  "ollama": {"base_url": "http://localhost:11434/v1", "key_env": None, "model": "gemma3"},
25
+ "llamacpp": {
26
+ "base_url": "http://127.0.0.1:8080/v1",
27
+ "base_url_env": "PERDE_LOCAL_BASE_URL",
28
+ "key_env": None,
29
+ "model": "gemma-4-E4B",
30
+ },
31
  "mock": {"base_url": None, "key_env": None, "model": "mock"},
32
  }
33
 
 
81
  key = os.environ.get(cfg["key_env"]) if cfg["key_env"] else "ollama"
82
  if not key:
83
  raise RuntimeError(f"Set {cfg['key_env']} to use provider '{provider}'.")
84
+ base_url = os.environ.get(cfg.get("base_url_env", "")) or cfg["base_url"]
85
+ client = OpenAI(base_url=base_url, api_key=key, timeout=60)
86
  resp = client.chat.completions.create(
87
  model=model,
88
  messages=[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": prompt}],
 
92
 
93
 
94
  def run(text: str, task: str, provider: str = "mock", entities=None, model: str | None = None) -> GatewayResult:
95
+ """Mask the task and the text together, block if anything leaks, send, restore the answer."""
96
+ combined = f"{task.strip()}\n\n---\n{text}" if task.strip() else text
97
+ if entities is not None and not entities and mask(combined).spans:
98
+ raise LeakBlocked("No data types are selected but personal data was detected; request blocked.")
99
+ m = mask(combined) if entities is None else mask(combined, tuple(entities))
100
+ outgoing = m.masked
101
  leaks = leaked_values(outgoing, m.mapping)
102
  if leaks:
103
  raise LeakBlocked(f"{len(leaks)} value(s) would leave unmasked; request blocked.")
 
110
 
111
 
112
  def _audit(m: MaskResult, outgoing: str, provider: str, model: str, latency_ms: int) -> None:
113
+ """Append-only log. Stores counts and a hash of the already-masked outgoing text, never raw
114
+ personal data. No hash of the raw input: a short input (one phone number) could be brute-forced."""
115
  record = {
116
  "ts": time.strftime("%Y-%m-%dT%H:%M:%S"),
117
  "provider": provider,
118
  "model": model,
119
  "entities_masked": m.counts(),
 
120
  "outgoing_sha256": hashlib.sha256(outgoing.encode()).hexdigest(),
121
  "leak_check": "passed",
122
  "latency_ms": latency_ms,
perde/recognizers.py CHANGED
@@ -60,7 +60,7 @@ def _context(text: str, start: int, end: int, words: tuple[str, ...], window: in
60
  # --- pattern detectors ------------------------------------------------------
61
 
62
  _PHONE = re.compile(
63
- r"(?<![\d+])(?:\+?\s?994|8|0)[\s\-.]?\(?(?:10|50|51|55|60|70|77|99|12|18|2\d|36)\)?"
64
  r"[\s\-.]?\d{3}[\s\-.]?\d{2}[\s\-.]?\d{2}(?!\d)"
65
  )
66
  _EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+(?:\.[\w-]+)+\b")
@@ -88,7 +88,7 @@ _NAME_WORD = rf"(?:[{AZ_UPPER}]\.\s?)?[{AZ_UPPER}][{AZ_UPPER}{AZ_LOWER}]+"
88
  _NUM = r"\d{1,4}[A-Za-z]?"
89
  _ADDRESS = re.compile(
90
  rf"(?:\b{_NUM},?\s)?{_NAME_WORD}(?:\s{_NAME_WORD}){{0,2}}\s{_STREET_WORD}"
91
- rf"(?:,?\s*(?:ev|bina|mənzil|m\.)?\s*{_NUM}(?![\d.]))?"
92
  rf"(?:\s*,?\s*(?:mənzil|m\.)\s*\d{{1,4}})?"
93
  r"|(?:ул\.|улица|проспект|пр-т)\s?[А-ЯЁ][а-яё]+(?:\s[А-ЯЁ][а-яё]+)?(?:,?\s*(?:д\.)?\s*\d{1,4})?"
94
  )
 
60
  # --- pattern detectors ------------------------------------------------------
61
 
62
  _PHONE = re.compile(
63
+ r"(?<![\d+(])\(?(?:\+?\s?994|8|0)[\s\-.]?\(?(?:10|50|51|55|60|70|77|99|12|18|2\d|36)\)?"
64
  r"[\s\-.]?\d{3}[\s\-.]?\d{2}[\s\-.]?\d{2}(?!\d)"
65
  )
66
  _EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+(?:\.[\w-]+)+\b")
 
88
  _NUM = r"\d{1,4}[A-Za-z]?"
89
  _ADDRESS = re.compile(
90
  rf"(?:\b{_NUM},?\s)?{_NAME_WORD}(?:\s{_NAME_WORD}){{0,2}}\s{_STREET_WORD}"
91
+ rf"(?:,?\s*(?:ev|bina|mənzil|m\.)?\s*{_NUM}(?!\d|\.\d))?"
92
  rf"(?:\s*,?\s*(?:mənzil|m\.)\s*\d{{1,4}})?"
93
  r"|(?:ул\.|улица|проспект|пр-т)\s?[А-ЯЁ][а-яё]+(?:\s[А-ЯЁ][а-яё]+)?(?:,?\s*(?:д\.)?\s*\d{1,4})?"
94
  )
perde/shield.py CHANGED
@@ -50,7 +50,7 @@ def mask(text: str, entities: tuple[str, ...] = ALL_ENTITIES, min_score: float =
50
  parts: list[str] = []
51
  cursor = 0
52
  for s in spans:
53
- key = (s.entity, _normalize(s.text))
54
  tag = value_to_tag.get(key)
55
  if tag is None:
56
  counters[s.entity] = counters.get(s.entity, 0) + 1
@@ -63,7 +63,7 @@ def mask(text: str, entities: tuple[str, ...] = ALL_ENTITIES, min_score: float =
63
  result = MaskResult(text, "".join(parts), spans)
64
  seen: set[str] = set()
65
  for s in spans:
66
- tag = value_to_tag[(s.entity, _normalize(s.text))]
67
  if tag not in seen:
68
  result.mapping[tag] = s.text
69
  seen.add(tag)
@@ -84,19 +84,27 @@ def restore(answer: str, mapping: dict[str, str]) -> str:
84
  def leaked_values(outgoing: str, mapping: dict[str, str]) -> list[str]:
85
  """Return original values that still appear in text about to leave the machine."""
86
  out_norm = _normalize(outgoing)
87
- out_digits = re.sub(r"\D", "", outgoing)
 
88
  leaks = []
89
  for value in mapping.values():
90
  v = _normalize(value)
91
  digits = re.sub(r"\D", "", value)
92
  # Compare the last 9 digits so "+994 55..." and "055..." count as the same number.
93
  core = digits[-9:]
94
- if v and v in out_norm:
95
  leaks.append(value)
96
- elif len(core) >= 7 and core in out_digits:
97
  leaks.append(value)
98
  return leaks
99
 
100
 
 
 
 
 
 
 
 
101
  def _normalize(s: str) -> str:
102
  return re.sub(r"\s+", " ", s).strip().lower()
 
50
  parts: list[str] = []
51
  cursor = 0
52
  for s in spans:
53
+ key = _key(s)
54
  tag = value_to_tag.get(key)
55
  if tag is None:
56
  counters[s.entity] = counters.get(s.entity, 0) + 1
 
63
  result = MaskResult(text, "".join(parts), spans)
64
  seen: set[str] = set()
65
  for s in spans:
66
+ tag = value_to_tag[_key(s)]
67
  if tag not in seen:
68
  result.mapping[tag] = s.text
69
  seen.add(tag)
 
84
  def leaked_values(outgoing: str, mapping: dict[str, str]) -> list[str]:
85
  """Return original values that still appear in text about to leave the machine."""
86
  out_norm = _normalize(outgoing)
87
+ # digits of each separate number in the outgoing text, so unrelated numbers never join up
88
+ out_numbers = [re.sub(r"\D", "", n) for n in re.findall(r"\+?\d[\d\s\-().]*\d", outgoing)]
89
  leaks = []
90
  for value in mapping.values():
91
  v = _normalize(value)
92
  digits = re.sub(r"\D", "", value)
93
  # Compare the last 9 digits so "+994 55..." and "055..." count as the same number.
94
  core = digits[-9:]
95
+ if v and re.search(rf"(?<!\w){re.escape(v)}(?!\w)", out_norm):
96
  leaks.append(value)
97
+ elif len(core) >= 7 and any(core in n for n in out_numbers):
98
  leaks.append(value)
99
  return leaks
100
 
101
 
102
+ def _key(span: Span) -> tuple[str, str]:
103
+ """Same person or number gets the same tag: phones compare by their last 9 digits."""
104
+ if span.entity == "PHONE":
105
+ return span.entity, re.sub(r"\D", "", span.text)[-9:]
106
+ return span.entity, _normalize(span.text)
107
+
108
+
109
  def _normalize(s: str) -> str:
110
  return re.sub(r"\s+", " ", s).strip().lower()
tests/test_shield.py CHANGED
@@ -15,6 +15,14 @@ def test_az_phone_formats():
15
  assert "PHONE" in entities(f"Əlaqə: {phone}"), phone
16
 
17
 
 
 
 
 
 
 
 
 
18
  def test_iban_checksum():
19
  assert iban_ok("AZ21NABZ00000000137010001944")
20
  assert not iban_ok("AZ21NABZ00000000137010001945")
@@ -43,6 +51,11 @@ def test_same_value_gets_same_tag():
43
  assert r.masked.count("[PERSON_1]") == 2
44
 
45
 
 
 
 
 
 
46
  def test_restore_tolerates_model_formatting():
47
  r = mask("Telefon +994 55 765 43 21")
48
  assert restore("Nömrə: [ phone_1 ]", r.mapping) == "Nömrə: +994 55 765 43 21"
@@ -54,6 +67,28 @@ def test_no_leak_after_masking():
54
  assert leaked_values(r.masked, r.mapping) == []
55
 
56
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57
  def test_leak_check_catches_unmasked_digits():
58
  r = mask("Telefon +994 55 765 43 21")
59
  assert leaked_values("call 0557654321", r.mapping)
 
15
  assert "PHONE" in entities(f"Əlaqə: {phone}"), phone
16
 
17
 
18
+ def test_bracketed_landline_masked_with_its_bracket():
19
+ assert entities("Tel: (012) 555 12 34")["PHONE"] == "(012) 555 12 34"
20
+
21
+
22
+ def test_house_number_before_full_stop_is_masked():
23
+ assert entities("Ünvan: Sülh küçəsi 12. Tel yoxdur")["ADDRESS"] == "Sülh küçəsi 12"
24
+
25
+
26
  def test_iban_checksum():
27
  assert iban_ok("AZ21NABZ00000000137010001944")
28
  assert not iban_ok("AZ21NABZ00000000137010001945")
 
51
  assert r.masked.count("[PERSON_1]") == 2
52
 
53
 
54
+ def test_same_phone_in_two_formats_gets_one_tag():
55
+ r = mask("Zəng: 050 123 45 67 və ya +994501234567")
56
+ assert r.masked.count("[PHONE_1]") == 2
57
+
58
+
59
  def test_restore_tolerates_model_formatting():
60
  r = mask("Telefon +994 55 765 43 21")
61
  assert restore("Nömrə: [ phone_1 ]", r.mapping) == "Nömrə: +994 55 765 43 21"
 
67
  assert leaked_values(r.masked, r.mapping) == []
68
 
69
 
70
+ def test_leak_check_ignores_values_inside_other_words_and_numbers():
71
+ r = mask("Əli zəng etdi, tel +994 55 765 43 21")
72
+ assert leaked_values("Bəli/xeyr cavabı ver", r.mapping) == []
73
+ assert leaked_values("Tarix 2024-55-76, kod 54 321", r.mapping) == []
74
+
75
+
76
+ def test_gateway_masks_personal_data_in_the_task_too(tmp_path, monkeypatch):
77
+ from perde import gateway
78
+ monkeypatch.setattr(gateway, "AUDIT_PATH", tmp_path / "audit.jsonl")
79
+ res = gateway.run("Tel 055 412 33 90", "Reply to Rəşad Həsənov, FIN 7XK2M9P", provider="mock")
80
+ for value in ("Rəşad Həsənov", "7XK2M9P", "055 412 33 90"):
81
+ assert value not in res.outgoing
82
+
83
+
84
+ def test_gateway_blocks_when_no_data_types_selected(tmp_path, monkeypatch):
85
+ import pytest
86
+ from perde import gateway
87
+ monkeypatch.setattr(gateway, "AUDIT_PATH", tmp_path / "audit.jsonl")
88
+ with pytest.raises(gateway.LeakBlocked):
89
+ gateway.run("Nərmin Quliyeva FIN 6TR9K2L", "", provider="mock", entities=[])
90
+
91
+
92
  def test_leak_check_catches_unmasked_digits():
93
  r = mask("Telefon +994 55 765 43 21")
94
  assert leaked_values("call 0557654321", r.mapping)
tools/video/.gitignore ADDED
@@ -0,0 +1 @@
 
 
1
+ shots/
tools/video/README.md ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ # Demo video tooling
2
+
3
+ Used to make `media/perde-demo.mp4`.
4
+
5
+ 1. Start the app: `streamlit run app.py` (port 8501).
6
+ 2. `python tools/video/capture.py tools/video/shots` takes real screenshots of the running app with headless Chrome.
7
+ 3. `python tools/video/render.py . media/perde-demo.mp4` renders 56 s at 1080p30, with a soundtrack synthesized in code.
8
+
9
+ Needs `pillow numpy imageio-ffmpeg websockets` and Google Chrome. Fonts: Segoe UI and Cascadia Mono (Windows).
tools/video/capture.py ADDED
@@ -0,0 +1,69 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Capture real screenshots of the running Pərdə app with headless Chrome (CDP)."""
2
+ import base64
3
+ import json
4
+ import subprocess
5
+ import sys
6
+ import time
7
+ import urllib.request
8
+ from pathlib import Path
9
+
10
+ from websockets.sync.client import connect
11
+
12
+ OUT = Path(sys.argv[1])
13
+ OUT.mkdir(parents=True, exist_ok=True)
14
+ CHROME = r"C:\Program Files\Google\Chrome\Application\chrome.exe"
15
+ PROFILE = OUT / "_chrome_profile"
16
+
17
+ proc = subprocess.Popen([
18
+ CHROME, "--headless=new", "--remote-debugging-port=9333", f"--user-data-dir={PROFILE}",
19
+ "--window-size=1920,1080", "--hide-scrollbars", "--force-dark-mode", "about:blank",
20
+ ])
21
+ try:
22
+ for _ in range(50):
23
+ try:
24
+ tabs = json.load(urllib.request.urlopen("http://127.0.0.1:9333/json"))
25
+ page = next(t for t in tabs if t["type"] == "page")
26
+ break
27
+ except Exception:
28
+ time.sleep(0.2)
29
+ ws = connect(page["webSocketDebuggerUrl"], max_size=50_000_000)
30
+ n = 0
31
+
32
+ def cmd(method, **params):
33
+ global n
34
+ n += 1
35
+ ws.send(json.dumps({"id": n, "method": method, "params": params}))
36
+ while True:
37
+ msg = json.loads(ws.recv())
38
+ if msg.get("id") == n:
39
+ return msg.get("result", {})
40
+
41
+ def js(expr):
42
+ return cmd("Runtime.evaluate", expression=expr, awaitPromise=True, returnByValue=True)
43
+
44
+ def shot(name):
45
+ data = cmd("Page.captureScreenshot", format="png")["data"]
46
+ (OUT / f"{name}.png").write_bytes(base64.b64decode(data))
47
+ print("saved", name)
48
+
49
+ cmd("Emulation.setDeviceMetricsOverride", width=1920, height=1080, deviceScaleFactor=1, mobile=False)
50
+ cmd("Emulation.setEmulatedMedia", features=[{"name": "prefers-color-scheme", "value": "dark"}])
51
+ cmd("Page.enable")
52
+ cmd("Page.navigate", url="http://localhost:8501/")
53
+ time.sleep(8)
54
+ shot("01_start")
55
+ js("[...document.querySelectorAll('button')].find(b => b.innerText.includes('Protect and send')).click()")
56
+ time.sleep(4)
57
+ shot("02_after_send")
58
+ js("window.scrollBy(0, 380)")
59
+ js("document.querySelector('section.main, [data-testid=stMain]')?.scrollBy(0, 380)")
60
+ time.sleep(1)
61
+ shot("03_scrolled")
62
+ js("[...document.querySelectorAll('button[role=tab]')].find(b => b.innerText.includes('Evidence')).click()")
63
+ time.sleep(2)
64
+ js("document.querySelector('[data-testid=stMain]')?.scrollTo(0, 0)")
65
+ time.sleep(1)
66
+ shot("04_evidence")
67
+ ws.close()
68
+ finally:
69
+ proc.terminate()
tools/video/render.py ADDED
@@ -0,0 +1,422 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Render the Pərdə promo video (1920x1080, 30 fps, with a synthesized soundtrack)."""
2
+ import math
3
+ import subprocess
4
+ import sys
5
+ import wave
6
+ from pathlib import Path
7
+
8
+ import numpy as np
9
+ from PIL import Image, ImageDraw, ImageFont, ImageFilter
10
+
11
+ HERE = Path(__file__).parent
12
+ REPO = Path(sys.argv[1])
13
+ OUT = Path(sys.argv[2])
14
+ sys.path.insert(0, str(REPO))
15
+ from perde import mask # noqa: E402 real masking output, not mocked
16
+ import imageio_ffmpeg # noqa: E402
17
+
18
+ W, H, FPS = 1920, 1080, 30
19
+ DUR = 56.0
20
+ FONTS = Path(r"C:\Windows\Fonts")
21
+ BOLD = str(FONTS / "segoeuib.ttf")
22
+ REG = str(FONTS / "segoeui.ttf")
23
+ LIGHT = str(FONTS / "segoeuisl.ttf")
24
+ MONO = str(FONTS / "CascadiaMono.ttf")
25
+
26
+ INK = (15, 23, 36)
27
+ PANEL = (24, 35, 58)
28
+ LIGHTC = (245, 241, 232)
29
+ MUTED = (174, 184, 201)
30
+ RED = (232, 80, 91)
31
+ AMBER = (242, 166, 90)
32
+ GREEN = (127, 200, 169)
33
+
34
+ _font_cache = {}
35
+
36
+
37
+ def F(path, size):
38
+ key = (path, size)
39
+ if key not in _font_cache:
40
+ _font_cache[key] = ImageFont.truetype(path, size)
41
+ return _font_cache[key]
42
+
43
+
44
+ def clamp(x, a=0.0, b=1.0):
45
+ return max(a, min(b, x))
46
+
47
+
48
+ def ease(x):
49
+ x = clamp(x)
50
+ return 1 - (1 - x) ** 3
51
+
52
+
53
+ def ease_io(x):
54
+ x = clamp(x)
55
+ return x * x * (3 - 2 * x)
56
+
57
+
58
+ def mix(c1, c2, t):
59
+ return tuple(int(a + (b - a) * clamp(t)) for a, b in zip(c1, c2))
60
+
61
+
62
+ def text_center(d, y, s, font, fill, alpha=1.0, bg=INK):
63
+ w = d.textlength(s, font=font)
64
+ d.text(((W - w) / 2, y), s, font=font, fill=mix(bg, fill, alpha))
65
+
66
+
67
+ # --- message layout --------------------------------------------------------
68
+
69
+ MSG = ("Salam, mən Nərmin Quliyeva. Kartımdan 2 dəfə 150 AZN çıxılıb, kart 4111 1111 1111 1111. "
70
+ "FIN kodum 6TR9K2L, hesabım AZ21NABZ00000000137010001944. Zəng edin: 055 412 33 90.")
71
+ RES = mask(MSG)
72
+ SPANS = RES.spans # real detections
73
+ MASKED = RES.masked
74
+
75
+
76
+ def layout(text, font, max_w, x0, y0, lh):
77
+ """Word-wrap; return list of (x, y, ch, index)."""
78
+ out, x, y = [], x0, y0
79
+ space = font.getlength(" ")
80
+ i = 0
81
+ for word in text.split(" "):
82
+ ww = font.getlength(word)
83
+ if x + ww > x0 + max_w and x > x0:
84
+ x, y = x0, y + lh
85
+ for ch in word:
86
+ out.append((x, y, ch, i))
87
+ x += font.getlength(ch)
88
+ i += 1
89
+ out.append((x, y, " ", i))
90
+ x += space
91
+ i += 1
92
+ return out
93
+
94
+
95
+ MSG_FONT = F(MONO, 40)
96
+ MSG_BOX = (180, 330, 1740, 600)
97
+ MSG_LAYOUT = layout(MSG, MSG_FONT, 1480, 220, 370, 64)
98
+ MASKED_LAYOUT = layout(MASKED, MSG_FONT, 1480, 220, 370, 64)
99
+
100
+
101
+ def span_of(idx):
102
+ for k, s in enumerate(SPANS):
103
+ if s.start <= idx < s.end:
104
+ return k
105
+ return None
106
+
107
+
108
+ def tag_ranges(text):
109
+ import re
110
+ return [(m.start(), m.end()) for m in re.finditer(r"\[[A-Z_]+_\d+\]", text)]
111
+
112
+
113
+ TAGS = tag_ranges(MASKED)
114
+
115
+ # --- scenes ----------------------------------------------------------------
116
+
117
+
118
+ def scene_hook(img, d, t):
119
+ """0-10s: typed message, then personal data lights up red."""
120
+ cap = "A bank clerk pastes a complaint into a chatbot…"
121
+ n = int(len(cap) * ease(t / 1.2))
122
+ text_center(d, 170, cap[:n], F(BOLD, 60), LIGHTC)
123
+ a = ease((t - 0.6) / 0.5)
124
+ d.rounded_rectangle(MSG_BOX, 28, fill=mix(INK, PANEL, a), outline=mix(INK, (44, 58, 85), a), width=2)
125
+ typed = int(len(MSG) * clamp((t - 1.0) / 3.2))
126
+ for x, y, ch, i in MSG_LAYOUT:
127
+ if i >= typed:
128
+ break
129
+ col = (220, 227, 238)
130
+ k = span_of(i)
131
+ if k is not None:
132
+ on = ease((t - (5.4 + 0.45 * k)) / 0.25)
133
+ col = mix(col, RED, on)
134
+ if on > 0:
135
+ d.rectangle((x, y + 52, x + MSG_FONT.getlength(ch), y + 56), fill=mix(PANEL, RED, on))
136
+ d.text((x, y), ch, font=MSG_FONT, fill=col)
137
+ if typed < len(MSG) and int(t * 4) % 2 == 0 and t > 1:
138
+ cx, cy, _, _ = MSG_LAYOUT[min(typed, len(MSG_LAYOUT) - 1)]
139
+ d.rectangle((cx, cy + 4, cx + 4, cy + 50), fill=AMBER)
140
+ if t > 8.2:
141
+ a = ease((t - 8.2) / 0.5)
142
+ text_center(d, 700, f"{len(SPANS)} pieces of personal data just left the building.", F(BOLD, 56), RED, a)
143
+
144
+
145
+ def scene_question(img, d, t):
146
+ """10-13s."""
147
+ s = 1 + 0.08 * ease(t / 3)
148
+ f = F(BOLD, int(110 * s))
149
+ a = ease(t / 0.4)
150
+ text_center(d, 380 - 20 * (s - 1) * 10, "What if the AI", f, LIGHTC, a)
151
+ text_center(d, 520 - 20 * (s - 1) * 10, "never saw them?", f, RED, ease((t - 0.5) / 0.4))
152
+
153
+
154
+ def curtain(d, t, reverse=False):
155
+ """Vertical crimson-to-amber stripes sweeping across."""
156
+ n = 12
157
+ sw = W / n
158
+ for k in range(n):
159
+ p = ease((t - k * 0.04) / 0.5)
160
+ if reverse:
161
+ p = 1 - p
162
+ h = H * p
163
+ col = mix(RED, AMBER, k / n)
164
+ d.rectangle((k * sw, 0, (k + 1) * sw + 1, h), fill=col)
165
+
166
+
167
+ def scene_logo(img, d, t):
168
+ """13-16s: curtain closes, logo appears."""
169
+ if t < 0.9:
170
+ curtain(d, t)
171
+ return
172
+ d.rectangle((0, 0, W, H), fill=INK)
173
+ for k, op in enumerate([1, 0.6, 0.35, 0.15]):
174
+ x = 1340 + k * 60
175
+ d.rectangle((x, 0, x + 28, H), fill=mix(INK, mix(RED, AMBER, 0.3), op * ease((t - 0.9) / 0.6)))
176
+ a = ease((t - 1.0) / 0.5)
177
+ d.text((180, 330), "Pərdə", font=F(BOLD, 230), fill=mix(INK, LIGHTC, a))
178
+ b = ease((t - 1.5) / 0.5)
179
+ d.text((190, 640), "the curtain between your customers", font=F(LIGHT, 60), fill=mix(INK, MUTED, b))
180
+ d.text((190, 715), "and the cloud.", font=F(LIGHT, 60), fill=mix(INK, MUTED, b))
181
+
182
+
183
+ def scene_morph(img, d, t):
184
+ """16-22s: personal data turns into tags, live."""
185
+ text_center(d, 170, "Personal data becomes tags — on your own machine.", F(BOLD, 56), LIGHTC, ease(t / 0.4))
186
+ d.rounded_rectangle(MSG_BOX, 28, fill=PANEL, outline=(44, 58, 85), width=2)
187
+ p = clamp((t - 0.8) / 3.0)
188
+ if p < 0.5:
189
+ for x, y, ch, i in MSG_LAYOUT:
190
+ k = span_of(i)
191
+ col = RED if k is not None else (220, 227, 238)
192
+ if k is not None and p > 0.08 * k:
193
+ col = mix(RED, PANEL, (p - 0.08 * k) * 8)
194
+ d.text((x, y), ch, font=MSG_FONT, fill=col)
195
+ else:
196
+ q = ease((p - 0.5) * 2.5)
197
+ for x, y, ch, i in MASKED_LAYOUT:
198
+ is_tag = any(a <= i < b for a, b in TAGS)
199
+ col = mix(PANEL, GREEN, q) if is_tag else (220, 227, 238)
200
+ d.text((x, y), ch, font=MSG_FONT, fill=col)
201
+ if t > 4.0:
202
+ a = ease((t - 4.0) / 0.5)
203
+ text_center(d, 700, "The AI provider sees only this.", F(BOLD, 56), GREEN, a)
204
+
205
+
206
+ APP = Image.open(HERE / "shots" / "02_after_send.png").convert("RGB")
207
+ # crop rectangles on the 1920x1080 app screenshot: (x, y, w) with 16:9 height
208
+ SHOTS = [
209
+ (0.0, (0, 0, 1920), None),
210
+ (2.2, (360, 590, 520), "1 · Detected on your device"),
211
+ (5.2, (852, 590, 520), "2 · The AI sees only tags"),
212
+ (8.2, (1344, 590, 520), "3 · Answer restored locally"),
213
+ (11.2, (360, 560, 1500), "Leak check: passed · 6 values masked"),
214
+ ]
215
+
216
+
217
+ def scene_app(img, d, t):
218
+ """22-36s: real app footage with zooms."""
219
+ for k in range(len(SHOTS) - 1, -1, -1):
220
+ if t >= SHOTS[k][0]:
221
+ break
222
+ t0, r1, cap = SHOTS[k]
223
+ r0 = SHOTS[k - 1][1] if k else r1
224
+ q = ease_io((t - t0) / 1.0)
225
+ x, y, w = (a + (b - a) * q for a, b in zip(r0, r1))
226
+ h = w * 9 / 16
227
+ x = clamp(x, 0, 1920 - w)
228
+ y = clamp(y, 0, 1080 - h)
229
+ frame = APP.crop((int(x), int(y), int(x + w), int(y + h))).resize((W, H), Image.BILINEAR)
230
+ img.paste(frame, (0, 0))
231
+ d2 = ImageDraw.Draw(img)
232
+ d2.rectangle((0, 0, 380, 64), fill=RED)
233
+ d2.text((28, 8), "REAL APP · LIVE", font=F(BOLD, 36), fill=LIGHTC)
234
+ if cap:
235
+ a = ease((t - t0 - 0.6) / 0.4)
236
+ bar_w = d2.textlength(cap, font=F(BOLD, 60)) + 96
237
+ d2.rounded_rectangle(((W - bar_w) / 2, 900, (W + bar_w) / 2, 1010), 24, fill=mix((255, 255, 255), INK, a))
238
+ d2.text(((W - bar_w) / 2 + 48, 912), cap, font=F(BOLD, 60), fill=mix((255, 255, 255), LIGHTC, a))
239
+
240
+
241
+ def scene_stats(img, d, t):
242
+ """36-45s: the measured result."""
243
+ text_center(d, 120, "Tested on 1,000 Azerbaijani texts it had never seen", F(BOLD, 56), LIGHTC, ease(t / 0.4))
244
+ rows = [("Pərdə", 94.9, RED, 0.6), ("Generic filter", 23.9, MUTED, 1.0)]
245
+ for k, (name, val, col, start) in enumerate(rows):
246
+ y = 330 + k * 230
247
+ p = ease((t - start) / 1.8)
248
+ d.text((180, y), name, font=F(BOLD, 52), fill=LIGHTC)
249
+ d.rounded_rectangle((180, y + 80, 1740, y + 150), 16, fill=PANEL)
250
+ if p > 0:
251
+ d.rounded_rectangle((180, y + 80, 180 + 1560 * val / 100 * p, y + 150), 16, fill=col)
252
+ num = f"{val * p:.1f}%"
253
+ nw = d.textlength(num, font=F(BOLD, 72))
254
+ d.text((1740 - nw, y - 10), num, font=F(BOLD, 72), fill=col if k == 0 else MUTED)
255
+ if t > 3.4:
256
+ a = ease((t - 3.4) / 0.5)
257
+ text_center(d, 820, "of personal data protected", F(LIGHT, 52), MUTED, a)
258
+ text_center(d, 900, "Leak rate: 76.1% → 5.1%", F(BOLD, 64), AMBER, ease((t - 4.4) / 0.5))
259
+
260
+
261
+ CHIPS = ["FIN 6TR9K2L", "VÖEN", "AZ IBAN", "+994", "AZE / AA ID", "oğlu · qızı", "Rənanın", "Кириллица"]
262
+
263
+
264
+ def scene_az(img, d, t):
265
+ """45-50s: built for Azerbaijan."""
266
+ text_center(d, 140, "Built for Azerbaijani data", F(BOLD, 72), LIGHTC, ease(t / 0.4))
267
+ f = F(MONO, 44)
268
+ x0, y, x = 180, 330, 180
269
+ for k, c in enumerate(CHIPS):
270
+ w = d.textlength(c, font=f) + 72
271
+ if x + w > W - 180:
272
+ x, y = x0, y + 140
273
+ a = ease((t - 0.3 - 0.18 * k) / 0.3)
274
+ if a > 0:
275
+ dy = 30 * (1 - a)
276
+ d.rounded_rectangle((x, y + dy, x + w, y + 100 + dy), 50, fill=mix(INK, PANEL, a), outline=mix(INK, RED, a), width=3)
277
+ d.text((x + 36, y + 22 + dy), c, font=f, fill=mix(INK, LIGHTC, a))
278
+ x += w + 28
279
+ if t > 2.4:
280
+ a = ease((t - 2.4) / 0.5)
281
+ text_center(d, 720, "0.14 ms per message · runs on-premises · no GPU", F(BOLD, 52), AMBER, a)
282
+ text_center(d, 810, "Works with OpenAI · Grok · Gemini · local Ollama", F(LIGHT, 48), MUTED, ease((t - 3.0) / 0.5))
283
+
284
+
285
+ def scene_close(img, d, t):
286
+ """50-56s."""
287
+ if t < 0.9:
288
+ curtain(d, t)
289
+ return
290
+ d.rectangle((0, 0, W, H), fill=INK)
291
+ for k, op in enumerate([1, 0.6, 0.35, 0.15]):
292
+ x = 1340 + k * 60
293
+ d.rectangle((x, 0, x + 28, H), fill=mix(INK, mix(RED, AMBER, 0.3), op))
294
+ d.text((180, 250), "Let the AI read the request.", font=F(BOLD, 88), fill=mix(INK, LIGHTC, ease((t - 1.0) / 0.5)))
295
+ d.text((180, 370), "Never the person.", font=F(BOLD, 88), fill=mix(INK, RED, ease((t - 1.6) / 0.5)))
296
+ a = ease((t - 2.4) / 0.5)
297
+ d.text((180, 600), "Pərdə", font=F(BOLD, 120), fill=mix(INK, LIGHTC, a))
298
+ d.text((186, 770), "Try it: huggingface.co/spaces/Traxic/perde", font=F(REG, 44), fill=mix(INK, MUTED, a))
299
+ d.text((186, 840), "NeuroBridge 2026 · Enterprise track", font=F(REG, 36), fill=mix(INK, MUTED, a))
300
+ if t > DUR - 50.0 - 0.8:
301
+ fade = clamp((t - (DUR - 50.0 - 0.8)) / 0.8)
302
+ img.paste(Image.blend(img, Image.new("RGB", (W, H), (0, 0, 0)), fade))
303
+
304
+
305
+ SCENES = [
306
+ (0.0, 10.0, scene_hook),
307
+ (10.0, 13.0, scene_question),
308
+ (13.0, 16.0, scene_logo),
309
+ (16.0, 22.0, scene_morph),
310
+ (22.0, 36.0, scene_app),
311
+ (36.0, 45.0, scene_stats),
312
+ (45.0, 50.0, scene_az),
313
+ (50.0, DUR, scene_close),
314
+ ]
315
+ TRANSITIONS = [s for s, _, _ in SCENES[1:]]
316
+ HIGHLIGHTS = [5.4 + 0.45 * k for k in range(len(SPANS))]
317
+
318
+
319
+ def frame(t):
320
+ img = Image.new("RGB", (W, H), INK)
321
+ d = ImageDraw.Draw(img)
322
+ for s, e, fn in SCENES:
323
+ if s <= t < e:
324
+ fn(img, d, t - s)
325
+ # quick fade-in at each scene start
326
+ if t - s < 0.15 and s > 0 and fn not in (scene_logo, scene_close):
327
+ img = Image.blend(Image.new("RGB", (W, H), INK), img, (t - s) / 0.15)
328
+ break
329
+ return img
330
+
331
+
332
+ # --- soundtrack --------------------------------------------------------------
333
+
334
+ def soundtrack(path):
335
+ sr = 44100
336
+ n = int(DUR * sr)
337
+ out = np.zeros(n)
338
+ rng = np.random.default_rng(7)
339
+ beat = 60 / 118
340
+
341
+ def add(sig, at):
342
+ i = int(at * sr)
343
+ j = min(n, i + len(sig))
344
+ if i < n:
345
+ out[i:j] += sig[: j - i]
346
+
347
+ tt = np.arange(int(0.25 * sr)) / sr
348
+ kick = np.sin(2 * np.pi * (45 + 90 * np.exp(-tt * 30)) * tt) * np.exp(-tt * 9)
349
+ hat_t = np.arange(int(0.04 * sr)) / sr
350
+ hat = rng.normal(0, 1, len(hat_t)) * np.exp(-hat_t * 90)
351
+ hat = np.diff(np.concatenate([[0], hat]))
352
+ bass_notes = [55.0, 55.0, 65.41, 49.0]
353
+ t_cur, k = 0.0, 0
354
+ while t_cur < DUR:
355
+ intro = t_cur < 10.0
356
+ if not intro or k % 2 == 0:
357
+ add(kick * (0.5 if intro else 0.9), t_cur)
358
+ add(hat * 0.12, t_cur + beat / 2)
359
+ if not intro:
360
+ f = bass_notes[(k // 4) % 4]
361
+ bt = np.arange(int(beat * 0.9 * sr)) / sr
362
+ bass = (np.sin(2 * np.pi * f * bt) + 0.3 * np.sin(4 * np.pi * f * bt)) * np.exp(-bt * 3) * 0.35
363
+ add(bass, t_cur)
364
+ t_cur += beat
365
+ k += 1
366
+ # typing clicks while the message types (1.0-4.2s)
367
+ ct = np.arange(int(0.012 * sr)) / sr
368
+ click = rng.normal(0, 1, len(ct)) * np.exp(-ct * 400) * 0.15
369
+ for at in np.arange(1.0, 4.2, 0.045):
370
+ add(click, at)
371
+ # pops on each red highlight
372
+ pt = np.arange(int(0.12 * sr)) / sr
373
+ pop = np.sin(2 * np.pi * 880 * pt) * np.exp(-pt * 30) * 0.35
374
+ for at in HIGHLIGHTS:
375
+ add(pop, at)
376
+ # whooshes on scene changes
377
+ wt = np.arange(int(0.6 * sr)) / sr
378
+ noise = rng.normal(0, 1, len(wt))
379
+ noise = np.convolve(noise, np.ones(30) / 30, mode="same")
380
+ whoosh = noise * np.sin(np.pi * wt / wt[-1]) ** 2 * 0.6
381
+ for at in TRANSITIONS:
382
+ add(whoosh, at - 0.3)
383
+ fade = np.ones(n)
384
+ fl = int(2.0 * sr)
385
+ fade[-fl:] = np.linspace(1, 0, fl)
386
+ out = out * fade
387
+ out = out / max(1e-9, np.abs(out).max()) * 0.8
388
+ pcm = (out * 32767).astype(np.int16)
389
+ stereo = np.repeat(pcm[:, None], 2, axis=1)
390
+ with wave.open(str(path), "wb") as w:
391
+ w.setnchannels(2)
392
+ w.setsampwidth(2)
393
+ w.setframerate(sr)
394
+ w.writeframes(stereo.tobytes())
395
+
396
+
397
+ def main():
398
+ wav = HERE / "track.wav"
399
+ soundtrack(wav)
400
+ ff = imageio_ffmpeg.get_ffmpeg_exe()
401
+ proc = subprocess.Popen(
402
+ [ff, "-y", "-f", "rawvideo", "-pix_fmt", "rgb24", "-s", f"{W}x{H}", "-r", str(FPS), "-i", "-",
403
+ "-i", str(wav), "-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p",
404
+ "-c:a", "aac", "-b:a", "160k", "-shortest", "-movflags", "+faststart", str(OUT)],
405
+ stdin=subprocess.PIPE, stderr=subprocess.DEVNULL,
406
+ )
407
+ total = int(DUR * FPS)
408
+ for i in range(total):
409
+ proc.stdin.write(frame(i / FPS).tobytes())
410
+ if i % 150 == 0:
411
+ print(f"frame {i}/{total}", flush=True)
412
+ proc.stdin.close()
413
+ proc.wait()
414
+ print("done", OUT)
415
+
416
+
417
+ if __name__ == "__main__":
418
+ if len(sys.argv) > 3: # preview stills: render.py REPO OUT t1 t2 ...
419
+ for ts in sys.argv[3:]:
420
+ frame(float(ts)).save(HERE / f"preview_{ts}.png")
421
+ else:
422
+ main()