simonlesaumon/voice-protection-gateway-private

Voice Protection Gateway

Presentation

Voice Protection Gateway is a CPU-first V0 for creators who want to process a voice recording before publishing it.

Reduce voice cloning risk and add traceable evidence before publishing.

This system does not make a voice impossible to clone. It provides layered risk reduction, fallback watermark evidence, non-consent metadata, reports, and local red-team checks. It must not be presented as perfect protection.

Architecture

The V0 flow is:

1. Upload: the FastAPI backend accepts .wav, .mp3, .m4a, and .flac files and creates a local job.
2. Pipeline: audio is converted to mono 16 kHz WAV, normalized, analyzed with a simple RMS/energy VAD, split into voiced chunks, and processed with a CPU-safe protection profile.
3. Watermark fallback: protected.wav receives a deterministic HMAC-based PCM fallback watermark. Verification compares uploaded audio against known report watermark messages.
4. Adversarial prototype: optional lightweight, CPU-safe perturbations are applied on voiced regions depending on the selected profile. This is not VoiceCloak, AudioSeal, ECAPA optimization, or a SOTA defense.
5. AudioSeal optional: disabled by default. Phase 13 can apply the real AudioSeal backend when optional ML dependencies are installed and AUDIOSEAL_BACKEND=audioseal is explicitly set. The fallback watermark remains active either way.
6. C2PA optional: disabled by default. Phase 14 writes a C2PA manifest definition and can call c2patool to create real Content Credentials when explicitly configured.
7. Speaker embedding optional: disabled by default. A deterministic mock backend exists for tests, and the Phase 12 speechbrain backend supports real ECAPA-TDNN scoring only when explicitly enabled. No model is downloaded automatically.
8. Red-team loop: local CPU checks simulate MP3 roundtrip when ffmpeg exists, resampling, trimming, and fallback watermark re-verification.

Outputs are written locally under data/, outputs/, reports/, and logs/.

Installation

Linux CPU-only prerequisites:

• Python 3.11+
• Node.js and npm
• ffmpeg
• Git LFS if cloning from Hugging Face with large files

Setup:

cp .env.example .env
cd backend
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
cd ../frontend
npm ci

Optional SpeechBrain/ECAPA dependencies:

cd backend
source .venv/bin/activate
python -m pip install -r requirements-ml.txt

The real SpeechBrain and AudioSeal backends remain opt-in. Keep ALLOW_MODEL_DOWNLOADS=false unless you intentionally want to download configured models, or point model variables at local paths.

Backend

Run from the repository root:

cd backend
source .venv/bin/activate
python -m uvicorn app.main:app --host 0.0.0.0 --port 8000

Health check:

curl http://localhost:8000/health

Main API:

• GET /ready
• GET /api/config
• POST /api/upload
• POST /api/jobs/{id}/process
• GET /api/jobs/{id}
• GET /api/jobs/{id}/download
• GET /api/jobs/{id}/report.json
• GET /api/jobs/{id}/report.html
• POST /api/verify

Frontend

Run from the repository root:

cd frontend
NEXT_PUBLIC_API_BASE_URL=http://localhost:8000 npm run dev -- --port 3000

Open:

http://localhost:3000

Build check:

cd frontend
npm run build

Docker

cp .env.example .env
docker compose up --build

Ports:

• Backend: http://localhost:8000
• Frontend: http://localhost:3000

The backend image installs ffmpeg. Compose mounts local data/, outputs/, logs/, and reports/ into the backend container.

Cloud deployment notes are in docs/deploy_cloud.md.

CLI

Run commands from backend/ with the virtualenv activated.

Protect:

python -m app.cli.protect ./sample.wav --profile fast --out ./out

Verify:

python -m app.cli.verify ./out/protected.wav --report ./out/report.json

Benchmark:

python -m app.cli.benchmark ./data/benchmark/input --profile fast --out ./outputs/benchmark

The benchmark writes benchmark_results.json and benchmark_results.csv.

Utility Scripts

From the repository root:

./scripts/run_dev.sh
./scripts/test_all.sh
./scripts/smoke_cli.sh
./scripts/clean_outputs.sh

clean_outputs.sh removes known temporary output folders, temporary logs, benchmark caches, frontend build cache, and pytest cache. It does not remove data/ uploads, the SQLite database, or normal generated reports/outputs.

Environment Variables

Core:

BACKEND_PORT=8000
FRONTEND_PORT=3000
NEXT_PUBLIC_API_BASE_URL=http://localhost:8000
NO_RETENTION=true
VPG_DATA_DIR=data
VPG_OUTPUTS_DIR=outputs
VPG_REPORTS_DIR=reports
VPG_DATABASE_PATH=data/jobs.sqlite3
MAX_UPLOAD_MB=250
CORS_ALLOW_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
CORS_ALLOW_ORIGIN_REGEX=https://.*\.(loca\.lt|trycloudflare\.com)

Speaker similarity:

SPEAKER_EMBEDDING_BACKEND=disabled
SPEAKER_MODEL_NAME=speechbrain/spkrec-ecapa-voxceleb
SPEAKER_MODEL_DIR=models/speaker-ecapa
ALLOW_MODEL_DOWNLOADS=false
DEVICE=auto

Red-team:

REDTEAM_MODE=basic
RVC_BENCHMARK_COMMAND=
RVC_BENCHMARK_TIMEOUT_SECONDS=300
NEURAL_CODEC_BENCHMARK_COMMAND=
NEURAL_CODEC_BENCHMARK_TIMEOUT_SECONDS=300

AudioSeal:

AUDIOSEAL_BACKEND=disabled
AUDIOSEAL_GENERATOR_NAME=audioseal_wm_16bits
AUDIOSEAL_DETECTOR_NAME=audioseal_detector_16bits
AUDIOSEAL_ALPHA=1.0
AUDIOSEAL_MAX_WAV_MB=40

C2PA:

C2PA_BACKEND=disabled
C2PA_TOOL_PATH=c2patool
C2PA_SIDECAR=true
C2PA_SIGNER_PATH=

Allowed SPEAKER_EMBEDDING_BACKEND values:

• disabled: default, no speaker model is loaded.
• mock: deterministic local proxy for tests and demos.
• speechbrain: explicit opt-in real ECAPA/SpeechBrain backend. It lazy-loads and caches the model, resolves DEVICE=auto to CUDA when available and CPU otherwise, and retries on CPU after CUDA out-of-memory. Keep ALLOW_MODEL_DOWNLOADS=false unless you intentionally want model downloads, or point SPEAKER_MODEL_NAME at a local model path.

Allowed AUDIOSEAL_BACKEND values:

• disabled: default, no AudioSeal model is loaded.
• audioseal: explicit opt-in real AudioSeal backend. It lazy-loads the generator and detector, uses DEVICE=auto|cpu|cuda, refuses remote model loading while ALLOW_MODEL_DOWNLOADS=false, and skips very large WAV files above AUDIOSEAL_MAX_WAV_MB to keep public demos responsive.

Allowed C2PA_BACKEND values:

• disabled: default, writes c2pa_manifest.json as sidecar evidence only.
• c2patool: explicit opt-in real C2PA backend. Requires c2patool available through C2PA_TOOL_PATH. Set C2PA_SIGNER_PATH when delegating claim signing to an external signer. Trust validation depends on the verifier trust list and signer configuration.

Allowed REDTEAM_MODE values:

• basic: MP3 roundtrip if ffmpeg exists, resampling, trimming, watermark re-checks.
• rvc: runs the basic checks plus an optional external benchmark command from RVC_BENCHMARK_COMMAND. The command must include {input} and {output} placeholders and produce a WAV. No RVC model is bundled or downloaded.
• neural_codec: runs the basic checks plus an optional external benchmark command from NEURAL_CODEC_BENCHMARK_COMMAND. The command must include {input} and {output} placeholders and produce a WAV. No codec model is bundled or downloaded.
• off: skip red-team checks.

Profiles

• evidence_only: watermark and evidence metadata only.
• fast: very light masked noise and subtle timbre jitter on voiced regions.
• balanced_cpu: moderate CPU-safe perturbation on voiced regions.
• high_protection_cpu: stronger CPU-safe perturbation with explicit quality warnings.

Reports

report.json includes:

• metrics
• watermark
• audioseal
• c2pa
• speaker_embedding
• adversarial_protection
• redteam
• warnings
• pipeline_version
• hashes
• metadata

report.html displays:

• summary
• scores
• watermark
• AudioSeal status
• C2PA status
• red-team checks
• warnings and limits
• hashes

Limitations

• The fallback watermark has weak robustness and may fail after compression, denoising, trimming, re-recording, or neural reconstruction.
• Speaker similarity is a benchmark proxy, not proof that a voice cannot be cloned.
• Real ECAPA/SpeechBrain scoring covers one encoder family only. It does not represent every attacker or every voice conversion/TTS system.
• AudioSeal is optional and disabled by default. It complements the fallback watermark when enabled, but does not guarantee clone resistance.
• The red-team loop is local and lightweight by default. RVC benchmarking is an optional external harness only; users must provide their own local command and consent-cleared test material.
• Neural codec attack benchmarking is also an optional external harness only. The project does not ship or download codec attack models.
• The adversarial prototype is CPU-safe and lightweight. It is not a SOTA anti-cloning system.
• C2PA is optional and disabled by default. A real C2PA credential requires c2patool, a supported asset/signing setup, and verifier trust configuration.
• This system does not make a voice impossible to clone.

Tests

Backend:

cd backend
source .venv/bin/activate
python -m pytest

Frontend:

cd frontend
npm run build
npm audit --audit-level=moderate

Full local suite:

./scripts/test_all.sh
./scripts/smoke_cli.sh

Roadmap

• Phase 12: ECAPA real / advanced SpeechBrain integration.
• Phase 13: AudioSeal.
• Phase 14: real C2PA.
• Phase 15: RVC benchmark.
• Phase 16: neural codec attacks.
• Phase 17: SaaS cloud.