Upload 3 files
Browse files
README.md
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Context-Aware Audio Correction
|
| 2 |
+
|
| 3 |
+
This Gradio app transcribes an audio sample, retrieves relevant passages from a reference document, and asks a language model to correct likely ASR mistakes using only document-backed evidence.
|
| 4 |
+
|
| 5 |
+
## Main Flow
|
| 6 |
+
|
| 7 |
+
```text
|
| 8 |
+
Upload document
|
| 9 |
+
-> extract text
|
| 10 |
+
-> split into passages
|
| 11 |
+
-> upload or record audio
|
| 12 |
+
-> transcribe with Whisper
|
| 13 |
+
-> retrieve related document passages
|
| 14 |
+
-> correct near-sound and domain-term errors
|
| 15 |
+
```
|
| 16 |
+
|
| 17 |
+
## Recognition Profiles
|
| 18 |
+
|
| 19 |
+
The app separates English, Chinese, and automatic recognition with explicit ASR profiles:
|
| 20 |
+
|
| 21 |
+
| Profile | Default model | Use case |
|
| 22 |
+
|---|---|---|
|
| 23 |
+
| English optimized | `openai/whisper-small.en` | English-only lectures and presentations |
|
| 24 |
+
| Chinese | `openai/whisper-small` | Mandarin recordings |
|
| 25 |
+
| Auto detect | `openai/whisper-small` | Unknown or mixed-language recordings |
|
| 26 |
+
|
| 27 |
+
## Local Run
|
| 28 |
+
|
| 29 |
+
```powershell
|
| 30 |
+
python -m venv .venv
|
| 31 |
+
.\.venv\Scripts\Activate.ps1
|
| 32 |
+
pip install -r requirements.txt
|
| 33 |
+
$env:HF_TOKEN="your Hugging Face token"
|
| 34 |
+
python app.py
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
Open:
|
| 38 |
+
|
| 39 |
+
```text
|
| 40 |
+
http://127.0.0.1:7860
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
## Hugging Face Spaces
|
| 44 |
+
|
| 45 |
+
Upload these files to the Space root directory:
|
| 46 |
+
|
| 47 |
+
```text
|
| 48 |
+
app.py
|
| 49 |
+
requirements.txt
|
| 50 |
+
packages.txt
|
| 51 |
+
README.md
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
Then add this secret in `Settings -> Variables and secrets`:
|
| 55 |
+
|
| 56 |
+
```text
|
| 57 |
+
HF_TOKEN=your Hugging Face token
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
## Optional Variables
|
| 61 |
+
|
| 62 |
+
```text
|
| 63 |
+
ASR_MODEL_EN=openai/whisper-small.en
|
| 64 |
+
ASR_MODEL_ZH=openai/whisper-small
|
| 65 |
+
ASR_MODEL_AUTO=openai/whisper-small
|
| 66 |
+
EMBEDDING_MODEL=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
|
| 67 |
+
LLM_MODEL=Qwen/Qwen2.5-7B-Instruct-1M
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
`ASR_MODEL` is still supported as the default multilingual ASR model for Chinese and Auto profiles.
|
| 71 |
+
|
| 72 |
+
## Notes
|
| 73 |
+
|
| 74 |
+
- Scanned PDFs need OCR before upload.
|
| 75 |
+
- Free CPU Spaces can be slow on the first run because models must be downloaded and loaded.
|
| 76 |
+
- Start with short audio samples, around 20 seconds to 2 minutes.
|
| 77 |
+
- The correction step is evidence-bound. It should not freely rewrite the transcript.
|
app.py
ADDED
|
@@ -0,0 +1,518 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import json
|
| 2 |
+
import os
|
| 3 |
+
import re
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
import gradio as gr
|
| 7 |
+
import numpy as np
|
| 8 |
+
import pdfplumber
|
| 9 |
+
from docx import Document
|
| 10 |
+
from openai import OpenAI
|
| 11 |
+
from sentence_transformers import SentenceTransformer
|
| 12 |
+
from transformers import pipeline
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
EMBEDDING_MODEL = os.getenv(
|
| 16 |
+
"EMBEDDING_MODEL", "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
|
| 17 |
+
)
|
| 18 |
+
LLM_MODEL = os.getenv("LLM_MODEL", "Qwen/Qwen2.5-7B-Instruct-1M")
|
| 19 |
+
HF_TOKEN = os.getenv("HF_TOKEN")
|
| 20 |
+
DEFAULT_MULTILINGUAL_ASR_MODEL = os.getenv("ASR_MODEL", "openai/whisper-small")
|
| 21 |
+
|
| 22 |
+
ASR_PROFILES = {
|
| 23 |
+
"English optimized - Whisper small.en": {
|
| 24 |
+
"model": os.getenv("ASR_MODEL_EN", "openai/whisper-small.en"),
|
| 25 |
+
"language": None,
|
| 26 |
+
"description": "Best default for English-only lectures and presentations.",
|
| 27 |
+
},
|
| 28 |
+
"Chinese - Whisper multilingual small": {
|
| 29 |
+
"model": os.getenv("ASR_MODEL_ZH", DEFAULT_MULTILINGUAL_ASR_MODEL),
|
| 30 |
+
"language": "chinese",
|
| 31 |
+
"description": "Use this for Mandarin recordings and Chinese documents.",
|
| 32 |
+
},
|
| 33 |
+
"Auto detect - Whisper multilingual small": {
|
| 34 |
+
"model": os.getenv("ASR_MODEL_AUTO", DEFAULT_MULTILINGUAL_ASR_MODEL),
|
| 35 |
+
"language": None,
|
| 36 |
+
"description": "Use this when the recording language is uncertain or mixed.",
|
| 37 |
+
},
|
| 38 |
+
}
|
| 39 |
+
|
| 40 |
+
asr_pipelines = {}
|
| 41 |
+
embedding_model = None
|
| 42 |
+
llm_client = None
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
APP_CSS = """
|
| 46 |
+
:root {
|
| 47 |
+
--brand: #0f766e;
|
| 48 |
+
--brand-strong: #115e59;
|
| 49 |
+
--ink: #111827;
|
| 50 |
+
--muted: #64748b;
|
| 51 |
+
--line: #d8ded9;
|
| 52 |
+
--paper: #ffffff;
|
| 53 |
+
--wash: #f6f7f2;
|
| 54 |
+
--accent: #c2410c;
|
| 55 |
+
}
|
| 56 |
+
|
| 57 |
+
body,
|
| 58 |
+
.gradio-container {
|
| 59 |
+
background: var(--wash) !important;
|
| 60 |
+
color: var(--ink);
|
| 61 |
+
font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
| 62 |
+
}
|
| 63 |
+
|
| 64 |
+
.main {
|
| 65 |
+
max-width: 1180px !important;
|
| 66 |
+
margin: 0 auto !important;
|
| 67 |
+
}
|
| 68 |
+
|
| 69 |
+
.app-shell {
|
| 70 |
+
padding: 28px 28px 12px;
|
| 71 |
+
border-bottom: 1px solid var(--line);
|
| 72 |
+
}
|
| 73 |
+
|
| 74 |
+
.app-kicker {
|
| 75 |
+
margin: 0 0 8px;
|
| 76 |
+
color: var(--brand-strong);
|
| 77 |
+
font-size: 12px;
|
| 78 |
+
font-weight: 700;
|
| 79 |
+
letter-spacing: 0.08em;
|
| 80 |
+
text-transform: uppercase;
|
| 81 |
+
}
|
| 82 |
+
|
| 83 |
+
.app-title {
|
| 84 |
+
margin: 0;
|
| 85 |
+
color: var(--ink);
|
| 86 |
+
font-size: 34px;
|
| 87 |
+
line-height: 1.12;
|
| 88 |
+
letter-spacing: 0;
|
| 89 |
+
}
|
| 90 |
+
|
| 91 |
+
.app-subtitle {
|
| 92 |
+
margin: 12px 0 0;
|
| 93 |
+
max-width: 780px;
|
| 94 |
+
color: var(--muted);
|
| 95 |
+
font-size: 16px;
|
| 96 |
+
line-height: 1.6;
|
| 97 |
+
}
|
| 98 |
+
|
| 99 |
+
.status-strip {
|
| 100 |
+
display: grid;
|
| 101 |
+
grid-template-columns: repeat(3, minmax(0, 1fr));
|
| 102 |
+
gap: 10px;
|
| 103 |
+
margin-top: 20px;
|
| 104 |
+
}
|
| 105 |
+
|
| 106 |
+
.status-item {
|
| 107 |
+
background: #ffffff;
|
| 108 |
+
border: 1px solid var(--line);
|
| 109 |
+
border-radius: 8px;
|
| 110 |
+
padding: 12px 14px;
|
| 111 |
+
}
|
| 112 |
+
|
| 113 |
+
.status-label {
|
| 114 |
+
color: var(--muted);
|
| 115 |
+
font-size: 12px;
|
| 116 |
+
margin-bottom: 4px;
|
| 117 |
+
}
|
| 118 |
+
|
| 119 |
+
.status-value {
|
| 120 |
+
color: var(--ink);
|
| 121 |
+
font-weight: 700;
|
| 122 |
+
font-size: 14px;
|
| 123 |
+
}
|
| 124 |
+
|
| 125 |
+
.gradio-container .block {
|
| 126 |
+
border-radius: 8px !important;
|
| 127 |
+
}
|
| 128 |
+
|
| 129 |
+
.gradio-container button.primary {
|
| 130 |
+
background: var(--brand) !important;
|
| 131 |
+
border-color: var(--brand) !important;
|
| 132 |
+
}
|
| 133 |
+
|
| 134 |
+
.gradio-container button.primary:hover {
|
| 135 |
+
background: var(--brand-strong) !important;
|
| 136 |
+
border-color: var(--brand-strong) !important;
|
| 137 |
+
}
|
| 138 |
+
|
| 139 |
+
textarea,
|
| 140 |
+
input,
|
| 141 |
+
.wrap {
|
| 142 |
+
border-radius: 8px !important;
|
| 143 |
+
}
|
| 144 |
+
|
| 145 |
+
.output-panel textarea {
|
| 146 |
+
font-size: 14px !important;
|
| 147 |
+
line-height: 1.55 !important;
|
| 148 |
+
}
|
| 149 |
+
|
| 150 |
+
.correction-notes,
|
| 151 |
+
.evidence-panel {
|
| 152 |
+
background: var(--paper);
|
| 153 |
+
}
|
| 154 |
+
|
| 155 |
+
@media (max-width: 760px) {
|
| 156 |
+
.app-shell {
|
| 157 |
+
padding: 22px 18px 8px;
|
| 158 |
+
}
|
| 159 |
+
|
| 160 |
+
.app-title {
|
| 161 |
+
font-size: 28px;
|
| 162 |
+
}
|
| 163 |
+
|
| 164 |
+
.status-strip {
|
| 165 |
+
grid-template-columns: 1fr;
|
| 166 |
+
}
|
| 167 |
+
}
|
| 168 |
+
"""
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
def get_asr_pipeline(model_id: str):
|
| 172 |
+
if model_id not in asr_pipelines:
|
| 173 |
+
asr_pipelines[model_id] = pipeline(
|
| 174 |
+
"automatic-speech-recognition",
|
| 175 |
+
model=model_id,
|
| 176 |
+
device=-1,
|
| 177 |
+
)
|
| 178 |
+
return asr_pipelines[model_id]
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
def get_embedding_model():
|
| 182 |
+
global embedding_model
|
| 183 |
+
if embedding_model is None:
|
| 184 |
+
embedding_model = SentenceTransformer(EMBEDDING_MODEL)
|
| 185 |
+
return embedding_model
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def get_llm_client():
|
| 189 |
+
global llm_client
|
| 190 |
+
if not HF_TOKEN:
|
| 191 |
+
return None
|
| 192 |
+
if llm_client is None:
|
| 193 |
+
llm_client = OpenAI(
|
| 194 |
+
base_url="https://router.huggingface.co/v1",
|
| 195 |
+
api_key=HF_TOKEN,
|
| 196 |
+
)
|
| 197 |
+
return llm_client
|
| 198 |
+
|
| 199 |
+
|
| 200 |
+
def read_text_file(path: Path) -> str:
|
| 201 |
+
for encoding in ("utf-8", "gb18030"):
|
| 202 |
+
try:
|
| 203 |
+
return path.read_text(encoding=encoding)
|
| 204 |
+
except UnicodeDecodeError:
|
| 205 |
+
continue
|
| 206 |
+
return path.read_text(errors="ignore")
|
| 207 |
+
|
| 208 |
+
|
| 209 |
+
def extract_document_text(file_path: str) -> str:
|
| 210 |
+
path = Path(file_path)
|
| 211 |
+
suffix = path.suffix.lower()
|
| 212 |
+
|
| 213 |
+
if suffix == ".txt":
|
| 214 |
+
text = read_text_file(path)
|
| 215 |
+
elif suffix == ".pdf":
|
| 216 |
+
pages = []
|
| 217 |
+
with pdfplumber.open(path) as pdf:
|
| 218 |
+
for page in pdf.pages:
|
| 219 |
+
pages.append(page.extract_text() or "")
|
| 220 |
+
text = "\n".join(pages)
|
| 221 |
+
elif suffix == ".docx":
|
| 222 |
+
doc = Document(path)
|
| 223 |
+
text = "\n".join(p.text for p in doc.paragraphs)
|
| 224 |
+
else:
|
| 225 |
+
raise ValueError("Only PDF, DOCX, and TXT documents are supported.")
|
| 226 |
+
|
| 227 |
+
text = re.sub(r"[ \t]+", " ", text)
|
| 228 |
+
text = re.sub(r"\n{3,}", "\n\n", text)
|
| 229 |
+
return text.strip()
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
def split_into_chunks(text: str, max_chars: int = 700, overlap: int = 90) -> list[str]:
|
| 233 |
+
paragraphs = re.split(r"\n\s*\n+", text)
|
| 234 |
+
pieces = []
|
| 235 |
+
for paragraph in paragraphs:
|
| 236 |
+
paragraph = paragraph.strip()
|
| 237 |
+
if not paragraph:
|
| 238 |
+
continue
|
| 239 |
+
pieces.extend(re.split(r"(?<=[.!?;:])\s+", paragraph))
|
| 240 |
+
|
| 241 |
+
pieces = [p.strip() for p in pieces if p and p.strip()]
|
| 242 |
+
|
| 243 |
+
chunks = []
|
| 244 |
+
current = ""
|
| 245 |
+
for piece in pieces:
|
| 246 |
+
if len(piece) > max_chars:
|
| 247 |
+
if current:
|
| 248 |
+
chunks.append(current)
|
| 249 |
+
current = ""
|
| 250 |
+
step = max_chars - overlap
|
| 251 |
+
for start in range(0, len(piece), step):
|
| 252 |
+
chunks.append(piece[start : start + max_chars])
|
| 253 |
+
continue
|
| 254 |
+
|
| 255 |
+
candidate = piece if not current else f"{current}\n{piece}"
|
| 256 |
+
if len(candidate) <= max_chars:
|
| 257 |
+
current = candidate
|
| 258 |
+
else:
|
| 259 |
+
chunks.append(current)
|
| 260 |
+
current = piece
|
| 261 |
+
|
| 262 |
+
if current:
|
| 263 |
+
chunks.append(current)
|
| 264 |
+
|
| 265 |
+
return [chunk for chunk in chunks if len(chunk) >= 20]
|
| 266 |
+
|
| 267 |
+
|
| 268 |
+
def resolve_asr_profile(profile_name: str) -> dict:
|
| 269 |
+
return ASR_PROFILES.get(profile_name, next(iter(ASR_PROFILES.values())))
|
| 270 |
+
|
| 271 |
+
|
| 272 |
+
def transcribe_audio(audio_path: str, profile_name: str) -> str:
|
| 273 |
+
profile = resolve_asr_profile(profile_name)
|
| 274 |
+
generate_kwargs = {"task": "transcribe"}
|
| 275 |
+
if profile["language"]:
|
| 276 |
+
generate_kwargs["language"] = profile["language"]
|
| 277 |
+
|
| 278 |
+
result = get_asr_pipeline(profile["model"])(audio_path, generate_kwargs=generate_kwargs)
|
| 279 |
+
if isinstance(result, dict):
|
| 280 |
+
return str(result.get("text", "")).strip()
|
| 281 |
+
return str(result).strip()
|
| 282 |
+
|
| 283 |
+
|
| 284 |
+
def retrieve_contexts(raw_transcript: str, chunks: list[str], top_k: int):
|
| 285 |
+
model = get_embedding_model()
|
| 286 |
+
doc_vectors = model.encode(chunks, normalize_embeddings=True)
|
| 287 |
+
query_vector = model.encode([raw_transcript], normalize_embeddings=True)[0]
|
| 288 |
+
scores = np.matmul(doc_vectors, query_vector)
|
| 289 |
+
top_indices = np.argsort(scores)[::-1][:top_k]
|
| 290 |
+
return [(int(i), float(scores[i]), chunks[int(i)]) for i in top_indices]
|
| 291 |
+
|
| 292 |
+
|
| 293 |
+
def build_correction_prompt(raw_transcript: str, contexts) -> list[dict]:
|
| 294 |
+
context_text = "\n\n".join(
|
| 295 |
+
f"[Document passage {idx + 1} | similarity {score:.3f}]\n{text}"
|
| 296 |
+
for idx, score, text in contexts
|
| 297 |
+
)
|
| 298 |
+
|
| 299 |
+
system_prompt = (
|
| 300 |
+
"You are a strict ASR correction assistant. Correct the transcript only when the "
|
| 301 |
+
"provided document context gives clear evidence. Focus on homophones, near-sound "
|
| 302 |
+
"mistakes, technical terms, names, acronyms, chapter titles, and domain-specific "
|
| 303 |
+
"phrases. Preserve the original sentence structure as much as possible. Do not "
|
| 304 |
+
"summarize, rewrite freely, or add information that was not spoken."
|
| 305 |
+
)
|
| 306 |
+
user_prompt = f"""
|
| 307 |
+
Correct the ASR transcript using the document passages below.
|
| 308 |
+
|
| 309 |
+
Rules:
|
| 310 |
+
1. Treat the raw transcript as the primary text.
|
| 311 |
+
2. Make only evidence-backed corrections.
|
| 312 |
+
3. Prefer keeping the original word when the document context is not strong enough.
|
| 313 |
+
4. Output JSON only. Do not output Markdown.
|
| 314 |
+
|
| 315 |
+
JSON schema:
|
| 316 |
+
{{
|
| 317 |
+
"corrected_text": "the complete corrected transcript",
|
| 318 |
+
"changes": [
|
| 319 |
+
{{
|
| 320 |
+
"original": "incorrect word or phrase",
|
| 321 |
+
"corrected": "corrected word or phrase",
|
| 322 |
+
"reason": "why the document supports this correction"
|
| 323 |
+
}}
|
| 324 |
+
]
|
| 325 |
+
}}
|
| 326 |
+
|
| 327 |
+
Document passages:
|
| 328 |
+
{context_text}
|
| 329 |
+
|
| 330 |
+
Raw ASR transcript:
|
| 331 |
+
{raw_transcript}
|
| 332 |
+
""".strip()
|
| 333 |
+
|
| 334 |
+
return [
|
| 335 |
+
{"role": "system", "content": system_prompt},
|
| 336 |
+
{"role": "user", "content": user_prompt},
|
| 337 |
+
]
|
| 338 |
+
|
| 339 |
+
|
| 340 |
+
def parse_json_response(text: str):
|
| 341 |
+
try:
|
| 342 |
+
return json.loads(text)
|
| 343 |
+
except json.JSONDecodeError:
|
| 344 |
+
match = re.search(r"\{.*\}", text, flags=re.S)
|
| 345 |
+
if match:
|
| 346 |
+
return json.loads(match.group(0))
|
| 347 |
+
raise ValueError("The language model did not return valid JSON.")
|
| 348 |
+
|
| 349 |
+
|
| 350 |
+
def correct_with_llm(raw_transcript: str, contexts):
|
| 351 |
+
client = get_llm_client()
|
| 352 |
+
if client is None:
|
| 353 |
+
return {
|
| 354 |
+
"corrected_text": raw_transcript,
|
| 355 |
+
"changes": [
|
| 356 |
+
{
|
| 357 |
+
"original": "LLM correction skipped",
|
| 358 |
+
"corrected": "LLM correction skipped",
|
| 359 |
+
"reason": "HF_TOKEN is not set. Add HF_TOKEN locally or in Hugging Face Spaces secrets.",
|
| 360 |
+
}
|
| 361 |
+
],
|
| 362 |
+
}
|
| 363 |
+
|
| 364 |
+
completion = client.chat.completions.create(
|
| 365 |
+
model=LLM_MODEL,
|
| 366 |
+
messages=build_correction_prompt(raw_transcript, contexts),
|
| 367 |
+
temperature=0.1,
|
| 368 |
+
max_tokens=1200,
|
| 369 |
+
)
|
| 370 |
+
content = completion.choices[0].message.content
|
| 371 |
+
return parse_json_response(content)
|
| 372 |
+
|
| 373 |
+
|
| 374 |
+
def format_contexts(contexts) -> str:
|
| 375 |
+
blocks = []
|
| 376 |
+
for rank, (idx, score, text) in enumerate(contexts, start=1):
|
| 377 |
+
blocks.append(f"### Passage {rank}\nSimilarity: `{score:.3f}`\n\n{text}")
|
| 378 |
+
return "\n\n---\n\n".join(blocks)
|
| 379 |
+
|
| 380 |
+
|
| 381 |
+
def format_changes(changes) -> str:
|
| 382 |
+
if not changes:
|
| 383 |
+
return "No document-backed correction was needed."
|
| 384 |
+
|
| 385 |
+
lines = []
|
| 386 |
+
for item in changes:
|
| 387 |
+
original = item.get("original", "")
|
| 388 |
+
corrected = item.get("corrected", "")
|
| 389 |
+
reason = item.get("reason", "")
|
| 390 |
+
lines.append(f"- `{original}` -> `{corrected}`: {reason}")
|
| 391 |
+
return "\n".join(lines)
|
| 392 |
+
|
| 393 |
+
|
| 394 |
+
def run_app(document_file, audio_file, profile_name, top_k):
|
| 395 |
+
if document_file is None:
|
| 396 |
+
raise gr.Error("Upload a PDF, DOCX, or TXT reference document first.")
|
| 397 |
+
if audio_file is None:
|
| 398 |
+
raise gr.Error("Upload or record an audio sample first.")
|
| 399 |
+
|
| 400 |
+
document_text = extract_document_text(document_file)
|
| 401 |
+
if not document_text:
|
| 402 |
+
raise gr.Error("No text was extracted from the document. Scanned PDFs need OCR first.")
|
| 403 |
+
|
| 404 |
+
chunks = split_into_chunks(document_text)
|
| 405 |
+
if not chunks:
|
| 406 |
+
raise gr.Error("The document is too short to build context.")
|
| 407 |
+
|
| 408 |
+
raw_transcript = transcribe_audio(audio_file, profile_name)
|
| 409 |
+
if not raw_transcript:
|
| 410 |
+
raise gr.Error("No speech text was recognized from the audio.")
|
| 411 |
+
|
| 412 |
+
contexts = retrieve_contexts(raw_transcript, chunks, int(top_k))
|
| 413 |
+
correction = correct_with_llm(raw_transcript, contexts)
|
| 414 |
+
|
| 415 |
+
corrected_text = correction.get("corrected_text", raw_transcript)
|
| 416 |
+
changes = correction.get("changes", [])
|
| 417 |
+
|
| 418 |
+
return (
|
| 419 |
+
raw_transcript,
|
| 420 |
+
corrected_text,
|
| 421 |
+
format_changes(changes),
|
| 422 |
+
format_contexts(contexts),
|
| 423 |
+
)
|
| 424 |
+
|
| 425 |
+
|
| 426 |
+
theme = gr.themes.Soft(
|
| 427 |
+
primary_hue="teal",
|
| 428 |
+
secondary_hue="orange",
|
| 429 |
+
neutral_hue="zinc",
|
| 430 |
+
radius_size="sm",
|
| 431 |
+
)
|
| 432 |
+
|
| 433 |
+
with gr.Blocks(
|
| 434 |
+
title="Context-Aware Audio Correction",
|
| 435 |
+
theme=theme,
|
| 436 |
+
css=APP_CSS,
|
| 437 |
+
) as demo:
|
| 438 |
+
gr.HTML(
|
| 439 |
+
"""
|
| 440 |
+
<section class="app-shell">
|
| 441 |
+
<p class="app-kicker">Hugging Face ASR + document retrieval</p>
|
| 442 |
+
<h1 class="app-title">Context-Aware Audio Correction</h1>
|
| 443 |
+
<p class="app-subtitle">
|
| 444 |
+
Upload a reference document and an audio clip. The app transcribes speech,
|
| 445 |
+
retrieves matching document passages, and corrects likely ASR mistakes using
|
| 446 |
+
only document-backed evidence.
|
| 447 |
+
</p>
|
| 448 |
+
<div class="status-strip">
|
| 449 |
+
<div class="status-item">
|
| 450 |
+
<div class="status-label">ASR profiles</div>
|
| 451 |
+
<div class="status-value">English / Chinese / Auto</div>
|
| 452 |
+
</div>
|
| 453 |
+
<div class="status-item">
|
| 454 |
+
<div class="status-label">Context engine</div>
|
| 455 |
+
<div class="status-value">Sentence embeddings</div>
|
| 456 |
+
</div>
|
| 457 |
+
<div class="status-item">
|
| 458 |
+
<div class="status-label">Correction policy</div>
|
| 459 |
+
<div class="status-value">Evidence-bound</div>
|
| 460 |
+
</div>
|
| 461 |
+
</div>
|
| 462 |
+
</section>
|
| 463 |
+
"""
|
| 464 |
+
)
|
| 465 |
+
|
| 466 |
+
with gr.Row():
|
| 467 |
+
with gr.Column(scale=1, min_width=320):
|
| 468 |
+
document_input = gr.File(
|
| 469 |
+
label="Reference document",
|
| 470 |
+
file_types=[".pdf", ".docx", ".txt"],
|
| 471 |
+
type="filepath",
|
| 472 |
+
)
|
| 473 |
+
audio_input = gr.Audio(
|
| 474 |
+
label="Audio sample",
|
| 475 |
+
sources=["upload", "microphone"],
|
| 476 |
+
type="filepath",
|
| 477 |
+
)
|
| 478 |
+
with gr.Column(scale=1, min_width=320):
|
| 479 |
+
profile_input = gr.Radio(
|
| 480 |
+
label="Recognition profile",
|
| 481 |
+
choices=list(ASR_PROFILES.keys()),
|
| 482 |
+
value="English optimized - Whisper small.en",
|
| 483 |
+
info=(
|
| 484 |
+
"English uses an English-only Whisper model. Chinese and Auto use "
|
| 485 |
+
"the multilingual Whisper model."
|
| 486 |
+
),
|
| 487 |
+
)
|
| 488 |
+
top_k_input = gr.Slider(
|
| 489 |
+
label="Document passages to retrieve",
|
| 490 |
+
minimum=1,
|
| 491 |
+
maximum=8,
|
| 492 |
+
value=4,
|
| 493 |
+
step=1,
|
| 494 |
+
)
|
| 495 |
+
submit_button = gr.Button("Transcribe and correct", variant="primary")
|
| 496 |
+
|
| 497 |
+
with gr.Row(elem_classes=["output-panel"]):
|
| 498 |
+
raw_output = gr.Textbox(label="Raw Whisper transcript", lines=9)
|
| 499 |
+
corrected_output = gr.Textbox(label="Context-corrected transcript", lines=9)
|
| 500 |
+
|
| 501 |
+
changes_output = gr.Markdown(
|
| 502 |
+
label="Correction notes",
|
| 503 |
+
elem_classes=["correction-notes"],
|
| 504 |
+
)
|
| 505 |
+
contexts_output = gr.Markdown(
|
| 506 |
+
label="Document evidence",
|
| 507 |
+
elem_classes=["evidence-panel"],
|
| 508 |
+
)
|
| 509 |
+
|
| 510 |
+
submit_button.click(
|
| 511 |
+
fn=run_app,
|
| 512 |
+
inputs=[document_input, audio_input, profile_input, top_k_input],
|
| 513 |
+
outputs=[raw_output, corrected_output, changes_output, contexts_output],
|
| 514 |
+
)
|
| 515 |
+
|
| 516 |
+
|
| 517 |
+
if __name__ == "__main__":
|
| 518 |
+
demo.launch(share=True)
|
网站搭建说明.md
ADDED
|
@@ -0,0 +1,472 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 基于 Hugging Face 的文档感知音频识别纠错网站搭建说明
|
| 2 |
+
|
| 3 |
+
## 1. 项目目标
|
| 4 |
+
|
| 5 |
+
本项目要实现一个网页应用:用户上传一份参考文档和一段音频后,系统先把音频识别成文字,再根据参考文档内容纠正识别结果中的错误。
|
| 6 |
+
|
| 7 |
+
普通语音识别系统经常会把专业词、缩写、人名、课程术语识别成发音相近但意思错误的内容。本项目的核心思路是:不只依赖语音模型本身,而是额外引入文档上下文,让系统知道这段录音可能在讲什么。
|
| 8 |
+
|
| 9 |
+
示例:
|
| 10 |
+
|
| 11 |
+
```text
|
| 12 |
+
原始识别结果:
|
| 13 |
+
This lecture explains back propagation and banishing gradients.
|
| 14 |
+
|
| 15 |
+
文档中出现:
|
| 16 |
+
backpropagation, vanishing gradients
|
| 17 |
+
|
| 18 |
+
纠错后:
|
| 19 |
+
This lecture explains backpropagation and vanishing gradients.
|
| 20 |
+
```
|
| 21 |
+
|
| 22 |
+
## 2. 系统整体思路
|
| 23 |
+
|
| 24 |
+
系统分为四个核心模块:
|
| 25 |
+
|
| 26 |
+
```text
|
| 27 |
+
文档上传
|
| 28 |
+
-> 文档文字提取
|
| 29 |
+
-> 文档切片
|
| 30 |
+
-> 文档语义向量化
|
| 31 |
+
|
| 32 |
+
音频上传
|
| 33 |
+
-> Whisper 语音识别
|
| 34 |
+
-> 得到原始转写文本
|
| 35 |
+
|
| 36 |
+
语义检索
|
| 37 |
+
-> 用原始转写文本检索最相关的文档片段
|
| 38 |
+
|
| 39 |
+
大模型纠错
|
| 40 |
+
-> 把原始转写和相关文档片段交给大模型
|
| 41 |
+
-> 要求大模型只根据文档证据纠正近音词和专业词
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
最终网页输出四部分内容:
|
| 45 |
+
|
| 46 |
+
```text
|
| 47 |
+
1. Raw Whisper transcript
|
| 48 |
+
2. Context-corrected transcript
|
| 49 |
+
3. Correction notes
|
| 50 |
+
4. Document evidence
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
## 3. 使用的技术
|
| 54 |
+
|
| 55 |
+
本项目主要使用以下技术:
|
| 56 |
+
|
| 57 |
+
```text
|
| 58 |
+
Python
|
| 59 |
+
Gradio
|
| 60 |
+
Hugging Face Spaces
|
| 61 |
+
Hugging Face Transformers
|
| 62 |
+
Whisper ASR model
|
| 63 |
+
SentenceTransformer embedding model
|
| 64 |
+
Hugging Face Router / Inference Provider
|
| 65 |
+
Qwen instruction model
|
| 66 |
+
```
|
| 67 |
+
|
| 68 |
+
各部分作用如下:
|
| 69 |
+
|
| 70 |
+
| 技术 | 作用 |
|
| 71 |
+
|---|---|
|
| 72 |
+
| Gradio | 快速搭建网页界面 |
|
| 73 |
+
| Hugging Face Spaces | 部署网页应用 |
|
| 74 |
+
| Transformers pipeline | 调用 Whisper 做语音识别 |
|
| 75 |
+
| Whisper | 把音频转成文字 |
|
| 76 |
+
| SentenceTransformer | 把文档片段和识别文本转换成向量 |
|
| 77 |
+
| NumPy | 计算文本向量相似度 |
|
| 78 |
+
| pdfplumber | 提取 PDF 文字 |
|
| 79 |
+
| python-docx | 提取 Word 文档文字 |
|
| 80 |
+
| Hugging Face Router | 调用在线大模型做纠错 |
|
| 81 |
+
|
| 82 |
+
## 4. 项目文件结构
|
| 83 |
+
|
| 84 |
+
项目根目录需要包含这些文件:
|
| 85 |
+
|
| 86 |
+
```text
|
| 87 |
+
app.py
|
| 88 |
+
requirements.txt
|
| 89 |
+
packages.txt
|
| 90 |
+
README.md
|
| 91 |
+
网站搭建说明.md
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
+
其中:
|
| 95 |
+
|
| 96 |
+
| 文件 | 作用 |
|
| 97 |
+
|---|---|
|
| 98 |
+
| app.py | 网站主程序 |
|
| 99 |
+
| requirements.txt | Python 依赖列表 |
|
| 100 |
+
| packages.txt | 系统依赖,例如 ffmpeg |
|
| 101 |
+
| README.md | 项目简要说明 |
|
| 102 |
+
| 网站搭建说明.md | 当前这份搭建和操作文档 |
|
| 103 |
+
|
| 104 |
+
上传到 Hugging Face Spaces 时,`app.py`、`requirements.txt`、`packages.txt` 必须放在 Space 根目录,不能放在子文件夹里。
|
| 105 |
+
|
| 106 |
+
## 5. 本地运行步骤
|
| 107 |
+
|
| 108 |
+
### 5.1 进入项目目录
|
| 109 |
+
|
| 110 |
+
在 PowerShell 中执行:
|
| 111 |
+
|
| 112 |
+
```powershell
|
| 113 |
+
cd "C:\Users\29697\Documents\Codex\2026-05-14\huggingface"
|
| 114 |
+
```
|
| 115 |
+
|
| 116 |
+
### 5.2 创建虚拟环境
|
| 117 |
+
|
| 118 |
+
```powershell
|
| 119 |
+
python -m venv .venv
|
| 120 |
+
```
|
| 121 |
+
|
| 122 |
+
### 5.3 安装依赖
|
| 123 |
+
|
| 124 |
+
如果可以激活虚拟环境,执行:
|
| 125 |
+
|
| 126 |
+
```powershell
|
| 127 |
+
.\.venv\Scripts\Activate.ps1
|
| 128 |
+
pip install -r requirements.txt
|
| 129 |
+
```
|
| 130 |
+
|
| 131 |
+
如果激活时报执行策略错误,直接用虚拟环境里的 Python 安装:
|
| 132 |
+
|
| 133 |
+
```powershell
|
| 134 |
+
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
|
| 135 |
+
```
|
| 136 |
+
|
| 137 |
+
### 5.4 设置 Hugging Face Token
|
| 138 |
+
|
| 139 |
+
到 Hugging Face 账号中创建 Access Token:
|
| 140 |
+
|
| 141 |
+
```text
|
| 142 |
+
https://huggingface.co/settings/tokens
|
| 143 |
+
```
|
| 144 |
+
|
| 145 |
+
然后在 PowerShell 中设置环境变量:
|
| 146 |
+
|
| 147 |
+
```powershell
|
| 148 |
+
$env:HF_TOKEN="hf_xxxxxxxxxxxxxxxxx"
|
| 149 |
+
```
|
| 150 |
+
|
| 151 |
+
注意:`HF_TOKEN` 不要写进代码,也不要发给别人。
|
| 152 |
+
|
| 153 |
+
### 5.5 启动网站
|
| 154 |
+
|
| 155 |
+
如果虚拟环境已激活:
|
| 156 |
+
|
| 157 |
+
```powershell
|
| 158 |
+
python app.py
|
| 159 |
+
```
|
| 160 |
+
|
| 161 |
+
如果没有激活虚拟环境:
|
| 162 |
+
|
| 163 |
+
```powershell
|
| 164 |
+
.\.venv\Scripts\python.exe app.py
|
| 165 |
+
```
|
| 166 |
+
|
| 167 |
+
终端出现下面内容说明启动成功:
|
| 168 |
+
|
| 169 |
+
```text
|
| 170 |
+
Running on local URL: http://127.0.0.1:7860
|
| 171 |
+
```
|
| 172 |
+
|
| 173 |
+
浏览器打开:
|
| 174 |
+
|
| 175 |
+
```text
|
| 176 |
+
http://127.0.0.1:7860
|
| 177 |
+
```
|
| 178 |
+
|
| 179 |
+
注意:PowerShell 停住不动是正常现象,因为它正在运行网站服务。如果要关闭网站,在 PowerShell 中按 `Ctrl + C`。
|
| 180 |
+
|
| 181 |
+
## 6. 网站使用步骤
|
| 182 |
+
|
| 183 |
+
打开网页后,按以下步骤操作:
|
| 184 |
+
|
| 185 |
+
1. 在 `Reference document` 上传参考文档。
|
| 186 |
+
2. 文档支持 `PDF`、`DOCX`、`TXT`。
|
| 187 |
+
3. 在 `Audio sample` 上传音频,或用麦克风录音。
|
| 188 |
+
4. 在 `Recognition profile` 选择识别配置。
|
| 189 |
+
5. 英文录音选择 `English optimized - Whisper small.en`。
|
| 190 |
+
6. 中文录音选择 `Chinese - Whisper multilingual small`。
|
| 191 |
+
7. 不确定语言选择 `Auto detect - Whisper multilingual small`。
|
| 192 |
+
8. 点击 `Transcribe and correct`。
|
| 193 |
+
9. 查看原始识别结果、纠错结果、修改说明和文档依据。
|
| 194 |
+
|
| 195 |
+
建议第一次测试时使用较短材料:
|
| 196 |
+
|
| 197 |
+
```text
|
| 198 |
+
英文文档:100 到 300 词
|
| 199 |
+
英文录音:20 到 60 秒
|
| 200 |
+
录音环境:安静、单人讲话
|
| 201 |
+
```
|
| 202 |
+
|
| 203 |
+
## 7. 英文测试样例
|
| 204 |
+
|
| 205 |
+
可以新建一个 `test.txt`,内容如下:
|
| 206 |
+
|
| 207 |
+
```text
|
| 208 |
+
This lecture explains backpropagation, vanishing gradients, convolutional neural networks, and attention mechanisms.
|
| 209 |
+
Backpropagation is a core algorithm for training neural networks.
|
| 210 |
+
Vanishing gradients can make deep neural networks difficult to train.
|
| 211 |
+
Attention mechanisms are widely used in natural language processing and speech recognition.
|
| 212 |
+
```
|
| 213 |
+
|
| 214 |
+
录音时可以读:
|
| 215 |
+
|
| 216 |
+
```text
|
| 217 |
+
This lecture explains backpropagation and vanishing gradients. It also introduces convolutional neural networks and attention mechanisms.
|
| 218 |
+
```
|
| 219 |
+
|
| 220 |
+
如果 Whisper 把某些专业词识别错,系统会尝试根据文档内容纠正。
|
| 221 |
+
|
| 222 |
+
## 8. 部署到 Hugging Face Spaces
|
| 223 |
+
|
| 224 |
+
### 8.1 创建 Space
|
| 225 |
+
|
| 226 |
+
1. 登录 Hugging Face。
|
| 227 |
+
2. 点击 `New Space`。
|
| 228 |
+
3. Space SDK 选择 `Gradio`。
|
| 229 |
+
4. Visibility 可以选择 `Public`。
|
| 230 |
+
5. 创建 Space。
|
| 231 |
+
|
| 232 |
+
### 8.2 上传文件
|
| 233 |
+
|
| 234 |
+
进入 Space 的 `Files` 页面,上传:
|
| 235 |
+
|
| 236 |
+
```text
|
| 237 |
+
app.py
|
| 238 |
+
requirements.txt
|
| 239 |
+
packages.txt
|
| 240 |
+
README.md
|
| 241 |
+
```
|
| 242 |
+
|
| 243 |
+
上传后点击:
|
| 244 |
+
|
| 245 |
+
```text
|
| 246 |
+
Commit changes to main
|
| 247 |
+
```
|
| 248 |
+
|
| 249 |
+
### 8.3 设置 Secret
|
| 250 |
+
|
| 251 |
+
进入:
|
| 252 |
+
|
| 253 |
+
```text
|
| 254 |
+
Settings -> Variables and secrets
|
| 255 |
+
```
|
| 256 |
+
|
| 257 |
+
添加 Secret:
|
| 258 |
+
|
| 259 |
+
```text
|
| 260 |
+
Name: HF_TOKEN
|
| 261 |
+
Value: hf_xxxxxxxxxxxxxxxxx
|
| 262 |
+
```
|
| 263 |
+
|
| 264 |
+
注意变量名必须是 `HF_TOKEN`,大小写要完全一致。
|
| 265 |
+
|
| 266 |
+
### 8.4 等待构建
|
| 267 |
+
|
| 268 |
+
回到 Space 页面查看状态:
|
| 269 |
+
|
| 270 |
+
```text
|
| 271 |
+
Building 正在安装依赖和启动应用
|
| 272 |
+
Running 应用运行成功
|
| 273 |
+
Build error 依赖安装失败
|
| 274 |
+
Runtime error 程序启动失败
|
| 275 |
+
```
|
| 276 |
+
|
| 277 |
+
如果出现错误,打开 `Logs`,查看最后几十行报错。
|
| 278 |
+
|
| 279 |
+
### 8.5 分享网站
|
| 280 |
+
|
| 281 |
+
如果 Space 是 Public,别人可以通过下面形式的链接访问:
|
| 282 |
+
|
| 283 |
+
```text
|
| 284 |
+
https://huggingface.co/spaces/用户名/Space名
|
| 285 |
+
```
|
| 286 |
+
|
| 287 |
+
或:
|
| 288 |
+
|
| 289 |
+
```text
|
| 290 |
+
https://用户名-Space名.hf.space
|
| 291 |
+
```
|
| 292 |
+
|
| 293 |
+
不要把本地地址发给别人:
|
| 294 |
+
|
| 295 |
+
```text
|
| 296 |
+
http://127.0.0.1:7860
|
| 297 |
+
```
|
| 298 |
+
|
| 299 |
+
这个地址只能在自己的电脑上打开。
|
| 300 |
+
|
| 301 |
+
## 9. 基础版 Hugging Face 的限制
|
| 302 |
+
|
| 303 |
+
免费 Hugging Face Spaces 通常适合作业展示和轻量 Demo,但不适合大规模高并发使用。常见限制包括:
|
| 304 |
+
|
| 305 |
+
```text
|
| 306 |
+
1. 第一次启动较慢。
|
| 307 |
+
2. 免费 Space 可能会休眠。
|
| 308 |
+
3. CPU 推理速度有限。
|
| 309 |
+
4. 大模型加载和音频识别可能需要等待。
|
| 310 |
+
5. 长音频处理较慢。
|
| 311 |
+
```
|
| 312 |
+
|
| 313 |
+
因此建议演示时:
|
| 314 |
+
|
| 315 |
+
```text
|
| 316 |
+
使用 20 秒到 2 分钟的短音频
|
| 317 |
+
使用 TXT 或文字版 PDF
|
| 318 |
+
避免扫描版 PDF
|
| 319 |
+
提前打开 Space,避免现场冷启动
|
| 320 |
+
准备本地运行截图或演示视频作为备用
|
| 321 |
+
```
|
| 322 |
+
|
| 323 |
+
## 10. 模型配置建议
|
| 324 |
+
|
| 325 |
+
当前页面已经把英文、中文、自动识别拆成了三个识别配置:
|
| 326 |
+
|
| 327 |
+
```text
|
| 328 |
+
English optimized - Whisper small.en: openai/whisper-small.en
|
| 329 |
+
Chinese - Whisper multilingual small: openai/whisper-small
|
| 330 |
+
Auto detect - Whisper multilingual small: openai/whisper-small
|
| 331 |
+
```
|
| 332 |
+
|
| 333 |
+
如果主要识别英文,可以在 Hugging Face Space 的 Variables 中添加:
|
| 334 |
+
|
| 335 |
+
```text
|
| 336 |
+
ASR_MODEL_EN=openai/whisper-small.en
|
| 337 |
+
```
|
| 338 |
+
|
| 339 |
+
如果要替换中文或自动识别使用的模型,可以添加:
|
| 340 |
+
|
| 341 |
+
```text
|
| 342 |
+
ASR_MODEL_ZH=openai/whisper-small
|
| 343 |
+
ASR_MODEL_AUTO=openai/whisper-small
|
| 344 |
+
```
|
| 345 |
+
|
| 346 |
+
旧的 `ASR_MODEL` 变量仍然可用,会作为中文和自动识别配置的默认多语言模型。
|
| 347 |
+
|
| 348 |
+
如果想要更高准确率,可以尝试:
|
| 349 |
+
|
| 350 |
+
```text
|
| 351 |
+
openai/whisper-medium
|
| 352 |
+
openai/whisper-large-v3
|
| 353 |
+
```
|
| 354 |
+
|
| 355 |
+
但这些模型在免费 CPU Space 上会更慢,甚至可能影响体验。
|
| 356 |
+
|
| 357 |
+
## 11. 常见问题
|
| 358 |
+
|
| 359 |
+
### 11.1 找不到 requirements.txt
|
| 360 |
+
|
| 361 |
+
报错:
|
| 362 |
+
|
| 363 |
+
```text
|
| 364 |
+
Could not open requirements file: requirements.txt
|
| 365 |
+
```
|
| 366 |
+
|
| 367 |
+
原因:当前目录没有 `requirements.txt`,或者文件没有上传到 Space 根目录。
|
| 368 |
+
|
| 369 |
+
解决:
|
| 370 |
+
|
| 371 |
+
```text
|
| 372 |
+
确认 app.py 和 requirements.txt 在同一个目录。
|
| 373 |
+
确认 requirements.txt 没有被命名成 requirements.txt.txt。
|
| 374 |
+
```
|
| 375 |
+
|
| 376 |
+
### 11.2 PowerShell 卡住不动
|
| 377 |
+
|
| 378 |
+
这是正常现象。`python app.py` 启动的是网页服务,PowerShell 会一直运行。
|
| 379 |
+
|
| 380 |
+
关闭服务:
|
| 381 |
+
|
| 382 |
+
```text
|
| 383 |
+
Ctrl + C
|
| 384 |
+
```
|
| 385 |
+
|
| 386 |
+
### 11.3 页面能打开,但点击按钮很慢
|
| 387 |
+
|
| 388 |
+
原因:
|
| 389 |
+
|
| 390 |
+
```text
|
| 391 |
+
第一次运行需要加载 Whisper 模型、embedding 模型和大模型接口。
|
| 392 |
+
免费 CPU Space 推理速度有限。
|
| 393 |
+
```
|
| 394 |
+
|
| 395 |
+
解决:
|
| 396 |
+
|
| 397 |
+
```text
|
| 398 |
+
先用短音频测试。
|
| 399 |
+
提前打开网页预热。
|
| 400 |
+
减少文档长度。
|
| 401 |
+
```
|
| 402 |
+
|
| 403 |
+
### 11.4 大模型没有纠错
|
| 404 |
+
|
| 405 |
+
可能原因:
|
| 406 |
+
|
| 407 |
+
```text
|
| 408 |
+
HF_TOKEN 没有设置。
|
| 409 |
+
原始识别已经足够正确。
|
| 410 |
+
文档里没有相关词。
|
| 411 |
+
检索到的文档片段不相关。
|
| 412 |
+
```
|
| 413 |
+
|
| 414 |
+
解决:
|
| 415 |
+
|
| 416 |
+
```text
|
| 417 |
+
检查 HF_TOKEN。
|
| 418 |
+
使用和录音内容更相关的文档。
|
| 419 |
+
把 Document passages to retrieve 调高到 5 或 6。
|
| 420 |
+
```
|
| 421 |
+
|
| 422 |
+
### 11.5 中国大陆同学打不开
|
| 423 |
+
|
| 424 |
+
Hugging Face 在中国大陆网络下访问可能不稳定。解决方式:
|
| 425 |
+
|
| 426 |
+
```text
|
| 427 |
+
准备演示视频。
|
| 428 |
+
准备本地运行截图。
|
| 429 |
+
让同学先测试链接。
|
| 430 |
+
必要时迁移到国内平台或云服务器。
|
| 431 |
+
```
|
| 432 |
+
|
| 433 |
+
## 12. 项目展示说明
|
| 434 |
+
|
| 435 |
+
展示时可以这样介绍:
|
| 436 |
+
|
| 437 |
+
```text
|
| 438 |
+
本项目不是简单调用语音识别模型,而是在普通 ASR 之后加入文档上下文检索和大模型纠错。
|
| 439 |
+
系统先用 Whisper 得到原始转写,再用语义向量检索与转写内容最相关的文档片段,
|
| 440 |
+
最后让大模型只根据这些文档证据纠正专业词、近音词和专有名词错误。
|
| 441 |
+
```
|
| 442 |
+
|
| 443 |
+
可以强调的创新点:
|
| 444 |
+
|
| 445 |
+
```text
|
| 446 |
+
1. 引入参考文档作为领域上下文。
|
| 447 |
+
2. 针对专业词和近音词错误进行纠正。
|
| 448 |
+
3. 输出修改依据,增强结果可信度。
|
| 449 |
+
4. 网页化部署,用户无需本地安装模型即可使用。
|
| 450 |
+
```
|
| 451 |
+
|
| 452 |
+
## 13. 后续可优化方向
|
| 453 |
+
|
| 454 |
+
如果需要继续完善,可以考虑:
|
| 455 |
+
|
| 456 |
+
```text
|
| 457 |
+
1. 增加高亮功能,标出被修改的词。
|
| 458 |
+
2. 增加音频分段,支持更长录音。
|
| 459 |
+
3. 增加 OCR,支持扫描版 PDF。
|
| 460 |
+
4. 增加用户自定义术语表。
|
| 461 |
+
5. 使用更强的 embedding 模型提高文档检索准确率。
|
| 462 |
+
6. 使用 GPU Space 提高推理速度。
|
| 463 |
+
7. 增加导出功能,把纠错结果导出为 TXT 或 DOCX。
|
| 464 |
+
```
|
| 465 |
+
|
| 466 |
+
## 14. 参考文档
|
| 467 |
+
|
| 468 |
+
- Hugging Face Gradio Spaces: https://huggingface.co/docs/hub/spaces-sdks-gradio
|
| 469 |
+
- Hugging Face Spaces Dependencies: https://huggingface.co/docs/hub/en/spaces-dependencies
|
| 470 |
+
- Hugging Face Space Secrets: https://huggingface.co/docs/huggingface_hub/v0.28.0/en/guides/manage-spaces
|
| 471 |
+
- Hugging Face Transformers Pipeline: https://huggingface.co/docs/transformers/v4.40.0/pipeline_tutorial
|
| 472 |
+
- Hugging Face Whisper Documentation: https://huggingface.co/docs/transformers/model_doc/whisper
|