Abid Ali Awan Codex commited on
Commit
aa2370d
·
1 Parent(s): 11407b6

Finalize hackathon submission documentation

Browse files

Co-authored-by: Codex <codex@openai.com>

Files changed (3) hide show
  1. FIELD_NOTES.md +0 -279
  2. README.md +18 -6
  3. SUBMISSION_TODO.md +0 -59
FIELD_NOTES.md DELETED
@@ -1,279 +0,0 @@
1
- # Field Notes
2
-
3
- ## Problem and scope
4
-
5
- People in Pakistan regularly receive messages that imitate banks, couriers,
6
- tax authorities, traffic police, utilities, and mobile operators. The difficult
7
- part is often not reading the message but deciding what to do next without
8
- opening an unsafe link, calling an untrusted number, or sharing an OTP.
9
-
10
- Pakistan Notice Helper is therefore a triage tool, not an authenticity checker.
11
- It accepts text or a screenshot and returns a risk label, a short explanation,
12
- visible red flags, and safe next steps. The wording deliberately avoids claims
13
- that a message is officially genuine or fraudulent.
14
-
15
- This scope fits the **Backyard AI** track: it addresses a specific local safety
16
- problem with a model small enough to run through `llama.cpp`.
17
-
18
- ## Why I built it this way
19
-
20
- I initially built with a larger Qwen model, then tested whether a much smaller
21
- model could preserve the safety behavior while reducing serving requirements.
22
- Qwen3.5 4B Q8 passed all high-risk scam cases and both screenshot cases in the
23
- ten-case evaluation, making it the practical production choice.
24
-
25
- The MTP variant was especially appealing because speculative multi-token
26
- prediction can improve generation speed when the draft tokens are accepted. In
27
- the deployed experiment, Qwen3.5 generated 440 draft tokens and accepted 222,
28
- for a 50.5% draft acceptance rate with `n_max=2`.
29
-
30
- I chose Modal because it provides a serverless GPU endpoint that scales to zero
31
- after inactivity and starts again on demand. That is a strong cost model for a
32
- demo with irregular traffic because compute is paid for when the deployment is
33
- actually used instead of keeping a GPU running continuously. Persistent
34
- Volumes avoid downloading the model and projector on every cold start.
35
-
36
- Control was another reason for using Modal. I build and pin the `llama.cpp`
37
- runtime, choose the model and quantization, manage the endpoint, and protect it
38
- with proxy authentication. Inputs are not sent to a third-party hosted LLM
39
- API. Modal still processes the request on its infrastructure, so this is
40
- self-managed inference rather than fully local inference, but the model-serving
41
- stack and API behavior remain under my control.
42
-
43
- ## Product decisions
44
-
45
- - Offer both English and Urdu instead of assuming every user is comfortable
46
- interpreting safety guidance in English.
47
- - Use simple, direct wording rather than legal or security terminology.
48
- - Support screenshots because many suspicious messages arrive through SMS,
49
- WhatsApp, and social media.
50
- - Treat every URL, phone number, and contact instruction in the submitted
51
- message as untrusted data.
52
- - Never invent an assessment when inference fails. The interface shows an
53
- explicit error instead of falling back to rules or cached samples.
54
- - Keep built-in examples fast with model-generated cached results, while any
55
- edited or new input always uses the live model.
56
- - Show a reply draft only for uncertain cases where clarification may be safe.
57
- Likely scams do not encourage further engagement.
58
-
59
- ## English and Urdu experience
60
-
61
- I added an English/Urdu switch to make the app more useful for people who can
62
- read a suspicious message but would understand the safety advice more clearly
63
- in Urdu. The selected language is remembered in the browser, so returning users
64
- do not have to switch it again.
65
-
66
- Urdu mode is more than a translated navigation layer. It changes the page to a
67
- right-to-left layout and translates headings, form labels, validation errors,
68
- status messages, risk labels, examples, disclaimers, and result-card controls.
69
- Live assessments also ask the model to return the explanation, red flags, safe
70
- next steps, and optional reply draft in clear Urdu script. English cached
71
- examples remain cached; selecting one in Urdu mode sends it through the live
72
- model so the result language matches the interface.
73
-
74
- The first Urdu layout exposed details that were easy to miss in English:
75
- headings needed different line heights, mixed Urdu and Latin model names could
76
- reorder unexpectedly, and mobile controls required more vertical space. I
77
- tested the interface at desktop and 390-pixel mobile widths, adjusted the RTL
78
- spacing, and rewrote literal translations into shorter, more natural Urdu.
79
-
80
- I also tested a bundled Nastaliq webfont. It made the interface feel less
81
- consistent and reduced readability at the sizes used by the app, so I removed
82
- it and returned to the earlier system Arabic font stack. This kept the improved
83
- Urdu copy and RTL layout without forcing a decorative typeface on every device.
84
-
85
- ## Small-model stack
86
-
87
- The primary model is `unsloth/Qwen3.5-4B-MTP-GGUF`, using the
88
- `Qwen3.5-4B-Q8_0.gguf` quantization and `mmproj-F16.gguf` vision projector.
89
- At 4B parameters it stays well below the hackathon's 32B limit.
90
-
91
- Modal supplies an L4 GPU and persistent model storage. A pinned,
92
- CUDA-enabled `llama.cpp` build runs `llama-server` and exposes an
93
- OpenAI-compatible endpoint. The OpenAI Python package is used only as an HTTP
94
- client with a custom `base_url`; no request is sent to OpenAI.
95
-
96
- Key measured results:
97
-
98
- | Measurement | Result |
99
- | --- | --- |
100
- | Initial evaluation strict passes | 9 of 10 |
101
- | Initial evaluation average score | 89.5/100 |
102
- | Final regression evaluation | 10 of 10 |
103
- | Final regression average score | 100/100 |
104
- | High-risk scam cases | All passed |
105
- | Screenshot cases | Both passed |
106
- | MTP draft acceptance | 222 of 440 tokens (50.5%) |
107
- | MTP draft limit | `n_max=2` |
108
-
109
- The final score comes from the same small ten-case suite after prompt and
110
- output-contract fixes. It is useful as a regression check, but it is not a
111
- real-world accuracy estimate.
112
-
113
- The full setup and measurements are documented in
114
- [the model experiment notes](docs/model_experiment_notes.md).
115
-
116
- ## Building with Codex
117
-
118
- Codex helped build most of the project, especially the custom frontend,
119
- Gradio Server integration, tests, trace pipeline, documentation, and repeated
120
- UI refinements. I was surprised by how quickly a fully custom HTML, CSS, and
121
- JavaScript interface could be connected to Gradio's queue and SSE protocol.
122
- This let me keep Hugging Face Spaces compatibility without settling for the
123
- default Gradio component layout.
124
-
125
- I used Codex as an engineering collaborator rather than only a code generator.
126
- It inspected the existing repository, implemented changes, ran tests, checked
127
- the live Space metadata, and helped keep the Modal and `llama.cpp`
128
- documentation consistent with the deployed system. The public Git history
129
- includes Codex co-author attribution for that work.
130
-
131
- ## Gradio and Space architecture
132
-
133
- `gradio.Server` provides queued API routes and Hugging Face Spaces hosting
134
- without imposing the default Gradio interface. A local HTML, CSS, and
135
- JavaScript frontend calls the Gradio POST and SSE protocol:
136
-
137
- ```text
138
- Browser
139
- -> custom mobile-first frontend
140
- -> English or Urdu interface and response-language request
141
- -> queued gradio.Server endpoint
142
- -> OpenAI-compatible client
143
- -> Modal proxy-authenticated web server
144
- -> CUDA llama.cpp
145
- -> Qwen3.5 4B Q8 MTP GGUF + vision projector
146
- ```
147
-
148
- The verified Server mode routes are `/gradio_api/call/{api_name}` and
149
- `/gradio_api/call/{api_name}/{event_id}`. The Space README pins the Gradio
150
- version and names `app.py` as its entry point.
151
-
152
- ## What failed and changed
153
-
154
- - Thinking mode initially consumed the 500-token output budget without
155
- returning final JSON. The production app disables thinking so the bounded
156
- completion budget is used for the final structured response.
157
- - A dense Roman Urdu screenshot reached the original completion limit. Image
158
- requests now receive a larger token budget.
159
- - The first Urdu interface used direct translations and generic spacing. It
160
- was revised with more natural wording, RTL-specific typography, responsive
161
- spacing, and cleaner handling of mixed Urdu and Latin text.
162
- - A bundled Nastaliq font looked worse in the product UI than the original
163
- system font stack. The font was removed while retaining the improved Urdu
164
- layout and copy.
165
- - One model response suggested an unverified official-looking domain. The
166
- system prompt now forbids invented URLs, phone numbers, organizations, and
167
- facts.
168
- - Modal CLI credentials returned HTTP 401 at the web endpoint. External calls
169
- require dedicated Modal Proxy Auth credentials.
170
- - Model availability cannot be hidden behind a rule-based fallback without
171
- making the product misleading. Failed requests therefore remain visible.
172
-
173
- ## Privacy-safe traces
174
-
175
- The optional public trace feature is enabled in the interface but can be
176
- disabled before each request. It records only allow-listed scalar metadata and
177
- deterministic summaries.
178
-
179
- Text is redacted and capped at 500 characters. Images are represented by a
180
- fixed description and are never stored. Public traces exclude raw screenshots,
181
- links, identifiers, generated explanations, reply text, errors, and
182
- credentials. Upload failures do not block the user response; pending JSONL
183
- shards remain available for retry.
184
-
185
- I designed the trace format this way because personal information can leak
186
- from more than the original input. A model explanation, reply draft, extracted
187
- phone number, URL, or exception can repeat private details even when the raw
188
- message is omitted. The trace serializer therefore does not publish the user's
189
- input, the model's free-form reasoning, the generated reply, or the final
190
- response text. It derives a limited set of redacted categories, booleans,
191
- counts, and fixed summaries instead.
192
-
193
- This minimizes what the application publishes, but it is not a claim that
194
- model inference itself is anonymous. Live text and images still travel to the
195
- private Modal endpoint for analysis. Users are warned not to submit sensitive
196
- personal data, and public trace sharing can be disabled before a request.
197
-
198
- The dataset and schema are published at
199
- [build-small-hackathon/pakistan-notice-helper-traces](https://huggingface.co/datasets/build-small-hackathon/pakistan-notice-helper-traces).
200
-
201
- ## Pakistan safety patterns
202
-
203
- The product focuses on patterns repeatedly present in Pakistani scam
204
- advisories and reported examples:
205
-
206
- - urgency, threats, suspension, arrest, disconnection, or parcel loss;
207
- - requests for OTPs, PINs, passwords, CVVs, CNIC details, or card data;
208
- - payment through message links, personal mobile numbers, wallets, or unusual
209
- channels;
210
- - impersonation of tax, telecom, banking, traffic, customs, and courier bodies;
211
- - prizes, refunds, jobs, rewards, or benefits requiring an advance fee.
212
-
213
- These signals are warnings, not proof. A familiar logo, sender name, or
214
- accurate personal detail does not establish authenticity.
215
-
216
- ## Remaining limitations
217
-
218
- - The live Space depends on the Modal endpoint and its cold-start behavior.
219
- - Screenshot quality and dense Roman Urdu text can affect visual recognition.
220
- - Urdu output quality still depends on the model, and mixed technical terms
221
- may occasionally need English wording for clarity.
222
- - The model can still miss subtle scams or flag legitimate notices.
223
- - The app does not query government, bank, courier, or telecom databases.
224
- - A result must always be confirmed through independently located official
225
- channels before payment or disclosure of personal information.
226
-
227
- ## Future direction
228
-
229
- The next major feature would be an agentic verification workflow. After
230
- understanding the submitted text or screenshot, the agent could search the web
231
- for current scam warnings, identify the organization being impersonated, and
232
- compare claims against independently discovered official sources. It could
233
- then present the evidence it found alongside the model's assessment.
234
-
235
- That workflow needs strict boundaries. It must never follow links or call
236
- numbers supplied by the suspicious message as if they were trusted. Search
237
- results would need source ranking, domain verification, citations, and a clear
238
- distinction between evidence and inference. The current project stops at
239
- triage because adding web access without those controls could make a safety
240
- tool less safe.
241
-
242
- ## Hackathon submission status
243
-
244
- The project meets the core technical constraints: a model at or below 32B,
245
- Gradio hosting as a Space under the hackathon organization, and a public app.
246
- It is intended for Backyard AI because the problem is specific and local, the
247
- 4B model is an honest fit for the task, and the interface is designed for the
248
- people receiving these messages. The final submission form is still pending.
249
-
250
- The strongest additional award targets are Modal Awards, Tiny Titan, Off-Brand
251
- Award, and the OpenAI Track. The implemented bonus-quest evidence clearly
252
- covers Llama Champion, Off-Brand, and Field Notes. The public trace dataset is
253
- a Sharing is Caring candidate, but the official wording says “agent trace”
254
- while this project publishes deterministic request traces rather than an
255
- autonomous-agent trajectory; that eligibility should be confirmed with the
256
- organizers. The project does not claim Off the Grid, Well-Tuned, the Nemotron
257
- Quest, or Best Agent.
258
-
259
- Before the June 15, 2026 deadline, the remaining submission work is to record
260
- target-user use and any resulting change, publish the short demo video and
261
- social post, verify that GitHub mirrors the final Space source, and submit the
262
- three required links. The operational checklist is in `SUBMISSION_TODO.md`.
263
-
264
- ## References
265
-
266
- - [Build Small Hackathon](https://huggingface.co/build-small-hackathon)
267
- - [Gradio Server mode](https://www.gradio.app/main/guides/server-mode)
268
- - [Gradio curl and SSE protocol](https://www.gradio.app/main/guides/querying-gradio-apps-with-curl)
269
- - [Hugging Face Spaces configuration](https://huggingface.co/docs/hub/spaces-config-reference)
270
- - [Qwen3.5 4B MTP GGUF](https://huggingface.co/unsloth/Qwen3.5-4B-MTP-GGUF)
271
- - [llama.cpp server](https://github.com/ggml-org/llama.cpp/tree/master/tools/server)
272
- - [Modal web servers](https://modal.com/docs/guide/webhooks)
273
- - [FBR fraudulent SMS warning](https://www.fbr.gov.pk/beware-fradulant-sms/152600)
274
- - [PTA Complaint Management System](https://complaint.pta.gov.pk/RegisterComplaint.aspx)
275
- - [State Bank of Pakistan](https://www.sbp.org.pk/)
276
- - [FIA complaint portal](https://complaint.fia.gov.pk/)
277
-
278
- Hackathon requirements, public links, and deployment claims were reviewed on
279
- June 8, 2026.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
README.md CHANGED
@@ -51,24 +51,36 @@ mixed-language inputs are also supported.
51
  | Live app | [Hugging Face Space](https://huggingface.co/spaces/build-small-hackathon/pakistan-notice-helper) |
52
  | Source | [GitHub repository](https://github.com/kingabzpro/pakistan-notice-helper) |
53
  | Open traces | [Privacy-safe trace dataset](https://huggingface.co/datasets/build-small-hackathon/pakistan-notice-helper-traces) |
54
- | Build report | [`FIELD_NOTES.md`](FIELD_NOTES.md) |
 
 
55
 
56
  ## Hackathon Fit
57
 
58
- This is an intended **Backyard AI** submission for the
59
  [Build Small. Play Big. Hackathon](https://huggingface.co/build-small-hackathon).
60
  It applies a small model to a practical regional safety problem.
61
 
62
  | Area | Project evidence |
63
  | --- | --- |
64
- | Core constraints | Public Gradio Space using Qwen3.5-4B, below the 7B limit |
65
- | Backyard AI | Notice and scam triage designed for Pakistani users |
66
  | Modal | Qwen3.5-4B is hosted on a Modal L4 GPU endpoint |
67
  | Tiny Titan | The 4B Q8 model passed the final 10-case internal regression suite |
68
  | Llama Champion | Qwen3.5-4B runs through a CUDA-enabled `llama.cpp` server using its OpenAI-compatible API |
69
  | Off-Brand | Custom mobile-first HTML, CSS, and JavaScript interface served through `gradio.Server` instead of the default Gradio UI |
70
- | Bonus quests | Runtime notes for Local Llama, privacy-safe traces, and published field notes |
71
- | Pending | Track form, user test, demo video, public announcement, and final repository sync |
 
 
 
 
 
 
 
 
 
 
72
 
73
  ## Architecture
74
 
 
51
  | Live app | [Hugging Face Space](https://huggingface.co/spaces/build-small-hackathon/pakistan-notice-helper) |
52
  | Source | [GitHub repository](https://github.com/kingabzpro/pakistan-notice-helper) |
53
  | Open traces | [Privacy-safe trace dataset](https://huggingface.co/datasets/build-small-hackathon/pakistan-notice-helper-traces) |
54
+ | Demo video | [Watch on YouTube](https://www.youtube.com/watch?v=EgYA3jrZTm0) |
55
+ | Build story | [Building Pakistan Notice Helper](https://huggingface.co/blog/build-small-hackathon/building-pakistan-notice-helper) |
56
+ | Social post | [Project announcement on LinkedIn](https://www.linkedin.com/posts/1abidaliawan_after-3-straight-days-of-building-testing-ugcPost-7469716521095794688-TlqZ/) |
57
 
58
  ## Hackathon Fit
59
 
60
+ This is a **Backyard AI** submission for the
61
  [Build Small. Play Big. Hackathon](https://huggingface.co/build-small-hackathon).
62
  It applies a small model to a practical regional safety problem.
63
 
64
  | Area | Project evidence |
65
  | --- | --- |
66
+ | Core constraints | Public Gradio Space using Qwen3.5-4B, below the 32B limit |
67
+ | Backyard AI | A real user tested the phone workflow and plans to use it for future suspicious messages |
68
  | Modal | Qwen3.5-4B is hosted on a Modal L4 GPU endpoint |
69
  | Tiny Titan | The 4B Q8 model passed the final 10-case internal regression suite |
70
  | Llama Champion | Qwen3.5-4B runs through a CUDA-enabled `llama.cpp` server using its OpenAI-compatible API |
71
  | Off-Brand | Custom mobile-first HTML, CSS, and JavaScript interface served through `gradio.Server` instead of the default Gradio UI |
72
+ | Sharing is Caring | Privacy-safe workflow traces are published as a Hugging Face dataset |
73
+ | Field Notes | Published a blog in the Build Small Hackathon organization sharing my opinions, development experience, model experiments, and lessons learned |
74
+
75
+ ## Demo and User Feedback
76
+
77
+ The [demo video](https://www.youtube.com/watch?v=EgYA3jrZTm0) includes a real
78
+ user using the app on a phone, taking screenshots of messages, and checking
79
+ them with Pakistan Notice Helper.
80
+
81
+ The user especially liked the app's speed and the accuracy of its responses.
82
+ They said they plan to use it in the future to check different scam or
83
+ suspicious messages.
84
 
85
  ## Architecture
86
 
SUBMISSION_TODO.md DELETED
@@ -1,59 +0,0 @@
1
- # Build Small Submission TODO
2
-
3
- Deadline: **June 15, 2026**
4
-
5
- ## Required submission package
6
-
7
- - [x] Host the Gradio app as a Space under the hackathon organization.
8
- - [ ] Record and publish a short demo video.
9
- - [ ] Publish a social-media post.
10
- - [ ] Submit the Space, demo video, and social-post links by June 15, 2026.
11
- - [ ] Save the submission confirmation.
12
-
13
- ## Backyard AI evidence
14
-
15
- - [ ] Ask at least one target user to try the Space.
16
- - [ ] Save a short, anonymized quote or note about their experience.
17
- - [ ] Add the user feedback and any resulting improvement to `FIELD_NOTES.md`.
18
-
19
- ## Final documentation and verification
20
-
21
- - [ ] Add the demo and social-post links to `README.md`.
22
- - [ ] Confirm the public GitHub repository contains the final Space source and
23
- Codex-attributed commits.
24
- - [ ] Test all public links and confirm the Space is running.
25
- - [ ] Test one English text, one Urdu text, and one screenshot through live
26
- inference.
27
- - [ ] Confirm the Modal endpoint is warm or starts successfully before
28
- recording and submitting.
29
-
30
- Current audit: the GitHub and Hugging Face repositories have different HEAD
31
- commits, so synchronization is still required.
32
-
33
- The official page requires a Space link, short demo video, and social post.
34
- Backyard AI also judges whether the intended person actually used the app.
35
- Social-media `@` mentions are not listed as a core requirement.
36
-
37
- ## Award and bonus-quest claims
38
-
39
- - [x] Backyard AI selected as the intended main track
40
- - [x] Modal-powered app
41
- - [x] Tiny Titan target: production model is 4B
42
- - [x] Off-Brand: custom `gradio.Server` frontend
43
- - [x] Llama Champion: inference through `llama.cpp`
44
- - [ ] Sharing is Caring: public privacy-safe request traces exist, but confirm
45
- that deterministic request traces qualify as an “agent trace”
46
- - [x] Field Notes: public build report
47
- - [x] OpenAI Track target: public repo with Codex-attributed commits
48
- - [ ] Best Demo: complete only after the video and social post are published
49
-
50
- Not claimed: Off the Grid, Well-Tuned, NVIDIA Nemotron Quest, Best Agent.
51
-
52
- ## Links
53
-
54
- - Space: https://huggingface.co/spaces/build-small-hackathon/pakistan-notice-helper
55
- - GitHub: https://github.com/kingabzpro/pakistan-notice-helper
56
- - Traces: https://huggingface.co/datasets/build-small-hackathon/pakistan-notice-helper-traces
57
- - Demo video: TODO
58
- - Social post: TODO
59
- - Submission confirmation: TODO