mzx commited on
Commit
f8aedd5
·
verified ·
1 Parent(s): 70fb85f

Harden public API and MCP inputs

Browse files
API_CONTRACT.md CHANGED
@@ -31,7 +31,14 @@ equal timestamps cannot cause duplicates or skips across pages.
31
  Canonical instrument IDs accept only alphanumeric, colon, period, underscore
32
  and hyphen characters and are capped at 240 characters.
33
 
34
- Full-text document search tokenizes at most 12 terms from a 120-character
35
- query, uses parameterized SQLite FTS, supports ticker/type/date constraints and
36
  returns at most 50 ranked results per page. Snippets are capped at 600
37
  characters; full markdown bodies are never returned by search.
 
 
 
 
 
 
 
 
31
  Canonical instrument IDs accept only alphanumeric, colon, period, underscore
32
  and hyphen characters and are capped at 240 characters.
33
 
34
+ Full-text document search rejects more than 12 terms or 120 characters, uses
35
+ parameterized SQLite FTS, supports ticker/type/date constraints and
36
  returns at most 50 ranked results per page. Snippets are capped at 600
37
  characters; full markdown bodies are never returned by search.
38
+
39
+ Public requests are protected by bounded global request and concurrency
40
+ budgets. Overload returns `429 rate_limited` or `503 service_busy` with a
41
+ one-second retry hint; URLs over 2,048 characters return `414
42
+ request_too_large`. Non-read API methods return `405 read_only_surface`.
43
+ File/upload/reset, login and monitoring routes are disabled. The complete
44
+ boundary is documented in [`SECURITY.md`](SECURITY.md).
MCP_TOOLS.md CHANGED
@@ -31,3 +31,8 @@ No MCP tool accepts arbitrary SQL, URLs or storage paths. The service cannot
31
  write data, mutate the publishing pipeline, list storage or expose credentials
32
  and worker state. Automated extraction can contain errors; verify research
33
  against the cited issuer announcement. This is not financial advice.
 
 
 
 
 
 
31
  write data, mutate the publishing pipeline, list storage or expose credentials
32
  and worker state. Automated extraction can contain errors; verify research
33
  against the cited issuer announcement. This is not financial advice.
34
+
35
+ Oversized values are rejected with clear tool errors instead of being silently
36
+ truncated. Search accepts at most 120 characters and 12 terms. Citation hosts
37
+ and public markdown artifact keys are fail-closed allowlists; service-level
38
+ request and concurrency budgets are described in [`SECURITY.md`](SECURITY.md).
README.md CHANGED
@@ -30,6 +30,8 @@ The typed REST contract is available at `/openapi.json`; pagination, cursor and
30
  structured-error rules are documented in [`API_CONTRACT.md`](API_CONTRACT.md).
31
  The exact MCP tool allowlist, response caps and provenance rules are documented
32
  in [`MCP_TOOLS.md`](MCP_TOOLS.md).
 
 
33
 
34
  The Space will load one validated commit of
35
  [`mzx/dilutionrisk-data`](https://huggingface.co/datasets/mzx/dilutionrisk-data)
@@ -47,10 +49,10 @@ The target update cadence is daily. Every data response will include the exact
47
  Dataset revision. Full PDFs/markdown are delivered as public artifact URLs and
48
  are not embedded into MCP responses.
49
 
50
- No arbitrary SQL, URL fetching, storage listing, pipeline mutation, credentials
51
- or operational worker state is exposed. Automated extraction may contain
52
- errors; verify against the linked issuer announcement. Nothing here is
53
- financial advice.
54
 
55
  The Space code is MIT-licensed. Dataset/source content has separate terms in
56
  the Dataset card and is not covered by the software licence.
 
30
  structured-error rules are documented in [`API_CONTRACT.md`](API_CONTRACT.md).
31
  The exact MCP tool allowlist, response caps and provenance rules are documented
32
  in [`MCP_TOOLS.md`](MCP_TOOLS.md).
33
+ Input allowlists, request/concurrency budgets and blocked mutation/file routes
34
+ are documented in [`SECURITY.md`](SECURITY.md).
35
 
36
  The Space will load one validated commit of
37
  [`mzx/dilutionrisk-data`](https://huggingface.co/datasets/mzx/dilutionrisk-data)
 
49
  Dataset revision. Full PDFs/markdown are delivered as public artifact URLs and
50
  are not embedded into MCP responses.
51
 
52
+ No arbitrary SQL, caller-supplied URL/path fetching, storage listing, file
53
+ upload, pipeline mutation, credentials or operational worker state is exposed.
54
+ Automated extraction may contain errors; verify against the linked issuer
55
+ announcement. Nothing here is financial advice.
56
 
57
  The Space code is MIT-licensed. Dataset/source content has separate terms in
58
  the Dataset card and is not covered by the software licence.
SECURITY.md ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Public-service security boundary
2
+
3
+ The Space is a public, read-only research service. Its accepted inputs are
4
+ tickers, canonical document identities, bounded search/filter values and opaque
5
+ cursors issued by the same Dataset revision. It does not accept SQL, URLs,
6
+ filesystem paths, storage keys or write/delete commands.
7
+
8
+ Controls enforced in the running service:
9
+
10
+ - ASX is the only exchange and all REST sorts/filters are typed allowlists;
11
+ - search text is limited to 120 characters and 12 terms, document type to 80
12
+ characters, canonical identities to 240 and cursors to 1,024;
13
+ - MCP result limits are clamped and REST limits are schema-bounded;
14
+ - citation URLs must be HTTPS on an approved ASX hostname;
15
+ - markdown reads are constructed only from canonical public HF keys matching
16
+ `artifacts/v1/asx/<TICKER>/...<FILE>.md`;
17
+ - legacy GCS paths, caller-supplied URLs and arbitrary local paths are never
18
+ fetched;
19
+ - Gradio upload/file/reset, login and monitoring routes are blocked;
20
+ - API write methods return a structured `405 read_only_surface` response;
21
+ - one bounded-memory global token bucket permits a burst of 240 requests and
22
+ refills at four requests per second;
23
+ - at most 16 public requests enter the query layer concurrently, while each
24
+ Gradio query function also has a four-call concurrency group; and
25
+ - overload returns structured `429 rate_limited` or `503 service_busy` errors
26
+ with `Retry-After: 1`.
27
+
28
+ The service exposes exact public build/data revisions and dependency versions,
29
+ but never serializes environment variables, tokens, credentials or worker
30
+ state. Public artifact and source content should still be treated as untrusted
31
+ text by downstream renderers.
app.py CHANGED
@@ -7,6 +7,7 @@ import os
7
  import platform
8
  import re
9
  import threading
 
10
  from datetime import date
11
  import urllib.parse
12
  import urllib.request
@@ -25,6 +26,7 @@ from company_query import filter_fingerprint,search_companies as query_companies
25
  from document_query import fingerprint as document_fingerprint,get_document,search_documents as query_documents
26
  from instrument_query import dilution_instruments
27
  from search_query import SearchInputError,document_citation,search as search_fts
 
28
  from cache import RevisionCache
29
 
30
 
@@ -35,6 +37,8 @@ CACHE = RevisionCache(
35
  "https://huggingface.co/buckets/mzx/dilutionrisk-public/resolve/manifests/v1/latest.json",
36
  RELEASE["dataset_id"],
37
  )
 
 
38
 
39
  if os.environ.get("SPACE_ID") and os.environ.get("DILUTIONRISK_SKIP_CACHE_BOOTSTRAP") != "1":
40
  threading.Thread(target=CACHE.bootstrap, name="dataset-cache-bootstrap", daemon=True).start()
@@ -114,6 +118,21 @@ def error_response(status:int,code:str,message:str,details:list[dict]|None=None)
114
  revision=CACHE.status()["dataset_revision"] or RELEASE["dataset_revision"]
115
  return JSONResponse(status_code=status,content=ErrorResponse(error=ErrorBody(code=code,message=message,details=details or [],dataset_revision=revision)).model_dump())
116
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
117
  @api.exception_handler(RequestValidationError)
118
  async def validation_error(_request:Request,error:RequestValidationError):
119
  details=[{"field":".".join(map(str,item["loc"][1:])),"type":item["type"],"message":item["msg"]} for item in error.errors()]
@@ -194,9 +213,9 @@ def search_companies(query:str="",min_risk:float|None=None,max_runway:float|None
194
  limit: Maximum companies to return; clamped to 1 through 25.
195
  cursor: Opaque next_cursor from the same query; empty for the first page.
196
  """
197
- cache,context=tool_context();limit=max(1,min(int(limit),25));q=query.strip()[:120] or None;minimum=max(0,min_risk) if min_risk is not None else None;runway=max(0,max_runway) if max_runway is not None else None;sort="risk" if min_risk is not None else "ticker";direction="desc" if min_risk is not None else "asc";filters={"q":q,"min_risk":minimum,"max_runway":runway,"min_expiry_3m":None,"min_expiry_6m":None,"min_expiry_12m":None,"needs_review":needs_review,"sort":sort,"direction":direction};fingerprint=filter_fingerprint(filters);offset=0
198
  if cursor:
199
- try:values=decode_cursor(cursor[:1025],cache["dataset_revision"],"mcp_companies")
200
  except CursorError as error:raise ValueError(str(error)) from error
201
  if len(values)!=2 or values[1]!=fingerprint or not isinstance(values[0],int) or not 0<=values[0]<=10000:raise ValueError("cursor does not match this company query")
202
  offset=values[0]
@@ -208,7 +227,7 @@ def get_company_snapshot(ticker:str)->dict:
208
  Args:
209
  ticker: Exact 2-6 character ASX ticker, for example BHP.
210
  """
211
- cache,context=tool_context();ticker=ticker.strip().upper()
212
  if not re.fullmatch(r"[A-Z0-9]{2,6}",ticker):raise ValueError("ticker must be 2-6 ASX characters")
213
  items,_,_=query_companies(cache["cache_dir"],q=ticker,sort="ticker",direction="asc",limit=10,offset=0);item=next((x for x in items if x["ticker"]==ticker),None);return {**context,"found":item is not None,"company":item}
214
 
@@ -222,10 +241,10 @@ def search_documents(query:str,ticker:str="",doc_type:str="",limit:int=10,cursor
222
  limit: Maximum documents to return; clamped to 1 through 25.
223
  cursor: Opaque next_cursor from the same query; empty for the first page.
224
  """
225
- cache,context=tool_context();limit=max(1,min(int(limit),25));ticker=ticker.strip().upper() or None;doc_type=doc_type.strip()[:80] or None;bounded_query=query[:120];fingerprint=document_fingerprint({"q":bounded_query,"ticker":ticker,"doc_type":doc_type});offset=0
226
  if ticker and not re.fullmatch(r"[A-Z0-9]{2,6}",ticker):raise ValueError("ticker must be 2-6 ASX characters")
227
  if cursor:
228
- try:values=decode_cursor(cursor[:1025],cache["dataset_revision"],"mcp_document_search")
229
  except CursorError as error:raise ValueError(str(error)) from error
230
  if len(values)!=2 or values[1]!=fingerprint or not isinstance(values[0],int) or not 0<=values[0]<=1000:raise ValueError("cursor does not match this document search")
231
  offset=values[0]
@@ -240,7 +259,7 @@ def get_document_metadata(document_id:str)->dict:
240
  Args:
241
  document_id: Canonical document identity such as ASX:CLA:03115626.
242
  """
243
- cache,context=tool_context();document_id=document_id.strip()
244
  if not re.fullmatch(r"ASX:[A-Z0-9]{2,6}:[A-Za-z0-9._-]+",document_id):raise ValueError("invalid canonical document ID")
245
  item=get_document(cache["cache_dir"],document_id);citation=document_citation(cache["cache_dir"],document_id);return {**context,"found":item is not None,"document":item,"citation":citation["source_link"] if citation else None}
246
 
@@ -251,13 +270,13 @@ def get_document_markdown(document_id:str,max_chars:int=6000)->dict:
251
  document_id: Canonical document identity such as ASX:CLA:03115626.
252
  max_chars: Maximum excerpt length; clamped to 500 through 12,000 characters.
253
  """
254
- cache,context=tool_context();document_id=document_id.strip();max_chars=max(500,min(int(max_chars),12000))
255
  if not re.fullmatch(r"ASX:[A-Z0-9]{2,6}:[A-Za-z0-9._-]+",document_id):raise ValueError("invalid canonical document ID")
256
  item=get_document(cache["cache_dir"],document_id);citation=document_citation(cache["cache_dir"],document_id);source_link=citation["source_link"] if citation else None
257
  if not item:return {**context,"found":False,"available":False,"markdown":None,"citation":None}
258
  key=item.get("markdown_artifact_key")
259
  if not key:return {**context,"found":True,"available":False,"reason":"HF markdown artifact not yet published","markdown":None,"citation":source_link}
260
- if not key.startswith("artifacts/v1/asx/") or ".." in key.split("/"):raise ValueError("invalid published artifact key")
261
  url="https://huggingface.co/buckets/mzx/dilutionrisk-public/resolve/"+urllib.parse.quote(key,safe="/=-._")
262
  with urllib.request.urlopen(url,timeout=20) as response:body=response.read(max_chars*4+1).decode("utf-8","replace")
263
  return {**context,"found":True,"available":True,"markdown":body[:max_chars],"truncated":len(body)>max_chars,"citation":source_link,"artifact_url":url}
@@ -269,7 +288,7 @@ def get_dilution_instruments(ticker:str,limit:int=50)->dict:
269
  ticker: Exact 2-6 character ASX ticker.
270
  limit: Maximum instruments to return; clamped to 1 through 100.
271
  """
272
- cache,context=tool_context();ticker=ticker.strip().upper()
273
  if not re.fullmatch(r"[A-Z0-9]{2,6}",ticker):raise ValueError("ticker must be 2-6 ASX characters")
274
  items=dilution_instruments(cache["cache_dir"],ticker,max(1,min(int(limit),100)));return {**context,"ticker":ticker,"items":items,"returned":len(items)}
275
 
 
7
  import platform
8
  import re
9
  import threading
10
+ import asyncio
11
  from datetime import date
12
  import urllib.parse
13
  import urllib.request
 
26
  from document_query import fingerprint as document_fingerprint,get_document,search_documents as query_documents
27
  from instrument_query import dilution_instruments
28
  from search_query import SearchInputError,document_citation,search as search_fts
29
+ from security import TokenBucket,bounded_text,finite_non_negative,valid_markdown_artifact_key
30
  from cache import RevisionCache
31
 
32
 
 
37
  "https://huggingface.co/buckets/mzx/dilutionrisk-public/resolve/manifests/v1/latest.json",
38
  RELEASE["dataset_id"],
39
  )
40
+ REQUEST_BUDGET = TokenBucket(capacity=240, refill_per_second=4)
41
+ QUERY_SLOTS = asyncio.Semaphore(16)
42
 
43
  if os.environ.get("SPACE_ID") and os.environ.get("DILUTIONRISK_SKIP_CACHE_BOOTSTRAP") != "1":
44
  threading.Thread(target=CACHE.bootstrap, name="dataset-cache-bootstrap", daemon=True).start()
 
118
  revision=CACHE.status()["dataset_revision"] or RELEASE["dataset_revision"]
119
  return JSONResponse(status_code=status,content=ErrorResponse(error=ErrorBody(code=code,message=message,details=details or [],dataset_revision=revision)).model_dump())
120
 
121
+ @api.middleware("http")
122
+ async def public_abuse_guard(request:Request,call_next):
123
+ path=request.url.path;protected=path.startswith(("/api/v1/","/gradio_api/"));blocked=path.startswith(("/gradio_api/upload","/gradio_api/file","/file=","/file/","/login","/logout","/monitoring")) or path.rstrip("/").endswith("/reset")
124
+ if blocked:return error_response(403,"read_only_surface","File, login, monitoring and mutation routes are disabled")
125
+ if path.startswith("/api/v1/") and request.method not in {"GET","HEAD","OPTIONS"}:return error_response(405,"read_only_surface","This API supports read-only methods only")
126
+ if protected and len(str(request.url))>2048:return error_response(414,"request_too_large","Request URL must be at most 2048 characters")
127
+ if protected and not REQUEST_BUDGET.consume():
128
+ response=error_response(429,"rate_limited","Public request budget exhausted; retry shortly");response.headers["Retry-After"]="1";return response
129
+ if not protected:return await call_next(request)
130
+ try:await asyncio.wait_for(QUERY_SLOTS.acquire(),timeout=0.25)
131
+ except TimeoutError:
132
+ response=error_response(503,"service_busy","Public query concurrency limit reached; retry shortly");response.headers["Retry-After"]="1";return response
133
+ try:return await call_next(request)
134
+ finally:QUERY_SLOTS.release()
135
+
136
  @api.exception_handler(RequestValidationError)
137
  async def validation_error(_request:Request,error:RequestValidationError):
138
  details=[{"field":".".join(map(str,item["loc"][1:])),"type":item["type"],"message":item["msg"]} for item in error.errors()]
 
213
  limit: Maximum companies to return; clamped to 1 through 25.
214
  cursor: Opaque next_cursor from the same query; empty for the first page.
215
  """
216
+ cache,context=tool_context();limit=max(1,min(int(limit),25));q=bounded_text(query,"query",120) or None;minimum=finite_non_negative(min_risk,"min_risk");runway=finite_non_negative(max_runway,"max_runway");cursor=bounded_text(cursor,"cursor",1024);sort="risk" if min_risk is not None else "ticker";direction="desc" if min_risk is not None else "asc";filters={"q":q,"min_risk":minimum,"max_runway":runway,"min_expiry_3m":None,"min_expiry_6m":None,"min_expiry_12m":None,"needs_review":needs_review,"sort":sort,"direction":direction};fingerprint=filter_fingerprint(filters);offset=0
217
  if cursor:
218
+ try:values=decode_cursor(cursor,cache["dataset_revision"],"mcp_companies")
219
  except CursorError as error:raise ValueError(str(error)) from error
220
  if len(values)!=2 or values[1]!=fingerprint or not isinstance(values[0],int) or not 0<=values[0]<=10000:raise ValueError("cursor does not match this company query")
221
  offset=values[0]
 
227
  Args:
228
  ticker: Exact 2-6 character ASX ticker, for example BHP.
229
  """
230
+ cache,context=tool_context();ticker=bounded_text(ticker,"ticker",6,required=True).upper()
231
  if not re.fullmatch(r"[A-Z0-9]{2,6}",ticker):raise ValueError("ticker must be 2-6 ASX characters")
232
  items,_,_=query_companies(cache["cache_dir"],q=ticker,sort="ticker",direction="asc",limit=10,offset=0);item=next((x for x in items if x["ticker"]==ticker),None);return {**context,"found":item is not None,"company":item}
233
 
 
241
  limit: Maximum documents to return; clamped to 1 through 25.
242
  cursor: Opaque next_cursor from the same query; empty for the first page.
243
  """
244
+ cache,context=tool_context();limit=max(1,min(int(limit),25));ticker=bounded_text(ticker,"ticker",6).upper() or None;doc_type=bounded_text(doc_type,"doc_type",80) or None;bounded_query=bounded_text(query,"query",120,required=True);cursor=bounded_text(cursor,"cursor",1024);fingerprint=document_fingerprint({"q":bounded_query,"ticker":ticker,"doc_type":doc_type});offset=0
245
  if ticker and not re.fullmatch(r"[A-Z0-9]{2,6}",ticker):raise ValueError("ticker must be 2-6 ASX characters")
246
  if cursor:
247
+ try:values=decode_cursor(cursor,cache["dataset_revision"],"mcp_document_search")
248
  except CursorError as error:raise ValueError(str(error)) from error
249
  if len(values)!=2 or values[1]!=fingerprint or not isinstance(values[0],int) or not 0<=values[0]<=1000:raise ValueError("cursor does not match this document search")
250
  offset=values[0]
 
259
  Args:
260
  document_id: Canonical document identity such as ASX:CLA:03115626.
261
  """
262
+ cache,context=tool_context();document_id=bounded_text(document_id,"document_id",240,required=True)
263
  if not re.fullmatch(r"ASX:[A-Z0-9]{2,6}:[A-Za-z0-9._-]+",document_id):raise ValueError("invalid canonical document ID")
264
  item=get_document(cache["cache_dir"],document_id);citation=document_citation(cache["cache_dir"],document_id);return {**context,"found":item is not None,"document":item,"citation":citation["source_link"] if citation else None}
265
 
 
270
  document_id: Canonical document identity such as ASX:CLA:03115626.
271
  max_chars: Maximum excerpt length; clamped to 500 through 12,000 characters.
272
  """
273
+ cache,context=tool_context();document_id=bounded_text(document_id,"document_id",240,required=True);max_chars=max(500,min(int(max_chars),12000))
274
  if not re.fullmatch(r"ASX:[A-Z0-9]{2,6}:[A-Za-z0-9._-]+",document_id):raise ValueError("invalid canonical document ID")
275
  item=get_document(cache["cache_dir"],document_id);citation=document_citation(cache["cache_dir"],document_id);source_link=citation["source_link"] if citation else None
276
  if not item:return {**context,"found":False,"available":False,"markdown":None,"citation":None}
277
  key=item.get("markdown_artifact_key")
278
  if not key:return {**context,"found":True,"available":False,"reason":"HF markdown artifact not yet published","markdown":None,"citation":source_link}
279
+ if not valid_markdown_artifact_key(key):raise ValueError("invalid published HF markdown artifact key")
280
  url="https://huggingface.co/buckets/mzx/dilutionrisk-public/resolve/"+urllib.parse.quote(key,safe="/=-._")
281
  with urllib.request.urlopen(url,timeout=20) as response:body=response.read(max_chars*4+1).decode("utf-8","replace")
282
  return {**context,"found":True,"available":True,"markdown":body[:max_chars],"truncated":len(body)>max_chars,"citation":source_link,"artifact_url":url}
 
288
  ticker: Exact 2-6 character ASX ticker.
289
  limit: Maximum instruments to return; clamped to 1 through 100.
290
  """
291
+ cache,context=tool_context();ticker=bounded_text(ticker,"ticker",6,required=True).upper()
292
  if not re.fullmatch(r"[A-Z0-9]{2,6}",ticker):raise ValueError("ticker must be 2-6 ASX characters")
293
  items=dilution_instruments(cache["cache_dir"],ticker,max(1,min(int(limit),100)));return {**context,"ticker":ticker,"items":items,"returned":len(items)}
294
 
release.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "service_version": "0.8.1",
3
  "schema_version": 1,
4
  "dataset_id": "mzx/dilutionrisk-data",
5
  "dataset_revision": "d2b8896cabf2489d1b2f82694376f90a3b38049c",
 
1
  {
2
+ "service_version": "0.9.0",
3
  "schema_version": 1,
4
  "dataset_id": "mzx/dilutionrisk-data",
5
  "dataset_revision": "d2b8896cabf2489d1b2f82694376f90a3b38049c",
search_query.py CHANGED
@@ -2,6 +2,7 @@
2
  from __future__ import annotations
3
  import re,sqlite3,time
4
  from pathlib import Path
 
5
 
6
  MAX_SNIPPET=600
7
  class SearchInputError(ValueError):pass
@@ -12,7 +13,8 @@ def database(cache_dir):
12
  def compile_query(value):
13
  terms=re.findall(r"[^\W_]+",value.casefold(),flags=re.UNICODE)
14
  if not terms:raise SearchInputError("Query has no searchable terms")
15
- return " AND ".join('"%s"'%term.replace('"','') for term in terms[:12])
 
16
  def search(cache_dir,query,*,ticker=None,doc_type=None,date_from=None,date_to=None,limit=20,offset=0,deadline_seconds=2.0):
17
  fts=compile_query(query);ticker_candidate=query.strip().upper() if re.fullmatch(r"[A-Za-z0-9]{2,6}",query.strip()) else "";conditions=["search MATCH ?"];params=[fts]
18
  if ticker:conditions.append("d.ticker=?");params.append(ticker)
@@ -24,7 +26,10 @@ def search(cache_dir,query,*,ticker=None,doc_type=None,date_from=None,date_to=No
24
  params=[MAX_SNIPPET,MAX_SNIPPET-1,*params,ticker_candidate,limit+1,offset];path=database(cache_dir);connection=sqlite3.connect("file:%s?mode=ro"%path,uri=True,timeout=2);connection.execute("PRAGMA query_only=ON");deadline=time.monotonic()+deadline_seconds;connection.set_progress_handler(lambda:1 if time.monotonic()>deadline else 0,10000)
25
  try:rows=connection.execute(sql,params).fetchall()
26
  finally:connection.close()
27
- names=["document_id","ticker","doc_type","announcement_date","title","source_link","snippet","score"]
28
- return [dict(zip(names,row)) for row in rows[:limit]],len(rows)>limit
 
 
 
29
  def document_citation(cache_dir,document_id):
30
- path=database(cache_dir);connection=sqlite3.connect("file:%s?mode=ro"%path,uri=True);row=connection.execute("SELECT coalesce(source_url,raw_source_uri),markdown_source_uri FROM documents WHERE document_id=?",(document_id,)).fetchone();connection.close();return {"source_link":row[0],"legacy_markdown_source":row[1]} if row else None
 
2
  from __future__ import annotations
3
  import re,sqlite3,time
4
  from pathlib import Path
5
+ from security import public_asx_url
6
 
7
  MAX_SNIPPET=600
8
  class SearchInputError(ValueError):pass
 
13
  def compile_query(value):
14
  terms=re.findall(r"[^\W_]+",value.casefold(),flags=re.UNICODE)
15
  if not terms:raise SearchInputError("Query has no searchable terms")
16
+ if len(terms)>12:raise SearchInputError("Query must contain at most 12 searchable terms")
17
+ return " AND ".join('"%s"'%term.replace('"','') for term in terms)
18
  def search(cache_dir,query,*,ticker=None,doc_type=None,date_from=None,date_to=None,limit=20,offset=0,deadline_seconds=2.0):
19
  fts=compile_query(query);ticker_candidate=query.strip().upper() if re.fullmatch(r"[A-Za-z0-9]{2,6}",query.strip()) else "";conditions=["search MATCH ?"];params=[fts]
20
  if ticker:conditions.append("d.ticker=?");params.append(ticker)
 
26
  params=[MAX_SNIPPET,MAX_SNIPPET-1,*params,ticker_candidate,limit+1,offset];path=database(cache_dir);connection=sqlite3.connect("file:%s?mode=ro"%path,uri=True,timeout=2);connection.execute("PRAGMA query_only=ON");deadline=time.monotonic()+deadline_seconds;connection.set_progress_handler(lambda:1 if time.monotonic()>deadline else 0,10000)
27
  try:rows=connection.execute(sql,params).fetchall()
28
  finally:connection.close()
29
+ names=["document_id","ticker","doc_type","announcement_date","title","source_link","snippet","score"];items=[]
30
+ for row in rows:
31
+ item=dict(zip(names,row));item["source_link"]=public_asx_url(item["source_link"])
32
+ if item["source_link"]:items.append(item)
33
+ return items[:limit],len(items)>limit
34
  def document_citation(cache_dir,document_id):
35
+ path=database(cache_dir);connection=sqlite3.connect("file:%s?mode=ro"%path,uri=True);row=connection.execute("SELECT source_url FROM documents WHERE document_id=?",(document_id,)).fetchone();connection.close();source_link=public_asx_url(row[0]) if row else None;return {"source_link":source_link} if source_link else None
security.py ADDED
@@ -0,0 +1,90 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Fail-closed validation and lightweight public-service abuse controls."""
2
+ from __future__ import annotations
3
+
4
+ import math
5
+ import re
6
+ import threading
7
+ import time
8
+ import urllib.parse
9
+
10
+
11
+ ASX_SOURCE_HOSTS = {"www.asx.com.au", "announcements.asx.com.au"}
12
+ MARKDOWN_ARTIFACT = re.compile(
13
+ r"^artifacts/v1/asx/[A-Z0-9]{2,6}/[A-Za-z0-9][A-Za-z0-9._=-]*(?:/[A-Za-z0-9][A-Za-z0-9._=-]*)*\.md$"
14
+ )
15
+
16
+
17
+ def bounded_text(value: str, name: str, maximum: int, *, required: bool = False) -> str:
18
+ """Strip a public string and reject, rather than truncate, oversized input."""
19
+
20
+ if not isinstance(value, str):
21
+ raise ValueError(f"{name} must be text")
22
+ normalized = value.strip()
23
+ if required and not normalized:
24
+ raise ValueError(f"{name} is required")
25
+ if len(normalized) > maximum:
26
+ raise ValueError(f"{name} must be at most {maximum} characters")
27
+ return normalized
28
+
29
+
30
+ def finite_non_negative(value: float | None, name: str) -> float | None:
31
+ """Reject NaN, infinity and negative public numeric filters."""
32
+
33
+ if value is None:
34
+ return None
35
+ number = float(value)
36
+ if not math.isfinite(number) or number < 0:
37
+ raise ValueError(f"{name} must be a finite non-negative number")
38
+ return number
39
+
40
+
41
+ def public_asx_url(value: str | None) -> str | None:
42
+ """Return only an HTTPS citation on an approved ASX hostname."""
43
+
44
+ if not value or len(value) > 2048:
45
+ return None
46
+ try:
47
+ parsed = urllib.parse.urlsplit(value)
48
+ port = parsed.port
49
+ except ValueError:
50
+ return None
51
+ if parsed.scheme != "https" or parsed.hostname not in ASX_SOURCE_HOSTS:
52
+ return None
53
+ if parsed.username or parsed.password or port not in (None, 443):
54
+ return None
55
+ return value
56
+
57
+
58
+ def valid_markdown_artifact_key(value: str | None) -> bool:
59
+ """Allow only canonical public markdown keys, never URLs or paths."""
60
+
61
+ return bool(value and len(value) <= 512 and MARKDOWN_ARTIFACT.fullmatch(value))
62
+
63
+
64
+ class TokenBucket:
65
+ """Thread-safe global token bucket with bounded memory."""
66
+
67
+ def __init__(self, capacity: int, refill_per_second: float):
68
+ if capacity < 1 or refill_per_second <= 0:
69
+ raise ValueError("invalid token bucket configuration")
70
+ self.capacity = float(capacity)
71
+ self.refill_per_second = float(refill_per_second)
72
+ self._tokens = float(capacity)
73
+ self._updated_at = time.monotonic()
74
+ self._lock = threading.Lock()
75
+
76
+ def consume(self, amount: float = 1.0, *, now: float | None = None) -> bool:
77
+ if amount <= 0 or amount > self.capacity:
78
+ return False
79
+ current = time.monotonic() if now is None else now
80
+ with self._lock:
81
+ elapsed = max(0.0, current - self._updated_at)
82
+ self._tokens = min(
83
+ self.capacity,
84
+ self._tokens + elapsed * self.refill_per_second,
85
+ )
86
+ self._updated_at = current
87
+ if self._tokens < amount:
88
+ return False
89
+ self._tokens -= amount
90
+ return True
tests/test_search_query.py CHANGED
@@ -5,8 +5,8 @@ from search_query import SearchInputError,search
5
  class SearchTests(unittest.TestCase):
6
  def test_safe_ranked_filtered_bounded_search(self):
7
  with tempfile.TemporaryDirectory() as td:
8
- p=Path(td)/"indexes/v1/x";p.mkdir(parents=True);db=p/"search.sqlite";c=sqlite3.connect(db);c.executescript("CREATE TABLE documents(rowid INTEGER PRIMARY KEY,document_id TEXT,ticker TEXT,doc_type TEXT,announced_date TEXT,title TEXT,source_url TEXT,raw_source_uri TEXT);CREATE VIRTUAL TABLE search USING fts5(title,ticker,doc_type,announced_date,body);INSERT INTO documents VALUES(1,'ASX:BHP:1','BHP','Report','2026-01-01','Annual Report','https://asx/1',NULL);INSERT INTO search(rowid,title,ticker,doc_type,announced_date,body) VALUES(1,'Annual Report','BHP','Report','2026-01-01','Capital raising details');");c.commit();c.close()
9
  lower,_=search(td,"annual report",limit=10);upper,_=search(td,"ANNUAL REPORT",limit=10);self.assertEqual(lower,upper);self.assertEqual(lower[0]["ticker"],"BHP");self.assertLessEqual(len(lower[0]["snippet"]),600)
10
- filtered,_=search(td,"capital-raising",ticker="BHP",doc_type="Report",date_from="2026-01-01",limit=10);self.assertEqual(filtered[0]["source_link"],"https://asx/1")
11
  with self.assertRaises(SearchInputError):search(td,"---___",limit=10)
12
  if __name__=="__main__":unittest.main()
 
5
  class SearchTests(unittest.TestCase):
6
  def test_safe_ranked_filtered_bounded_search(self):
7
  with tempfile.TemporaryDirectory() as td:
8
+ p=Path(td)/"indexes/v1/x";p.mkdir(parents=True);db=p/"search.sqlite";c=sqlite3.connect(db);c.executescript("CREATE TABLE documents(rowid INTEGER PRIMARY KEY,document_id TEXT,ticker TEXT,doc_type TEXT,announced_date TEXT,title TEXT,source_url TEXT,raw_source_uri TEXT);CREATE VIRTUAL TABLE search USING fts5(title,ticker,doc_type,announced_date,body);INSERT INTO documents VALUES(1,'ASX:BHP:1','BHP','Report','2026-01-01','Annual Report','https://www.asx.com.au/asx/1',NULL);INSERT INTO search(rowid,title,ticker,doc_type,announced_date,body) VALUES(1,'Annual Report','BHP','Report','2026-01-01','Capital raising details');");c.commit();c.close()
9
  lower,_=search(td,"annual report",limit=10);upper,_=search(td,"ANNUAL REPORT",limit=10);self.assertEqual(lower,upper);self.assertEqual(lower[0]["ticker"],"BHP");self.assertLessEqual(len(lower[0]["snippet"]),600)
10
+ filtered,_=search(td,"capital-raising",ticker="BHP",doc_type="Report",date_from="2026-01-01",limit=10);self.assertEqual(filtered[0]["source_link"],"https://www.asx.com.au/asx/1")
11
  with self.assertRaises(SearchInputError):search(td,"---___",limit=10)
12
  if __name__=="__main__":unittest.main()
tests/test_security.py ADDED
@@ -0,0 +1,102 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import math
4
+ import os
5
+ import unittest
6
+ from unittest.mock import patch
7
+
8
+ from fastapi.testclient import TestClient
9
+
10
+ import app
11
+ from search_query import SearchInputError,compile_query
12
+ from security import TokenBucket,bounded_text,finite_non_negative,public_asx_url,valid_markdown_artifact_key
13
+
14
+
15
+ REVISION = "d2b8896cabf2489d1b2f82694376f90a3b38049c"
16
+ CONTEXT = {
17
+ "dataset_revision": REVISION,
18
+ "published_at": "2026-07-17T12:49:16Z",
19
+ "source_watermark": {"provider": "ASX"},
20
+ "dataset_citation": "https://huggingface.co/datasets/mzx/dilutionrisk-data/tree/" + REVISION,
21
+ }
22
+
23
+
24
+ class SecurityTests(unittest.TestCase):
25
+ def test_public_text_and_numeric_inputs_fail_closed(self):
26
+ self.assertEqual(bounded_text(" BHP ", "ticker", 6), "BHP")
27
+ with self.assertRaisesRegex(ValueError, "at most 6"):
28
+ bounded_text("TOO-LONG", "ticker", 6)
29
+ for value in (-1, math.inf, -math.inf, math.nan):
30
+ with self.subTest(value=value), self.assertRaises(ValueError):
31
+ finite_non_negative(value, "risk")
32
+ with self.assertRaises(SearchInputError):
33
+ compile_query("one two three four five six seven eight nine ten eleven twelve thirteen")
34
+
35
+ def test_citations_and_markdown_keys_are_strictly_allowlisted(self):
36
+ self.assertEqual(
37
+ public_asx_url("https://www.asx.com.au/asx/v2/statistics/announcements.do?id=1"),
38
+ "https://www.asx.com.au/asx/v2/statistics/announcements.do?id=1",
39
+ )
40
+ for value in ("http://www.asx.com.au/a", "https://evil.example/a", "file:///etc/passwd", "gs://private/key"):
41
+ with self.subTest(value=value):
42
+ self.assertIsNone(public_asx_url(value))
43
+ self.assertTrue(valid_markdown_artifact_key("artifacts/v1/asx/CLA/03115626.md"))
44
+ for value in (
45
+ "artifacts/v1/asx/CLA/../../etc/passwd.md",
46
+ "https://evil.example/file.md",
47
+ "/etc/passwd.md",
48
+ "artifacts/v1/asx/CLA/file.pdf",
49
+ ):
50
+ with self.subTest(value=value):
51
+ self.assertFalse(valid_markdown_artifact_key(value))
52
+
53
+ def test_markdown_tool_never_fetches_caller_controlled_targets(self):
54
+ document = {"document_id": "ASX:CLA:03115626", "markdown_artifact_key": "artifacts/v1/asx/CLA/../../etc/passwd.md"}
55
+ cache = {"cache_dir": "/tmp/cache", "dataset_revision": REVISION}
56
+ with patch.object(app, "tool_context", return_value=(cache, CONTEXT)), patch.object(app, "get_document", return_value=document), patch.object(app, "document_citation", return_value={"source_link": "https://www.asx.com.au/a"}), patch.object(app.urllib.request, "urlopen") as urlopen:
57
+ with self.assertRaisesRegex(ValueError, "invalid published HF"):
58
+ app.get_document_markdown(document["document_id"])
59
+ urlopen.assert_not_called()
60
+
61
+ def test_oversized_mcp_inputs_raise_clear_errors(self):
62
+ cache = {"cache_dir": "/tmp/cache", "dataset_revision": REVISION}
63
+ with patch.object(app, "tool_context", return_value=(cache, CONTEXT)):
64
+ with self.assertRaisesRegex(ValueError, "at most 120"):
65
+ app.search_companies(query="x" * 121)
66
+ with self.assertRaisesRegex(ValueError, "at most 80"):
67
+ app.search_documents(query="capital", doc_type="x" * 81)
68
+
69
+ def test_token_bucket_degrades_repeated_requests_without_state_growth(self):
70
+ bucket = TokenBucket(capacity=2, refill_per_second=1)
71
+ self.assertTrue(bucket.consume(now=100.0))
72
+ self.assertTrue(bucket.consume(now=100.0))
73
+ self.assertFalse(bucket.consume(now=100.0))
74
+ self.assertTrue(bucket.consume(now=101.0))
75
+
76
+ def test_http_rate_limit_and_read_only_guards_are_structured(self):
77
+ client = TestClient(app.app)
78
+ with patch.object(app.REQUEST_BUDGET, "consume", return_value=False):
79
+ response = client.get("/api/v1/status")
80
+ self.assertEqual(response.status_code, 429)
81
+ self.assertEqual(response.json()["error"]["code"], "rate_limited")
82
+ self.assertEqual(response.headers["retry-after"], "1")
83
+
84
+ response = client.post("/api/v1/status")
85
+ self.assertEqual(response.status_code, 405)
86
+ self.assertEqual(response.json()["error"]["code"], "read_only_surface")
87
+ response = client.post("/gradio_api/upload")
88
+ self.assertEqual(response.status_code, 403)
89
+ self.assertEqual(response.json()["error"]["code"], "read_only_surface")
90
+ response = client.get("/file=/etc/passwd")
91
+ self.assertEqual(response.status_code, 403)
92
+ self.assertEqual(response.json()["error"]["code"], "read_only_surface")
93
+
94
+ def test_public_status_does_not_disclose_environment_secrets(self):
95
+ cache = {"cache_dir": "/tmp/cache", "dataset_revision": REVISION}
96
+ with patch.dict(os.environ, {"HF_TOKEN": "SECRET_SENTINEL", "OPENAI_API_KEY": "SECRET_SENTINEL"}), patch.object(app, "tool_context", return_value=(cache, CONTEXT)), patch.object(app, "health", return_value={"ready": True}):
97
+ payload = app.get_dataset_status()
98
+ self.assertNotIn("SECRET_SENTINEL", repr(payload))
99
+
100
+
101
+ if __name__ == "__main__":
102
+ unittest.main()