PrajwalN-AIS commited on
Commit
d7fd43f
·
verified ·
1 Parent(s): 7037153

Upload folder using huggingface_hub

Browse files
Files changed (4) hide show
  1. Dockerfile +12 -0
  2. README.md +42 -4
  3. config.yaml +17 -0
  4. trace_logger.py +40 -0
Dockerfile ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # LiteLLM proxy for the Bolt Instruct V2 hackathon.
2
+ # Routes student requests to the Bolt endpoint, issues per-team keys,
3
+ # and logs every prompt->response pair for V3 training data.
4
+ FROM ghcr.io/berriai/litellm:main-latest
5
+
6
+ # HF Spaces serve on port 7860
7
+ EXPOSE 7860
8
+
9
+ COPY config.yaml /app/config.yaml
10
+ COPY trace_logger.py /app/trace_logger.py
11
+
12
+ CMD ["--config", "/app/config.yaml", "--port", "7860"]
README.md CHANGED
@@ -1,10 +1,48 @@
1
  ---
2
  title: Bolt Hackathon Proxy
3
- emoji: 🐢
4
- colorFrom: red
5
- colorTo: red
6
  sdk: docker
 
7
  pinned: false
8
  ---
9
 
10
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  title: Bolt Hackathon Proxy
3
+ emoji: ⚡
4
+ colorFrom: blue
5
+ colorTo: indigo
6
  sdk: docker
7
+ app_port: 7860
8
  pinned: false
9
  ---
10
 
11
+ # Bolt Instruct V2 — Hackathon Proxy
12
+
13
+ A LiteLLM proxy that routes student requests to the Bolt Instruct V2 endpoint,
14
+ issues one API key per team, and logs every prompt→response pair for V3
15
+ training data.
16
+
17
+ ## Required Space secrets
18
+
19
+ | secret | value |
20
+ |---|---|
21
+ | `HF_TOKEN` | a scoped token that can call the Bolt endpoint |
22
+ | `MASTER_KEY` | admin key for minting team keys, e.g. `sk-hack-master-...` |
23
+ | `DATABASE_URL` | managed Postgres URL (Supabase / Neon) — enables per-team keys |
24
+
25
+ ## Durable traces
26
+
27
+ Add **persistent storage** to the Space so `/data/traces.jsonl` survives
28
+ restarts. Without it, traces are lost when the Space rebuilds.
29
+
30
+ ## Mint a team key
31
+
32
+ ```bash
33
+ curl https://<space-url>/key/generate \
34
+ -H "Authorization: Bearer $MASTER_KEY" \
35
+ -d '{"key_alias": "team-01", "max_budget": 20}'
36
+ ```
37
+
38
+ ## What students use
39
+
40
+ ```python
41
+ from openai import OpenAI
42
+ client = OpenAI(base_url="https://<space-url>/v1", api_key="<team key>")
43
+ r = client.chat.completions.create(
44
+ model="bolt-instruct-v2",
45
+ messages=[{"role": "user", "content": "Hello!"}],
46
+ )
47
+ print(r.choices[0].message.content)
48
+ ```
config.yaml ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model_list:
2
+ - model_name: bolt-instruct-v2
3
+ litellm_params:
4
+ model: openai/aisquared/bolt-instruct-v2
5
+ api_base: https://6ad27aecd2d0ab8296e58e32.endpoints.huggingface.cloud/v1
6
+ api_key: os.environ/HF_TOKEN
7
+ # Default generation length; keeps one long request from hogging a slot.
8
+ max_tokens: 1024
9
+
10
+ litellm_settings:
11
+ callbacks: trace_logger.trace_logger
12
+ drop_params: true
13
+
14
+ general_settings:
15
+ # One shared key for the whole hackathon — no database needed.
16
+ # Students use this value as their API key.
17
+ master_key: os.environ/MASTER_KEY
trace_logger.py ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # LiteLLM custom callback: append every prompt->response pair to traces.jsonl.
2
+ # This is the V3 training-data capture.
3
+ import json
4
+ import time
5
+ from litellm.integrations.custom_logger import CustomLogger
6
+
7
+ OUT = "/data/traces.jsonl"
8
+
9
+
10
+ class TraceLogger(CustomLogger):
11
+ def _write(self, kwargs, response_obj, start, end):
12
+ try:
13
+ meta = (kwargs.get("litellm_params", {}) or {}).get("metadata", {}) or {}
14
+ try:
15
+ lat = (end - start).total_seconds()
16
+ except Exception:
17
+ lat = None
18
+ rec = {
19
+ "ts": time.time(),
20
+ "team": meta.get("user_api_key_alias") or meta.get("user_api_key"),
21
+ "messages": kwargs.get("messages"),
22
+ "tools": (kwargs.get("optional_params", {}) or {}).get("tools"),
23
+ "response": response_obj.model_dump()
24
+ if hasattr(response_obj, "model_dump") else str(response_obj),
25
+ "latency_s": lat,
26
+ }
27
+ with open(OUT, "a") as f:
28
+ f.write(json.dumps(rec, default=str) + "\n")
29
+ except Exception as e:
30
+ with open(OUT + ".err", "a") as f:
31
+ f.write(f"{type(e).__name__}: {e}\n")
32
+
33
+ async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
34
+ self._write(kwargs, response_obj, start_time, end_time)
35
+
36
+ def log_success_event(self, kwargs, response_obj, start_time, end_time):
37
+ self._write(kwargs, response_obj, start_time, end_time)
38
+
39
+
40
+ trace_logger = TraceLogger()