--- title: FabAgent emoji: ๐ŸŸฆ colorFrom: blue colorTo: indigo sdk: streamlit sdk_version: 1.36.0 app_file: app.py pinned: false license: mit short_description: ๋ฐ˜๋„์ฒด ๊ณต์ • ์ด์ƒ์˜ ํƒ์ง€ยท์›์ธยท์˜ํ–ฅยท๋Œ€์‘์„ ์ž‡๋Š” ๋ฉ€ํ‹ฐ ์—์ด์ „ํŠธ ์šด์˜ ํ”Œ๋žซํผ --- # FabAgent ๋ฐ˜๋„์ฒด ๊ณต์ • ์ด์ƒ์˜ **ํƒ์ง€ โ†’ ์›์ธ ๋ถ„์„ โ†’ ์˜ํ–ฅ ํ‰๊ฐ€ โ†’ ๋Œ€์‘ ๊ถŒ๊ณ **๋ฅผ ํ•˜๋‚˜์˜ ๋ฉ€ํ‹ฐ ์—์ด์ „ํŠธ ํŒŒ์ดํ”„๋ผ์ธ์œผ๋กœ ํ†ตํ•ฉํ•˜๋Š” ์šด์˜ ํ”Œ๋žซํผ์ž…๋‹ˆ๋‹ค. ๊ฐ Tier๊ฐ€ **์ž์œจ ๋„๊ตฌ ํ˜ธ์ถœ(tool calling)** ๊ณผ **์กฐ๊ฑด๋ถ€ ๋ผ์šฐํŒ…** ์„ ํ†ตํ•ด ์˜์‚ฌ๊ฒฐ์ •์„ ์ง„ํ–‰ํ•˜๋Š” ์ง„์งœ LLM agent๋กœ ๊ตฌ์„ฑ๋˜์–ด, ๋‹จ์ผ LLM ์ฑ—๋ด‡๊ณผ ๋‹ฌ๋ฆฌ ์ถ”์  ๊ฐ€๋Šฅํ•˜๊ณ  ๋ชจ๋“ˆํ™”๋œ ์˜์‚ฌ๊ฒฐ์ • ํ๋ฆ„๊ณผ auditableํ•œ reasoning trace๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ## ํ•ต์‹ฌ ํŠน์ง• - **4-Tier multi-agent system** - ํƒ์ง€(ML) ยท ์›์ธ(agentic RAG) ยท ์˜ํ–ฅ(tool-using) ยท ๋Œ€์‘(tool-using) - **Plan-and-Execute ํŒจํ„ด** (๊ธฐ๋ณธ) - Central Planner Agent๊ฐ€ ์ „์ฒด ์›Œํฌํ”Œ๋กœ์šฐ plan์„ 1ํšŒ ์‚ฐ์ถœ, Tier executor๊ฐ€ plan๋Œ€๋กœ ์‹คํ–‰ โ†’ ํ†ต์‹  ์˜ค๋ฒ„ํ—ค๋“œ ์ตœ์†Œํ™” (LLM ํ˜ธ์ถœ -60%, latency -54%) - **Tool-using agent** (์˜ต์…˜) - 7๊ฐœ ๋„๋ฉ”์ธ ๋„๊ตฌ๋ฅผ LLM์ด ์ž์œจ ์„ ํƒยท๋ฐ˜๋ณต ํ˜ธ์ถœ (`AGENT_MODE=autonomous`) - **Supervisor agent** (autonomous ๋ชจ๋“œ) - LLM์ด Tier 2 ๊ฒฐ๊ณผ๋ฅผ ๋ณด๊ณ  ํ›„์† workflow path(proceed_full / fast_track / escalate) ๋™์  ๊ฒฐ์ • - **์กฐ๊ฑด๋ถ€ ๋ผ์šฐํŒ…** - LangGraph์—์„œ severity gate + cause confidence retry + supervisor ๋ถ„๊ธฐ (3๋‹จ๊ณ„) - **CRAG self-correction** - retrieval grader๊ฐ€ ๊ฒ€์ƒ‰ ํ’ˆ์งˆ ํ‰๊ฐ€, ์ž„๊ณ„์น˜ ๋ฏธ๋‹ฌ ์‹œ ์ฟผ๋ฆฌ ์ž๋™ ์žฌ์ž‘์„ฑ - **LangSmith observability** - ๋ชจ๋“  LLMยทtoolยทagent ํ˜ธ์ถœ ์ž๋™ ํŠธ๋ ˆ์ด์Šค (production-grade) - **Production RAG** - BM25 + FAISS + Reciprocal Rank Fusion (5๋‹จ๊ณ„ paradigm ablation์œผ๋กœ ๊ฒ€์ฆ) - **์‹ค๋ฐ์ดํ„ฐ ๊ธฐ๋ฐ˜** - UCI SECOM (590 ์ต๋ช… ์„ผ์„œ) + PHM 2016 CMP (25๊ฐœ ๋ช…๋ช… ์„ผ์„œ, ์‹ค์ธก) - **์ž๊ฐ€ ํ•™์Šต ๋ฃจํ”„** - ์šด์˜์ž ์Šน์ธ ์‹œ ๋ถ„์„ ๊ฒฐ๊ณผ๊ฐ€ ์ธ์‹œ๋˜ํŠธ DB(.md)์— ์ž๋™ ๊ธฐ๋ก โ†’ ๋‹ค์Œ RAG์— ์ฆ‰์‹œ ๋ฐ˜์˜ - **์ •๋Ÿ‰ ๊ทผ๊ฑฐ** - ํ•ต์‹ฌ ์˜์‚ฌ๊ฒฐ์ •๋งˆ๋‹ค ablation/๋ฒค์น˜๋งˆํฌ/์ฐจํŠธ (`experiments/*/results.md`) ## ์•„ํ‚คํ…์ฒ˜ ### 4-Tier Multi-Agent Pipeline ``` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ ์•Œ๋žŒ ์ธ๋ฐ•์Šค โ”‚ ์‚ฌ์ด๋“œ๋ฐ”์—์„œ ์•Œ๋žŒ ์„ ํƒ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ–ผ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ agents/orchestrator.py - LangGraph StateGraph (์กฐ๊ฑด๋ถ€ + LLM-driven ๋ผ์šฐํŒ…) โ”‚ โ”‚ โ”‚ โ”‚ START โ†’ [detect] โ”€โ”€โ”€โ”€ score < 0.30 โ”€โ”€โ†’ [noise] โ†’ END (severity gate) โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ score โ‰ฅ 0.30 โ”€โ”€โ†’ [cause] โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€ max pct < 40% โ”€โ”€โ”€โ”€โ”€โ†’ [cause_retry] โ”€โ”€โ” โ”‚ โ”‚ โ”‚ (output threshold) โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€ max pct โ‰ฅ 40% โ”€โ”€โ†’ [supervisor] โ†โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€ LLM ๊ฒฐ์ • โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ โ”‚ โ”‚ proceed_full โ”‚ โ”‚ โ”‚ โ”‚ escalate โ–ผ โ”‚ โ”‚ โ”‚ [impact] โ”€โ”€โ” โ”‚ โ”‚ โ”‚ โ–ผ โ”‚ โ”‚ โ”‚ fast_track โ”€โ”€โ†’ [fast_impact] โ†’ [response] โ†’ END โ”‚ โ”‚ โ”‚ โ–ฒ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚ โ”‚ Tier 1: IsolationForest (ML) - SECOM/PHM ๋ฐ์ดํ„ฐ ๋ถ„๊ธฐ โ”‚ โ”‚ Tier 2: agentic RAG (tools: search_knowledge, lookup_incident, get_pm) โ”‚ โ”‚ Supervisor (gpt-4o-mini): actionยทseverityยทreasoning์„ LLM์ด ๊ฒฐ์ • โ”‚ โ”‚ Tier 3: tool-using (query_wip, get_downstream, get_yield_baseline, pm) โ”‚ โ”‚ fast_impact: LLM ํ˜ธ์ถœ ์—†๋Š” deterministic ๊ฒฝ๋Ÿ‰ ์ฒ˜๋ฆฌ (fast_track ์ „์šฉ) โ”‚ โ”‚ Tier 4: tool-using (search, lookup_incident, get_pm, check_pm_schedule) โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ–ผ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ TierData (๋‹จ์ผ ๊ณ„์•ฝ) โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ–ผ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ Streamlit UI (cascade) โ”‚ โ”‚ Tier 4 ์Šน์ธ โ†’ ์ž‘์—…์ง€์‹œ โ”‚ โ”‚ + ์ž๊ฐ€ ํ•™์Šต ๋ฃจํ”„ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` ### 7๊ฐœ Agent Tools | ๋„๊ตฌ | ๋ฐ˜ํ™˜ | ์‚ฌ์šฉ Tier | |---|---|---| | `search_knowledge` | INC/FMEA/SOP/FLOW ๋ฌธ์„œ hybrid ๊ฒ€์ƒ‰ ๊ฒฐ๊ณผ | 2, 4 | | `lookup_incident_history` | ๊ณผ๊ฑฐ incident ๊ตฌ์กฐํ™” ๋ ˆ์ฝ”๋“œ (์›์ธยทํ•ด๊ฒฐ์ฑ…ยทyield ํšŒ๋ณต๋ฅ ) | 2, 4 | | `get_pm_history` | ์žฅ๋น„ ๋งˆ์ง€๋ง‰ PM ์ผ์žยท๊ฒฝ๊ณผ์ผยทoverdue ์—ฌ๋ถ€ | 2, 3, 4 | | `check_pm_schedule` | ๋‹ค์Œ 7์ผ ๊ฐ€์šฉ PM ์œˆ๋„์šฐ | 4 | | `query_wip_status` | ์˜ํ–ฅ ๋ฐ›๋Š” WIP lot/wafer ์ˆ˜ | 3 | | `get_downstream_steps` | ํ›„๊ณต์ • ์˜์กด์„ฑ (typical deltaยทseverity) | 3 | | `get_yield_baseline` | ๊ณต์ • ์ตœ๊ทผ 30์ผ yield ๊ธฐ์ค€์„  (%) | 3 | LLM์ด ์–ด๋–ค ๋„๊ตฌ๋ฅผ ์–ธ์ œ ํ˜ธ์ถœํ• ์ง€ ์ž์œจ ๊ฒฐ์ •ํ•ฉ๋‹ˆ๋‹ค. ํ˜ธ์ถœ ๋กœ๊ทธ๊ฐ€ reasoning trace = production audit trail. ## ๋ฐ์ดํ„ฐ & ๋ชจ๋ธ | ์•Œ๋žŒ | ๊ณต์ • step | ๋ฐ์ดํ„ฐ | ๋ชจ๋ธ / ๊ธฐ์ˆ  | |---|---|---|---| | A1 | Photo (๋…ธ๊ด‘) | UCI SECOM (590 ์ต๋ช… ์„ผ์„œ) | IsolationForest + gpt-5-mini agent + Hybrid RAG | | A2 | Etch (์‹๊ฐ) | UCI SECOM (๋‹ค๋ฅธ fail row) | IsolationForest + gpt-5-mini agent + Hybrid RAG | | A3 | CMP (์—ฐ๋งˆ) | **PHM 2016 CMP** (25๊ฐœ ๋ช…๋ช… ์„ผ์„œ, SLURRY_FLOW ๋“ฑ) | IsolationForest + gpt-5-mini agent + Hybrid RAG | SECOM์€ ์ต๋ช… ์ฒ˜๋ฆฌ๋œ ํ‘œ์ค€ ๋ฒค์น˜๋งˆํฌ๋ผ ๊ณต์ • step ๋ผ๋ฒจ์ด narrative์ž…๋‹ˆ๋‹ค (ํ•œ๊ณ„ ๋ช…์‹œ). PHM 2016 CMP๋Š” ์‹ค์ œ CMP ๊ณต์ • ์„ผ์„œ ๋ฐ์ดํ„ฐ๋กœ step-specific ์ถ”๋ก ์ด ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค. ## ์ •๋Ÿ‰ ํ‰๊ฐ€ ์š”์•ฝ ํ•ต์‹ฌ ์˜์‚ฌ๊ฒฐ์ •๋งˆ๋‹ค ablation ์‹คํ—˜์„ ์ˆ˜ํ–‰ํ•˜๊ณ  ๊ฒฐ๊ณผยท์ฐจํŠธยท์‹œํ–‰์ฐฉ์˜ค๋ฅผ ๊ธฐ๋กํ–ˆ์Šต๋‹ˆ๋‹ค. ์ƒ์„ธ๋Š” [experiments/README.md](experiments/README.md). | ID | ์‹คํ—˜ | ๊ฒฐ์ • | ํ•ต์‹ฌ ์ˆ˜์น˜ | |---|---|---|---| | **D1** | Tier 1 ๋ชจ๋ธ ๋น„๊ต | IsolationForest | PR-AUC 0.129 (LOF 0.089, OC-SVM 0.098, baseline 0.119) | | **D2** | Retrieval ๋ฐฑ์—”๋“œ latency | 4 backend ๊ฒ€์ฆ | keyword 0.5ms / faiss 60ms / **hybrid 54ms** / +rerank 326ms | | **D5** | Multi vs Single LLM | Multi-Agent | ๋น„์šฉ $0.018 vs $0.008 (2.2x), ๊ถŒ๊ณ  ๊นŠ์ด 1.6~1.9๋ฐฐ, ๋ชจ๋“ˆํ™” ๊ฒฐ์ •์  | | **D6** | RAG paradigm 5๋‹จ๊ณ„ ablation | **Hybrid** ์ฑ„ํƒ | faithfulness: No RAG 0.32 โ†’ Hybrid 0.82 (2.5x). Hybrid๊ฐ€ ๋ชจ๋“  ์ง€ํ‘œ 1์œ„ | | **D7** | Workflow vs Agentic ๋น„๊ต | **Agentic** ์ฑ„ํƒ | tool 0โ†’13, ์ธ์šฉ ๊นŠ์ด +25%, ๋น„์šฉ 2.6x, latency 2.3x, reasoning trace ํ™•๋ณด | | **D8** | CRAG (Self-correction) ON vs OFF | CRAG **ํ™œ์„ฑ ์œ ์ง€** (๊ด€์ธก ๊ฐ€์น˜) | ํ’ˆ์งˆ ๋ณ€ํ™” -0.1%p (๋™๊ธ‰), refinement ๋ฐœ๋™๋ฅ  20%, relevance_score ๋…ธ์ถœ, ๋น„์šฉ +31% | | **D9** | ํ•œ๊ตญ์–ด reranker (Dongjin-kr/ko-reranker) vs ์˜์–ด(BAAI) vs hybrid (12 docs) | ๋‘˜ ๋‹ค hybrid์— ๋ฏธ๋‹ฌ | hybrid 0.734 / BAAI 0.714 / ko 0.703 | | **D10** | **D9 ํ›„์†**: ์ฝ”ํผ์Šค 12โ†’34 ํ™•์žฅ ํ›„ reranker ์žฌํ‰๊ฐ€ | **๊ฐ€์„ค ๊ฒ€์ฆ - ํšจ๊ณผ ์™„์ „ ๋ฐ˜์ „** | hybrid **0.592** / BAAI **0.709 (+0.117)** / ko **0.675 (+0.083)** | | **D11** | Conductor (Plan-and-Execute) vs Autonomous (tool-using loop) | **Conductor ์ฑ„ํƒ** | LLM -60%, Latency **131โ†’60์ดˆ (-54%)**, ๋น„์šฉ -58%, **์ธ์šฉ ๋™๋“ฑ(6.0)** | ### D6 ํ•ต์‹ฌ ๊ทธ๋ž˜ํ”„ ![RAG Paradigm Evolution](experiments/rag_paradigm/charts/ragas_comparison.png) ![Quality vs Latency Trade-off](experiments/rag_paradigm/charts/tradeoff.png) ### D7 ํ•ต์‹ฌ ๊ทธ๋ž˜ํ”„ ![Workflow vs Agentic - ํ˜ธ์ถœ ํšŸ์ˆ˜](experiments/agentic_vs_workflow/charts/calls_citations.png) ![Tier๋ณ„ Latency](experiments/agentic_vs_workflow/charts/latency_per_tier.png) ### D8 ํ•ต์‹ฌ ๊ทธ๋ž˜ํ”„ ![CRAG ์ž๊ฐ€ ์ •์ • ํ™œ๋™](experiments/crag_eval/charts/crag_activity.png) ![CRAG ํšจ๊ณผ - ๋‹ต๋ณ€ ํ’ˆ์งˆ](experiments/crag_eval/charts/quality.png) ### D10 ํ•ต์‹ฌ ๊ทธ๋ž˜ํ”„ (ํ™•์žฅ ์ฝ”ํผ์Šค์—์„œ reranker ํšจ๊ณผ ๋ฐ˜์ „ ๊ฒ€์ฆ) ![Reranker ๋น„๊ต (34 docs)](experiments/reranker_compare/charts/reranker_comparison.png) ### D11 ํ•ต์‹ฌ ๊ทธ๋ž˜ํ”„ (Conductor vs Autonomous) ![ํ˜ธ์ถœ ํšŸ์ˆ˜ ๋น„๊ต](experiments/conductor_vs_autonomous/charts/calls_comparison.png) ![Latency ๋น„๊ต](experiments/conductor_vs_autonomous/charts/latency_comparison.png) ![๋น„์šฉ ๋น„๊ต](experiments/conductor_vs_autonomous/charts/cost_comparison.png) ## ์‹œํ–‰์ฐฉ์˜ค ์ด ์‹œ์Šคํ…œ์ด ์ฒ˜์Œ๋ถ€ํ„ฐ ์ด ๋ชจ์–‘์ด์—ˆ๋˜ ๊ฑด ์•„๋‹™๋‹ˆ๋‹ค. ์‹ค์ œ๋กœ ๋‹ค์Œ ๋‹ค์„ฏ ๋ฒˆ์˜ ํฐ ๋ฐฉํ–ฅ ์ „ํ™˜์„ ๊ฑฐ์ณค์Šต๋‹ˆ๋‹ค. ### 1. ๋”๋ฏธ ๋ฐ์ดํ„ฐ โ†’ ์‹ค๋ฐ์ดํ„ฐ (M1~M3) - **์‹œ์ž‘**: `data/demo.TIER_DATA`์— ํ•˜๋“œ์ฝ”๋”ฉ๋œ Tier 1~4 ๊ฒฐ๊ณผ๋กœ UI๋งŒ ์‹œ์—ฐ - **๋ฌธ์ œ**: ๋ฐ์ดํ„ฐ ๊ธฐ๋ฐ˜ ์˜์‚ฌ๊ฒฐ์ •์ด ์•„๋‹ˆ๋ผ "์‹œ๋‚˜๋ฆฌ์˜ค ์˜์ƒ"์— ๊ฐ€๊นŒ์›€. ์ธํ„ฐ๋ทฐ ๋ถ€์ ํ•ฉ - **ํ•ด๊ฒฐ**: SECOM(A1ยทA2) + PHM 2016 CMP(A3) ์‹ค๋ฐ์ดํ„ฐ ๋กœ๋” ๊ตฌ์ถ•, Tier 1์„ IsolationForest๋กœ ์‹ค์ œ ์ถ”๋ก  ### 2. ํ‚ค์›Œ๋“œ RAG โ†’ Hybrid โ†’ +Rerank โ†’ **๋‹ค์‹œ Hybrid** (D2, D6) - **์‹œ์ž‘**: ์ฝ”ํผ์Šค๊ฐ€ ์ž‘์•„์„œ(~10๋ฌธ์„œ) ๋‹จ์ˆœ ํ‚ค์›Œ๋“œ ๋งค์นญ์œผ๋กœ ์ถฉ๋ถ„ - **์ถ”๊ฐ€**: BM25(sparse) + FAISS(dense) + Reciprocal Rank Fusion ๊ฒฐํ•ฉ. Anthropic Contextual Retrieval ํŒจํ„ด ์ ์šฉ - **์š•์‹ฌ**: Cross-encoder rerank(`BAAI/bge-reranker-base`)๊นŒ์ง€ ์ถ”๊ฐ€, "production ์ •๋ฐ€" ์–ดํ•„ ์‹œ๋„ - **๋ฐ˜์ „**: 5๋‹จ๊ณ„ paradigm ablation (D6) ๊ฒฐ๊ณผ `Hybrid + Rerank`๊ฐ€ `Hybrid`๋ณด๋‹ค ์˜คํžˆ๋ ค ๋ชปํ•จ - `Hybrid` faithfulness 0.821, answer_relevancy 0.394 - `Hybrid + Rerank` faithfulness 0.819, answer_relevancy **0.167** (๋‚™ํญ ํผ) - **์›์ธ ๋ถ„์„**: โ‘  ์ฝ”ํผ์Šค๊ฐ€ ~10๋ฌธ์„œ๋กœ ์ž‘์•„ Hybrid top-3์ด ์ด๋ฏธ ์ •๋‹ต์— ๊ทผ์ ‘ โ‘ก `bge-reranker-base`๊ฐ€ ์˜์–ด ํ•™์Šต ๋ชจ๋ธ์ด๋ผ ํ•œ๊ตญ์–ด ๋„๋ฉ”์ธ ํ…์ŠคํŠธ์—์„œ ์ ์ˆ˜ ์‹ ํ˜ธ๊ฐ€ ์žก์Œ - **๊ตํ›ˆ**: production ํŒจํ„ด์„ ๋ธ”๋ผ์ธ๋“œ ์ ์šฉํ•˜๋ฉด ์—ญํšจ๊ณผ. **์ •๋Ÿ‰ ํ‰๊ฐ€(RAGAS) ์—†์ด 'rerank๊ฐ€ ์ข‹๋‹ค'๋Š” ํ†ต๋…์„ ๊ทธ๋Œ€๋กœ ๋Œ๊ณ  ๊ฐˆ ๋ป”ํ•จ** - **๊ฒฐ์ •**: ๊ธฐ๋ณธ backend๋ฅผ `hybrid_rerank` โ†’ `hybrid`๋กœ ๋ณ€๊ฒฝ. `Hybrid + Rerank`๋Š” ํ™˜๊ฒฝ๋ณ€์ˆ˜ ์˜ต์…˜์œผ๋กœ ์œ ์ง€ (์ฝ”ํผ์Šค 100+ ํ™•์žฅ ์‹œ ์žฌํ‰๊ฐ€ ๊ถŒ์žฅ) ### 3. "์ด๊ฒŒ ์ง„์งœ agent์ธ๊ฐ€?" ์ž๊ธฐ ๊ฒ€์ฆ (M5) - **์ƒํ™ฉ**: 4-Tier๊ฐ€ LLM ํ˜ธ์ถœํ•˜๋‹ˆ multi-agent๋ผ๊ณ  ์ฃผ์žฅํ•˜๊ณ  ์žˆ์—ˆ์Œ - **๋ƒ‰์ •ํ•œ ์ ๊ฒ€**: Anthropic์˜ ["Building Effective Agents"](https://www.anthropic.com/engineering/building-effective-agents) ์ •์˜ ๊ธฐ์ค€์œผ๋กœ ๋ณด๋ฉด ํ˜„์žฌ๋Š” **workflow** (๊ฐ Tier๊ฐ€ ์‚ฌ์ „ ์ •์˜๋œ RAG ํ•œ ๋ฒˆ + LLM ํ•œ ๋ฒˆ) - **agent์˜ ์ •์˜**: LLM์ด ๋„๊ตฌ๋ฅผ ์ž์œจ ์„ ํƒยท๋ฐ˜๋ณต ํ˜ธ์ถœ, ์กฐ๊ฑด๋ถ€ ๋ถ„๊ธฐ, reasoning loop - **๊ฐญ**: ๋„๊ตฌ ํ˜ธ์ถœ ์—†์Œ / ๋ฃจํ”„ ์—†์Œ / ์กฐ๊ฑด๋ถ€ ๋ผ์šฐํŒ… ์—†์Œ ### 4. Workflow โ†’ Agentic ์ „ํ™˜ (ํ˜„์žฌ) - **๊ตฌํ˜„**: 7๊ฐœ ๋„๋ฉ”์ธ ๋„๊ตฌ ์ •์˜ (`agents/tools/`), Tier 2/3/4๋ฅผ tool-calling loop + synthesis ํ˜ธ์ถœ ํŒจํ„ด์œผ๋กœ ์žฌ์ž‘์„ฑ - **conditional routing**: LangGraph์— severity gate (Tier 1 score < 0.3 โ†’ noise) + cause confidence retry (max pct < 40 โ†’ ํ•œ ๋ฒˆ ์žฌ์‹œ๋„) ๋ถ„๊ธฐ ์ถ”๊ฐ€ - **์ •๋Ÿ‰ ๊ฒ€์ฆ (D7)**: workflow vs agentic 3 ์•Œ๋žŒ ๋น„๊ต - LLM ํ˜ธ์ถœ: 3 โ†’ 9 (x3.0) - Tool ํ˜ธ์ถœ: 0 โ†’ 13 (agentic์˜ reasoning trace) - ์œ ๋‹ˆํฌ ์ธ์šฉ: 4 โ†’ 5 (+25%, ์†”์งํžˆ ์ ๋‹นํ•œ ์ˆ˜์ค€) - ๋น„์šฉ: $0.012 โ†’ $0.030 / ์•Œ๋žŒ (x2.6) - Latency: 83s โ†’ 194s (x2.3) - **์†”์งํ•œ trade-off**: ๋น„์šฉ 2.6๋ฐฐ๊ฐ€ ์ •๋‹นํ™”๋˜๋Š” ์ด์œ ๋Š” ์ธ์šฉ +25%๊ฐ€ ์•„๋‹ˆ๋ผ **tool ํ˜ธ์ถœ ๋กœ๊ทธ ์ž์ฒด๊ฐ€ production audit trail**. fab ํ™˜๊ฒฝ์—์„œ "์ด ๊ถŒ๊ณ ๊ฐ€ ์™œ ๋‚˜์™”๋Š”๊ฐ€"์˜ ๊ฐ์‚ฌ ์ถ”์ ์ด ๊ฒฐ์ •์  ### 5. LangGraph ๋„์ž… (M4) - **์ด์ „**: orchestrator๊ฐ€ ๊ฒฐ์ •๋ก ์  ํ•จ์ˆ˜ ํ˜ธ์ถœ ์‹œํ€€์Šค - **ํ˜„์žฌ**: LangGraph StateGraph + `lru_cache`๋กœ ์ปดํŒŒ์ผ๋œ ๊ทธ๋ž˜ํ”„ - **์ด๋“**: โ‘  mermaid ๋‹ค์ด์–ด๊ทธ๋žจ ์ž๋™ ์ถ”์ถœ โ†’ ๋ถ„๊ธฐ ์‹œ๊ฐํ™” โ‘ก ํ–ฅํ›„ ๋™์  ๋ผ์šฐํŒ…ยท์žฌ์‹œ๋„ยท์ธํ„ฐ๋ŸฝํŠธ ํ™•์žฅ ์‹œ ๋™์ผ ๊ตฌ์กฐ ์œ„์—์„œ ์ ์ง„ ๊ฐ€๋Šฅ ### 6. CRAG (Self-correction) ๋„์ž… - ๋‘ ๋ฒˆ์งธ "์ •๋Ÿ‰์œผ๋กœ ํ†ต๋… ๋ฐ˜๋ฐ•" ์‚ฌ๋ก€ (D8) - **์‹œ์ž‘**: AnthropicยทLangChain์ด ์ž์ฃผ ์–ธ๊ธ‰ํ•˜๋Š” CRAG (Corrective RAG) ํŒจํ„ด ๋„์ž…. retrieval grader๊ฐ€ ๊ฒ€์ƒ‰ ๊ฒฐ๊ณผ๋ฅผ ์ž์ฒด ํ‰๊ฐ€ํ•˜๊ณ , ์ž„๊ณ„์น˜ ๋ฏธ๋‹ฌ ์‹œ ์ฟผ๋ฆฌ๋ฅผ ์žฌ์ž‘์„ฑํ•ด ์žฌ๊ฒ€์ƒ‰ - **๊ตฌํ˜„**: `agents/rag/crag.py` - gpt-4o-mini ๊ธฐ๋ฐ˜ grader/refiner, `search_knowledge` ๋„๊ตฌ์— transparent ํ†ตํ•ฉ (ํ™˜๊ฒฝ๋ณ€์ˆ˜ `CRAG_ENABLED` ํ† ๊ธ€) - **smoke test ๊ฒฐ๊ณผ ์ธ์ƒ์ **: gibberish ์ฟผ๋ฆฌ(`์•Œ์ˆ˜์—†์Œ xyzzy foobar`)์— ๋Œ€ํ•ด grader๊ฐ€ avg score 0.0 ๋ถ€์—ฌ โ†’ LLM์ด `CMP ๊ณต์ • ์‹คํŒจ ๋ชจ๋“œ ๋ถ„์„ ๋ฐ ์Šฌ๋Ÿฌ๋ฆฌ ๊ด€๋ฆฌ ์ ˆ์ฐจ ๊ด€๋ จ ์ •๋ณด` ๋กœ ์ž๋™ ์žฌ์ž‘์„ฑ โ†’ avg score 0.68 ํšŒ๋ณต. ์ง„์งœ ์ž๊ฐ€ ์ •์ • ๋ฉ”์ปค๋‹ˆ์ฆ˜ ์ž‘๋™ - **๋ฐ˜์ „**: 3 ์•Œ๋žŒ ์ •๋Ÿ‰ ๋น„๊ต(D8)์—์„œ ํ’ˆ์งˆ ์ฐจ์ด ์‚ฌ์‹ค์ƒ ์—†์Œ (faithfulness -0.1%p, relevancy -3.3%p) - **์›์ธ ๋ถ„์„**: 1. ์ฝ”ํผ์Šค๊ฐ€ ~10๋ฌธ์„œ๋กœ ์ž‘์•„ hybrid ๊ฒ€์ƒ‰์ด ์ด๋ฏธ ์ž˜ ์ž‘๋™ 2. agentic loop ์ž์ฒด๊ฐ€ ์ด๋ฏธ self-correction ์ผ๋ถ€ ์ˆ˜ํ–‰ (LLM์ด ์ฒซ ๊ฒ€์ƒ‰ ๊ฒฐ๊ณผ๊ฐ€ ๋ถ€์กฑํ•˜๋ฉด ๋‹ค๋ฅธ ์ฟผ๋ฆฌ๋กœ ๋‹ค์‹œ ํ˜ธ์ถœ). CRAG์™€ ๋ถ€๋ถ„ ์ค‘๋ณต 3. Refinement ๋ฐœ๋™๋ฅ  20% (5๋ฒˆ ์ค‘ 1๋ฒˆ) - ์ •์ƒ ์ฟผ๋ฆฌ์—์„  ๋ฌด๋ฐœ๋™ - **๊ตํ›ˆ**: **D6 (Rerank) ์‹œํ–‰์ฐฉ์˜ค์™€ ๊ฐ™์€ ํŒจํ„ด** - production ํŒจํ„ด์„ ์ž‘์€ ๋„๋ฉ”์ธ ์ฝ”ํผ์Šค์— ๋ธ”๋ผ์ธ๋“œ ์ ์šฉํ•˜๋ฉด ROI ๋‚ฎ์Œ. ์ •๋Ÿ‰ ํ‰๊ฐ€ ์—†์ด๋Š” "CRAG ๋„์ž…ํ–ˆ์Œ" ๋งˆ์ผ€ํŒ…์œผ๋กœ ๋๋‚ฌ์„ ๊ฒƒ - **๊ฒฐ์ •**: CRAG **ํ™œ์„ฑ ์œ ์ง€** (`CRAG_ENABLED=true`). - ํ’ˆ์งˆ ํ–ฅ์ƒ ๋ฏธ๋ฏธํ•˜์ง€๋งŒ ์ธ์šฉ ์‹ ๋ขฐ๋„(0~1 relevance_score) ๊ฐ€์‹œํ™”๊ฐ€ production observability์— ๊ฐ€์น˜ - ๋น„์šฉ +31% ์ ˆ๋Œ€๊ฐ’ ๋ฏธ๋ฏธ (1000 ์•Œ๋žŒ๋‹น +$2.90) - ์ฝ”ํผ์Šค 100+ ํ™•์žฅ ๋˜๋Š” ํ•œ๊ตญ์–ด reranker ๋„์ž… ์‹œ ์žฌํ‰๊ฐ€ ๊ถŒ์žฅ ### 7. ํ•œ๊ตญ์–ด reranker ํ›„์† ํ‰๊ฐ€ - D6 ๊ฐ€์„ค ๊ฒ€์ฆ (D9) - **์‹œ์ž‘**: D6์—์„œ "์˜์–ด ํ•™์Šต reranker๊ฐ€ ํ•œ๊ตญ์–ด ์ฝ”ํผ์Šค์—์„œ ํšจ๊ณผ ์—†์Œ" ๊ฐ€์„ค ์ œ์‹œ. `Dongjin-kr/ko-reranker`๋กœ ๊ฒ€์ฆ - **๋ฐฉ๋ฒ•**: `rerank.py`์— `RERANK_MODEL` ํ™˜๊ฒฝ๋ณ€์ˆ˜ ์ถ”๊ฐ€, 6๊ฐœ ๋Œ€ํ‘œ ์ฟผ๋ฆฌ ร— 3 ๋ชจ๋“œ(hybrid / BAAI / ko-reranker)๋กœ CRAG grader ์ฑ„์  - **๊ฒฐ๊ณผ**: hybrid 0.734 (baseline), BAAI 0.714 (-0.020), **ko-reranker 0.703 (-0.031)** - **๋ฐ˜์ „ ์•ˆ์— ๋ฐ˜์ „**: ์ฟผ๋ฆฌ๋ณ„๋กœ ๋ณด๋ฉด ko-reranker๊ฐ€ CMP(+0.10), ์˜๋ฏธ ์šฐํšŒ 1(+0.083)์—์„  ์šฐ์œ„. Etch(-0.18), ์˜๋ฏธ ์šฐํšŒ 2(-0.20)์—์„  ์†์‹ค. ์ „์ฒด ํ‰๊ท ์€ ๋ฌด์Šน๋ถ€ - **ํ•ด์„**: ํ•œ๊ตญ์–ด reranker๊ฐ€ ์˜์–ด๋ณด๋‹จ ๋„๋ฉ”์ธ ์ ํ•ฉ์„ฑ ์•ฝ๊ฐ„ ์šฐ์œ„์ง€๋งŒ, **๋ณธ ์ฝ”ํผ์Šค ๊ทœ๋ชจ(~10๋ฌธ์„œ)์—์„  hybrid top-3์ด ์ด๋ฏธ ์ถฉ๋ถ„ํžˆ ์ •๋ฐ€ํ•ด ์–ด๋–ค reranker๋„ ์˜๋ฏธ ์žˆ๋Š” ์ด๋“ ์—†์Œ** - **๊ฒฐ๋ก (์ž ์ •)**: D6 ๊ฐ€์„ค ๋ถ€๋ถ„์  ์žฌํ™•์ธ - ์ฝ”ํผ์Šค ๊ทœ๋ชจ๊ฐ€ ์ง„์งœ ์›์ธ์ด๋ผ๋Š” ๋” ํฐ ๊ฐ€์„ค์„ ์ œ์‹œ ### 8. Supervisor agent - LLM-driven ๋™์  workflow routing - **์‹œ์ž‘**: ๊ธฐ์กด conditional edge๋Š” threshold ๊ธฐ๋ฐ˜(`score < 0.3` โ†’ skip, `max pct < 40` โ†’ retry). "์ง„์งœ agent๋ผ๋ฉด LLM์ด ๋งฅ๋ฝ์„ ๋ณด๊ณ  ๊ฒฐ์ •ํ•ด์•ผ ํ•œ๋‹ค" - **๊ตฌํ˜„**: `agents/supervisor.py` - Tier 2 ๊ฒฐ๊ณผ๋ฅผ ๋ฐ›์•„ 3๊ฐ€์ง€ action ๊ฒฐ์ • - `proceed_full` (ํ‘œ์ค€): Tier 3 โ†’ Tier 4 - `fast_track` (๋‹จ์ผ ์›์ธ ์šฐ์„ธ): Tier 3 LLM skip, deterministic ๊ฒฝ๋Ÿ‰ ์ฒ˜๋ฆฌ๋กœ ๋น„์šฉ ์ ˆ๊ฐ - `escalate` (๊ณ ์œ„ํ—˜): ์ •์ƒ ์ง„ํ–‰ + human review ํ”Œ๋ž˜๊ทธ (Tier 4 immediate ์ฒซ ํ•ญ๋ชฉ์— ๐Ÿšจ prepend) - **๋ชจ๋ธ**: gpt-4o-mini (์˜์‚ฌ๊ฒฐ์ • ์†Œ์ž‘์—…, ๋น„์šฉ ์ ˆ๊ฐ) - **LangGraph ์‹œ๊ฐํ™”**: 3๊ฐœ ๋ถ„๊ธฐ ๋…ธ๋“œ (detect, cause, supervisor) - ์ •์  + LLM-driven ๋ผ์šฐํŒ…์˜ ์กฐํ•ฉ - **smoke test**: A1 (medium severity, 3 causes) โ†’ proceed_full, A2/A3 (high severity) โ†’ proceed_full. ํ˜„ ๋ฐ์ดํ„ฐ์—์„  ๋ชจ๋‘ ํ‘œ์ค€ ๊ฒฝ๋กœ ์„ ํƒ (์•ˆ์ „ํ•œ ๊ธฐ๋ณธ) ### 9. LangSmith observability ํ†ตํ•ฉ - **๋ชฉ์ **: production-grade ํŠธ๋ ˆ์ด์Šค ๋Œ€์‹œ๋ณด๋“œ. ์•Œ๋žŒ๋ณ„ LLM ํ˜ธ์ถœ ํŠธ๋ฆฌ, tool ํ˜ธ์ถœ ์‹œํ€€์Šค, latencyยทtoken ๋ถ„์„ - **๊ตฌํ˜„**: `wrap_openai`๋กœ ๋ชจ๋“  chat.completions.create ์ž๋™ ํŠธ๋ ˆ์ด์Šค, `@traceable` ๋ฐ์ฝ”๋ ˆ์ดํ„ฐ๋กœ 4-Tier agent + Supervisor + tool dispatcher๋ฅผ nested run์œผ๋กœ ์‹œ๊ฐํ™” - **ํ† ๊ธ€**: ํ™˜๊ฒฝ๋ณ€์ˆ˜ `LANGSMITH_TRACING=true/false`๋กœ on/off, ๋น„ํ™œ์„ฑ ์‹œ no-op (์„ฑ๋Šฅยท๊ธฐ๋Šฅ ์˜ํ–ฅ ์—†์Œ) - **ํฌํŠธํด๋ฆฌ์˜ค ๊ฐ€์น˜**: ์ธํ„ฐ๋ทฐ์—์„œ "๊ฐ LLM ํ˜ธ์ถœยทtool ํ˜ธ์ถœยทtokenยทlatency ๋‹ค ๋ณด์ž…๋‹ˆ๋‹ค" ํ•œ ์ค„๋กœ production-grade ์ธ์ƒ ### 10. ์ฝ”ํผ์Šค 12 โ†’ 34 ํ™•์žฅ + D10์œผ๋กœ ๊ฐ€์„ค ๊ฒ€์ฆ - **์‹œ์ž‘**: D9๊นŒ์ง€ ๋ˆ„์ ๋œ ๊ฐ€์„ค "์ฝ”ํผ์Šค ๊ทœ๋ชจ๊ฐ€ reranker ํšจ์šฉ์˜ ์„ ๊ฒฐ์กฐ๊ฑด". ์ด๋ฅผ ์ •๋Ÿ‰ ๊ฒ€์ฆํ•˜๋ ค๋ฉด ์ฝ”ํผ์Šค ํ™•์žฅ ํ•„์ˆ˜ - **๋ฐฉ๋ฒ•**: ํ•ฉ๋ฒ•์  ๊ณต๊ฐœ ์ž๋ฃŒ๋กœ 12๊ฐœ โ†’ 34๊ฐœ ํ™•์žฅ - ํ•œ๊ตญ์–ด ์œ„ํ‚ค๋ฐฑ๊ณผ 12๊ฐœ (๋ฐ˜๋„์ฒด ๊ณต์ • ์ „๋ฐ˜: Photo/Etch/CMP/์ด์˜จ์ฃผ์ž…/๋ฐ•๋ง‰/ํฌํ† ๋ ˆ์ง€์ŠคํŠธ/EUV ๋“ฑ) - SKํ•˜์ด๋‹‰์Šค ๋‰ด์Šค๋ฃธ 6๊ฐœ (์‹ค์ œ ์‚ฐ์—… ์šด์˜ ๊ด€์ : ์‹๊ฐยทํฌํ† ยทCMPยท์„ธ์ •) - ์‚ผ์„ฑ๋ฐ˜๋„์ฒด ๊ณต์‹ 3๊ฐœ (8๋Œ€ ๊ณต์ •ยท์šฉ์–ด์ง‘ยทEUV) - SKC ์†Œ์žฌ 1๊ฐœ, PHM Society 2016 ์ฑŒ๋ฆฐ์ง€ 1๊ฐœ - ๋ชจ๋“  ์ถœ์ฒ˜๋Š” CC BY-SA ๋˜๋Š” ๊ณต๊ฐœ ์ž๋ฃŒ, ํŒŒ์ผ๋ณ„ attribution ๋ช…์‹œ - **D10 ๊ฒฐ๊ณผ (re-run D9 with 34 docs)**: - hybrid: 0.734 โ†’ **0.592** (-0.142, noise ์ฆ๊ฐ€) - BAAI: -0.020 โ†’ **+0.117** (๋ฐ˜์ „!) - ko-reranker: -0.031 โ†’ **+0.083** (๋ฐ˜์ „!) - **๊ฒ€์ฆ ์™„๋ฃŒ**: ๊ฐ€์„ค ์ •ํ™•ํžˆ ์ž…์ฆ. ํ™•์žฅ ์ฝ”ํผ์Šค์—์„œ๋Š” BM25/FAISS๊ฐ€ ์ผ๋ฐ˜ ์ž๋ฃŒ noise๋ฅผ ํก์ˆ˜ํ•˜์ง€๋งŒ cross-encoder๊ฐ€ ๊ทธ์ค‘์—์„œ ์ •๋‹ต์„ ๊ณจ๋ผ๋ƒ„ - **๊ฒฐ์ •**: ์ฝ”ํผ์Šค 30+ ํ™˜๊ฒฝ์—์„œ๋Š” `RAG_BACKEND=hybrid_rerank` ๊ถŒ์žฅ. ๋ฐ๋ชจ์šฉ 12๊ฐœ์—์„  hybrid ์œ ์ง€ - **์‹œ๋ฆฌ์ฆˆ ์˜์˜**: D6 โ†’ D9 โ†’ D10์ด portfolio narrative๋กœ ์™„์„ฑ. "ํ†ต๋… โ†’ ์ •๋Ÿ‰ ๋ฐ˜๋ฐ• โ†’ ๊ฐ€์„ค โ†’ ์ •๋Ÿ‰ ๊ฒ€์ฆ"์˜ ์‚ฌ์ดํด์ด ์ •๋Ÿ‰ ํ‰๊ฐ€์˜ ๊ฐ€์น˜ ์ž์ฒด๋ฅผ ์ฆ๋ช… ### 11. Conductor ํŒจํ„ด (Plan-and-Execute) ๋„์ž… - "agentic โ†’ conductor" ์ „ํ™˜ - **์ƒํ™ฉ**: 4-tier agentic ์‹œ์Šคํ…œ(D7) ๋„์ž… ํ›„ ์•Œ๋žŒ๋‹น ์•ฝ 3๋ถ„(194์ดˆ)ยทLLM 10~13ํšŒ๋กœ ๋ฐ๋ชจ UX ์ €ํ•˜. ์ฝ”ํผ์Šค ํ™•์žฅ(D10)์œผ๋กœ ์ปจํ…์ŠคํŠธ ๋” ๋ฌด๊ฑฐ์›Œ์ง - **์ง„๋‹จ**: Anthropic์˜ [Building Effective Agents](https://www.anthropic.com/engineering/building-effective-agents)๊ฐ€ ๋ช…์‹œํ•˜๋“ฏ "autonomous" ํŒจํ„ด์€ ์ ์‘์„ฑ์ด ๊ฐ•์ ์ด์ง€๋งŒ ํ†ต์‹  ์˜ค๋ฒ„ํ—ค๋“œ(iteration ๋ˆ„์ ยท์žฌ๊ท€ ํ˜ธ์ถœ ์œ„ํ—˜) ํผ - **์ „ํ™˜**: Plan-and-Execute ํŒจํ„ด - Central Planner Agent๊ฐ€ ์•Œ๋žŒ+Tier 1์„ ๋ณด๊ณ  ์ „์ฒด ์›Œํฌํ”Œ๋กœ์šฐ(Tier 2/3/4 ๊ฐ tool ํ˜ธ์ถœ plan)๋ฅผ 1ํšŒ ์‚ฐ์ถœ, ๊ฐ Tier executor๋Š” plan๋Œ€๋กœ tool ์ง์ ‘ ์‹คํ–‰ + LLM 1ํšŒ synthesis - **๊ตฌํ˜„**: `agents/planner.py` ์‹ ๊ทœ + `cause/impact/response.py`์— conductor ๊ฒฝ๋กœ ์ถ”๊ฐ€ (autonomous๋Š” ์˜ต์…˜ ์œ ์ง€). ํ™˜๊ฒฝ๋ณ€์ˆ˜ `AGENT_MODE=conductor` (๊ธฐ๋ณธ) / `autonomous`๋กœ ํ† ๊ธ€ - **D11 ์ •๋Ÿ‰ ๊ฒฐ๊ณผ (3 ์•Œ๋žŒ, CRAG OFF ๋™์ผ ์กฐ๊ฑด)**: - LLM ํ˜ธ์ถœ: 10.0 โ†’ 4.0 (**-60%**, Planner 1 + Tierร—3 synthesis) - Latency: 131์ดˆ โ†’ **60์ดˆ** (**-54%**) - ๋น„์šฉ: $33.23 โ†’ $13.80 / 1000์•Œ๋žŒ (**-58%**) - ์ธ์šฉ ๊นŠ์ด: 6.0 โ†’ 6.0 (**๋™๋“ฑ**, ํ’ˆ์งˆ ์†์‹ค ์—†์Œ) - **ํ•ต์‹ฌ narrative**: D7(workflow โ†’ agentic์œผ๋กœ ์ž์œจ์„ฑ ํ™•๋ณด) โ†’ D11(agentic โ†’ conductor๋กœ ํšจ์œจ ํšŒ๋ณต). ๋‘ ๋‹จ๊ณ„ ๋ชจ๋‘ ์ •๋Ÿ‰ ๋ฐ์ดํ„ฐ ๊ธฐ๋ฐ˜ ์˜์‚ฌ๊ฒฐ์ •. **autonomous์˜ ์ž์œจ์„ฑ vs conductor์˜ ํšจ์œจ์„ฑ์„ trade-off๋กœ ๋ช…์‹œ ์ฑ„ํƒ** - **์žฌ๊ท€ ์œ„ํ—˜ ์›์ฒœ ์ฐจ๋‹จ**: ๊ธฐ์กด `MAX_TOOL_ITERATIONS=4` ์บก์— ์˜์กดํ•˜๋˜ ๋ฌดํ•œ๋ฃจํ”„ ๋ฐฉ์ง€๊ฐ€ plan ๊ณ ์ • ์‹คํ–‰์œผ๋กœ ๋ณธ์งˆ์ ์œผ๋กœ ํ•ด๊ฒฐ ## ์‹คํ–‰ ### ๋กœ์ปฌ ```bash pip install -r requirements.txt cp .env.example .env # OPENAI_API_KEY ์ž…๋ ฅ streamlit run app.py --server.port 8501 ``` PHM 2016 CMP ์บ์‹œ(`data/phm2016/phm_cmp_features.csv`)๋Š” ์ €์žฅ์†Œ์— ํฌํ•จ๋˜์–ด ๋ณ„๋„ ๋‹ค์šด๋กœ๋“œ๊ฐ€ ํ•„์š” ์—†์Šต๋‹ˆ๋‹ค. raw trajectory ๋ฐ์ดํ„ฐ๋ฅผ ์ง์ ‘ ๋‹ค๋ฃจ๋ ค๋ฉด `data/phm2016/README.md` ์ฐธ๊ณ . ### RAG ๋ฐฑ์—”๋“œ ์ „ํ™˜ ```bash RAG_BACKEND=hybrid streamlit run app.py # ๊ธฐ๋ณธ๊ฐ’ (์‹ค์ธก ๋ฐ์ดํ„ฐ ๊ทผ๊ฑฐ ์ฑ„ํƒ) RAG_BACKEND=hybrid_rerank streamlit run app.py # ์˜ต์…˜: ์ฝ”ํผ์Šค ํ™•์žฅ ์‹œ RAG_BACKEND=faiss streamlit run app.py # ์˜ต์…˜: ์˜๋ฏธ ์œ„์ฃผ RAG_BACKEND=keyword streamlit run app.py # ์˜ต์…˜: ์˜์กด์„ฑ ์ตœ์†Œ ``` ### Agent ๋ชจ๋“œ ํ† ๊ธ€ (D11) ```bash AGENT_MODE=conductor streamlit run app.py # ๊ธฐ๋ณธ - Plan-and-Execute (๋น ๋ฅด๊ณ  ์ €๋ ด, ํ†ต์‹  ์ตœ์†Œ) AGENT_MODE=autonomous streamlit run app.py # ์˜ต์…˜ - tool-using agent loop (์ ์‘์„ฑ ์šฐ์œ„) ``` ### CRAG (Self-correction) ํ† ๊ธ€ ```bash CRAG_ENABLED=true streamlit run app.py # ๊ธฐ๋ณธ๊ฐ’ - retrieval grader + ์ž๋™ refinement CRAG_ENABLED=false streamlit run app.py # ๋น„ํ™œ์„ฑ - latency critical ์‹œ๋‚˜๋ฆฌ์˜ค ``` ### LangSmith Observability (์„ ํƒ) ๋ชจ๋“  LLMยทtoolยทagent ํ˜ธ์ถœ์ด [LangSmith](https://smith.langchain.com) ๋Œ€์‹œ๋ณด๋“œ๋กœ ์ž๋™ ์ „์†ก๋ฉ๋‹ˆ๋‹ค. ```bash # .env ์— ์ถ”๊ฐ€ (https://smith.langchain.com ์—์„œ API ํ‚ค ๋ฐœ๊ธ‰) LANGSMITH_TRACING=true LANGSMITH_API_KEY=lsv2_pt_... LANGSMITH_PROJECT=fabagent ``` ํ™œ์„ฑ ์‹œ: - `wrap_openai`๊ฐ€ ๋ชจ๋“  `chat.completions.create` ํ˜ธ์ถœ์„ ์ž๋™ ํŠธ๋ ˆ์ด์Šค (Tier 2/3/4 agent, CRAG grader, RAGAS ํ‰๊ฐ€ ๋“ฑ) - `@traceable` ๋ฐ์ฝ”๋ ˆ์ดํ„ฐ๋กœ 4-Tier orchestratorยท๊ฐ agentยทtool dispatcher๋ฅผ nested run์œผ๋กœ ์‹œ๊ฐํ™” - production observability: ์•Œ๋žŒ๋ณ„ latency / ๋น„์šฉ / tool ํ˜ธ์ถœ ํŠธ๋ฆฌ / error ์ถ”์  ### Hugging Face Spaces ๋ฐฐํฌ 1. HF Space ์ƒ์„ฑ (SDK: Streamlit) 2. ๋ณธ ์ €์žฅ์†Œ ์—ฐ๊ฒฐ (๋˜๋Š” push) 3. Space settings โ†’ Variables and secrets โ†’ `OPENAI_API_KEY` ์ถ”๊ฐ€ 4. ์ž๋™ ๋นŒ๋“œยท๋ฐฐํฌ ์ƒ๋‹จ frontmatter๊ฐ€ HF Space ์„ค์ •์œผ๋กœ ์ž๋™ ์ธ์‹๋ฉ๋‹ˆ๋‹ค. ## ํŒŒ์ผ ๊ตฌ์กฐ ``` fabagent/ โ”œโ”€โ”€ app.py # Streamlit ์—”ํŠธ๋ฆฌํฌ์ธํŠธ โ”œโ”€โ”€ components/ # UI ์ปดํฌ๋„ŒํŠธ (์‚ฌ์ด๋“œ๋ฐ”ยทํ—ค๋”ยทTier ์นด๋“œยทcascade) โ”œโ”€โ”€ core/ โ”‚ โ”œโ”€โ”€ schema.py # Tier1~4 TypedDict ๊ณ„์•ฝ โ”‚ โ””โ”€โ”€ pipeline.py # ์•Œ๋žŒ โ†’ Tier ๋ฐ์ดํ„ฐ ๋ผ์šฐํ„ฐ โ”œโ”€โ”€ agents/ โ”‚ โ”œโ”€โ”€ orchestrator.py # LangGraph StateGraph + ์กฐ๊ฑด๋ถ€ ๋ผ์šฐํŒ… โ”‚ โ”œโ”€โ”€ detection.py # Tier 1 IsolationForest (SECOM/PHM ๋””์ŠคํŒจ์น˜) โ”‚ โ”œโ”€โ”€ planner.py # Central Planner Agent (Plan-and-Execute, conductor ๋ชจ๋“œ) โ”‚ โ”œโ”€โ”€ cause.py # Tier 2 (conductor: plan ๋ฐ›์Œ / autonomous: tool loop) โ”‚ โ”œโ”€โ”€ impact.py # Tier 3 (conductor / autonomous ์–‘ ๋ชจ๋“œ) โ”‚ โ”œโ”€โ”€ response.py # Tier 4 (conductor / autonomous ์–‘ ๋ชจ๋“œ) โ”‚ โ”œโ”€โ”€ supervisor.py # autonomous ๋ชจ๋“œ ์ „์šฉ LLM-driven router โ”‚ โ”œโ”€โ”€ llm.py # OpenAI ํด๋ผ์ด์–ธํŠธ + LangSmith wrap_openai ํ†ตํ•ฉ โ”‚ โ”œโ”€โ”€ tools/ # 7๊ฐœ agent ๋„๊ตฌ โ”‚ โ”‚ โ”œโ”€โ”€ knowledge.py # search_knowledge (RAG ๊ฒ€์ƒ‰) โ”‚ โ”‚ โ”œโ”€โ”€ incident.py # lookup_incident_history โ”‚ โ”‚ โ”œโ”€โ”€ equipment.py # get_pm_history, check_pm_schedule โ”‚ โ”‚ โ””โ”€โ”€ process.py # query_wip_status, get_downstream_steps, get_yield_baseline โ”‚ โ””โ”€โ”€ rag/ โ”‚ โ”œโ”€โ”€ store.py # ๋ฐฑ์—”๋“œ dispatch (keyword/faiss/hybrid/hybrid_rerank) โ”‚ โ”œโ”€โ”€ faiss_store.py # FAISS ๋ฒกํ„ฐ ๊ฒ€์ƒ‰ โ”‚ โ”œโ”€โ”€ hybrid_store.py # BM25 + FAISS + Reciprocal Rank Fusion โ”‚ โ”œโ”€โ”€ rerank.py # Cross-encoder ์žฌ์ •๋ ฌ (BAAI/bge-reranker-base) โ”‚ โ”œโ”€โ”€ crag.py # CRAG self-correction (grader + query refiner) โ”‚ โ”œโ”€โ”€ learn.py # ์ž๊ฐ€ ํ•™์Šต ๋ฃจํ”„ (INC-AUTO-*.md ์ž๋™ ๊ธฐ๋ก) โ”‚ โ””โ”€โ”€ knowledge/ # ๋„๋ฉ”์ธ ๋ฌธ์„œ (INC/FMEA/SOP/FLOW) โ”œโ”€โ”€ data/ โ”‚ โ”œโ”€โ”€ demo.py # ์•Œ๋žŒ ์ •์˜ โ”‚ โ”œโ”€โ”€ wip.py # ์˜ํ–ฅ WIP ๊ฒฐ์ •๋ก  ๋ฐ์ดํ„ฐ โ”‚ โ”œโ”€โ”€ secom/ # SECOM ๋กœ๋” + ์ „์ฒ˜๋ฆฌ + raw .data โ”‚ โ””โ”€โ”€ phm2016/ # PHM 2016 CMP ๋กœ๋” + ์‚ฌ์ „ ์ง‘๊ณ„ CSV โ”œโ”€โ”€ experiments/ # ์ •๋Ÿ‰ ๋น„๊ต ์‹คํ—˜ + ์ฐจํŠธ โ”‚ โ”œโ”€โ”€ tier1_detection/ # D1: IsoForest / LOF / OC-SVM โ”‚ โ”œโ”€โ”€ retrieval_compare/ # D2: keyword / FAISS / hybrid / +rerank โ”‚ โ”œโ”€โ”€ multi_vs_single/ # D5: multi-agent vs single LLM โ”‚ โ”œโ”€โ”€ rag_eval/ # RAGAS ํ‰๊ฐ€ (hybrid vs hybrid_rerank) โ”‚ โ”œโ”€โ”€ rag_paradigm/ # D6: 5๋‹จ๊ณ„ paradigm ablation โ”‚ โ”œโ”€โ”€ agentic_vs_workflow/ # D7: workflow vs agentic ๋น„๊ต โ”‚ โ”œโ”€โ”€ crag_eval/ # D8: CRAG self-correction ํšจ๊ณผ ํ‰๊ฐ€ โ”‚ โ”œโ”€โ”€ reranker_compare/ # D9ยทD10: ํ•œ๊ตญ์–ด reranker + ํ™•์žฅ ์ฝ”ํผ์Šค ๊ฐ€์„ค ๊ฒ€์ฆ โ”‚ โ””โ”€โ”€ conductor_vs_autonomous/ # D11: Conductor vs Autonomous (latency -54%, ์ธ์šฉ ๋™๋“ฑ) โ”œโ”€โ”€ docs/orchestrator_graph.mmd # LangGraph ์ž๋™ ์ถ”์ถœ mermaid โ”œโ”€โ”€ styles/main.css # ๋””์ž์ธ ์‹œ์Šคํ…œ โ””โ”€โ”€ tests/ ``` ## ๊ธฐ์ˆ  ์Šคํƒ - **ํ”„๋ก ํŠธ**: Streamlit 1.36+ - **๋ฐฑ์—”๋“œ**: OpenAI SDK (`gpt-5-mini`, structured output + function calling) - **Orchestration**: LangGraph StateGraph (์กฐ๊ฑด๋ถ€ + LLM-driven ๋ผ์šฐํŒ…, mermaid ์ถ”์ถœ) - **Observability**: LangSmith (`wrap_openai` + `@traceable`, ์˜ต์…˜) - **ML**: scikit-learn (IsolationForest, LOF, OC-SVM) - **RAG**: rank-bm25 + sentence-transformers + FAISS + RRF (์˜ต์…˜: cross-encoder rerank) - **ํ‰๊ฐ€**: RAGAS (faithfulness, answer_relevancy, context_precision) with gpt-4o-mini - **๋ฐ์ดํ„ฐ**: pandas, UCI SECOM, PHM 2016 CMP Data Challenge - **Python**: 3.11+ ## ํ•œ๊ณ„์™€ ํ–ฅํ›„ ํ™•์žฅ - **SECOM์˜ ์ต๋ช…์„ฑ**: 590๊ฐœ ์„ผ์„œ๊ฐ€ ์–ด๋А ๊ณต์ •ยท๋ฌผ๋ฆฌ๋Ÿ‰์ธ์ง€ ๋น„๊ณต๊ฐœ๋ผ A1/A2์˜ step ๋ผ๋ฒจ์€ ์‹œ์—ฐ์šฉ narrative - **knowledge ๋ฌธ์„œ**: ํ•ฉ์„ฑ 12๊ฐœ + ๊ณต๊ฐœ ์ž๋ฃŒ(์œ„ํ‚ค/SKํ•˜์ด๋‹‰์Šค/์‚ผ์„ฑ/SKC/PHM) 22๊ฐœ = **์ด 34๊ฐœ**. ์‹ค fab ์ˆ˜์ฒœ ๋ฌธ์„œ ๋Œ€๋น„ ์—ฌ์ „ํžˆ ์ž‘์ง€๋งŒ, D10์—์„œ ์ฝ”ํผ์Šค ๊ทœ๋ชจ์™€ reranker ํšจ์šฉ์˜ ๊ด€๊ณ„๋Š” ์ •๋Ÿ‰ ๊ฒ€์ฆ๋จ - **๋„๊ตฌ mock data**: PM ์ด๋ ฅยทyield baselineยทdownstream ์˜์กด์„ฑ์€ in-memory mock (์‹ค fab์€ MES/EAP/YMS ์–ด๋Œ‘ํ„ฐ๋กœ ๊ต์ฒด) - **ํ•œ๊ตญ์–ด reranker ๊ฒ€์ฆ ์™„๋ฃŒ(D9)**: hybrid ๋‹จ๋…์— ๋ฏธ๋‹ฌ, ์ฝ”ํผ์Šค ํ™•์žฅ์ด reranker ํšจ์šฉ์˜ ์„ ๊ฒฐ์กฐ๊ฑด์ž„์„ ํ™•์ธ - **Supervisor fast_track ์‹œ์—ฐ ๋ถ€์žฌ**: ํ˜„ 3๊ฐœ ๋ฐ๋ชจ ์•Œ๋žŒ์€ ๋ชจ๋‘ proceed_full ์„ ํƒ - fast_track/escalate ์‹œ์—ฐ์„ ์œ„ํ•ด์„  ๋” ๋‹ค์–‘ํ•œ ์•Œ๋žŒ ์‹œ๋‚˜๋ฆฌ์˜ค ํ•„์š” - **๊ณ ๋„ํ™” ๋ฐฉํ–ฅ**: GraphRAG(๊ณต์ • ์˜์กด์„ฑ ๋…ธ๋“œ ๊ทธ๋ž˜ํ”„) ยท ์‹ค MES/EAP/YMS ์–ด๋Œ‘ํ„ฐ ยท ํ•œ๊ตญ์–ด reranker ์ „์šฉ fine-tuning ยท ๋ฉ€ํ‹ฐ ์—์ด์ „ํŠธ negotiation (ํ˜„์žฌ๋Š” supervisor๊ฐ€ ์ผ๋ฐฉํ–ฅ ๋ผ์šฐํŒ…)