StockEx / StockEx_Developer_Guide.html
RayMelius's picture
Initial commit: StockEx trading platform
9e5fa5b
Raw
History Blame
30.5 kB
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>StockEx - Developer Guide</title>
<style>
@page { margin: 1.5cm; size: A4; }
body { font-family: 'Segoe UI', Arial, sans-serif; max-width: 900px; margin: 0 auto; padding: 20px; line-height: 1.5; color: #333; font-size: 11pt; }
h1 { color: #1a1a2e; border-bottom: 3px solid #4CAF50; padding-bottom: 10px; font-size: 24pt; }
h2 { color: #2e7d32; margin-top: 25px; border-bottom: 1px solid #ddd; padding-bottom: 5px; font-size: 16pt; page-break-after: avoid; }
h3 { color: #1565c0; margin-top: 18px; font-size: 13pt; }
h4 { color: #555; margin-top: 12px; font-size: 11pt; }
table { width: 100%; border-collapse: collapse; margin: 12px 0; font-size: 10pt; }
th, td { border: 1px solid #ddd; padding: 8px; text-align: left; }
th { background: #f5f5f5; font-weight: bold; }
tr:nth-child(even) { background: #fafafa; }
.header { text-align: center; margin-bottom: 25px; }
.subtitle { color: #666; font-size: 14pt; }
.version { color: #999; font-size: 10pt; }
.section { page-break-inside: avoid; margin-bottom: 20px; }
.highlight { background: #e8f5e9; padding: 12px; border-radius: 5px; margin: 10px 0; border-left: 4px solid #4CAF50; }
.info { background: #e3f2fd; padding: 12px; border-radius: 5px; margin: 10px 0; border-left: 4px solid #2196F3; }
.warning { background: #fff3e0; padding: 12px; border-radius: 5px; margin: 10px 0; border-left: 4px solid #ff9800; }
.code-block { background: #263238; color: #aed581; padding: 15px; border-radius: 5px; font-family: 'Consolas', monospace; font-size: 10pt; overflow-x: auto; margin: 10px 0; white-space: pre; }
.json { background: #37474f; color: #80cbc4; }
code { background: #f5f5f5; padding: 2px 6px; border-radius: 3px; font-family: 'Consolas', monospace; font-size: 10pt; }
.green { color: #2e7d32; }
.red { color: #c62828; }
.blue { color: #1565c0; }
hr { border: none; border-top: 1px solid #ddd; margin: 25px 0; }
.footer { text-align: center; color: #666; font-size: 9pt; margin-top: 30px; padding-top: 15px; border-top: 1px solid #ddd; }
.toc { background: #fafafa; padding: 15px; border-radius: 5px; margin: 15px 0; columns: 2; }
.toc ul { margin: 0; padding-left: 20px; }
.toc li { margin: 4px 0; font-size: 10pt; }
.arch-diagram { background: #f5f5f5; padding: 15px; border-radius: 5px; font-family: 'Consolas', monospace; white-space: pre; font-size: 9pt; overflow-x: auto; line-height: 1.3; }
.module-box { border: 1px solid #ddd; border-radius: 6px; padding: 12px; margin: 12px 0; background: #fafafa; page-break-inside: avoid; }
.module-box h4 { margin: 0 0 8px 0; color: #1a1a2e; border-bottom: 1px solid #eee; padding-bottom: 5px; }
.port { display: inline-block; background: #e3f2fd; padding: 2px 8px; border-radius: 3px; font-family: monospace; font-size: 9pt; margin-right: 5px; }
.tech { display: inline-block; background: #f3e5f5; padding: 2px 8px; border-radius: 3px; font-size: 9pt; margin-right: 5px; }
.endpoint { font-family: monospace; background: #e8f5e9; padding: 2px 6px; border-radius: 3px; }
.two-col { display: flex; gap: 20px; }
.two-col > div { flex: 1; }
.screenshot { text-align: center; margin: 15px 0; }
.screenshot img { max-width: 100%; border: 1px solid #ddd; border-radius: 5px; }
</style>
</head>
<body>
<div class="header">
<h1>StockEx Trading Platform</h1>
<p class="subtitle">Developer & Technical Guide</p>
<p class="version">Version 1.0 | Euronext OPTIQ Inspired</p>
</div>
<div class="toc">
<strong>Contents</strong>
<ul>
<li>1. Overview</li>
<li>2. Architecture</li>
<li>3. Modules</li>
<li>4. Database Persistence</li>
<li>5. Data Flow Diagrams</li>
<li>6. Kafka Topics</li>
<li>7. Message Formats</li>
<li>8. SSE Events</li>
<li>9. Configuration</li>
<li>10. Development</li>
</ul>
</div>
<hr>
<div class="section">
<h2>1. Overview</h2>
<p><strong>StockEx</strong> is a real-time trading simulation platform providing complete order-to-trade lifecycle emulation. Built with microservices architecture using Docker containers.</p>
<div class="two-col">
<div class="highlight">
<strong>Core Features</strong>
<ul style="margin:5px 0; padding-left:20px;">
<li>FIX 4.4 protocol gateway</li>
<li>Price-time priority matching</li>
<li>Kafka event streaming</li>
<li>SSE real-time dashboard</li>
<li>SQLite persistence</li>
</ul>
</div>
<div class="info">
<strong>Tech Stack</strong>
<ul style="margin:5px 0; padding-left:20px;">
<li>Python 3.11 / Flask</li>
<li>QuickFIX/Python</li>
<li>Apache Kafka 7.5</li>
<li>Docker Compose</li>
<li>SQLite</li>
</ul>
</div>
</div>
</div>
<div class="section">
<h2>2. System Architecture</h2>
<div class="arch-diagram">
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ FIX UI Client │ │ FIX UI Client │ │ Frontend │
│ (Port 5002) │ │ (Port 5003) │ │ (Port 5000) │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ FIX 4.4 │ FIX 4.4 │ HTTP/JSON
└───────────────┬───────┴───────────────────────┘
┌───────────────────────────────┐
│ FIX OEG (Port 5001) │
│ QuickFIX Order Gateway │
└───────────────┬───────────────┘
│ JSON
┌────────────────────────────────────────────────────────────────────┐
│ APACHE KAFKA (Port 9092) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ orders │ │ trades │ │ snapshots│ │
│ └──────────┘ └──────────┘ └──────────┘ │
└─────┬─────────────────┬─────────────────┬─────────────────────────┘
│ │ │
▼ │ ▼
┌─────────────┐ │ ┌─────────────┐
│ Matcher │─────────┘ │ MD Feeder │
│ (Port 6000) │ │ (MDF) │
│ │ │ │
│ Order Book │◄──────────────────│ Price Sim │
│ Trade Exec │ │ BBO Publish │
└─────────────┘ └─────────────┘
▼ REST API
┌─────────────────────────────────────────┐
│ Dashboard (Port 5005) │
│ Orders │ Trades │ Book │ Chart │ Stats │
└─────────────────────────────────────────┘
</div>
</div>
<div class="section">
<h2>3. Module Specifications</h2>
<div class="module-box">
<h4>Kafka Message Broker</h4>
<span class="port">9092</span> <span class="port">29092 (host)</span> <span class="tech">Confluent 7.5.0</span>
<p>Central event bus. All order flow distributed via topics. Zookeeper (port 2181) for coordination.</p>
</div>
<div class="module-box">
<h4>FIX Order Entry Gateway</h4>
<span class="port">5001</span> <span class="tech">QuickFIX/Python</span>
<p>FIX 4.4 acceptor. Receives NewOrderSingle (D), OrderCancelRequest (F), OrderCancelReplaceRequest (G). Normalizes to JSON → Kafka <code>orders</code>.</p>
</div>
<div class="module-box">
<h4>Matcher Engine</h4>
<span class="port">6000</span> <span class="tech">Python/Flask/SQLite</span>
<p>Consumes <code>orders</code>, matches with price-time priority, publishes <code>trades</code>. Maintains order book per symbol. SQLite persistence.</p>
<table>
<tr><th>Endpoint</th><th>Method</th><th>Description</th></tr>
<tr><td><span class="endpoint">/orderbook/&lt;symbol&gt;</span></td><td>GET</td><td>Order book depth</td></tr>
<tr><td><span class="endpoint">/trades</span></td><td>GET</td><td>Recent trades</td></tr>
<tr><td><span class="endpoint">/health</span></td><td>GET</td><td>Health + stats</td></tr>
</table>
</div>
<div class="module-box">
<h4>Market Data Feeder (MDF)</h4>
<span class="tech">Python</span>
<p>Simulates market activity. 90% passive orders (book building), 10% aggressive (trades). Publishes BBO snapshots.</p>
<p>Output: <code>orders</code> + <code>snapshots</code> topics</p>
</div>
<div class="module-box">
<h4>Dashboard</h4>
<span class="port">5005</span> <span class="tech">Flask/SSE/JavaScript</span>
<p>Real-time web UI. Consumes Kafka + Matcher API. Server-Sent Events for live streaming. Edit/Cancel order management.</p>
<table>
<tr><th>Endpoint</th><th>Method</th><th>Description</th></tr>
<tr><td><span class="endpoint">/stream</span></td><td>GET</td><td>SSE event stream</td></tr>
<tr><td><span class="endpoint">/data</span></td><td>GET</td><td>Polling fallback</td></tr>
<tr><td><span class="endpoint">/order/cancel</span></td><td>POST</td><td>Cancel order</td></tr>
<tr><td><span class="endpoint">/order/amend</span></td><td>POST</td><td>Amend order</td></tr>
</table>
</div>
<div class="module-box">
<h4>FIX UI Clients</h4>
<span class="port">5002</span> <span class="port">5003</span> <span class="tech">QuickFIX/Flask</span>
<p>Web-based FIX initiators. Connect to FIX OEG for institutional order submission.</p>
</div>
</div>
<div class="section">
<h2>4. Database Persistence</h2>
<div class="info">
<strong>Storage:</strong> SQLite database at <code>/app/data/matcher.db</code><br>
<strong>Docker Volume:</strong> <code>stockex_matcher_data</code> (survives container restarts)
</div>
<h4>4.1 Database Schema</h4>
<div class="module-box">
<h4>order_book — All Orders</h4>
<table>
<tr><th>Column</th><th>Type</th><th>Description</th></tr>
<tr><td><code>id</code></td><td>INTEGER</td><td>Auto-increment primary key</td></tr>
<tr><td><code>cl_ord_id</code></td><td>TEXT</td><td>Unique client order ID</td></tr>
<tr><td><code>symbol</code></td><td>TEXT</td><td>Security (ALPHA, EXAE, etc.)</td></tr>
<tr><td><code>side</code></td><td>TEXT</td><td>BUY / SELL</td></tr>
<tr><td><code>price</code></td><td>REAL</td><td>Limit price</td></tr>
<tr><td><code>quantity</code></td><td>INTEGER</td><td>Original order quantity</td></tr>
<tr><td><code>remaining_qty</code></td><td>INTEGER</td><td>Unfilled quantity</td></tr>
<tr><td><code>status</code></td><td>TEXT</td><td>OPEN / FILLED / CANCELLED</td></tr>
<tr><td><code>timestamp</code></td><td>REAL</td><td>Order entry time (Unix)</td></tr>
<tr><td><code>created_at</code></td><td>DATETIME</td><td>DB insert timestamp</td></tr>
</table>
</div>
<div class="module-box">
<h4>trades — Executed Trades</h4>
<table>
<tr><th>Column</th><th>Type</th><th>Description</th></tr>
<tr><td><code>id</code></td><td>INTEGER</td><td>Auto-increment primary key</td></tr>
<tr><td><code>symbol</code></td><td>TEXT</td><td>Traded security</td></tr>
<tr><td><code>price</code></td><td>REAL</td><td>Execution price</td></tr>
<tr><td><code>quantity</code></td><td>INTEGER</td><td>Traded quantity</td></tr>
<tr><td><code>buy_order_id</code></td><td>TEXT</td><td>Buyer's cl_ord_id</td></tr>
<tr><td><code>sell_order_id</code></td><td>TEXT</td><td>Seller's cl_ord_id</td></tr>
<tr><td><code>timestamp</code></td><td>REAL</td><td>Trade time (Unix)</td></tr>
<tr><td><code>created_at</code></td><td>DATETIME</td><td>DB insert timestamp</td></tr>
</table>
</div>
<h4>4.2 Database Functions</h4>
<table>
<tr><th>Function</th><th>Description</th></tr>
<tr><td><code>save_order(order)</code></td><td>Insert new order into order_book</td></tr>
<tr><td><code>update_order_quantity(id, qty)</code></td><td>Update remaining_qty after partial fill</td></tr>
<tr><td><code>cancel_order(cl_ord_id)</code></td><td>Set status = 'CANCELLED'</td></tr>
<tr><td><code>save_trade(trade)</code></td><td>Insert executed trade</td></tr>
<tr><td><code>get_open_orders(symbol, side)</code></td><td>Query open orders for matching</td></tr>
<tr><td><code>load_order_books()</code></td><td>Restore order books on startup</td></tr>
<tr><td><code>get_trades(symbol, limit)</code></td><td>Retrieve recent trades</td></tr>
<tr><td><code>delete_filled_orders(days)</code></td><td>Cleanup old filled/cancelled orders</td></tr>
</table>
<h4>4.3 Indexes</h4>
<div class="code-block">CREATE INDEX idx_trades_symbol ON trades(symbol);
CREATE INDEX idx_trades_timestamp ON trades(timestamp);
CREATE INDEX idx_orderbook_symbol_side ON order_book(symbol, side);
CREATE INDEX idx_orderbook_status ON order_book(status);
CREATE INDEX idx_orderbook_cl_ord_id ON order_book(cl_ord_id);</div>
</div>
<div class="section">
<h2>5. Data Flow Diagrams</h2>
<h4>4.1 Order Entry Flow</h4>
<div class="arch-diagram">
┌──────────────┐
│ FIX Client │
└──────┬───────┘
│ FIX 4.4 NewOrderSingle (35=D)
┌──────────────┐
│ FIX OEG │ Validate → Normalize → Generate cl_ord_id
└──────┬───────┘
│ JSON
┌──────────────┐
│ Kafka │ Topic: orders
│ [orders] │
└──────┬───────┘
┌─────┴─────┐
▼ ▼
┌────────┐ ┌───────────┐
│Matcher │ │ Dashboard │
└────────┘ └───────────┘
</div>
<h4>4.2 Order Matching Flow</h4>
<div class="arch-diagram">
┌─────────────────┐
│ Incoming Order │
│ (from Kafka) │
└────────┬────────┘
┌─────────────────┐
│ Parse & Validate│
└────────┬────────┘
┌──────────────┴──────────────┐
▼ ▼
┌────────────┐ ┌────────────┐
│ BUY Order │ │ SELL Order │
└──────┬─────┘ └──────┬─────┘
│ │
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ Check SELL book │ │ Check BUY book │
│ for price ≤ bid │ │ for price ≥ ask │
└────────┬─────────┘ └────────┬─────────┘
│ │
┌──────┴──────┐ ┌──────┴──────┐
▼ ▼ ▼ ▼
┌───────┐ ┌────────┐ ┌───────┐ ┌────────┐
│ Match │ │No Match│ │ Match │ │No Match│
│ Found │ │ │ │ Found │ │ │
└───┬───┘ └───┬────┘ └───┬───┘ └───┬────┘
│ │ │ │
▼ ▼ ▼ ▼
┌───────┐ ┌────────┐ ┌───────┐ ┌────────┐
│Execute│ │Add to │ │Execute│ │Add to │
│ Trade │ │BUY Book│ │ Trade │ │SELLBook│
└───┬───┘ └────────┘ └───┬───┘ └────────┘
│ │
└───────────┬───────────────┘
┌────────────┐
│Kafka:trades│
└────────────┘
</div>
<h4>4.3 Real-time Dashboard Flow</h4>
<div class="arch-diagram">
┌─────────────────────────────────────────────────────────────────┐
│ BROWSER │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Dashboard UI │ │
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────────────┐│ │
│ │ │ Orders │ │ Trades │ │ Book │ │ Statistics ││ │
│ │ └────▲────┘ └────▲────┘ └────▲────┘ └────────▲────────┘│ │
│ └───────┼───────────┼───────────┼───────────────┼──────────┘ │
│ │ │ │ │ │
│ └───────────┴─────┬─────┴───────────────┘ │
│ │ │
│ ┌───────▼────────┐ │
│ │ EventSource │ SSE Connection │
│ │ /stream │ │
│ └───────┬────────┘ │
└────────────────────────────┼────────────────────────────────────┘
│ HTTP (SSE)
┌────────────────────────────────────────────────────────────────┐
│ DASHBOARD SERVER │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────┐ │
│ │Kafka Consumer│─────▶│ SSE Broadcast│────▶│ Clients │ │
│ │ (orders, │ │ Queue │ │ Queue[] │ │
│ │ trades, │ └──────────────┘ └────────────┘ │
│ │ snapshots) │ │
│ └──────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ REST API │◀────▶│ Matcher │ /orderbook, /trades │
│ │ /data │ │ Proxy │ │
│ └──────────────┘ └──────────────┘ │
└────────────────────────────────────────────────────────────────┘
</div>
<h4>4.4 Complete System Interaction</h4>
<div class="arch-diagram">
┌─────────┐ ┌─────────┐ ┌─────────┐
│FIX Cli 1│ │FIX Cli 2│ │Frontend │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
└─────┬─────┴───────────┘
┌─────────────┐
│ FIX OEG │◄──── FIX 4.4 Protocol
└──────┬──────┘
┌────────────────────────────────────┐
│ KAFKA CLUSTER │
│ ┌────────┐┌────────┐┌──────────┐ │
│ │orders ││trades ││snapshots │ │
│ └───┬────┘└───▲────┘└────▲─────┘ │
└──────┼─────────┼──────────┼───────┘
│ │ │
┌─────┼─────────┼──────────┼─────┐
│ ▼ │ │ │
│ ┌────────┐ │ │ │
│ │MATCHER │────┘ │ │
│ │ │ │ │
│ │ Book │ │ │
│ │ Match │ │ │
│ │ Trade │ │ │
│ └────────┘ │ │
│ ▲ │ │
│ │ REST │ │
│ │ │ │
│ ┌───┴────────────────────┴──┐ │
│ │ DASHBOARD │ │
│ │ (SSE + Kafka Consumer) │ │
│ └───────────────────────────┘ │
│ │
│ ┌───────────────────────────┐ │
│ │ MD FEEDER (MDF) │──┘
│ │ Orders + Snapshots │
│ └───────────────────────────┘
│ DOCKER NETWORK
└────────────────────────────────┘
</div>
<h4>4.5 Message Sequence: New Order to Trade</h4>
<div class="arch-diagram">
FIX Client FIX OEG Kafka Matcher Dashboard
│ │ │ │ │
│──35=D────▶│ │ │ │
│NewOrder │ │ │ │
│ │──JSON────▶│ │ │
│ │ [orders] │ │ │
│ │ │──consume──▶│ │
│ │ │ │ │
│ │ │ │──match() │
│ │ │ │ │
│ │ │◀──trade────│ │
│ │ │ [trades] │ │
│ │ │ │ │
│ │ │──────────────consume───▶│
│ │ │ │ │
│ │ │ │ render()
│ │ │ │ │
</div>
</div>
<div class="section">
<h2>6. Kafka Topics</h2>
<table>
<tr><th>Topic</th><th>Producers</th><th>Consumers</th><th>Content</th></tr>
<tr><td><code>orders</code></td><td>FIX OEG, MDF, Frontend</td><td>Matcher, Dashboard</td><td>New/Cancel/Amend orders</td></tr>
<tr><td><code>trades</code></td><td>Matcher</td><td>Dashboard, Consumer</td><td>Executed trades</td></tr>
<tr><td><code>snapshots</code></td><td>MDF</td><td>Dashboard</td><td>BBO updates</td></tr>
</table>
</div>
<div class="section">
<h2>7. Message Formats</h2>
<h4>Order (New)</h4>
<div class="code-block json">{
"symbol": "ALPHA",
"side": "BUY",
"price": 25.50,
"quantity": 100,
"cl_ord_id": "MDF-1234567890-1",
"timestamp": 1234567890.123,
"source": "MDF"
}</div>
<h4>Order (Cancel)</h4>
<div class="code-block json">{
"type": "cancel",
"orig_cl_ord_id": "MDF-1234567890-1",
"symbol": "ALPHA",
"timestamp": 1234567890.456
}</div>
<h4>Order (Amend)</h4>
<div class="code-block json">{
"type": "amend",
"orig_cl_ord_id": "MDF-1234567890-1",
"cl_ord_id": "amend-1234567890",
"symbol": "ALPHA",
"quantity": 150,
"price": 25.45,
"timestamp": 1234567890.789
}</div>
<h4>Trade</h4>
<div class="code-block json">{
"symbol": "ALPHA",
"price": 25.50,
"quantity": 100,
"buy_order_id": "order-123",
"sell_order_id": "order-456",
"timestamp": 1234567890.123
}</div>
<h4>Snapshot (BBO)</h4>
<div class="code-block json">{
"symbol": "ALPHA",
"best_bid": 25.45,
"best_ask": 25.55,
"bid_size": 500,
"ask_size": 300,
"timestamp": 1234567890.123,
"source": "MDF"
}</div>
</div>
<div class="section">
<h2>8. SSE Events</h2>
<table>
<tr><th>Event</th><th>Data</th><th>Trigger</th></tr>
<tr><td><code>connected</code></td><td>{status}</td><td>Client connects</td></tr>
<tr><td><code>init</code></td><td>{orders, bbos, trades}</td><td>Initial state dump</td></tr>
<tr><td><code>order</code></td><td>Order JSON</td><td>New order received</td></tr>
<tr><td><code>trade</code></td><td>Trade JSON</td><td>Trade executed</td></tr>
<tr><td><code>snapshot</code></td><td>BBO JSON</td><td>Price update</td></tr>
</table>
</div>
<div class="section">
<h2>9. Configuration</h2>
<table>
<tr><th>Variable</th><th>Default</th><th>Description</th></tr>
<tr><td><code>KAFKA_BOOTSTRAP</code></td><td>kafka:9092</td><td>Broker address</td></tr>
<tr><td><code>MATCHER_URL</code></td><td>http://matcher:6000</td><td>Matcher API</td></tr>
<tr><td><code>TICK_SIZE</code></td><td>0.05</td><td>Min price increment</td></tr>
<tr><td><code>ORDERS_PER_MIN</code></td><td>8</td><td>MDF rate</td></tr>
<tr><td><code>KAFKA_RETRIES</code></td><td>30</td><td>Connection retries</td></tr>
</table>
<h4>Securities (shared_data/securities.txt)</h4>
<div class="code-block">#SYMBOL start_price current_price
ALPHA 25.00 25.00
EXAE 42.00 42.00
PEIR 18.50 18.50
QUEST 12.75 12.75</div>
</div>
<div class="section">
<h2>10. Development Commands</h2>
<h4>Build & Run</h4>
<div class="code-block">docker compose up --build # Start all
docker compose up -d --build # Background
docker compose logs -f dashboard # Follow logs
docker compose down # Stop all</div>
<h4>Reset Data</h4>
<div class="code-block">docker compose down
docker volume rm stockex_matcher_data
docker compose up -d</div>
<h4>Container Access</h4>
<div class="code-block">docker exec -it matcher bash
docker exec -it dashboard bash
docker logs matcher --tail 50</div>
<h4>API Testing</h4>
<div class="code-block">curl http://localhost:6000/orderbook/ALPHA
curl http://localhost:6000/trades
curl http://localhost:5005/data</div>
</div>
<div class="screenshot">
<img src="screenshots/dashboard.png" alt="StockEx Dashboard">
<p style="font-size:10pt; color:#666;"><em>StockEx Trading Dashboard - Real-time Market View</em></p>
</div>
<hr>
<div class="footer">
<strong>StockEx Trading Platform v1.0</strong><br>
Developer Guide | Euronext OPTIQ Inspired<br>
<p style="margin-top:10px;">Print to PDF: Ctrl+P → Save as PDF</p>
</div>
</body>
</html>