system_knowledge_base/rag_index.jsonl

{"source": "dataset_generation_rules.md", "chunk_id": 0, "text": "TRADING LLM DATASET GENERATION RULES\n\nThe dataset creator must distinguish between:\nA. Strategy knowledge / documentation datasets.\nB. Numerical market-data datasets.\nC. Image/chart understanding datasets.\nD. Conversational trading-analysis datasets.\nE. Trade-decision approval or denial datasets.\n\nNever mix these formats without explicitly labeling the source type and target\ntraining objective.\n\nA strategy library should preserve semantic structure. A PDF containing a\nstrategy, rulebook, risk model, or trading methodology should be extracted,\ncleaned, separated into coherent chunks, and represented with metadata such as\ndocument name, section, page when available, topic, strategy name, timeframe,\nmarket, and source type.\n\nA numerical dataset must preserve numerical values exactly. Do not replace\nOHLCV or indicator values with vague natural-language summaries when the target\nmodel is expected to learn quantitative relationships. Preserve timestamps,\nsymbol, timeframe, OHLCV, indicator columns, and labels. Rows must be sorted\nchronologically within each symbol/timeframe series.\n\nIf timestamps are missing, the system must not silently invent real-world dates\nthat could be mistaken for historical market data. A generated sequence must be\nexplicitly marked as synthetic/generated and must preserve row order.\n\nEvery training example should have a clearly defined target objective. Examples:\nmarket-state classification, setup detection, trade-plan generation, trade\napproval/denial, chart captioning, strategy explanation, or multimodal chart\nanalysis.\n\nAvoid look-ahead leakage. Features used at decision time must not contain future\ninformation. If a label uses future candles, the label horizon and future window\nmust be explicitly documented in metadata.\n\nThe system should preserve provenance. Each output row should identify the\nsource file and dataset type.", "terms": ["ahead", "analysis", "and", "approval", "are", "available", "avoid", "between", "candles", "captioning", "chart", "chronologically", "chunks", "classification", "cleaned", "clearly", "coherent", "columns", "contain", "containing", "conversational", "could", "creator", "data", "dataset", "datasets", "dates", "decision", "defined", "denial", "detection", "distinguish", "document", "documentation", "documented", "each", "every", "exactly", "example", "examples", "expected", "explanation", "explicitly", "extracted", "features", "file", "for", "formats", "future", "generated", "generation", "have", "historical", "horizon", "identify", "image", "indicator", "information", "into", "invent", "knowledge", "label", "labeling", "labels", "language", "leakage", "learn", "library", "llm", "look", "marked", "market", "metadata", "methodology", "missing", "mistaken", "mix", "model", "multimodal", "must", "name", "natural", "never", "not", "numerical", "objective", "ohlcv", "order", "output", "page", "pdf", "plan", "preserve", "provenance", "quantitative", "real", "relationships", "replace", "represented", "risk", "row", "rows", "rulebook", "rules", "section", "semantic", "separated", "sequence", "series", "setup", "should", "silently", "sorted", "source", "state", "strategy", "structure", "such", "summaries", "symbol", "synthetic", "system", "target", "that", "the", "these", "time", "timeframe", "timestamps", "topic", "trade", "trading", "training", "type", "understanding", "used", "uses", "vague", "values", "when", "window", "with", "within", "without", "world"]}
{"source": "image_chart_rules.md", "chunk_id": 0, "text": "TRADING IMAGE AND CHART DATASET RULES\n\nChart screenshots and candlestick images can be used for:\n1. image-to-text captioning,\n2. visual question answering,\n3. chart-analysis conversations,\n4. multimodal supervised fine-tuning.\n\nDo not claim exact price values from a chart image unless they are legible or\nprovided as structured metadata. Do not infer hidden candles or indicators.\n\nA chart-image example should preserve:\nimage path or image identifier,\nsymbol if known,\ntimeframe if known,\nvisible chart context,\nvisual observations,\nmarket structure,\nliquidity observations,\nindicators visible,\nsetup classification,\nentry/stop/target only when provided or clearly inferable,\nand uncertainty notes.\n\nFor a multimodal dataset, the image reference must remain associated with the\ntext response. For a text-only dataset, OCR and/or a vision caption may be\nused, but the generated text must be marked as image-derived.\n\nA chart-analysis response should distinguish observation from inference and\nshould not manufacture certainty.", "terms": ["analysis", "and", "answering", "are", "associated", "but", "can", "candles", "candlestick", "caption", "captioning", "certainty", "chart", "claim", "classification", "clearly", "context", "conversations", "dataset", "derived", "distinguish", "entry", "exact", "example", "fine", "for", "from", "generated", "hidden", "identifier", "image", "images", "indicators", "infer", "inferable", "inference", "known", "legible", "liquidity", "manufacture", "marked", "market", "may", "metadata", "multimodal", "must", "not", "notes", "observation", "observations", "ocr", "only", "path", "preserve", "price", "provided", "question", "reference", "remain", "response", "rules", "screenshots", "setup", "should", "stop", "structure", "structured", "supervised", "symbol", "target", "text", "the", "they", "timeframe", "trading", "tuning", "uncertainty", "unless", "used", "values", "visible", "vision", "visual", "when", "with"]}
{"source": "market_data_rules.md", "chunk_id": 0, "text": "NUMERICAL MARKET DATA RULES\n\nNumerical CSV/XLSX data should be treated as structured time-series data.\n\nRecommended metadata:\nsymbol, timeframe, timestamp, source_file, row_id, dataset_type,\nfeature_columns, label_columns, synthetic_timestamp.\n\nRecognize common columns case-insensitively:\ntimestamp/date/datetime/time, symbol/ticker, open, high, low, close,\nvolume, vwap, atr, rsi, macd, ema, sma, adx, bbands, and other indicators.\n\nDo not round values unless the user explicitly requests rounding.\nDo not convert numeric data into prose as the only representation.\nFor numerical fine-tuning, retain structured fields and optionally add a\nnatural-language market_state field.\n\nRows must be sorted by timestamp. Duplicate timestamps should be preserved\nonly when the dataset has a valid reason, such as multiple symbols or levels.\n\nA generated timestamp must be labeled synthetic_timestamp=true. It is a\nsequence index, not evidence of a real historical date.\n\nThe dataset creator should support labels such as:\nsetup_present, setup_type, market_regime, direction, entry, stop_loss,\ntake_profit, risk_reward, outcome, approval, denial_reason, and confidence.\nLabels must come from the source data or user-provided annotations.", "terms": ["add", "adx", "and", "annotations", "approval", "atr", "bbands", "case", "close", "columns", "come", "common", "confidence", "convert", "creator", "csv", "data", "dataset", "dataset_type", "date", "datetime", "denial_reason", "direction", "duplicate", "ema", "entry", "evidence", "explicitly", "feature_columns", "field", "fields", "fine", "for", "from", "generated", "has", "high", "historical", "index", "indicators", "insensitively", "into", "label_columns", "labeled", "labels", "language", "levels", "low", "macd", "market", "market_regime", "market_state", "metadata", "multiple", "must", "natural", "not", "numeric", "numerical", "only", "open", "optionally", "other", "outcome", "preserved", "prose", "provided", "real", "reason", "recognize", "recommended", "representation", "requests", "retain", "risk_reward", "round", "rounding", "row_id", "rows", "rsi", "rules", "sequence", "series", "setup_present", "setup_type", "should", "sma", "sorted", "source", "source_file", "stop_loss", "structured", "such", "support", "symbol", "symbols", "synthetic_timestamp", "take_profit", "the", "ticker", "time", "timeframe", "timestamp", "timestamps", "treated", "true", "tuning", "unless", "user", "valid", "values", "volume", "vwap", "when", "xlsx"]}
{"source": "strategy_library_rules.md", "chunk_id": 0, "text": "STRATEGY LIBRARY DATASET RULES\n\nStrategy documents are knowledge assets, not ordinary prose blobs. The\nextraction pipeline should preserve:\n- strategy name\n- market/instrument\n- timeframe\n- session\n- setup conditions\n- entry conditions\n- invalidation conditions\n- stop-loss logic\n- take-profit logic\n- position sizing/risk rules\n- confirmation requirements\n- no-trade conditions\n- examples and counterexamples\n\nWhen a document contains rules, the dataset should not invent missing rules.\nWhen a rule is ambiguous, preserve the ambiguity in the source text or flag it\nfor review.\n\nA strategy-library knowledge-base row should be suitable for retrieval and\nshould contain a coherent context chunk rather than an arbitrary cut through a\ntable or rule.\n\nFor conversational fine-tuning, use instruction/input/response or the target\nchat format selected by the user. Do not fabricate a profitable trade outcome.", "terms": ["ambiguity", "ambiguous", "and", "arbitrary", "are", "assets", "base", "blobs", "chat", "chunk", "coherent", "conditions", "confirmation", "contain", "contains", "context", "conversational", "counterexamples", "cut", "dataset", "document", "documents", "entry", "examples", "extraction", "fabricate", "fine", "flag", "for", "format", "input", "instruction", "instrument", "invalidation", "invent", "knowledge", "library", "logic", "loss", "market", "missing", "name", "not", "ordinary", "outcome", "pipeline", "position", "preserve", "profit", "profitable", "prose", "rather", "requirements", "response", "retrieval", "review", "risk", "row", "rule", "rules", "selected", "session", "setup", "should", "sizing", "source", "stop", "strategy", "suitable", "table", "take", "target", "text", "than", "the", "through", "timeframe", "trade", "tuning", "use", "user", "when"]}
{"source": "unsloth_formats.md", "chunk_id": 0, "text": "UNSLOTH DATASET FORMAT RULES\n\nThe dataset creator should support multiple output schemas rather than forcing\nall data into one universal format.\n\nSupported logical dataset types:\n- strategy_knowledge\n- market_time_series\n- chart_image_caption\n- chart_analysis_conversation\n- trade_decision\n- mixed_multimodal\n\nPreferred instruction dataset fields:\ninstruction, input, output, metadata\n\nPreferred conversational field:\nconversations: [{\"from\":\"system\",\"value\":\"...\"},{\"from\":\"human\",\"value\":\"...\"},{\"from\":\"gpt\",\"value\":\"...\"}]\n\nFor multimodal data, retain an image field or image reference according to the\ntraining pipeline selected by the user.\n\nEach row should contain provenance metadata whenever possible. Dataset rows\nmust be valid JSONL and independently parseable.\n\nThe creator should not promise that a dataset is automatically suitable for\nevery Unsloth model. The selected base model, tokenizer, chat template,\nsequence length, and trainer configuration must match the generated schema.", "terms": ["according", "all", "and", "automatically", "base", "chart_analysis_conversation", "chart_image_caption", "chat", "configuration", "contain", "conversational", "conversations", "creator", "data", "dataset", "each", "every", "field", "fields", "for", "forcing", "format", "from", "generated", "gpt", "human", "image", "independently", "input", "instruction", "into", "jsonl", "length", "logical", "market_time_series", "match", "metadata", "mixed_multimodal", "model", "multimodal", "multiple", "must", "not", "one", "output", "parseable", "pipeline", "possible", "preferred", "promise", "provenance", "rather", "reference", "retain", "row", "rows", "rules", "schema", "schemas", "selected", "sequence", "should", "strategy_knowledge", "suitable", "support", "supported", "system", "template", "than", "that", "the", "tokenizer", "trade_decision", "trainer", "training", "types", "universal", "unsloth", "user", "valid", "value", "whenever"]}