docs/ORGANIZATION.md

Flash Attention RTL - Repository Organization

Overview

This document describes the reorganized directory structure of the Flash Attention RTL project after the three-agent optimization and consolidation effort.

Directory Structure

Root Directory (/)

The root directory contains:

• consolidated_agent.nim - Main executable implementing the consolidated SUBLEQ tactical agent
• hybrid_cli.nim - Command-line interface for the hybrid compiler
• refuge.nim - Reference implementation (legacy, from original RTL project)
• subleq_bf.nim (removed, now in /src/)
• Documentation files:
  • README.md - Main project documentation
  • AGENT3_INTEGRATION_REPORT.md - Agent 3 integration summary
  • AGENT3_FINAL_INTEGRATION.md - Final Agent 3 work summary
  • COMPILER_SPEC.md - Compiler specification
  • HYBRID_COMPILER_README.md - Hybrid compiler documentation
  • SIMULATION_SETUP.md - Simulation setup instructions

Source Directory (/src/)

Core compiler and runtime components:

• befunge_codegen.nim - Befunge 2D language code generator
• fa_model.nim - Flash Attention model definitions
• hybrid_ast.nim - Abstract syntax tree definitions
• hybrid_benchmark.nim - Performance benchmarking utilities
• hybrid_codegen.nim - Code generation backend
• hybrid_compiler.nim - Orchestrator for lexer → parser → codegen pipeline
• hybrid_ir.nim - Intermediate representation
• hybrid_lexer.nim - Tokenization for hybrid languages
• hybrid_normalizer.nim - IR normalization
• hybrid_optimizer.nim - Code optimization passes
• hybrid_parser.nim - Parser for hybrid syntax
• parser_pipeline.nim - Complete front-end pipeline
• subleq_bf.nim - SUBLEQ interpreter + Brainfuck transpiler

Tests Directory (/tests/)

Comprehensive test suite:

• test_agent3_conformance.nim - Final conformance test suite
• test_agent3_determinism.nim - Deterministic build verification
• test_agent3_differential.nim - Differential verification (reference vs compiled)
• test_agent3_e2e.nim - End-to-end tests (18+ test programs)
• test_bf_to_subleq_j.nim - Brainfuck transpilation tests
• test_fa_opcode.nim - Flash Attention opcode verification
• test_hybrid_codegen.nim - Code generator tests
• test_hybrid_compiler.nim - Full compiler pipeline tests
• test_hybrid_frontend.nim - Lexer/parser/AST tests

Other Directories

• /cstack/ - Call stack utilities
• /rtl/ - Register transfer level implementations
• /sim/ - Simulation infrastructure and test vectors
• /software/ - Software utilities

Import Paths

Root-level files importing from /src/

import src/module_name

Example (consolidated_agent.nim):

import src/subleq_bf

Test files importing from /src/

import ../src/module_name

Example (tests/test_agent3_e2e.nim):

import ../src/subleq_bf
import ../src/hybrid_compiler

Internal imports within /src/

Modules within /src/ import each other directly (same directory):

import module_name

Example (src/hybrid_compiler.nim):

import hybrid_ast
import hybrid_codegen
import subleq_bf

Building

Compile Root Executable

cd /home/user/flash-attention-rtl
nim c -d:release consolidated_agent.nim
nim c -d:release hybrid_cli.nim

Compile Tests

cd /home/user/flash-attention-rtl/tests
nim c -d:release test_agent3_e2e.nim
nim c -d:release test_hybrid_compiler.nim

Check Compilation (without building)

nim check consolidated_agent.nim
nim check tests/test_agent3_e2e.nim

Module Relationships

Three-Layer Architecture

1. Frontend Layer (/src/hybrid_*.nim):

   • hybrid_lexer.nim - Tokenization
   • hybrid_parser.nim - Parsing
   • hybrid_ast.nim - AST definitions
   • parser_pipeline.nim - Complete pipeline

2. Intermediate Layer (/src/hybrid_*.nim):

   • hybrid_ir.nim - IR representation
   • hybrid_normalizer.nim - IR normalization
   • hybrid_optimizer.nim - Optimization passes

3. Codegen & Runtime (/src/*_codegen.nim, /src/subleq_bf.nim):

   • hybrid_codegen.nim - Main code generator
   • befunge_codegen.nim - Befunge 2D support
   • subleq_bf.nim - SUBLEQ interpreter + Brainfuck transpiler

Compilation Pipeline

Source Code
    ↓
hybrid_lexer → Tokens
    ↓
hybrid_parser → AST
    ↓
hybrid_normalizer → IR
    ↓
hybrid_optimizer → Optimized IR
    ↓
hybrid_codegen → SUBLEQ Machine Code
    ↓
subleq_bf.runSubleq → Execution

Testing Strategy

Test Categories

• Conformance Tests: Verify all features work correctly
• Determinism Tests: Ensure reproducible builds
• Differential Tests: Compare reference vs compiled execution
• End-to-End Tests: Full pipeline with 18+ test programs
• Unit Tests: Individual component verification

Running Tests

cd /home/user/flash-attention-rtl/tests
nim c -d:release test_agent3_e2e.nim && ./test_agent3_e2e
nim c -d:release test_hybrid_compiler.nim && ./test_hybrid_compiler

Import Validation

All imports have been verified to work correctly:

• Root files (consolidated_agent.nim, hybrid_cli.nim) import from /src/
• Test files import from /src/ using ../src/ prefix
• Internal /src/ imports work directly (same directory)
• No circular dependencies
• All modules compile without import errors

Migration Notes

This organization was implemented by:

• Agent A: File reorganization and import path updates
• Agent B: Test integration and fix verification
• Agent C: Import validation and documentation

The reorganization maintains 100% backward compatibility with the compilation and test infrastructure while improving modularity and clarity of dependencies.