# FortWin Operations Manual Version: `1.0` Audience: operators, defenders, platform engineers, contributors Scope: single-node production loop currently implemented in this repository. --- ## 1. Purpose This manual explains how to operate, validate, and troubleshoot FortWin end-to-end: `detect/causality -> forge generation -> pre-council filter -> council verdict -> signed feed -> enforcer activation -> attestation` Use this guide for: - day-1 environment setup - day-2 operations - debugging and incident reconstruction - soak testing and reliability checks --- ## 2. System Components ### 2.1 Control Plane (`src/FortWin.ControlPlane`) - Command API and promotion transitions - Artifact registry and feed export/build - Strategy council evaluation and verdicts - Incident timeline endpoint by `trace_id` ### 2.2 Forge Plane (`src/FortWin.Forge`) - Ingest -> generate -> validate -> submit pipeline - Forge Manager orchestration: - context builder - router - candidate normalizer - pre-council filter - outcome learner - Typed compiler/validator blocking codes (`CompilerBlockingCode`) - Supports `local` or `remote` neural mode for bounded attack interpretation, pattern reranking, and candidate scoring assistance ### 2.3 Forge GPU Service (`src/FortWin.ForgeGpu`) - Dedicated ONNX/DirectML inference service - Exposes `/api/neural/status`, `/api/neural/service-status`, `/api/neural/interpret`, `/api/neural/rerank-patterns`, and `/api/neural/rank` - Keeps GPU inference isolated from control logic, artifact compilation, council, and enforcement ### 2.4 Feed Plane (`src/FortWin.Feed`) - Publish/list/get/pull/verify feed envelopes ### 2.5 Enforcement Plane (`src/FortWin.Enforcer`) - Verify/admit/stage/canary/activate or rollback - Attestation output and runtime state updates ### 2.6 Unified Dashboard (`src/FortWin.Dashboard`) - Single operator console executable - Tabs for overview, throughput, forges, events, history, incidents, commands - Command proxy to Control Plane and promoted artifact export ### 2.7 Shared - Contracts: `src/FortWin.Contracts` - Eventing/projections: `src/FortWin.Eventing` - Signing/key management: `src/FortWin.Security` --- ## 3. Prerequisites - Windows with PowerShell - `.NET 9 SDK` - Ports available for local services (examples below) Check: ```powershell dotnet --version ``` --- ## 4. Repository Layout Key directories: - `docs/` architecture and whitepaper docs - `scripts/` verification and soak scripts - `src/` all services/libraries - `data/events/` append-only event JSONL files - `data/projections/` projection read models - `data/feed/` feed envelopes - `data/runtime/` runtime state (enforcer, forge pipelines, keys, soak logs) --- ## 5. Quick Start Build: ```powershell dotnet build FortWin.sln ``` Run core verification suite: ```powershell powershell -ExecutionPolicy Bypass -File scripts\verify-phase1.ps1 powershell -ExecutionPolicy Bypass -File scripts\verify-sprint1.ps1 powershell -ExecutionPolicy Bypass -File scripts\verify-strategy-council.ps1 powershell -ExecutionPolicy Bypass -File scripts\verify-production-loop.ps1 ``` Build executable bundles: ```powershell powershell -ExecutionPolicy Bypass -File scripts\publish-executables.ps1 ``` Single-command live runtime (starts services + drives forge builds + feeds enforcer): ```powershell powershell -ExecutionPolicy Bypass -File scripts\start-forge-dashboard-enforcer.ps1 ``` GPU-service-backed runtime: ```powershell powershell -ExecutionPolicy Bypass -File scripts\start-forge-dashboard-enforcer.ps1 ``` Default startup now: - prefers the GPU service - bootstraps repo-local dev ONNX models under `models\` when missing - falls back to deterministic CPU mode if GPU startup is unavailable Strict GPU mode: ```powershell powershell -ExecutionPolicy Bypass -File scripts\start-forge-dashboard-enforcer.ps1 -GpuMode required -GpuExecutionProvider directml ``` Open dashboard: - `http://localhost:5090` --- ## 6. Service Startup (Manual) Start services in separate terminals: ```powershell dotnet run --project src/FortWin.ControlPlane --urls http://localhost:5077 dotnet run --project src/FortWin.Feed --urls http://localhost:5081 dotnet run --project src/FortWin.ForgeGpu --urls http://localhost:5083 -- --FortWin:ForgeGpu:RequireReadyOnStartup false --FortWin:ForgeGpu:Ranker:Enabled true --FortWin:ForgeGpu:Ranker:ModelPath C:\Users\r2d2\Desktop\FortWin\models\forge-ranker.onnx --FortWin:ForgeGpu:Ranker:ExecutionProvider directml --FortWin:ForgeGpu:Interpretation:Enabled true --FortWin:ForgeGpu:Interpretation:ModelPath C:\Users\r2d2\Desktop\FortWin\models\forge-interpret.onnx --FortWin:ForgeGpu:Interpretation:ExecutionProvider directml --FortWin:ForgeGpu:Rerank:Enabled true --FortWin:ForgeGpu:Rerank:ModelPath C:\Users\r2d2\Desktop\FortWin\models\forge-rerank.onnx --FortWin:ForgeGpu:Rerank:ExecutionProvider directml dotnet run --project src/FortWin.Forge --urls http://localhost:5082 dotnet run --project src/FortWin.Dashboard --urls http://localhost:5090 ``` Health checks: ```powershell Invoke-RestMethod http://localhost:5077/health Invoke-RestMethod http://localhost:5081/health Invoke-RestMethod http://localhost:5083/health Invoke-RestMethod http://localhost:5082/health Invoke-RestMethod http://localhost:5090/health ``` --- ## 7. API Runbook ### 7.1 Control Plane - `GET /api/state` - `POST /api/commands` - `POST /api/artifacts/register` - `POST /api/feed/build` - `POST /api/feed/export` - `POST /api/feed/verify` - `POST /api/strategy/evaluate` - `GET /api/incidents/{traceId}` ### 7.2 Forge - `POST /api/pipeline/ingest` - `POST /api/pipeline/generate/{traceId}` - `POST /api/pipeline/validate/{traceId}` - `POST /api/pipeline/submit/{traceId}` - `GET /api/pipeline/{traceId}` - `GET /api/pipeline` - `GET /api/forge/neural/status` - `GET /api/forge/neural/telemetry` ### 7.3 Forge GPU Service - `GET /api/neural/status` - `GET /api/neural/service-status` - `POST /api/neural/interpret` - `POST /api/neural/rerank-patterns` - `POST /api/neural/rank` ### 7.4 Feed - `POST /api/feed/publish` - `GET /api/feed/current` - `GET /api/feed/list` - `GET /api/feed/{feedId}` - `GET /api/feed/pull` - `POST /api/feed/verify` ### 7.5 Dashboard - `GET /api/dashboard/overview` - `GET /api/dashboard/runtime-artifacts?page=&pageSize=` - `GET /api/dashboard/runtime-archive?page=&pageSize=` - `GET /api/dashboard/recent-events?page=&pageSize=` - `GET /api/dashboard/pipelines?page=&pageSize=` - `POST /api/dashboard/commands` - `GET /api/dashboard/incidents/{traceId}` - `GET /api/dashboard/export/promoted` ### 7.6 Forge Neural Inference (Optional GPU) Forge remains deterministic by default, with optional ONNX neural assistance for: - `TraceSummary` interpretation through the remote service boundary - pattern retrieval reranking before candidate generation - bounded neural score adjustment during candidate scoring If neural inference is unavailable, Forge stays operational and falls back to deterministic-only interpretation, retrieval order, and ranking. Configuration keys (`FortWin:Forge:Neural`): - `Enabled` - `Mode` - `ServiceUrl` - `RequestTimeoutSeconds` - `ModelPath` - `ExecutionProvider` (`cpu`, `cuda`, `directml`) - `InputName` (optional) - `OutputName` (optional) - `MaxScoreAdjustment` PowerShell example: ```powershell $env:FortWin__Forge__Neural__Enabled="true" $env:FortWin__Forge__Neural__ModelPath="C:\Users\r2d2\Desktop\FortWin\models\forge-ranker.onnx" $env:FortWin__Forge__Neural__ExecutionProvider="directml" dotnet run --project src/FortWin.Forge --urls http://localhost:5082 ``` Verify status: ```powershell .\scripts\verify-forge-neural.ps1 -ForgeUrl http://localhost:5082 ``` Dedicated GPU service verification: ```powershell .\scripts\verify-forge-gpu-service.ps1 -GpuServiceUrl http://localhost:5083 ``` Model preflight: ```powershell .\scripts\verify-gpu-models.ps1 ``` Generate or refresh repo-local trained local surrogate models: ```powershell .\scripts\generate-dev-gpu-models.ps1 -Overwrite ``` Verify Control Plane paging: ```powershell .\scripts\verify-controlplane-paging.ps1 ``` Verify dashboard paging and overview trimming: ```powershell .\scripts\verify-dashboard-paging.ps1 ``` GPU service restart helper: ```powershell .\scripts\restart-forge-gpu.ps1 -GpuServiceUrl http://localhost:5083 -ExecutionProvider directml -BootstrapDevModels ``` Dashboard visibility: - Overview shows Forge neural readiness and GPU service readiness separately. - Overview includes an `Inference Health` summary card with fallback rate, latest execution path, and latest latency. - Overview keeps heavy collections out of the payload; artifacts, events, pipelines, and archive history are loaded through separate paged endpoints. - History tab exposes archived runtime artifacts directly for operator inspection. - Forges tab shows model-specific provider state, model path, and whether interpretation, rerank, or rank requests used remote inference or deterministic fallback. - `GET /api/forge/neural/telemetry` exposes recent inference records and operation summaries for operator review and incident reconstruction. --- ## 8. Artifact Lifecycle (Operational) 1. Register artifact in Control Plane. 2. Forge generates candidates and validates them. 3. Forge Manager applies quality/diversity filtering. 4. Strategy Council evaluates survivors and emits verdict. 5. Control Plane exports signed feed envelope. 6. Enforcer verifies feed signature and admission policy. 7. Enforcer stages -> canary -> activates or rolls back. 8. Attestation events/results are persisted for learning and audit. --- ## 9. Forge Manager and Blocking Codes ### 9.1 Manager outputs Pipeline state includes: - `forgeContext` - `executionPlan` - `preCouncilCandidates` - `traceSummary` - `attackTransitions` - `repairIntents` - `repairCandidates` - `scoreCards` - `lineage` ### 9.2 Compiler/validator rejection typing Compiler and validator paths use `CompilerBlockingCode` (no free-form failure strings in these paths). Examples: - `IntentMissingGoal` - `IntentConstraintInvalid` - `CandidateNoArtifacts` - `CandidateActionNotAllowedForKind` - `CandidatePerfBudgetExceeded` - `CandidateDifferentialDriftExceeded` - `CandidateGeneralizationWeak` --- ## 10. Command Model Primary command types: - `Approve` - `Stage` - `Canary` - `Promote` - `Rollback` - `Suppress` - `SetAutonomyLevel` - `SetEmergencyOverride` All commands use `CommandEnvelope` and are applied through Control Plane mutation + event emission. --- ## 11. Runtime Files (Authoritative/Operational) ### 11.1 Events - `data/events/events-YYYY-MM-DD.jsonl` ### 11.2 Projections - `data/projections/runtime-state.json` - `data/projections/artifact-registry.json` - `data/projections/strategy-council.json` ### 11.3 Forge - `data/runtime/forge-pipelines/{traceId}.json` - Incompatible legacy state auto-backs up to: - `*.invalid-.bak` ### 11.4 Feed - `data/feed/current-feed.json` - `data/feed/envelopes/*.json` ### 11.5 Enforcer - `data/runtime/enforcer-policy.json` - `data/runtime/enforcer-state.json` - `data/runtime/enforcer-last-run.json` - `data/runtime/attestations-YYYY-MM-DD.jsonl` ### 11.6 Signing - `data/runtime/keys/signing-keys.json` --- ## 12. Monitoring and Debugging ### 12.1 Event inspection (latest) ```powershell Invoke-RestMethod "http://localhost:5077/api/events?limit=200" ``` ### 12.2 Incident reconstruction ```powershell $traceId = "trace-prod-loop" Invoke-RestMethod "http://localhost:5077/api/incidents/$traceId" ``` ### 12.3 Forge pipeline inspection ```powershell $traceId = "trace-prod-loop" Invoke-RestMethod "http://localhost:5082/api/pipeline/$traceId" ``` ### 12.4 Typical failure classes - `command_rejected` - `promotion_blocked` - `activation_blocked_emergency_override` - `forge_candidate_refined` (not a fatal error; indicates iterative refinement) - `enforcer_artifact_processed` with non-`Applied` status --- ## 13. Soak Testing Use the soak runner: ```powershell powershell -ExecutionPolicy Bypass -File scripts\run-soak.ps1 -Minutes 30 ``` Output: - `data/runtime/soak/soak-.log` - `data/runtime/soak/soak--summary.json` - `data/runtime/soak/soak--errors.log` (only when failures occur) Recommended acceptance criteria: - `fails == 0` - no unhandled process crashes - no persistent event-type error spikes --- ## 14. Enforcer Node Installation (Executable) Install packaged enforcer executable to a workstation/node (run elevated PowerShell): ```powershell powershell -ExecutionPolicy Bypass -File scripts\install-enforcer-node.ps1 -PackagePath build\publish\enforcer-win-x64 ``` Defaults: - install root: `C:\Program Files\FortWin\Enforcer` - data root: `C:\ProgramData\FortWin\data` - scheduled task: `FortWinEnforcer` (periodic execution) Uninstall: ```powershell powershell -ExecutionPolicy Bypass -File scripts\uninstall-enforcer-node.ps1 ``` --- ## 15. Security and Trust - Feed envelopes are signed and verified by the security layer. - Enforcer rejects invalid signature or policy-incompatible artifacts. - High-risk operations remain bounded by policy and transition rules. Operational rule: - never manually edit generated feed signatures or projection files to force state. --- ## 16. Recovery Procedures ### 15.1 If a service will not start - rebuild solution: - `dotnet build FortWin.sln` - verify health endpoint after restart - inspect latest events and runtime files ### 15.2 If forge pipeline file is incompatible - Forge auto-backs up invalid JSON and recreates a clean state object. - inspect `*.invalid-*.bak` in `data/runtime/forge-pipelines/` ### 15.3 If enforcer starts rejecting artifacts - inspect: - `data/runtime/enforcer-policy.json` - `data/runtime/enforcer-last-run.json` - verify feed signature through: - `POST /api/feed/verify` --- ## 17. Contributor Workflow 1. Update contracts first (`src/FortWin.Contracts`) 2. Update service logic 3. Build and run verification scripts 4. Update docs (`README.md`, `docs/ARCHITECTURE_README.md`, this manual) Minimum check before merge: ```powershell dotnet build FortWin.sln powershell -ExecutionPolicy Bypass -File scripts\verify-production-loop.ps1 ``` --- ## 18. Non-Goals (Current Phase) - Full multi-node cell/federation rollout automation - Full production UI cockpit project in this repository - Unbounded autonomous code execution outside artifact model --- ## 19. Appendix: Useful Commands Build: ```powershell dotnet build FortWin.sln ``` Run Control Plane only: ```powershell dotnet run --project src/FortWin.ControlPlane --urls http://localhost:5077 ``` Run Forge only: ```powershell dotnet run --project src/FortWin.Forge --urls http://localhost:5082 ``` Run Feed only: ```powershell dotnet run --project src/FortWin.Feed --urls http://localhost:5081 ``` Run Unified Dashboard only: ```powershell dotnet run --project src/FortWin.Dashboard --urls http://localhost:5090 ``` Run Enforcer against current feed: ```powershell dotnet run --project src/FortWin.Enforcer -- --data-root data --feed data/feed/current-feed.json --mode apply ``` Start integrated live loop for forge/dashboard/enforcer: ```powershell powershell -ExecutionPolicy Bypass -File scripts\start-forge-dashboard-enforcer.ps1 -Cycles 1 ``` Stop live loop services via PID file: ```powershell powershell -ExecutionPolicy Bypass -File scripts\stop-forge-dashboard-enforcer.ps1 ``` --- End of manual.