# BuildFlow Architecture Closure And Backend SOP Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task in the current session. Do not dispatch subagents for this plan. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Complete the fifth optimization batch by enforcing source-wide architecture rules, moving the remaining endpoints and module integrations to MediatR CQRS, normalizing business error codes and file organization, and producing a fully isolated, executable, evidence-archiving backend SOP. **Architecture:** `BuildFlow.Host` remains the only HTTP/composition root. Endpoints only translate HTTP/SSE contracts to target-module Application commands or stream requests; every cross-module call is an `IMediator` send to a target-owned Query/Command/DTO. Roslyn architecture tests scan all current source files, and the SOP starts isolated random-port infrastructure before running migrations, integration, concurrency, recovery, and large-data gates. **Tech Stack:** .NET 10, FastEndpoints 8.1, MediatR 12.5, FluentValidation 12.1, EF Core 10/Npgsql, Roslyn 5.0, xUnit, PostgreSQL/pgvector, Redis, MongoDB, MinIO, Docker Compose, Bash. --- ## File Map The implementation creates or changes the following ownership units: - `tests/BuildFlow.Architecture.Tests/SourceArchitectureTests.cs`: source-wide Roslyn guards for endpoint dependencies, Host bridges, error-code arguments, and top-level type count. - `src/modules/platform/Application/Auth/**`: change-password command, handler, and validator. - `src/modules/rag/Application/RagQA/**`: RAG stream request, stream event, handler, and validator. - `src/modules/platform/Application/Users/**`: Platform-owned batch user-summary query and DTO consumed by IM. - `src/modules/file/Application/Features/Files/**`: File-owned batch preview-path query and DTO consumed by Platform. - IM/RAG/Platform handlers: inject `IMediator` and send target-owned contracts directly. - `src/BuildFlow.Host/Services/Integration/InProcess*.cs`: deleted after all consumers move to CQRS. - Workflow/RAG/File throw sites: explicit semantic `ErrorCode` arguments. - Consul/workflow graph/tree and four test support files: split to one top-level class/record/enum per file. - `scripts/backend-optimization-compose.yml`: isolated PostgreSQL, Redis, MongoDB, and MinIO services with random host ports. - `scripts/backend-optimization-sop.sh`: fail-fast orchestrator and artifact writer. - `docs/backend-optimization-sop.md`: operator contract, prerequisites, phases, outputs, and failure handling. - `artifacts/backend-optimization-sop//`: generated only at runtime and ignored by Git. ### Task 1: Add Source-Wide Architecture Guards In RED State **Files:** - Modify: `tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj` - Create: `tests/BuildFlow.Architecture.Tests/SourceArchitectureTests.cs` - [x] **Step 1: Add Roslyn to the architecture test project** Add the cached repository-compatible package version: ```xml ``` - [x] **Step 2: Write the four source-wide failing tests** `SourceArchitectureTests` must locate `BuildFlow.slnx`, enumerate non-generated `.cs` files under `src` and `tests`, parse each with `CSharpSyntaxTree.ParseText`, and expose four facts: ```csharp [Fact] public void EndpointConstructorsShouldNotDependOnDbContexts(); [Fact] public void HostShouldNotContainInProcessBridges(); [Fact] public void BusinessExceptionsShouldSpecifySemanticErrorCodes(); [Fact] public void SourceFilesShouldContainAtMostOneTopLevelClassRecordOrEnum(); ``` The tests must report relative path and 1-based line number. The exception guard applies to `BusinessException`, `NotFoundException`, and `UnauthorizedException`, requires at least two arguments, and requires the first expression to be `ErrorCode.`. The file guard counts only top-level `class`, `record`, and `enum`; nested declarations remain allowed. - [x] **Step 3: Run each guard and verify the expected RED inventory** Run: ```bash dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore --filter FullyQualifiedName~SourceArchitectureTests --logger "console;verbosity=normal" ``` Expected: FAIL and report exactly the current categories: two DbContext endpoints, three Host `InProcess*` bridges, 95 one-argument business exceptions, and seven multi-top-level-type files. The 95th site is a namespace-qualified exception in `UploadDocumentEndpoint` that the earlier regular-expression audit did not detect. - [x] **Step 4: Commit the RED guard** ```bash git add tests/BuildFlow.Architecture.Tests git commit -m "test: enforce backend source architecture" ``` ### Task 2: Move Change Password To Platform CQRS **Files:** - Create: `src/modules/platform/Application/Auth/Commands/ChangePasswordCommand.cs` - Create: `src/modules/platform/Application/Auth/Handlers/ChangePasswordCommandHandler.cs` - Create: `src/modules/platform/Application/Auth/Validators/ChangePasswordCommandValidator.cs` - Modify: `src/BuildFlow.Host/Endpoints/Platform/Auth/ChangePasswordEndpoint.cs` - Create: `tests/BuildFlow.Platform.Tests/Auth/ChangePasswordCommandHandlerTests.cs` - Create: `tests/BuildFlow.Platform.Tests/Auth/ChangePasswordEndpointTests.cs` - Modify: `tests/BuildFlow.Platform.Tests/BuildFlow.Platform.Tests.csproj` - [x] **Step 1: Write handler and endpoint tests first** Cover these behaviors: ```text valid old password -> password hash changes and command completes wrong old password -> BusinessException(ErrorCode.Business) same old/new password -> BusinessException(ErrorCode.Business) missing current user -> UnauthorizedException(ErrorCode.Unauthorized) endpoint -> sends ChangePasswordCommand and writes an OK response ``` The endpoint test must use `FastEndpoints.Factory.Create` with a mocked `IMediator`; it must not construct a `PlatformDbContext`. - [x] **Step 2: Run tests to verify RED** ```bash dotnet test tests/BuildFlow.Platform.Tests/BuildFlow.Platform.Tests.csproj --no-restore --filter "FullyQualifiedName~ChangePassword" --logger "console;verbosity=normal" ``` Expected: FAIL because the command/handler do not exist and the endpoint still requires `PlatformDbContext`. - [x] **Step 3: Implement the Platform command boundary** Use this contract: ```csharp public sealed record ChangePasswordCommand(string OldPassword, string NewPassword) : IRequest; ``` The handler injects `PlatformDbContext` and `ICurrentUserContext`, verifies the current user and BCrypt hash, throws explicit semantic exceptions, updates the hash, and calls `SaveChangesAsync(ct)`. The validator enforces non-empty old/new passwords and the existing password policy. The endpoint becomes: ```csharp public class ChangePasswordEndpoint(IMediator mediator) : Endpoint { public override async Task HandleAsync(ChangePasswordRequest req, CancellationToken ct) { await mediator.Send(new ChangePasswordCommand(req.OldPassword, req.NewPassword), ct); await Send.OkAsync(ct); } } ``` - [x] **Step 4: Run focused tests and the endpoint architecture guard** ```bash dotnet test tests/BuildFlow.Platform.Tests/BuildFlow.Platform.Tests.csproj --no-restore --filter "FullyQualifiedName~ChangePassword" dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore --filter FullyQualifiedName~EndpointConstructorsShouldNotDependOnDbContexts ``` Expected: change-password tests PASS; the architecture test still reports only `RAGStreamEndpoint`. - [x] **Step 5: Commit** ```bash git add src/modules/platform/Application/Auth src/BuildFlow.Host/Endpoints/Platform/Auth tests/BuildFlow.Platform.Tests git commit -m "refactor: move password changes to platform cqrs" ``` ### Task 3: Move RAG SSE Generation To An Application Stream Handler **Files:** - Create: `src/modules/rag/Application/RagQA/Queries/RAGStreamQuery.cs` - Create: `src/modules/rag/Application/RagQA/DTOs/RAGStreamEvent.cs` - Create: `src/modules/rag/Application/RagQA/Handlers/RAGStreamQueryHandler.cs` - Create: `src/modules/rag/Application/RagQA/Validators/RAGStreamQueryValidator.cs` - Modify: `src/BuildFlow.Host/Endpoints/Rag/RAG/RAGStreamEndpoint.cs` - Create: `tests/BuildFlow.Rag.Tests/RagStreamQueryHandlerTests.cs` - Create: `tests/BuildFlow.Host.Tests/BuildFlow.Host.Tests.csproj` - Create: `tests/BuildFlow.Host.Tests/RagStreamEndpointTests.cs` - Modify: `BuildFlow.slnx` - [x] **Step 1: Write stream handler and endpoint protocol tests** Use a target-owned request: ```csharp public sealed record RAGStreamQuery(Guid KnowledgeBaseId, string Question) : IStreamRequest; ``` Test event order for hits (`Sources`, content chunks, `Done`), no hits (fallback content, `Done`), missing knowledge base (`NotFoundException(ErrorCode.NotFound)`), and cancellation propagation. Use `Factory.Create` to assert exact SSE output including `event: sources`, JSON `data:` content frames, and final `data: [DONE]`. - [x] **Step 2: Run tests to verify RED** ```bash dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~RagStream" dotnet test tests/BuildFlow.Host.Tests/BuildFlow.Host.Tests.csproj --no-restore --filter "FullyQualifiedName~RagStream" ``` Expected: FAIL because the request/event/handler do not exist and the endpoint owns retrieval/chat logic. - [x] **Step 3: Implement the stream handler** `RAGStreamEvent` must be the only top-level type in its file and expose source/content/done factories without adding an enum file: ```csharp public sealed record RAGStreamEvent( IReadOnlyList? Sources, string? Content, bool IsDone); ``` The handler injects `RagDbContext`, `IRagRetrievalService`, and `IAIChatAgent`; it retrieves the knowledge base, emits source summaries truncated at 200 characters, builds the bounded prompt with the current retrieval result, streams chat chunks, and emits Done. The endpoint injects only `IMediator`, sets SSE headers before enumeration, encodes events, flushes after every frame, and preserves the current wire protocol byte-for-byte. - [x] **Step 4: Run focused tests and endpoint guard** ```bash dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~RagStream" dotnet test tests/BuildFlow.Host.Tests/BuildFlow.Host.Tests.csproj --no-restore --filter "FullyQualifiedName~RagStream" dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore --filter FullyQualifiedName~EndpointConstructorsShouldNotDependOnDbContexts ``` Expected: PASS; no endpoint constructor depends on a DbContext. - [x] **Step 5: Commit** ```bash git add BuildFlow.slnx src/modules/rag/Application/RagQA src/BuildFlow.Host/Endpoints/Rag/RAG tests/BuildFlow.Rag.Tests tests/BuildFlow.Host.Tests git commit -m "refactor: stream rag answers through cqrs" ``` ### Task 4: Replace IM User Information Bridge With A Platform Batch Query **Files:** - Create: `src/modules/platform/Application/Users/DTOs/UserSummaryDto.cs` - Create: `src/modules/platform/Application/Users/Queries/GetUserSummariesQuery.cs` - Create: `src/modules/platform/Application/Users/Handlers/GetUserSummariesQueryHandler.cs` - Modify: `src/modules/im/Application/IM.Application.csproj` - Modify: seven handlers under `src/modules/im/Application/Chat/Handlers/` that inject `IUserInfoService` - Delete: `src/modules/im/Domain/Common/IUserInfoService.cs` - Delete: `src/BuildFlow.Host/Services/Integration/InProcessUserInfoService.cs` - Modify: `src/BuildFlow.Host/BuildFlowHostModule.cs` - Modify: IM unit tests using `Mock` - Replace: `tests/BuildFlow.IM.Integration.Tests/InMemoryUserInfoService.cs` with `InMemoryUserSummaryQueryHandler.cs` - Modify: `tests/BuildFlow.IM.Integration.Tests/ImServerFixture.cs` - [x] **Step 1: Write Platform query tests and update IM tests to expect mediator sends** The Platform query returns an `IReadOnlyDictionary` for distinct requested IDs and performs one database query. IM tests must configure `IMediator.Send(new GetUserSummariesQuery(...), ct)` and verify that empty input avoids a send where practical. - [x] **Step 2: Run focused tests to verify RED** ```bash dotnet test tests/BuildFlow.Platform.Tests/BuildFlow.Platform.Tests.csproj --no-restore --filter FullyQualifiedName~GetUserSummaries dotnet test tests/BuildFlow.IM.Tests/BuildFlow.IM.Tests.csproj --no-restore --filter "FullyQualifiedName~GetConversations|FullyQualifiedName~SendMessage" ``` Expected: FAIL until the target-owned query and IM handler changes exist. - [x] **Step 3: Implement the direct CQRS calls** Add a Platform Application project reference to IM Application. Each IM handler injects `IMediator`, sends one `GetUserSummariesQuery` for its deduplicated IDs, and reads results by `Guid` key. Do not catch all exceptions and do not introduce a replacement service/interface bridge. The integration fake implements `IRequestHandler>`; `ImServerFixture` registers both the IM Application assembly and the test assembly with MediatR. - [x] **Step 4: Remove bridge interface, Host implementation, and registration** Delete `IUserInfoService` and `InProcessUserInfoService`; remove the Host `AddScoped()` registration and stale using directives/comments. - [x] **Step 5: Run IM unit and SignalR integration tests** ```bash dotnet test tests/BuildFlow.IM.Tests/BuildFlow.IM.Tests.csproj --no-restore dotnet test tests/BuildFlow.IM.Integration.Tests/BuildFlow.IM.Integration.Tests.csproj --no-restore ``` Expected: PASS with the same private/group conversation display names and notification behavior. - [x] **Step 6: Commit** ```bash git add src/modules/platform/Application/Users src/modules/im src/BuildFlow.Host tests/BuildFlow.IM.Tests tests/BuildFlow.IM.Integration.Tests git commit -m "refactor: query platform users through mediator" ``` ### Task 5: Replace File And Avatar Bridges With File Application Contracts **Files:** - Create: `src/modules/file/Application/Features/Files/DTOs/FilePreviewPathDto.cs` - Create: `src/modules/file/Application/Features/Files/Queries/ResolveFilePreviewPathsQuery.cs` - Create: `src/modules/file/Application/Features/Files/Handlers/ResolveFilePreviewPathsQueryHandler.cs` - Create: `tests/BuildFlow.File.Tests/ResolveFilePreviewPathsQueryHandlerTests.cs` - Modify: `src/modules/platform/Application/Platform.Application.csproj` - Modify: `src/modules/platform/Application/Auth/Handlers/GetCurrentUserQueryHandler.cs` - Modify: `src/modules/platform/Application/Auth/Handlers/UpdateProfileCommandHandler.cs` - Modify: `src/modules/platform/Application/Users/Handlers/GetUserSummariesQueryHandler.cs` - Delete: `src/modules/platform/Domain/Services/IAvatarUrlResolver.cs` - Modify: `src/modules/rag/Application/BuildFlow.Application.csproj` - Modify: RAG document upload/process/preview handlers and their tests - Delete: `src/modules/rag/Domain/Interfaces/IFileStorageClient.cs` - Delete: `src/modules/rag/Infrastructure/Files/FileStorageClient.cs` - Modify: `src/modules/rag/Infrastructure/RAGInfrastructureModule.cs` - Delete: `src/BuildFlow.Host/Services/Integration/InProcessFileStorageClient.cs` - Delete: `src/BuildFlow.Host/Services/Integration/InProcessAvatarUrlResolver.cs` - Modify: `src/BuildFlow.Host/BuildFlowHostModule.cs` - [x] **Step 1: Write File preview-path query tests** Test null/blank references, existing HTTP/HTTPS URLs, object-key URL escaping, duplicate inputs, and batch output. The target-owned handler must return `/api/files/preview?objectKey=` without touching S3. - [x] **Step 2: Update Platform tests to expect one File batch query** `GetCurrentUserQueryHandler`, `UpdateProfileCommandHandler`, and `GetUserSummariesQueryHandler` inject `IMediator`. A batch user query sends one `ResolveFilePreviewPathsQuery` for distinct non-empty avatar references to avoid N+1 path resolution. - [x] **Step 3: Update RAG tests to expect direct File Application sends** RAG upload sends `UploadFileCommand(objectKey, stream, contentType, fileSize)`; process and preview send `DownloadFileQuery(objectKey)` and dispose the returned stream. Remove `IHttpContextAccessor`, auth-token extraction, and `IFileStorageClient` stubs from these paths. - [x] **Step 4: Run focused tests to verify RED** ```bash dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter FullyQualifiedName~ResolveFilePreviewPaths dotnet test tests/BuildFlow.Platform.Tests/BuildFlow.Platform.Tests.csproj --no-restore --filter "FullyQualifiedName~CurrentUser|FullyQualifiedName~UpdateProfile|FullyQualifiedName~GetUserSummaries" dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~Document|FullyQualifiedName~PreviewDocument" ``` Expected: FAIL before implementation and PASS after the direct mediator changes. - [x] **Step 5: Delete the old interfaces, implementations, and DI registrations** Remove all three Host `InProcess*` classes and both source-module service interfaces. Remove the obsolete RAG HTTP `FileStorageClient` registration and class. `rg` must return no production references: ```bash rg -n "InProcessUserInfoService|InProcessFileStorageClient|InProcessAvatarUrlResolver|IUserInfoService|IFileStorageClient|IAvatarUrlResolver" src ``` Expected: no matches. - [x] **Step 6: Run the Host bridge guard and affected test projects** ```bash dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore --filter FullyQualifiedName~HostShouldNotContainInProcessBridges dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore dotnet test tests/BuildFlow.Platform.Tests/BuildFlow.Platform.Tests.csproj --no-restore dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore ``` Expected: PASS. - [x] **Step 7: Commit** ```bash git add src/modules/file src/modules/platform src/modules/rag src/BuildFlow.Host tests/BuildFlow.File.Tests tests/BuildFlow.Platform.Tests tests/BuildFlow.Rag.Tests git commit -m "refactor: remove host module service bridges" ``` ### Task 6: Normalize Every Business Exception Error Code **Files:** - Modify: 95 throw/return sites reported by the architecture guard in Host, File, RAG, and Workflow. - Modify tests only where they assert exception codes. - [x] **Step 1: Add focused semantic-code assertions before production edits** Extend existing tests for representative categories: duplicate resource, missing resource, invalid state, invalid parameter/schema, and forbidden operation. Assert the exception `Code` property, not only the type/message. - [x] **Step 2: Run the representative tests and architecture guard in RED** ```bash dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter "FullyQualifiedName~HandlerTests|FullyQualifiedName~ProcessEngineTests" dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore --filter FullyQualifiedName~BusinessExceptionsShouldSpecifySemanticErrorCodes ``` Expected: representative code assertions and the architecture guard fail on old one-argument calls. - [x] **Step 3: Apply the semantic mapping deliberately** Use this mapping at every reported site: ```text missing/not found -> NotFoundException(ErrorCode.NotFound, message) duplicate/already exists -> BusinessException(ErrorCode.AlreadyExists, message) disabled/deleted/already processed/unsupported transition -> BusinessException(ErrorCode.InvalidState, message) invalid JSON/schema/value/path/missing action argument -> BusinessException(ErrorCode.InvalidParam, message) permission denied -> UnauthorizedException(ErrorCode.Forbidden, message) rule conflict without a narrower category -> BusinessException(ErrorCode.Business, message) ``` Convert existing `BusinessException` calls that describe a missing resource to `NotFoundException` where the surrounding contract already treats absence as 404. Do not replace all calls with a single code. - [x] **Step 4: Run the guard and module tests** ```bash dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore --filter FullyQualifiedName~BusinessExceptionsShouldSpecifySemanticErrorCodes dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore ``` Expected: PASS and zero old one-argument calls. - [x] **Step 5: Commit** ```bash git add src/modules/file src/modules/rag src/modules/workflow tests git commit -m "refactor: classify backend business errors" ``` ### Task 7: Split Every Multi-Type Source File **Files:** - Split `src/BuildFlow.Host/Configuration/ConsulConfigurationProvider.cs` into `ConsulConfigurationSource.cs`, `ConsulOptions.cs`, `ConsulConfigurationProvider.cs`, and `ConsulKvItem.cs`. - Split `src/modules/workflow/Domain/Engine/WorkflowGraph.cs` into `WorkflowGraph.cs`, `GraphNode.cs`, `GraphEdge.cs`, `GraphEndpoint.cs`, `GraphPoint.cs`, and `GraphJsonExtensions.cs`. - Split `src/modules/workflow/Domain/Engine/WorkflowTree.cs` into `WorkflowTree.cs`, `TreeNode.cs`, and `TreeBranch.cs`. - Create `tests/BuildFlow.Platform.Tests/DictionaryConfiguration.cs` from `PlatformTestDb.cs`. - Create `tests/BuildFlow.Rag.Tests/Stub/StubDocumentExtractorFactory.cs` from `StubDocumentExtractor.cs`. - Create `tests/BuildFlow.Workflow.Tests/Form/FormTestFixtureClassFixture.cs` from `FormTestFixture.cs`. - Create `tests/BuildFlow.Workflow.Tests/Notification/NoOpPlatformNotificationPublisher.cs` from `NotificationServiceTests.cs`. - [x] **Step 1: Move declarations without changing behavior** Preserve namespaces, modifiers, XML comments, JSON attributes, extension methods, and member order. Each original file retains its namesake type; each moved declaration gets a same-named file. - [x] **Step 2: Run the file organization guard** ```bash dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore --filter FullyQualifiedName~SourceFilesShouldContainAtMostOneTopLevelClassRecordOrEnum ``` Expected: PASS with zero violations under `src` and `tests`. - [x] **Step 3: Run affected projects** ```bash dotnet test tests/BuildFlow.Platform.Tests/BuildFlow.Platform.Tests.csproj --no-restore dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore dotnet build src/BuildFlow.Host/BuildFlow.Host.csproj --no-restore ``` Expected: PASS, 0 warning, 0 error. - [x] **Step 4: Commit** ```bash git add src/BuildFlow.Host/Configuration src/modules/workflow/Domain/Engine tests git commit -m "refactor: enforce one top-level type per file" ``` ### Task 8: Add Isolated Infrastructure, Real S3 Coverage, And SOP Metrics **Files:** - Create: `scripts/backend-optimization-compose.yml` - Create: `tests/BuildFlow.File.Tests/S3StorageServiceIntegrationTests.cs` - Modify: `tests/BuildFlow.IM.Integration.Tests/ImServerFixture.cs` - Modify: `tests/BuildFlow.IM.Tests/Chat/SendMessageCommandHandlerTests.cs` - Modify: selected performance tests to emit `SOP_METRIC key=value` through `ITestOutputHelper` - Create: `tests/BuildFlow.Workflow.Tests/Performance/OpenXmlMemoryProfileTests.cs` - [x] **Step 1: Add environment-driven isolated connection support** Replace fixed IM test endpoints with these environment variables and retain current values only as local fallbacks: ```text BUILDFLOW_IM_TEST_POSTGRES_CONNECTION BUILDFLOW_IM_TEST_MONGO_CONNECTION BUILDFLOW_IM_TEST_REDIS_CONNECTION BUILDFLOW_WORKFLOW_TEST_CONNECTION BUILDFLOW_MONITORING_TEST_CONNECTION BUILDFLOW_RAG_REDIS_TEST_CONNECTION BUILDFLOW_FILE_S3_TEST_ENDPOINT BUILDFLOW_FILE_S3_TEST_ACCESS_KEY BUILDFLOW_FILE_S3_TEST_SECRET_KEY ``` - [x] **Step 2: Add a real MinIO S3 integration test** The test must create a unique bucket, upload a non-empty object, download and byte-compare it, verify a zero-byte object, delete objects/bucket in `finally`, and emit object count/bytes as `SOP_METRIC`. It must fail rather than skip when its configured endpoint is unavailable. - [x] **Step 3: Add deterministic metric output** Emit at least these values from passing tests: ```text form_aggregate_select_count import_commit_count redis_concurrent_append_count redis_list_length rag_context_message_count rag_context_estimated_tokens file_size_max_s3_concurrency excel_export_peak_managed_bytes excel_export_rows ``` `OpenXmlMemoryProfileTests` writes a large export to a temporary file from an `IAsyncEnumerable`, samples `GC.GetTotalMemory(false)` while writing, asserts a conservative bounded peak delta, emits the observed bytes, and always deletes the file. - [x] **Step 4: Add random-port Docker Compose infrastructure** Use pinned images already compatible with the repository: ```text pgvector/pgvector:0.8.0-pg17 redis:7-alpine mongo:7 minio/minio:RELEASE.2025-04-22T22-12-26Z ``` Publish container ports with Docker-assigned random loopback ports, add health checks, use test-only credentials, and name volumes inside the per-run Compose project so teardown removes only SOP resources. - [x] **Step 5: Start the compose stack manually and run infrastructure tests** Use a unique Compose project name, discover ports with `docker compose port`, export the variables above, then run PostgreSQL reliability/migration tests, Redis tests, IM SignalR integration tests, Monitoring UPSERT tests, and the MinIO test. Expected: all PASS, zero skipped, and no access to developer databases or existing containers. - [x] **Step 6: Commit** ```bash git add scripts/backend-optimization-compose.yml tests git commit -m "test: add isolated backend integration evidence" ``` ### Task 9: Implement The Executable Closure SOP And Operator Guide **Files:** - Create: `scripts/backend-optimization-sop.sh` - Create: `docs/backend-optimization-sop.md` - Modify: `.gitignore` - Create: `tests/BuildFlow.Architecture.Tests/BackendOptimizationSopTests.cs` - [x] **Step 1: Write SOP contract tests in RED** The architecture test reads the script and Compose file and asserts: ```text set -euo pipefail EXIT trap performs project-scoped docker compose down -v all four isolated services exist and use random host ports stages appear in design order dotnet build requires no restore and warnings-as-errors TRX results and logs go under a timestamped artifact directory skipped > 0 causes failure static scans include password, one-argument exception, endpoint DbContext, InProcess, multi-type, DateTime.Now no appsettings.Secrets.json write and no developer database name ``` - [x] **Step 2: Run the contract test to verify RED** ```bash dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore --filter FullyQualifiedName~BackendOptimizationSopTests ``` Expected: FAIL because the script and guide do not exist. - [x] **Step 3: Implement the fail-fast SOP** The script must: ```text 1. verify dotnet 10, docker, docker compose, git, rg, curl, and repository cleanliness policy 2. run Roslyn/static guards and explicit final-debt scans 3. restore once, then build with --no-restore and capture 0 warning/0 error 4. run non-external unit and architecture tests 5. start isolated Compose services and wait for health 6. create isolated databases and run empty + previous-version migration paths 7. run PostgreSQL, Redis, MongoDB/SignalR, MinIO, and endpoint/SSE tests 8. run two-worker claim, crash/lease recovery, restart-equivalent recovery tests 9. run long-context, large-form, large-Excel, and large-directory/performance gates 10. run the complete solution test suite, reject any skipped test, aggregate TRX counts and SOP_METRIC lines, and write summary.md ``` Every stage runs through a helper that records command, start/end timestamps, duration, exit status, and log path. Any command failure exits non-zero; the final success marker is written only after every gate passes. The cleanup trap removes only the unique Compose project. - [x] **Step 4: Document operation and evidence semantics** `docs/backend-optimization-sop.md` must explain prerequisites, exact invocation, optional artifact root, isolated resource behavior, environment variables, stage meanings, report layout, failure triage, and the rule that a failed/skipped gate is not completion. - [x] **Step 5: Ignore generated evidence while preserving the directory convention** Add: ```gitignore artifacts/backend-optimization-sop/ ``` - [x] **Step 6: Run syntax and contract tests** ```bash bash -n scripts/backend-optimization-sop.sh dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore --filter FullyQualifiedName~BackendOptimizationSopTests ``` Expected: PASS. - [x] **Step 7: Commit** ```bash git add scripts/backend-optimization-sop.sh docs/backend-optimization-sop.md .gitignore tests/BuildFlow.Architecture.Tests git commit -m "chore: add backend optimization closure sop" ``` ### Task 10: Run The Clean Full SOP And Close The Audit **Files:** - Runtime output only: `artifacts/backend-optimization-sop//` - Modify documentation only if the run reveals a missing command or ambiguous failure path. - [ ] **Step 1: Confirm tracked worktree state and run the SOP from repository root** ```bash git status --short scripts/backend-optimization-sop.sh ``` Expected: the script starts a unique isolated Compose project, does not touch existing `infra-*` containers, and exits 0. - [ ] **Step 2: Verify the archived report** The report must contain: ```text all commands and stage durations build 0 warning / 0 error total passed / failed / skipped counts with skipped = 0 empty and upgrade migration evidence SSE source/content/DONE protocol evidence PostgreSQL/Redis/MongoDB/MinIO integration evidence concurrency and recovery test evidence query-count, remote-call-concurrency, Redis, context-window, transaction, and memory metrics failure log paths (empty on success, populated on failure) ``` - [ ] **Step 3: Run final debt scans independently of the SOP** ```bash rg -n "Task\.Run" src/modules/workflow src/BuildFlow.Host -g '*.cs' rg -n "InProcess(UserInfoService|FileStorageClient|AvatarUrlResolver)" src -g '*.cs' rg -n "DateTime\.Now" src -g '*.cs' dotnet test BuildFlow.slnx --no-restore dotnet build BuildFlow.slnx --no-restore ``` Expected: no forbidden debt matches; all tests PASS, 0 skipped; build reports 0 warning and 0 error. - [ ] **Step 4: Commit any run-discovered SOP corrections** ```bash git add scripts docs tests git commit -m "test: close backend optimization audit" ``` Skip this commit when the clean run requires no tracked correction. ## Self-Review - Spec coverage: Tasks 2-3 cover Endpoint CQRS; Tasks 4-5 remove all three bridges; Task 6 covers all explicit error codes; Task 7 covers all seven multi-type files; Task 1 enforces the four new source-wide guards while existing `HostArchitectureTests` retains Program/launchSettings/Api checks; Tasks 8-10 implement and execute the full SOP. - Placeholder scan: the plan contains no unresolved placeholders. Every task identifies exact files, commands, expected RED/GREEN outcomes, and commit boundaries. - Type consistency: `RAGStreamQuery` returns `RAGStreamEvent`; `GetUserSummariesQuery` returns `Dictionary`; `ResolveFilePreviewPathsQuery` returns File-owned preview DTOs; callers only inject `IMediator`. - Safety: all external tests accept explicit environment variables; Docker assigns random host ports; the unique Compose project and volumes are removed by an EXIT trap; no local Secrets file or developer database is written.