# BuildFlow Context And I/O Performance Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Bound RAG chat context and Redis append cost, remove S3 writes from file GET paths, and stream workflow Excel imports and exports without changing HTTP routes or response formats. **Architecture:** Chat generation uses one scoped context-window builder backed by a bounded PostgreSQL query, while Redis stores each cached message as one List element and appends atomically with TTL refresh. File size repair moves to a bounded Host scheduler that sends a File Application command. Workflow imports pass a bounded stream to an OpenXML reader, and exports page database rows into an `OpenXmlWriter` temporary file that the existing download endpoint streams and deletes. **Tech Stack:** .NET 10, EF Core 10, PostgreSQL 17, StackExchange.Redis 2.8, MediatR 12, DocumentFormat.OpenXml 3.1.1, ClosedXML 0.104.2, xUnit, FluentAssertions, Moq. --- ## Scope And Compatibility - Keep all existing chat, file, import, template, and export routes, request fields, response JSON, permissions, and SSE events unchanged. - Add `Rag:Chat:MaxContextMessages = 50`, `MaxContextTokens = 8000`, and `CacheTtlMinutes = 60`. - Chat context always contains the newest messages that fit both limits and is returned in chronological order. - Keep complete chat history in PostgreSQL and complete conversation-detail history in Redis after a cache miss. - Keep the Redis key exactly `chat:messages:{conversationId}`. - Add `File:SizeVerification` with a 60-second poll, 15-second startup delay, batch size 100, and maximum concurrency 4. - Add `SizeVerified`; all newly uploaded files set it to `true`. Migration marks historical `Size > 0` rows verified and leaves `Size == 0` pending. - Add workflow import limits: 50 MiB file, 100,000 data rows, 32,767 characters per cell, and export page size 500. - Keep template generation in ClosedXML because it contains only header and sample rows. Replace data import and export paths with OpenXML streaming. - Do not add cross-module service bridges. Host background work sends a File Application command through MediatR. ### Task 1: Record The Fourth-Batch Plan **Files:** - Create: `docs/superpowers/plans/2026-07-23-buildflow-context-and-io-performance.md` - [ ] **Step 1: Review the plan against design sections 7.1 through 7.4** Confirm bounded chat SQL, token trimming, shared generation behavior, Redis List atomic append, cache rebuild, `SizeVerified`, read-only GET paths, bounded S3 repair, pre-read upload rejection, OpenXML row limits, keyset export paging, and temporary-file cleanup all have RED tests and verification gates. - [ ] **Step 2: Commit the plan** ```bash git add docs/superpowers/plans/2026-07-23-buildflow-context-and-io-performance.md git commit -m "docs: plan context and io performance" ``` ### Task 2: Build One Bounded Chat Context Window **Files:** - Create: `src/modules/rag/Infrastructure/Cache/ChatOptions.cs` - Create: `src/modules/rag/Application/Chat/Services/ChatContextWindowBuilder.cs` - Create: `tests/BuildFlow.Rag.Tests/Chat/ChatContextWindowBuilderTests.cs` - Create: `tests/BuildFlow.Rag.Tests/Chat/RecordingChatAgent.cs` - Modify: `src/modules/rag/Application/Chat/Handlers/SendMessageCommandHandler.cs` - Modify: `src/modules/rag/Application/Chat/Handlers/StreamMessageCommandHandler.cs` - Modify: `src/modules/rag/Application/Chat/Handlers/EditMessageStreamCommandHandler.cs` - Modify: `src/modules/rag/Application/RAGApplicationModule.cs` - Modify: `src/modules/rag/Infrastructure/RAGInfrastructureModule.cs` - Modify: `src/modules/rag/Infrastructure/Persistence/Configurations/ChatMessageConfiguration.cs` - Create: `src/modules/rag/Infrastructure/Persistence/Migrations/20260724014757_AddChatContextIndex.cs` - Create: `src/modules/rag/Infrastructure/Persistence/Migrations/20260724014757_AddChatContextIndex.Designer.cs` - Modify: `src/modules/rag/Infrastructure/Persistence/Migrations/RagDbContextModelSnapshot.cs` - Modify: `src/BuildFlow.Host/appsettings.json` - Modify: existing RAG handler tests for the new builder dependency. - [ ] **Step 1: Write failing context-window tests** Cover these behaviors: ```csharp [Fact] public async Task ReturnsNewestMessagesWithinMessageLimitInChronologicalOrder() [Fact] public async Task StopsBeforeAddingAMessageThatWouldExceedTokenLimit() [Fact] public async Task SendStreamAndEditHandlersUseTheSameWindowBuilder() ``` Seed 100 messages, configure `MaxContextMessages = 6`, and assert the builder returns message 95 through 100 in ascending `(CreatedAt, Id)` order. For token trimming, use UTF-8 content whose estimated total crosses the limit and assert only the newest fitting suffix remains. Use `RecordingChatAgent` to prove all three handlers receive bounded history rather than all database rows. - [ ] **Step 2: Run and verify RED** ```bash dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter FullyQualifiedName~ChatContextWindowBuilderTests ``` Expected: compilation fails because `ChatOptions`, `ChatContextWindowBuilder`, and the new handler constructor parameters do not exist. - [ ] **Step 3: Add validated chat options** ```csharp public sealed class ChatOptions { public const string SectionName = "Rag:Chat"; public int MaxContextMessages { get; set; } = 50; public int MaxContextTokens { get; set; } = 8000; public int CacheTtlMinutes { get; set; } = 60; } ``` Bind with `AddOptions().Bind(...).Validate(...)` and reject non-positive values at startup. Register `ChatContextWindowBuilder` as scoped in `RAGApplicationModule`. - [ ] **Step 4: Implement one bounded builder** The builder must query only the latest configured rows: ```csharp var latest = await db.ChatMessages .AsNoTracking() .Where(x => x.ConversationId == conversationId) .OrderByDescending(x => x.CreatedAt) .ThenByDescending(x => x.Id) .Take(options.Value.MaxContextMessages) .Select(x => new ContextMessage(x.Role, x.Content, x.CreatedAt, x.Id)) .ToListAsync(ct); ``` Walk `latest` from newest to oldest, estimate each message with `Math.Max(1, (Encoding.UTF8.GetByteCount(content) + 3) / 4)`, stop before exceeding `MaxContextTokens`, reverse the accepted suffix, and return `Microsoft.Extensions.AI.ChatMessage` values. Keep helper records private or place each top-level type in its own file. - [ ] **Step 5: Reuse the builder in all generation handlers** After the user message is saved, both send handlers call `BuildAsync(conversation.Id, ct)`. After edit truncation is committed, the edit handler calls the same method. Delete all three unbounded `OrderBy(...).ToListAsync(...)` history queries. Conversation-detail loading remains complete and is not routed through the generation window. - [ ] **Step 6: Add the composite index and migration** Add: ```csharp builder.HasIndex(x => new { x.ConversationId, x.CreatedAt }); ``` Generate and inspect the migration: ```bash dotnet ef migrations add AddChatContextIndex \ --project src/modules/rag/Infrastructure \ --startup-project src/BuildFlow.Host \ --context RagDbContext ``` The migration must only replace the old `ConversationId` index with `ix_b_chat_messages_conversation_id_created_at`; it must not alter tables, columns, or foreign keys. - [ ] **Step 7: Verify and commit** ```bash dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~ChatContextWindowBuilderTests|FullyQualifiedName~ChatEndToEndTests|FullyQualifiedName~ChatStreamOwnershipTests" dotnet build src/modules/rag/Application/BuildFlow.Application.csproj --no-restore git add src/modules/rag src/BuildFlow.Host/appsettings.json tests/BuildFlow.Rag.Tests git commit -m "perf: bound rag chat context" ``` ### Task 3: Store Chat Cache As An Atomic Redis List **Files:** - Modify: `src/modules/rag/Infrastructure/Cache/ChatMessageCache.cs` - Modify: `tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj` - Create: `tests/BuildFlow.Rag.Tests/Chat/RedisChatMessageCacheCollection.cs` - Create: `tests/BuildFlow.Rag.Tests/Chat/RedisChatMessageCacheFixture.cs` - Create: `tests/BuildFlow.Rag.Tests/Chat/RedisChatMessageCacheTests.cs` - [ ] **Step 1: Write failing real-Redis tests** Use `BUILDFLOW_RAG_REDIS_TEST_CONNECTION` or `127.0.0.1:56379,abortConnect=false`. Every test uses a new conversation ID and deletes only `chat:messages:{id}` in cleanup. Cover: ```csharp [Fact] public async Task ConcurrentAppendsPreserveEveryMessage() [Fact] public async Task AppendAddsOneListElementAndRefreshesTtl() [Fact] public async Task SetAndGetRoundTripCompleteHistoryInOrder() ``` Run 100 concurrent appends and assert `ListLength == 100`, IDs are unique, Redis key type is `List`, and TTL is positive. Seed 500 elements, append one, and assert length becomes 501 without replacing the key. - [ ] **Step 2: Run and verify RED** ```bash dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter FullyQualifiedName~RedisChatMessageCacheTests ``` Expected: current String-based cache either reports key type `String` or loses concurrent appends. - [ ] **Step 3: Implement List reads and atomic writes** `GetMessagesAsync` uses `ListRangeAsync(key, 0, -1)` and deserializes each element independently. `AppendMessageAsync` executes one Lua script that performs `RPUSH` and `PEXPIRE`: ```lua redis.call('RPUSH', KEYS[1], ARGV[1]) redis.call('PEXPIRE', KEYS[1], ARGV[2]) return redis.call('LLEN', KEYS[1]) ``` `SetMessagesAsync` executes one Lua script that deletes the old key, pushes all serialized elements in order, and sets TTL. For an empty list, delete the key. Use `ChatOptions.CacheTtlMinutes`; do not hard-code one hour. Check cancellation before and after StackExchange.Redis calls. - [ ] **Step 4: Verify cache miss rebuild behavior** Run existing conversation-detail tests and add an assertion that a cache miss reads all database messages, rebuilds a Redis List, and a second read does not query PostgreSQL messages again. - [ ] **Step 5: Verify and commit** ```bash dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~RedisChatMessageCacheTests|FullyQualifiedName~ChatEndToEndTests" rg -n "StringGetAsync|StringSetAsync|GetMessagesAsync\(conversationId" src/modules/rag/Infrastructure/Cache/ChatMessageCache.cs git add src/modules/rag/Infrastructure/Cache tests/BuildFlow.Rag.Tests git commit -m "perf: append chat cache atomically" ``` Expected: tests pass and the scan has no output. ### Task 4: Mark Persisted File Sizes As Verified **Files:** - Modify: `src/modules/file/Domain/Entities/FileMeta/FileMetaEntity.cs` - Modify: `src/modules/file/Infrastructure/Persistence/Configurations/FileMetaConfiguration.cs` - Modify: `src/modules/file/Application/Features/Files/Handlers/UploadFileCommandHandler.cs` - Modify: `src/modules/file/Application/Features/Folders/Handlers/UploadToFolderCommandHandler.cs` - Modify: `src/modules/file/Application/Features/Multipart/Handlers/CompleteMultipartCommandHandler.cs` - Create: `src/modules/file/Infrastructure/Migrations/20260724093400_AddFileSizeVerified.cs` - Create: `src/modules/file/Infrastructure/Migrations/20260724093400_AddFileSizeVerified.Designer.cs` - Modify: `src/modules/file/Infrastructure/Migrations/FileDbContextModelSnapshot.cs` - Create: `tests/BuildFlow.File.Tests/FileSizeVerifiedTests.cs` - [ ] **Step 1: Write failing entity and upload tests** Assert direct upload, folder upload, and completed multipart records set `SizeVerified = true`, including a valid zero-byte upload. Add a model assertion that the pending-scan index is `(SizeVerified, CreatedAt)`. - [ ] **Step 2: Run and verify RED** ```bash dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter FullyQualifiedName~FileSizeVerifiedTests ``` Expected: compilation fails because `SizeVerified` does not exist. - [ ] **Step 3: Add the field, mapping, index, and upload assignments** ```csharp /// 对象大小是否已由上传结果或后台 S3 查询确认。 public bool SizeVerified { get; set; } ``` Map it with `.HasDefaultValue(false).HasComment("文件大小是否已确认")`, add `HasIndex(x => new { x.SizeVerified, x.CreatedAt })`, and set it to `true` in every new `FileMetaEntity` upload path. - [ ] **Step 4: Generate and inspect the migration** ```bash dotnet ef migrations add AddFileSizeVerified \ --project src/modules/file/Infrastructure \ --startup-project src/BuildFlow.Host \ --context FileDbContext ``` After `AddColumn`, include: ```csharp migrationBuilder.Sql("UPDATE b_files SET size_verified = TRUE WHERE size > 0"); ``` The migration must leave zero-byte historical records pending and create the composite index. - [ ] **Step 5: Verify and commit** ```bash dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter FullyQualifiedName~FileSizeVerifiedTests dotnet build src/modules/file/Application/File.Application.csproj --no-restore git add src/modules/file tests/BuildFlow.File.Tests/FileSizeVerifiedTests.cs git commit -m "feat: track verified file sizes" ``` ### Task 5: Move File Size Repair Off GET Paths **Files:** - Delete: `src/modules/file/Application/Common/FileSizeBackfillHelper.cs` - Modify: `src/modules/file/Application/Features/Folders/Handlers/GetRootFilesQueryHandler.cs` - Modify: `src/modules/file/Application/Features/Folders/Handlers/GetFolderQueryHandler.cs` - Modify: `src/modules/file/Application/Features/Folders/Handlers/GetFolderTreeQueryHandler.cs` - Create: `src/modules/file/Application/Common/FileSizeVerificationOptions.cs` - Create: `src/modules/file/Application/Features/Files/Commands/ProcessUnverifiedFileSizesCommand.cs` - Create: `src/modules/file/Application/Features/Files/Handlers/ProcessUnverifiedFileSizesCommandHandler.cs` - Create: `src/modules/file/Application/Features/Files/Services/FileSizeVerificationProcessor.cs` - Modify: `src/modules/file/Application/FileApplicationModule.cs` - Create: `src/BuildFlow.Host/Services/BackgroundServices/FileSizeVerificationService.cs` - Modify: `src/BuildFlow.Host/BuildFlowHostModule.cs` - Modify: `src/BuildFlow.Host/appsettings.json` - Create: `tests/BuildFlow.File.Tests/FileSizeVerificationProcessorTests.cs` - Modify: `tests/BuildFlow.File.Tests/FoldersBucketsSettingsTests.cs` - [ ] **Step 1: Write failing read-path and processor tests** Read tests seed `Size = 0, SizeVerified = false`, call all three GET handlers, and verify `IS3StorageService.GetObjectSizeAsync` and `SaveChangesAsync` side effects never occur. Processor tests cover: - a positive object size updates `Size` and marks verified; - a zero-byte object marks verified with `Size = 0`; - a missing object (`null`) logs one warning and marks verified; - a transient exception logs one warning and remains unverified; - batch size limits S3 calls; - an instrumented S3 fake observes no more than `MaxConcurrency` simultaneous calls. - [ ] **Step 2: Run and verify RED** ```bash dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter "FullyQualifiedName~FileSizeVerificationProcessorTests|FullyQualifiedName~FoldersBucketsSettingsTests" ``` Expected: GET handlers call S3 and the processor types are missing. - [ ] **Step 3: Make GET handlers database-only** Remove `IS3StorageService` constructor dependencies and every `FileSizeBackfillHelper.BackfillAsync` call. Delete the helper. Folder-size totals use persisted `Size`; pending historical values remain zero until the background processor repairs them. - [ ] **Step 4: Add the bounded module use case** The command is: ```csharp public sealed record ProcessUnverifiedFileSizesCommand( int BatchSize, int MaxConcurrency) : IRequest; ``` The processor validates `BatchSize` between 1 and 1000 and `MaxConcurrency` between 1 and 32, loads the oldest unverified rows with `Take(BatchSize)`, and performs S3 calls through `Parallel.ForEachAsync` using the configured maximum concurrency. Store outcomes in a thread-safe collection, mutate tracked EF entities only after all parallel S3 calls complete, and call `SaveChangesAsync` once. - [ ] **Step 5: Add the Host scheduler** Bind `File:SizeVerification`, validate positive intervals/batch/concurrency, register the processor scoped, and add a `BackgroundService` that creates a scope and sends `ProcessUnverifiedFileSizesCommand` through MediatR once per interval. Follow the cancellation handling used by `ImportJobProcessorService`. - [ ] **Step 6: Verify and commit** ```bash dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore rg -n "FileSizeBackfillHelper|GetObjectSizeAsync" src/modules/file/Application/Features/Folders git add src/modules/file src/BuildFlow.Host tests/BuildFlow.File.Tests git commit -m "perf: repair file sizes in background" ``` Expected: File tests pass and the scan has no output. ### Task 6: Stream And Bound Excel Imports **Files:** - Modify: `src/modules/workflow/Application/Workflow.Application.csproj` - Modify: `src/modules/workflow/Application/Reliability/WorkflowJobOptions.cs` - Modify: `src/modules/workflow/Application/BuildFlow/Common/Services/IFormSchemaExcelService.cs` - Modify: `src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs` - Modify: `src/modules/workflow/Application/BuildFlow/Imports/Commands/CreateImportJobCommand.cs` - Modify: `src/modules/workflow/Application/BuildFlow/Imports/Handlers/CreateImportJobCommandHandler.cs` - Modify: `src/modules/workflow/Application/BuildFlow/Imports/Services/ImportJobProcessor.cs` - Modify: `src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Imports/CreateImportJobEndpoint.cs` - Modify: `src/BuildFlow.Host/BuildFlowHostModule.cs` - Modify: `src/BuildFlow.Host/appsettings.json` - Modify: workflow Excel and import test fakes for stream signatures. - Create: `tests/BuildFlow.Workflow.Tests/Performance/TrackingReadStream.cs` - Create: `tests/BuildFlow.Workflow.Tests/Performance/OpenXmlImportBoundaryTests.cs` - [ ] **Step 1: Write failing upload and parser boundary tests** Desired command contract: ```csharp public sealed record CreateImportJobCommand( Guid FormDefinitionId, Stream FileStream, long FileLength, string OriginalFileName) : IRequest; ``` Cover pre-read rejection when `FileLength > MaxFileBytes`, successful stream-to-temp-file copy without `MemoryStream.ToArray`, rejection at row `MaxRows + 1`, rejection when a cell exceeds `MaxCellTextLength`, and parsing shared-string, inline-string, number, boolean, and blank cells. - [ ] **Step 2: Run and verify RED** ```bash dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter FullyQualifiedName~OpenXmlImportBoundaryTests ``` Expected: compilation fails for the stream command/parser contract and missing limits. - [ ] **Step 3: Add explicit OpenXML and import options** Add `DocumentFormat.OpenXml` version `3.1.1` directly to `Workflow.Application.csproj`. Extend `WorkflowJobOptions.ImportJobSection` with: ```csharp public long MaxFileBytes { get; set; } = 50 * 1024 * 1024; public int MaxRows { get; set; } = 100_000; public int MaxCellTextLength { get; set; } = 32_767; public int ExportPageSize { get; set; } = 500; ``` Validate all values, with `ExportPageSize <= 5000`. - [ ] **Step 4: Stop buffering uploads** The endpoint opens `IFormFile.OpenReadStream()` and sends it with `IFormFile.Length`. The handler rejects an oversized length with `TooLargeException` before the first read, copies to `{TempPath}/buildflow-imports/{jobId}.xlsx` with an 81,920-byte buffer, and deletes the temp file on database failure, cancellation, or copy failure. It must not allocate a full file byte array. - [ ] **Step 5: Implement the OpenXML row iterator** Change the parser contract to: ```csharp IEnumerable Parse( string schemaJson, Stream xlsxStream, int maxRows, int maxCellTextLength); ``` Open the workbook read-only with `SpreadsheetDocument.Open`, resolve shared strings once, and use `OpenXmlReader` over the first worksheet. Build the header map from row 1. Yield each subsequent row immediately, enforcing row count before yielding and cell text length before conversion. Throw `BusinessException(ErrorCode.TooLarge, ...)` for either limit. Do not use `XLWorkbook`, `ReadAllBytesAsync`, `ToArray()`, worksheet `Descendants()`, or a result list in the import path. - [ ] **Step 6: Open the job file as a stream** `ImportJobProcessor` uses `FileStream` with asynchronous/sequential-scan options and passes the configured row/cell limits to `Parse`. Keep existing committed-cursor skipping and batch transactions unchanged. - [ ] **Step 7: Verify and commit** ```bash dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter "FullyQualifiedName~OpenXmlImportBoundaryTests|FullyQualifiedName~ImportJobBatchingTests|FullyQualifiedName~ImportJobRecoveryTests|FullyQualifiedName~FormSchemaExcelServiceTests" rg -n "ReadAllBytesAsync|new XLWorkbook\(ms\)|\.ToArray\(\)" src/modules/workflow/Application/BuildFlow/Imports src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Imports git add src/modules/workflow src/BuildFlow.Host tests/BuildFlow.Workflow.Tests git commit -m "perf: stream bounded excel imports" ``` Expected: tests pass; any remaining `XLWorkbook`/`ToArray` match is only the small template-generation method, not import parsing. ### Task 7: Page And Stream Excel Exports **Files:** - Modify: `src/modules/workflow/Application/BuildFlow/Common/Services/IFormSchemaExcelService.cs` - Modify: `src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs` - Modify: `src/modules/workflow/Application/BuildFlow/Exports/Handlers/ExportFormDataQueryHandler.cs` - Modify: `src/modules/workflow/Application/BuildFlow/Exports/Queries/ExportFormDataResult.cs` - Modify: `src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Exports/DownloadExportEndpoint.cs` - Create: `tests/BuildFlow.Workflow.Tests/Performance/CountingExportPageInterceptor.cs` - Create: `tests/BuildFlow.Workflow.Tests/Performance/CancellingExportExcelService.cs` - Create: `tests/BuildFlow.Workflow.Tests/Performance/OpenXmlExportBoundaryTests.cs` - [ ] **Step 1: Write failing paged-export and cleanup tests** Cover 1,201 rows with page size 500 and assert exactly three `b_form_data` SELECT commands, stable output row count/order, and a result that contains a temporary file path rather than `byte[]`. Add cancellation and writer-exception tests that assert no `buildflow-export-*.xlsx` file remains. - [ ] **Step 2: Run and verify RED** ```bash dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter FullyQualifiedName~OpenXmlExportBoundaryTests ``` Expected: current handler performs one unbounded `ToListAsync`, returns `byte[]`, and has no temporary-file cleanup contract. - [ ] **Step 3: Add the sequential writer contract** ```csharp Task WriteExportAsync( string schemaJson, string formName, IAsyncEnumerable dataJsonRows, Stream output, CancellationToken ct); ``` Implement with `SpreadsheetDocument.Create` and `OpenXmlWriter`. Write workbook/sheet metadata, one header row, then each JSON row in order. Check cancellation at least once per row. Do not create `XLWorkbook`, a complete row list, or a final byte array. - [ ] **Step 4: Page rows by a stable keyset** The handler creates a unique file under `Path.GetTempPath()/buildflow-exports`, then exposes an async row iterator. Each page orders by `CreatedAt` and `Id`, filters after the last pair, selects only `CreatedAt`, `Id`, and `DataJson`, and uses `Take(ExportPageSize)`. The iterator yields page rows before querying the next page. Return: ```csharp public sealed record ExportFormDataResult( string FilePath, string FileName, string ContentType); ``` On writer failure or cancellation, the handler deletes the temporary file before rethrowing. - [ ] **Step 5: Stream and always delete in the endpoint** The endpoint keeps the current status, content type, and Content-Disposition. It opens the result path with asynchronous sequential-scan options, copies to `Response.Body`, flushes, and deletes the file in `finally`, including client cancellation and response-copy failure. - [ ] **Step 6: Verify and commit** ```bash dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter "FullyQualifiedName~OpenXmlExportBoundaryTests|FullyQualifiedName~FormSchemaExcelServiceTests" rg -n "ToListAsync|byte\[\]|MemoryStream|BuildExport" src/modules/workflow/Application/BuildFlow/Exports src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Exports git add src/modules/workflow/Application/BuildFlow src/BuildFlow.Host/Endpoints/Workflow tests/BuildFlow.Workflow.Tests git commit -m "perf: stream paged excel exports" ``` ### Task 8: Run The Fourth-Batch Verification Gate **Files:** - Verify only. - [ ] **Step 1: Run focused suites** ```bash dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~Chat|FullyQualifiedName~RedisChatMessageCacheTests" dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter "FullyQualifiedName~FileSize|FullyQualifiedName~FoldersBucketsSettingsTests" dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter "FullyQualifiedName~OpenXml|FullyQualifiedName~ImportJob|FullyQualifiedName~FormSchemaExcelServiceTests" ``` - [ ] **Step 2: Run affected projects** ```bash dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore ``` - [ ] **Step 3: Run full-solution gates** ```bash dotnet test BuildFlow.slnx --no-restore dotnet build BuildFlow.slnx --no-restore ``` Expected: every project passes, zero skipped, and build has 0 warnings and 0 errors. - [ ] **Step 4: Run mandatory static scans** ```bash rg -n "OrderBy\(.*CreatedAt\).*ToListAsync|StringGetAsync|StringSetAsync" src/modules/rag/Application/Chat src/modules/rag/Infrastructure/Cache rg -n "FileSizeBackfillHelper|GetObjectSizeAsync" src/modules/file/Application/Features/Folders rg -n "ReadAllBytesAsync|BuildExport\(|ToListAsync" src/modules/workflow/Application/BuildFlow/Imports src/modules/workflow/Application/BuildFlow/Exports rg -n "new XLWorkbook" src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs git diff --check git status --short ``` Expected: the first three scans have no output; the ClosedXML scan finds only template generation; diff check has no output; worktree is clean. - [ ] **Step 5: Record batch evidence** Report bounded context SQL row count and token count, Redis concurrent append count/key type/TTL, GET-path S3 call count, file repair maximum concurrency and missing/transient outcomes, import byte/row/cell limit results, absence of `ReadAllBytesAsync`, export page count for 1,201 rows, cancellation cleanup result, affected-project totals, full-solution total, build warning/error counts, and all fourth-batch commit hashes. Do not start the fifth-batch architecture plan until every command above is green. ## Self-Review Results - Spec coverage: design sections 7.1, 7.2, 7.3, and 7.4 each have RED tests, implementation tasks, commits, and deterministic gates. - Subsystem boundaries: chat changes remain inside RAG; file repair is owned by File Application and triggered from Host through MediatR; workflow Excel changes remain module-internal. - HTTP compatibility: routes, permissions, request field names, SSE events, content types, and download disposition remain unchanged. - Database impact: RAG adds one composite index; File adds one boolean column plus one composite index and an explicit historical-data update. - Type consistency: `ChatOptions`, `SizeVerified`, stream import signatures, and temporary-file export result types are introduced before all consumers are modified. - Placeholder scan: no deferred implementation markers, unnamed edge cases, or unspecified verification commands remain.