xiangning 3b918f5c0b perf: bound rag chat context

2026-07-24 09:50:16 +08:00

28 KiB

Raw Blame History

BuildFlow Context And I/O Performance Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Bound RAG chat context and Redis append cost, remove S3 writes from file GET paths, and stream workflow Excel imports and exports without changing HTTP routes or response formats.

Architecture: Chat generation uses one scoped context-window builder backed by a bounded PostgreSQL query, while Redis stores each cached message as one List element and appends atomically with TTL refresh. File size repair moves to a bounded Host scheduler that sends a File Application command. Workflow imports pass a bounded stream to an OpenXML reader, and exports page database rows into an OpenXmlWriter temporary file that the existing download endpoint streams and deletes.

Tech Stack: .NET 10, EF Core 10, PostgreSQL 17, StackExchange.Redis 2.8, MediatR 12, DocumentFormat.OpenXml 3.1.1, ClosedXML 0.104.2, xUnit, FluentAssertions, Moq.

Scope And Compatibility

Keep all existing chat, file, import, template, and export routes, request fields, response JSON, permissions, and SSE events unchanged.
Add Rag:Chat:MaxContextMessages = 50, MaxContextTokens = 8000, and CacheTtlMinutes = 60.
Chat context always contains the newest messages that fit both limits and is returned in chronological order.
Keep complete chat history in PostgreSQL and complete conversation-detail history in Redis after a cache miss.
Keep the Redis key exactly chat:messages:{conversationId}.
Add File:SizeVerification with a 60-second poll, 15-second startup delay, batch size 100, and maximum concurrency 4.
Add SizeVerified; all newly uploaded files set it to true. Migration marks historical Size > 0 rows verified and leaves Size == 0 pending.
Add workflow import limits: 50 MiB file, 100,000 data rows, 32,767 characters per cell, and export page size 500.
Keep template generation in ClosedXML because it contains only header and sample rows. Replace data import and export paths with OpenXML streaming.
Do not add cross-module service bridges. Host background work sends a File Application command through MediatR.

Task 1: Record The Fourth-Batch Plan

Files:

Create: docs/superpowers/plans/2026-07-23-buildflow-context-and-io-performance.md
Step 1: Review the plan against design sections 7.1 through 7.4

Confirm bounded chat SQL, token trimming, shared generation behavior, Redis List atomic append, cache rebuild, SizeVerified, read-only GET paths, bounded S3 repair, pre-read upload rejection, OpenXML row limits, keyset export paging, and temporary-file cleanup all have RED tests and verification gates.

Step 2: Commit the plan

git add docs/superpowers/plans/2026-07-23-buildflow-context-and-io-performance.md
git commit -m "docs: plan context and io performance"

Task 2: Build One Bounded Chat Context Window

Files:

Create: src/modules/rag/Infrastructure/Cache/ChatOptions.cs
Create: src/modules/rag/Application/Chat/Services/ChatContextWindowBuilder.cs
Create: tests/BuildFlow.Rag.Tests/Chat/ChatContextWindowBuilderTests.cs
Create: tests/BuildFlow.Rag.Tests/Chat/RecordingChatAgent.cs
Modify: src/modules/rag/Application/Chat/Handlers/SendMessageCommandHandler.cs
Modify: src/modules/rag/Application/Chat/Handlers/StreamMessageCommandHandler.cs
Modify: src/modules/rag/Application/Chat/Handlers/EditMessageStreamCommandHandler.cs
Modify: src/modules/rag/Application/RAGApplicationModule.cs
Modify: src/modules/rag/Infrastructure/RAGInfrastructureModule.cs
Modify: src/modules/rag/Infrastructure/Persistence/Configurations/ChatMessageConfiguration.cs
Create: src/modules/rag/Infrastructure/Persistence/Migrations/20260724014757_AddChatContextIndex.cs
Create: src/modules/rag/Infrastructure/Persistence/Migrations/20260724014757_AddChatContextIndex.Designer.cs
Modify: src/modules/rag/Infrastructure/Persistence/Migrations/RagDbContextModelSnapshot.cs
Modify: src/BuildFlow.Host/appsettings.json
Modify: existing RAG handler tests for the new builder dependency.
Step 1: Write failing context-window tests

Cover these behaviors:

[Fact]
public async Task ReturnsNewestMessagesWithinMessageLimitInChronologicalOrder()

[Fact]
public async Task StopsBeforeAddingAMessageThatWouldExceedTokenLimit()

[Fact]
public async Task SendStreamAndEditHandlersUseTheSameWindowBuilder()

Seed 100 messages, configure MaxContextMessages = 6, and assert the builder returns message 95 through 100 in ascending (CreatedAt, Id) order. For token trimming, use UTF-8 content whose estimated total crosses the limit and assert only the newest fitting suffix remains. Use RecordingChatAgent to prove all three handlers receive bounded history rather than all database rows.

Step 2: Run and verify RED

dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter FullyQualifiedName~ChatContextWindowBuilderTests

Expected: compilation fails because ChatOptions, ChatContextWindowBuilder, and the new handler constructor parameters do not exist.

Step 3: Add validated chat options

public sealed class ChatOptions
{
    public const string SectionName = "Rag:Chat";
    public int MaxContextMessages { get; set; } = 50;
    public int MaxContextTokens { get; set; } = 8000;
    public int CacheTtlMinutes { get; set; } = 60;
}

Bind with AddOptions<ChatOptions>().Bind(...).Validate(...) and reject non-positive values at startup. Register ChatContextWindowBuilder as scoped in RAGApplicationModule.

Step 4: Implement one bounded builder

The builder must query only the latest configured rows:

var latest = await db.ChatMessages
    .AsNoTracking()
    .Where(x => x.ConversationId == conversationId)
    .OrderByDescending(x => x.CreatedAt)
    .ThenByDescending(x => x.Id)
    .Take(options.Value.MaxContextMessages)
    .Select(x => new ContextMessage(x.Role, x.Content, x.CreatedAt, x.Id))
    .ToListAsync(ct);

Walk latest from newest to oldest, estimate each message with Math.Max(1, (Encoding.UTF8.GetByteCount(content) + 3) / 4), stop before exceeding MaxContextTokens, reverse the accepted suffix, and return Microsoft.Extensions.AI.ChatMessage values. Keep helper records private or place each top-level type in its own file.

Step 5: Reuse the builder in all generation handlers

After the user message is saved, both send handlers call BuildAsync(conversation.Id, ct). After edit truncation is committed, the edit handler calls the same method. Delete all three unbounded OrderBy(...).ToListAsync(...) history queries. Conversation-detail loading remains complete and is not routed through the generation window.

Step 6: Add the composite index and migration

Add:

builder.HasIndex(x => new { x.ConversationId, x.CreatedAt });

Generate and inspect the migration:

dotnet ef migrations add AddChatContextIndex \
  --project src/modules/rag/Infrastructure \
  --startup-project src/BuildFlow.Host \
  --context RagDbContext

The migration must only replace the old ConversationId index with ix_b_chat_messages_conversation_id_created_at; it must not alter tables, columns, or foreign keys.

Step 7: Verify and commit

dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~ChatContextWindowBuilderTests|FullyQualifiedName~ChatEndToEndTests|FullyQualifiedName~ChatStreamOwnershipTests"
dotnet build src/modules/rag/Application/BuildFlow.Application.csproj --no-restore
git add src/modules/rag src/BuildFlow.Host/appsettings.json tests/BuildFlow.Rag.Tests
git commit -m "perf: bound rag chat context"

Task 3: Store Chat Cache As An Atomic Redis List

Files:

Modify: src/modules/rag/Infrastructure/Cache/ChatMessageCache.cs
Modify: tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj
Create: tests/BuildFlow.Rag.Tests/Chat/RedisChatMessageCacheCollection.cs
Create: tests/BuildFlow.Rag.Tests/Chat/RedisChatMessageCacheFixture.cs
Create: tests/BuildFlow.Rag.Tests/Chat/RedisChatMessageCacheTests.cs
Step 1: Write failing real-Redis tests

Use BUILDFLOW_RAG_REDIS_TEST_CONNECTION or 127.0.0.1:56379,abortConnect=false. Every test uses a new conversation ID and deletes only chat:messages:{id} in cleanup.

Cover:

[Fact]
public async Task ConcurrentAppendsPreserveEveryMessage()

[Fact]
public async Task AppendAddsOneListElementAndRefreshesTtl()

[Fact]
public async Task SetAndGetRoundTripCompleteHistoryInOrder()

Run 100 concurrent appends and assert ListLength == 100, IDs are unique, Redis key type is List, and TTL is positive. Seed 500 elements, append one, and assert length becomes 501 without replacing the key.

Step 2: Run and verify RED

dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter FullyQualifiedName~RedisChatMessageCacheTests

Expected: current String-based cache either reports key type String or loses concurrent appends.

Step 3: Implement List reads and atomic writes

GetMessagesAsync uses ListRangeAsync(key, 0, -1) and deserializes each element independently. AppendMessageAsync executes one Lua script that performs RPUSH and PEXPIRE:

redis.call('RPUSH', KEYS[1], ARGV[1])
redis.call('PEXPIRE', KEYS[1], ARGV[2])
return redis.call('LLEN', KEYS[1])

SetMessagesAsync executes one Lua script that deletes the old key, pushes all serialized elements in order, and sets TTL. For an empty list, delete the key. Use ChatOptions.CacheTtlMinutes; do not hard-code one hour. Check cancellation before and after StackExchange.Redis calls.

Step 4: Verify cache miss rebuild behavior

Run existing conversation-detail tests and add an assertion that a cache miss reads all database messages, rebuilds a Redis List, and a second read does not query PostgreSQL messages again.

Step 5: Verify and commit

dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~RedisChatMessageCacheTests|FullyQualifiedName~ChatEndToEndTests"
rg -n "StringGetAsync|StringSetAsync|GetMessagesAsync\(conversationId" src/modules/rag/Infrastructure/Cache/ChatMessageCache.cs
git add src/modules/rag/Infrastructure/Cache tests/BuildFlow.Rag.Tests
git commit -m "perf: append chat cache atomically"

Expected: tests pass and the scan has no output.

Task 4: Mark Persisted File Sizes As Verified

Files:

Modify: src/modules/file/Domain/Entities/FileMeta/FileMetaEntity.cs
Modify: src/modules/file/Infrastructure/Persistence/Configurations/FileMetaConfiguration.cs
Modify: src/modules/file/Application/Features/Files/Handlers/UploadFileCommandHandler.cs
Modify: src/modules/file/Application/Features/Folders/Handlers/UploadToFolderCommandHandler.cs
Modify: src/modules/file/Application/Features/Multipart/Handlers/CompleteMultipartCommandHandler.cs
Create: src/modules/file/Infrastructure/Migrations/20260724093400_AddFileSizeVerified.cs
Create: src/modules/file/Infrastructure/Migrations/20260724093400_AddFileSizeVerified.Designer.cs
Modify: src/modules/file/Infrastructure/Migrations/FileDbContextModelSnapshot.cs
Create: tests/BuildFlow.File.Tests/FileSizeVerifiedTests.cs
Step 1: Write failing entity and upload tests

Assert direct upload, folder upload, and completed multipart records set SizeVerified = true, including a valid zero-byte upload. Add a model assertion that the pending-scan index is (SizeVerified, CreatedAt).

Step 2: Run and verify RED

dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter FullyQualifiedName~FileSizeVerifiedTests

Expected: compilation fails because SizeVerified does not exist.

Step 3: Add the field, mapping, index, and upload assignments

/// <summary>对象大小是否已由上传结果或后台 S3 查询确认。</summary>
public bool SizeVerified { get; set; }

Map it with .HasDefaultValue(false).HasComment("文件大小是否已确认"), add HasIndex(x => new { x.SizeVerified, x.CreatedAt }), and set it to true in every new FileMetaEntity upload path.

Step 4: Generate and inspect the migration

dotnet ef migrations add AddFileSizeVerified \
  --project src/modules/file/Infrastructure \
  --startup-project src/BuildFlow.Host \
  --context FileDbContext

After AddColumn, include:

migrationBuilder.Sql("UPDATE b_files SET size_verified = TRUE WHERE size > 0");

The migration must leave zero-byte historical records pending and create the composite index.

Step 5: Verify and commit

dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter FullyQualifiedName~FileSizeVerifiedTests
dotnet build src/modules/file/Application/File.Application.csproj --no-restore
git add src/modules/file tests/BuildFlow.File.Tests/FileSizeVerifiedTests.cs
git commit -m "feat: track verified file sizes"

Task 5: Move File Size Repair Off GET Paths

Files:

Delete: src/modules/file/Application/Common/FileSizeBackfillHelper.cs
Modify: src/modules/file/Application/Features/Folders/Handlers/GetRootFilesQueryHandler.cs
Modify: src/modules/file/Application/Features/Folders/Handlers/GetFolderQueryHandler.cs
Modify: src/modules/file/Application/Features/Folders/Handlers/GetFolderTreeQueryHandler.cs
Create: src/modules/file/Application/Common/FileSizeVerificationOptions.cs
Create: src/modules/file/Application/Features/Files/Commands/ProcessUnverifiedFileSizesCommand.cs
Create: src/modules/file/Application/Features/Files/Handlers/ProcessUnverifiedFileSizesCommandHandler.cs
Create: src/modules/file/Application/Features/Files/Services/FileSizeVerificationProcessor.cs
Modify: src/modules/file/Application/FileApplicationModule.cs
Create: src/BuildFlow.Host/Services/BackgroundServices/FileSizeVerificationService.cs
Modify: src/BuildFlow.Host/BuildFlowHostModule.cs
Modify: src/BuildFlow.Host/appsettings.json
Create: tests/BuildFlow.File.Tests/FileSizeVerificationProcessorTests.cs
Modify: tests/BuildFlow.File.Tests/FoldersBucketsSettingsTests.cs
Step 1: Write failing read-path and processor tests

Read tests seed Size = 0, SizeVerified = false, call all three GET handlers, and verify IS3StorageService.GetObjectSizeAsync and SaveChangesAsync side effects never occur. Processor tests cover:

a positive object size updates Size and marks verified;
a zero-byte object marks verified with Size = 0;
a missing object (null) logs one warning and marks verified;
a transient exception logs one warning and remains unverified;
batch size limits S3 calls;
an instrumented S3 fake observes no more than MaxConcurrency simultaneous calls.
Step 2: Run and verify RED

dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter "FullyQualifiedName~FileSizeVerificationProcessorTests|FullyQualifiedName~FoldersBucketsSettingsTests"

Expected: GET handlers call S3 and the processor types are missing.

Step 3: Make GET handlers database-only

Remove IS3StorageService constructor dependencies and every FileSizeBackfillHelper.BackfillAsync call. Delete the helper. Folder-size totals use persisted Size; pending historical values remain zero until the background processor repairs them.

Step 4: Add the bounded module use case

The command is:

public sealed record ProcessUnverifiedFileSizesCommand(
    int BatchSize,
    int MaxConcurrency) : IRequest<int>;

The processor validates BatchSize between 1 and 1000 and MaxConcurrency between 1 and 32, loads the oldest unverified rows with Take(BatchSize), and performs S3 calls through Parallel.ForEachAsync using the configured maximum concurrency. Store outcomes in a thread-safe collection, mutate tracked EF entities only after all parallel S3 calls complete, and call SaveChangesAsync once.

Step 5: Add the Host scheduler

Bind File:SizeVerification, validate positive intervals/batch/concurrency, register the processor scoped, and add a BackgroundService that creates a scope and sends ProcessUnverifiedFileSizesCommand through MediatR once per interval. Follow the cancellation handling used by ImportJobProcessorService.

Step 6: Verify and commit

dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore
rg -n "FileSizeBackfillHelper|GetObjectSizeAsync" src/modules/file/Application/Features/Folders
git add src/modules/file src/BuildFlow.Host tests/BuildFlow.File.Tests
git commit -m "perf: repair file sizes in background"

Expected: File tests pass and the scan has no output.

Task 6: Stream And Bound Excel Imports

Files:

Modify: src/modules/workflow/Application/Workflow.Application.csproj
Modify: src/modules/workflow/Application/Reliability/WorkflowJobOptions.cs
Modify: src/modules/workflow/Application/BuildFlow/Common/Services/IFormSchemaExcelService.cs
Modify: src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs
Modify: src/modules/workflow/Application/BuildFlow/Imports/Commands/CreateImportJobCommand.cs
Modify: src/modules/workflow/Application/BuildFlow/Imports/Handlers/CreateImportJobCommandHandler.cs
Modify: src/modules/workflow/Application/BuildFlow/Imports/Services/ImportJobProcessor.cs
Modify: src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Imports/CreateImportJobEndpoint.cs
Modify: src/BuildFlow.Host/BuildFlowHostModule.cs
Modify: src/BuildFlow.Host/appsettings.json
Modify: workflow Excel and import test fakes for stream signatures.
Create: tests/BuildFlow.Workflow.Tests/Performance/TrackingReadStream.cs
Create: tests/BuildFlow.Workflow.Tests/Performance/OpenXmlImportBoundaryTests.cs
Step 1: Write failing upload and parser boundary tests

Desired command contract:

public sealed record CreateImportJobCommand(
    Guid FormDefinitionId,
    Stream FileStream,
    long FileLength,
    string OriginalFileName) : IRequest<ImportJobDto>;

Cover pre-read rejection when FileLength > MaxFileBytes, successful stream-to-temp-file copy without MemoryStream.ToArray, rejection at row MaxRows + 1, rejection when a cell exceeds MaxCellTextLength, and parsing shared-string, inline-string, number, boolean, and blank cells.

Step 2: Run and verify RED

dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter FullyQualifiedName~OpenXmlImportBoundaryTests

Expected: compilation fails for the stream command/parser contract and missing limits.

Step 3: Add explicit OpenXML and import options

Add DocumentFormat.OpenXml version 3.1.1 directly to Workflow.Application.csproj. Extend WorkflowJobOptions.ImportJobSection with:

public long MaxFileBytes { get; set; } = 50 * 1024 * 1024;
public int MaxRows { get; set; } = 100_000;
public int MaxCellTextLength { get; set; } = 32_767;
public int ExportPageSize { get; set; } = 500;

Validate all values, with ExportPageSize <= 5000.

Step 4: Stop buffering uploads

The endpoint opens IFormFile.OpenReadStream() and sends it with IFormFile.Length. The handler rejects an oversized length with TooLargeException before the first read, copies to {TempPath}/buildflow-imports/{jobId}.xlsx with an 81,920-byte buffer, and deletes the temp file on database failure, cancellation, or copy failure. It must not allocate a full file byte array.

Step 5: Implement the OpenXML row iterator

Change the parser contract to:

IEnumerable<ExcelParsedRow> Parse(
    string schemaJson,
    Stream xlsxStream,
    int maxRows,
    int maxCellTextLength);

Open the workbook read-only with SpreadsheetDocument.Open, resolve shared strings once, and use OpenXmlReader over the first worksheet. Build the header map from row 1. Yield each subsequent row immediately, enforcing row count before yielding and cell text length before conversion. Throw BusinessException(ErrorCode.TooLarge, ...) for either limit. Do not use XLWorkbook, ReadAllBytesAsync, ToArray(), worksheet Descendants<Row>(), or a result list in the import path.

Step 6: Open the job file as a stream

ImportJobProcessor uses FileStream with asynchronous/sequential-scan options and passes the configured row/cell limits to Parse. Keep existing committed-cursor skipping and batch transactions unchanged.

Step 7: Verify and commit

dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter "FullyQualifiedName~OpenXmlImportBoundaryTests|FullyQualifiedName~ImportJobBatchingTests|FullyQualifiedName~ImportJobRecoveryTests|FullyQualifiedName~FormSchemaExcelServiceTests"
rg -n "ReadAllBytesAsync|new XLWorkbook\(ms\)|\.ToArray\(\)" src/modules/workflow/Application/BuildFlow/Imports src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Imports
git add src/modules/workflow src/BuildFlow.Host tests/BuildFlow.Workflow.Tests
git commit -m "perf: stream bounded excel imports"

Expected: tests pass; any remaining XLWorkbook/ToArray match is only the small template-generation method, not import parsing.

Task 7: Page And Stream Excel Exports

Files:

Modify: src/modules/workflow/Application/BuildFlow/Common/Services/IFormSchemaExcelService.cs
Modify: src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs
Modify: src/modules/workflow/Application/BuildFlow/Exports/Handlers/ExportFormDataQueryHandler.cs
Modify: src/modules/workflow/Application/BuildFlow/Exports/Queries/ExportFormDataResult.cs
Modify: src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Exports/DownloadExportEndpoint.cs
Create: tests/BuildFlow.Workflow.Tests/Performance/CountingExportPageInterceptor.cs
Create: tests/BuildFlow.Workflow.Tests/Performance/CancellingExportExcelService.cs
Create: tests/BuildFlow.Workflow.Tests/Performance/OpenXmlExportBoundaryTests.cs
Step 1: Write failing paged-export and cleanup tests

Cover 1,201 rows with page size 500 and assert exactly three b_form_data SELECT commands, stable output row count/order, and a result that contains a temporary file path rather than byte[]. Add cancellation and writer-exception tests that assert no buildflow-export-*.xlsx file remains.

Step 2: Run and verify RED

dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter FullyQualifiedName~OpenXmlExportBoundaryTests

Expected: current handler performs one unbounded ToListAsync, returns byte[], and has no temporary-file cleanup contract.

Step 3: Add the sequential writer contract

Task WriteExportAsync(
    string schemaJson,
    string formName,
    IAsyncEnumerable<string> dataJsonRows,
    Stream output,
    CancellationToken ct);

Implement with SpreadsheetDocument.Create and OpenXmlWriter. Write workbook/sheet metadata, one header row, then each JSON row in order. Check cancellation at least once per row. Do not create XLWorkbook, a complete row list, or a final byte array.

Step 4: Page rows by a stable keyset

The handler creates a unique file under Path.GetTempPath()/buildflow-exports, then exposes an async row iterator. Each page orders by CreatedAt and Id, filters after the last pair, selects only CreatedAt, Id, and DataJson, and uses Take(ExportPageSize). The iterator yields page rows before querying the next page.

Return:

public sealed record ExportFormDataResult(
    string FilePath,
    string FileName,
    string ContentType);

On writer failure or cancellation, the handler deletes the temporary file before rethrowing.

Step 5: Stream and always delete in the endpoint

The endpoint keeps the current status, content type, and Content-Disposition. It opens the result path with asynchronous sequential-scan options, copies to Response.Body, flushes, and deletes the file in finally, including client cancellation and response-copy failure.

Step 6: Verify and commit

dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter "FullyQualifiedName~OpenXmlExportBoundaryTests|FullyQualifiedName~FormSchemaExcelServiceTests"
rg -n "ToListAsync|byte\[\]|MemoryStream|BuildExport" src/modules/workflow/Application/BuildFlow/Exports src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Exports
git add src/modules/workflow/Application/BuildFlow src/BuildFlow.Host/Endpoints/Workflow tests/BuildFlow.Workflow.Tests
git commit -m "perf: stream paged excel exports"

Task 8: Run The Fourth-Batch Verification Gate

Files:

Verify only.
Step 1: Run focused suites

dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~Chat|FullyQualifiedName~RedisChatMessageCacheTests"
dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter "FullyQualifiedName~FileSize|FullyQualifiedName~FoldersBucketsSettingsTests"
dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter "FullyQualifiedName~OpenXml|FullyQualifiedName~ImportJob|FullyQualifiedName~FormSchemaExcelServiceTests"

Step 2: Run affected projects

dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore
dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore
dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore
dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore

Step 3: Run full-solution gates

dotnet test BuildFlow.slnx --no-restore
dotnet build BuildFlow.slnx --no-restore

Expected: every project passes, zero skipped, and build has 0 warnings and 0 errors.

Step 4: Run mandatory static scans

rg -n "OrderBy\(.*CreatedAt\).*ToListAsync|StringGetAsync|StringSetAsync" src/modules/rag/Application/Chat src/modules/rag/Infrastructure/Cache
rg -n "FileSizeBackfillHelper|GetObjectSizeAsync" src/modules/file/Application/Features/Folders
rg -n "ReadAllBytesAsync|BuildExport\(|ToListAsync" src/modules/workflow/Application/BuildFlow/Imports src/modules/workflow/Application/BuildFlow/Exports
rg -n "new XLWorkbook" src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs
git diff --check
git status --short

Expected: the first three scans have no output; the ClosedXML scan finds only template generation; diff check has no output; worktree is clean.

Step 5: Record batch evidence

Report bounded context SQL row count and token count, Redis concurrent append count/key type/TTL, GET-path S3 call count, file repair maximum concurrency and missing/transient outcomes, import byte/row/cell limit results, absence of ReadAllBytesAsync, export page count for 1,201 rows, cancellation cleanup result, affected-project totals, full-solution total, build warning/error counts, and all fourth-batch commit hashes.

Do not start the fifth-batch architecture plan until every command above is green.

Self-Review Results

Spec coverage: design sections 7.1, 7.2, 7.3, and 7.4 each have RED tests, implementation tasks, commits, and deterministic gates.
Subsystem boundaries: chat changes remain inside RAG; file repair is owned by File Application and triggered from Host through MediatR; workflow Excel changes remain module-internal.
HTTP compatibility: routes, permissions, request field names, SSE events, content types, and download disposition remain unchanged.
Database impact: RAG adds one composite index; File adds one boolean column plus one composite index and an explicit historical-data update.
Type consistency: ChatOptions, SizeVerified, stream import signatures, and temporary-file export result types are introduced before all consumers are modified.
Placeholder scan: no deferred implementation markers, unnamed edge cases, or unspecified verification commands remain.

28 KiB Raw Blame History

BuildFlow Context And I/O Performance Implementation Plan

Scope And Compatibility

Task 1: Record The Fourth-Batch Plan

Task 2: Build One Bounded Chat Context Window

Task 3: Store Chat Cache As An Atomic Redis List

Task 4: Mark Persisted File Sizes As Verified

Task 5: Move File Size Repair Off GET Paths

Task 6: Stream And Bound Excel Imports

Task 7: Page And Stream Excel Exports

Task 8: Run The Fourth-Batch Verification Gate

Self-Review Results

28 KiB

Raw Blame History