549 lines
28 KiB
Markdown
549 lines
28 KiB
Markdown
# BuildFlow Context And I/O Performance Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Bound RAG chat context and Redis append cost, remove S3 writes from file GET paths, and stream workflow Excel imports and exports without changing HTTP routes or response formats.
|
|
|
|
**Architecture:** Chat generation uses one scoped context-window builder backed by a bounded PostgreSQL query, while Redis stores each cached message as one List element and appends atomically with TTL refresh. File size repair moves to a bounded Host scheduler that sends a File Application command. Workflow imports pass a bounded stream to an OpenXML reader, and exports page database rows into an `OpenXmlWriter` temporary file that the existing download endpoint streams and deletes.
|
|
|
|
**Tech Stack:** .NET 10, EF Core 10, PostgreSQL 17, StackExchange.Redis 2.8, MediatR 12, DocumentFormat.OpenXml 3.1.1, ClosedXML 0.104.2, xUnit, FluentAssertions, Moq.
|
|
|
|
---
|
|
|
|
## Scope And Compatibility
|
|
|
|
- Keep all existing chat, file, import, template, and export routes, request fields, response JSON, permissions, and SSE events unchanged.
|
|
- Add `Rag:Chat:MaxContextMessages = 50`, `MaxContextTokens = 8000`, and `CacheTtlMinutes = 60`.
|
|
- Chat context always contains the newest messages that fit both limits and is returned in chronological order.
|
|
- Keep complete chat history in PostgreSQL and complete conversation-detail history in Redis after a cache miss.
|
|
- Keep the Redis key exactly `chat:messages:{conversationId}`.
|
|
- Add `File:SizeVerification` with a 60-second poll, 15-second startup delay, batch size 100, and maximum concurrency 4.
|
|
- Add `SizeVerified`; all newly uploaded files set it to `true`. Migration marks historical `Size > 0` rows verified and leaves `Size == 0` pending.
|
|
- Add workflow import limits: 50 MiB file, 100,000 data rows, 32,767 characters per cell, and export page size 500.
|
|
- Keep template generation in ClosedXML because it contains only header and sample rows. Replace data import and export paths with OpenXML streaming.
|
|
- Do not add cross-module service bridges. Host background work sends a File Application command through MediatR.
|
|
|
|
### Task 1: Record The Fourth-Batch Plan
|
|
|
|
**Files:**
|
|
- Create: `docs/superpowers/plans/2026-07-23-buildflow-context-and-io-performance.md`
|
|
|
|
- [ ] **Step 1: Review the plan against design sections 7.1 through 7.4**
|
|
|
|
Confirm bounded chat SQL, token trimming, shared generation behavior, Redis List atomic append, cache rebuild, `SizeVerified`, read-only GET paths, bounded S3 repair, pre-read upload rejection, OpenXML row limits, keyset export paging, and temporary-file cleanup all have RED tests and verification gates.
|
|
|
|
- [ ] **Step 2: Commit the plan**
|
|
|
|
```bash
|
|
git add docs/superpowers/plans/2026-07-23-buildflow-context-and-io-performance.md
|
|
git commit -m "docs: plan context and io performance"
|
|
```
|
|
|
|
### Task 2: Build One Bounded Chat Context Window
|
|
|
|
**Files:**
|
|
- Create: `src/modules/rag/Infrastructure/Cache/ChatOptions.cs`
|
|
- Create: `src/modules/rag/Application/Chat/Services/ChatContextWindowBuilder.cs`
|
|
- Create: `tests/BuildFlow.Rag.Tests/Chat/ChatContextWindowBuilderTests.cs`
|
|
- Create: `tests/BuildFlow.Rag.Tests/Chat/RecordingChatAgent.cs`
|
|
- Modify: `src/modules/rag/Application/Chat/Handlers/SendMessageCommandHandler.cs`
|
|
- Modify: `src/modules/rag/Application/Chat/Handlers/StreamMessageCommandHandler.cs`
|
|
- Modify: `src/modules/rag/Application/Chat/Handlers/EditMessageStreamCommandHandler.cs`
|
|
- Modify: `src/modules/rag/Application/RAGApplicationModule.cs`
|
|
- Modify: `src/modules/rag/Infrastructure/RAGInfrastructureModule.cs`
|
|
- Modify: `src/modules/rag/Infrastructure/Persistence/Configurations/ChatMessageConfiguration.cs`
|
|
- Create: `src/modules/rag/Infrastructure/Persistence/Migrations/20260724014757_AddChatContextIndex.cs`
|
|
- Create: `src/modules/rag/Infrastructure/Persistence/Migrations/20260724014757_AddChatContextIndex.Designer.cs`
|
|
- Modify: `src/modules/rag/Infrastructure/Persistence/Migrations/RagDbContextModelSnapshot.cs`
|
|
- Modify: `src/BuildFlow.Host/appsettings.json`
|
|
- Modify: existing RAG handler tests for the new builder dependency.
|
|
|
|
- [ ] **Step 1: Write failing context-window tests**
|
|
|
|
Cover these behaviors:
|
|
|
|
```csharp
|
|
[Fact]
|
|
public async Task ReturnsNewestMessagesWithinMessageLimitInChronologicalOrder()
|
|
|
|
[Fact]
|
|
public async Task StopsBeforeAddingAMessageThatWouldExceedTokenLimit()
|
|
|
|
[Fact]
|
|
public async Task SendStreamAndEditHandlersUseTheSameWindowBuilder()
|
|
```
|
|
|
|
Seed 100 messages, configure `MaxContextMessages = 6`, and assert the builder returns message 95 through 100 in ascending `(CreatedAt, Id)` order. For token trimming, use UTF-8 content whose estimated total crosses the limit and assert only the newest fitting suffix remains. Use `RecordingChatAgent` to prove all three handlers receive bounded history rather than all database rows.
|
|
|
|
- [ ] **Step 2: Run and verify RED**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter FullyQualifiedName~ChatContextWindowBuilderTests
|
|
```
|
|
|
|
Expected: compilation fails because `ChatOptions`, `ChatContextWindowBuilder`, and the new handler constructor parameters do not exist.
|
|
|
|
- [ ] **Step 3: Add validated chat options**
|
|
|
|
```csharp
|
|
public sealed class ChatOptions
|
|
{
|
|
public const string SectionName = "Rag:Chat";
|
|
public int MaxContextMessages { get; set; } = 50;
|
|
public int MaxContextTokens { get; set; } = 8000;
|
|
public int CacheTtlMinutes { get; set; } = 60;
|
|
}
|
|
```
|
|
|
|
Bind with `AddOptions<ChatOptions>().Bind(...).Validate(...)` and reject non-positive values at startup. Register `ChatContextWindowBuilder` as scoped in `RAGApplicationModule`.
|
|
|
|
- [ ] **Step 4: Implement one bounded builder**
|
|
|
|
The builder must query only the latest configured rows:
|
|
|
|
```csharp
|
|
var latest = await db.ChatMessages
|
|
.AsNoTracking()
|
|
.Where(x => x.ConversationId == conversationId)
|
|
.OrderByDescending(x => x.CreatedAt)
|
|
.ThenByDescending(x => x.Id)
|
|
.Take(options.Value.MaxContextMessages)
|
|
.Select(x => new ContextMessage(x.Role, x.Content, x.CreatedAt, x.Id))
|
|
.ToListAsync(ct);
|
|
```
|
|
|
|
Walk `latest` from newest to oldest, estimate each message with `Math.Max(1, (Encoding.UTF8.GetByteCount(content) + 3) / 4)`, stop before exceeding `MaxContextTokens`, reverse the accepted suffix, and return `Microsoft.Extensions.AI.ChatMessage` values. Keep helper records private or place each top-level type in its own file.
|
|
|
|
- [ ] **Step 5: Reuse the builder in all generation handlers**
|
|
|
|
After the user message is saved, both send handlers call `BuildAsync(conversation.Id, ct)`. After edit truncation is committed, the edit handler calls the same method. Delete all three unbounded `OrderBy(...).ToListAsync(...)` history queries. Conversation-detail loading remains complete and is not routed through the generation window.
|
|
|
|
- [ ] **Step 6: Add the composite index and migration**
|
|
|
|
Add:
|
|
|
|
```csharp
|
|
builder.HasIndex(x => new { x.ConversationId, x.CreatedAt });
|
|
```
|
|
|
|
Generate and inspect the migration:
|
|
|
|
```bash
|
|
dotnet ef migrations add AddChatContextIndex \
|
|
--project src/modules/rag/Infrastructure \
|
|
--startup-project src/BuildFlow.Host \
|
|
--context RagDbContext
|
|
```
|
|
|
|
The migration must only replace the old `ConversationId` index with
|
|
`ix_b_chat_messages_conversation_id_created_at`; it must not alter tables, columns, or foreign keys.
|
|
|
|
- [ ] **Step 7: Verify and commit**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~ChatContextWindowBuilderTests|FullyQualifiedName~ChatEndToEndTests|FullyQualifiedName~ChatStreamOwnershipTests"
|
|
dotnet build src/modules/rag/Application/BuildFlow.Application.csproj --no-restore
|
|
git add src/modules/rag src/BuildFlow.Host/appsettings.json tests/BuildFlow.Rag.Tests
|
|
git commit -m "perf: bound rag chat context"
|
|
```
|
|
|
|
### Task 3: Store Chat Cache As An Atomic Redis List
|
|
|
|
**Files:**
|
|
- Modify: `src/modules/rag/Infrastructure/Cache/ChatMessageCache.cs`
|
|
- Modify: `tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj`
|
|
- Create: `tests/BuildFlow.Rag.Tests/Chat/RedisChatMessageCacheCollection.cs`
|
|
- Create: `tests/BuildFlow.Rag.Tests/Chat/RedisChatMessageCacheFixture.cs`
|
|
- Create: `tests/BuildFlow.Rag.Tests/Chat/RedisChatMessageCacheTests.cs`
|
|
|
|
- [ ] **Step 1: Write failing real-Redis tests**
|
|
|
|
Use `BUILDFLOW_RAG_REDIS_TEST_CONNECTION` or `127.0.0.1:56379,abortConnect=false`. Every test uses a new conversation ID and deletes only `chat:messages:{id}` in cleanup.
|
|
|
|
Cover:
|
|
|
|
```csharp
|
|
[Fact]
|
|
public async Task ConcurrentAppendsPreserveEveryMessage()
|
|
|
|
[Fact]
|
|
public async Task AppendAddsOneListElementAndRefreshesTtl()
|
|
|
|
[Fact]
|
|
public async Task SetAndGetRoundTripCompleteHistoryInOrder()
|
|
```
|
|
|
|
Run 100 concurrent appends and assert `ListLength == 100`, IDs are unique, Redis key type is `List`, and TTL is positive. Seed 500 elements, append one, and assert length becomes 501 without replacing the key.
|
|
|
|
- [ ] **Step 2: Run and verify RED**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter FullyQualifiedName~RedisChatMessageCacheTests
|
|
```
|
|
|
|
Expected: current String-based cache either reports key type `String` or loses concurrent appends.
|
|
|
|
- [ ] **Step 3: Implement List reads and atomic writes**
|
|
|
|
`GetMessagesAsync` uses `ListRangeAsync(key, 0, -1)` and deserializes each element independently. `AppendMessageAsync` executes one Lua script that performs `RPUSH` and `PEXPIRE`:
|
|
|
|
```lua
|
|
redis.call('RPUSH', KEYS[1], ARGV[1])
|
|
redis.call('PEXPIRE', KEYS[1], ARGV[2])
|
|
return redis.call('LLEN', KEYS[1])
|
|
```
|
|
|
|
`SetMessagesAsync` executes one Lua script that deletes the old key, pushes all serialized elements in order, and sets TTL. For an empty list, delete the key. Use `ChatOptions.CacheTtlMinutes`; do not hard-code one hour. Check cancellation before and after StackExchange.Redis calls.
|
|
|
|
- [ ] **Step 4: Verify cache miss rebuild behavior**
|
|
|
|
Run existing conversation-detail tests and add an assertion that a cache miss reads all database messages, rebuilds a Redis List, and a second read does not query PostgreSQL messages again.
|
|
|
|
- [ ] **Step 5: Verify and commit**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~RedisChatMessageCacheTests|FullyQualifiedName~ChatEndToEndTests"
|
|
rg -n "StringGetAsync|StringSetAsync|GetMessagesAsync\(conversationId" src/modules/rag/Infrastructure/Cache/ChatMessageCache.cs
|
|
git add src/modules/rag/Infrastructure/Cache tests/BuildFlow.Rag.Tests
|
|
git commit -m "perf: append chat cache atomically"
|
|
```
|
|
|
|
Expected: tests pass and the scan has no output.
|
|
|
|
### Task 4: Mark Persisted File Sizes As Verified
|
|
|
|
**Files:**
|
|
- Modify: `src/modules/file/Domain/Entities/FileMeta/FileMetaEntity.cs`
|
|
- Modify: `src/modules/file/Infrastructure/Persistence/Configurations/FileMetaConfiguration.cs`
|
|
- Modify: `src/modules/file/Application/Features/Files/Handlers/UploadFileCommandHandler.cs`
|
|
- Modify: `src/modules/file/Application/Features/Folders/Handlers/UploadToFolderCommandHandler.cs`
|
|
- Modify: `src/modules/file/Application/Features/Multipart/Handlers/CompleteMultipartCommandHandler.cs`
|
|
- Create: `src/modules/file/Infrastructure/Migrations/20260724093400_AddFileSizeVerified.cs`
|
|
- Create: `src/modules/file/Infrastructure/Migrations/20260724093400_AddFileSizeVerified.Designer.cs`
|
|
- Modify: `src/modules/file/Infrastructure/Migrations/FileDbContextModelSnapshot.cs`
|
|
- Create: `tests/BuildFlow.File.Tests/FileSizeVerifiedTests.cs`
|
|
|
|
- [ ] **Step 1: Write failing entity and upload tests**
|
|
|
|
Assert direct upload, folder upload, and completed multipart records set `SizeVerified = true`, including a valid zero-byte upload. Add a model assertion that the pending-scan index is `(SizeVerified, CreatedAt)`.
|
|
|
|
- [ ] **Step 2: Run and verify RED**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter FullyQualifiedName~FileSizeVerifiedTests
|
|
```
|
|
|
|
Expected: compilation fails because `SizeVerified` does not exist.
|
|
|
|
- [ ] **Step 3: Add the field, mapping, index, and upload assignments**
|
|
|
|
```csharp
|
|
/// <summary>对象大小是否已由上传结果或后台 S3 查询确认。</summary>
|
|
public bool SizeVerified { get; set; }
|
|
```
|
|
|
|
Map it with `.HasDefaultValue(false).HasComment("文件大小是否已确认")`, add `HasIndex(x => new { x.SizeVerified, x.CreatedAt })`, and set it to `true` in every new `FileMetaEntity` upload path.
|
|
|
|
- [ ] **Step 4: Generate and inspect the migration**
|
|
|
|
```bash
|
|
dotnet ef migrations add AddFileSizeVerified \
|
|
--project src/modules/file/Infrastructure \
|
|
--startup-project src/BuildFlow.Host \
|
|
--context FileDbContext
|
|
```
|
|
|
|
After `AddColumn`, include:
|
|
|
|
```csharp
|
|
migrationBuilder.Sql("UPDATE b_files SET size_verified = TRUE WHERE size > 0");
|
|
```
|
|
|
|
The migration must leave zero-byte historical records pending and create the composite index.
|
|
|
|
- [ ] **Step 5: Verify and commit**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter FullyQualifiedName~FileSizeVerifiedTests
|
|
dotnet build src/modules/file/Application/File.Application.csproj --no-restore
|
|
git add src/modules/file tests/BuildFlow.File.Tests/FileSizeVerifiedTests.cs
|
|
git commit -m "feat: track verified file sizes"
|
|
```
|
|
|
|
### Task 5: Move File Size Repair Off GET Paths
|
|
|
|
**Files:**
|
|
- Delete: `src/modules/file/Application/Common/FileSizeBackfillHelper.cs`
|
|
- Modify: `src/modules/file/Application/Features/Folders/Handlers/GetRootFilesQueryHandler.cs`
|
|
- Modify: `src/modules/file/Application/Features/Folders/Handlers/GetFolderQueryHandler.cs`
|
|
- Modify: `src/modules/file/Application/Features/Folders/Handlers/GetFolderTreeQueryHandler.cs`
|
|
- Create: `src/modules/file/Application/Common/FileSizeVerificationOptions.cs`
|
|
- Create: `src/modules/file/Application/Features/Files/Commands/ProcessUnverifiedFileSizesCommand.cs`
|
|
- Create: `src/modules/file/Application/Features/Files/Handlers/ProcessUnverifiedFileSizesCommandHandler.cs`
|
|
- Create: `src/modules/file/Application/Features/Files/Services/FileSizeVerificationProcessor.cs`
|
|
- Modify: `src/modules/file/Application/FileApplicationModule.cs`
|
|
- Create: `src/BuildFlow.Host/Services/BackgroundServices/FileSizeVerificationService.cs`
|
|
- Modify: `src/BuildFlow.Host/BuildFlowHostModule.cs`
|
|
- Modify: `src/BuildFlow.Host/appsettings.json`
|
|
- Create: `tests/BuildFlow.File.Tests/FileSizeVerificationProcessorTests.cs`
|
|
- Modify: `tests/BuildFlow.File.Tests/FoldersBucketsSettingsTests.cs`
|
|
|
|
- [ ] **Step 1: Write failing read-path and processor tests**
|
|
|
|
Read tests seed `Size = 0, SizeVerified = false`, call all three GET handlers, and verify `IS3StorageService.GetObjectSizeAsync` and `SaveChangesAsync` side effects never occur. Processor tests cover:
|
|
|
|
- a positive object size updates `Size` and marks verified;
|
|
- a zero-byte object marks verified with `Size = 0`;
|
|
- a missing object (`null`) logs one warning and marks verified;
|
|
- a transient exception logs one warning and remains unverified;
|
|
- batch size limits S3 calls;
|
|
- an instrumented S3 fake observes no more than `MaxConcurrency` simultaneous calls.
|
|
|
|
- [ ] **Step 2: Run and verify RED**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter "FullyQualifiedName~FileSizeVerificationProcessorTests|FullyQualifiedName~FoldersBucketsSettingsTests"
|
|
```
|
|
|
|
Expected: GET handlers call S3 and the processor types are missing.
|
|
|
|
- [ ] **Step 3: Make GET handlers database-only**
|
|
|
|
Remove `IS3StorageService` constructor dependencies and every `FileSizeBackfillHelper.BackfillAsync` call. Delete the helper. Folder-size totals use persisted `Size`; pending historical values remain zero until the background processor repairs them.
|
|
|
|
- [ ] **Step 4: Add the bounded module use case**
|
|
|
|
The command is:
|
|
|
|
```csharp
|
|
public sealed record ProcessUnverifiedFileSizesCommand(
|
|
int BatchSize,
|
|
int MaxConcurrency) : IRequest<int>;
|
|
```
|
|
|
|
The processor validates `BatchSize` between 1 and 1000 and `MaxConcurrency` between 1 and 32, loads the oldest unverified rows with `Take(BatchSize)`, and performs S3 calls through `Parallel.ForEachAsync` using the configured maximum concurrency. Store outcomes in a thread-safe collection, mutate tracked EF entities only after all parallel S3 calls complete, and call `SaveChangesAsync` once.
|
|
|
|
- [ ] **Step 5: Add the Host scheduler**
|
|
|
|
Bind `File:SizeVerification`, validate positive intervals/batch/concurrency, register the processor scoped, and add a `BackgroundService` that creates a scope and sends `ProcessUnverifiedFileSizesCommand` through MediatR once per interval. Follow the cancellation handling used by `ImportJobProcessorService`.
|
|
|
|
- [ ] **Step 6: Verify and commit**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore
|
|
rg -n "FileSizeBackfillHelper|GetObjectSizeAsync" src/modules/file/Application/Features/Folders
|
|
git add src/modules/file src/BuildFlow.Host tests/BuildFlow.File.Tests
|
|
git commit -m "perf: repair file sizes in background"
|
|
```
|
|
|
|
Expected: File tests pass and the scan has no output.
|
|
|
|
### Task 6: Stream And Bound Excel Imports
|
|
|
|
**Files:**
|
|
- Modify: `src/modules/workflow/Application/Workflow.Application.csproj`
|
|
- Modify: `src/modules/workflow/Application/Reliability/WorkflowJobOptions.cs`
|
|
- Modify: `src/modules/workflow/Application/BuildFlow/Common/Services/IFormSchemaExcelService.cs`
|
|
- Modify: `src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs`
|
|
- Modify: `src/modules/workflow/Application/BuildFlow/Imports/Commands/CreateImportJobCommand.cs`
|
|
- Modify: `src/modules/workflow/Application/BuildFlow/Imports/Handlers/CreateImportJobCommandHandler.cs`
|
|
- Modify: `src/modules/workflow/Application/BuildFlow/Imports/Services/ImportJobProcessor.cs`
|
|
- Modify: `src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Imports/CreateImportJobEndpoint.cs`
|
|
- Modify: `src/BuildFlow.Host/BuildFlowHostModule.cs`
|
|
- Modify: `src/BuildFlow.Host/appsettings.json`
|
|
- Modify: workflow Excel and import test fakes for stream signatures.
|
|
- Create: `tests/BuildFlow.Workflow.Tests/Performance/TrackingReadStream.cs`
|
|
- Create: `tests/BuildFlow.Workflow.Tests/Performance/OpenXmlImportBoundaryTests.cs`
|
|
|
|
- [ ] **Step 1: Write failing upload and parser boundary tests**
|
|
|
|
Desired command contract:
|
|
|
|
```csharp
|
|
public sealed record CreateImportJobCommand(
|
|
Guid FormDefinitionId,
|
|
Stream FileStream,
|
|
long FileLength,
|
|
string OriginalFileName) : IRequest<ImportJobDto>;
|
|
```
|
|
|
|
Cover pre-read rejection when `FileLength > MaxFileBytes`, successful stream-to-temp-file copy without `MemoryStream.ToArray`, rejection at row `MaxRows + 1`, rejection when a cell exceeds `MaxCellTextLength`, and parsing shared-string, inline-string, number, boolean, and blank cells.
|
|
|
|
- [ ] **Step 2: Run and verify RED**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter FullyQualifiedName~OpenXmlImportBoundaryTests
|
|
```
|
|
|
|
Expected: compilation fails for the stream command/parser contract and missing limits.
|
|
|
|
- [ ] **Step 3: Add explicit OpenXML and import options**
|
|
|
|
Add `DocumentFormat.OpenXml` version `3.1.1` directly to `Workflow.Application.csproj`. Extend `WorkflowJobOptions.ImportJobSection` with:
|
|
|
|
```csharp
|
|
public long MaxFileBytes { get; set; } = 50 * 1024 * 1024;
|
|
public int MaxRows { get; set; } = 100_000;
|
|
public int MaxCellTextLength { get; set; } = 32_767;
|
|
public int ExportPageSize { get; set; } = 500;
|
|
```
|
|
|
|
Validate all values, with `ExportPageSize <= 5000`.
|
|
|
|
- [ ] **Step 4: Stop buffering uploads**
|
|
|
|
The endpoint opens `IFormFile.OpenReadStream()` and sends it with `IFormFile.Length`. The handler rejects an oversized length with `TooLargeException` before the first read, copies to `{TempPath}/buildflow-imports/{jobId}.xlsx` with an 81,920-byte buffer, and deletes the temp file on database failure, cancellation, or copy failure. It must not allocate a full file byte array.
|
|
|
|
- [ ] **Step 5: Implement the OpenXML row iterator**
|
|
|
|
Change the parser contract to:
|
|
|
|
```csharp
|
|
IEnumerable<ExcelParsedRow> Parse(
|
|
string schemaJson,
|
|
Stream xlsxStream,
|
|
int maxRows,
|
|
int maxCellTextLength);
|
|
```
|
|
|
|
Open the workbook read-only with `SpreadsheetDocument.Open`, resolve shared strings once, and use `OpenXmlReader` over the first worksheet. Build the header map from row 1. Yield each subsequent row immediately, enforcing row count before yielding and cell text length before conversion. Throw `BusinessException(ErrorCode.TooLarge, ...)` for either limit. Do not use `XLWorkbook`, `ReadAllBytesAsync`, `ToArray()`, worksheet `Descendants<Row>()`, or a result list in the import path.
|
|
|
|
- [ ] **Step 6: Open the job file as a stream**
|
|
|
|
`ImportJobProcessor` uses `FileStream` with asynchronous/sequential-scan options and passes the configured row/cell limits to `Parse`. Keep existing committed-cursor skipping and batch transactions unchanged.
|
|
|
|
- [ ] **Step 7: Verify and commit**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter "FullyQualifiedName~OpenXmlImportBoundaryTests|FullyQualifiedName~ImportJobBatchingTests|FullyQualifiedName~ImportJobRecoveryTests|FullyQualifiedName~FormSchemaExcelServiceTests"
|
|
rg -n "ReadAllBytesAsync|new XLWorkbook\(ms\)|\.ToArray\(\)" src/modules/workflow/Application/BuildFlow/Imports src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Imports
|
|
git add src/modules/workflow src/BuildFlow.Host tests/BuildFlow.Workflow.Tests
|
|
git commit -m "perf: stream bounded excel imports"
|
|
```
|
|
|
|
Expected: tests pass; any remaining `XLWorkbook`/`ToArray` match is only the small template-generation method, not import parsing.
|
|
|
|
### Task 7: Page And Stream Excel Exports
|
|
|
|
**Files:**
|
|
- Modify: `src/modules/workflow/Application/BuildFlow/Common/Services/IFormSchemaExcelService.cs`
|
|
- Modify: `src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs`
|
|
- Modify: `src/modules/workflow/Application/BuildFlow/Exports/Handlers/ExportFormDataQueryHandler.cs`
|
|
- Modify: `src/modules/workflow/Application/BuildFlow/Exports/Queries/ExportFormDataResult.cs`
|
|
- Modify: `src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Exports/DownloadExportEndpoint.cs`
|
|
- Create: `tests/BuildFlow.Workflow.Tests/Performance/CountingExportPageInterceptor.cs`
|
|
- Create: `tests/BuildFlow.Workflow.Tests/Performance/CancellingExportExcelService.cs`
|
|
- Create: `tests/BuildFlow.Workflow.Tests/Performance/OpenXmlExportBoundaryTests.cs`
|
|
|
|
- [ ] **Step 1: Write failing paged-export and cleanup tests**
|
|
|
|
Cover 1,201 rows with page size 500 and assert exactly three `b_form_data` SELECT commands, stable output row count/order, and a result that contains a temporary file path rather than `byte[]`. Add cancellation and writer-exception tests that assert no `buildflow-export-*.xlsx` file remains.
|
|
|
|
- [ ] **Step 2: Run and verify RED**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter FullyQualifiedName~OpenXmlExportBoundaryTests
|
|
```
|
|
|
|
Expected: current handler performs one unbounded `ToListAsync`, returns `byte[]`, and has no temporary-file cleanup contract.
|
|
|
|
- [ ] **Step 3: Add the sequential writer contract**
|
|
|
|
```csharp
|
|
Task WriteExportAsync(
|
|
string schemaJson,
|
|
string formName,
|
|
IAsyncEnumerable<string> dataJsonRows,
|
|
Stream output,
|
|
CancellationToken ct);
|
|
```
|
|
|
|
Implement with `SpreadsheetDocument.Create` and `OpenXmlWriter`. Write workbook/sheet metadata, one header row, then each JSON row in order. Check cancellation at least once per row. Do not create `XLWorkbook`, a complete row list, or a final byte array.
|
|
|
|
- [ ] **Step 4: Page rows by a stable keyset**
|
|
|
|
The handler creates a unique file under `Path.GetTempPath()/buildflow-exports`, then exposes an async row iterator. Each page orders by `CreatedAt` and `Id`, filters after the last pair, selects only `CreatedAt`, `Id`, and `DataJson`, and uses `Take(ExportPageSize)`. The iterator yields page rows before querying the next page.
|
|
|
|
Return:
|
|
|
|
```csharp
|
|
public sealed record ExportFormDataResult(
|
|
string FilePath,
|
|
string FileName,
|
|
string ContentType);
|
|
```
|
|
|
|
On writer failure or cancellation, the handler deletes the temporary file before rethrowing.
|
|
|
|
- [ ] **Step 5: Stream and always delete in the endpoint**
|
|
|
|
The endpoint keeps the current status, content type, and Content-Disposition. It opens the result path with asynchronous sequential-scan options, copies to `Response.Body`, flushes, and deletes the file in `finally`, including client cancellation and response-copy failure.
|
|
|
|
- [ ] **Step 6: Verify and commit**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter "FullyQualifiedName~OpenXmlExportBoundaryTests|FullyQualifiedName~FormSchemaExcelServiceTests"
|
|
rg -n "ToListAsync|byte\[\]|MemoryStream|BuildExport" src/modules/workflow/Application/BuildFlow/Exports src/BuildFlow.Host/Endpoints/Workflow/BuildFlow/Exports
|
|
git add src/modules/workflow/Application/BuildFlow src/BuildFlow.Host/Endpoints/Workflow tests/BuildFlow.Workflow.Tests
|
|
git commit -m "perf: stream paged excel exports"
|
|
```
|
|
|
|
### Task 8: Run The Fourth-Batch Verification Gate
|
|
|
|
**Files:**
|
|
- Verify only.
|
|
|
|
- [ ] **Step 1: Run focused suites**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore --filter "FullyQualifiedName~Chat|FullyQualifiedName~RedisChatMessageCacheTests"
|
|
dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore --filter "FullyQualifiedName~FileSize|FullyQualifiedName~FoldersBucketsSettingsTests"
|
|
dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore --filter "FullyQualifiedName~OpenXml|FullyQualifiedName~ImportJob|FullyQualifiedName~FormSchemaExcelServiceTests"
|
|
```
|
|
|
|
- [ ] **Step 2: Run affected projects**
|
|
|
|
```bash
|
|
dotnet test tests/BuildFlow.Rag.Tests/BuildFlow.Rag.Tests.csproj --no-restore
|
|
dotnet test tests/BuildFlow.File.Tests/BuildFlow.File.Tests.csproj --no-restore
|
|
dotnet test tests/BuildFlow.Workflow.Tests/BuildFlow.Workflow.Tests.csproj --no-restore
|
|
dotnet test tests/BuildFlow.Architecture.Tests/BuildFlow.Architecture.Tests.csproj --no-restore
|
|
```
|
|
|
|
- [ ] **Step 3: Run full-solution gates**
|
|
|
|
```bash
|
|
dotnet test BuildFlow.slnx --no-restore
|
|
dotnet build BuildFlow.slnx --no-restore
|
|
```
|
|
|
|
Expected: every project passes, zero skipped, and build has 0 warnings and 0 errors.
|
|
|
|
- [ ] **Step 4: Run mandatory static scans**
|
|
|
|
```bash
|
|
rg -n "OrderBy\(.*CreatedAt\).*ToListAsync|StringGetAsync|StringSetAsync" src/modules/rag/Application/Chat src/modules/rag/Infrastructure/Cache
|
|
rg -n "FileSizeBackfillHelper|GetObjectSizeAsync" src/modules/file/Application/Features/Folders
|
|
rg -n "ReadAllBytesAsync|BuildExport\(|ToListAsync" src/modules/workflow/Application/BuildFlow/Imports src/modules/workflow/Application/BuildFlow/Exports
|
|
rg -n "new XLWorkbook" src/modules/workflow/Application/BuildFlow/Common/Services/FormSchemaExcelService.cs
|
|
git diff --check
|
|
git status --short
|
|
```
|
|
|
|
Expected: the first three scans have no output; the ClosedXML scan finds only template generation; diff check has no output; worktree is clean.
|
|
|
|
- [ ] **Step 5: Record batch evidence**
|
|
|
|
Report bounded context SQL row count and token count, Redis concurrent append count/key type/TTL, GET-path S3 call count, file repair maximum concurrency and missing/transient outcomes, import byte/row/cell limit results, absence of `ReadAllBytesAsync`, export page count for 1,201 rows, cancellation cleanup result, affected-project totals, full-solution total, build warning/error counts, and all fourth-batch commit hashes.
|
|
|
|
Do not start the fifth-batch architecture plan until every command above is green.
|
|
|
|
## Self-Review Results
|
|
|
|
- Spec coverage: design sections 7.1, 7.2, 7.3, and 7.4 each have RED tests, implementation tasks, commits, and deterministic gates.
|
|
- Subsystem boundaries: chat changes remain inside RAG; file repair is owned by File Application and triggered from Host through MediatR; workflow Excel changes remain module-internal.
|
|
- HTTP compatibility: routes, permissions, request field names, SSE events, content types, and download disposition remain unchanged.
|
|
- Database impact: RAG adds one composite index; File adds one boolean column plus one composite index and an explicit historical-data update.
|
|
- Type consistency: `ChatOptions`, `SizeVerified`, stream import signatures, and temporary-file export result types are introduced before all consumers are modified.
|
|
- Placeholder scan: no deferred implementation markers, unnamed edge cases, or unspecified verification commands remain.
|