fix(streaming): reapply checkpoint advancement with safe recovery - #11269

Purpose and release sequencing

This PR re-attempts the checkpoint advancement introduced by #11249, which was reverted by #11330. The original optimization could remain disabled for an agent's lifetime after a recoverable failure. The re-attempt establishes explicit recovery, ownership and retention guarantees before advancing checkpoints.

Release gate: cut a release from the reverted state before considering this PR for a later release. The reverted tree is a standalone release state. It restores subscription-based checkpoints while preserving #11268's shutdown-draining and handshake-ownership guarantees. Idle subscriptions can still pin conservative checkpoint progress, with the associated replay tradeoff accepted for that release.

#11330 is merged as 39ea49999e. This branch has now been rebased onto that actual main commit, retaining the explicit reapplication and recovery rework. All 13 reapplication/fix commits are patch-identical to the prepared series. The ordinary PR diff now includes the complete reapplication against the reverted baseline. When the intervening release is published, renew compatibility validation against that released baseline before landing this PR.

Solution

The built-in Event Hubs components cooperate through an internal certified-prefix protocol. Cursors scan unrelated records and retain selected matching records until acknowledged, filtered, or explicitly skipped by delivery-failure policy. Recovery reconciles receiver-owned source positions and staged notifications; registration and handshake ownership remain independent barriers. The minimum subscription prefix, bounded by the last fully accounted read, governs checkpoint publication and Event Hubs eviction.

Unresolved work retries on pump ticks. Certified progress is published before every read-loop capacity check, so sustained nonempty reads refresh checkpoint and purge authority. Same-cursor re-handshakes retain and rewind unsettled selections when older deliveries finish; replacement cursors retain their requested replay positions. Removing the last unresolved subscriber releases a settled registration pin. Sticky rewind misses preserve the unresolved range while the consumer and failure handler receive the error. The shutdown-draining and handshake-ownership guarantees from #11268 are preserved.

Capacity controls new reception. Already-accepted read accounting, read recovery, and registration retries continue at zero capacity, including when a receiver notification handoff or the agent's admission/discovery accounting filled the cache.

Failure policy and bounded retention

StreamPullingAgentOptions.RetryFailedDeliveries defaults to false. After the existing delivery retry budget is exhausted, the built-in Event Hubs path notifies the consumer and failure handler, skips that selected batch, and continues. Those events can be absent from the subscription's output, and checkpointing can advance past them.

Setting the option to true retains the batch for later pump retries and holds checkpointing and reclamation until it is resolved or the subscription is removed. Subscription faulting/removal follows the configured failure handler. Both settings preserve failed-read, incomplete-selection, cache-miss, and handshake replay obligations. Ownership changes defer skip decisions and preserve replay.

Native certified admission reserves one possible new pool buffer per requested record, independently of averaged consumer pressure. The existing cache constructor's defaultMaxAddCount supplies both maximum read size and buffer budget. The native factory permits 1,000 buffers of 1 MiB per partition. Deployment sizing also includes metadata, SDK prefetch, free pooled buffers and partition count. A factory returning the native cache can choose a smaller constructor budget. Age-eligible, certified records release capacity; staged handoffs can finish while new reception is paused. Packing rollback and exception-safe purge cleanup preserve actual buffer ownership, including when observers throw.

Public surface and compatibility

Recovery, cursor progress, filtering and checkpoint capabilities are internal. Publication reuses the released IQueueCache.UpdateDeliveryProgress callback. Public additions remain the opt-in delivery-policy boolean and IPurgeObservable.TryRemoveOldestMessage, whose default calls the existing removal method and returns true. The Event Hubs API diff records an explicit implementation of the existing ICacheDataAdapter.Compare member.

The Orleans 10.3.1 nullable callback/default and existing custom or derived cache, transport, adapter and eviction behavior are preserved. Ordinary receipt providers keep their notification/skip policy. Certified processing is selected for the native Event Hubs composition and stays fixed for the provider run. Normal Release package validation and unchanged 10.3.1 binary implementations/callers pass without compatibility suppressions.

On the actual post-revert baseline 39ea49999e, 314 shared/Functional and 78 Event Hubs/monitor cases pass on each of net8.0 and net10.0. Coverage includes handshake ownership, both failure policies, error notification, registration cleanup, continuous publication, partial records, atomic admission, full-cache recovery, capacity reopening and guarded reclamation. The rebase preserves the streaming implementation, tests and documentation unchanged.

Runtime cost

Already-accounted cursors stay idle while the read boundary is unchanged. Mutation-safe traversals reuse pooled snapshots. Certified bookkeeping is enabled only for checkpoint-capable owners; ordinary grouped delivery tracks node ranges with explicit mappings only for sliced records.

A historical 22-case actual-build BenchmarkDotNet comparison measured the per-read publication/agent repairs against 660fba8e3d. Warm idle allocation remained 208 B/tick at 1, 32 and 256 subscriptions, and every active allocation result was unchanged. Elapsed differences ranged from approximately 8% lower to 7% higher; the one-subscription certified case increased from 22.10 to 23.71 microseconds per 64-input operation. The repaired fixture verified certification through the current completed read, whereas the prior implementation reported the preceding pump's prefix.

Earlier original-base comparisons showed certified processing approximately 6–15% above the successful-delivery baseline, and ordinary receipt cases from 3% faster to 11% slower. These measurements predate the final buffer-cleanup/full-cache-gate changes and subsequent rebases; they are not fresh-head throughput claims.

Measurements use .NET 10.0.12, BenchmarkDotNet 0.15.8, Windows Hyper-V and real agent/SimpleQueueCache code with synchronous local consumers. They exclude Event Hubs pooled-cache admission, SDK/network, remote RPC, serialization and checkpoint-store I/O. VM variation applies. The earlier additional approximately 214 KB/s of warm idle allocation was eliminated.

Shared boundaries remain separate from #10588's positioning work, #10316's adaptive pressure policy and #7464's persisted checkpoint reset. This update does not approve, merge, release, or enable auto-merge on this PR.