Conversation
On retry, the resumable sender injects a completion carrying the persisted
size returned by QueryWriteStatus. That counts bytes durably written by an
earlier, aborted attempt, so w.bufBaseOffset can advance past everything the
current attempt has sent.
Two indices derived from it were used unchecked to reslice buffers:
toCopyIn := cmdBuf[int(w.bufBaseOffset-offset):]
w.bufUnsentIdx = int(sentOffset - w.bufBaseOffset)
The first panics with "slice bounds out of range" once the persisted offset
passes the end of the command buffer. The second can go negative and panic
on the next reslice of w.buf. Both are runtime panics, so they take down the
calling process rather than failing the upload.
Clamp both to their slice bounds. A skip past the end of cmdBuf means the
service already holds all of it, so there is nothing left to stage and the
next write resumes from the offset the service reported. writeLoopAttempt
already clamps the same index for the same reason.
Contributor
There was a problem hiding this comment.
Code Review
This pull request prevents a potential 'slice bounds out of range' panic in the resumable write path of the gRPC writer. When a reconnect occurs, the service may report a persisted size from an earlier attempt that exceeds what the current attempt has sent. The changes clamp the buffer indices derived from this offset to ensure they stay within valid bounds. Additionally, a regression test TestGRPCWriter_ServiceAcksBeyondSentBytes has been added to verify this behavior. There are no review comments, so we have no further feedback to provide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a resumable gRPC write is retried, the sender reconnects and injects a
completion carrying the persisted size returned by
QueryWriteStatus. Thatfigure counts bytes durably written by the earlier, aborted attempt, so
w.bufBaseOffsetcan advance past everything the current attempt has sent.Two indices derived from it were used unchecked to reslice buffers in
(*gRPCWriterCommandWrite).handle:The first panics once the persisted offset passes the end of the command
buffer:
The second can go negative, which panics on the next reslice of
w.buf.Both are runtime panics, so they take down the calling process rather than
failing the upload.
Why this has not been reported
The resume path is hard to reach today. Plain gRPC uploads are non-idempotent,
so
run()gives them zero retries, and an attempt that hangs never ends. Itdoes become reachable for uploads that are idempotent —
IfGenerationMatch,DoesNotExist, or an appendable takeover — where a transient mid-upload errortriggers a genuine retry of a multi-chunk write.
Fix
Clamp both indices to their slice bounds. A skip past the end of
cmdBufmeansthe service already holds all of it, so there is nothing left to stage and the
next write resumes from the offset the service reported — nothing is dropped
and nothing is re-sent.
writeLoopAttemptalready clamps the same index for the same reason, with acomment noting that a completion can land beyond all of
w.buf; this appliesthat treatment to the two sites in the command handler that lacked it.
The change only ever narrows a slice range. It cannot widen one, reorder sends,
or alter
bufBaseOffset/flushOffset. In any state where the clamps do notengage, behaviour is unchanged; where they do, the previous behaviour was a
panic.
Tests
TestGRPCWriter_ServiceAcksBeyondSentBytesdriveswriteLoopwith a senderthat reports a persisted offset two chunks beyond what was sent, which is what
mockSender's newoverAckAfter/overAckByfields model. Both default tozero, so existing tests are unaffected.
The test asserts more than survival. The payload is a position-dependent
pattern, and every buffer the sender receives must carry exactly the payload
bytes belonging at the offset it was sent to, with no region sent twice —
a clamp that shifted the stream would be silent object corruption, which is
worse than the panic. Reverting the fix reproduces the panic:
Verified:
go test -short -race ./...across thestoragemodule — pass-race -count=5— pass*Emulatedtests against the storage-testbench atmain— pass(
TestWriterRetryAttrsEmulatedandTestRetryNeverEmulatedfail in my localenvironment, but they fail identically on a clean checkout and one of them is
the HTTP subtest, so they are unrelated to this change)