Skip to content

fix(storage): clamp resume offsets in the gRPC write path - #20530

Open
cpriti-os wants to merge 1 commit into
googleapis:mainfrom
cpriti-os:fix-grpc-write-resume-offset-panic
Open

cpriti-os wants to merge 1 commit into
googleapis:mainfrom
cpriti-os:fix-grpc-write-resume-offset-panic

Conversation

@cpriti-os

Copy link
Copy Markdown
Contributor

Problem

When a resumable gRPC write is retried, the sender reconnects and injects a
completion carrying the persisted size returned by QueryWriteStatus. That
figure counts bytes durably written by the earlier, aborted attempt, so
w.bufBaseOffset can advance past everything the current attempt has sent.

Two indices derived from it were used unchecked to reslice buffers in
(*gRPCWriterCommandWrite).handle:

toCopyIn := cmdBuf[int(w.bufBaseOffset-offset):]
w.bufUnsentIdx = int(sentOffset - w.bufBaseOffset)

The first panics once the persisted offset passes the end of the command
buffer:

panic: runtime error: slice bounds out of range [2097152:608579]
  grpc_writer.go  (*gRPCWriterCommandWrite).handle

The second can go negative, which panics on the next reslice of w.buf.

Both are runtime panics, so they take down the calling process rather than
failing the upload.

Why this has not been reported

The resume path is hard to reach today. Plain gRPC uploads are non-idempotent,
so run() gives them zero retries, and an attempt that hangs never ends. It
does become reachable for uploads that are idempotent — IfGenerationMatch,
DoesNotExist, or an appendable takeover — where a transient mid-upload error
triggers a genuine retry of a multi-chunk write.

Fix

Clamp both indices to their slice bounds. A skip past the end of cmdBuf means
the service already holds all of it, so there is nothing left to stage and the
next write resumes from the offset the service reported — nothing is dropped
and nothing is re-sent.

writeLoopAttempt already clamps the same index for the same reason, with a
comment noting that a completion can land beyond all of w.buf; this applies
that treatment to the two sites in the command handler that lacked it.

The change only ever narrows a slice range. It cannot widen one, reorder sends,
or alter bufBaseOffset / flushOffset. In any state where the clamps do not
engage, behaviour is unchanged; where they do, the previous behaviour was a
panic.

Tests

TestGRPCWriter_ServiceAcksBeyondSentBytes drives writeLoop with a sender
that reports a persisted offset two chunks beyond what was sent, which is what
mockSender's new overAckAfter / overAckBy fields model. Both default to
zero, so existing tests are unaffected.

The test asserts more than survival. The payload is a position-dependent
pattern, and every buffer the sender receives must carry exactly the payload
bytes belonging at the offset it was sent to, with no region sent twice —
a clamp that shifted the stream would be silent object corruption, which is
worse than the panic. Reverting the fix reproduces the panic:

clamps reverted → panic: runtime error: slice bounds out of range [786432:262144]
restored        → ok  cloud.google.com/go/storage

Verified:

  • go test -short -race ./... across the storage module — pass
  • New test under -race -count=5 — pass
  • 50 *Emulated tests against the storage-testbench at main — pass
    (TestWriterRetryAttrsEmulated and TestRetryNeverEmulated fail in my local
    environment, but they fail identically on a clean checkout and one of them is
    the HTTP subtest, so they are unrelated to this change)

On retry, the resumable sender injects a completion carrying the persisted
size returned by QueryWriteStatus. That counts bytes durably written by an
earlier, aborted attempt, so w.bufBaseOffset can advance past everything the
current attempt has sent.

Two indices derived from it were used unchecked to reslice buffers:

    toCopyIn := cmdBuf[int(w.bufBaseOffset-offset):]
    w.bufUnsentIdx = int(sentOffset - w.bufBaseOffset)

The first panics with "slice bounds out of range" once the persisted offset
passes the end of the command buffer. The second can go negative and panic
on the next reslice of w.buf. Both are runtime panics, so they take down the
calling process rather than failing the upload.

Clamp both to their slice bounds. A skip past the end of cmdBuf means the
service already holds all of it, so there is nothing left to stage and the
next write resumes from the offset the service reported. writeLoopAttempt
already clamps the same index for the same reason.
@cpriti-os
cpriti-os requested review from a team as code owners September 16, 2026 16:33
@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Sep 16, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request prevents a potential 'slice bounds out of range' panic in the resumable write path of the gRPC writer. When a reconnect occurs, the service may report a persisted size from an earlier attempt that exceeds what the current attempt has sent. The changes clamp the buffer indices derived from this offset to ensure they stay within valid bounds. Additionally, a regression test TestGRPCWriter_ServiceAcksBeyondSentBytes has been added to verify this behavior. There are no review comments, so we have no further feedback to provide.

@cpriti-os cpriti-os added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 16, 2026
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants