To be able to split formats like Avro that are splittable, we need to be able to go back and forth between offsets in the underlying data stream and in the AGS stream. I think that's possible since this has a strict format. The header structure is 8 bytes (4 magic + 4 block length) and then there is 24 bytes of overhead per CipherBlock (12 nonce + 12 tag).

For any given offset in the underlying stream, we can find the block where it will be found by dividing by the block length:

def plain_block_index(plain_offset, block_length):
    return offset // block_length

Then we can skip to a given block:

def cipher_block_offset(block_index, block_length):
    return 8 + block_index * (24 + block_length)

But the problem is that we don't get split offsets in the underlying file, we get split offsets in the AGS stream and need to figure out where in the underlying file those translate to, so that we don't have overlapping ranges that tasks are responsible for. We have a picture like this for a block size of 1024, 3 blocks, and a last block of size 553:

To convert an AGS offset into a stream offset, I think we need to produce the the block index and an offset in that block. Producing the block index is the opposite of the cipher_block_offset function:

def cipher_block_index(cipher_offset, block_length):
    if (cipher_offset < 8):
        raise ValueError()
    return (cipher_offset - 8) // (block_length + 24)

Then the offset in the plaintext block can be produced by subtracting the ciphertext offset from the start of the ciphertext block and some extra accounting:

def plaintext_offset(cipher_offset, block_length):
    block_index = cipher_block_index(cipher_offset, block_length)
    block_start = cipher_block_offset(block_index, block_length)
    local_offset = min(cipher_offset - block_start, block_length)
    return block_index * block_length + local_offset

The min with block_length ensures that the local block offsets never overlap. Offsets that are between the ciphertext blocks (like 1048 in tag-1 for the example above) are mapped to the end of the plaintext block. This gives us a stable way to recover plaintext offsets.

Using the plaintext offsets, I think we can reliably split AGS files that store Avro data:

1. Convert AGS byte range splits into plaintext splits, removing any 0-byte splits (unlikely but possible)
2. Implement an AGS stream that can seek to plaintext offsets (using the translation methods above)
3. Pass the plaintext splits into the Avro reader with the plaintext seeking AGS stream

@ggershinsky, what do you think?

@rdblue , thanks! I need to think deeper about this; currently on vacation; will respond the first week of September.

@rdblue I'm back in office, went through this carefully. I believe we are on the same page. The technical approach you describe is the same as used in the spec implementation PR

At a high level, this should be fully transparent to the current Avro readers. The API they use (file getLength; stream skip, seek, getPos) return the same values and work the same as before/without AGS encryption. This is the AGS code job to translate the plaintext<->ciphertext stuff behind the scenes. So, as long as the Avro split functionality depends on these APIs, it should continue to work as usual with the AGS encrypted files.

@rdblue what do you think?

I think that this should work.