OneCopy: High-Performance Sub-File Deduplication Engine & WebAssembly Bridge

OneCopy is an enterprise-grade, systems-level deduplication engine engineered to reduce storage overhead and network ingress latency across high-redundancy data environments. Utilizing Content-Defined Chunking (FastCDC v2020) paired with SIMD-accelerated BLAKE3 cryptographic digests, OneCopy systematically identifies and eliminates duplicate byte sequences at the sub-file level without relying on brittle static-block dividing strategies.

Designed around a zero-copy memory pipeline in idiomatic Rust, OneCopy is architecture-agnostic. It runs natively as a high-throughput Desktop/Sever Command Line tool and executes deterministically inside client web browsers via a WebAssembly (WASM) Bridge, persisting content-addressable storage directly to local IndexedDB databases with zero data exfiltration.

Traditional static-block segmentation suffers from boundary shifting: inserting or deleting a single byte at the head of a file alters all subsequent block divisions, resulting in zero deduplication across file revisions. OneCopy avoids this by implementing a Gear-hash rolling bitmask that identifies natural, shift-resistant chunk boundaries based on internal content entropy.

• Minimum Chunk Size: 2,048 bytes (2 KB) — Prevents chunk inflation overhead.
• Target / Average Chunk Size: 8,192 bytes (8 KB) — Optimal trade-off between dictionary indexing granularity and deduplication savings.
• Maximum Chunk Size: 65,536 bytes (64 KB) — Enforces bounded memory limits on high-entropy streams.

The computational core (src/lib.rs) operates strictly over borrowed memory slices (&[u8]). During the extraction of chunks and the evaluation of rolling hashes, zero internal byte array duplications (memcpy) occur. Allocations are restricted strictly to lightweight metadata struct representations (FileManifest and ChunkInfo).

When running in browser environments via WebAssembly, all file stream ingestion, chunking, hashing, and re-assembly happen strictly within client memory space. Chunks are persisted to an IndexedDB wrapper utilizing reference counting (refCount). Duplicate chunks merely increment storage counters rather than writing duplicate bytes to physical drives.

                     Raw Ingestion Stream (&[u8] / Uint8Array)
                                        │
                                        ▼
                   ┌─────────────────────────────────────────┐
                   │    FastCDC v2020 Rolling Hash Engine    │
                   └────────────────────┬────────────────────┘
                                        │
                    Emits Variable-Length Chunks (2 KB - 64 KB)
                                        │
                                        ▼
                   ┌─────────────────────────────────────────┐
                   │        SIMD BLAKE3 Digest Hashing       │
                   └────────────────────┬────────────────────┘
                                        │
               ┌────────────────────────┴────────────────────────┐
               ▼                                                 ▼
      Unique Content Hash                              Hash Collision Detected
               │                                                 │
               ▼                                                 ▼
     Persist Binary Chunk Blob                        Increment RefCount Pointer
   (Disk / IndexedDB 'chunks')                         (Instant Storage Savings)
               │                                                 │
               └────────────────────────┬────────────────────────┘
                                        ▼
                   ┌─────────────────────────────────────────┐
                   │       Emit FileManifest JSON Schema     │
                   └─────────────────────────────────────────┘

In structured benchmarking trials across standardized real-world operational datasets, the OneCopy architecture achieved substantial storage utilization improvements, proving commercial viability across enterprise storage environments:

Empirical evaluation confirmed that applying sliding-window dictionary compression (e.g., DEFLATE / .zip) prior to ingestion intentionally destroys deduplication efficiency across file versions. A single altered word in a plaintext source file radically scrambles the resulting compressed binary byte stream due to the Avalanche Effect, rendering block hashing ineffective.

Enterprise Implementation Standard: For optimal system performance and Total Cost of Ownership (TCO) mitigation, data pipelines must feed raw, uncompressed, or natively structured streams into the OneCopy deduplication engine first, applying cold-storage block compression individually to unique physical chunks post-deduplication.

• Rust Toolchain: v1.80.0+ (Install via Rustup)
• Node.js Environment: v18.0.0+ & NPM (Node Official)
• WebAssembly Pack: (cargo install wasm-pack)

The browser application demonstrates live, client-side deduplication against local machine storage without external server overhead.

# Clone repository
git clone https://github.com/Gevry/OneCopy.git
cd OneCopy/web

# Install frontend workspace dependencies
npm install

# Start local Vite engineering studio
npm run dev

Access the dashboard via http://localhost:5173. Drag and drop single files or archival directories onto the dropzone to evaluate live storage efficiency and examine generated manifests.

For large-scale server ingestion, pipeline automation, or batch folder archiving, compile and interact directly with the optimized native target.

# Navigate to workspace root and build optimization binaries
cd OneCopy
cargo build --release

# Ingest, chunk, and deduplicate a target file or system directory
.\target\release\onecopy.exe store "C:\Enterprise\Archives\Invoices"

# Reconstruct original source asset and verify end-to-end BLAKE3 integrity
.\target\release\onecopy.exe restore ".\manifests\sample_invoice.pdf.json"

OneCopy/
├── src/
│   ├── lib.rs                 # Core FastCDC chunking engine & WASM interoperability layer
│   └── main.rs                # Native systems CLI implementation (store / restore processing)
├── pkg/                       # Compiled WebAssembly production distributions
├── web/
│   ├── src/
│   │   ├── lib/
│   │   │   ├── db.js          # IndexedDB transactional storage & garbage collection layer
│   │   │   └── wasmBridge.js  # Zero-copy Javascript-to-WASM memory serialization bridge
│   │   ├── App.jsx            # React 19 Executive Dashboard & analytical metrics suite
│   │   └── App.css            # Tailwind CSS v4 design system tokens & typography rules
│   ├── vite.config.js         # Build configuration & WASM MIME headers
│   └── package.json           # Frontend dependencies & workspace runtime execution
├── Cargo.toml                 # Workspace dependency definitions & target compilation rules
└── README.md                  # Enterprise system documentation