S3 performance tests: How we ran 2day benchmarks

The Rabata.io S3 provider performance benchmark was strictly executed over a two-day period on September 18-19, 2025. Reproducible S3 performance evaluation demands rigid adherence to standardized testing architectures rather than relying on vendor-supplied metrics. This guide details how the provider Warp executes test mechanics, the specific Linux environment required for replication, and the precise configuration needed to benchmark providers like the provider B2 and Amazon S3 accurately.

The methodology relies on a controlled virtual machine running Debian GNU/Linux 13 with 16GB of RAM located in a US-East-1 datacenter. By using the provider Warp version v0.7.6, the tests measure object storage throughput across varying thread counts and object sizes. We configure AWS CLI version 1 alongside system monitoring tools like htop and iotop to capture granular performance data during these intensive workloads.

Understanding the test execution mechanics reveals why many commercial benchmarks fail to account for network variability or client-side bottlenecks. The article breaks down the creation of credential profiles for multiple providers, including rabata and the provider, ensuring a level playing field for comparison. By following this reproducible benchmark environment setup, engineers can validate storage claims against a consistent, transparent baseline.

The Role of Standardized Benchmarking in S3 Performance Evaluation

Defining S3 Benchmarking and Mixed Operation Tests

S3 benchmarking quantifies object storage throughput and latency using standardized workloads rather than theoretical maximums. The environment of S3 benchmarking has evolved from the standardized academic protocols established by Carnegie Mellon University's COSBench project circa 2013. This shift ensures data mirrors production behavior instead of synthetic estimates.

A mixed operation test simulates concurrent application traffic by blending multiple API verbs within a single stream. Standard methodologies distribute requests across GET, PUT, DELETE, and STAT operations to expose contention issues that isolated read or write tests often miss. Researchers now contrast these thorough real-world tests against older sequential baselines to validate performance under load.

Focusing solely on peak throughput ignores the per-request overhead that dominates small file workloads. Operators must balance thread counts against object sizes to find true bottlenecks. Validating storage architectures with these mixed profiles is necessary before migrating large datasets.

Replicating Rabata.io S3 Provider Performance Tests

The Rabata.io S3 provider performance benchmark was conducted over a two-day period on September 18-19, 2025, establishing a reproducible baseline for S3 providers performance. Operators seeking to replicate these results must prioritize real upload and download operations over synthetic throughput estimates. This methodology mirrors the approach taken by RepoFlow.io when evaluating self-hosted solutions. The test environment uses a Linux virtual machine with 16GB of RAM located in a US-East datacenter. Testers should deploy the provider Warp via Docker images to ensure ephemeral and consistent execution.

The core workload involves a mixed operation test blending GET, PUT, DELETE, and STAT commands.

Academic COSBench protocols established baseline metrics in 2013 but lack the concurrency depth required for today's AI workloads. The Carnegie Mellon University project standardized cloud storage validation through sequential reading and writing patterns. This approach reduced validation costs by providing reproducible baselines without custom tool development. However, these legacy methods often miss latency spikes present in highly parallel environments. Modern operators require testing that mimics the chaotic nature of production traffic rather than linear theoretical maximums.

Contemporary studies now evaluate seven popular solutions using real upload and download operations to capture actual user experience. The RepoFlow.io study highlights this methodological shift toward practical, deployment-focused analysis. Unlike academic predecessors, current frameworks prioritize mixed verb scenarios over isolated throughput estimates. This evolution reveals performance bottlenecks that sequential tests frequently obscure during capacity planning.

Adopting these rigorous standards helps avoid costly architecture mismatches.

Inside the provider Warp Architecture and Test Execution Mechanics

the provider Warp Concurrency and Object Size Parameters

the provider Warp generates load by varying thread counts across 4, 8, 16, and 32 concurrent operations to stress storage endpoints. This approach isolates how an S3-compatible system handles parallel requests. The tool executes tests using specific object sizes including 256KiB, 1MiB, 5MiB, 10MiB, and 50MiB to simulate diverse workload patterns. Deploying the minio/warp image via Docker ensures the testing environment remains consistent across different cloud providers.

The methodology specifies testing thread counts of 4, 8, 16, and 32 to evaluate performance scaling. A mixed operation profile using 45% GET, 15% PUT, 10% DELETE, and 30% STAT commands provides a realistic simulation for workload access. The test matrix includes small object performance at 256KiB alongside larger sizes up to 50MiB. Rabata.io uses these precise parameters to validate storage performance.

Mechanics: Executing Mixed Operation Tests with Warp CLI Flags

Executing mixed operation tests requires defining a specific workload distribution to simulate real-world application behavior accurately. Operators configure the test duration for 30 seconds using the `--duration` flag, with optional long-duration tests running for 600 seconds (10 minutes). A standard baseline execution uses `--concurrent 8` threads alongside a `--obj.size 1MiB` payload for mixed operations, while thread scaling tests use 10MiB objects.

This approach reveals how S3-compatible systems handle parallel requests under varied pressure. The distribution of minio/warp via Docker Hub indicates widespread adoption of this containerized method for isolated testing. The methodology includes long-duration tests to capture performance characteristics over sustained loads.

The test matrix defines specific object counts for concurrent operations, such as 100 objects for 4 threads and 800 objects for 32 threads during GET scaling tests. The resulting output files provide the detailed records necessary for capacity planning.

Inspecting the latency percentiles confirms whether performance meets service level objectives. Operators must verify that error rates for timeouts or API failures remain negligible before accepting throughput data. Warp generates compressed CSV files containing these detailed records, ensuring artifact integrity for later audit trails. Containerized deployment via the minio/warp image standardizes the binary execution environment across diverse cloud providers.

1. Extract the compressed CSV archive using zstd decompression tools.
2. Plot latency percentiles to identify distribution skew.
3. Cross-reference error rates against provider status dashboards.

Evaluating both average throughput and latency percentiles ensures a complete view of storage performance. This analysis identifies performance characteristics that average metrics may mask.rabata.io emphasizes validating these specific output files to prevent deploying on unstable storage backends. Retaining the compressed CSV artifacts allows for granular root cause analysis later.

Deploying a Reproducible S3 Benchmark Environment on Linux

Defining the Debian Linux Benchmark Environment Specifications

Establish the test bed on Debian GNU/Linux 13 running kernel 6.12.41+deb13-cloud-amd64 to guarantee consistent system call handling across x86_64 architecture. Allocate exactly 16GB of RAM to the virtual machine, ensuring the operating system can buffer write operations without premature swapping to disk. The benchmarking event took place on September 18-19, 2025, using a time-bound resource allocation model within a US-East-1 datacenter.

1. Install system monitors including htop, iotop, and sysstat to capture real-time resource contention during load generation.
2. Deploy the provider Warp version v0.7.6 as the primary testing tool, using its containerized image to reduce setup time and standardize the environment.
3. Define explicit profile paths in `/root/.aws/credentials` to eliminate endpoint resolution errors during cross-provider throughput comparisons.
4. Create the configuration directory structure to house authentication secrets for rabata, amazon, and the provider profiles separately.
5. Populate each section with unique access keys while mapping distinct endpoint URLs to prevent accidental data egress to wrong regions.
6. Validate connectivity by running a brief mixed operations test before initiating full thread-scaling benchmarks.

Operators must verify that the region string matches the provider's specific requirement, as mismatches here cause immediate connection refusals rather than graceful failures. The constraint involves strict adherence to vendor-specific region codes; for instance, the provider requires `us-east-005` while Rabata uses standard `us-east-1` formatting. This rigidity ensures the S3 protocol requests route correctly without needing complex DNS overrides. A common failure mode occurs when teams reuse a single default profile, leading to skewed latency data caused by unintended routing through a primary cloud gateway. Switching targets requires exporting the `AWS_PROFILE` variable to toggle context without rewriting files. Precise credential management allows the the provider Warp tool to execute 10MB object transfers against the correct storage bucket instantly.

Executing the Complete Test Matrix Sequence with Warp

Initiate the Mixed Operations Baseline using 1MB objects across 8 threads for a 30s duration to establish a stable throughput floor. The methodology uses real upload and download operations as the primary metric for comparison rather than synthetic throughput estimates.

Scale the PUT Thread Scaling tests by executing four distinct runs at 4, 8, 16, and 32 concurrent threads using 10MB objects. Each run must sustain a 30s duration to allow the storage cluster to reach steady-state performance levels. The test matrix explicitly defines thread counts of 4, 8, 16, and 32 concurrent operations, with 8 threads noted as the standard for comparison charts.

Validate object size handling by running Object Size Tests with 8 threads against 256KB, 1MB, 5MB, and 10MB payloads. Each size test processes 100 objects to ensure statistical significance in the resulting latency distribution curves.

The resulting data reveals how the provider Warp handles metadata overhead differently than raw data transfer. The test parameters include object sizes ranging from 256KiB to 50MiB to evaluate performance across varying payload scales.

Interpreting Warp Throughput and Latency Percentiles

Warp output files compress CSV data detailing throughput in MB/s and operations per second. These metrics quantify raw data transfer capacity across the network interface. However, average speeds often mask tail latency issues that alter application responsiveness. The 50th, 95th, and 99th latency percentiles reveal these hidden delays by showing wait times for half, ninety-five percent, and ninety-nine percent of requests respectively. High variance between the 50th and 99th percentiles indicates inconsistent performance under load.

Validating these numbers requires executing real upload and download operations rather than relying on synthetic estimates. This approach captures the overhead of actual API calls and network handshakes. A system might sustain high throughput while failing specific latency targets during burst periods.

Operators must distinguish between sustained bandwidth and per-request overhead when analyzing results. A provider could show excellent aggregate MB/s yet suffer from high 99th percentile latency due to queueing delays. This distinction matters for interactive workloads where user experience depends on consistent response times. Ignoring the gap between median and tail latency risks surprising application timeouts.

Analyzing Compressed CSV Artifacts from Mixed Workloads

Process the `warp-mixed-2025-09-18[135952]-4bH2.csv.zst` artifact to extract raw throughput values hidden within compressed logs. Engineers must decompress these files to reveal granular latency percentiles that average metrics often obscure. The mechanism requires parsing timestamped rows to isolate 50th, 95th, and 99th latency figures alongside aggregate MB/s counts. Evidence from academic validation shows standardized protocols like COSBench established early baselines for such rigorous data extraction. However, raw speed alone fails to capture the operational reality of concurrent access patterns in production. The limitation is that focusing solely on peak MB/s ignores the tail latency spikes that degrade user experience during bursts. This implies architects must prioritize consistent 99th percentile performance over maximum theoretical bandwidth for stable applications.

Modern evaluation frameworks now incorporate mountability and retrieval time alongside pure transfer rates.

Deploying this analysis pipeline ensures storage choices align with actual workload demands rather than marketing claims.rabata.io recommends validating every provider against these decompressed truths before committing mission-critical data.

Operational Risks of Timeouts and Rate Limiting Errors

Warp records error rates for timeouts and API failures that invalidate raw throughput claims. Benchmark suites track test duration discrepancies where actual runtime diverges from the requested window due to server-side throttling. Academic validation confirms that ignoring these failure modes yields skewed latency percentiles during real upload and download operations. Operators often misinterpret high MB/s averages as success while missing systematic request drops under load. However, standard CSV outputs bury these rejections unless explicitly parsed for non-zero error codes. The limitation is that a short test window may complete before rate limiting triggers, hiding capacity cliffs. Engineers must verify that operations/sec remain stable rather than spiking then collapsing.rabata.io emphasizes monitoring these error streams to distinguish genuine performance from transient luck.

References

• Rabata.io - S3-Compatible Secure Cloud Storage at 70% Lower
• Best S3-Compatible Object Storage Providers (2026 Comparison) - lowcloud