Zero-Copy vs. Copy-Based RDMA Data Transfer

This repository is a fork of the ibverbs crate created to benchmark and compare the performance of different Copy-Based vs. Zero-Copy RDMA data transfer strategies.

Project Overview

The core goal is to measure the latency and throughput of various methods for performing RDMA READ operations. The primary division between strategies is:

• Copy-Based: These strategies pre-allocate and register a large MR with the RDMA device. To read data, the application first copies data from its source buffer into this pre-registered MR, and then initiates the RDMA operation.
• Zero-Copy: These strategies attempt to avoid the memory copy. They do this by registering the user's data buffer on-the-fly, right before the RDMA operation, and deregistering it immediately after. This saves on CPU copy time but incurs the overhead of dynamic memory registration.

This project benchmarks several variations of these two fundamental approaches.

---

Strategies Benchmarked

The ibverbs-perf/benches directory contains the client-side implementations for each strategy:

• copy_blocking: A simple, single-threaded copy-based approach. It copies data into a pre-registered MR and blocks until the RDMA READ completes.
• copy_non_blocking: A multi-threaded copy-based approach. It uses a pool of worker threads to perform the memory copies concurrently.
• naive_blocking: A zero-copy strategy that sequentially registers/read/deregisters in a blocking manner.
• naive_non_blocking: The non-blocking, multi-threaded version of the naive strategy.
• pipeline_blocking: A zero-copy strategy that splits a large request into smaller chunks and "pipelines" the registration/read/deregistration operations in a blocking manner.
• pipeline_non_blocking: The non-blocking, multi-threaded version of the pipeline strategy.
• blocking_ideal / non_blocking_ideal: Baseline strategies used to measure the "best-case" scenario. They omit data validation and other overheads to provide a theoretical performance ceiling. --skip-validation is required for these.

---

Building and Running Benchmarks

Prerequisites

You must have the libibverbs development libraries installed. On Debian-based systems:

sudo apt-get install libibverbs-dev

By default, the benchmarks also use hwloc for optimized memory allocation in copy-based strategies. This is recommended for best performance.

sudo apt-get install libhwloc-dev

To build without hwloc, you can use the --no-default-features flag.

Quick Start: Running a Benchmark

All benchmarks require a server instance to be running. It's recommended to restart the server between long benchmark runs, as it does not currently clean up old QPs.

1. On the Server Machine (e.g., 192.168.1.100)

# Start the server
cargo run --release --bin server

2. On the Client Machine

# Run the 'copy_blocking' benchmark against the server
cargo bench --bench copy_blocking -- 192.168.1.100

# Run a different benchmark (e.g., 'naive_non_blocking')
cargo bench --bench naive_non_blocking -- 192.168.1.100

# See all configuration options for a specific benchmark
cargo bench --bench copy_blocking -- --help

---

Alternative: Using Pre-Compiled Binaries

You can also build the binaries first and run them directly.

1. Build the Binaries

# Build the server
cargo build --release --bin server

# Build a specific benchmark
cargo build --release --bench copy_blocking

2. Run the Binaries

# Start the server
./target/release/server

# Run the client benchmark
# Note: The suffix after the benchmark name is auto-generated by Cargo
./target/release/deps/copy_blocking-<generated-suffix> <REMOTE_IP>

---

Configuration Options

Flags can be passed to the server and client binaries to control the benchmark parameters.

Server Configuration

• --size: Specifies the size (in bytes) of the local Memory Region (MR) containing the data to be read by clients. This should be larger than any single client request size.

Common Client Configuration

• --size: The request size (in bytes) to read from the server. Must be less than or equal to the server's --size and less than the maximum single RDMA READ size (2GiB - 1).
• --warmup: Duration (in seconds) to run the benchmark before starting measurements.
• --measure: Duration (in seconds) for the actual measurement period.
• --skip-validation: Disables validation of the read data. Required for "Ideal" clients.
• --skip-latency: Disables latency measurements.
• --skip-throughput: Disables throughput measurements.
• --memory-max: Maximum size (in bytes) of all pre-allocated requests in the VecDeque.
• --operations-max: Maximum number of pre-allocated requests in the VecDeque.
• --logging: Enables logging (currently only for non-blocking clients).

copy_blocking & copy_non_blocking

• --mr-size: Size of the pre-allocated MRs used for copying.
• --concurreny (non-blocking only): The number of copy worker threads to use.

naive_non_blocking & pipeline_non_blocking

• --concurrency-reg: The number of registration worker threads to use.
• --concurrency-dereg: The number of deregistration worker threads to use.

ideal_blocking, ideal_non_blocking, pipeline_blocking, pipeline_non_blocking

• --chunk-size: The size (in bytes) that a large request will be split into.

---

Repository Structure

• ibverbs-sys: Contains the low-level ffi bindings for the C libibverbs library.
• ibverbs: Contains mid-level, safe Rust wrappers for the C library's structs and functions.
• ibverbs-perf: Contains all high-level application logic.
  • src/: Implementation of the server binary and the different transfer strategies.
  • benches/: Contains the client-side benchmark logic for comparing the strategies.