Python API Usage¶
This guide covers Python initialization, RX/TX workflows, buffer lifecycle calls,
file writes, and the bindings for utility, socket, and RDMA helpers. The function
calls below follow the six-step lifecycle introduced in the API Guide:
init → RX/TX → access → free → shutdown. The bindings are produced from
python/daqiri_common_pybind.cpp
and mirror the C++ free-function API in C++ API Usage.
For the terminology used here (burst, segment, flow, queue, memory region, zero-copy ownership, RX reorder), keep the Concepts page open in a second tab.
Functions that can fail return a daqiri.Status value, or a tuple whose first
element is a Status and whose remaining elements are output values. Packet
storage is owned by DAQIRI, so received and transmitted bursts must be explicitly
freed. See
Concepts → Zero-Copy Ownership for why a
missed free causes RX drops.
For a complete executable Python benchmark, see
examples/daqiri_bench_raw_tx_rx.py.
Building and Importing¶
Follow the build flow in Getting Started, container
or bare-metal, and add -DDAQIRI_BUILD_PYTHON=ON to the CMake configure step
(the default is OFF). The container path is recommended, and it already provides
pybind11 and the rest of the build dependencies. For example, inside the
container:
cmake -S . -B build \
-DBUILD_SHARED_LIBS=ON \
-DDAQIRI_BUILD_PYTHON=ON \
-DDAQIRI_ENGINE="dpdk ibverbs"
cmake --build build -j
cmake --install build --prefix /opt/daqiri
A bare-metal build uses the same commands but additionally requires pybind11
on the host.
The installed package is placed under the configured library directory, in
python/daqiri. Add the parent directory (for example
/opt/daqiri/lib/python) to PYTHONPATH before importing:
The daqiri package re-exports the compiled _daqiri pybind11 module, so
application code uses one import.
DAQIRI package versions use CalVer in YYYY.MM.PATCH form. The Python package
exposes daqiri.__version__, daqiri.__version_info__, and
daqiri.__abi_version__; the ABI version is intentionally separate from the
CalVer package version. The YAML common.version field remains the
configuration schema version.
PyYAML is required at runtime when calling daqiri_init() with a Python dict
or a config-like object that provides as_dict().
Initialization¶
daqiri.daqiri_init(config) accepts more input forms than the C++ overload:
- YAML file path
- YAML string
- Python
dict, converted to YAML with PyYAML daqiri.NetworkConfiginstance- config-like object with a
valueattribute oras_dict()method
import daqiri
status = daqiri.daqiri_init("daqiri_bench_raw_tx_rx.yaml")
if status != daqiri.Status.SUCCESS:
raise RuntimeError(f"DAQIRI init failed: {status}")
try:
# RX/TX work here.
pass
finally:
daqiri.print_stats()
daqiri.shutdown()
Only one engine may be active at a time. A second daqiri_init() before shutdown()
returns Status.INTERNAL_ERROR without changing the live engine. After shutdown(), a
later initialization starts with a fresh engine instance and fresh resources.
Initialize from a Python dictionary:
import daqiri
config = {
"version": 1,
"stream_type": "raw",
"master_core": 3,
"log_level": "warn",
"memory_regions": [
{
"name": "RX_CPU",
"kind": "huge",
"affinity": 0,
"access": ["local"],
"num_bufs": 51200,
"buf_size": 1024,
}
],
"interfaces": [
{
"name": "rx_port",
"address": "0000:03:00.0",
"rx": {
"queues": [
{
"name": "rxq0",
"id": 0,
"cpu_core": "8",
"batch_size": 1024,
"memory_regions": ["RX_CPU"],
}
]
},
}
],
}
status = daqiri.daqiri_init(config)
Application-owned CUDA buffers use the same requirements-and-bindings workflow as C++. Addresses
are integers, and the Python allocation owner must stay alive until after shutdown():
status, parsed = daqiri.parse_network_config("config.yaml")
status, requirements = daqiri.get_memory_region_requirements(parsed)
rx_req = requirements["RX_GPU"]
# For example, gpu_owner may be a CuPy allocation made in the desired context.
bindings = {
"RX_GPU": daqiri.ExternalMemoryRegion(int(gpu_owner.data.ptr), rx_req.capacity),
}
status = daqiri.daqiri_init(parsed, bindings)
# Return outstanding bursts first.
daqiri.shutdown()
del gpu_owner
Bindings may cover only a subset of regions. They are supported by DPDK, raw ibverbs, and
RoCE/RDMA, but not direct TCP/UDP sockets. DPDK rejects external huge regions because EAL cannot
preserve ownership of hugepage mappings that predate initialization; use host, host_pinned, or
ibverbs/RDMA for caller-owned CPU hugepages.
Parse without starting the engine:
status, parsed = daqiri.parse_network_config("daqiri_bench_raw_tx_rx.yaml")
if status == daqiri.Status.SUCCESS:
print(parsed.common.engine_type)
If GPU RX reorder_configs are configured for the DPDK engine, set one CUDA
stream per GPU reorder plan before pulling reordered bursts. Pass the CUDA
stream as an integer address. Pass 0 to use the default stream. See the
Configuration YAML Reference
for reorder configuration constraints.
status = daqiri.set_reorder_cuda_stream("rx_port", "rx_reorder_0", 0)
if status != daqiri.Status.SUCCESS:
raise RuntimeError(status)
Receiving Packets¶
RX Step 1: Get a burst¶
get_rx_burst() is non-blocking. It returns (Status, BurstParams | None).
Status.SUCCESS means a burst is ready. Status.NULL_PTR and
Status.NOT_READY both mean no burst is ready; other statuses report errors.
Available overloads:
status, burst = daqiri.get_rx_burst(port_id, queue_id)
status, burst = daqiri.get_rx_burst(port_id) # any queue on a port
status, burst = daqiri.get_rx_burst() # any queue on any port
status, burst = daqiri.get_rx_burst_for_connection(conn_id, server=True)
RX Step 2: Access packet data¶
For a single-segment burst (CPU-only or batched GPU), copy data out with
get_packet_bytes:
for idx in range(daqiri.get_num_packets(burst)):
status, payload = daqiri.get_packet_bytes(burst, idx)
if status == daqiri.Status.SUCCESS:
process_packet(payload)
For header-data split (two segments):
for idx in range(daqiri.get_num_packets(burst)):
status, header = daqiri.get_segment_packet_bytes(burst, 0, idx)
status, payload = daqiri.get_segment_packet_bytes(burst, 1, idx)
Other per-packet accessors:
length = daqiri.get_packet_length(burst, idx)
seg_len = daqiri.get_segment_packet_length(burst, 0, idx)
flow = daqiri.get_packet_flow_id(burst, idx)
status, rx_ts_ns = daqiri.get_packet_rx_timestamp(burst, idx)
RX hardware timestamps are available only when DAQIRI is configured with
rx.hardware_timestamps: true and the NIC and driver support hardware timestamps.
DAQIRI returns unsigned 64-bit PTP epoch nanoseconds in the same clock domain as
a PTP-synchronized CLOCK_REALTIME. Device-clock ticks are not part of the public API.
On the raw ibverbs engine, DAQIRI requests mlx5 real-time CQ timestamps only when
the device advertises that format. Devices without that capability stay on the
default mlx5 device-clock CQ format, and DAQIRI converts those raw ticks to
nanoseconds before returning them.
For RDMA/RoCE, DAQIRI requests both device-clock and wall-clock completion
fields so mlx5 preserves the ordinary TX CQ's clock format and converts receive
timestamps to epoch nanoseconds. The getter never exposes packed seconds/nanoseconds.
This works with direct, indirect, and mixed RX/TX polling modes without enabling
accurate TX scheduling.
Enabling rx.hardware_timestamps fails connection setup if the NIC or driver
cannot create a timestamp-capable receive CQ. Timestamps are available on
successful RECEIVE and receiver-side RDMA_WRITE_IMM completions;
TX completions return NOT_SUPPORTED.
An RDMA timestamp describes the receive completion of the message, which may
span multiple wire packets.
WARNING: PTP synchronization is required. DAQIRI does not validate the NIC or system clock
configuration. Timestamp values are invalid if the clocks are not PTP-synchronized.
RxQueueConfig.poll_mode accepts QueuePollMode.INDIRECT (the default) or
QueuePollMode.DIRECT. Direct mode is available on raw ibverbs and RDMA (roce://)
queues. The Python thread calling get_rx_burst() checks for ready packets while the GIL is
released. For RDMA, use get_rx_burst_for_connection(conn_id, server) to drive direct RX and TX completions;
direct SEND and RECEIVE postings contain exactly one packet each.
RX Step 3: Free buffers¶
When you are done with a burst, free it:
Or free incrementally:
daqiri.free_packet(burst, idx) # one packet (all segments)
daqiri.free_packet_segment(burst, seg, idx) # one segment of one packet
daqiri.free_all_segment_packets(burst, seg) # one segment across the burst
daqiri.free_rx_burst(burst) # metadata only
Pointer and Buffer Semantics¶
daqiri.get_packet_ptr() and daqiri.get_segment_packet_ptr() return integer
addresses. They do not return Python-managed memory objects and they do not
transfer ownership to Python.
Prefer the copy helpers unless the application deliberately needs raw addresses
for CUDA or foreign-function interop. The copy helpers handle both CPU pointers
and CUDA device pointers. Device copies use cudaMemcpy internally.
status = daqiri.copy_buffer_to_segment_packet(
burst,
seg=0,
idx=0,
data=b"payload",
nbytes=None,
src_offset=0,
dst_offset=0,
)
status, data = daqiri.get_segment_packet_bytes(
burst,
seg=0,
idx=0,
nbytes=None,
src_offset=0,
)
Runtime Queues and Memory Regions¶
The raw ibverbs engine exposes the same runtime resource lifecycle in Python.
Each accepted request returns (Status.SUCCESS, op_id). Completions from
independent requests may be interleaved, so retain results for operations other
than the one currently being awaited:
completed = {}
def wait_for_resource(op_id):
while op_id not in completed:
status, result = daqiri.poll_resource_op()
if status == daqiri.Status.NOT_READY:
continue
if status != daqiri.Status.SUCCESS:
raise RuntimeError("resource polling failed")
completed[result.op_id] = result
return completed.pop(op_id)
status, op_id = daqiri.add_memory_region_async(memory_region_config)
if status != daqiri.Status.SUCCESS:
raise RuntimeError("resource request was rejected")
result = wait_for_resource(op_id)
if result.status != daqiri.Status.SUCCESS:
raise RuntimeError("resource operation failed")
Runtime RX queues receive traffic only after a dynamic flow targets them. Delete those dynamic flows before deleting the queue; static flows cannot be removed and keep their queues in use. Queue deletion drains application-held RX buffers and pending TX work, so continue releasing bursts while polling. A memory region cannot be deleted until every queue or reorder output that references it has been removed. Metadata supports runtime batches up to at least 256 packets, or the largest startup queue batch if greater. Raw ibverbs dynamic flows share one internal matcher priority, so use non-overlapping match criteria until explicit flow priorities are exposed.
Reordered RX Bursts¶
For an overview of what RX reorder is and when to use it, see Concepts → RX Packet Aggregation and Reorder. This section covers how to consume reordered bursts from Python.
Reordered RX bursts are identified by flags on burst.hdr.hdr.burst_flags:
DAQIRI_BURST_FLAG_REORDERED: burst contains one aggregated reorder buffer.DAQIRI_BURST_FLAG_REORDER_TIMEOUT: the aggregate was emitted by the timeout path rather than by filling the configuredpackets_per_batch.DAQIRI_BURST_FLAG_DIRECT_PLACED: the ibverbs hardware path placed payloads directly in their final aggregate slots. The slots are not rearmed untilfree_rx_burst()releases the burst.
For reordered bursts, burst.hdr.hdr.max_pkt is the logical number of source
packets in the aggregate, while burst.hdr.hdr.num_pkts remains 1 because the
consumer receives one aggregate buffer.
status, burst = daqiri.get_rx_burst()
if status == daqiri.Status.SUCCESS and burst is not None:
try:
if burst.hdr.hdr.burst_flags & daqiri.DAQIRI_BURST_FLAG_REORDERED:
daqiri.synchronize_burst_event(burst)
status, info = daqiri.get_reorder_burst_info(burst)
if status == daqiri.Status.SUCCESS:
print(info.batch_id, info.aggregate_len)
status, missing_indices = daqiri.get_reorder_missing_info(burst)
if status == daqiri.Status.SUCCESS:
print(missing_indices)
finally:
daqiri.free_all_packets_and_burst_rx(burst)
synchronize_burst_event() waits for the CUDA event attached to the burst, if
any.
Transmitting Packets¶
TX Step 1: Allocate a burst¶
port_id = daqiri.get_port_id("tx_port")
batch_size = 1024
burst = daqiri.create_tx_burst_params()
daqiri.set_header(burst, port_id, 0, batch_size, 1)
if not daqiri.is_tx_burst_available(burst):
daqiri.free_tx_metadata(burst)
# retry later
else:
status = daqiri.get_tx_packet_burst(burst)
if status != daqiri.Status.SUCCESS:
daqiri.free_tx_metadata(burst)
# retry later
TX Step 2: Fill packets¶
Header helpers are available when DAQIRI is responsible for filling packet
headers. set_ipv4_header() takes the source and destination hosts as
integers, not dotted-quad strings. Convert with ipaddress.IPv4Address (or
your own packer) before calling:
import ipaddress
src_host = int(ipaddress.IPv4Address("1.2.3.4"))
dst_host = int(ipaddress.IPv4Address("5.6.7.8"))
daqiri.set_eth_header(burst, idx, "aa:bb:cc:dd:ee:ff")
daqiri.set_ipv4_header(burst, idx, ip_len, daqiri.IPPROTO_UDP, src_host, dst_host)
daqiri.set_udp_header(burst, idx, udp_len, 4096, 4096)
daqiri.set_udp_payload(burst, idx, payload_bytes)
Or copy a complete packet from a Python buffer:
packet = bytes([0] * 1064)
for idx in range(daqiri.get_num_packets(burst)):
status = daqiri.copy_buffer_to_packet(burst, idx, packet)
status = daqiri.set_packet_lengths(burst, idx, [len(packet)])
TX Step 3: Send¶
status = daqiri.send_tx_burst(burst)
# Do not free `burst` after a SUCCESS or NO_SPACE_AVAILABLE return. See the ownership note below.
For raw ibverbs, send_tx_burst() takes ownership of the burst on success and when
transmit capacity is temporarily exhausted. TxQueueConfig.poll_mode = QueuePollMode.DIRECT selects single-packet raw
ibverbs direct TX. The Python thread submits the packet and manages transmit progress; no worker
or handoff ring touches the packet. Direct TX requires exactly one packet, one pending handle,
and one thread per queue. On NO_SPACE_AVAILABLE DAQIRI has already freed the packet reservation
and burst internally. In
both cases the application must not free or otherwise access the burst
afterwards. Other failures leave ownership with the caller, who must retry or free
the burst.
For raw ibverbs indirect TX, set TxQueueConfig.buffer_mode = TxBufferMode.SHARED
(or YAML buffer_mode: shared). max_inflight_sends bounds accepted sends
independently of each region's num_bufs. Clone before submitting either burst:
status, second = daqiri.clone_tx_burst(original, 1 << 2)
# Three segments: private network header, private message header, shared payload.
The helper releases the GIL, returns a null burst on failure, and leaves the source unchanged. Bits select shared segments; unselected CPU segments receive private copies. Shared bytes must remain immutable. Submit or cancel each burst separately; storage is recycled only after all references complete or cancel. See Shared TX segments for ownership and limitations. Python packet-copy helpers expose raw writes and require the same immutability discipline.
For RDMA (roce://), direct TX posts the SEND on the Python thread and
get_rx_burst_for_connection(conn_id, server) drives its completion. A successful submission
retains the burst until DAQIRI returns it as a completion. Keep polling even
for send-only applications, then free completed bursts as usual. Direct SENDs
from huge or host_pinned memory use inline payloads when they fit the QP's
inline capacity (128 bytes requested); GPU payloads use DMA. A failed direct
RDMA posting retains caller ownership so it can be retried or freed. For a fully
direct RDMA connection, the same Python thread must submit and poll both
directions; calls from another thread return NOT_READY. Each direction has
1,024 request credits. NO_SPACE_AVAILABLE on this path retains caller ownership;
poll completions and retry.
Once get_rx_burst_for_connection() returns a completion, reuse its registered
buffers and metadata by setting the next RDMA header and packet lengths, then
resubmitting it. Acquire packet buffers only once and free the handle after its
last completion.
Timed Transmission¶
Raw Ethernet transport supports ConnectX-6 Dx and later. Packet pacing and precise timed transmission require ConnectX-7 or later:
RDMA SEND, RDMA_READ, RDMA_WRITE, and RDMA_WRITE_IMM support timed
transmission with tx.accurate_send: true in either TX polling mode on supported mlx5 hardware.
Set the RDMA header and acquire buffers before setting timestamps. Zero clears a
timestamp, and rdma_set_header() clears scheduling on reuse. Submit ahead of the
target time. RDMA targets must be strictly less than four seconds from the current
PTP-synchronized CLOCK_REALTIME at assignment and posting. A scheduled READ
gates request issuance, and zero-byte WRITE_IMM notifications can be scheduled.
Subsequent SQ operations on the connection wait behind scheduled work; RC backpressure and retries
can delay transmission further.
ptp_timestamp_ns is an unsigned 64-bit PTP epoch-nanosecond value in the same clock
domain as a PTP-synchronized CLOCK_REALTIME. Device-clock ticks are not accepted.
WARNING: PTP synchronization is required. DAQIRI does not validate the NIC or system clock
configuration. Scheduled transmission is invalid if the clocks are not PTP-synchronized.
Transmit Completion Timestamps¶
Raw ibverbs and RDMA/RoCE expose hardware TX completion timestamps without a YAML option:
import time
# After acquiring packet buffers, before submitting the burst:
status, request_id = daqiri.request_packet_tx_timestamp(burst, packet_index)
if status == daqiri.Status.SUCCESS:
daqiri.send_tx_burst(burst) # Follow the engine's usual completion ownership rules.
try:
deadline = time.monotonic() + 1.0
while True:
status, tx_ns = daqiri.get_packet_tx_timestamp(request_id)
if status != daqiri.Status.NOT_READY or time.monotonic() >= deadline:
break
if status == daqiri.Status.SUCCESS:
print(tx_ns) # NIC completion timestamp in PTP epoch ns.
finally:
daqiri.release_tx_timestamp(request_id)
The request ID remains valid after raw burst metadata is recycled or an RDMA
completion is freed. Reads are
nonblocking and repeatable; release every ID, even when abandoning a pending
request. There are 4096 request slots per engine. Direct TX requires the queue's
owner thread to call the getter to poll completions. Indirect TX uses the worker.
Scheduled sends are supported on both engines; raw ibverbs additionally supports
gathered segments, named endpoints, and shared-buffer clones. Clones need separate
requests. Requested operations get individual signaled WQEs, which increases CQ
traffic. Raw bursts with requests bypass multi-packet packing. DPDK and plain
UDP/TCP return NOT_SUPPORTED.
For RDMA, packet_index selects a SEND, RDMA_READ, RDMA_WRITE, or
RDMA_WRITE_IMM work request, which may span multiple wire packets. A READ timestamp
includes response arrival. Continue dequeuing normal completions with
get_rx_burst_for_connection() and releasing them with free_tx_burst(); reading
timestamps preserves these completions and their buffer ownership. Reusing an
RDMA header cancels its previous unsent requests. No RX timestamp or accurate-send
option is needed. When the provider cannot create a timestamp-capable TX CQ,
ordinary TX remains available and timestamp requests return NOT_SUPPORTED.
Cancelled/dropped operations, CQ errors, queue removal, and RDMA disconnection return GENERIC_FAILURE;
shutdown invalidates all IDs. The value is sampled by the NIC for the TX CQE,
independently of the requested send time. The hardware defines its measurement
point, so it is not a calibrated wire-departure timestamp. PTP synchronization is
required for comparisons with system time or another device. See the
C++ timestamp contract for details.
Writing Bursts to Storage¶
Received bursts can be written to local storage as raw packet data or as a classic pcap capture. The Python bindings expose the synchronous write helpers:
status = daqiri.daqiri_write_raw_to_file(
burst, "/mnt/nvme/capture", "packet_group_0", packet_data_offset=60,
)
status = daqiri.daqiri_write_pcap_to_file(
burst, "/mnt/nvme/capture", "packet_group_0",
)
CUDA device-backed packet segments require DAQIRI_ENABLE_GDS=ON and working
NVIDIA cuFile support. See
C++ API Usage → Writing Bursts to Storage
for the full GDS constraints. The asynchronous file-write API is C++-only at
this time.
Socket and RDMA¶
Socket helpers return connection IDs and queue routing information. The returned port and queue can then be used with the normal burst APIs:
import socket
status, conn_id = daqiri.socket_connect_to_server("192.0.2.10", 5000)
if status == daqiri.Status.SUCCESS:
status, port, queue = daqiri.socket_get_port_queue(conn_id)
# Socket options use OS constants directly; DAQIRI does not map option names.
daqiri.socket_setsockopt(conn_id, socket.SOL_SOCKET, socket.SO_RCVBUF, 8 * 1024 * 1024)
status, rx_burst = daqiri.get_rx_burst(port, queue)
if status == daqiri.Status.SUCCESS and rx_burst is not None:
try:
for idx in range(daqiri.get_num_packets(rx_burst)):
status, data = daqiri.get_packet_bytes(rx_burst, idx)
finally:
daqiri.free_all_packets_and_burst_rx(rx_burst)
tx_burst = daqiri.create_tx_burst_params()
daqiri.set_header(tx_burst, port, queue, 1, 1)
if daqiri.get_tx_packet_burst(tx_burst) == daqiri.Status.SUCCESS:
daqiri.copy_buffer_to_packet(tx_burst, 0, b"hello")
daqiri.set_packet_lengths(tx_burst, 0, [5])
# send_tx_burst() takes ownership of tx_burst on success and on a
# full-ring failure, so do not free it here.
daqiri.send_tx_burst(tx_burst)
else:
daqiri.free_tx_metadata(tx_burst)
socket_setsockopt(conn_id, level, optname, value) applies Linux setsockopt
to an existing TCP/UDP socket connection. value may be bool, int, str,
bytes, bytearray, or another bytes-like buffer. For structured options such
as timeouts, pack the native struct yourself and pass it as bytes.
RDMA follows the same tuple-returning style:
status, conn_id = daqiri.rdma_connect_to_server("192.0.2.10", 5000)
if status == daqiri.Status.SUCCESS:
burst = daqiri.create_tx_burst_params()
daqiri.rdma_set_header(
burst,
daqiri.RDMAOpCode.SEND,
conn_id,
False,
1,
0,
"TX_GPU",
)
For one-sided reads and writes, the peer exports its registration with
status, region = daqiri.rdma_get_memory_region(conn_id, "REMOTE_MEMORY").
Exchange the descriptor's integer address, length, and rkey through your
application's control channel. The requester selects RDMAOpCode.RDMA_READ
or RDMAOpCode.RDMA_WRITE in rdma_set_header, allocates local buffers with
get_tx_packet_burst, sets every packet length, then calls
rdma_set_remote_memory(burst, region, offset=0) before send_tx_burst.
A batch transfers consecutive remote ranges using the packet lengths.
Poll get_rx_burst_for_connection(conn_id, is_server) for the original request. Check
completion.rdma_status == daqiri.Status.SUCCESS before consuming READ data,
then call free_tx_burst(completion) once. completion.rdma_is_tx is True
for both reads and writes. The peer does not need to post receives for these
operations. Registration permissions, descriptor lifetime, ownership, and
partial failure behavior follow the
C++ read/write API.
RDMA_READ_IMM returns NOT_SUPPORTED.
READ requires RC; WRITE and WRITE_IMM support RC and UC. UD supports SEND/RECEIVE
only, with SEND payloads limited to the active port MTU. Receive pointers and
lengths exclude the UD GRH. UC and UD provide no delivery or RNR retry guarantee.
Select RDMAOpCode.RDMA_WRITE_IMM explicitly to write with a notification,
providing the immediate value in the creation call:
daqiri.rdma_set_header(
request, daqiri.RDMAOpCode.RDMA_WRITE_IMM,
conn_id, False, 1, wr_id, "LOCAL_MEMORY",
immediate_data=slot_id,
)
Only RDMA_WRITE_IMM accepts the immediate_data argument. Omitting it on
RDMA_WRITE_IMM uses zero, which is a valid notification value. Allocate and
submit the request normally; a packet length of zero sends only the immediate
notification. Submit it after plain writes on the same RC connection when you
need a separate notification. Plain writes do not automatically notify.
Each packet in a write-with-immediate burst consumes one peer receive credit
and produces one peer notification with the supplied value.
On UC, post those credits before sending: missing credits or lost packets can
drop notifications, and a successful local completion does not confirm delivery.
The receiver pre-posts RECEIVE credits; lengths may be zero for notification
use. Poll get_rx_burst_for_connection(conn_id, is_server) and check the status
and opcode. status, slot_id = daqiri.rdma_get_immediate_data(notification)
returns the host-order value. The notification has rdma_is_tx == False and
local packet length zero; written data resides in the separately registered
target. Batched receive credits produce independently owned completions, each
released once with free_tx_burst. With rx.hardware_timestamps: true,
status, timestamp_ns = daqiri.get_packet_rx_timestamp(notification, 0) returns
the notification completion time, including for zero-byte notifications.
Plain READ/WRITE operations have no receiver completion or RX timestamp. See the
C++ notification semantics.
RDMAMemoryRegion.access carries the exported access mask and must be exchanged
with the address, length, and rkey. Exported target MRs explicitly configure remote
read/write access. To retrieve a notification's remote write byte count, call
status, immediate, written_bytes = daqiri.rdma_get_immediate_data(notification, with_length=True).
Without with_length, the existing (status, immediate) return shape is preserved.
The notification's local packet length remains zero.
Look up server-side connection IDs:
status, conn_id = daqiri.rdma_get_server_conn_id("192.0.2.10", 5000)
status, conn_id = daqiri.socket_get_server_conn_id("192.0.2.10", 5000)
Utility Functions¶
status, mac_str = daqiri.get_mac_addr(port_id) # "aa:bb:cc:dd:ee:ff"
mac_bytes = daqiri.format_eth_addr("aa:bb:cc:dd:ee:ff")
port = daqiri.get_port_id("rx_port")
num_queues = daqiri.get_num_rx_queues(port_id)
daqiri.drop_all_traffic(port_id)
daqiri.allow_all_traffic(port_id)
daqiri.flush_port_queue(port_id, queue_id)
daqiri.print_stats()
daqiri.shutdown()
GIL Behavior¶
The bindings release the Python GIL around blocking or long-running DAQIRI
calls where the pybind11 wrapper declares py::gil_scoped_release. This
currently includes:
daqiri_init: all input forms (YAML path/string, dict,NetworkConfig,value/as_dictconfig-like objects). The GIL is released around the underlying DAQIRI call. Any Python-side conversion (PyYAMLdump,as_dict()invocation) still runs with the GIL held.daqiri_init_from_yaml_stringanddaqiri_init_from_yaml_fileget_rx_burst(all overloads) andget_rx_burst_for_connectionis_tx_burst_available,get_tx_packet_burst,send_tx_burst,clone_tx_burstcopy_buffer_to_packet,copy_buffer_to_segment_packet: released around the underlying memory copy (host orcudaMemcpy).get_packet_bytes,get_segment_packet_bytes: released around the copy out of the DAQIRI buffer.daqiri_write_raw_to_file,daqiri_write_pcap_to_filesynchronize_burst_event
The TX timestamp request, getter, and release calls keep the GIL held. The getter does not wait for completion; direct queues poll their CQ during the call.
Python code still needs its own synchronization for shared Python objects used by worker threads.
Function Reference¶
This section summarizes the Python functions exposed by the daqiri module.
The workflow sections above show the common call order and ownership rules.
Initialization, Parsing, and Lifecycle¶
| Function | Returns |
|---|---|
daqiri_init(config) |
Status |
daqiri_init(config, bindings) |
Status |
get_memory_region_requirements(config) |
(Status, dict[str, MemoryRegionRequirement]) |
daqiri_init_from_yaml_string(yaml_string) |
Status |
daqiri_init_from_yaml_file(yaml_path) |
Status |
parse_network_config(yaml_string_or_path) |
(Status, NetworkConfig) |
parse_network_config_from_yaml_string(yaml_string) |
(Status, NetworkConfig) |
parse_network_config_from_yaml_file(yaml_path) |
(Status, NetworkConfig) |
get_engine_type() |
EngineType |
shutdown() |
None |
print_stats() |
None |
String and Type Helpers¶
| Function | Purpose |
|---|---|
version_string() |
Return the DAQIRI package version as YYYY.MM.PATCH. |
version_year() / version_month() / version_patch() |
Return the CalVer components. |
abi_version() |
Return the DAQIRI shared-library ABI version. |
engine_type_from_string(str) / engine_type_to_string(type) |
Convert engine types. |
stream_type_from_string(str) / stream_type_to_string(type) |
Convert stream types. |
reorder_data_type_from_string(str) / reorder_data_type_to_string(type) |
Convert reorder data types. |
reorder_endianness_from_string(str) / reorder_endianness_to_string(endianness) |
Convert reorder endianness values. |
log_level_from_string(str) / log_level_to_string(level) |
Convert log levels. |
Burst Metadata and Packet Access¶
| Function | Purpose |
|---|---|
create_burst_params() |
Allocate generic burst metadata. |
create_tx_burst_params() |
Allocate TX burst metadata. |
set_header(burst, port, q, num, segs) |
Set burst port, queue, packet count, and segment count. |
set_num_packets(burst, num) / get_num_packets(burst) |
Set or read packet count. |
get_q_id(burst) |
Read queue ID. |
get_burst_tot_byte(burst) |
Read total byte count for a burst. |
get_packet_ptr(burst, idx) |
Return segment 0 packet address as an integer. |
get_segment_packet_ptr(burst, seg, idx) |
Return segment packet address as an integer. |
get_packet_length(burst, idx) / get_segment_packet_length(burst, seg, idx) |
Read packet or segment length. |
get_packet_flow_id(burst, idx) |
Read matched flow ID, or 0 when no flow matched. |
get_packet_rx_timestamp(burst, idx) |
Return (Status, timestamp_ns) with a PTP epoch-nanosecond timestamp. |
request_packet_tx_timestamp(burst, idx) |
Return (Status, request_id) before TX submission (raw ibverbs or RDMA). |
get_packet_tx_timestamp(request_id) |
Return (Status, timestamp_ns) from the requested NIC TX CQE, or NOT_READY. |
release_tx_timestamp(request_id) |
Release a pending or completed timestamp request; return Status. |
copy_buffer_to_packet(burst, idx, data, nbytes=None, src_offset=0, dst_offset=0) |
Copy a Python buffer into segment 0. |
copy_buffer_to_segment_packet(burst, seg, idx, data, nbytes=None, src_offset=0, dst_offset=0) |
Copy a Python buffer into a specific segment. |
get_packet_bytes(burst, idx, nbytes=None, src_offset=0) |
Return (Status, bytes) from segment 0. |
get_segment_packet_bytes(burst, seg, idx, nbytes=None, src_offset=0) |
Return (Status, bytes) from a segment. |
set_packet_lengths(burst, idx, lens) |
Set segment lengths for one packet. |
set_all_packet_lengths(burst, lens) |
Set segment lengths for every packet. |
set_packet_tx_time(burst, idx, time) |
Set one packet's scheduled TX time as PTP epoch nanoseconds (ConnectX-7+). |
RX and Reorder¶
| Function | Purpose |
|---|---|
get_rx_burst(port, queue) |
Dequeue an RX burst from a specific port and queue. |
get_rx_burst(port) |
Dequeue from any queue on a port. |
get_rx_burst() |
Dequeue from any queue on any port. |
get_rx_burst_for_connection(conn_id, server) |
Dequeue for a socket/RDMA connection. |
get_connection_id(burst) |
Read the transport connection ID recorded on an RX burst. |
set_reorder_cuda_stream(interface_name, reorder_name, stream=0) |
Set CUDA stream for a GPU reorder plan. |
get_reorder_burst_info(burst) |
Return (Status, ReorderBurstInfo). |
get_reorder_missing_info(burst) |
Return (Status, list[int]) of missing sequence slots. |
synchronize_burst_event(burst) |
Wait for the CUDA event attached to a burst, if any. |
TX and Header Fill¶
| Function | Purpose |
|---|---|
is_tx_burst_available(burst) |
Check whether TX packet buffers are available. |
get_tx_packet_burst(burst) |
Populate a TX burst with packet buffers. |
set_connection_id(burst, conn_id) |
Attach a transport connection ID to a TX burst (socket/RDMA). |
clone_tx_burst(source, shared_segment_mask) |
Return (Status, clone) with shared selected segments and private CPU copies. |
send_tx_burst(burst) |
Enqueue a populated TX burst. |
set_eth_header(burst, idx, dst_addr) |
Fill the Ethernet header destination address. |
set_ipv4_header(burst, idx, ip_len, proto, src_host, dst_host) |
Fill an IPv4 header. |
set_udp_header(burst, idx, udp_len, src_port, dst_port) |
Fill a UDP header. |
set_udp_payload(burst, idx, data) |
Copy UDP payload bytes. |
rdma_set_header(burst, op_code, conn_id, is_server, num_pkts, wr_id, local_mr_name) |
Fill RDMA TX metadata. |
rdma_get_memory_region(conn_id, mr_name) |
Return (Status, RDMAMemoryRegion) for the connection's registration. |
rdma_set_remote_memory(burst, region, offset=0) |
Set a checked remote range for RDMA read/write after setting packet lengths. |
rdma_set_header(burst, RDMAOpCode.RDMA_WRITE_IMM, conn_id, is_server, num_pkts, wr_id, local_mr_name, immediate_data) |
Create an explicit write-with-immediate request. |
rdma_get_immediate_data(burst, with_length=False) |
Return (Status, value); with_length=True returns (Status, value, written_bytes) for receiver notifications. |
rdma_get_opcode(burst) |
Return the RDMA operation code for a burst. |
Buffer Release¶
| Function | Purpose |
|---|---|
free_packet(burst, idx) |
Free all segments for one packet. |
free_packet_segment(burst, seg, idx) |
Free one segment for one packet. |
free_all_segment_packets(burst, seg) |
Free one segment across all packets. |
free_segment_packets_and_burst(burst, seg) |
Free one segment across all packets and free burst metadata. |
free_all_packets_and_burst_rx(burst) |
Free all RX packets and RX burst metadata. |
free_all_packets_and_burst_tx(burst) |
Free all TX packets and TX burst metadata. |
free_rx_burst(burst) / free_tx_burst(burst) |
Free burst metadata. For RDMA request completions, free_tx_burst also returns the packet buffers. |
free_rx_metadata(burst) / free_tx_metadata(burst) |
Free only RX or TX metadata. |
File I/O¶
| Function | Purpose |
|---|---|
daqiri_write_raw_to_file(burst, absolute_path, file_prefix, packet_data_offset) |
Write burst packets as raw files. |
daqiri_write_pcap_to_file(burst, absolute_path, file_prefix) |
Append burst packets to a pcap file. |
Ports, Traffic, Socket, and RDMA¶
Raw userspace TX bypasses the automatic route and ARP processing performed for
Linux sockets. With the raw ibverbs engine, call
resolve_ipv4_mac(port, dst, timeout_ms) after initialization, then reuse the
returned MAC in packet templates. This is a one-shot lookup: DAQIRI does not
cache or refresh the returned value, so the application must re-query after a
peer, gateway, route, link, or network-namespace change, or according to its
own failure/refresh policy. Configure rx.flow_isolation: true so unmatched
ARP traffic remains on the kernel path when the port also has DAQIRI RX queues.
A TX-only interface with no rx.queues needs no rx section. See
Destination MAC resolution.
| Function | Purpose |
|---|---|
get_mac_addr(port) |
Return (Status, "aa:bb:cc:dd:ee:ff"). |
resolve_ipv4_mac(port, dst, timeout_ms) |
Return (Status, mac_string) after resolving a host-order integer or dotted-decimal IPv4 destination through the raw ibverbs port's Linux route and ARP table. The binding releases the GIL while waiting. |
format_eth_addr(addr) |
Return six MAC-address bytes from a xx:xx:xx:xx:xx:xx MAC string. Invalid input returns zero bytes. |
get_port_id(key) |
Resolve an interface name or PCIe address to a port ID. |
get_num_rx_queues(port_id) |
Return configured RX queue count for a port. |
drop_all_traffic(port) |
Install a high-priority drop rule on a port. |
allow_all_traffic(port) |
Remove a drop rule installed by drop_all_traffic. |
flush_port_queue(port, queue) |
Drain stale packets from a port queue. |
add_rx_flow_async(port, flow) |
Return (Status, op_id) after enqueueing one dynamic RX flow create. |
add_rx_flows_async(port, flows) |
Return (Status, op_id) after enqueueing a dynamic RX flow batch create. One completion returns flow_ids in input order. |
delete_flow_async(flow_id) |
Return (Status, op_id) after enqueueing deletion of one dynamic flow. |
poll_flow_op() |
Return (Status, FlowOpResult), or NOT_READY when no dynamic flow operation has completed. |
add_memory_region_async(config[, binding]) |
Add an owned or externally bound ibverbs memory region and return (Status, op_id). |
delete_memory_region_async(name) |
Remove an unused runtime memory region and return (Status, op_id). |
add_rx_queue_async(port, config) / add_tx_queue_async(port, config) |
Add a raw-ibverbs queue and return (Status, op_id). |
delete_rx_queue_async(port, queue_id) / delete_tx_queue_async(port, queue_id) |
Begin drain-based queue removal and return (Status, op_id). |
poll_resource_op() |
Return (Status, ResourceOpResult), or NOT_READY while no operation has completed. |
socket_connect_to_server(server_addr, server_port[, src_addr]) |
Return (Status, conn_id). |
socket_get_port_queue(conn_id) |
Return (Status, port, queue). |
socket_get_server_conn_id(server_addr, server_port) |
Return (Status, conn_id). |
socket_setsockopt(conn_id, level, optname, value) |
Apply a Linux socket option to an existing TCP/UDP socket connection. |
rdma_connect_to_server(server_addr, server_port[, src_addr]) |
Return (Status, conn_id). |
rdma_get_port_queue(conn_id) |
Return (Status, port, queue). |
rdma_get_server_conn_id(server_addr, server_port) |
Return (Status, conn_id). |
Dynamic RX flows are RX-only in v1. The action attribute remains the queue-action
shorthand. Set flow.action.id = 0 for direct steering, or set
flow.action.ids = [0, 1] to enable automatic flow-affine IPv4/UDP five-tuple
RSS. A one-entry ids list is direct steering. Lists must be non-empty,
duplicate-free, and contain configured RX queue IDs. An unchanged five tuple
stays on one queue; balanced packet totals require enough distinct, reasonably
balanced tuples. Multi-queue RSS is unsupported for eCPRI and reorder-owned
flows, and hardware creation errors are returned rather than falling back to a
single queue. Use ordered actions when a raw DPDK or raw ibverbs dynamic rule needs hardware
VLAN pop or VXLAN/GRE/NVGRE decapsulation before the final queue action. Static TX
encapsulation/push rules are configured in YAML under tx.flows.
Constants¶
| Name | Description |
|---|---|
ADV_NETWORK_HEADER_SIZE_BYTES |
Advanced network header size constant. |
MAX_NUM_RX_QUEUES |
Maximum configured RX queues. |
MAX_NUM_TX_QUEUES |
Maximum configured TX queues. |
MAX_INTERFACES |
Maximum configured interfaces. |
MAX_NUM_SEGS |
Maximum packet segments per burst. |
DAQIRI_VERSION |
DAQIRI package version as YYYY.MM.PATCH. |
DAQIRI_VERSION_YEAR / DAQIRI_VERSION_MONTH / DAQIRI_VERSION_PATCH |
CalVer components. |
DAQIRI_ABI_VERSION |
DAQIRI shared-library ABI version. |
DAQIRI_BURST_FLAG_REORDERED |
Burst flag indicating a reordered aggregate. |
DAQIRI_BURST_FLAG_REORDER_TIMEOUT |
Burst flag indicating a reorder timeout aggregate. |
DAQIRI_BURST_FLAG_DIRECT_PLACED |
Burst flag indicating an ibverbs hardware-placed aggregate. |
MEM_ACCESS_LOCAL |
Local memory access flag. |
MEM_ACCESS_RDMA_WRITE |
RDMA write memory access flag. |
MEM_ACCESS_RDMA_READ |
RDMA read memory access flag. |
MEM_ACCESS_RELAXED_ORDERING |
Opt-in PCIe relaxed-ordering flag for NIC DMA into the region (raw ibverbs and RoCE engines). |
Enums¶
| Enum | Values |
|---|---|
Status |
SUCCESS, NULL_PTR, NO_FREE_BURST_BUFFERS, NO_FREE_PACKET_BUFFERS, NOT_READY, INVALID_PARAMETER, NO_SPACE_AVAILABLE, NOT_SUPPORTED, GENERIC_FAILURE, CONNECT_FAILURE, INTERNAL_ERROR, RESOURCE_IN_USE, ALREADY_EXISTS |
RDMAOpCode |
CONNECT, SEND, RECEIVE, RDMA_WRITE, RDMA_WRITE_IMM, RDMA_READ, RDMA_READ_IMM, INVALID |
RDMACompletionType |
RX, TX, INVALID |
EngineType |
UNKNOWN, DEFAULT, DPDK, SOCKET, RDMA, IBVERBS |
Direction |
RX, TX, TX_RX |
BufferLocation |
CPU, GPU, CPU_GPU_SPLIT |
MemoryKind |
HOST, HOST_PINNED, HUGE, DEVICE, NULL_MR, INVALID |
StreamType |
RAW, SOCKET, INVALID |
QueuePollMode |
INDIRECT, DIRECT, INVALID |
TxBufferMode |
CYCLIC, SHARED |
LoopbackType |
DISABLED, LOOPBACK_TYPE_SW, LOOPBACK_TYPE_HW |
RDMAMode |
CLIENT, SERVER, INVALID |
RDMATransportMode |
RC, UC, UD, INVALID |
SocketMode |
CLIENT, SERVER, INVALID |
FlowType |
QUEUE, VLAN_PUSH, VLAN_POP, TUNNEL_ENCAP, TUNNEL_DECAP |
TunnelType |
NONE, VXLAN, GRE, NVGRE |
FlowMatchType |
IPV4_UDP, FLEX_ITEM, ECPRI, ETHERNET |
FlowOpType |
ADD_RX, ADD_RX_BATCH, DELETE |
ResourceOpType |
ADD_MEMORY_REGION, DELETE_MEMORY_REGION, ADD_RX_QUEUE, DELETE_RX_QUEUE, ADD_TX_QUEUE, DELETE_TX_QUEUE |
ResourceState |
CREATING, ACTIVE, DRAINING, REMOVED, FAILED |
ReorderMethod |
INVALID, SEQ_BATCH_NUMBER, SEQ_PACKETS_PER_BATCH |
ReorderDataType |
SAME, INT4, INT8, INT16, INT32, FP16, BF16, FP32, FP64, INVALID |
ReorderEndianness |
HOST, NETWORK, INVALID |
LogLevel |
TRACE, DEBUG, INFO, WARN, ERROR, CRITICAL, OFF |
ErrorGlobalStats |
OUT_OF_RX_BUFFERS, RX_QUEUE_FULL, METADATA_BUF_DEPLETED, SENTINEL |
Data and Config Classes¶
All exposed classes have a default constructor and read/write attributes.
Config classes mirror the C++ configuration structs, with Python attribute
names that mostly omit the trailing underscore from the C++ member name (e.g.
name_ → name). A few fields are renamed for clarity, for example mrs_
→ memory_regions and ifs_ → interfaces.
| Class | Purpose |
|---|---|
BurstParams |
Opaque burst handle used by RX, TX, free, and packet-access helpers. Exposes hdr, connection_id, rdma_wr_id, and read-only rdma_status / rdma_is_tx. |
BurstHeader |
Wrapper for BurstHeaderParams. |
BurstHeaderParams |
Burst metadata: packet count, port, queue, segment count, byte totals, and reorder flags. |
ReorderBurstInfo |
Metadata for reordered aggregate bursts. |
NetworkConfig |
Top-level parsed DAQIRI configuration. |
CommonConfig |
Global stream type, engine selection, direction, loopback, and core settings. (Transport protocol is derived from the endpoint URI scheme.) |
InterfaceConfig |
Per-interface address, socket/RoCE/RDMA, RX, and TX configuration. |
RxConfig |
RX flow isolation, timestamps, queues, flows, flex items, and reorder configs. |
TxConfig |
TX accurate-send flag, queues, and flows. |
CommonQueueConfig |
Shared queue fields: name, ID, batch size, split boundary, CPU core, memory regions, and offloads. |
RxQueueConfig |
RX queue wrapper with common queue fields, timeout, QueuePollMode, and the raw ibverbs cqe_compression flag. |
TxQueueConfig |
TX queue wrapper with common fields, QueuePollMode, TxBufferMode, and max_inflight_sends. |
RDMAMemoryRegion |
Registered address, length, rkey, and access to exchange with a connected RDMA peer. |
MemoryRegionConfig |
Memory region kind, affinity, access flags, sizes, counts, and ownership. |
ExternalMemoryRegion |
Non-owning integer address and byte capacity. |
MemoryRegionRequirement |
Effective kind, slot size, buffer count, capacity, and alignment. |
VlanActionConfig |
VLAN push parameters: VLAN ID, priority, DEI, and ethertype. |
TunnelConfig |
VXLAN, GRE, or NVGRE tunnel template fields for hardware encap/decap actions. |
FlowAction |
Flow action type, scalar queue target id, queue-list target ids, optional VLAN config, and optional tunnel config. |
EthernetMatch |
Optional source and destination MAC-address match fields. |
FlowMatch |
Flow match fields for Ethernet, UDP, IPv4, eCPRI, and flex-item matching. |
FlowConfig |
Static named flow rule combining legacy action, ordered actions, and match fields. |
FlowRuleConfig |
Dynamic RX flow rule combining legacy action, ordered actions, and match fields. |
FlowOpResult |
Dynamic flow operation completion. Batch adds return flow_ids in input order. |
ResourceOpResult |
Runtime memory-region or queue completion with operation ID, type, state, status, and resource identity. |
FlexItemConfig |
Flexible parser item configuration. |
FlexItemMatch |
Flexible parser match value and mask. |
SocketConfig |
Socket client/server endpoint URI, legacy IP/port, and timing settings. |
RoCEConfig |
RoCE transport settings. |
RDMAConfig |
RDMA mode, transport mode, and port. |
ReorderConfig |
Reorder engine/cyclic-sequence contract, type, memory region, payload offset, flows, method, and data type conversion. |
ReorderBitFieldConfig |
Bit offset and width for extracting reorder fields. |
ReorderSeqBatchNumberConfig |
Sequence-number, batch-number, and packets-per-batch field config. |
ReorderSeqPacketsPerBatchConfig |
Sequence-number and packets-per-batch field config. |
ReorderDataTypesConfig |
Optional reorder input/output data type conversion settings. |
MemoryKind.HUGE explicitly requires hugetlb-backed memory. Initialization fails if a
compatible hugepage pool cannot satisfy a DAQIRI-owned region; DAQIRI does not silently use
regular pages or transparent hugepages. Select MemoryKind.HOST when regular host memory is
intended.
MemoryKind.NULL_MR represents a raw ibverbs RX discard region with no backing storage.
Set MemoryRegionConfig.buf_size to the discard span and reference the region by name
in CommonQueueConfig.memory_regions, just as in YAML and dictionary configurations.
A positive buf_size is required only when another MR follows. The final null MR may
omit buf_size in YAML/dictionary configs or use 0 to discard the remainder. See the
null-region example.