Writing a Zero-Copy Protocol Parser in Rust

Key Takeaways

• Use repr(transparent) to make a view type that is a slice, and use the type system to handle ownership and mutability.
  
  

• Be intentional about what you validate, and when. Only validate what is required for memory safety, and leave the rest to the user.
  
  

• Use bitshifting to read and write unaligned bitfields, ideally with a generic helper that takes a bit range and a type.
  
  

• Use a Layout type to describe the wire format, and derive all offsets and lengths from it, so that the view and model cannot drift apart.
  
  

• Only use unsafe when you have a pattern that guarantees memory safety, and test it thoroughly with fuzzing and property testing.

Networking needs to be fast. A router should reach line rate, which today means hundreds of Gbps. At those rates, a small packet arrives every few nanoseconds, so any code on the hot path has only a few dozen cycles per packet.

Traditional parsers extracts every field of a buffer, has to allocate memory, and creates an owned struct. With routers reading only a few fields to decide what to do, most of that work is wasted and time is spent which we simply don't have.

Zero-copy parsing avoids this. Instead of copying any data, we directly access the packet buffer, wherever it is, and read fields only when they are needed.

From our benchmarks (which can be found at the end of this post), this is 10x faster than parsing into an owned struct, 20 ns per packet instead of 217 ns.

If you have worked with C, you already know the idea. This post will show how to do it in Rust, using its type system to make it safe and ergonomic.

Existing solutions

Rust has a good ecosystem for parsing, and you should check it out before writing your own parser. Writing your own means writing unsafe code, and the existing crates are already tested and fuzzed. We looked at three groups of crates:

• zerocopy and bytemuck map a Rust struct onto a byte buffer with no per-field cost. The Rust struct layout is the wire layout, so the format must be expressible as a repr(C) struct, e.g. byte-aligned fields of fixed size.
  
  

• modular-bitfield and bitfield handle bitfields, which can supplement zerocopy for some formats.
  
  

• deku, binrw and nom handle bitfields and length-prefixed formats well, but they parse into owned structures which is what we wanted to avoid.

SCION headers have two properties which make the first group somewhat clunky. Fields can be smaller than a byte, and some fields have a length determined by an earlier field.

The third group is ruled out because we want to avoid copying, and they all parse into owned structures. So we wrote our own parser, inspired by etherparse.

Views: a type that is a slice

To begin our zero-copy journey, we need a type that represents a typed byte slice. These we call views.

The obvious way to write a view type is to wrap a slice.

struct MyView<'a> {
    data: &'a [u8],
}

This works, and we can add accessors to MyView. But mutability and ownership are now part of our type, so we must write out every combination ourselves.

struct MyView<'a>     { data: &'a [u8] }
struct MyViewMut<'a>  { data: &'a mut [u8] }
struct MyViewOwned    { data: Box<[u8]> }

That is three types per protocol element, each with its own accessors. Macros or traits can reduce the boilerplate, but with the many different elements in a SCION packet it is still a lot of code to write and maintain.

Instead of wrapping the slice, we can make our type a slice.

#[repr(transparent)]
struct MyView([u8]);

Two things happen here.

• The single field is an unsized [u8], which makes MyView unsized as well, so we can only hold it behind a pointer.
  
  

• #[repr(transparent)] tells the compiler that MyView has the same layout and pointer metadata as the field it wraps.

Together this means a reference to MyView and a reference to [u8] are the same thing at runtime. Thanks to this, Rust's type system handles mutability and ownership for us. We can use the same type in all four situations.

&[u8]      =  &MyView          // shared, borrowed
&mut [u8]  =  &mut MyView      // exclusive, borrowed
Box<[u8]>  =  Box<MyView>      // owned
Rc<[u8]>   =  Rc<MyView>       // shared, refcounted

This is similar to how str, Path and CStr work.

There are some disadvantages to this approach.

Views are !Sized. We can never hold one by value, so no view on the stack, in a struct field, or returned from a function. Everything is &, &mut or Box.

Views cannot hold extra data; e.g., a cached offset or a parsed length.

For us, these are acceptable. Every packet is already behind a pointer, because it lives in a receive buffer we do not own. And we don't plan to cache anything in the view since the accessors are cheap enough to compute on demand.

Creating a view

Our views are not exactly normal structs, so we cannot build one field by field. Instead, we validate a byte slice and relabel it.

This is done in three steps:

• Check that the data is large enough for every field the type exposes.
  
  

• Split off exactly that many bytes.
  
  

• Transmute the reference to the slice into a reference to our view type.
  

#[repr(transparent)]
pub struct HeaderView([u8]);

impl HeaderView {
    const LEN: usize = 10;

    pub fn try_parse_mut(data: &mut [u8]) -> Result<(&mut Self, &mut [u8]), Error> {
        if data.len() < Self::LEN {
            return Err(Error::TooShort);
        }

        let (head, rest) = data.split_at(Self::LEN);

        // SAFETY: `Self` is a `repr(transparent)` wrapper around `[u8]`, so
        // `&[u8]` and `&Self` have the same layout and pointer metadata.
        let view = unsafe { std::mem::transmute::<&[u8], &Self>(head) };

        Ok((view, rest))
    }

    pub fn try_parse(data: &[u8]) -> Result<(&Self, & [u8]), Error> {
       //... Immutable variant
    }
}

We also return the unused rest of the slice, ideally in two variants: mut and immutable. This lets callers parse one header after another without tracking offsets themselves.

In compiled code this is a single length comparison. The transmute produces no instructions at all, it only exists to satisfy the type system.

The length check is the one thing which guarantees our core invariant for views.

A &HeaderView only exists if its bytes are large enough for every field HeaderView exposes.

As long as we follow this invariant, every accessor can skip bounds checks.

What is transmute?

std::mem::transmute reinterprets the bits of a value as a different type. The compiler only checks that both types have the same size. Beyond that it takes our word that the reinterpretation is correct. Getting it wrong is undefined behaviour, so it is unsafe.

Note what we transmute here. Not [u8] into HeaderView, but &[u8] into &HeaderView. We relabel a pointer, which is valid because:

• &[u8] is a fat pointer, an address plus a usize element count, so two usizes wide.
  
  

• HeaderView's only field is an unsized [u8], which makes the struct unsized too. A reference to it carries the same element count metadata and is also two usizes wide.
  
  

• #[repr(transparent)] guarantees that the wrapper's layout and pointer metadata are exactly those of the field it wraps.

Both references have the same size and interpret both halves the same way, so the transmute is sound. No instructions are emitted, because we relabel a pointer instead of moving data.

The third point is arguably the most important one. Without #repr(transparent), Rust promises nothing about the struct's representation, and the transmute becomes undefined behaviour.

If you prefer to be explicit about what happens, a pointer cast says the same thing, missing the size similarity transmute enforces.

let view = unsafe { &*(head as *const [u8] as *const Self) };

Reading and writing fields

Byte aligned Fields

If you know how to read bytes from a slice, you know how we read fields. Byte-aligned fields at fixed offsets are straightforward.

impl HeaderView {
    pub fn field1(&self) -> u32 {
        u32::from_be_bytes(self.0[0..4].try_into().unwrap())
    }

    pub fn field3(&self) -> &[u8] {
        &self.0[6..10]
    }
}

field3 returns a borrow of the underlying bytes. There is no copy, and the lifetime ties the result back to the original buffer.

Unaligned bit Fields

Fields smaller than a byte need bit manipulation. Say we want a 4 bit field at bit offset 14, which crosses a byte boundary.

let buf = [0b0000_0000, 0b0000_0011, 0b1100_0000, 0b0000_0000];
//                               ^^    ^^
//                           The bits we want, 14..18

First we load the bytes that contain the field. Bits 14 to 18 are in bytes 1 and 2, so we read those two bytes into a temporary storage large enough to hold the field. We call this the lane.

let lane = u16::from_be_bytes([buf[1], buf[2]]);

//                byte 0        byte 1       byte 2      byte 3
//  buffer    | xxxx xxxx | xxxx xx11 | 11xx xxxx | xxxx xxxx |  bits 14..18, across the boundary
//  lane                  | xxxx xx11   11xx xxxx |               loaded as one u16

Next we shift the field down to bit 0, fully to the right of our lane.

The lane covers bits 8 to 24, and our field ends at bit 18, so there are 24 - 18 = 6 bits below it.

let lane = lane >> 6;

//  before   | xxxx xx11   11xx xxxx |
//  after    | 0000 00xx   xxxx 1111 |  field is now in the low 4 bits

Finally we apply a mask to remove the bits above the field.

let lane = lane & 0b0000_0000_0000_1111;

//  before    | 0000 00xx   xxxx 1111 |
//  after     | 0000 0000   0000 1111 |

The full read looks like this.

fn read_field(buf: &[u8]) -> u16 {
    // Copy the two bytes that contain the field into a temporary storage
    let mut lane = u16::from_be_bytes([buf[1], buf[2]]);
    lane = lane >> 6;  // Shift right 6 bits to get the field to bit 0
    lane & 0b1111  // Mask any bit above the field, leaving only the low 4 bits
}

To write a value back, we use the same logic in reverse.

We load the lane, clear the bits of the field, shift the value into position, and set the bits.

fn write_field(buf: &mut [u8], value: u16) {
    let mut value = value & 0b1111; // Truncate the value to the field width
    value = value << 6; // Align the value to its position in the lane

    // Copy the relevant bytes into a temporary storage
    let mut lane = u16::from_be_bytes([buf[1], buf[2]]);
    let mask = 0b1111 << 6; // Create a mask at the field position

    lane = lane & !mask; // Zero out the field from the lane, keeping the other bits
    lane = lane | value; // Set the field in the lane

    let [b1, b2] = lane.to_be_bytes(); // Write the lane back to the buffer
    buf[1] = b1;
    buf[2] = b2;
}

To not have to write these bit manipulations by hand for every field, we wrote a generic helper that takes a bit range and a type, unchecked_bit_range_be_read and unchecked_bit_range_be_write. They are generic over the field type and handle any bit range up to 128 bits. The actual instructions generated by them are roughly the same as the hand-written code above, even on aligned fields.

// SAFETY: the caller guarantees the bit range lies within `buf`.
let port = unsafe { unchecked_bit_range_be_read::<u16>(&self.0, 16..32) };

These functions are unsafe because they use get_unchecked internally, which removes the bounds check on every access, this is optional and trades safety for some additional performance.

Fields without a fixed size

Two things get harder when the length of a field depends on the data.

• We have to calculate the total size of the element before we can create the view, because the invariant covers the whole element.
  
  

• We have to calculate the offset of every field that follows a variable-length field.

As an example, here is a socket address whose size depends on a type field.

//  0                   1                   2                   3
//  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// |     Type      |      RSV      |             Port              |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// |                                                               |
// +               Address (4 or 16 bytes, per Type)               +
// |                                                               |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
#[repr(transparent)]
pub struct SocketAddrView([u8]);
impl SocketAddrView {
  const STATIC_LEN: usize = 4; // Type, RSV, Port
  pub fn try_parse(data: &[u8]) -> Result<(&Self, &[u8]), Error> {
        // First check that the fixed part is present.
      if data.len() < Self::STATIC_LEN {
          return Err(Error::TooShort);
      }
        // Then read the field that determines the length of the rest.
      let total_len = match data[0] {
          1 => Self::STATIC_LEN + 4,  // IPv4
          2 => Self::STATIC_LEN + 16, // IPv6
          _ => return Err(Error::UnknownAddrType), // Since the type is length relevant, we cannot continue without it.
      };
        // Validate the dynamic length before we create the view.
      if data.len() < total_len {
          return Err(Error::TooShort);
      }
        // Convert the slice into a view, and return the rest of the slice.
      let (head, rest) = data.split_at(total_len);
        // SAFETY: `head` is exactly `total_len` bytes, which covers every field.
      let view = unsafe { std::mem::transmute::<&[u8], &Self>(head) };
      Ok((view, rest))
  }
  pub fn addr_bytes(&self) -> &[u8] {
      let len = match self.0[0] {
          1 => Self::STATIC_LEN + 4,  // IPv4
          2 => Self::STATIC_LEN + 16, // IPv6
          _ => unreachable!("validated on construction"),
      };
      &self.0[Self::STATIC_LEN..len]
  }
}

Here we first check that the static part is present, then read the type field to calculate the total length of the element, and finally check that the slice is large enough for that length.

For this example, an unknown type is a parse error. Without the type we cannot calculate a length, so there is no view to return.

One note about this strategy. We check the size of the slice only once, when we create the view, and we check it against a length derived from a field in the data.

Mutating a field through a &mut SocketAddrView could invalidate the check afterwards, because the view would still cover bytes that are now wrong.

Setters for size-determining fields therefore have to be unsafe, or must not exist.

Views with a known size

Last edge to cover, if we know the exact size of a view, we can wrap an array instead of a slice.

#[repr(transparent)]
pub struct TimestampView([u8; 8]);
impl TimestampView {
  pub const LEN: usize = 8;
  pub fn try_parse(data: &[u8]) -> Result<(&Self, &[u8]), Error> {
      if data.len() < Self::LEN {
          return Err(Error::TooShort);
      }
      let (head, rest) = data.split_at(Self::LEN);
      // SAFETY: `head` is exactly LEN bytes, and `Self` is a transparent
      // wrapper around `[u8; LEN]`, which has alignment 1.
      let view = unsafe { &*(head.as_ptr() as *const Self) };
      Ok((view, rest));
  }
  pub fn micros(&self) -> u64 {
      u64::from_be_bytes(self.0)
  }
}

Since the compiler knows the size of the Array, it knows the size of the view, making it Sized. This allows us to use it e.g. in a slice of repeated elements.

//  0                   1                   2                   3
//  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// |     Count     |                                               |
// +-+-+-+-+-+-+-+-+                                               +
// |                                                               |
// +               Timestamps (Count * 8 bytes each)               +
// |                                                               |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

#[repr(transparent)]
pub struct TimestampListView([u8]);
impl TimestampListView {
    pub fn try_parse(data: &[u8]) -> Result<(&Self, &[u8]), Error> {
           // ..
    }
    pub fn timestamps(&self) -> &[TimestampView] {
        let count = self.0[0] as usize;
        let timestamps_start = 1;
        let timestamps_end = timestamps_start + count * TimestampView::LEN;
        let timestamp_slice = &self.0[timestamps_start..timestamps_end];
        // SAFETY: TimestampView is #[repr(transparent)] over [u8; 8], and timestamp_slice is exactly count * 8 bytes long.
        unsafe {
            std::slice::from_raw_parts(
                timestamp_slice.as_ptr() as *const TimestampView,
                count,
            )
        }
    }
}

for ts in list.timestamps() now works, and iterating costs nothing beyond advancing the pointer.

Layout, View and Model

A pattern shows up in the sections above. Every view needs field offsets and lengths, both when it is created and when a field is read. And so far, nothing lets a user build a packet from scratch, which they will want to do.

This is where our architecture splits into three layers.

• A View is the zero-copy type from the rest of this post. It is the hot path.
  
  

• A Model is a regular owned Rust struct with real types. It is the cold path, used for construction, tests and logging.
  
  

• A Layout describes the wire format as data, e.g. where fields are and how long the element is. It is used by both the view and the model, so they cannot drift apart.

bytes ── parse ─▶ &View ── accessors ──▶ field values     hot path
   ▲                 │  \
   │                 │   ── informed by ── Layout
   │                 │  /
   └──── encode ── Model ── accessors ──▶ field values     cold path

We separated the Layout because parsing and encoding are the same calculations in opposite directions. If this is implemented at multiple places, changes will be missed and cause problems. The layout as a single source of truth for offsets avoids this. 

For a user this gives an API where they can either read fields in place or build a new packet.

// Build a packet with a Model
let packet = ScionUdpPacket::new(
    "[1-1,2.2.2.2]:1234".parse::<ScionSocketAddr>()?,
    "[1-1,3.3.3.3]:5678".parse::<ScionSocketAddr>()?,
    DpPath::Empty,
    b"payload".to_vec(),
);
let encoded = packet.try_encode_to_vec()?;

// Parse it back into a View, with no allocation and no copy
let (view, _rest) = ScionUdpPacketView::try_from_slice(&encoded)?;
let dst_ia = view.header().dst_ia();

// Or pay once to get an owned Model back
let model = view.try_to_model()?;

Below is the socket address from the previous section, written out across all three layers.

The Layout

First is our source of truth. We write out all the field positions and lengths, deriving everything else from that.

We use bits instead of bytes in general, since fields are not always byte-aligned.

For static fields, the ranges are constants. For dynamic fields, whose position or size depends on other data, they become methods on the layout.

use std::ops::Range;

#[derive(Copy, Clone, PartialEq, Eq, Debug)]
#[repr(u8)]
pub enum AddrType {
    V4 = 1,
    V6 = 2,
}

impl AddrType {
    const fn from_wire(raw: u8) -> Option<Self> {
        match raw {
            1 => Some(Self::V4),
            2 => Some(Self::V6),
            _ => None,
        }
    }
}

#[derive(Copy, Clone)]
pub struct SocketAddrLayout {
    addr_type: AddrType,
}

impl SocketAddrLayout {
    // Fixed positions, in bits from the start of the element.
    pub const TYPE: Range<usize> = 0..8;
    pub const RSV: Range<usize> = 8..16;
    pub const PORT: Range<usize> = 16..32;

    /// Bytes we need before we can read `TYPE` and decide the rest.
    pub const STATIC_LEN: usize = 4;

    // The layout depends on the address type, so we store it in the struct.
    pub const fn new(addr_type: AddrType) -> Self {
        Self { addr_type }
    }

    // The position depends on the data, so this is a method, not a constant.
    pub const fn address(&self) -> Range<usize> {
        match self.addr_type {
            AddrType::V4 => 32..64,   // 4 bytes
            AddrType::V6 => 32..160,  // 16 bytes
        }
    }

    pub const fn total_len(&self) -> usize {
        /// Derived, so changes in address() are reflected here.
        self.address().end / 8
    }
}

These layouts should be as DRY as possible, all methods should derive their values from other methods, so that a change in one place is reflected everywhere.

The View

#[repr(transparent)]
pub struct SocketAddrView([u8]);

impl SocketAddrView {
    pub fn try_parse(data: &[u8]) -> Result<(&Self, &[u8]), Error> {
        if data.len() < SocketAddrLayout::STATIC_LEN {
            return Err(Error::TooShort);
        }

        // SAFETY: STATIC_LEN bytes are present, and TYPE lies within them.
        let raw = unsafe {
            unchecked_bit_range_be_read::<u8>(data, SocketAddrLayout::TYPE)
        };
        let addr_type = AddrType::from_wire(raw).ok_or(Error::UnknownAddrType)?;

        // Only now can we build the layout, and with it know the length.
        let layout = SocketAddrLayout::new(addr_type);
        if data.len() < layout.total_len() {
            return Err(Error::TooShort);
        }

        let (head, rest) = data.split_at(layout.total_len());
        // SAFETY: `head` covers every range the layout describes, and `Self` is
        // a `repr(transparent)` wrapper around `[u8]`.
        let view = unsafe { std::mem::transmute::<&[u8], &Self>(head) };
        Ok((view, rest))
    }

    pub fn addr_type(&self) -> AddrType {
        // SAFETY: the length was checked when the view was created.
        let raw = unsafe {
            unchecked_bit_range_be_read::<u8>(&self.0, SocketAddrLayout::TYPE)
        };
        AddrType::from_wire(raw).expect("validated on construction")
    }

    /// Safe, because `PORT` has a fixed position and cannot change the length.
    pub fn set_port(&mut self, port: u16) {
        // SAFETY: the length was checked when the view was created.
        unsafe {
            unchecked_bit_range_be_write::<u16>(&mut self.0, SocketAddrLayout::PORT, port)
        }
    }

    /// # Safety
    /// The new type must imply the same encoded length as the current one.
    /// Otherwise the length checked on construction no longer holds, and every
    /// later field access may read out of bounds.
    pub unsafe fn set_addr_type(&mut self, addr_type: AddrType) {
        unsafe {
            unchecked_bit_range_be_write::<u8>(
                &mut self.0, SocketAddrLayout::TYPE, addr_type as u8,
            )
        }
    }

    pub fn address(&self) -> &[u8] {
        let bits = SocketAddrLayout::new(self.addr_type()).address();
        debug_assert!(bits.start.is_multiple_of(8), "We are doing a read without a helper, must be aligned");
        debug_assert!(bits.end.is_multiple_of(8), "We are doing a read without a helper, must be aligned");

        &self.0[bits.start / 8..bits.end / 8]
        // Optionally, this may also convert to a `std::net::IpAddr`, for this example we leave it as a byte slice.
    }
}

There are two things worth pointing out here:

set_port and set_addr_type both write a fixed-position field with the same helper, but one is safe and the other is not. Any setter, which touches size-relevant fields must be unsafe, the caller must understand that this changes the length of the view and invalidates the length check done on construction.

Since we need to work on bit ranges, the layout gives us bits which we need to convert to bytes for the slice. In our actual implementation, this is done with a helper struct BitRange.

The Model

The model is a normal Rust struct with real types. It's not meant to be used on the hot path, so it can allocate and copy freely.

pub struct SocketAddrModel {
    pub port: u16,
    pub address: IpAddr,
}

impl SocketAddrModel {
    fn addr_type(&self) -> AddrType {
        match self.address {
            IpAddr::V4(_) => AddrType::V4,
            IpAddr::V6(_) => AddrType::V6,
        }
    }

    pub fn encode_to(&self, buf: &mut [u8]) -> Result<usize, Error> {
        // The same layout the parser uses, in the opposite direction.
        let layout = SocketAddrLayout::new(self.addr_type());

        if buf.len() < layout.total_len() {
            return Err(Error::BufferTooSmall);
        }

        // SAFETY: the check above covers every range the layout hands out.
        unsafe {
            unchecked_bit_range_be_write::<u8>(
                buf, SocketAddrLayout::TYPE, self.addr_type() as u8,
            );
            unchecked_bit_range_be_write::<u8>(buf, SocketAddrLayout::RSV, 0);
            unchecked_bit_range_be_write::<u16>(buf, SocketAddrLayout::PORT, self.port);
        }

        let bits = layout.address();
        let dst = &mut buf[bits.start / 8..bits.end / 8];
        match self.address {
            IpAddr::V4(v4) => dst.copy_from_slice(&v4.octets()),
            IpAddr::V6(v6) => dst.copy_from_slice(&v6.octets()),
        }

        Ok(layout.total_len())
    }
}

The encode_to method is the opposite of try_parse. It uses the same layout, but writes to a buffer instead of reading from one.

Validation

When designing a parser we have to decide what we validate. There are three main kinds of validation.

• Validate that the data is safe to access, e.g. the data is large enough to contain all fields.
  
  

• Validate that the data is semantically correct, e.g. a field has a valid value.
  
  

• Validate that the data is consistent, e.g. a checksum is correct.

Our views do the first validation and nothing else. It is the only kind that is required for memory safety, so it is the only one we cannot make optional.

Semantical and consistency checks are optional, and we leave them to the user.

However, as you have seen before, some exceptions do exist: in our example, the AddrType field is validated during parsing, because it determines the length of the rest of the element. Without it we cannot safely create a view. 

Performance

Now that we have a parser, the question is, how much do we actually gain from zero-copy parsing? For this example, we can look at some of our micro-benchmarks.

Here we compare parsing a SCION packet into a view, versus parsing it into an owned model. The test is run over 10k randomly generated packets, and the total time is measured.

The tests were run on a developer machine (using an Intel(R) i7-14700K). So they should be taken with a big grain of salt. However this paints a clear picture of the relative difference between the two approaches.

The per-packet time is derived from the total time divided by the number of packets.

Parsing

That is a speedup of about 10.85x, at 20 ns per packet instead of 217 ns. 20 ns per packet is roughly 50 million packets per second on one core.

Encoding

This benchmark is a little cheeky, as it compares doing nothing (the view is already a slice) to encoding the model into a new buffer. However, if we had to modify a field on the hot-path, which a SCION router has to do, this time would be added to the 217 ns above, so it is still relevant.

Field Access

The final benchmark measures repeated field reads: iterating the path of a SCION packet and summing the ingress and egress interface of every hop field.

This is a case where the view is expected to lose. The model has already decoded every field into a native struct, so a read is a struct member access. The view holds only bytes, so reads cost additional offset arithmetic to locate the field and bit manipulation to extract it.

Comparison

The benchmarks above measure isolated operations. In the actual hot path, these operations need to be combined.

Here, we need to decode the packet, mutate at least one field, and put it back on the wire.

Combining the numbers for this path, we get: 

Translating to throughput at a near-worst-case 120-byte packet, on a single core:

This is the difference between a single core falling way short of 10Gbps, and it handling 40 Gbps.

However, the framing needs one honest qualification. These are parser ceilings – a real router needs to do much more than just this. With a fast parser, we just buy more headroom for the rest of the forwarding path, which is the point of this exercise.

Testing

One of the biggest costs when writing a protocol library like this is ensuring that it is correct and safe.
We won’t go too deep into testing strategies in this blog, since it deserves much more attention.

We choose cargo-fuzz and Proptest to validate our code. Miri is also a must use. 

Setting up fuzzing can be as easy as this: 

fuzz_target!(|data: &[u8]| {
    let mut data = data.to_vec(); // We need mutable access to the data

    match ScionRawPacketView::from_mut_slice(&mut data) {
        Ok((view, _rest)) => {
            exec_every_view_function(view);
        }
        Err(ViewConversionError::BufferTooSmall { .. }) 
        | Err(ViewConversionError::Other(_)) => {}
    }
});

Where we get random data from the fuzzer, try to parse it, and then execute all functions we have on the parsed data. One important note is to watch out to correctly use black_box to ensure function calls are not optimized away.

If you want to see more of how we did our testing, you can take a look at our repo.

Caveats and dangers

When we wrote this parser we accepted unsafe code in two places.

• When transmuting a slice into a view type.
  
  

• When reading and writing fields without bounds checks.

The first is easy to argue. Our type is a transparent wrapper around a slice, so converting back and forth is sound, and the claim is contained in a single type declaration.

The second is more dangerous and can easily lead to undefined behaviour. We can still do this responsibly because the slice length is checked once, when the view is created. This means every subsequent accessor is guaranteed to be in bounds. However, if this invariant is broken, e.g. by doing wrong calculations – or by forgetting to mark a length-relevant setter as unsafe – we have undefined behaviour. This is a class of mistakes that our tests and fuzzing will catch.

It is completely optional to use unsafe in the accessors, it removes additional out of bounds checks, which might add some overhead, but if you don't need to squeeze out the maximum performance, you can just use safe accessors.

If you go the unsafe route, testing is most important, and you should spend time to create a good testing strategy for your code.

Summary

With this post you have learned tricks on how to write a zero-copy parser in Rust, and how to use the type system to make it safe and ergonomic.

You now know how to use repr(transparent) to make a type that is a slice, and how to use the type system to handle ownership and mutability rather than writing out every combination yourself, with the caveats mentioned.

We have discussed the importance of being intentional about what you validate, and when, especially when writing high-performance networking code. The key is, do as little as possible to guarantee memory safety, and leave the rest to the user.

The post has shown you a possible architecture for a zero-copy parser, with three layers: a View for the hot path, a Model for the cold path, and a Layout as a single source of truth for offsets and lengths. This architecture can be used to avoid duplicated calculations and to provide a safe API for users to build packets.

Furthermore, you have read how bitshifting is used to read and write unaligned bitfields, and how to handle fields whose length is determined by an earlier field.

We have also discussed how certain mutations must be treated as unsafe, because they can invalidate the length check done on construction, and how to use the type system to make this clear.

Finally, you have seen that unsafe code is mostly optional for zero-copy parsing, but it can be used to remove bounds checks from accessors and squeeze out a little more performance.