2.3.2. Dynamic Byte Order#

In addition to traditional byte order types, caterpillar supports a dynamic byte order based on the current pack or unpack context.

Added in version 2.6.4: This feature is available in starting from version 2.6.4

Added in version 2.10.0: Added the Inherit byteorder type.

There are various use-cases that require a struct to handle both big-endian and little-endian. In order to reduce the amount of code for these structs, caterpillar introduces a special byte order type: DynByteOrder. It supports two different configuration levels:

  1. global: endianess is configured when calling pack() or unpack().

  2. struct-level: byte order is applied per struct (handed down to all fields)

  3. field-level: byte order is applied per field

Each of these configuration levels support various methods of selecting the target endian:

  • global configuration using an additional keyword argument in pack() or unpack().

  • context-key: configuration based on a value within the current context

  • custom function: endian is derived from a custom function

2.3.2.1. Example: struct-wide dynamic byte order#

Let’s consider the following struct definition. The dynamic endian configuration will be applied to all fields that haven’t got an endian already set.

 1@struct(order=Dynamic)
 2class Format:
 3    a: uint16           # litte endian or big endian is decided using
 4    b: uint32           # a global context variable
 5
 6obj = Format(a=0x1234, b=0x56789ABC)
 7
 8# pack the object using BigEndian
 9pack(obj, order=BigEndian)
10
11# now pack with little endian
12pack(obj, order=LittleEndian)

 1@struct(order=Dynamic)
 2class Format:
 3    a: uint16_t         # litte endian or big endian is decided using
 4    b: uint32_t         # a global context variable
 5
 6obj = Format(a=0x1234, b=0x56789ABC)
 7
 8# pack the object using BigEndian
 9pack(obj, order=BigEndian)
10
11# now pack with little endian
12pack(obj, order=LittleEndian)

Here we pass an additional global context variable named CTX_ORDER ("_order") to the packing and unpacking process. The dynamic endian will automatically infer the order based on this global variable.

2.3.2.2. Example: field-level dynamic byte order#

The same concept as shown above can be applied to single fields too. By default, the endian to use must be given as a global context variable as described before.

1@struct(order=LittleEndian)
2class Format:
3    a: uint16
4    b: Dynamic + uint32   # only this field will be affected
5
6# packing and unpacking is the same as in the previous example

1@struct(order=LittleEndian)
2class Format:
3    a: uint16_t
4    b: f[int, Dynamic + uint32]   # only this field will be affected
5
6# packing and unpacking is the same as in the previous example

2.3.2.3. Example: context key reference#

Sometimes format specifications use a special field indicating whether all following fields are using big or little endian. To implement this kind of endian selection, a so called context key can be specified, which can take one of the following forms:

  • direct reference: just a string reference

    # ...
spec: uint8
number: Dynamic(key="spec") + uint32
# ...

    # ...
spec: uint8_t
number: f[int, Dynamic(key="spec") + uint32]
# ...

  • context-lambda: a function that takes the current context as its first parameter and returns the target endian configuration value.

    # ...
spec: uint8
number: Dynamic(key=this.spec) + uint32
# ...

    # ...
spec: uint8
number: f[int, Dynamic(key=this.spec) + uint32]
# ...

The target endianess is decided based on the context value:

  • str will be applied directly as the format character

  • any object storing a ch string value

  • any other object is converted to a bool and the following mapping is applied:

As an example, consider the following definition:

 1@struct(order=BigEndian)
 2class Format:
 3    spec: uint8 = 0
 4    a: DynByteOrder(key=this.spec) + uint16
 5    # alternatively
 6    # a: Dynamic(this.spec) + uint16
 7    b: uint32
 8
 9# packing and unpacking does not require the extra endian value
10obj = Format(spec=0, a=0x1234, b=0x56789ABC)
11# 0 -> False, results in BigEndian
12data_be = pack(obj)
13
14# 1 -> True, results in LittleEndian
15obj.spec = 1
16data_le = pack(obj)

 1@struct(order=BigEndian)
 2class Format:
 3    spec: uint8_t = 0
 4    a: f[int, DynByteOrder(key=this.spec) + uint16]
 5    # alternatively
 6    # a: f[int, uint16, Dynamic(this.spec)]
 7    b: uint32_t
 8
 9# packing and unpacking does not require the extra endian value
10obj = Format(spec=0, a=0x1234, b=0x56789ABC)
11# 0 -> False, results in BigEndian
12data_be = pack(obj)
13
14# 1 -> True, results in LittleEndian
15obj.spec = 1
16data_le = pack(obj)

2.3.2.4. Example: byte order selected by an ancestor#

A nested model can select its byte order from an enclosing model by chaining the parent context path. In this example, Inner is nested through Middle, so parent.parent.byte_order resolves Outer.byte_order:

 1@struct(order=Dynamic(parent.parent.byte_order))
 2class Inner:
 3    value: uint32
 4
 5@struct
 6class Middle:
 7    inner: Inner
 8
 9@struct
10class Outer:
11    byte_order: uint8  # 0 selects BigEndian; 1 selects LittleEndian
12    middle: Middle

When unpacking, declare byte_order before middle so its value is available when Inner resolves its dynamic byte order.

2.3.2.5. Inheriting the enclosing struct’s byte order#

Added in version 2.10.0.

A struct that is reused in multiple places (e.g. a shared header or record type) sometimes needs to be decoded using whatever byte order the embedding struct declares, rather than a byte order fixed at its own definition site. The global Inherit enables exactly this:

 1@struct(order=Inherit)
 2class Inner:
 3    value: uint32
 4
 5@struct(order=BigEndian)
 6class Outer:
 7    inner: Inner  # value decoded using BigEndian
 8
 9@struct(order=LittleEndian)
10class Format:
11    inner: Inner  # value decoded using LittleEndian

 1@struct(order=Inherit)
 2class Inner:
 3    value: uint32_t
 4
 5@struct(order=BigEndian)
 6class Outer:
 7    inner: Inner  # value decoded using BigEndian
 8
 9@struct(order=LittleEndian)
10class Format:
11    inner: Inner  # value decoded using LittleEndian

A struct declared with order=Inherit resolves its byte order from the field that embeds it, at any nesting depth (including arrays and chains of Inherit structs). If there is no enclosing struct - e.g. the struct is used standalone or passed directly to pack() / unpack() - it falls back to the regular default byte order, exactly like a struct declared with order=None.

Wrappers can use an independently ordered metadata field without changing the order inherited by their payload. For example, this frame has a little-endian length prefix but a big-endian inner value:

1@struct(order=BigEndian)
2class Outer:
3    inner: Prefixed(LittleEndian + uint16, Inner)

inner.value is decoded using BigEndian; the explicit LittleEndian applies only to the prefix.

To override the byte order for a single embedding site instead of changing Inner itself, combine order=Inherit with the ByteOrder + Struct operator, mirroring the existing per-field ByteOrder + <atom> syntax:

1@struct
2class Format:
3    inner: LittleEndian + Inner  # value decoded using LittleEndian

This also lets sibling fields referencing the same Inherit-enabled struct type resolve to different byte orders independently of one another - including a plain (non-overridden) reference that keeps inheriting from the enclosing struct alongside two explicitly-pinned siblings:

1@struct(order=BigEndian)
2class Format:
3    little: LittleEndian + Inner  # always little
4    big: BigEndian + Inner        # always big
5    inherited: Inner              # follows Format's BigEndian

Changing Format’s own order (e.g. to LittleEndian) only affects the inherited field; little and big stay pinned to their explicit override since they no longer carry Inherit at that embedding site.

As with Dynamic, this can be composed with the extended annotation syntax, e.g. f[Inner, Inner, LittleEndian].

Note

This feature is unrelated to the struct class inheritance described in Struct (subclassing a @struct-decorated class via normal Python MRO). order=Inherit is about how a struct picks its byte order when it is embedded as a field in another struct, regardless of whether either struct participates in Python class inheritance at all.

2.3.2.6. Nesting depth and mixed byte orders#

order=Inherit resolves through arbitrarily deep nesting, and struct chains may freely mix explicit byte orders with Inherit at different levels. Each Inherit struct resolves against the nearest enclosing struct that has a concrete (non-``Inherit``) byte order - not necessarily its immediate parent and not necessarily the outermost struct:

 1@struct(order=Inherit)
 2class Inner:
 3    value: uint32
 4
 5@struct(order=LittleEndian)  # concrete order - breaks the chain here
 6class Middle:
 7    inner: Inner
 8
 9@struct(order=BigEndian)
10class Outer:
11    middle: Middle
12
13# Inner.value is decoded using Middle's LittleEndian, not Outer's
14# BigEndian, because Middle already provides
15# a concrete order for anything it embeds.

This also means a chain of several Inherit structs in a row correctly propagates the first concrete order found further up the chain, however deep:

 1@struct(order=Inherit)
 2class Inner:
 3    value: uint32
 4
 5@struct(order=Inherit)
 6class Middle:
 7    inner: Inner
 8
 9@struct(order=Inherit)
10class Outer:
11    middle: Middle
12
13@struct(order=BigEndian)
14class Format:
15    outer: Outer
16
17# Format.outer.middle.inner.value is decoded using BigEndian, propagated
18# through two intermediate Inherit levels.

The same resolution rule applies uniformly regardless of how a struct is nested, which has been verified for:

  • Arrays, including arrays nested inside other Inherit structs (items: Inner[2], or an array of a struct that itself contains an array of Inherit structs).

  • bitfield(), which supports order=Inherit the same way regular structs do, whether used as a scalar member, an array member, or nested multiple levels deep.

  • Prefixed, since it delegates directly to the wrapped struct using the same context rather than introducing its own field wrapper.

  • Unions structs, both as a container holding an Inherit-enabled member and as the Inherit-enabled struct itself.