Operators#

TODO: describe all supported operators

Caterpillar supports various operators for Sequence, Struct, Field, and custom struct implementations [1]. This section provides an overview of the operators that can be used with structs, including special context operations.

Struct operators#

All functions described below return a new Field instance if not already called on a field object.

struct.__getitem__(self, length)#

This function acts as an array definition and supports different kinds of length types.

Static (int)

A constant value is used to mark static arrays with a fixed length. Float values are not accepted.

>>> array = uint8[100]

Dynamic (_ContextLambda)

A callable function may be used to return the length as an integer. If you choose not to use a context value, the provided callable function must take the Context instance as its first argument.

>>> array = uint8[this.length] # where this.length would store 100

Greedy (_GreedyType)

Used when the amount of elements to be unpacked is unknown. The struct tries to unpack an object of the given type and stops either on EOF or if an exception is thrown.

>>> array = uint8[...]

Prefixed (_PrefixedType, slice)

In cases where there are length-prefixed structures and you don’t want to store the length in an extra variable, you can use Prefixed to improve packing data.

>>> prefixed_array = CString[uint8::]

Note

This function is applicable to all classes annotated with @struct or @bitfield.

struct.__rshift__(self, options)#

Invoked to apply a set of options to a Field. This function changes the behavior of a struct. It unpacks a value using the field’s underlying struct and passes it directly into the given options to retrieve the final struct.

The described behavior has a limitation: if the underlying struct is not a _ContextLambda, there is no possibility of packing an object back to binary data because the initial value is not known.

>>> field = Field(this.foo) >> {
...     "bar": uint16,
...     "baz": uint32,
... }

You can also include a default option using DEFAULT_OPTION from within the fields submodule.

>>> field = Field(this.foo) >> {
...     "bar": uint16,
...     DEFAULT_OPTION: ctx._value
... }

The previously parsed value is accessible from within the current context, not the current object context.

struct.__matmul__(self, offset)#

Another special operator (@) is used to re-position the current field to a specified offset position, where the offset can be static or dynamic.

When unpacking objects from a stream, the reader will temporarily jump to the given offset. Using the F_KEEP_POSITION option, the reader will continue parsing at the resulting position.

Packing is tricky as we don’t want to lose any data when jumping to an offset position. Internally, a dictionary with offset-data mappings is created and will be applied when all normal fields have been written to the stream. Alternatively, there is an option to firstly write everything into a temporary file and copy the final result into the given stream.

Static (int)

Integer values are accepted to be static, and therefore the default behaviour is applied.

>>> field = uint8 @ 0x1234

Dynamic (_ContextLambda)

Callables and context lambdas are accepted as well.

>>> field = uint8 @ this.offset

Caution

This operator is not applicable on raw struct classes. Therefore, the class has to be turned into a field first. A shortcut can help you with that.

>>> from caterpillar.shortcuts import F
>>> field = F(Format) @ 0x1233

struct.__floordiv__(self, condition)#

Experimental. Invoked to link the current field or struct with a certain condition, which can be either a static boolean value or a context lambda.

>>> field = uint8 // (lenof(this.array) > 0)

Developer’s note

This feature is proposed to be replaced by an if-else structure chain in the future.

struct.__rsub__(self, bits)#

Invoked to specify the amount of bits this field uses. (Only applicable in classes decorated with @bitfield)

>>> field = 3 - uint8   # 3 of 8 bits are used

struct.__and__(self, other)#

Invoked to create a chain of two structs. It is important that only the last element doesn’t need to return a bytes object after unpacking.

>>> chain = ZLibCompressed(...) & Format

Important

The returned object is not a Field instance, but a Chain instance to support more than two elements.

Field specific operations#

The following methods are specifically designed for fields and will only affect those. Even though these operations are also supported on sequence objects, only the returning Field object will be affected.

field.__or__(self, flag)#

field.__ior__(self, flag)#

With the logical OR operation, a so-called Flag can be set. Some flags are defined globally, and their meaning is described in Options.

>>> field = uint8 | F_KEEP_POSITION

field.__xor__(self, flag)#

field.__ixor__(self, flag)#

The logical XOR operation is designed to remove a flag/option from the specified field.

>>> field = uint8 ^ F_KEEP_POSITION

Sequence specific operators#

The following functions have been implemented to provide a user-friendly interface for extending Sequence objects. All subclasses inherit this functionality as well.

sequence.__add__(self, sequence)#

sequence.__iadd__(self, sequence)#

Called to import all fields from the given sequence into this instance. Note that the fields will be added to the end of the current field list.

>>> seq = Sequence({"a": uint8}) + Sequence({"b": uint8})

sequence.__sub__(self, sequence)#

sequence.__isub__(self, sequence)#

Invoked to remove all fields in this sequence that are also stored in the given sequence. This operation does not alter the used model but only affects the internal model representation.

>>> seq = Sequence({"a": uint8, "b": uint8}) - Sequence({"b": uint8})

Context specific operations#

The Context implements attribute-like access on top of a dictionary, where the requested path is resolved recursively. For instance, the path "foo.bar.baz" will be split into three parts and then getattr is called until the final element has been reached or an error occurs.

To enhance the facilities of a Context instance, there are special classes with even more special operations.

Context path#

The context path takes a special place, as it can provide lazy execution of almost all operators on definition. Its attribute-access model results in a new ContextPath instance.

>>> path = ContextPath("foo").bar.baz
Path('foo.bar.baz')

To enable list-like access and function calls, there are special methods:

path.__call__(self, **kwargs)#

Calling the path without the context instance results in a special state. The path stores the keyword arguments given in the call and executes them after retrieving the value from the context.

>>> path = this.foo.bar(x=19)
Path('_obj.foo.bar', call(x=10))

path.__getitem__(self, key)#

This method has the same effect on the path, as it stores the key argument and executes the __getitem__() method on the retrieved value afterward.

>>> path = this.foo.bar(x=19)[10]
Path('_obj.foo.bar', call(x=19), getitem(10))