# Data type extensibility and composition

**URL:** <https://forum.opencyphal.org/t/data-type-extensibility-and-composition/827>\
**Category:** Specification\
**Created:** [May 10, 2020, 12:31pm UTC](https://forum.opencyphal.org/t/data-type-extensibility-and-composition/827 "2020-05-10T12:31:11Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\
**Post date:** [May 10, 2020, 12:31pm UTC](https://forum.opencyphal.org/t/data-type-extensibility-and-composition/827/1 "2020-05-10T12:31:12Z")

</div>

The ability to mutate data type definitions as the systems they describe evolve is essential because here on Earth there is no such thing as a stable design unless it is obsolete. Being designed for high-integrity deeply embedded systems where a software update may be costly to execute, UAVCAN has to provide developers with means of modifying data type definitions without breaking wire compatibility with existing deployments.

The current v1.0-alpha specification is a huge step forward compared to the original v0 because it supports renaming, extension, and truncation of data type definitions without breaking backward compatibility (as explained with examples in the [The Cyphal Guide](https://forum.opencyphal.org/t/uavcan-v1-crash-course/778) and discussed in [UAVCAN v1.0 and ArduPilot](https://forum.opencyphal.org/t/uavcan-v1-0-and-ardupilot/671)). Extension and truncation combined enable one to replace existing fields with new ones, albeit at the cost of increased footprint (the space occupied by the removed field has to be padded out). Overall, the available data type modification tools are similar to those offered by conventional non-real-time protocols such as Cap’n’Proto or CORBA CDR, although the details differ due to the hard real-time constraints of DSDL.

While working on my write-up on the recommended data type design practices for our drone-minded colleagues I realized that the above-described facilities may not be sufficient. There is a problem that follows from our overly direct approach to serialization.

The offset of a field upon serialization is dependent on the set of the preceding fields, which in turn is determined by the preceding composite fields. Modification of any of the composites may alter the placement of the following fields, possibly originating from other composites, which introduces undesirable coupling across independent data types, rendering the involved data types non-extensible excepting the topmost one. Example:

```python
# X.1.0
bool a

# X.1.1
bool a
bool d

# Y.1.0
bool b
bool c

# Z.1.0
X.1.0 x
Y.1.0 y

# Z.1.1
X.1.1 x # <-- problem
Y.1.0 y

```

The above-defined versions of `Z` serialize as:

| Type | Field pattern |
| --- | --- |
| `Z.1.0` | `abc` |
| `Z.1.1` | `adbc` |

Despite the modification to `X.1.1` being backward-compatible by virtue of the implicit extension/truncation rules, the layout of the top-level container `Z` is broken.

The coupling can be eliminated by serializing composites orthogonally:

 ![image](https://forum.opencyphal.org/uploads/default/original/1X/20dda44e0b1429f51b4d047c031bf8eb31a372fa.png)

_Diagram: top half – v1.0, bottom half – v1.1; the rightmost column shows orthogonal serialization; the case for more than two nested composites would be difficult to illustrate on a flat medium._

Such orthogonal axes can be mapped onto the final linear representation by augmenting it with the length information per axis. In the following example for v1.1, the length augmentation information is shown as `#`:

| Type | Pattern |
| --- | --- |
| `Z.1.0` | `#a#bc` |
| `Z.1.1` | `#ad#bc` |

Such augmentation allows one to reconstruct orthogonal dimensions without knowledge of the specific data type definition, thus eliminating the undesirable coupling.

The idea is trivial and it can be found in other serialization protocols, although sometimes the length-based augmentation is replaced with direct addressing of each axis relative to the origin of the serialization buffer – the approaches are functionally equivalent. Its disadvantage is the length augmentation overhead introduced per composite-typed field. The practical significance of the overhead is the highest for data types whose footprint is low. At the same time, such data types tend to be simple, and therefore the probability of such a data type requiring modification is low, which reduces the value of orthogonality. An obvious way to mitigate the length overhead is to provide the data type designer with an option to suppress orthogonal serialization at the data type definition time. In the following example, type Y is serialized non-orthogonally (inline):

| Type | Pattern |
| --- | --- |
| `Z.1.0` | `#abc` |
| `Z.1.1` | `#adbc` |

Similarities can be found with some OOP languages that distinguish between value types and reference types.

The length augmentation is a field that defines the size of the following segment (axis) in the serialization buffer. Its range should support the largest sensible object size and the type should be a standard-size integer, which leaves `uint32` and `uint64`. A serialized representation that exceeds 4 GiB is unlikely to be ever encountered in practice, but for the sake of extensibility, it is rational to reserve the upper half of the range \left[2^{31}, 2^{32}-1\right] where the most significant bit is set to possibly support 64-bit length fields in the future (similar to the UTF-8 multi-byte encoding format).

Then, a composite-typed field like `X.1.1 x` is a DSDL equivalent of:

```python
uint8[<2**32] axis

```

That is unless the type of the field is defined as non-extensible. Borrowing liberally from the widely adopted terminology in the OOP world, the non-orthogonality of a type can be declared with the help of a directive like `@final`. Example:

```python
# Y.1.0
@final
bool b
bool c

```

Here, the practical effect of the declaration is not that the type is non-extensible or immutable, but rather that the DSDL language will be unable to guarantee backward compatibility automatically, requiring the developer to take the necessary precautions manually. The existing standard types available under `uavcan.primitive` and `uavcan.si` will require such annotation.

Besides the above-listed concerns, the orthogonal serialization is vital for implementing the much-discussed structured data type design approach where the common functionality that is relevant for different specialized types is extracted into a base supertype. As a made-up example, consider a pair of types that model the status of a battery management system and an on-board electric turbogenerator. Imagine that the fields describing the current output power (watt) and the available energy reserves (joule) are extracted into a base supertype. Once such extraction is done, the base supertype (which may be highly complex) becomes impossible to extend without breaking the wire compatibility of every subtype, despite the pledge to support extensibility through implicit extension and truncation. The orthogonal serialization solves that problem.

I am currently evaluating how critical this addition is considering the ongoing efforts to define an industry-specific messaging standard based on UAVCAN. The effort required to implement this is low.

---

_[View the full topic](https://forum.opencyphal.org/t/data-type-extensibility-and-composition/827)._
