So I’ve made some changes to PyDSDL which are currently on a branch. It is more or less functionally complete but the tests are not yet updated. I had to introduce certain modifications to the proposal I shared here earlier because a closer look revealed some flaws.
One minor change is that I renamed “capacity” into “extent” in order to avoid overloading the term “capacity” as applied to arrays. Other than that, it is the same: extent is defined as the amount of buffer memory that is required for storing a serialized representation of the type, and thus it also limits the maximum bit length of compatible versions. It is never smaller than the maximum serialized bit length of all involved fields combined.
The length field that prepends a non-final type is termed “delimiter header” – borrowed from DDS-XTYPES. Contrary to the early proposal, it is fixed at 32 bits (with the option to make it configurable in the future). One could deduce the size of the header from the size of the data type, but that means that we arrive at an evil corner case where the data type designer might decide to add a few more fields into an existing type, inadvertently breaking the 64 KiB barrier and thus breaking the binary compatibility. That issue alone is easy to manage at the data type level itself (just remove some fields), but what if there is a hierarchy of nested types, where the top type has a 16-bit prefix, and the bottom type adds extra fields that push the footprint of the TOP type above the 64 K barrier? There are two options:
- Always use a large prefix (like in DDS-XTYPES), with the possibility of manually constraining it when needed.
- Use a variable-length prefix, like in Protobuf or some representations of ASN.1.
Here, I chose the first option because it is simpler and more in-line with the rest of the DSDL specification. The extra overhead only affects those types that are not @final, which tend to be large and complex, so in relative terms, the negative effects are tolerable.
If a type contains a field of an appendable type, its author is effectively unable to make strong assumptions about the alignment of the fields that are positioned after the appendable composite because its length may be changed arbitrarily:
Inner.1.0 foo
uint64 aligned # Alignment unknown unless Inner.1.0 is @final
Yet, as we know from the experience, the byte alignment guarantee is vital for high-performance serialization and deserialization. Up to now, before the delimited serialization support, in most cases it was possible to enforce the required alignment by strategically placing the void padding fields where necessary. With the runtime variability that is brought in by the delimited serialization, the byte alignment in the presence of appendable types would be impossible to guarantee unless we require that the length of an appendable type is always a multiple of one byte (8 bits). Hence, as I mentioned in the OP post here, the delimiter header specifies the length in bytes, and the serialized representation of an appendable type is always padded out to one byte (with zeros). So that in the example above the serialization logic can always rely on faster byte-level operations rather than slow bit-level alternatives.
These considerations highlight a certain deficiency of the current design. Originally, DSDL was purely a bit-level format without any notion of byte or padding/alignment. This model was broken with the introduction of the rule that requires implicit array length prefixes and union tag fields to be of a standard size (8, 16, 32, or 64 bits). This change has somewhat altered the inner consistency of the logic, and now, the addition of the byte-padded appendable composites distorts the design even further.
First, our design guidelines mention that data types should be constructed such that their bit length sets are byte-aligned because it simplifies composition. Second, non-byte-aligned composites are difficult to serialize/deserialize using code generation, because if the composite is unaligned, it is impossible to delegate (de)serialization to a dedicated routine because the buffer is unaligned.
These considerations forced me to introduce the concept of data alignment into the standard such that composite-typed fields (whether final or appendable) are always aligned at one byte, and each member of their bit length set is always a multiple of one byte. This is achieved by dynamically padding the preceding space before a composite-typed field with up to 7 bits, and likewise padding the space after the last field of the nested composite such that its length is a multiple of 8. This matter was first discussed on this forum a long time ago in New DSDL directive for simple compile-time checks.
The alignment requirements for all type categories are as follows:
- Primitive: 1 bit (i.e., no alignment)
- Void: 1 bit (i.e., no alignment)
- Composite: 8 bit (one byte)
- Array: same as the element type.
Observe that by virtue of having the maximum alignment of 8 bit and the fact that the size of an implicit array prefix or union tag field is always a multiple of 8 bits, such implicit fields never break the alignment of the following data. Should we ever decide to introduce the alignment requirement that is greater than 8 bits (I doubt it’s ever going to make sense), we will need to alter the logic of deriving the size of such implicit fields to ensure that they are wide enough to not break the alignment of the following elements.
Suddenly this enables one trick for dynamically aligning data fields at one byte, as discussed in the above-linked thread:
bool[<=123] some_bits
@assert _offset_ % 8 != {0} # Not aligned here due to the element size of the array.
uavcan.primitive.Empty.1.0 _ # This field is zero bits long
@assert _offset_ % 8 == {0} # ...but the offset is now byte-aligned
Going back to the extent again, it essentially defines the degree to which a data type can expand in future revisions without breaking the compatibility. A greater limit obviously provides more opportunities for modifications; on the other hand, large values are costly for memory-constrained devices, so a sensible compromise is needed. While we could require the author of the data type to always define the extent for every data type definition explicitly, it harms the usability of the standard because generally the designer of a network service has a lot of other things to keep in mind. So unless the extent is set explicitly, it defaults to:
Where B is the bit length set as if the type was final. Essentially, we multiply the maximum size by 1.5 and pad the result to byte. This definition has an undesirable side effect of increasing the extent exponentially at each level of nesting (because each level introduces the 1.5 multiplication factor), but a more stable definition that would be trivial to compute does not appear to exist.
The bit length set of an appendable type is then the same as that of uint8[<=(extent+7)//8], except that the length prefix is always 32-bit. The definition of the bit length set for @final types is unchanged.
Suppose we have two types that are identical except that one of them is final and the other is appendable:
# Final.1.0
uint64 foo
@assert _offset_ == {64}
@final
# Appendable.1.0
uint64 foo
@assert _offset_ == {64}
The resulting offsets are then as follows:
Final.1.0 foo
@assert _offset_ == {64}
Appendable.1.0 foo
@assert _offset_ == {32, 40, 48, 56, 64, 72, 80, 88, 96}
There remains one case of inner inconsistency, though: the definition of extent does not include the delimiter header, whereas the bit length set does. This is because the delimiter header is not involved when the composite is serialized as the top-level object.