I have moved my rewrite of PyUAVCAN v1.0 from my personal fork back into the UAVCAN org: https://github.com/UAVCAN/pyuavcan/tree/uavcan-v1.0
At this point, the most critical parts of the library seem more or less finished and reasonably well tested (branch coverage ~94%; or 90% if you consider the auto-generated code which is not that well-tested), although they are still far from being API-stable:
- Python package generation from DSDL namespaces using Nunavut.
- Virtually zero-copy data serialization and deserialization.
- Extensible transport layer architecture that is sufficiently generic to support at least CAN 2.0, CAN FD, UDP, and IEEE 802.15.4, including heterogeneous redundant configurations.
- A concrete implementation of the CAN transport (2.0/FD universal).
One can generate code from DSDL, construct an object, serialize it, and send it over a simulated CAN bus (the reverse is also possible). Higher-level abstractions that would tie these operations together and provide a convenient application-level API are still missing. My next goal is to implement the so called “presentation layer” which provides the usual pub/sub and RPC abstractions; this seems straightforward and I don’t anticipate any issues here. Actually, I don’t anticipate any issues at all since the hardcore parts (namely, the serialization and the extensible transport layer arch) are already finished.
As I wrote earlier on this forum, I could really use help with development. At the moment, the most pressing issues are the lack of CAN media drivers and also the lack of adequate test rigging for them. I would like someone (perhaps @adolfogc and @ldg since they have expressed interest earlier, or whoever is available) to help move the project forward by contributing CAN media drivers for:
- SocketCAN
- Generic SLCAN adapters, with the loopback extensions as documented in the Zubax Babel datasheet
- Python-CAN library
Some code may be borrowed from the old library; however, much of it will have to be rewritten from scratch because the old library has certain fatal issues in its architecture and does not provide asynchronous API. Media implementations are abstracted through an interface pyuavcan.transport.can.media.Media which seems kinda stable so far since I managed to write a working CAN transport implementation and a mock driver without changing anything in it:
class Media:
ReceivedFramesHandler = Callable[[Iterable[TimestampedDataFrame]], None]
@property
def interface_name(self) -> str: ...
@property
def max_data_field_length(self) -> int: ...
@property
def number_of_acceptance_filters(self) -> int: ...
def set_received_frames_handler(self, handler: ReceivedFramesHandler) -> None: ...
async def configure_acceptance_filters(self, configuration: Sequence[FilterConfiguration]) -> None: ...
async def enable_automatic_retransmission(self) -> None: ...
async def send(self, frames: Iterable[DataFrame]) -> None: ...
async def close(self) -> None: ...
There is a stub for SocketCAN located under pyuavcan.transport.can.media.socketcan.SocketCAN which would be a good place to start with.
Besides CAN media drivers, there is also an outstanding issue of increasing the coverage of auto-generated code. As far as serialization and deserialization goes, the existing tests seem adequate: there are some basic carefully crafted manual serialization and deserialization tests and also a colossal randomized test which just generates tons of random objects and random serialized representations and throws that at the marshaling logic, ensuring that the random data survives roundtrip serialization/deserialization without distortions and that the serializer is able to handle random (insecure) inputs from the network correctly (it’s not allowed to throw exceptions). What’s missing is the error handling on the side of internal API (such as trying to initialize a union with more than one active field or trying to assign a string where numpy.array(dtype=uint8) is expected), and also perhaps it’s worth exploring whether it would be possible to generate an automatic test suite which would compare generated serialization/deserialization code against its source DSDL definition and prove its correctness. I am far from certain that it is necessary for PyUAVCAN since Python is not applicable in high-integrity applications (it’s not a high-integrity platform itself), but this could be a way to gather relevant experience which could be applied then in other implementations such as libcanard, libuavcan, or uavcan.rs (nudge nudge @kjetilkjeka) which are designed for high(er)-reliability systems. I will spare you the details, but the general idea is that the current code generator emits special hooks in the comments that could potentially allow a sufficiently intelligent piece of software to automatically trace every line of generated code back to its DSDL specification (such requirement tracing either in automatic or manual form is one of the cornerstones of safety-critical software development). Here is an example for uavcan.node.port.GetInfo.0.1.Request, the annotations follow the form BEGIN .+ (DE)?SERIALIZATION:
def _serialize_aligned_(self, _ser_: GetInfo_0_1.Request._SerializerTypeVar_) -> None:
assert _ser_.current_bit_length % 8 == 0, 'Serializer is not byte-aligned'
_orig_bit_length_ = _ser_.current_bit_length
# BEGIN COMPOSITE SERIALIZATION: uavcan.node.port.GetInfo.Request.0.1
# BEGIN STRUCTURE FIELD SERIALIZATION: void7
assert _ser_.current_bit_length % 8 == 0, 'self.: void7'
_ser_.skip_bits(7)
# END STRUCTURE FIELD SERIALIZATION: void7
# BEGIN STRUCTURE FIELD SERIALIZATION: uavcan.node.port.ID.1.0 port_id
# Object self.port_id is not always byte-aligned, serializing in-place.
# BEGIN COMPOSITE SERIALIZATION: uavcan.node.port.ID.1.0
# Tag field byte-aligned: False; values byte-aligned: True
# BEGIN UNION FIELD SERIALIZATION: uavcan.node.port.ServiceID.1.0 service_id
if self.port_id.service_id is not None:
_ser_.add_unaligned_unsigned(0, 1) # Tag 0
assert _ser_.current_bit_length % 8 == 0, 'self.port_id.service_id: uavcan.node.port.ServiceID.1.0'
self.port_id.service_id._serialize_aligned_(_ser_) # Delegating because this object is always byte-aligned.
# END UNION FIELD SERIALIZATION: uavcan.node.port.ServiceID.1.0 service_id
# BEGIN UNION FIELD SERIALIZATION: uavcan.node.port.SubjectID.1.0 subject_id
elif self.port_id.subject_id is not None:
_ser_.add_unaligned_unsigned(1, 1) # Tag 1
assert _ser_.current_bit_length % 8 == 0, 'self.port_id.subject_id: uavcan.node.port.SubjectID.1.0'
self.port_id.subject_id._serialize_aligned_(_ser_) # Delegating because this object is always byte-aligned.
# END UNION FIELD SERIALIZATION: uavcan.node.port.SubjectID.1.0 subject_id
else:
raise RuntimeError('Malformed union uavcan.node.port.ID.1.0')
# END COMPOSITE SERIALIZATION: uavcan.node.port.ID.1.0
# END STRUCTURE FIELD SERIALIZATION: uavcan.node.port.ID.1.0 port_id
# END COMPOSITE SERIALIZATION: uavcan.node.port.GetInfo.Request.0.1
assert 24 <= (_ser_.current_bit_length - _orig_bit_length_) <= 24, \
'Bad serialization of uavcan.node.port.GetInfo.Request.0.1'
@staticmethod
def _deserialize_aligned_(_des_: GetInfo_0_1.Request._DeserializerTypeVar_) -> GetInfo_0_1.Request:
assert _des_.consumed_bit_length % 8 == 0, 'Deserializer is not byte-aligned'
_bit_length_base_ = _des_.consumed_bit_length
_des_.require_remaining_bit_length(24)
# BEGIN COMPOSITE DESERIALIZATION: uavcan.node.port.GetInfo.Request.0.1
# BEGIN STRUCTURE FIELD DESERIALIZATION: void7
_des_.require_remaining_bit_length(7)
assert _des_.consumed_bit_length % 8 == 0, 'void7'
_des_.skip_bits(7)
# END STRUCTURE FIELD DESERIALIZATION: void7
# BEGIN STRUCTURE FIELD DESERIALIZATION: uavcan.node.port.ID.1.0 port_id
# The temporary _f0_ holds the value of the field "port_id"
_des_.require_remaining_bit_length(17)
# The object is not always byte-aligned, deserializing in-place.
# BEGIN COMPOSITE DESERIALIZATION: uavcan.node.port.ID.1.0
# Tag field byte-aligned: False; values byte-aligned: True
_tag1_ = _des_.fetch_unaligned_unsigned(1)
# BEGIN UNION FIELD DESERIALIZATION: uavcan.node.port.ServiceID.1.0 service_id
if _tag1_ == 0:
_des_.require_remaining_bit_length(16)
assert _des_.consumed_bit_length % 8 == 0, 'uavcan.node.port.ServiceID.1.0'
_uni2_ = uavcan.node.port.ServiceID_1_0._deserialize_aligned_(_des_)
_f0_ = uavcan.node.port.ID_1_0(service_id=_uni2_)
# END UNION FIELD DESERIALIZATION: uavcan.node.port.ServiceID.1.0 service_id
# BEGIN UNION FIELD DESERIALIZATION: uavcan.node.port.SubjectID.1.0 subject_id
elif _tag1_ == 1:
_des_.require_remaining_bit_length(16)
assert _des_.consumed_bit_length % 8 == 0, 'uavcan.node.port.SubjectID.1.0'
_uni3_ = uavcan.node.port.SubjectID_1_0._deserialize_aligned_(_des_)
_f0_ = uavcan.node.port.ID_1_0(subject_id=_uni3_)
# END UNION FIELD DESERIALIZATION: uavcan.node.port.SubjectID.1.0 subject_id
else:
raise _des_.FormatError(f'uavcan.node.port.ID.1.0: Union tag value {_tag1_} is invalid')
# END COMPOSITE DESERIALIZATION: uavcan.node.port.ID.1.0
# END STRUCTURE FIELD DESERIALIZATION: uavcan.node.port.ID.1.0 port_id
self = GetInfo_0_1.Request(port_id=_f0_)
# END COMPOSITE DESERIALIZATION: uavcan.node.port.GetInfo.Request.0.1
assert 24 <= (_des_.consumed_bit_length - _bit_length_base_) <= 24, \
'Bad deserialization of uavcan.node.port.GetInfo.Request.0.1'
assert isinstance(self, GetInfo_0_1.Request)
return self
Also, some information on DSDL code generation (particularly related to in-depth data layout analysis for generation of highly efficient (de)serialization code) is provided here: https://github.com/UAVCAN/pydsdl/pull/24
Perhaps it’s also worth looking into the mutability concerns in the context of (near) zero-copy serialization. Generally, zero-copy stacks require immutability of objects due to the issues with shared memory, which the following snippet (not pseudocode but an actual example that can be reproduced in REPL) demonstrates:
>>> import pyuavcan
>>> import uavcan.primitive.array # Assuming that the package is already generated
>>> b = bytearray([2, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> msg = pyuavcan.dsdl.try_deserialize(uavcan.primitive.array.Natural32_1_0, b)
>>> msg
uavcan.primitive.array.Natural32.1.0(value=[67305985, 134678021])
>>> msg.value[0] = 0xFFFFFFFF
>>> list(b) # Source array has been updated
[2, 255, 255, 255, 255, 5, 6, 7, 8, 9]
Since immutability cannot be statically enforced in Python, this could potentially cause someone to shoot themselves in the foot by mutating objects whose data is still being processed by the stack, leading to potentially hard-to-trace data corruption issues. I am not yet sure how important this property is, but any feedback and ideas would be appreciated.
I am referring to the serialization logic as “near zero-copy” because the current implementation actually does copy the data during deserialization if the underlying transport emits the data in several fragments rather than one contiguous chunk. The primitive serialization classes should be updated to support fragmented inputs to eliminate this special case, so this is a yet another aspect where contributions would be welcome.
Let me summarize. The library is a work-in-progress, but it is nearing a usable state fast and at this point the hardest parts seem to be done. It could use help with (ordered from most critical to least critical):
- Async CAN media drivers for SocketCAN, SLCAN, and Python-CAN.
- Support for fragmented serialization buffers.
- Automatic verification of generated serialization code by tracing it back to the source DSDL definition.
I would like PyUAVCAN to become a full-featured implementation of the stack, a first-class citizen of the UAVCAN ecosystem (which can’t be said about the old implementation due to its architectural shortcomings and limitations) which could be used for testing new protocol features and building sophisticated monitoring and testing tools on top of, such as Yukon.
There will be more opportunities for contribution once I’m done with the higher-level API which I am currently working on.
Paging @adolfogc @ldg @Zarkopafilis @scottdixon @erik.rainey
