Stockholm Summit recap

Development & Maintenance
1

This post is a brief recap of the major design decisions that were made during the Stockholm Summit (Oct 2nd-3rd, 2018) and during the preceding discussions on Github. The objective of the summit was to discuss the main goals related to the transition from the current pre-release version of the specification (termed “UAVCAN v0”) to the first major stable release (termed “UAVCAN v1.0”). The experience accumulated in the three years since the release of v0 showed that certain major breaking changes have to be introduced in order to eliminate early design mistakes (which are too serious to ignore).

For future reference, here is the list of the most relevant conversations that contain virtually the entirety of the context:

(Side note: technical discussions used to be spread out thinly over the Google Group, several Github repos, and the Gitter chat room. From now on, high-level technical discussions should be conducted only on this forum for the sake of ease of tracking and searchability.)

Here is an abridged list of the changes in v1 compared to v0, loosely ordered from most significant to least significant:

  • The concept of Data Type ID (and its special cases: Message Type ID and Service Type ID) will be removed from the specification. Identification and separation of data streams will be managed by two new concepts instead: Subject ID (for publish/subscribe links) and Service ID (for RPC). This change alone will have a profound effect on UAVCAN applications and its usage patterns. It will be reviewed later in this post in more detail.
  • Added support for CAN FD alongside CAN 2.0B.
  • Semantic versioning for data type definitions. In UAVCAN v0, there was no clear method of updating data type definitions. Once a data type was released, its definition had to be frozen forever in order to not break the compatibility with other nodes. Now, there is a well-defined process of data type advancement and maintenance. The full context is available in this megathread on Github: https://github.com/UAVCAN/dsdl/issues/35
  • The concept of Data Type Signature (and the related concept of DSDL Signature) will be removed from the specification. The compatibility guarantees will be enforced statically using optional static analysis tooling for DSDL definitions.
  • Data integrity checking will be decoupled from data type compatibility checking: multi-frame transfer CRC will no longer be pre-seeded with the data type signature (this is a consequence of the above).
  • New CAN ID format. Brand new.
  • Tail Array Optimization (TAO) will be removed from the specification. The new approach is to apply same serialization principles uniformly, regardless of the context.
  • Some rewording and restructuring of the specification to make the protocol compatible with other transports (not only CAN). Seeing as the protocol is now focused neither on UAV nor on CAN, the name “UAVCAN” is a misnomer. Are there any interpretations that do not involve Unmanned Aerial Vehicles and Controller Area Networks?

A major current discussion topic is whether there is a need to enforce code compatibility under the same major DSDL version. The discussion can be found here: https://github.com/UAVCAN/specification/issues/9. Basically, there are two options on the table:

  • Require that the code generated from any DSDL definition be API-compatible with the code generated from any other DSDL definition under the same major version. Meaning that an application can switch from v1.2 to v1.3 keeping its source code unchanged.
  • Make DSDL completely unaware of generated code. That means that an application might end up being broken if a DSDL definition v1.2 were to be replaced with v1.3. The implication is that it may no longer be possible to switch between different minor DSDL versions without releasing a new major release of the application. (my personal take on the matter is that the question boils down to whether it is acceptable to version applications and the protocol differently)

Now I would like to briefly recap the major effects of the transition from Data Type ID to Subject ID, mentioned earlier. This is not intended to be an in-depth explanation (that will be published in the spec draft in a few days hopefully); rather, this is a hands-on overview.

The decision was guided by our perception that the protocol v0 was generally well-designed from the physical layer up to the transport layer. The upper layers consisting of DSDL were largely a collection of poorly defined overspecified schemas that forced one to design their applications in a very particular way. Any attempt to step out of the bounds outlined by the protocol specification would make its use challenging if not impossible. This was considered (at least by me personally) a major problem, the most severe issue that had to be resolved lest the protocol would never find its way outside of the domain of small drones.

Now, here is the example. Consider that we have to integrate two motor controllers that accept their inputs using the message uavcan.equipment.esc.RawCommand and report their status information using the message uavcan.equipment.esc.Status. The data flow diagram for both of them would look as follows:

 uavcan.equipment.esc.RawCommand  +--------------------+ uavcan.equipment.esc.Status
--------------------------------->|        ESC         |----------------------------->
 data type ID: 1030               |    (esc_index)     |  data type ID: 1034
                                  +--------------------+

Each ESC is identified by its ESC index. A command message contains an array of setpoints; each ESC extracts the setpoint intended for it using the pre-configured index parameter. A status message identifies the source ESC by its index, while also carrying the node ID information which in this specific setting is redundant and not used by the application.

This (current) approach has several major issues, the most important of them being that the protocol merges data flows distributed across different nodes into the same logical channel referenced by the Data Type ID. The Data Type ID is used at the same time to also convey the data type information, thus fusing together two completely different levels of abstraction and creating severe usability issues. The resulting unified data flow is then to be parsed and split by each node separately, which creates scalability and latency issues. The problem becomes even more severe in certain other application scenarios. For example, the standard message uavcan.equipment.gnss.Fix2 (notice the versioning kludge added out of necessity) requires that there be no more than one GNSS receiver per node. A benign-looking definition uavcan.equipment.device.Temperature fuses together completely different data streams: an outside air temperature would end up on the same logical data channel with a CPU temperature, requiring each subscriber to carefully sift through the message stream to pick up what’s needed and to ignore the rest. The go-to approach to that problem was to create more narrowly-specialized types to avoid unnecessary data flow unification.

As I said above, that was identified to be the worst problem with the protocol v0. The solution that we are introducing in v1 is to add an additional abstraction layer which untangles message types from physical processes they describe. With the new framework, the ESC data flow problem could be approached as follows:

 uavcan.si.angular_velocity.Raw  +--------------------+ uavcan.si.angular_velocity.Timestamped
-------------------------------->|        ESC 0       |-------------------------------->
 subject ID: 100                 |                    | subject ID: 201
                                 |                    |
                                 |                    | uavcan.si.voltage.Timestamped
                                 |                    |-------------------------------->
                                 |                    | subject ID: 301
                                 |                    |
                                 |                    | uavcan.si.current.Timestamped
                                 |                    |-------------------------------->
                                 |                    | subject ID: 401
                                 |                    |
                                 |                    | uavcan.si.temperature.Raw
                                 |                    |-------------------------------->
                                 |                    | subject ID: 501
                                 +--------------------+

 uavcan.si.angular_velocity.Raw  +--------------------+ uavcan.si.angular_velocity.Timestamped
-------------------------------->|        ESC 1       |-------------------------------->
 subject ID: 102                 |                    | subject ID: 203
                                 |                    |
                                 |                    | uavcan.si.voltage.Timestamped
                                 |                    |-------------------------------->
                                 |                    | subject ID: 303
                                 |                    |
                                 |                    | uavcan.si.current.Timestamped
                                 |                    |-------------------------------->
                                 |                    | subject ID: 403
                                 |                    |
                                 |                    | uavcan.si.temperature.Raw
                                 |                    |-------------------------------->
                                 |                    | subject ID: 503
                                 +--------------------+

The new framework avoids the problem of data flow unification completely, thus obviating the need for esc_index and related instance identifiers (such as sensor_id and so on). Each ESC has its own separate Subject ID for each incoming and outcoming data stream. The subject ID values are not standardized, and each application is free to choose them arbitrarily. Cross-vendor and cross-application compatibility for COTS hardware will be achieved through manual or automatic pre-configuration of Subject Identifiers.

The new specification will likely introduce an optional recommendation (convention) to assign even values of Subject ID for data streams carrying commands, and odd values of Subject ID for data streams carrying state estimates (such as sensor measurements and other actionable data).

The implications of the change are major, and it is recognized that the transition to v1 will make the life of early adopters hard. Further, it is also understood that the new protocol does not have the same level of support for common applications, and truly plug-and-play nodes will be harder to implement, at least until there are additional higher-level provisions for that (could be added in v1.1 perhaps?). The move also reflects our commitment to provide better support for vendor-specific and application-specific closed ecosystems.

I hope the above was a satisfactory description of the upcoming changes and I also hope that I didn’t forget anything important. My fine colleagues @kjetilkjeka and @scottdixon might add corrections or amendments later in this thread.

2

Well, crap. I just spend the last hour editing my own summary instead of reading yours. I’ll just post my summary as well which has far less detail but does touch on some other topics discussed at the summit:

On October 2nd and 3rd three of the UAVCAN maintainers; Pavel Kirienko (BDFL), Kjetil Kjeka (rustifarian), and Scott Dixon (the FD guy), held a summit in Stockholm Sweden to discuss version 1 of the UAVCAN specification. At the meeting’s conclusion the attendees agreed that a first draft of UAVCAN v1 should be ready by the end of October. An announcement will be made on this forum when the draft is ready for public comment and feedback. After the draft is released work will begin to adapt the four reference implementations; pyuavcan, uavcan.rs, libcanard, and libuavcan. As the maintainers gain insight into the impact the new specification has on each implementation, and as changes are made in response to public comments, estimated milestone dates will be released for each project. Pavel Kirienko (BDFL) will setup and run a weekly developer call Mondays at 6pm (UTC) to coordinate these efforts.

The GPL licensing encumberment found in the gui_tool project was discussed. The consensus was to scrap the current pyqt implementation in favor of a UI technology that offered better licensing terms. It was suggested that a python implementation that exposed its GUI via a web server (e.g. In the same manner Jupyter notebooks are served locally) should be designed. This would allow the use of a more modern and flexible JavaScript GUI toolkit but would still leverage pyuavcan as its core implementation. This approach would also easily allow UI remoting from containers or virts to host platforms that do not support CAN-FD (e.g. OSX). The major concerns were a lack of experience in developing JavaScript UIs by the current UAVCAN contributors and a lack of time given the large amount of work the v1 port will require for the core implementations. This lead to a discussion about soliciting more active contributions from the community to accelerate the v1 port and to help implement the new GUI tool. Scott Dixon agreed to ask for more support from his organization but was not able to make a hard commitment on this point. Pavel Kirienko, of Zubax robotics, offered that his company may be able to offer limited, additional help with the pyuavcan port.

Kjetil re-raised his proposal to require “code compatibility” between all minor revisions within a given major version of the protocol (see the github thread here). This was discussed at length but it was agreed that the group would not come to consensus on this issue at the summit.

The majority of the two days was spent discussing Data Types in general and the standard UAVCAN type definitions specifically. The following points are the key decisions emerging from this lengthy discussion:

  • All types under the “protocol” namespace will remain for v1 and will remain, largely, unchanged.
  • All vehicle-specific types will initially be omitted from the v1 specification. As vendors manufacturer v1 compliant hardware Subjects will be added to support their product class.
    • It was agreed that the v0 standard types were deficient because they lacked the refinement that comes from use. Because of this v1 should only add standard Subjects if and when real products require support for them.
    • This decision requires a dedicated section in the specification detailing the manufacturer engagement process.
  • The standard will no longer use the term “Data Type ID” but will rename this concept to “Subject”. Likewise, Service Type ID will be referred to as simply Service ID.
  • V1 will adopt a model of System State, Physical Processes, Physical Quantities, Data Types, and Bit streams.
    • This model will obviate the need for sensor and actuator indices in messages.
    • All physical quantities will be float32
  • Because v1 will sacrifice some bus efficiency to present a more generic “publish and subscribe” semantic it will become even more important that v1 implementations improve support for custom Subjects and Services. Scott and Kjetil both stressed the importance of custom DSDL to the success of the v1 protocol.

Finally, it was resolved that the Swiss make the best chocolates and that Amazon Web Service’s† Stockholm office was a very gracious host. We would like to thank them again for the coffee and the whiteboards.

†Please note that neither Amazon Web Services, its parent companies, nor its subsidiaries sponsor, directly support, nor make any claims for this open-source software project. We were guests as a benefit of my (Scott Dixon) employment and as part of a project that I contribute to for my own professional curiosity and sense of fair play.

1 Like
3

Looks like Pavel and Scott got it all. I do not have any additions but will add to the custom DSDL importance point.

The new super generic definitions are hopefully going to be very nice to use for Plug and Play applications, when you just want things to work smooth out of the box. For applications with specialized demands: Where perhaps bus efficiency is of the essence, where data need to be synchronized, etc. They’re not going to be ideal.

Using Uavcan with custom DSDL needs to be a first class use case. It also means that manufacturers should have the opportunity to create custom types for their equipment (also supporting the generic types) and let users onfigure the generic or the custom types on a subject. This means that we need to start thinking about things such as namespace reservation in the near future.

1 Like
4
5
6