UAVCAN Drone Application Layer SIG guidelines

Consortium UDRAL: Cyphal Drone Application Layer
1

This document sets the guidelines for the Special Interest Group (SIG) dedicated to the development and maintenance of the UAVCAN Drone Application Layer (UDRAL). The rules set out in this post are derived from our experience managing a prior standard draft — DS-015 (now defunct) — with appropriate precautions taken to avoid repeating past mistakes.

Scope

The UAVCAN v1.0 Specification does not standardize the application layer (L7 per the OSI model) except for certain basic, domain-agnostic utility functions such as firmware update or configuration management. Likewise, it does not define the physical layer (L1 per the OSI model), limiting its scope to approximately L2 through L6, inclusive. This is done to widen the applicability limits of the core protocol without putting undue constraints on practical implementations.

This SIG will develop, and then maintain for an indefinite period, an open and royalty-free standard that defines an application-layer protocol for use in avionic systems of autonomous aerial vehicles. Due to shared underlying principles and common concepts, the resulting standard may be applicable (perhaps to a limited extent) in other types of systems such as robotics and manned aviation, which is to be encouraged as long as such widening of the scope is not in contradiction with the main objective of the SIG.

The SIG will not attempt to define the entirety of the application layer, leaving the possibility for vendors and implementers to introduce additional capabilities and interfaces beside this standard as necessary.

This SIG will not engage in the development of the physical layer connectivity standard. A separate SIG will be organized for that purpose instead.

This SIG will not engage in the development or modification of the UAVCAN v1.0 standard itself, although it is welcome to bring proposals to the protocol maintainers if desired.

The scope is summarized on the following diagram:

image
image1158×355 42.2 KB

The SIG will take the design recommendations provided in the official UAVCAN Guide into consideration.

The main work product of the SIG is a set of DSDL definitions published in the official public regulated data types repository along with enclosed technical documentation and design rationale.

Organization

The term “open”, as used in “open standard” above, denotes that the final standard is to be accessible to the general public under a permissive license on a royalty-free basis. Despite the final product being available to everyone free of charge, participation in its development process will be limited only to the members of the UAVCAN Consortium. This move is intended to reduce the workload on the contributors, eliminate random opinions from the conversation, and incentivize paid membership in the Consortium which helps us sustain the project. Observe that open-source projects and non-profits are entitled to free membership.

The discussion of the standard will be conducted chiefly in the closed category of this forum. The forum will also be used to facilitate voting (more on this later). The SIG can convene on regular or ad-hoc voice calls at its discretion, provided that active participation in the call is limited to the SIG participants only (others may be allowed to listen). However, key decisions and agreements reached using any means of communication other than this forum will be documented on the latter to ensure that the entire group has full access to the relevant information regardless of one’s ability to participate in voice conversations or any other off-forum communications.

The existing Drone SIG subcategory will be migrated into the closed section of the forum to accommodate the above.

The number of representatives that a UAVCAN Consortium member is allowed to bring into the conversation depends on its membership tier:

  • Sponsored or Expert members: up to 5 representatives.
  • Regular members: up to 2 representatives.

Representatives can be registered/unregistered by sending a private message to the forum admins.

Any active member of the Consortium can become a participant of this SIG by merely informing the admins. The admins may remove a Consortium member from the SIG in the event of its representatives being unreachable for a considerable amount of time in order to prevent such members from delaying the group’s progress. An up-to-date list of the current SIG participants will be maintained in a dedicated post on this forum for bookkeeping purposes.

Major technical questions and technical disagreements between the SIG participants will be resolved by voting. The voting process will be managed using the forum’s built-in capability.

Each participant of the SIG is allowed to cast up to 1 (one) vote (this implies that representatives should internally select one delegate who is to cast the vote for the represented SIG participant).

A Consortium member who is not a participant of the SIG may still cast a vote, which would automatically register it as a SIG participant.

Unless explicitly specified otherwise, each voting session will be open for 14 calendar days or until each current SIG member has cast its vote or expressly desisted from doing so (whichever happens first). A voting session is considered successful if at least 75% of the active SIG participants have either registered their votes or expressly desisted from doing so.

Besides the SIG participants, each of the two core developers of the protocol (@pavel.kirienko and @scottdixon) are allocated one vote each.

2

Workflow

Main design decisions and constraints will be documented in a single forum thread. The same thread will be used for voting, shall the group fail to reach consensus on any specific issue.

Technical proposals and counter-proposals are to be brought up in dedicated threads in the same forum category. For example, suppose that Participant A proposed certain Design X. A new topic is started which elaborates Design X in fine detail, provides rationale and known issues. Design X is then submitted for evaluation by other members of the SIG. If the proposal is accepted (perhaps after some technical debate and modifications until the group reached consensus), the iteration is finished. Otherwise, Participant B might bring counter-proposal — Design Y, in a dedicated thread. The group would then discuss the options and perhaps implement amendments. Shall the group reach consensus, the iteration is finished. If no consensus could be reached, the SIG would choose X vs. Y by simple majority voting (see quorum & time limits defined above).

Voting is intended to avoid delaying the progress of the SIG shall it be impossible to reach consensus on any specific matter.

3

as I’ve mentioned previously, I’d prefer our discussions to be public. I think it will lead to a better result. If you really can’t do that, then making it writeable only by members and readable by everyone would be 2nd best option.

4

I don’t think these options are on the table, unfortunately. We will, however, accommodate #2 partially by publishing the result of our work at the public GitHub repository.

Do you have any feedback regarding the workflow?

5

would it be published during the development process, or only at the end?
I don’t want to be in the position where I have to keep development secret from other members of the ArduPilot developer community, or from our partners and from our weekly development call attendees.

it sounds like the normal operation of mailing lists and forums in all sorts of technical communities, but with secrecy. It is the secrecy that doesn’t sit well with me, as it will result in a worse outcome.
I think the most useful thing I could do to start is put together a suggested message set for the core messages, like I suggested in my CANDevices proposal. I could put that in a git repo and the other members here could discuss and poke holes in it on this forum. I think we could make a lot of progress very quickly that way.
The big thing that I really want to know is how we will handle the type safety issue that I raised previously. I proposed using a field in the CANDevices msgs for that, but what I’d much prefer is to reserve a range of subject IDs for these CANDevices, so given a subject ID we instantly know what type of msg it is. That would make configuration vastly simpler and much more robust.

6

This is probably the biggest immediate problem I have with embracing UAVCANv1 for the wider ecosystem right now.

I understand the flexibility in untangling the combined data type, subject, and ID (/priority) we had in UAVCANv0, but I struggle with the jump to it being so free-form like in differential_pressure_sensor demo or subject names coming from comments in service files (eg air_data_computer).

Can we find something in between those extremes combining the type safety and simplicity we enjoy with current UAVCAN with respecting the UAVCANv1 interface guidelines (no fixed port IDs, etc)? It seems to me for the majority of the usage we actually need we should be discussing things at the level of a simple directory of subjects (mapping to data types).

Simple map of subject name to type

  • port id would come from a known register naming convention using the subject name
  • resolves type safety issues
  • port id could even have acceptable defaults in many cases (I know this might be contentious)
  • code generators could directly consume this and produce libuavcan style Publisher and Subscriber helpers that have things like the port id from register part handled automatically
  • this could be extending the existing services idea, but digestible by tooling, and probably the goal of each service being much simpler/coherent at least at the start
7

My expectation is that the repository will be updated incrementally as the group progresses, so outside observers will be able to see roughly the current WIP at all times.

This is fine in general but I think it would save much effort to you and other participants if we first agreed on what exactly we are building before actually starting the work. This requires that the technical requirements are put forth and agreed upon. Can you please briefly describe what are you trying to achieve before publishing any definitions? Otherwise, we are risking facing two negative consequences:

  • Psychological anchoring might make the group reluctant to accept competitive design goals once an implementation proposal is already in place.

  • The discussion may become less organized as core goals will be discussed alongside minute technicalities of the implementation.

Case in point: you are implicitly proposing here that the ease of manual configuration be a critical design goal. Let’s make such proposals explicit.

This thread is not intended to be the place for technical discussions, it is a meta-discussion. You and @dagar appear to be interested in this specific technical matter, so let us discuss it separately. EDIT: see Port type safety enforcement

8

I think @tridge approach of having the forum category read only for non members is best. It avoids double effort keeping up with the repository and keeps everyone in the loop, as it stands there’s enough confusion with DS-015 gone, and the move to this SIG, if we continue working in the dark it will be hard for the communities to follow our progress and understand the decision making. I don’t see any reason why would this not be possible. Can you please explain?

9

Keeping the conversation private helps us eliminate ill-informed opinions and incentivize membership. The former saves time and effort; the latter helps us sustain our activities. Keeping everyone in the loop is a questionable objective by itself; it is sufficient to ensure that the core members are kept up-to-date, which is already the case.

Based on my prior experience, I don’t think there are any significant disadvantages that come with making the design process closed.

10

I agree we need to keep the conversations focused and keep outsiders from interfering. But they doesn’t mean closing the door to the community to read/see our discussions. Transparency builds trust.

1 Like
11

Let’s return to this questoin after the first rough draft of the standard is finished.

12

@pavel.kirienko to state the obvious, the only participant in this discussion who doesn’t want it to be public is you. Read-only for the general public/broader forum is a good compromise. We’d like that to be implemented ASAP, not after a draft is ready.

3 Likes
13

Hey all, just wanted to check in on where we are in the development of the new standard. It seems like we’ve kind of stalled on discussion, and it’s been around two weeks since we last discussed standards development. I just want to ask a couple of clarifying questions so I can be sure where we stand on various points:

  • Are we going to open UDRAL discussions to the general public as read-only? It seems that the majority of members are in support of this, as it allows everyone to know how the standard is progressing while still keeping the core discussion focused. I think it would be a good idea to make a decision on this now, as opening the discussions after a rough draft is finished somewhat undermines the reasons for opening the discussion in the first place (to build a transparent environment around the development of the new standard). While we could update a separate repo with the rough draft of the standard, in my opinion it is not necessarily a substitute for opening discussion.

  • @pavel.kirienko mentioned that we should agree on technical requirements before starting work on a message set. Do we have any sort of document for drafting these technical requirements? Perhaps we should start a Google Doc or something similar for drafting to centralize ideas in one place?

  • Do we have any sort of timeline in mind for developing the UDRAL standard? General time goals (e.g. outline a draft of the technical requirements 2 weeks from now) may help to anchor development by giving us a specific time frame to work with.

1 Like
14

Hey Vincent!

I do expect this to happen at one point, yes. I don’t think doing it now is a sensible idea because I don’t want to risk deterioration of the discourse similar to what took place in the DS-015 discussion we had a few months ago. I don’t think we are compromising on trust or openness by keeping the conversation closed for now.

The point of providing read-only access to this forum afterward is to provide the “paper trail” for all major design decisions that shape the standard.

We are working on that already. The first item to decide upon is the problem of port-ID assignment, it is being discussed here: Port type safety enforcement Right now we are waiting for @dagar to submit his feedback on my proposal.

Having a well-defined timeline would certainly help. I am not at liberty to define one so perhaps we should seek input from @dagar and @tridge.

2 Likes
15

Thanks for the response.

Perhaps having some sort of regular (weekly or bi-weekly?) call to sync on UDRAL between developers of the standard would help us move forward on development. It definitely feels right now like the forum is not currently effective at facilitating discussion and reconciliation, and that there is a lack of communication.

I want to see this standard move forward. I feel that this is currently difficult because the different members of this SIG currently disagree on what the standard should be - this is not just a disagreement on technical requirements, but a disagreement in the philosophy of how UDRAL should operate.

From my (limited) idea of the current Ardupilot-aligned community (@tridge and co ), it seems that AP is simply seeking a project that will be UAVCANv0 But Better™. This would, in my understanding, mean a simple, flat message standard that closely emulates UAVCANv0’s standard data types, likely rigidly defined with fixed port IDs.

On the other hand, @pavel.kirienko seems to be seeking a much more ambitious distributed computing solution, in the vein of ROS. This includes integrator-defined port IDs and a more complex, possibly nested set of message types - essentially something similar to DS-015. Such a solution would likely be more flexible but raise complexity.

In summary, @tridge wants UAVCANv0; @pavel.kirienko wants DS-015.

Currently, I believe these two views of where UDRAL should go are fundamentally incompatible. They must be reconciled as soon as possible; otherwise, this standard will die in the SIG before ever being adopted. We have already seen continued conflict between these two views undermining the development of the standard in this thread alone. Technical requirements and architectural discussions are largely pointless when this group fundamentally disagrees on the philosophy of the new standard; inevitably, all technical disagreements will revolve around this divide.

I believe a meeting/call is necessary to resolve these issues, as previous attempts to discuss this on the forum have resulted in failure and entrenchment. Perhaps a meeting where we can actually hear each other’s voices will make it easier for us to come to a solution.

I want to see this new standard succeed and enter adoption. We need to discuss the future of UDRAL thoroughly and resolve these ideological disagreements before it tears apart the standard.

1 Like
16

I am also in favor of a dev call; I think the forum is “too asynchronous” causing us to get frustrated at each other instead of coming to a consensus. I would suggest a weekly call for now. I am not at liberty to choose a time, but invite @tridge @pavel.kirienko and @dagar to suggest a time that works for everyone (Daniel, Vincent, and I are on the same continent so thats easier). We can always transition to a forum once we have something in place.

I agree that technical discussions don’t mean much if the members fundamentally agree on what they want to see in the standard - but I think we can come to a conclusion, especially though a call. The fact is, UAVCANv0 exists - and that’s our starting point. While my personal feelings on the matter lean towards Pavel’s vision of a distributed, abstract and flexible network, we need to recognize that v0 is what is currently vendors/adopters as well as autopilot implementors are used to (and quite honestly, it works reasonable well).

All of this said, nothing moves a project forward like some tangible progress. After a certain point, technical discussions start to stagnate and don’t bring us closer to a UDRAL standard that anyone can use. I feel that it would be better to 1) outline some basic requirements (which has already largely been done) and 2) throw a rough draft together, with the full disclosure that it is not ready for stable use.

At the risk of being a bit too rushed, I have created a google document: https://docs.google.com/document/d/1oD-ngXTAG6-qQePqgSZkpUgQCgGF4Bc7O19cXCfDaK4/edit?usp=sharing It is currently owned by AmadorUAVs (my drone team, which is also a consortium member) but if Pavel or someone else would like to transfer ownership that is fine with me. We may end up scrapping the document 2 weeks from now but I feel that at least we will have something to work with. I have given edit access to anyone with the link.

On the bright side, I do think there’s some things we can agree on:

  • (maybe not completely in agreement) fixed port IDs don’t look like they will happen
  • We really need some type safety standards. Currently I like @pavel.kirienko’s UDRAL class idea, but it will likely need some revision before we end up adopting anything. For the sake of having something concrete, I will start to put together device classes on the google document.
  • Nesting can be useful, but too much nesting (as seen in DS-015) causes unnecessary complexity, overhead, and developer headaches. Please, let us try to non-rigidly constrain the amount of nesting.
  • The need for a good automatic integration system that captures 90% of user requirements is paramount. It has become clear that this is a prerequisite to adoption, not something we can add on later. I like the ideas being discussed in Port type safety enforcement but I think we are waiting on @dagar and @tridge to give their opinions on the matter.
  • Let us please make a compromise between efficiency and architectural values. As @tridge pointed out, DS-015 makes a complete mess of this, wasting bytes on what is, after all, a limited bus (classic CAN). Designing something that only works properly on CAN-FD is IMO not a good idea at all. It will be many more years before CAN-FD becomes a standard on the market; I am yet to see any vendors even supporting it, even on newly released hardware. Besides, keeping an eye on efficiency is healthy. Now, that said, making poor abstraction choices with the goal of saving a couple bytes so that a message can be crammed into fewer CAN frames, is also not a good solution. It makes a mess of the datatype definitions and makes the standard less future proof. I think a good compromise would be to split up messages that are semantically separate while also questioning the addition of fields to a message beyond what is necessary (for example, the controversial covariances in DS-015, which I personally see little use for, are quite large).
17

I think we are mostly on the same page.

My original proposal to define the high-level business requirements before switching to the technical means is intended to avoid the problem of fundamental misalignment you mentioned. As I said, we are working on that already in that other thread, pending input from Daniel. It was discussed at today’s dev call.

Speaking of calls, perhaps you are right: having a periodic sync call would help us maintain some cadence. I would suggest piggybacking on the existing weekly call but I’m not sure how compatible it is with @tridge’s timezone and schedule. Feel free to propose a new time although I don’t expect it to be easy considering our time zones.

I think it’s less of an issue now than it used to be thanks to Vincent’s documentation tool. Do give it a look if you haven’t already (it’s already merged into Nunavut). I am obviously biased but it seems like with this tool one can comprehend complex hierarchies easily.

18

I think it’s less of an issue now than it used to be thanks to Vincent’s documentation tool. Do give it a look if you haven’t already (it’s already merged into Nunavut). I am obviously biased but it seems like with this tool one can comprehend complex hierarchies easily.

Yeah, I’ve been following the development of this, and it’s definitely a lifesaver. However, adding too much nesting can still lead to a loss of the big picture and decrease efficiency, so it’s just something to keep in mind.

My original proposal to define the high-level business requirements before switching to the technical means is intended to avoid the problem of fundamental misalignment you mentioned. As I said, we are working on that already in that other thread, pending input from Daniel. It was discussed at today’s dev call.

I understand what you’re saying, but in this particular case, I think it would be beneficial to jump start progress with some tangible ideas as well as lay out the high level requirements on the document. I request that you (@pavel.kirienko) and of course, everyone else, try this out to see if it helps our work. If we find ourselves following the wrong priorities as a result, we can change course. Besides, my idea was to lay out what we need, in essence, before we start the discussing what types are actually needed.

19

Thank you for the reply. Apologies if I may have sounded a little overly frustrated or alarmed.

For the issue of a periodic sync call, I think perhaps piggybacking on the current weekly call (which takes place at 3AM in Australia) would probably not work. Unfortunately, as you pointed out our time zones make it inconvenient to meet synchronously. The best single time I can come up with is 9PM PDT = 12:00AM EDT = 7AM Estonia = 2PM Australia, which is definitely not ideal (although I have no clue as to what everyone else’s schedules and availability are).

As for the points Kalyan made above, I am in agreement that perhaps some tangible progress will move the standard forward and lift it out of stagnation. Our team has proposed some general design goals and device class requirements here; perhaps you’d like to review it and offer suggestions and corrections?

20

I am reluctant to provide my input regarding the high-level goals because I already did that for DS-015, and that proposal still stands. I don’t think there was anything fundamentally wrong with DS-015 — it was designed on solid and well-known principles, which I expect to work well for UDRAL, too. Yet, I would be the first to admit that the implementation (not the high-level design) of DS-015 could have been done better.

My position concerning the high-level principles can be found in the Interface Design Guidelines that all of you are already familiar with. Let me also add the deprecated DS-015 document here for the sake of completeness (it doesn’t discuss the architecture much though and as such it is of little value in this conversation but I would like it here for posterity, since the Dronecode repository that used to contain this doc is no longer available): DS-015 UAVCAN Drone Standard.pdf (2.3 MB)