nope. In CS the syntax of the following two structures is the same:
struct {
uint8_t instance;
int16_t mag_field_mgauss[3];
} struct1;
struct {
uint8_t instance;
int16_t acceleration_mg[3];
} struct2;
but their semantics is different, as the meaning of the fields is different.
Neither the syntax or the semantics tells you anything about what someone will do with the data.
Thinking about this some more, I think we are missing one key factor in the discussion, which is the distinction between regulated and unregulated data types.
I have been focused on regulated data types, such as the core sensor messages for use by ArduPilot and PX4. For those regulated data types the use of a centrally administratively assigned ID is essential as the whole point of the regulation is to create a shared use standard with strong type safety.
For unregulated data types the situation is completely different. In that case a central assignment of IDs would be very counter productive, as it would make v1 much less attractive for experimentation. Having to get IDs assigned by a central administrative authority would be a major pain.
I suspect the examples @pavel.kirienko was talking about on the last call would have been for the unregulated case. For unregulated messages having a registry based lookup to assign subject IDs would be fine. It would actually be very nice, as it means you could mix and match unregulated messages while avoiding collisions either by manual configuration of devices or by string ID matching for the semantic name of the message. We couldn’t do that in v0 without risk of ID collisions.
So I think what we should do is:
This approach has one big advantage of what I proposed before with @semantic_id, which is the ability for even regulated messages to be checked for complete syntactic compatibility via registry lookups (eg. could be done by arming checks). We would just need a structural checksum of the DSDL available as a registry entry on the sending node. Ideally it would actually be 2 structural checksums, one for the unextended message, and one including any extensions that the sending node includes.
This ability to check the structural checksum of messages could be very useful if a vendor releases a bad firmware that sends a regulated message with bad DSDL (eg. through a mixup with git versions). We could catch it and report an error.
I like where this is going, but I do have one other concern:
UAVCANv1 has the intention of being applicable to many vehicular and robotics projects, not just drones/unmanned aerial vehicles. Hence transport layer rigidity such as switching bits and fixing port IDs makes me uncomfortable as it shows a strong preference towards the UAV use case in the regulated types. I’m not sure whether or not this will actually end up being a problem, but it’s something to keep in mind.
for any regulated set of messages you really want those fixed IDs. It isn’t specific to UAVs.
For unregulated messages having the mapping done via registry lookups is a win for sharing messages between loosely connected projects.
Do we want to have a call tomorrow? I can do the same time or earlier.
First I should restate the idea I shared at the call. The old v0 (aka Andrew’s approach) builds the standard in a holistic way, such that the standard envisions the entire architecture of the vehicle. The v1 approach is focused on isolated and composable network services instead to manage complexity and allow reuse. I am not sure analogies can be of help here but perhaps you can conceptualize it roughly as the difference between a highly complex monolithic program versus idiomatic OOP.
In v1, the design of the vehicular system is finalized by the integrator that links various network services together to achieve the required behaviors. This approach is superior compared to the legacy but it is still being misunderstood. I will try another way to explain it, this time speaking in very practical & hands-on terms instead of abstract ideas.
You can grab a COTS product like this and use it as an altimeter:
The underlying lidar is well-suited for any other task involving rangefinding, which is also stated by the manufacturer. Yet, its UAVCAN v0 interface stands in the way. The v0 approach would interface it with the help of a dedicated message type with a fixed-ID roughly like this:
uint8 lidar_id
float32 altitude
You can’t use that for measuring any distance other than AGL.
Okay, so what if you defined a more generic data type that carries just the raw range reading like so:
uint8 lidar_id
float32 range # not altitude but just range to whatever
That lets you remove the assumption about the direction the sensor is facing, cool. Now, suppose that on your (flying) robot there are multiple sensors and your application needs to read the data of select few sensors. You make a subscriber and go roughly like:
def handle_reading(msg):
if msg.lidar_id not in (10, 11, 12, 13):
return # This is not the reading you are looking for.
# Process the reading...
<...>
Problems galore:
In idiomatic UAVCAN v1, you solve this problem as follows:
float32 range
(or you just use uavcan.si.unit.length.Scalar in order to enhance compatibility with 3rd-party software and avoid reinventing this particular wheel)
See the instance-ID? Me neither. In order to interface your sensor with the component that is intended to receive its data, you pick any free subject-ID and assign it to both the sensor and the subscriber. The protocol stack will ensure that the data is delivered from one to the other without introducing undue logical coupling with other parts of the system and without leaking the transport layer details up to your application layer. Shall any new data consumers appear later, you string them onto the same subject-ID.
You might say here that the sensor-ID is unnecessary in v0 because you can just differentiate sources by node-ID. This is also a terrible design because it leaks abstractions the other way: 1. you have to assume that there be at most one data source per node, which is not a valid assumption at the design stage; 2. you can’t rely on the well-known and well-understood principles of data-centric publish-subscribe (DCPS) that would let you isolate data consumers from data providers with the help of the network (which effectively serves as the data broker).
Why is the second point important? Because it reduces the cognitive load on the person designing the network services — in v1, when you design a service you are focused on the service itself and you don’t need to care how the entire network is going to integrate with it. It surely doesn’t matter with this lidar example but it becomes seriously critical when you get to highly complex networks involving multiple nodes interoperating via a dozen of subjects. At this point, people experienced with UAVCAN v0 start asking if there is a way to run DDS over (UAV)CAN because v0 is clearly unfit for the purpose.
Idiomatic DCPS is much simpler conceptually because entities that belong to other layers of the protocol (node-ID from the transport layer and instance-ID) are not manifested in the application. Instead, you have one robust identifier — the port-ID (or the “topic name” in other systems) — that is solely responsible for addressing the data within the system. If you prefer a more traditional CS analogy, think of the network as the address space in a program, where the port-ID is the pointer that points to your data structure in memory. You don’t care who puts that structure where you are reading it from, because it’s irrelevant — all you care about is the data itself. Aside from making the system more flexible, this approach also simplifies failure mode analysis by virtue of involving fewer variable states.
So there is one vendor of quadplane VTOL that once told me this: “We have two groups of ESC in our design: one drives the tractor propellers in the plane mode, the other is used in the hover mode only. We need to run one at a slow rate at the other at a high rate. How do we implement that?”
They were (still are) using v0. The answer I gave them was roughly like: “You don’t – UAVCAN v0 does not support that because this use case was not envisioned when the standard was designed. Go define your own data types to fix it.”
IIRC they are using RCPWM for one of the groups now.
The design broke because the v0 architecture is too rigid. It is, quite literally, its fatal flaw, and I don’t want you to carry this broken design into the new standard. To see how this use case is to be addressed in UAVCAN v1, go look at the DS-015 ESC service, which manages it properly:
The setpoint message does not have a fixed port-ID, allowing you to implement an arbitrary number of groups.
The interface to ESC and servo is largely unified. Commonalities are easy to extract when you are following sensible design practices instead of jamming bits together in one large fixed message.
The feedback from ESC/servo is published in logically pure messages which do not contain any means of instance segregation — this job is delegated to the broker (the networking stack) as it should be. We don’t have an “ESC status” message with everything in it; instead, we have a message for kinematic states, one for electric states, and so on. This is also invaluable for bandwidth-limited networks where the possibility to disable or throttle unnecessary publications is critical.
How can you “re use” this as anything other than a magnetic field?
Magnetic field readings are not only used for navigation. You seem to insist that the only way a (flying) robot or a vehicle may use magnetic field readings is for navigation, which is obviously not true. A magnetometer installed on an electropermanent magnet or on the gimbal obviously has no relevance for the navigation system, yet the published data contains magnetic field readings nonetheless.
This example is derived from my today’s chat with @Dima, he might be able to add more info on this as soon as he caught up with this thread. His team is working on the Innopolis VTOL dynamics simulator (among other things).
In UAVCAN v0, there is an interface for reporting the status of the fuel injection system of an internal combustion engine. The interface aggregates a large number of loosely coupled parameters into one large message, which is a violation of the interface segregation principle and a major design problem in itself, but I spoke about it above already so let’s skip this part. Another major problem is that the interface lacks any controls for starting/stopping the engine and commanding its power setting.
To fix the problem, a v0 system will have to either abuse the ESC messages (which they do in their simulator at the moment) or to define ad-hoc fixed messages I spoke about several posts earlier. A v1 system would not have this problem because the idiomatic way to accept command is to subscribe to a highly generic type like uavcan.si.unit... or uavcan.primitive..., which don’t necessitate allocation of a fixed-ID. New types do need to be designed occasionally, but by virtue of being architecturally pure, they are much more reusable, shielding the end-user of the protocol from problems that arise when the protocol designer failed to envision a specific use case.
Idiomatic v1 would also split the old fuel injection status message into a group of smaller messages arranged roughly as follows:
reg.drone.physics.dynamics.rotation.PlanarTs.reg.drone.physics.electricity.PowerTs
Reliance on highly generic types widens the scope of the standard drastically.
Take a product like this:
Currently, its reuse in applications that do not involve traditional small UAVs is hindered by the rigidity of the v0 interfaces it supports. Let’s focus on the seemingly benign feature: the safety switch button.
In v0, there is a dedicated message type that may be leveraged to publish the state of this button as the global system arming state. But the button is such a basic UI feature, surely I could reuse it for something different?
In UAVCAN v1, it is trivially implementable by publishing uavcan.primitive.scalar.Bit. Any subscriber can be configured to use this button for any purpose by merely setting its subject-ID accordingly. In v0, the button can only be used for those applications that were envisioned by the author of the interface.
Forget the button. What if I have several subsystems that need to be armed/disarmed selectively, as is the case in more complex vehicles? In DS-015, this is addressed by extracting relevant concerns into the readiness service, which can be instantiated as necessary. For example, one instance can be configured to control the avionics while a dedicated one arms the propulsion system (details in the documentation). In UAVCAN v0, none of these options are available because the designer of that service failed to think about more complicated usage scenarios; fixing that requires the introduction of new data types that will have to co-exist with the legacy ones.
I want you spend an hour to launch this demo on your machine (doing that should not require removal of UAVCAN v0 from your system anymore thanks to @coder_kalyan):
https://pyuavcan.readthedocs.io/en/stable/pages/demo.html
Try to answer the following question: how do you implement the same behaviors using fixed port-IDs without defining data types specific to this application only?
We can’t dispatch blobs of bytes to a subsystem in the flight controller based on what amounts to a hint.
This is only true as long as you treat port-IDs as a hint rather than a robust system-defining parameter. There are safety-critical production systems all over the world running diverse protocol stacks ranging from CANopen (with variable/dynamic PDO) and CANKingdom up to DDS and your typical message queues. They do just fine without fixed predefined identifiers because they manage configuration properly. Saying that non-fixed port-IDs are unfit for UDRAL amounts to discarding the extensive experience from the industry, which is hardly a sensible thing to do. Only inherently limited protocols that are useless outside of their extremely narrow domains, such as CANaerospace or MAVLink, can afford to rely on fixed identifiers and be blind to the general trends in the wider world out there.
Your requirement of having fixed identifiers at the cost of reusability and application flexibility is not justifiable. The failure modes you are concerned about are only manifested at configuration time and therefore they do not affect the operational safety of the vehicle. At the configuration stage, they are trivial to mitigate using mechanisms already discussed.
We are not going to re-introduce the same broken design back into v1, so fixed semantic-IDs are not happening. Instead, I suggest we give ArduPilot a closer look to try and find out how to integrate the non-fixed port-IDs into it without breaking the existing logic and without putting undue strain on the maintainers.
strong typing is one of the bedrocks of reliable computing. It is why we use classes, types etc in languages like C++.
I don’t think this analogy is particularly relevant, but since you brought it up — in native languages like C, C++, Rust, or perhaps any language without mandatory RTTI, types do not exist at runtime. Even Java generics are implemented via type erasure so they lack detailed type information at runtime. You obviously know that. In these terms, interfaces in C++ do work like subjects in UAVCAN v1 — the compiler guarantees that the types are correct before the runtime, just like the configuration stage guarantees that subjects are linked correctly before the vehicle is operational. At runtime, all you have are binary blobs.
For unregulated data types the situation is completely different. In that case a central assignment of IDs would be very counter productive, as it would make v1 much less attractive for experimentation.
<…>
I suspect the examples @pavel.kirienko was talking about on the last call would have been for the unregulated case.
The fact that you mention experimentation suggests that you still don’t understand what port-IDs (topic names) are for. I have attempted to correct this in this post; let me know if I succeeded or not. In the context of this discussion, I make no distinction between regulated and unregulated types.
Remember that implementing one method and applying it throughout the stack is also easier than implementing two separate approaches for regulated and unregulated types.
on receipt regulated messages can be immediately dispatched without any registry lookups
Neither of the approaches requires registry lookups at runtime.
Andrew, if I were to put this post into one sentence: I urge you to be a little visionary and look beyond the immediate needs you have in front of you right now. Lest one day somebody will ask you how to run DDS over CAN.
I would like to, however tonight does not work for me. Could we do tomorrow night?
I think we should have this call, because it seemed like we were close to making some progress at the end of the last call, before Andrew had to drop off.
@dagar @pavel.kirienko @bbworld1 Could you please state your availability?
Also, I was wondering if we could move it a bit earlier. I understand that it would be difficult for @pavel.kirienko, but every member of the call in North America was having a hard time keeping focus (especially @dagar, for whom it was 3AM).
Moving the call 30 minutes earlier is not a huge problem. I should be available at this time for the entire week, including the weekend.
Perhaps we should spend some time now analyzing the possible failure modes at configuration time (due to autoconfiguration, user error, hotplug failures, etc) and addressing them in our design before we move forward. Maybe that will make others feel more comfortable?
This sounds like we’ve looped back to the start point.
I’m available essentially any time tomorrow night. I would prefer a slightly earlier time than last time though.
I don’t necessarily see this as a problem of showing a strong preference towards the UAV use case. It is true that there is now a rigid divide between fixed and non-fixed messages - or rather regulated and unregulated messages, in tridge’s new proposal. However, I don’t see anything specific about this proposal that would disadvantage using UAVCAN for other purposes.
I think an interesting point was made about port-IDs - that they are not necessarily 1:1 with subject+node ID. The primary weakness of tridge’s proposal in my opinion is that it ties a single subject - and further than that, a single type - directly to a single port ID; as Pavel mentioned, however, a port ID in UAVCANv1 could be considered to be more consistent with a topic name in another system. This is a drastically different concept from v0’s data type IDs. I think perhaps this is a mismatch in expectations that would be best resolved during the call.
I disagree; for one, we now understand each other a lot better, and there are more concrete examples on both sides. I know it appears like we aren’t getting anywhere, but I do feel that we were getting closer to resolving something last week - I’d like to have another call to see if we can get anywhere.
looks like you didn’t read my previous messages in this topic. I am in no way insisting that magnetic field readings only be used for navigation. Even in v0 the 1002.MagneticFieldStrength2 message is not in any way tied to being used for navigation. In fact, I’ve used it for non-navigation tasks with existing vehicles and ArduPilot.
Please understand that my proposals on both @semantic_id and @regulated_id are not at all about the purpose of the data, they are about the semantics. Semantics is quite different from purpose.
Once you understand that then you should see that using @semantic_id or @regulated_id in no way reduces the utility of the protocol for composability.
arrgh! no!!!
A rangefinder has no way of knowing the altitude. This is like your earlier push for an airspeed sensor in DS-015 to publish a calibrated airspeed. The rangefinder cannot know the altitude and differential pressure sensor cannot know the airspeed.
In both v0 and v1 you would publish a “range” from the rangefinder. You can’t turn that into an altitude without having a lot more information. You need to know:
no, you can’t even use it for AGL … there just isn’t enough information
no! That is not how you do it.
We have a per-rangefinder orientation in ArduPilot. You assign rangerfinders to slots in our AP_RangeFinder library, and each is associated with its own orientation. Then you can do sane queries on the rangefinder library like “give me the range of the first healthy rangefinfer that has an orientation of pitch 270 degrees”. That is exactly what our API does right now with v0.
but we want the association between the particular rangefinder instance to an orientation to be in the autopilot logic, not in the network protocol routing. They both carry exactly the same data and achieve the same goal, but having an instance ID fits how we structure the code perfectly and fits with all out existing serial, i2c and UAVCAN v0 supported sensors.
I know you want to push all that logic into the network. That is not what ArduPilot or px4 want, and it is just counter productive.
no, it actually completely ruins the usefulness of the protocol. That part is what led to the horrors of DS-015. We want the individual elements to be tightly coupled into a single message so we know all the data comes from a single point in time. We don’t want the fuel flow of the EFI message to come separately from the throttle level, because we really do care that we get a single data point that has the throttle and fuel flow from the same sample of the EFI system.
You could do that with your proposal of lots of separate subjects only if you added a timestamp to every single component so you can line them up. That is enormously wasteful and just a major pain to re-assemble into a coherent set of one-point-in-time data on the receiving end.
absolute rot. You can use a button in v0 just as generically as you can use a button in v1. You keep conflating semantics and purpose, they are not the same thing.
nope. We just will not use v1, or we’ll use a fork of it with sane design.
Your long message just convinces me that you really haven’t listened or thought about the issues I’ve been raising for weeks at all.
Thank you @pavel.kirienko for reiterating on the design-intent behind not having identifiers enclosed within the message. I feel that @tridge has two main pain points which are configuration at startup and network debugging tooling.
@pavel.kirienko What possible solutions can you think of to solve Andrews concerns?
@tridge What, fixed-IDs aside, would solve your pain points? Do you have other pain points (Probably to be discussed in a separate thread)?
It may appear this way but I have the distinct impression that progress is being made. In the Japanese martial arts progress is likened to ascending a spiral staircase. You are progressing from basic to ever more advanced techniques until you arrive at the basics again. But your understanding of basics is deeper this time so you are able to perceive and appreciate them with a new clarity which leads to additional understanding not previously possible.
The bottom line is: From the top it appears if you are going in circles. Only when viewed from the (in)side you can see that despite seemingly going in circles you also move upwards 
.
It appears to me that you (Andrew, as acting spokesperson for Ardupilot) do not want to end up with a UAVCAN that requires ArduPilot to adapt its existing implementation because it’s inconvenient. I get that. In fact you’ve got my sympathy. But I think you are seeing this to narrow. UAVCAN V1 will take a while to happen, and the transition will take probably a longer period of time than anyone wishes to admit. Therefore there will be plenty of time to accommodate those changes.
I suggest to let the representative(s) of PX4 speak for themselves. Perhaps even other developers at ArduPilot even share a slightly different view?
Finally, let’s keep in mind that all of us are human beings, with different technical and cultural backgrounds, some even using English as a secondary language. Misunderstandings are bound to happen (Trust me on this, I’m married to a foreigner 
). This can be only countered by adopting an open mindset, really considering different viewpoints, asking question for clarification and giving everyone the benefit of the doubt.
Network debugging is trivial to solve. All we have to do is think about how we do protocol analysis a bit differently. As @pavel.kirienko stated, using wireshark to monitor a V1 network is like using it to decipher the contents of a ROS/DDS transfer. Wireshark is a transport layer analyzer, while message contents are an application layer problem. To view network traffic in something like yukon (WIP), you could configure it with a table associating each port ID with the data type on that port. With autoconfiguration, such a table could even be exported automatically by the allocation authority and imported directly into the tool. Note that it only has to be done once - if the system configuration does not change, the port IDs won’t either. This is, to an extent, how yakut works already - although a convenient table is not (yet) implemented.
@tridge did have at least one other concern about the general robustness of not being able to know the packet semantics by just looking at each message payload. This is where I highly recommend we do an analysis of the real failure modes rather than resort to high level speculation like “this is not robust” or “this is too complex”.
I wanted to mention one more theme I’m seeing here:
The concern of “this is how it’s currently implemented in ardupilot/px4” seems to be coming up often, especially with ardupilot. Now, don’t get me wrong: I am not trying to undervalue/underappreciate the importance of ardupilot, px4, or the possible difficulty of implementing V1 services in either. However, keep in mind that ardupilot and PX4 are just single implementors of a larger protocol which is supposed to be standard across a much wider domain. Let us be cautious about what our design priorities are - the network should be designed to serve the network/standard architecture’s best interests, not the interests of a particular implementation.
The thing about the way Pavel wants to do things is it is based on a false premise. It assumes that semantics == purpose, and then assumes that by adding all this extra complexity of going via remote registry lookups for a regulated data types that we gain some great new capability. The fact is we don’t. Composability and re-use is just as easy with the @regulated_id way as with the registry, but the regulated_id is much much simpler.
please remember that the UI tool is not just for developers - it is a user tool for solving problems in the field. What you are describing is complex and has failure modes that just aren’t justifiable. We want this tool to be as simple and robust as we can make it.
Having the regulated_id in the packet is dirt simple, can’t go wrong, and gives us exactly the behavior we want for regulated message sets where we want to guarantee that all the vendors are using the same DSDL definitions.
It doesn’t matter how the data that the subscriber is receiving is used inside the subscriber.
This part of my example makes assumptions about service orientation, let’s omit it, it is not relevant in this context. The next example under the one you referenced assumes that the sensor is to publish raw range, how do you solve that using your approach? I don’t think a sensible solution exists.
The way you treat data inside ArduPilot matters little because different implementations are expected to utilize the protocol differently. What you described is an okay approach except for the fact that you have to pollute your models with instance-ID and forcing the flight controller to process data it may not need (e.g., how do I connect a rangefinder whose readings are utilized by another node that is not the flight controller?). You can fix the way you handle sensor readings by merely replacing sensor-ID with subject-ID.
I don’t care how it fits your code. I care about sensible architecture that is not a tangled-up mess of different layers.
You can obtain the exact same results by merely swapping sensor-ID with port-ID in your code. Doing this will actually reduce the logical coupling between your application and the network layer.
Generally speaking, bandwidth concerns are grossly overstated. You end up over-optimizing messages like global position or EFI status which only take about 1% of bus load, resulting in bad architecture with no tangible benefits for the network utilization. Regardless, you only need timestamping for reassembly if the data validity period is less than the publication period, which is not the case with applications we are discussing at the moment.
How exactly? By adding a sensor-ID? If you actually read my posts about this you should see how this is a bad idea.
I don’t think you are really receptive to the values I am trying to communicate here. In my last post I asked you a very practical question on how do you address the specific use case presented in the PyUAVCAN demo, but you ignored it. Can you look into it now? Can you provide an equally hands-on illustration of how your proposal doesn’t break in the scenarios I have outlined above?
Hi all,
I’ve been following this topic for some time already. Lots of arguments stated here do make sense, however it seems that either proposal can’t work with the other, thus making a compromise seems to be hard right now.
So let’s start with the fixed vs non-fixed port-ID discussion here. When I started implementing UAVCANv1 on my systems I really disliked the idea not having an easy Plug-and-play experience anymore and the requirement of the configuration by the integrator that would complicate a simple drone setup.
I’ve had some discussion with @pavel.kirienko about this and he was very keen that a fixed port-ID wouldn’t be possible and would break the very thing he believes that UAVCANv1 improves over UAVCANv0 which is the focus on isolated and composable network services in a democratic distributed network.
The focus of UAVCANv1 gives a completely different take on distributed networking compared to UAVCANv0. Where in UAVCANv0 you had fixed portID for messages now a integrator has to setup this manually. Which currently from a end-user perspective would be a huge downgrade compared to UAVCANv0.
I don’t think Pavel is ever going to advocate fixed port-ID’s, thus using the uavcan.register/cookie approach seems to be a compromise to get easy plug-and-play like experience on UAVCANv1 setup.
Regarding robustness and safety this might not the best solution, but then the idea of a system integrator that setups the network would be a robust solution. However tooling (e.g. Yukon) is going to be very important for a good end-user experience.