Simpler and better DSDL versioning
I think this suggestion also is the best of both when it comes to code compatibility vs fixing field names and layout
Instead of versioning DSDL types individually we version the namespace as a whole. Rules to make this work better than the alternative will be set in place. This allow things as protocols that use several data types to be well defined.
This is also a more complete proposal including practical solutions for storing/releasing/managing standard and vendor specific DSDL.
There are a few motivating factors for this proposal. The most important is that when creating protocols that use multiple types there is no way to express how the types rely upon each other. With this proposal types in the same major version namespace must work together, in addition they can be sustituted with any version they are
There was a big discussion what changes should be allowed “between” minor version bumps. This suggestion allows a more natural workflow, changing the type as needed and releasing versions when happy.
This suggestion is also a “best of both worlds” in regards to code compatibility. Allowing the breakage of code compatbility while remaining bit/semantic compatibility without introducing minor versions. The only inconvinience is that code breaking changes must be scheduled, which is a blessing in disguise as it gives users more predictibility of when code maintenance is required.
The organization of DSDL
This section describes operational concerns around DSDL. None of this should be put in the specification. This section is included to demonstrate that this solution work well in practice.
For the time being, we’re using Github to distribute code, dsdl and documentation. This proposal will work with Github but also not be problematic to move to other solutions, git based or not. The only requirement is that it’s possible to create separate repositories and run automated checks on them.
We create a new GitHub organization to namespace DSDL where every repo, except those starting with an underscore, will be a root namespace. Initially, only the Uavcan organization controlled “uavcan” namespace will exist but other organizations can apply for their root namespace as well. The relevant branches of these repos must be protected so only changes will be accepted through pull requests.
One of the repositories in the DSDL organization, that is one of the previously mentioned exceptions to root namespace, will be called “_applications”. Anyone can fork this repo and add their application (using a template) for a namespace or vendor Port ID. Obtaining a namespace should be relatively easy, but require some documentation while ID’s would already require a namespace and a bit more justification. An application can be discussed in the PR and the PR can be merged when the application is accepted. That way there will be a public registry of all applications for namespaces and IDs available.
Some CI system (like Travis CI or Gitlab CI) will be used to verify certain properties of the PRs. This way we have the possibility of giving certain key people rights to merging PRs to a namespace they own (as long as tests passes). However, their rights must be limited, as force pushing or merging when tests failed would be dangerous.
Furthermore, all DSDL repos should (at least) be dual licensed under Apache2 and MIT to maximize compatibility and community acceptance.
Versioning of namespaces
This section should not be enforced through the standard, only describes good practice. It should be mentioned explicitly as a non-normative part.
Every namespace is recommended to be versioned with a
<major>.<minor> version number. A convention will be created where it’s recommended to keep a definition in the root namespace containing constants defining the version number and other convenient things. If versioning of the namespaces is used, it’s good practice to keep “version tags” in the repositories to allow easier fetching of a concrete version.
uavcan namespace will follow the previously mentioned recommendations.
Definitions do not have separate version numbers. Since things as early experimentation and tracking of long-term stability are important the following (meta) directives will be used.
@compatible (<major> | 0.<minor>)- “compatible with versions (since or of the same tag)”
@unstable exist for early experimentation, and a definition tagged with this directive do not adhere to any stabilization guarantee and may be changed in arbitrary ways only updating the minor version of the definition. A DSDL compiler should either not compile such definition by default or map it to language mechanism for unstable features.
Since breaking fielded systems is something that can be damaging for the ecosystem. We introduce a
@compatible tag as a first class citizen of DSDL. If versioning is used on some level (namespace or datatype),
@compatible (<major> | 0.<minor>) can be used to express the oldest version in this system the definitions is compatible with.
Two definitions with the exact same namespace, fully qualified path and
@compatible “tag” must be bit and semantic (not necessarily code) compatible.
This “little trick” is what makes this system, even though simpler and more complete, as powerful as “separate versioning of datatypes”. Instead of having the
GetInfo service return a concrete version it will return a “Compatible since” version. If this “compatible since” version matches with your own version everything is good and jolly even though the major version of the namespace is different.
If the workings of a “micro-protocol” changes in a way that breaks semantically compatibility the “compatible since” version must be updated to reflect this. This can only be done during a major version bump, as minor versions must be semantically compatible.
A minor version bump is allowed to introduce the
@deprecated directive. This schedule the type for removal at the next possible moment (if using namespace versioning, this will be next major version bump). This directive can peacefully coexist with the
@compatible x.y and
@unstable directives. But a type tagged with
@unstable could also be removed without deprecation.
Allowed changes for a minor version bump
This section should not be enforced through the standard, it only describes good practice. It should be mentioned explicitly as a non-normative part.
To ensure that updates between minor versions can be done without fear. For a stable definition, every change that increments the minor version must be:
- Bit compatible
- Semantically compatible
- Code compatible
Requiring code compatibility between minor versions is not a problem as the
@compatible x.y directive can be used to express bit and semantically compatibility between major versions. This will allow the code that is produced also to be semantically versioned equal to the DSDL and be distributed conveniently.
Changes to field layout and such (that is semantically and bit compatible but not code compatible) must be scheduled during a major version bump, and mean that such change cannot be done “this week”. This system schedules major version bumps of the uavcan namespace regularly, to allow such small changes can be collected and applied synchronized so only a single code update is required for users. This seems to give excellent possibilities to improve things while giving users excellent stability guarantees.
In addition, removing the
@unstable tag or adding the
@deprecated tag is also allowed for a minor version bump. Changing the
@compatible tag is not allowed as this is a breach of semantic compatibility.
Minor versions do not follow a strict release schedule and are bumped as the uavcan developers feel necessary.
Major version releases guidelines
This section should not be enforced through the standard, it only describes good practice. It should be mentioned in some sort of operation documentation, but not in the specification itself.
The organizations controlling a namespace are allowed to do major version releases as they please. This section describes good practices that they are recommended to follow. The Uavcan organization should do its best to follow this for the official namespace (uavcan namespace) but smaller divergences are tolerated when justified.
First of all, releasing a major version does not necessarily break fielded units or the ecosystem. Units are highly recommended to use the
@compatible directive for compatibility checking of relevant types, and a bump in the major version should not break anything other than the concrete types that are broken. Even though a major version bump is technically free to break all the datatypes it wants to, this should not be done lightly. Stability of fielded units is extremely important, and breaking datatype compatibility must be avoided unless there exist excellent reasons for doing so.
The namespaces should follow a schedule for updating the major version rather than doing it unexpectedly. Even though fielded units or the ecosystem will most likely not break for a major version bump (except in cases where this is absolutely deemed necessary) the code is expected to break. Scheduling removal for non-functional changes, like deprecated types and doing changes that break code compatibility, is highly valuable as this code maintenance can be expected and done efficiently. These non-functional changes can be very nice, but is generally not something that needs to happen overnight, due to this adhering to a release schedule for major versions is something that seems to be acceptable.
The release schedule for major versions for the official uavcan namespace should be every three (3) years with the following exceptions. When version 1.0 is released, version 2.0 is expected to release in one (1) year after. And version 3.0 is expected to release two (2) years after version 2.0. The more frequent releases early on is due to the increased demand for small fixes early on. And the need for deprecating mistakes made early on more quickly.
Comparison to seperate versioning of datatypes
The arguments are denoted as following:
- (+) in favor of namespace versioning
- (-) in favor of datatype versioning
- (=) neutral arguments or statements
Arguments for/against namespace versioning
I will add/remove arguments to this list as the proposal is being discussed.
- (-) Changes requiring code compatibility breakage can not be done “any time of day”.
- (=) Changes to fix names, field-layout etc can be done without breaking bit/semantic compatibility in both alternatives.
- (+) Changes that require code maintenance is expected and scheduled.
- (+) Less surprising, and more similar to what the rest of the world is doing.
- (+) Allows easy and safe distribution of generated code as well as DSDL.
- (+) Gives us deadlines to work towards, motivating continuous improvement.
- (+) Semantic compatibility can be broken elegantly for micro-protocols.
- (+) It’s clear when deprecated types can be removed.
- Exactly what to call the DSDL github repo. I propose
- What semantics should we use on the
@stabledirective. Either “Stable since version x.y” or “Compatbile with version x.y and above”
- Figure out the detail of handing out namespaces/IDs and how the templates should look.
- Gitlab has good CD (continous deployment) integrated, this is in my experience much better than what travis has to offer. Should we consider using gitlab for development with a github mirror?