Proposal for a type manifest specification

Development & Maintenance Specification
1

When consuming large pools of Cyphal messages new problems arise that were not foreseen. Specifically, pydsdl and Nunavut currently collect and generate all types found under a given root namespace. This quickly becomes untenable as the number of messages grows. This post proposes a format that allows software projects to declare a set of messages or patterns that match a set of messages within a given set of namespaces as being “of interest” to them. Various tools like pydsdl and Nunavut can then use this format to apply rules like “check for semantic compatibility errors between Corn_2_1 and Corn_2_2” or “only c headers for messages selected and for the types these messages use”. The post goes further to call for proposals for a minified message description format to allow build-time (and possibly runtime) interoperability checks between software projects. As such we propose a new format; “DSDL Manifests”

DSDL Manifests

DSDL Manifests describe the DSDL definitions that are available to a system. A rules engine processes this manifest when discovering types within a root namespace and uses it to select specific DSDL versions and to ignore uninteresting types. The manifest may also contain additional rules that describe the compatibility of the selected definitions with previously encountered definitions.

Message Selection

The manifest may contain rules that describe which DSDL definitions are of interest to a system. These rules are used to include or exclude definitions as discovery takes place within a namespace. For example, given the following root namespace farm and the following directory structure:

farm/
    ├── livestock/
    │   ├── Cow_1_0.dsdl
    │   ├── Pig_1_0.dsdl
    │   ├── Pig_1_1.dsdl
    ├── crop/
    │   ├── Corn_2_1.dsdl
    │   ├── Corn_3_0.dsdl
    │   ├── Wheat_0_1.dsdl

A manifest may describes rules like this:

    "farm.livestock.Pig": "1.0"
    "farm.crop.Corn"    : "^2.0"
    "farm.crop.Wheat"   : "*"

Where the rules are:

  • Pig: 1.0 selects the exact version 1.0 so Pig_1_0.dsdl is selected
  • Corn: ^2.0 selects the newest version that is semantically compatible so Corn_2_1.dsdl is selected
  • Wheat: * selects the newest version without any compatibility checking. Wheat_0_1.dsdl is selected but
    warnings should be issued if Wheat 1.0 or greater is selected and --warnings-are-errors should be
    an available option to manifest processors.
  • Cow: is not selected because it is not in the manifest.

Message Verification

Manifests may (optional) contain a (TBD) minimal description of the DSDL definitions previously encountered by a subscriber or utilized by a publisher. This allows additional processing of selected definitions to find transcription errors or illegal changes to versioned types. This compressed representation of the DSDL would not contain constants, comments, whitespace, or other etherial information making it suitable as an input to automated compatibility checkers.

Format

The manifest is a JSON Lines<https://jsonlines.org/>_ format file with each line being one of three types:

  • Header – The first line of the file and only one header is allowed. It contains metadata about the manifest like the version of the manifest format and the default action for unselected types.
    • Exclude - Any type not selected by a rule is ignored.
    • IncludeGreedy - Any type not selected by a rule has all versions of the type selected.
    • IncludeLatest - Any type not selected by a rule has the latest version of the type selected.
    • IncludeReleased - Same as InclusiveLatest but ignores unreleased types (i.e. 0.x versioned types).
  • Signature - Describes a compressed representation of a DSDL definition that has been previously encountered. This part of the proposal is not fully developed but a place for it is reserved and the spirit of this functionality should be included in the discussion of the first version of the manifest specification.
  • Selector - Each describes a single rule for including or excluding types found under a root namespace.
    • Exclude - All versions type selected by a rule are ignored.
    • Include - The rule selects the best, single version of the type that matches.
    • IncludeGreedy - The rule selects all type versions that match.

(IncludeLatest and IncludeReleased are redundant for selectors since they are a subset of version matching rules)

The header must be the first line and must specify how many signatures and selectors are in the manifest.
If there are any signatures they must follow the header and be grouped together. If there are any selectors
they must follow the signatures and be grouped together. The order of signatures and selectors is not important. Tools should detect and report contradicting rules when processing the manifest and treat these as fatal errors.

Fixed port IDs
The manifest ignores fixed port identifiers as they are not part of the type signature.

Example

{"type":"header", "version":"1.0", "default-action": "Exclude", "selectors": 6}
{"type": "selector", action="Include", "parts": ["uavcan", "node", "ExecuteCommand"], "version":"^1.0"}
{"type": "selector", action="Include", "parts": ["uavcan", "node", "Heartbeat"], "version":"^1.0"}
{"type": "selector", action="Include", "parts": ["uavcan", "primitive", "String"], "version":"^1.0"}
{"type": "selector", action="Exclude", "parts": ["farm", "livestock", "Cow"], "version":"*", comments="No red meat please"}
{"type": "selector", action="IncludeGreedy", "parts": ["farm", "livestock", "Pig"], "version":"^1.0", comments="The other white meat"}
{"type": "selector", action="IncludeGreedy", "parts": ["farm", "crop", "Corn"], "version":">=2.0"}

The types are expressed in parts to allow search trees to be built and encouraging performant processing of the selectors. We specifically avoid regular expressions for this reason.

While not a Python-specific format, the version matching rules are borrowed from Python (todo: PEP???)

Comments are optional but should be emitted as console output in CLIs or in log files when encountered.

1 Like
2

I think that the signature must contain:

  • type
  • semantic name assigned
  • counts for arrays
  • version
  • Port ID if present in name

If any of these things are different, a semantic difference could be present. Is this in line with your thinking?

3

Not port ID. I’m pretty sure the type is completely decoupled from these identifiers. The manifest is very specifically about the DSDL type system and it will not scale to become a system configuration language. There are other candidates for that like YANG models

1 Like
4

My initial thought was that we’re approaching the problem a little backwards through negative filtering (exclusion of irrelevant types) rather than positive selection (inclusion of the types of interest only), as is commonly done in build systems out there. But then I realized that if we make our DSDL processing front-ends (PyDSDL, Canadensis) aware of the manifest model, as already proposed by Scott for PyDSDL, the difference between inclusion and exclusion is eliminated as it is up to the frontend implementation to decide whether the irrelevant types even need to be looked at.

I find the general idea quite sensible but there are a few questions regarding the model itself, and then we need to bikeshed about the less important details like the proposed file format, which I don’t like.

Model issues

Do we need a variety of default inclusion rules?

Existing applications out there are leveraging DSDL without manifests and thus, in terms of this manifest, are practicing the ultimate greedy behavior where everything is included; this option will remain available in the interest of retaining backward compatiblity. This is one extreme on the greediness spectrum. The opposite extreme is a manifest file that specifies that only the types that explicitly match the patterns are included and everything else is not needed.

Do we need to support anything else but the two extremes? My suggestion is to limit the scope of this proposal to explicit inclusion only (which corresponds to choosing the “Exclude” default action), at least at this stage, as the format is extensible and can be amended with missing features later on. Looking at other configuration systems out there, however, I predict it won’t be necessary.

Emergent redundancy in type version specification

This point pertains mostly to Nunavut rather than the manifest model but it should be discussed here.

If an application were to adopt DSDL manifests, a redundancy in the major data type version selection will appear, as the type of a message will need to be specified not only in the application sources but also in the manifest file:

#include <uavcan/node/ExecuteCommand_1_1.h>

{"type":"header", "version":"1.0", "default-action": "Exclude", "selectors": 1}
{"type": "selector", action="Include", "parts": ["uavcan", "node", "ExecuteCommand"], "version":"1.1"}

If the type is already selected in the manifest, restating it in the sources might be undesirable in some scenarios.

Some target languages already provide aliases that allow one to omit the minor version, defaulting to the latest available version:

from uavcan.node import ExecuteCommand_1  # Selects v1.1 because it's specified in the manifest

With the manifest support in place, it would make sense to allow one to omit the version specification in the source code completely, as this is taken care of by the manifest; this will require a new set of aliases that resolve to the latest available major version:

from uavcan.node import ExecuteCommand  # Selects v1.1 because it's specified in the manifest

Obviously, this does not eliminate the possibility to specify the version explicitly if desired.

Automatic compatibility verification

It is challenging to define a sensible set of rules for strict compatibility checking. To solve this problem in general, one has to possess domain knowledge. There have been attempts to build automatic checkers, both within the Cyphal project and elsewhere (e.g., see the DDS CDR specification); in my opinion, they end up too restrictive, preventing some of the safe alterations of existing data types. There are some specific examples where strict rules are difficult to define in the Specification and one in the Guide.

Given the high domain-sensitivity of the problem, how about we approach it from the opposite end: instead of extracting the compatibility knowledge into a set of strict rules, we use automated reasoning?

What I mean by that is that you can upload the Cyphal specification PDF into GPT4 and ask it whether a set of DSDL definitions are semantically compatible with each other. It mostly works, especially if you amend your request with specific pitfalls to pay attention to. Obviously, in this case, building compressed representations is going to be counter-productive, as LLMs can make use of the comments and such.

Now, this is not really within the scope of this proposal, but what I’m trying to say is that maybe we need to approach the matter of semantic compatibility slightly differently.

File format bikeshedding

Use a human-friendly format

The way it is proposed, it seems like there’s been some strong rationale behind this design that is not explained in the OP post. For example, the format is clearly defined as machine-friendly at the expense of human-friendliness; why? The different record types (header, signature, selector) appear to have been introduced in an attempt to work around the lack of support for a global document structure; why not choose a format that doesn’t lack it? JSON Lines is clearly designed for stream processing of uniform records which doesn’t map well to this use case. Last but not least, JSON is not a human-friendly format in itself.

We need YAML (a superset of JSON).

Would anyone like to submit a competing YAML-based proposal before I publish mine?

If we accept my earlier comment about minimizing the scope of the manifest such that it only specifies the types that are needed, then there is probably no need to specify action and default-action at this point (this may change in a future version).

Do not split type names

I think we should treat type names as atomic entities instead of splitting them up because this way they are easier to read and edit by a human and this also makes them full-text searchable. Write uavcan.node.ExecuteCommand, not ["uavcan", "node", "ExecuteCommand"].

Star is not needed

It is unlikely that one will ever want to allow the selection of any version of a datatype without restrictions. If we accept this (and, optionally, treat the major version of zero specially such that it is considered compatible with v1.x, but there are disadvantages to this), then there is apparently no need for the star * specifier as it can be replaced with ^0.0.

Warnings are not needed

Arguably we should stick to fatal errors only.

3rd-party compatibility

While not a Python-specific format, the version matching rules are borrowed from Python (todo: PEP???)

The hat ^ seems to be borrowed from NPM, not Python. The Python way would be to use a tilde.

1 Like
5

The proposal is very much concerned with defining targets (positive inclusion) which is the primary purpose of having a manifest. The negative filtering is a necessary implementation detail given our current tooling’s emphasis on globular inclusion of folder trees.

I think there’s value in being able to express something like “include all uavcan/si types”.

I think the two versions here have distinct functions. The manifest versions are about all available types. The header versioning is about the specific types in use.

What you are describing here seems like a separable feature albeit one that doesn’t require much in the form of a specification. It could be implemented in Nunavut with a simple switch like --unversioned-types that generated types like foo.bar.Type instead of foo.bar.Type_1_0 but still put these in versioned files. A build system could do the rest using the manifest. For example, a project might express a desire to use the latest 1.0 of our Type and its build system could generate appropriate macros for inclusions, could generate inclusion code automatically, or could simply copy/rename or symlink to the appropriate version before compiling the code.

The best approach would be layered with the first (and only, currently) layer being just the core versioning scheme itself. From there we can store and check hashes of all semantically significant code in the DSDL which is where my proposal ends for now. This covers both human error (i.e. changing a published DSDL definition without bumping the version number) and unintentional modification (e.g. data corruption or malfunctioning tools). I think your idea of using LLMs is an excellent third layer but is outside the scope of this proposal.

I’ll admit this format was optimized for cmake (cmake can read json natively) but even Python can read json as part of the core language whereas yaml et-al require additional libraries.

I’d prefer one of two directions here: either we utilize some sort of schema-defined json (It doesn’t have to be json-lines) or we define an incredibly primitive text file format that can be parsed with a few, light-weight regular expressions (e.g. [record.split(fields_separator_pattern) for record in document.split(records_separator_pattern)]). The manifest is about build and packaging systems so automation should be the primary author and consumer.

I agree. Can we add to the specification that the “canonical” name for a dsdl type is in this format? I think we’ve implied that dot-separated text is the preferred representation but I don’t remember if we formalized this including how to represent the version and if the fixed port ID is part of this canonical identifier format.

I see now we fully define this in section 3.1.2 of the specification.

Actually, Poetry is using the hat now as well.

6

Okay, then it would appear that we don’t need to introduce the concept of the default action for unspecified types; it is to be always set to “exclude” per the OP post.

There is value, probably, in eliminating the type “Selector” as well, but we can return to this point later.

Okay.

By this, you mean the DSDL language rules that require type definitions sharing the same major version to be wire-compatible with each other?

Okay, I see that here you want something similar to file digests that ignores comments and whitespace; this is not really directly related to the matter of wire compatibility. If we go this way, we should be extra clear about the purpose and limitations of this utility, otherwise one might mistakenly assume that you can’t modify the DSDL structure without breaking wire compatibility.

Can you expand on this? The consumption part is obvious but what are the typical scenarios that assume automatic generation of the DSDL manifests?

7

@scottdixon can you please look at the questions in my previous post?

I recall a few weekly calls back, we discussed that you have a PoC generator of the manifest files; I am very interested to learn more about that because right now, I can’t really fully assess the proposal, as one key part is missing: how the manifest files are supposed to be automatically generated.

I understand your rationale for JSON but I think there is room for a significant simplification of the format. The first step is the removal of “default action” (the default being “exclude” as I covered earlier) and another one is perhaps the migration toward the standard JSON instead of JSON-lines. But this is comparatively unimportant right now as we have higher-level questions open.

8

My thinking as evolved on this recently. The more obvious part of my proposal, simple dependency declarations, should simply utilize the dep-file format. I can have Nunavut support generation of these files.

The compatibility checking part of this proposal is somewhat underdeveloped. I think it can be withdrawn for the time being until we have dep-file support in Nunavut and the changes needed in pydsdl to support this which are in development in pull-101

9

lovely