Central repository of DSDL namespaces

cc @scottdixon @chemicstry

Concept

When operating Cyphal tools that manage DSDL, the user is required to provide the DSDL sources manually. Examples of this include the use of Yakut, Yukon, or Nunavut. The tool UX could be improved if they were able to locate the required DSDL definitions automatically and the user could utilize the semantic versioning features of DSDL directly without the need to manage the DSDL source versions manually. This is easily implementable through a centralized repository of DSDL namespaces.

The workflow I envision is as follows: when a tool encounters an unknown DSDL data type name, it would automatically fetch the list of all known DSDL namespaces — not only standard but also vendor-specific — from some URI on the web, parse it, and check if the required root namespace is available there. If it is, it would automatically fetch the DSDL source archive from the URI associated with that namespace and process it.

This approach would, ideally, obviate the need for manual namespace management completely, as if all of the well-known DSDL namespaces in existence were always available to the tools locally. One might see how this approach mirrors the package repository design for some programming languages, such as PyPI or the Rust crate registry.

To enable vending of custom repositories, the user should be able to specify a custom URI for the repository. The default repository is to contain strictly only the public regulated namespaces.

A repository definition can inherit namespaces from other repositories. The public regulated repository is not allowed to depend on other repositories so as to prevent unintended inclusion of third-party namespaces and the formation of cyclic dependencies.

Proposed implementation

Define the entry schema for the repository in YAML; variables are shown in ALL_CAPS:

# List of repositories whose contents are entirely included into this one.
# Cyclic dependencies are not permitted.
extends:  # Assume empty if missing.
- uri: REPOSITORY_URI
- ...

# List of namespaces defined in this repository.
# If a namespace defined here is also defined in any of the inherited repositories,
# the local definition overrides the inherited one.
namespaces:  # Assume empty if missing.
  ROOT_NAMESPACE_NAME:
    source:
      zip:
        uri: ZIP_ARCHIVE_URI
        path: PATH_INSIDE_ZIP_ARCHIVE
    dependencies:
      namespaces: [ROOT_NAMESPACE_NAME, ...]
  ...

Definition of the root repository https://dsdl.opencyphal.org/public_regulated.yaml:

# The root repository is not allowed to depend on any other repository.
# The root repository is only allowed to define public regulated namespaces.
namespaces:
  uavcan:
    source:
      zip:
        uri: https://github.com/OpenCyphal/public_regulated_data_types/archive/refs/heads/master.zip
        path: /
    dependencies:
      namespaces: []
  reg:
    source:
      zip:
        uri: https://github.com/OpenCyphal/public_regulated_data_types/archive/refs/heads/master.zip
        path: /
    dependencies:
      namespaces: [uavcan]

The root public_regulated.yaml repository file will be hosted in the existing public regulated data types GitHub repository.

Example of a vendor-specific repository, say, https://dsdl.zubax.com/repository.yaml:

extends:
- uri: https://dsdl.opencyphal.org/public_regulated.yaml
namespaces:
  zubax:
    source:
      zip:
        uri: https://github.com/Zubax/zubax_dsdl/archive/refs/heads/master.zip
        path: /
    dependencies:
      namespaces: [uavcan]

Users are expected to define their own local repository.yaml file which normally should only contain links to external repositories and define none of the namespaces by itself:

# ~/.dsdl.yaml
extends:
- uri: https://dsdl.opencyphal.org/public_regulated.yaml
- uri: https://dsdl.zubax.com/repository.yaml
- ...

Cyphal tools would ingest the location of the repository file via the environment variable like CYPHAL_REPO, somewhat similar to CYPHAL_PATH; the default value should be https://dsdl.opencyphal.org/public_regulated.yaml.

Usage in tools

Yakut

When the tool encounters a command like this:

y pub -T 0.1 10:reg.udral.service.common.readiness 3

It would automatically detect the root namespace reg, fetch the repo file, fetch the archive, find the root namespace directory in it, compile it, and cache the compiled definitions locally. This will only be done once and the cached data will persist until the user references a data type version that is not present in the local cache, which will force the cache to be invalidated. Thus, the versioning capability built into DSDL will drive the cache invalidation automatically.

The “compile” command and the associated installation steps will thus be eliminated.

Yukon

When the tool encounters a type name register matching uavcan\.(pub|sub|cln|srv)\..*\.type that contains a data type name unknown to the tool (not present in the local cache), it would automatically consult with the registry and only alert the user of the unknown data type if it is not available in the registry.

Nunavut

While not a priority, I will list this possibility for completeness anyway. Currently, Nunavut is invoked with the specific DSDL root namespace path (or several):

nnvg --target-language c public_regulated_data_types/reg --lookup-dir public_regulated_data_types/uavcan

With this proposal implemented, one could do this instead; observe that it is no longer necessary to specify the lookup directories:

nnvg --target-language c --namespace=reg
2 Likes

I think we should define a schema and host one for the public_regulated types but tools should have ways to specify several services that vend their own repository.yaml files. We’ll have to define rules for collisions but a decentralized approach prevents OpenCyphal from becoming an API management organization and it allows organizations to host private, internal repository.yamls along-side the public regulated data types.

Vending custom repository.yamls is certainly a sensible idea but am I reading it correctly that you are against having any vendor-specific namespaces in the central repository.yaml?

Would you find it acceptable if we were to allow a repository to inherit namespaces from other repositories (by way of a schema extension), such that there would be several repositories ranging from the most bare-bones root repository that only contains the public regulated namespaces, up to some hypothetical extras.yaml that inherits the former and adds arbitrary vendor-specific namespaces? This design has certain similarities with APT package repositories.

Correct

Yeah, that sounds fine.

1 Like

I have updated the OP post accordingly.

Let’s make sure our tools are using https properly.