# Central repository of DSDL namespaces

**URL:** <https://forum.opencyphal.org/t/central-repository-of-dsdl-namespaces/1817>\
**Category:** Development & Maintenance\
**Created:** [January 16, 2023, 11:10am UTC](https://forum.opencyphal.org/t/central-repository-of-dsdl-namespaces/1817 "2023-01-16T11:10:00Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\
**Post date:** [January 16, 2023, 11:10am UTC](https://forum.opencyphal.org/t/central-repository-of-dsdl-namespaces/1817/1 "2023-01-16T11:10:00Z")

</div>

cc @scottdixon @chemicstry

## Concept

When operating Cyphal tools that manage DSDL, the user is required to provide the DSDL sources manually. Examples of this include the use of Yakut, Yukon, or Nunavut. The tool UX could be improved if they were able to locate the required DSDL definitions automatically and the user could utilize the semantic versioning features of DSDL directly without the need to manage the DSDL source versions manually. This is easily implementable through a centralized repository of DSDL namespaces.

The workflow I envision is as follows: when a tool encounters an unknown DSDL data type name, it would automatically fetch the list of all known DSDL namespaces — not only standard but also vendor-specific — from some URI on the web, parse it, and check if the required root namespace is available there. If it is, it would automatically fetch the DSDL source archive from the URI associated with that namespace and process it.

This approach would, ideally, obviate the need for manual namespace management completely, as if all of the well-known DSDL namespaces in existence were always available to the tools locally. One might see how this approach mirrors the package repository design for some programming languages, such as PyPI or the Rust crate registry.

To enable vending of custom repositories, the user should be able to specify a custom URI for the repository. The default repository is to contain strictly only the public regulated namespaces.

A repository definition can inherit namespaces from other repositories. The public regulated repository is not allowed to depend on other repositories so as to prevent unintended inclusion of third-party namespaces and the formation of cyclic dependencies.

## Proposed implementation

Define the entry schema for the repository in YAML; variables are shown in `ALL_CAPS`:

```yaml
# List of repositories whose contents are entirely included into this one.
# Cyclic dependencies are not permitted.
extends: # Assume empty if missing.
- uri: REPOSITORY_URI
- ...

# List of namespaces defined in this repository.
# If a namespace defined here is also defined in any of the inherited repositories,
# the local definition overrides the inherited one.
namespaces: # Assume empty if missing.
  ROOT_NAMESPACE_NAME:
    source:
      zip:
        uri: ZIP_ARCHIVE_URI
        path: PATH_INSIDE_ZIP_ARCHIVE
    dependencies:
      namespaces: [ROOT_NAMESPACE_NAME, ...]
  ...

```

Definition of the root repository `https://dsdl.opencyphal.org/public_regulated.yaml`:

```auto
# The root repository is not allowed to depend on any other repository.
# The root repository is only allowed to define public regulated namespaces.
namespaces:
  uavcan:
    source:
      zip:
        uri: https://github.com/OpenCyphal/public_regulated_data_types/archive/refs/heads/master.zip
        path: /
    dependencies:
      namespaces: []
  reg:
    source:
      zip:
        uri: https://github.com/OpenCyphal/public_regulated_data_types/archive/refs/heads/master.zip
        path: /
    dependencies:
      namespaces: [uavcan]

```

The root `public_regulated.yaml` repository file will be hosted in the existing public regulated data types GitHub repository.

Example of a vendor-specific repository, say, `https://dsdl.zubax.com/repository.yaml`:

```auto
extends:
- uri: https://dsdl.opencyphal.org/public_regulated.yaml
namespaces:
  zubax:
    source:
      zip:
        uri: https://github.com/Zubax/zubax_dsdl/archive/refs/heads/master.zip
        path: /
    dependencies:
      namespaces: [uavcan]

```

Users are expected to define their own local `repository.yaml` file which normally should only contain links to external repositories and define none of the namespaces by itself:

```yaml
# ~/.dsdl.yaml
extends:
- uri: https://dsdl.opencyphal.org/public_regulated.yaml
- uri: https://dsdl.zubax.com/repository.yaml
- ...

```

Cyphal tools would ingest the location of the repository file via the environment variable like `CYPHAL_REPO`, somewhat similar to `CYPHAL_PATH`; the default value should be `https://dsdl.opencyphal.org/public_regulated.yaml`.

## Usage in tools

### Yakut

When the tool encounters a command like this:

```bash
y pub -T 0.1 10:reg.udral.service.common.readiness 3

```

It would automatically detect the root namespace `reg`, fetch the repo file, fetch the archive, find the root namespace directory in it, compile it, and cache the compiled definitions locally. This will only be done once and the cached data will persist until the user references a data type version that is not present in the local cache, which will force the cache to be invalidated. Thus, the versioning capability built into DSDL will drive the cache invalidation automatically.

The “compile” command and the [associated installation steps will thus be eliminated](https://github.com/OpenCyphal/yakut/issues/60).

### Yukon

When the tool encounters a type name register matching **`uavcan\.(pub|sub|cln|srv)\..*\.type`** that contains a data type name unknown to the tool (not present in the local cache), it would automatically consult with the registry and only alert the user of the unknown data type if it is not available in the registry.

### Nunavut

While not a priority, I will list this possibility for completeness anyway. Currently, Nunavut is invoked with the specific DSDL root namespace path (or several):

```bash
nnvg --target-language c public_regulated_data_types/reg --lookup-dir public_regulated_data_types/uavcan

```

With this proposal implemented, one could do this instead; observe that it is no longer necessary to specify the lookup directories:

```bash
nnvg --target-language c --namespace=reg

```

---

<div class="post-metadata">

**Author:** ![scottdixon](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/scottdixon/32/39_2.png) [@scottdixon](https://forum.opencyphal.org/u/scottdixon)\
**Post date:** [February 3, 2023, 5:38pm UTC](https://forum.opencyphal.org/t/central-repository-of-dsdl-namespaces/1817/2 "2023-02-03T17:38:33Z")

</div>

I think we should define a schema and host one for the public\_regulated types but tools should have ways to specify several services that vend their own repository.yaml files. We’ll have to define rules for collisions but a decentralized approach prevents OpenCyphal from becoming an API management organization and it allows organizations to host private, internal repository.yamls along-side the public regulated data types.

---

<div class="post-metadata">

**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\
**Post date:** [February 3, 2023, 5:47pm UTC](https://forum.opencyphal.org/t/central-repository-of-dsdl-namespaces/1817/3 "2023-02-03T17:47:28Z")

</div>

> [@scottdixon](#):
>
> I think we should define a schema and host one for the public\_regulated types but tools should have ways to specify several services that vend their own repository.yaml files.

Vending custom repository.yamls is certainly a sensible idea but am I reading it correctly that you are against having any vendor-specific namespaces in the central repository.yaml?

Would you find it acceptable if we were to allow a repository to inherit namespaces from other repositories (by way of a schema extension), such that there would be several repositories ranging from the most bare-bones root repository that only contains the public regulated namespaces, up to some hypothetical `extras.yaml` that inherits the former and adds arbitrary vendor-specific namespaces? This design has certain similarities with APT package repositories.

---

<div class="post-metadata">

**Author:** ![scottdixon](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/scottdixon/32/39_2.png) [@scottdixon](https://forum.opencyphal.org/u/scottdixon)\
**Post date:** [February 3, 2023, 5:51pm UTC](https://forum.opencyphal.org/t/central-repository-of-dsdl-namespaces/1817/4 "2023-02-03T17:51:30Z")

</div>

> [@pavel.kirienko](#):
>
> Vending custom repository.yamls is certainly a sensible idea but am I reading it correctly that you are against having any vendor-specific namespaces in the central repository.yaml?

Correct

> [@pavel.kirienko](#):
>
> Would you find it acceptable if we were to allow a repository to inherit namespaces from other repositories

Yeah, that sounds fine.

---

<div class="post-metadata">

**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\
**Post date:** [February 3, 2023, 6:15pm UTC](https://forum.opencyphal.org/t/central-repository-of-dsdl-namespaces/1817/5 "2023-02-03T18:15:24Z")

</div>

I have updated the OP post accordingly.

---

<div class="post-metadata">

**Author:** ![scottdixon](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/scottdixon/32/39_2.png) [@scottdixon](https://forum.opencyphal.org/u/scottdixon)\
**Post date:** [February 3, 2023, 9:09pm UTC](https://forum.opencyphal.org/t/central-repository-of-dsdl-namespaces/1817/6 "2023-02-03T21:09:47Z")

</div>

> [@pavel.kirienko](#):
>
> Usage in tools

Let’s make sure our tools are using https properly.
