Library-independent code generation

[pavel.kirienko]

@scottdixon would you say that extracting the C++ code generator from libuavcan into a separate, independently reusable entity, is feasible? My attempt to identify items that generated code might depend on shows that most of those come from the C++ standard library rather than the UAVCAN implementation library, meaning that, proper design provided, one can easily reuse code generated for one UAVCAN library with another UAVCAN library.

This is valuable not only from the standpoint of code reusability but also for consistent application design. For example, a node may have both CAN and serial interfaces exposed; suppose that UAVCAN is used over both of those. There is libuavcan for CAN; for serial, either an ad-hoc implementation or a separate library (if there is one) will need to be used. The transports are quite different, but the DSDL layer is identical, so it would make sense to reuse the same generated code with all involved transports.

Another practical example of code reuse between different implementations is libuavcan vs. libcanard. Having a common C++ code generator for both (regardless of the fact that libcanard is a C99 implementation) reduces the code maintenance effort considerably (applications that cannot leverage C++ will not benefit from this, obviously).

If library-independent code generation is implemented, it would automatically resolve this issue as well:

Support low-level transport access

…including the following implied sub-issue, assuming that the virtualized application is implemented in C++:

A possible solution is to let the virtualized systems perform DSDL object serialization on their own, thus effectively eliminating the type information, and interact with the UAVCAN bus by cutting directly into the transport layer of the host’s stack. So that, for example, when a virtualized application needs to emit a particular transfer, it would serialize it internally and then hand over the chunk of bytes to the host accompanied by the port-ID, destination node-ID, priority, and whatever else is needed to let the transport layer dispatch the data correctly. Data reception is the reverse of that: the virtualized application instructs the host which port-ID it is interested in, and the host takes care to route all incoming transfers accordingly.

I am weakly considering right now if there is merit in having the code generator I made for PyUAVCAN transformed into a library-independent one, too.

I seem to remember @kjetilkjeka planning on implementing a similar approach for his Rust implementation (couldn’t find the discussion but it’s certainly somewhere public).

If we were to do this, it could become a slight paradigm shift, because UAVCAN libraries would be essentially just implementations of transports only rather than of the full protocol stack as they are now. Continuing this line of reason, one might see the appeal of the following proposal because it underlines the weakness of the logical coupling between DSDL and the rest of the stack:

Uavcan file extension

I’m working through the chapter on dsdl in the v1 specification and realized that I don’t know why dsdl files have a .uavcan extension instead of a .dsdl extension? Should we support either?

[scottdixon]

Totally on board with this. How do you want to handle the logistics? Should this be its own repo, a sub-folder of libuavcan, or perhaps this becomes something we add to Nunavut?

We could also decide to implement this as c99 (i.e. just factor this out of libcanard) and use c99 dsdl data structures in libuavcan?

[pavel.kirienko]

Totally on board with this.

Sigh. I was hoping for a lengthy debate with much deliberation on either side. (no not really)

How do you want to handle the logistics? Should this be its own repo, a sub-folder of libuavcan, or perhaps this becomes something we add to Nunavut?

Can this be just a collection of predefined templates shipped with Nunavut? We could put them somewhere under https://github.com/UAVCAN/nunavut/tree/master/src/nunavut with a subfolder per language. Basically, all this would move under nunavut/predefined_templates/python: https://github.com/UAVCAN/pyuavcan/tree/uavcan-v1.0/pyuavcan/dsdl/_templates

Come to think of it, the above might be insufficient, because looking at the PyUAVCAN code generator I see that there is like a hundred lines of logic around the templates that appear quite necessary: https://github.com/UAVCAN/pyuavcan/blob/uavcan-v1.0/pyuavcan/dsdl/_compiler.py. So maybe instead of templates we could ship dedicated Python modules that can be also run as CLI tools, one per target language? nnvg-python, nnvg-cpp, etc.

I don’t yet know.

We could also decide to implement this as c99 (i.e. just factor this out of libcanard) and use c99 dsdl data structures in libuavcan?

When the ancient proto-civilizations first invented the C language, they certainly did not plan for such use. This is confirmed by a recent discovery of a cave painting which said (my translation into modern English may be imperfect) that the lack of an adequate object model and namespacing in C makes it unfit for use with DSDL.

[scottdixon]

I’ll play around with nunavut to see how it’ll fit.

[scottdixon]

@pavel.kirienko

Come to think of it, the above might be insufficient, because looking at the PyUAVCAN code generator I see that there is like a hundred lines of logic around the templates that appear quite necessary: https://github.com/UAVCAN/pyuavcan/blob/uavcan-v1.0/pyuavcan/dsdl/_compiler.py. So maybe instead of templates we could ship dedicated Python modules that can be also run as CLI tools, one per target language? nnvg-python , nnvg-cpp , etc.

nunavut already has a concept of language specific modules so this isn’t a stretch. I think I like this idea. My concern is around the complexity of the testing framework as each language is added. We’ll now need to add several C++ compilers and build targets to cover the c++ types. We can reuse these if we pull libcanard’s types into nunavut but if we (perhaps when) we support rust we’ll need to add that compiler for testing and if someone wants go or javascript or ruby, etc. You can see how this would get a bit out of hand perhaps?

I guess what we’re proposing is that nunavut be expanded from a simple code generator to a full-fledged, polyamorous transpiler. This suggests we might want to design an intermediate language for DSDL to allow testing and verification of common logic before the code-gen step. We’d then only have to test the python->DIL (DSDL Intermediate Language) logic and could provide fairly simple semantic verification of the language transforms instead of doing a full build and execute to test each language.

I’m not arguing here (sorry), I’m just thinking out loud.

[pavel.kirienko]

Indeed, we could probably generalize much of the work via DIL (we don’t really want another three-letter acronym, do we? Can we call it DSDIL?), it seems like an interesting idea. The hardest part of code generation is the serialization/deserialization logic where the generator has to statically determine field alignment, and since this property is DSDL-related rather than target-programming-language related, DSDIL could perhaps help here.

I don’t quite understand this part about testing though:

We’d then only have to test the python->DIL (DSDL Intermediate Language) logic and could provide fairly simple semantic verification of the language transforms instead of doing a full build and execute to test each language.

End-to-end testing per language/compiler seems necessary for validation purposes regardless of DSDIL. Am I misunderstanding this part?

The question of source control organization is also relevant. I suppose you would be arguing for a mono-repository containing Nunavut and all language-specific components?

[scottdixon]

We’d then only have to test the python->DIL (DSDL Intermediate Language) logic and could provide fairly simple semantic verification of the language transforms instead of doing a full build and execute to test each language.

(Let’s just ignore this for now)

It occurs to me that pydsdl’s AST is partway to being an intermediate language. Rather than invent something new what if we were to simply expand the capability of the expression syntax when used outside of a dsdl document to support describing serialization and deserialization in an abstract way? Nunavut would then continue to act as a transpiler backend and by simply providing language-specific transforms of these expressions.

[pavel.kirienko]

@scottdixon

Rather than invent something new what if we were to simply expand the capability of the expression syntax when used outside of a dsdl document to support describing serialization and deserialization in an abstract way?

Do proceed