I’m proposing replacing pydsdl and nunavut both with a new project that does all code generation using MLIR and LLVM
I’ve code-generated a version that will generate c, c++, c++/pmr, Rust (std), Go, Python, Python w/ C bindings, and Typescript.
This is still a proof of concept but I’ve reached a level of comfort with the idea to propose it here. I’d like to move it into the garage and start maturing it. My plan, as the Nunavut maintainer, is to deprecate Nunavut as soon as I’ve insured llvm-dsdl has reached parity (and more) and has a release pipeline for windows, mac, and linux. This is a native compiler and will be released as binaries. It should perform significantly faster because of this.
The use of MLIR means there is a significant amount of shared structure in the compiler where adding new languages is no longer a major undertaking (It took codex about 22 minutes to add both pure Python and Python with C bindings to the project).
What’s left to do?
Python testing wasn’t finished. Low-risk grunt work for the codegen
release pipelines and CI
Peer reviews of the design
The frontend and CLI need more features
documentation needs to be cleanedup
Some hand-tuning of the generated types (cosmetic work)
I am now going to experiment with this a little with the help of my own agents. The immediate feedback I have so far after opening the repo is that there’s too much red herring in the README, it looks LLM-generated and a little noisy, I think some cleanup would help.
For sure. The documentation is a mess. This is all LLM generated. It’s a bit of an experiment in just how far I can push LLMs to work autonomously since I simply don’t have time to do all of this myself.
As for Rust no-std? Easy. I’ll do that next. I’m just having codex finish up it’s work on Python right now.
I also just pushed dsdld last night. This is a full-featured language server for DSDL. I’ve only just generated the framework but haven’t had time to debug it so it probably has big gaps but the idea is there which is a clangd-like daemon for DSDL.
What does pydsdl do that this package couldn’t do better? For one thing the performance is fantastic and it’s not even been aggressively optimized. It can chew through 2000+ DSDL types and generate c code for them in ~6 seconds.
The front end hasn’t been developed much at all so there’s a lot more clean up to do there but, in all, the fact that we’re building LLVM AST should mean this code base can be whatever pydsdl can be and can do it better simply because it’s native code.
Finally, the distribution story is going to be a bit of a pain for me but for users it’ll be so much better then python. I’m completely over Python’s broken distribution story. It’s just been a nightmare of incompatible runtimes and ten different ways to setup an environment. For llvm-dsdl my plan is to push it through all the standard distribution channels for such tools; brew, apt-get, maybe WinGet.
Of course. But I meant that the LLMs took about 20 hours to add the first language then then it only took the LLM 22 minutes to add Python after establishing all that work. The 20 hours of building a common representation with 22 minutes to add a new language on top of that shows how much logic is shared and how, relatively, little new logic is needed to add language back ends on top of it.
It exposes the AST in Python. This is occasionally useful, such as for example for serialization without code generation (see linked PR). It is also useful for linting, although you could argue that dsdld could also do that so not sure about this point specifically. It can also be used as a relatively clean reference implementation, at least for a while. Overall I see no point deprecating it unless we introduce major DSDL spec changes that would require massive rework (but it doesn’t seem like we’re going to do that soon).
Yeah, I’ve implemented linting in dsdld. I haven’t fully developed a ruleset yet but I have it doing the basics with quick fixes and refactoring. Dsdld will provide syntax highlighting, linting, autocomplete/tooltips, quick fixes, full namespace indexing, AI integration via MCP, and possibly language generation target expansion, etc. It is designed to be the only thing one needs to work with DSDL and, because it is using the same AST and MLIR as dsdlc, it will guarantee high-fidelity between the editing state of dsdl files and the code generated for it.
That’s good, but my point is still standing – I want to keep a pure-Python parser (with serializer) in parallel with llvm-dsdl because it has utility of its own that is not directly related to code generation.
I would like to re-float the idea of abandoning the old segregation between “message” types and “RPC-service” types based on the presence of the --- separator. In Cyphal v1.1 the distinction is no longer meaningful, and it is even less meaningful considering the imminent divorce of DSDL from the Cyphal pubsub protocol. I propose that ----containing types simply define a pair of messages that go together. The immediate use for it in Cyphal v1.1 is to conveniently define a message (the first part) and its (optional) response. If there’s use for more than 2 messages in a group, we could double down and allow an arbitrary number of --- separators as well.
There seems to be some value in service types for CAN though, isn’t there? I agree, for higher order systems more sophisticated mechanisms are needed but for CAN bus there’s value in having an innate and primitive RPC system that is suitable for even the most resource constrained systems.
We’re not removing request-response types, we merely need to model them differently to match v1.1 better. Request-response exchanges are still available in v1.1, but they are designed differently – instead of a dedicated transport entity, they are just ordinary messages that you can unicast-respond to.
That is where my proposal ends and uncertainty begins. The main idea is that I want to get rid of the “RPC-service” types as a separate entity, because they have no representation in v1.1, and they make even less sense now that DSDL is going to be a separate standard not directly coupled with Cyphal.
One option is to make the code generator output suffixed types when an --- token is encountered in the definition, similar to your pseudocode. Or suffix Response to the second definition.
The thing is we’d be introducing a DSDL incompatibility with v1.0. I’m not sure what that buys us other than aesthetics. V1.0 can’t just evaporate. We have to support it along with v1.1
Any change is non-breaking and acceptable as long as we retain wire compatibility. Applications using RPC-services in v1.0 will still be able to interoperate with applications using the new notation because it doesn’t affect wire representations. The change is mostly to remove the heavy special casing around RPC-service types in the specification and the reference implementations (PyDSDL and llvm-dsdl), optionally with a footnote that interop with v1.0 services is maintained.
If I have a dsdl v1.1 file using — and I want to also use this for v1.0 I need to either have to different files or I have to special case this. It’s not worth making type definitions incompatible over is what I’m getting at. Sure, it cleans things up a bit but it’s v1.1 not v2.0 here. Especially when it comes to CAN, the choice of which of the two versions to support should be a simple compile-time flag and not a set of different document types.
The syntax is not affected by this, so per my proposal the file will be understood both by v1.0 and v1.1 parsers, but the parsers will present it differently to the consumer: the v1.0 one will infer a service type, while v1.1 doesn’t have service types so it will infer a pair of message types.
Oh I see. Supporting more than one separator is entirely optional. We can drop that feature completely or just make it the user’s responsibility to ensure that if they want compatibility they have to use at most one separator.
