# API proposal for libcyphal

**URL:** <https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850>\
**Category:** LibCyphal\
**Created:** [March 18, 2023, 4:41pm UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850 "2023-03-18T16:41:57Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\
**Post date:** [March 18, 2023, 4:41pm UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850/1 "2023-03-18T16:41:57Z")

</div>

I apologize for the clickbait title as there’s actually no proposal here. I just want to share these two files here per @scottdixon’s suggestion:

[can.hpp](https://forum.opencyphal.org/uploads/short-url/z9nej6PzFosIODv5rpPN4jS6lLq.hpp) (27.5 KB)  
[generic.hpp](https://forum.opencyphal.org/uploads/short-url/1OEM84ehgs4RZ23kKFSbX4k8xdn.hpp) (15.4 KB)

`generic.hpp` contains the transport-agnostic part and `can.hpp` contains the Cyphal/CAN specific part. The macros are legacy cruft for compatibility with Nunavut C; ignore them. The key idea is to use type erasure with factory methods to create publishers/subscribers/etc and to decouple the application from the transport specifics via static polymorphism — assume that a hypothetical `udp.hpp` would implement the same methods.

If we can tolerate dynamic memory allocation during initialization, implementing this interface should not be a problem. I think we should embrace type erasure because it allows the construction of concise API that is easy to understand and hard to misuse. In general, I would like libcyphal to somewhat approximate the API of PyCyphal: [Architecture — PyCyphal 1.27.1 documentation](https://pycyphal.readthedocs.io/en/stable/pages/architecture.html)

---

<div class="post-metadata">

**Author:** ![scottdixon](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/scottdixon/32/39_2.png) [@scottdixon](https://forum.opencyphal.org/u/scottdixon)\
**Post date:** [June 10, 2023, 5:27am UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850/2 "2023-06-10T05:27:06Z")

</div>

Pulling a chapter from a very old book, what about this idea to get type-erasure without resorting to templates:

```auto
struct IComposite
{

    virtual SerializeResult serialize(nunavut::support::bitspan out_buffer) const = 0;

    virtual SerializeResult deserialize(nunavut::support::const_bitspan in_buffer) = 0;

    virtual HResult QueryInterface( const char* type_name, void *& out_object) noexcept = 0;
);
protected:
    ~IComposite() = default;
};

```

and then a helper that hides the unsafe stuff:

```auto
template<typename T>
std::optional<std::reference_wrapper<T>> dsdl_cast(IComposite& obj)
{
    void* obj_as_i;
    if(0 == obj.QueryInterface(dsdl_traits<T>::type_name(), obj_as_i))
    {
        return std::optional<std::reference_wrapper<T>>{*reinterpret_cast<T*>(obj_as_i)};
    }
    return std::optional<std::reference_wrapper<T>>{};
}

```

so you end up writing code like this:

```auto
void printIfFoo(IComposite& unknown)
{
    auto obj = dsdl_cast<Foo_1_0>(unknown);
    if (obj)
    {
        std::cout << "unknown was a Foo_1_0. data = " << static_cast<Foo_1_0>(*obj).data << std::endl;
    }
    else
    {
        std::cout << "unknown remains a mystery." << std::endl;
    }
}

```

(full, crappy, example: [Compiler Explorer](https://godbolt.org/z/8EcT8zjhj))

@lydiagh ? @pavel.kirienko ?

---

<div class="post-metadata">

**Author:** ![scottdixon](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/scottdixon/32/39_2.png) [@scottdixon](https://forum.opencyphal.org/u/scottdixon)\
**Post date:** [June 13, 2023, 6:08pm UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850/3 "2023-06-13T18:08:11Z")

</div>

Here’s a full proposal mockup for libcyphal itself: [Compiler Explorer](https://godbolt.org/z/hTr5xqrq5)

This should extend to DSDL types once we promote the `IPolymorphicType` to CETL and integrate it with Nunavut.

---

<div class="post-metadata">

**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\
**Post date:** [June 14, 2023, 11:21am UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850/4 "2023-06-14T11:21:06Z")

</div>

I think the static cast here detracts attention from the key part; if I understand correctly, it can be avoided by replacing the optional ref wrapper with a raw pointer:

```diff
-std::cout << "unknown was a Foo_1_0. data = " << static_cast<Foo_1_0>(*obj).data << std::endl;
+std::cout << "unknown was a Foo_1_0. data = " << obj->data << std::endl;

```

I understand that implementing this solution would require raising at least one deviation. The expectation is that a faithfully implemented ordinary type erasure would require raising more deviations which is why we’re avoiding it. Right?

I am not super happy about this design. Before we lose all hope and resort to it, I would like to talk more about the regular type erasure with metaprogramming confined to 4 (four!) critical methods within the library — the factory methods that construct the presentation-level port objects, namely `Publisher`, `Subscriber`, `Client`, and `Server`. In this case, the factory class (which would be named `Presentation` if we were to follow the PyCyphal design) would look roughly as follows:

 

```cpp
/// Polymorphic base classes; definitions omitted for brevity. Implementations are Voldemort types.
class Publisher;
class Subscriber;
class Client;
class Server;

class Presentation final
{
public:
    /// This factory method requires raising a deviation!
    /// The returned type can be used as-is or converted to the Publisher base and used with IComposite.
    template <typename T>
    auto* makePublisher(const std::uint16_t port_id)
    {
        class VoldemortPublisher final : public Publisher
        {
        public:
            VoldemortPublisher(...) {...}

            bool publish(const T& msg)
            {
                std::array<std::byte, T::SerializationBufferSize> buf;
                serialize(msg, buf);
                return iface.publish(port_id_, buf); // This call is made up but you get the point.
            }

            bool publish(const IComposite& msg) override // Compatibility with the IComposite interface.
            {
                if (auto obj = dsdl_cast<T>(msg))
                {
                    publish(*obj);
                    return true;
                }
                return false;
            }

        private:
            const std::uint16_t port_id_;
        };
        return allocator_.construct<VoldemortPublisher>(port_id, ...);
    }
};

```

 

The advantage of this approach is that many (most?) applications will be able to rely strictly on the concrete types rather than using templates or `IComposite` because the DSDL type is specialized in the type of object returned by the factory. The disadvantage is that it requires a few more deviations.

---

<div class="post-metadata">

**Author:** ![scottdixon](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/scottdixon/32/39_2.png) [@scottdixon](https://forum.opencyphal.org/u/scottdixon)\
**Post date:** [June 21, 2023, 5:52am UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850/5 "2023-06-21T05:52:15Z")

</div>

it seems to me there are two problem domains:

1. DSDL types and runtime introspection
2. Concrete object lifecycle in libcyphal

> **This post only discusses item (2) and a technique for erasing platform-specific, concrete types from objects vended through libcyphal.**

I’m working on the first version of libcyphal that we’ll ask for an in-depth design review on and I have the following idea for shielding the transport layer from types defined for a specific platform.

And so I give you … the Voldemort Pointer?

```auto
/// TODO integrate this functionality into cetl::pmr::Factory
struct VoldemortPointer final
{
    template <typename T>
    using allocator_t = cetl::pmr::polymorphic_allocator<T>;

    template <typename T>
    using unique_ptr_t = std::unique_ptr<T, cetl::pmr::PolymorphicDeleter<allocator_t<T>>>;

    /// Construct a new concrete type but return a unique_ptr to an interface type for the concrete object.
    /// This becomes a polymorphic "voldemort" of sorts where only RTTI provides a safe way to refer to the
    /// concrete type after construction.
    template <typename InterfaceType, typename ConcreteType, typename... Args>
    static unique_ptr_t<InterfaceType> make_unique(allocator_t<ConcreteType>& concrete_allocator, Args&&... args)
    {
        ConcreteType* s = concrete_allocator.allocate(1);
        if (s)
        {
#if __cpp_exceptions
            try
            {
                concrete_allocator.construct(s, std::forward<Args>(args)...);
            } catch (...)
            {
                concrete_allocator.deallocate(s, 1);
                throw;
            }
#else
            concrete_allocator.construct(s, std::forward<Args>(args)...);
#endif
        }

        typename unique_ptr_t<InterfaceType>::deleter_type td{allocator_t<InterfaceType>{concrete_allocator.resource()},
                                                              1};

        return unique_ptr_t<InterfaceType>{s, td};
    }
    VoldemortPointer() = delete;
};

```

for example:

```auto
#include "CatToy.hpp"
#include "VoldemortPointer.hpp"

struct ICat
{
    virtual void meow() const noexcept = 0;

protected:
    // Yes, this is a wart. You could make the dtor public but then anyone could delete it.
    // This, at least, prevents a casual mistake and has the added benefit of describing
    // the one place in the system to look for who is going to delete a cat (soul-less bastards!)
    friend class cetl::pmr::PolymorphicDeleter<cetl::pf17::pmr::polymorphic_allocator<ICat>>;
    ~ICat() = default;
};
...

using CatPointer = VoldemortPointer::unique_ptr_t<ICat>;

extern CatPointer make_cat(const CatToy&);

```

… then, elsewhere we implement `make_cat`…

```auto
class TomMarvoloRiddle: public ICat
{
public:
    TomMarvoloRiddle(const CatToy& toy) : toy_{toy} {}
    virtual ~TomMarvoloRiddle() = default;

    void meow() const noexcept override
    {
        // implement meow.
    }
private:
    CatToy toy_;
};

cetl::pf17::pmr::polymorphic_allocator<TomMarvoloRiddle> voldemort_allocator_{cetl::pf17::pmr::new_delete_resource()};

CatPointer make_cat(const CatToy& toy)
{
    VoldemortPointer::make_unique<ICat, TomMarvoloRiddle>(voldemort_allocator_, toy);
}

```

This technique takes advantage of the polymorphic part of the `polymorphic_allocator` in that the deleter is able to use polymorphism to delete the concrete type without a need to refer to the concrete type (i.e. the concrete type is erased in the pointer). This technique also avoids `auto` or hiding anything other than the object’s `vtbl` which is the sole source of dark magic in this configuration.

Okay, true, this isn’t very novel (and the Voldemort reference is contrived) but it’s subtly different from just passing up pointers and deleting through base class types in that the concrete allocator doesn’t need to be much of an allocator at all. In fact, our `TomMarvoloRiddle` object may be an item in a static array where we are using a `monotonic_buffer_resource` or could even be a singleton where we provide a no-op deleter. This pattern completely separates memory management of the concrete type from the APIs that vend references to its interfaces. Furthermore, ownership is transferred to the caller of `make_unique` and the factory context does not maintain any access to the concrete type.

---

<div class="post-metadata">

**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\
**Post date:** [July 3, 2023, 12:12pm UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850/6 "2023-07-03T12:12:25Z")

</div>

> This post only discusses item (2) and a technique for erasing platform-specific, concrete types from objects vended through libcyphal.

This is a slight detour from the topic of my post. I wanted to discuss the erasure of the DSDL type from port objects, such as where a concrete `ConcreteSubscriber<T>` for some DSDL type T is represented via some non-template base `Subscriber`. I mean, it is still type erasure, but applied to a different problem.

> This technique takes advantage of the polymorphic part of the polymorphic\_allocator in that the deleter is able to use polymorphism to delete the concrete type without a need to refer to the concrete type (i.e. the concrete type is erased in the pointer).  
> \<…\>  
> our TomMarvoloRiddle object may be an item in a static array where we are using a monotonic\_buffer\_resource or could even be a singleton where we provide a no-op deleter. This pattern completely separates memory management of the concrete type from the APIs that vend references to its interfaces. Furthermore, ownership is transferred to the caller of make\_unique and the factory context does not maintain any access to the concrete type.

Do I understand correctly that the advantage of this approach is that it allows one to customize the lifecycle management of the allocated object (such as making it static or using some special allocation strategy)? I am trying to assess the implications of having or not having this ability. We do know that the operator delete is always provided with the true size of the concrete object, even if it is being deleted through a base pointer, so that addresses at least part of the problem.

> the Voldemort reference is contrived

Indeed it is. I think it is appropriate to call your pointer a `Horcrux`, to be more technically correct.

---

<div class="post-metadata">

**Author:** ![scottdixon](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/scottdixon/32/39_2.png) [@scottdixon](https://forum.opencyphal.org/u/scottdixon)\
**Post date:** [July 3, 2023, 6:04pm UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850/7 "2023-07-03T18:04:59Z")

</div>

> [@pavel.kirienko](#):
>
> Do I understand correctly that the advantage of this approach is that it allows one to customize the lifecycle management of the allocated object (such as making it static or using some special allocation strategy)?

Correct. These “Horcrux” pointers have a deleter that can abstract the “delete” action from the user of the pointer. The deleter can call delete, it can call deallocate on a PMR allocator, it can decrement the reference count on a shared buffer, or it could even set a bit in a hardware register to indicate that a given DMA buffer was no longer in use, etc. But this is a property one gets with any `unique_ptr`. The subtle difference in this pattern is that, like the Voldemort type, only the creator of the pointer is using the concrete type and once the pointer is created and ownership is transferred the owner of the pointer cannot know (i.e. cannot name) the concrete type. This concrete type does live on in the deleter but that too is opaque to the owner of the pointer. Finally, using the `friend` declaration, any attempt by the owner (or other objects the owner lends a raw pointer to) to delete this pointer will fail since only the deleter is allowed to use the protected destructor.

```auto
class SomeDumbObject final
{
public:
    SomeDumbObject(ICat* a_cat) : cat_{a_cat} {}
    ~SomeDumObject()
    {
        delete cat_; //< nope. ICat::~ICat is protected and SomeDumbObject is not
                     // a friend.
    }
private:
    ICat* cat_;
};

CatPointer tabby = make_cat(some_toy);
SomeDumbObject foo{tabby.get()}; //< nope (see above)

```

---

<div class="post-metadata">

**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\
**Post date:** [July 3, 2023, 6:15pm UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850/8 "2023-07-03T18:15:20Z")

</div>

That’s neat, indeed. Can we settle on some kind of plan regarding the design of the port factory methods? Are we generally okay with using Voldemort types, perhaps vended via your Horcrux pointers to allow for deep customization of the object lifecycle?

---

<div class="post-metadata">

**Author:** ![scottdixon](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/scottdixon/32/39_2.png) [@scottdixon](https://forum.opencyphal.org/u/scottdixon)\
**Post date:** [July 3, 2023, 7:44pm UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850/9 "2023-07-03T19:44:47Z")

</div>

As long as we agree that we need an `IComposite` to allow generic serialization and deserialization in libcyphal

---

<div class="post-metadata">

**Author:** ![pavel.kirienko](https://forum.opencyphal.org/user_avatar/forum.opencyphal.org/pavel.kirienko/32/9_2.png) [@pavel.kirienko](https://forum.opencyphal.org/u/pavel.kirienko)\
**Post date:** [July 3, 2023, 8:22pm UTC](https://forum.opencyphal.org/t/api-proposal-for-libcyphal/1850/10 "2023-07-03T20:22:06Z")

</div>

As long as the ability to use the concrete type of a DSDL object remains an option, I see no harm in that.