You seem to be correct at guessing that your issues are caused by the TX queue overflow. If lowering the publishing rate is not an option, consider increasing the size of the transmission queue (by allocating more memory for the library) and also make sure that your outgoing transfers are not being preempted by higher-priority frames on the bus (adjust priorities if necessary).
Oh. If you are on Linux (for some reason I assumed that you are running that on an embedded platform), the queue size is probably not an issue, because it is sized by default to be 512 kibibyte large:
You can still increase it by instantiating your node manually instead of using the helpers, but it is unlikely to help. I am now suspecting that the culprit is your SocketCAN backend driver. Do I recall correctly that some months ago you were asking on this forum about the SocketCAN driver getting stuck because of malfunctioning frame loopback logic? How did you go about fixing that? If you simply disabled the loopback checks, then I am inclined to suspect that your modified SocketCAN driver simply overwhelms the underlying driver with too many frames; i.e., the problem lies not within libuavcan but rather within SocketCAN.
Okay, what if your underlying SocketCAN driver returns your loopback frames before the corresponding outgoing frames are actually emitted to the bus? If that were the case, libuavcan would likely unload its TX queue faster than the controller can emit frames, leading to the behavior you are observing with frame losses. Do you think that might be the case? Can you inspect the driver code or perhaps conduct a test to establish whether the loopback return actually waits for the completion of the corresponding transmission?