The following text is an excerpt from a private message exchange where the origins of the protocol were discussed.
So back in the old days having electronics onboard a vehicle was weird. Eventually, vehicle designers realized that onboard electronics can do cool stuff if used properly. In order to be useful, an electronic device (actually, any machine or human, including VR systems) needs some means of assessing and/or influencing the world around it; hence, it needs to communicate with other units which are capable of doing so (such as sensors or actuators). At first, this problem was addressed in the most direct way possible by laying out wires between electronic blocks and their inferior devices (sensors/actuators, operator interface such as buttons/knobs/instruments). This picture comes from my childhood when I was helping my grandfather repair electrical equipment in his cars (Russian language):
The above is the wiring diagram of VAZ 21099: Lada Samara - Wikipedia. Those old Russian cars had spectacular service manuals and their electrics were always fascinating.
So then Germans realized that this mess of wires is getting out of hand and they made CAN bus. People who cared deeply about industrial automation realized that this is highly relevant so they jumped onboard too, although they also had stuff like RS-485. People from the airplane business arrived to a pretty much the same realization, but being restricted to different design constraints they came up with different solutions, such as ARINC 429. While implementations differed, their underlying philosophy was the same: replace many wires with few wires, keep everything else unchanged.
The CAN bus, RS-485, etc. are useless unless we can agree how to structure and interpret stuff, so then there were higher-level protocols such as CANopen or J1939 or MODBUS or whatever. These first vehicular and industrial communication protocols were designed by mechanical/electrical engineers who had a particular way of thinking: they were thinking in the mindset of mechanical engineering (quite what you would expect from mechanical engineers, I guess) and basic signals; what they came up with was essentially the same mess of wires but virtualized away from their eyes. Same applies to relatively newer protocols such as CANaerospace or Profibus, which despite their later appearance have actually failed to bring about any innovation.
Now, those were cars and airplanes and factories and stuff. In parallel to that, computer stuff was experiencing explosive growth, so explosive that nowadays having anything without a computer is weird. People started asking computers to do complex stuff with their vehicles, like driving around a city full of other cars and pedestrians, or flying their drone over the Alps. Complex stuff requires complex abstractions, yet the onboard communication business is stuck in the 80’s, being a virtualized mess of wires rather than a sophisticated high-level network that a software engineer would expect to have. At this point, relying on a mechanical engineer to build us an adequate onboard communication medium would be stupid, because our new objectives and requirements and goals have mostly shifted into the domain of state-of-the-art software engineering. Software engineers don’t think in terms of wires, they think in terms of abstract data entities representing complex real-world objects and processes. Say, they think about “vehicle dynamics” instead of “roll, pitch, yaw, climb rate, etc”; they think about “objects around the car” instead of “the radar return is such-and-such”.
So clearly there is some unmet demand here. There are people trying to use ROS in real-time settings (which is a bad idea) or define questionable standards like SOME/IP or abuse CAN bus to meet their needs. There seems to be a lack of a robust high-level communication solution that would be able to represent vehicular data streams in a sufficiently abstract way to make software engineers comfortable, and yet it should be simple enough to stay robust, deterministic, and suitable for safety-critical hard-realtime systems.