[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users


Hi,

I am start looking at the source code of ROS2 to understand how message are serialized.
My goal is to update [ros_type_introspection](https://github.com/facontidavide/ros_type_introspection) to ROS2.

I GUESS that serialization details are defined by the middleware (OpenSplice, eProsilica, etc), aren't them?

Is there anything equivalent to [topic_tools:ShapeShifter](http://docs.ros.org/diamondback/api/topic_tools/html/classtopic__tools_1_1ShapeShifter.html).? Is it even possible to have something that works similarly?

Is there any way in ROS2 to introspect the content of a message at run-time?

Cheers

Davide





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/1) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users


No, unfortunately we don't have that yet. @karsten was working on it for this release, but we ran into some issues figuring out how to most efficiently get data in and out of the underlying DDS vendors without (de)serializing. The API we'll provide will be relatively simple, something like `rmw_publish_raw` and `rmw_take_raw` and some wrapper like `ShapeShifter` in our C++ API, but it's the "how" that's held us up.

But we're still looking at this issue and it's in our critical path as we need it to implement rosbag and to implement "type masquerading".

I'll try to remember to reply here when we get the first version of that new API figured out.

----

The other thing I suppose you'll need is a way to parse the "ROS IDL" files, e.g. `.msg`? We don't currently have that in anything but Python. I see that you have one for ROS 1 in C++:

https://github.com/facontidavide/ros_type_introspection/blob/master/src/ros_introspection.cpp

As part of your package, which is awesome :smile:, but it would likely need to be updated to support the new syntax we support in ROS 2 (see http://design.ros2.org/articles/interface_definition.html, specifically the part about bounded strings and arrays).

Also, it would be awesome to have a separate library for parsing the various interface files in C/C++, so it could be reused everywhere.

There's also the possibility to have another, more machine friendly, version of the interfaces. For example, as part of generation at build time we could generate an xml or json file which represents the contents of the corresponding `.msg` file. Then you could lookup and load that file instead (with an xml or json library), which would save you from needing to writing parsing code, and instead you would only need to interpret the contents correctly.

----

This is a feature I'd really like to see well supported in both ROS 1 and ROS 2, it would be especially helpful in making the bridge between the two more convenient to use with non-standard types. So, once we get you unblocked by providing something like ShapeShifter, I'd be really interested in helping you with anything you're missing to implement it.





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/2) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users
In reply to this post by Ahmed Ali via ros-users


Awesome, I am looking forward for it.

The "machine friendly" format you mentioned would (of course) be VERY welcome but not strictly necessary, since I have done already most of the heavy lifting in terms of parsing.

One quick question. My runtime deserialization was created doing a "reverse engineering" of how ROS1 messages are created.
In DDS is the serialization protocol standardized or middleware dependent? Since DDS is known to have good interoperability, I am tempted to think that the former is true...





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/3) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users
In reply to this post by Ahmed Ali via ros-users


The DDS serialization protocol is indeed standardized. It uses a format called Extended Common Data Representation (XCDR). The most up-to-date reference is the DDS-XTYPES specification version 1.2. See http://www.omg.org/spec/DDS-XTypes/





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/4) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users
In reply to this post by Ahmed Ali via ros-users


Well, we're going to need to provide a function which can serialize and deserialize a `unint8_t[]` given an output structure. For example, with `sensor_msgs/Image.msg` it might be like:

```c++
unint8_t[] buffer = somehow_get_serialized_message_data();
sensor_msgs::msg::Image my_image;
deserialize_message<sensor_msgs::msg::Image>(buffer, my_image);
```

The implementation of this imagined `deserialize_message` function may end up calling an implementation specific function, or at the very least how it is deserialized should be an implementation detail.

But of course what you want is more like:

```c++
std::string sensor_msgs_image_definition = somehow_get_message_definition();
unint8_t[] buffer = somehow_get_serialized_message_data();
// some, as yet, imagined class...
auto message_object = MessageTypeIntrospector(sensor_msgs_image_definition, buffer);
assert(message_object.has_field("height") && message_object.field_is<uint32>("height"));
uint32  height = message_object.get_field<uint32>("height");
```

We have the beginnings of this in https://github.com/ros2/rosidl/tree/master/rosidl_typesupport_introspection_cpp, but I'm 100% certain how much is still needed to support this at the user level. We might not be able to use any of that package at the user level, since it isn't always used in the implementation, but it is at least representing the structure of a message as C++ structs and objects which is part of what we'd need to do. My initial guess is that we would need to have a way to generate the same typesupport structures but given an input string or structure which describes the type.

So I'd actually say we should try to avoid your package needing to reverse engineer how to deserialize, instead that should be provided by our abstraction layer. This allows us to protect the stuff that comes on top from underlying changes, e.g. if we changed from one version of CDR (the serialization standard used in DDS and ROS 2) to another.





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/5) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users
In reply to this post by Ahmed Ali via ros-users


Just two side notes:

- The roadmap currently contains a task to revisit the message definition format. The `.msg` format currently used with some minor extensions from ROS 1 might be replaced with something more powerful since the format can't handle several of the pending feature requests. Just to be consider when spending effort on implementing the current format in different languages.

- While the serialization protocol of the DDS implementations follows a single standard you can't rely on the rmw impl. to be DDS. There has e.g. be efforts to implement the rmw interface with other middlewares like OPC UA.





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/6) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users
In reply to this post by Ahmed Ali via ros-users


Summarizing:

1) Type erased messages are needed for applications such as rosbag, rqt_plot, PlotJuggler, MATLAB importers, etc. Any generic topic subscriber indeed.
2) ShapeShifter had all the informations needed to do this: raw bytes buffer, Message definition and MD5Sum.
3) ROS1 had just one protocol, whilst ROS2 can potentially have DDS or others.

THINGS WE MIGHT DO:

A) I guess that the first step is to add to ROSIDL the equivalent of __ros::message_traits::Definition< Type >::value()__. Preferably expressed as JSON or XML format.

B) It must be possible to subscribe to a topic in a "generic" way, i.e. bypassing the deserialization step and accessign the raw bytes of the message.

C) To implement a run-time deserializer, I don't see any solution but implementing a different code for each protocol :frowning: No magic wands.





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/7) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users
In reply to this post by Ahmed Ali via ros-users


[quote="facontidavide, post:7, topic:3393"]
A) I guess that the first step is to add to ROSIDL the equivalent of ros::message_traits::Definition&lt; Type &gt;::value(). Preferably expressed as JSON or XML format.
[/quote]

I assume the value should contain the message description in a readable format. Why should that be provided through a language specific symbol? I would suggest to provide this information through the ament resource index which would make it available across languages.





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/8) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users
In reply to this post by Ahmed Ali via ros-users


what I meant is that ros::message_traits::Definition< Type >::value() return a __const char*__

But this static C string contains the information in a easy to parse format that any language can easily parse.

Something like this https://gist.github.com/facontidavide/4ade0fb2d3cd48a64eb47c5c1a927de6





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/9) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users
In reply to this post by Ahmed Ali via ros-users


[quote="facontidavide, post:9, topic:3393"]
what I meant is that ros::message_traits::Definition&lt; Type &gt;::value() return a const char*
[/quote]

I think exposing this information (even a preprocessed version to actually store e.g. yaml) through the ament resource index would be better in order to make it available to any language.





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/10) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users
In reply to this post by Ahmed Ali via ros-users


ok, I got your point.





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/11) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] ROS2: anything equivalent to topic_tools:ShapeShifter?

Ahmed Ali via ros-users
In reply to this post by Ahmed Ali via ros-users


Well, it seems that at least for eProsima, there are some open source pieces of software that can help me with the this task. But it will take me time to dig into the source code





---
[Visit Topic](https://discourse.ros.org/t/ros2-anything-equivalent-to-topic-tools-shapeshifter/3393/12) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>