[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users


The paper [Maruyama, Yuya et al. Exploring the performance of ROS2. 2016 International Conference on Embedded Software (EMSOFT) (2016): 1-10.](https://www.semanticscholar.org/paper/Exploring-the-performance-of-ROS2-Maruyama-Kato/07b895f3b584dea4f64e91844f243de382026b20) evaluates the latency of ROS2 (different node configurations, different DDS implementations, different QoS policies, etc.).

Does someone know if these results have been updated by some more recent work(s)?





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/1) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users


We will release a plugin-based tool that will let you evaluate latency and memory resource while altering number of pubs/subs, QoS, RT settings, large and small message types, publishing frequency, security settings, inter and intra process communication, etc. We need 2-3 more weeks, stay tuned.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/2) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users


That's fantastic @Dejan_Pangercic, looking forward to have a look.

On our side, we recently released a first tech report (https://arxiv.org/abs/1804.07643) that's part of a series that hopefully will characterize better the latencies and throughput in ROS 2 while taking in consideration Real-Time aspects. This first tech report treats OSI layers 1 and 2.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/3) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


Hi all, we released the performance_test tool: https://github.com/ApexAI/performance_test to e.g. benchmark latency, jitter, lost samples, etc. in different DDS implementations.
Currently supported benchmarking is for communication mean over:
1. FastRTPS directly
1. ROS 2 rmw layer (and thus any supported rmw_* implementation)
1. Connext DDS Micro directly

We plan to extend it to use ROS1 comms as well.

Hope you find it useful, any feedaback is more than welcome.

D.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/4) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


Hello !

Nice tool @Dejan_Pangercic !

What are the requirements on the RMW layer to be tested ?
I have a [non-DDS-based RMW](https://github.com/astralien3000/rmw_ndn), but it currently only support simple pub/sub.
It would be interesting to measure performance with your tool and compare it to DDS implementations.

Also (for latency), is it only measuring the end-to-end latency, or can it be more comprehensive ? For example measuring the time spent in each layer (RCL/RMW/DDS/Network).





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/5) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


Hi @astralien3000,

I am the maintainer of the performance test tool.

If you already have RMW implementation which supports pub/sub you should be able to directly test your communication mean without any additional work. You just need to set the proper environment variable `RMW_IMPLEMENTATION=rmw_ndn` before starting the tool.

It can not messure latency based on application layers unfortunately, for this it would have to be invasive in all these layers.

But you can create a new communication plugin for the NDN transport as I did for FastRTPS here:  https://github.com/ApexAI/performance_test/blob/master/performance_test/src/communication_abstractions/fast_rtps_communicator.hpp.

This will allow you to compare only the communication frameworks performance and will also give you some insight over the overhead the various RMW layers introduce.

If you run into issues implementing the plugin I will be glad to support you.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/6) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


Thank you for your answer @andreaspasternak,

I was lacking some features (typesupport, proper management of multithreading, etc...), but in the end I was able to test my stack with your package. Also, I already implemented the invasive solution for measuring the latency of each layers. I will ask you if I need help, thank you again.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/7) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users


Hi @astralien3000,  
> Also, I already implemented the invasive solution for measuring the latency of each layers. I will ask you if I need help, thank you again.

Would you mind sharing some more details about your solution for measuring latency in each layer? Maybe we could even integrate it into the performance_test itself.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/8) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


I didn't do anything complex, but since it is an invasive way of measuring, I don't know if it is easy to integrate into the performance_test. [I only printed events with their timestamp on stdio](https://github.com/astralien3000/rmw_fastrtps/blob/phd_meas_latency/rmw_fastrtps_cpp/src/rmw_publish.cpp#L53), and then post processed with a python script. It is not very precise, but since I don't want the real latency (only being able to compare the two implementations), it is ok for me. To avoid the print extra cost during the experiment, you can register the events+timestamp in a (pre-allocated) table, and print everything at the end.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/9) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users


Very nice. I was thinking if someone could do a  ros message extension to add-in the latencies at each layers in a private area of the message at each layer (encode as a BLOB) and could be retrieved at the subscriber to measure-in.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/10) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users


For those interested, a new technical report studying this topic is available: https://arxiv.org/pdf/1808.10821.pdf.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/11) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users


Here's another update: https://arxiv.org/pdf/1809.02595.pdf

> **Towards a distributed and real-time framework for robots: evaluation of ROS 2.0 communications for real-time robotic applications**
>
> In this work we present an experimental setup to show the suitability of ROS 2.0 for real-time robotic applications. We disclose an evaluation of ROS 2.0 communications in a robotic inter-component (hardware) communication case on top of Linux. We benchmark and study the worst case latencies and missed deadlines to characterize ROS 2.0 communications for real-time applications. We demonstrate experimentally how computation and network congestion impacts the communication latencies and ultimately, propose a setup that, under certain conditions, mitigates these delays and obtains bounded traffic.

Compared to other results:
- All the measurements have been made in embedded devices.
- We measure latencies in a inter-component scenario. Given the lack of synchronization mechanisms (in this particular work we did not set them up), we use round-trip (ping-pong).
- Previous work focuses on the measurement of local latencies while we measure distributed ones.
- We measure how communications are affected in stressed conditions. This is the best way to show if the communication stack is well configured for real-time (which connects to previous work https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/11?u=vmayoral and https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/3?u=vmayoral).





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/12) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


Hello Victor, thanks a lot for your reports, I'm reading them right now and I came to a sentence that I don't understand. On https://arxiv.org/abs/1804.07643 on the 4th page you say:

> Additionally, as Ethernet is asynchronous, the high priority frames sharing the same link can content between them.

What does **content** mean in that context?

Sorry if it's a bit of a picky question.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/13) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users


Oops, later on I found the usage of **contend** which now makes sense to me. I guess that one was a typo :)





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/14) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users


Hey there @awesomebytes! It certainly sounds like a typo. Many thanks for reporting. Let us review it internally and report back if our mistakes go beyond that.

Cheers!





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/15) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users


An another one:

> **Time Synchronization in modular collaborative robots, _M-cobots_**
>
> A new generation of robot systems which are modular, flexible and safe for human-robot interaction are needed. Existing cobots seem to meet only the later and require a modular approach to improve their reconfigurability and interoperability. We propose a new sub-class of cobots named M-cobots which tackle these problems. In particular, we discuss the relevance of synchronization for these systems, analyze it and demonstrate how with a properly configured M-cobot, we are able to obtain a) distributed sub-microsecond clock synchronization accuracy among modules, b) timestamping accuracy of ROS 2.0 messages under 100 microseconds and c) millisecond-level end-to-end communication latencies, even when disturbed with networking overloads of up to 90 % of the network capacity.

Read the tech report at https://arxiv.org/pdf/1809.07295.pdf





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/16) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


Hi Victor,

Thank you for the reports related to latency using RT_PREMPT linux and ROS 2 with various network settings. It was very interesting to read.

Have couple of questions.

1) In the base RT_PREEMPT linux kernel report (https://arxiv.org/pdf/1808.10821.pdf), I understand Table-III and Table-IV is what matters. But, while looking at, Table-II (Roundtrip latency results with RT normal), was curious if you know what might be the reason for TX traffic at 100Mbps, the MAX latency is considerably high at 25ms? I would expect latency to be high when RX traffic at 100Mbps

2) In the ROS 2 evaluation report (https://arxiv.org/pdf/1809.02595.pdf), in Fig 5-a, when the system is idle, DDS2 has high MAX latency(4ms) compared to others. Was just curious which DDS implementation is this and what might be the reason?

3)  In the Fog 6-f of the ROS 2 evaluation report, at 80Mbps, where it cannot meet the deadlines and dropped packets, was curious, is ksoftirqd processing the packets the primary cause for the latency or can it be the DDS layer causing the latency. Also, about packets being dropped, would setting the size of kernel socket buffers ( net.core.rmem*, net.core.wmem*) would help too.

Thanks
Anup





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/17) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


@vmayoral see 3 questions above by folks from Apex.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/18) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


Hi @Anup_Pemmaiah, @Dejan_Pangercic  

Thanks for your feedback, I will try to clarify some of your doubts:

1. Both TX and RX paths are suffering from the context change to the ksoftirqd threads but in a different way. In the transmission path both streams are going trough the same Qdisc queue. When there are packets pending to be transmitted in the Qdisc queue they are sent from the ksoftirqd context. At some point the fair scheduler decides that the ksoftirqd thread has consumed enough CPU and it is preempted. During this time, packets are accumulated and we observe high latencies in the order of milliseconds. For 100 Mbps it looks like the RX path, packet are processed more efficiently. This is probably because the Ksoftirqd context is not triggered all the time and part of these packets are processed in the Ethernet IRQ thread which has real-time priority. However, when we increased the network load of the concurrent traffic (>200Mbps) we observed also high latencies even in the RT normal case.

2. For fig 5a and 5b we were using the default configuration of each DDS. In the case of that DDS the default configuration might not be optimized for low bounded latencies but for other purposes. However, when for the real-time settings (fig 5c and 5d) we customized the configuration of that DDS and the problem was solved.

3. In this case we had 80 Mbps non-ROS 2.0 concurrent traffic with the ROS 2.0 round-trip traffic. As there is no contention in the DDS layers the problem was very likely caused in the kernel level. Posterior analysis tracing the kernel confirmed our suspects. Changing the socket queues may prevent packet drop but would not solve the root of the problem which is going to cause latency. The real problem is caused by how the net processing is deferred to ksoftirqd context. For the moment we can only mitigate these problems and expect this is solved in the new kernel releases.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/19) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Latency and throughput in ROS2

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


My colleague @carlossv just answered @Dejan_Pangercic. Ping me if you guys are around IROS and would like to discuss this face to face.





---
[Visit Topic](https://discourse.ros.org/t/latency-and-throughput-in-ros2/4367/20) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
12