[Discourse.ros.org] [ROS Projects] Feedback Wanted: Test framework for verifying topic output (both unit and integration testing)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [ROS Projects] Feedback Wanted: Test framework for verifying topic output (both unit and integration testing)

Tully Foote via ros-users


I've been working on a project that (I believe) simplifies testing nodes. I have a working proof of concept that permits one to simply supply inputs (currently, just messages but extensible) and verify outputs to expected values. I've also been experimenting with methods of supplying parametric rules (near mappings) between input and output for testing a full parameter space.

I'm looking for feedback, and in the future some testers, to see which direction I should grow this project.

The first and biggest pain point I see in my mind is defining tests. Rather than supplying a few test cases, I'd love to be able to just give the functional mapping of input to output and say "make sure the node follows this". However for even relatively few degrees of freedom, this becomes costly in time.

Take for example a node that filters a Twist message `cmd_vel`. Let's propose it does two things, first it clamps the message to a certain velocity magnitude. Then, given another message it subscribes to, it attenuates the acceleration





---
[Visit Topic](https://discourse.ros.org/t/feedback-wanted-test-framework-for-verifying-topic-output-both-unit-and-integration-testing/5336/1) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [ROS Projects] Feedback Wanted: Test framework for verifying topic output (both unit and integration testing)

Tully Foote via ros-users


It could probably be interesting for you to integrate with [https://github.com/boschresearch/ros_fmu]() "(... which provides) wrapping functional mockup units (FMUs) for co-simulation of physical models into ROS nodes." to generate input space...





---
[Visit Topic](https://discourse.ros.org/t/feedback-wanted-test-framework-for-verifying-topic-output-both-unit-and-integration-testing/5336/2) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [ROS Projects] Feedback Wanted: Test framework for verifying topic output (both unit and integration testing)

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


I like your ideas and think this could be very useful. However, one should keep in mind that writing a complete specification of the expected behavior of a node is as complex as writing the node itself; if you had a complete mapping from input to output, you could just execute it instead of running the node. So writing a full functional mapping of input to output is probably infeasible. In your example, you're doing something else: You specify invariants that should hold (velocity magnitude is clamped, and dv/dt corresponds with accel_max), while leaving open some degrees of freedom in the behavior of the node. I think this is the way to do it.

By the way, are you familiar with [rostest](http://wiki.ros.org/rostest)? An example where it's used are the  [tests for robot_pose_ekf](https://github.com/ros-planning/navigation/tree/kinetic-devel/robot_pose_ekf/test). Here they just test some simple properties of the node's output topics and check that the final estimated pose is correct. This is probably enough to catch 95% of regression bugs.





---
[Visit Topic](https://discourse.ros.org/t/feedback-wanted-test-framework-for-verifying-topic-output-both-unit-and-integration-testing/5336/3) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [ROS Projects] Feedback Wanted: Test framework for verifying topic output (both unit and integration testing)

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


This is quite interesting, thanks for this! I'll have to look into it further.





---
[Visit Topic](https://discourse.ros.org/t/feedback-wanted-test-framework-for-verifying-topic-output-both-unit-and-integration-testing/5336/4) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [ROS Projects] Feedback Wanted: Test framework for verifying topic output (both unit and integration testing)

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users




[quote="Martin_Guenther, post:3, topic:5336"]

I like your ideas and think this could be very useful. However, one should keep in mind that writing a complete specification of the expected behavior of a node is as complex as writing the node itself; if you had a complete mapping from input to output, you could just execute it instead of running the node. So writing a full functional mapping of input to output is probably infeasible. In your example, youre doing something else: You specify invariants that should hold (velocity magnitude is clamped, and dv/dt corresponds with accel_max), while leaving open some degrees of freedom in the behavior of the node. I think this is the way to do it.

[/quote]



This is a great way to put it. My concern is precisely how to express invariants generally enough such that development of the node (or nodes) isn't hindered by the tests. Cumbersome tests are likely to be ignored if they break and are hard to fix.



[quote="Martin_Guenther, post:3, topic:5336"]

By the way, are you familiar with [rostest ](http://wiki.ros.org/rostest)?

[/quote]



Yes, should have mentioned that this will ideally become a `rostest` node where pubs/subs/invariants are supplied via yaml.











---

[Visit Topic](https://discourse.ros.org/t/feedback-wanted-test-framework-for-verifying-topic-output-both-unit-and-integration-testing/5336/5) or reply to this email to respond.







If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>