[Discourse.ros.org] [Quality Assurance] Pytest support for ROS / rostest

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Quality Assurance] Pytest support for ROS / rostest

Ian McMahon via ros-users


Hello everyone,

I want to stir some interest in supporting pytest for ROS. [pytest](https://docs.pytest.org/en/latest/) is a very popular unit testing framework for Python.

In my opinion, pytest is far easier to use than Python unittest and nosetests supported by ROS as of now. My personal experience says the easier something is to do, the likelier it is to be done. As we are talking about unit tests, an essential part of professional software development and QA, I think it's even more important here. I have outline more about why this is necessary in a [blog post.](https://machinekoder.com/testing-ros-powered-robots-pytest/)

The easiest way to support pytest within ROS is to use a new test runner that wraps pytest and extracts the JUnit XML output. Here is my attempt at doing that [ros_pytest](https://github.com/machinekoder/ros_pytest)

ROS seems to pick up the test data very well. However, I noticed that failing tests still don't trigger a failure when running rostests (but the log/terminal output shows the error). I would appreciate any input on why this is the case.

I joined the ROS community a few months ago and I want to make my first contribution to the project. So please let me know how if you think this is a good idea and how I can proceed.

Alex





---
[Visit Topic](https://discourse.ros.org/t/pytest-support-for-ros-rostest/4990/1) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Quality Assurance] Pytest support for ROS / rostest

Ian McMahon via ros-users


BTW: Congratulations on winning the Austrian Open Source Award 2017 with Machinekit :)





---
[Visit Topic](https://discourse.ros.org/t/pytest-support-for-ros-rostest/4990/2) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Quality Assurance] Pytest support for ROS / rostest

Ian McMahon via ros-users
In reply to this post by Ian McMahon via ros-users




Great to see an initiative like this in testing! A few questions after a glance:



In your [blog post](https://machinekoder.com/testing-ros-powered-robots-pytest/) you said:



> The really great thing about pytest is that it really makes unit testing easy and quick. To shorten you feedback loops I recommend you to clearly distinguish between unit tests and ROS node or integration tests.



After reading the blog, I didn't quite get how `pytest` makes  unit testing "easy and quick". I mean it looks like it does, but did you find it easier and quicker than `nosetests` that ROS is [already integrated with](http://docs.ros.org/api/catkin/html/howto/format2/python_nose_configuration.html)? I'm simply curious, as ROS allows to run unit testing by `nosetests` as well as integrate it into Node testing.



> ROS seems to pick up the test data very well. However, I noticed that failing tests still dont trigger a failure when running rostests (but the log/terminal output shows the error). I would appreciate any input on why this is the case.



Not sure what you mean by "ROS seems to pick up the test data very well". But I'd agree the test result is not the easiest to read. In your particular case, have you tried [`catkin_test_results`](http://docs.ros.org/api/catkin/html/howto/format2/run_tests.html)?











---

[Visit Topic](https://discourse.ros.org/t/pytest-support-for-ros-rostest/4990/3) or reply to this email to respond.







If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Quality Assurance] Pytest support for ROS / rostest

Ian McMahon via ros-users




> After reading the blog, I didnt quite get how pytest makes unit testing easy and quick. I mean it looks like it does, but did you find it easier and quicker than nosetests that ROS is already integrated with? Im simply curious, as ROS allows to run unit testing by nosetests as well as integrate it into Node testing.



I should have elaborated on this. I have only tried `nosetests` for testing parts of Machinekit. From what I know it just extends Python unittest, which is not great after all. `nosetests` also supports test functions (no need to write a complex test runner class), which is great. However, `pytest` seems to be a lot easier when it comes to writing and fixtures. Additionally, it makes great use of assert, including meaningful error messages. Writing pytests feels in my opinion just more natural and pythonic.



> Not sure what you mean by ROS seems to pick up the test data very well. But Id agree the test result is not the easiest to read. In your particular case, have you tried catkin_test_results?



    ros$ catkin_test_results

    build/ros_pytest/test_results/ros_pytest/rosunit-test_lib.xml: 2 tests, 0 errors, 1 failures, 0 skipped

    Summary: 3 tests, 0 errors, 1 failures, 0 skipped



Actually looks good. I just wondered why it does not "crash" or abort with an error message when the tests are run. But I'm not that familiar with ROS yet.











---

[Visit Topic](https://discourse.ros.org/t/pytest-support-for-ros-rostest/4990/4) or reply to this email to respond.







If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Quality Assurance] Pytest support for ROS / rostest

Ian McMahon via ros-users




[quote="machinekoder, post:4, topic:4990"]

```

ros$ catkin_test_results

build/ros_pytest/test_results/ros_pytest/rosunit-test_lib.xml: 2 tests, 0 errors, 1 failures, 0 skipped

Summary: 3 tests, 0 errors, 1 failures, 0 skipped

```



Actually looks good. I just wondered why it does not crash or abort with an error message when the tests are run. But Im not that familiar with ROS yet.

[/quote]





I'm afraid I'm not sure what you mean, but you don't want the whole chain of tests to stop in the middle just because of a single test case failed, do you? I agree that the output can be improved to be more intuitive though.



Apparently `PyTest` seems to be getting popular. It would be nice for it to be integrated into core library so that we can use it without defining special dependency etc (AFAIUC, your current approach requires a special test runner script). Would you interested in going further down this road (if so I'd definitely like to help)?











---

[Visit Topic](https://discourse.ros.org/t/pytest-support-for-ros-rostest/4990/5) or reply to this email to respond.







If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Quality Assurance] Pytest support for ROS / rostest

Ian McMahon via ros-users
In reply to this post by Ian McMahon via ros-users


I concur that `pytest` feels "easier" / more natural to use (obviously a subjective point). It certainly is more actively being developed and offers more features through the numerous available plugins. For ROS 2 we are using `pytest` over `nose` for a while for these reasons.

The new build tool [colcon](http://colcon.readthedocs.io) also invokes Python tests with `pytest`. Since `pytest` is capable of running tests written for `nose` this is usually compatible with exiting tests.

There are three parts to Python unit testing to consider:
* the API used to write unit tests
* the tool used to perform the testing of a package
* the tool used by `rostest` internally

I think the three parts are fairly independent from each other. E.g. using `colcon` you can run the tests of a ROS package with `pytest` even though they use `nose` / `unittest` API.





---
[Visit Topic](https://discourse.ros.org/t/pytest-support-for-ros-rostest/4990/6) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Quality Assurance] Pytest support for ROS / rostest

Ian McMahon via ros-users


[quote="dirk-thomas, post:6, topic:4990"]
It certainly is more actively being developed and offers more features through the numerous available plugins.
[/quote]

In the Python web dev domain pytest is the standard tool for testing distributed systems. If one wants to get an idea about in what non public projects it could be used I suggest to read about the history of the pytest initiator (Holger Krekel).

I would help out in adding pytest support in ROS1.





---
[Visit Topic](https://discourse.ros.org/t/pytest-support-for-ros-rostest/4990/7) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Quality Assurance] Pytest support for ROS / rostest

Ian McMahon via ros-users


The `ros_pytest` package is ready. I just created a PR for the Kinetic release: https://github.com/ros/rosdistro/pull/18443





---
[Visit Topic](https://discourse.ros.org/t/pytest-support-for-ros-rostest/4990/8) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Quality Assurance] Pytest support for ROS / rostest

Ian McMahon via ros-users


The PR has been merged. ros_pytest is now available as package!





---
[Visit Topic](https://discourse.ros.org/t/pytest-support-for-ros-rostest/4990/9) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>