[Discourse.ros.org] [Next Generation ROS] Design By Contract

classic Classic list List threaded Threaded
38 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users


[quote="asmodehn, post:20, topic:2405"]
Dont build a distributed (==multiprocess) system if you dont have to. Programming language elements (functions, classes, libraries, packages) are made for composing correctly in all sorts of ways, and there is usually theoretical background, tooling, conventions, processes, to help you satisfy the cognitive biases you didnt know you had. No distributed software system that allows you to control the distribution graph, has anything equivalent to that currently. ROS is no exception (actually erlang might be the only exception).
Example : A whole part of Operating System design is to prevent processes interactions, and most recent OSes are preempting ? This is opposed to the features suitable for a distributed system, which by definition needs process cooperation, and where controlling when each process can be interrupted, or not, is really useful. In one process, in one language, all these problems vanish.
[/quote]

ROS is a multi process system on the robot/(multi-)processor level itself. I guess you mean multi device system instead of multiprocess system. Right, distributed systems are beyond robotics. However robotics and distributed systems have already merged into distributed robotics ([Kiva robots in an Amazon warehouse in 2011](https://www.youtube.com/watch?v=6KRjuuEVEZs)). Isn't it the time to ease the development of such systems (and single robots) by providing better framework capabilities?

As Amazon already did non real time distributed robotics in 2011 I think considerations w.r.t. preemption and inter-process interaction beyond the robot level is more critical in real-time (soft/firm/hard) applications.

[quote="asmodehn, post:20, topic:2405"]
Regarding ROS, the best bet is likely to integrate/interface/implement ROS with the existing programming language that provide the feature that you need, instead of trying to integrate that awesome language feature into ROS (because it implies re-implementation and proactive maintenance from ROS community for something that is not purely robotics related)
[/quote]

I would argue that if a framework implementation lacks conceptual features this cannot be compensated with the choice of suitable programming languages which address lower levels of abstractions only. But you are right, it is better to improve a framework w.r.t. to it's given capabilities instead of trying to integrate language features (at least in the short term).





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/21) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


Instead of waiting for more capabilities in ROS2 it is probably more valuable to add more capabilities to the current ROS1 framework. Without DbC on the ROS node level one can verify the ROS node interface with rostest. Currently the set of reusable test nodes is limited. What about adding more generic test nodes like [fake topic publishers](https://github.com/fkromer/ros_comm/tree/faketopicpublisher) (draft state) to `ros_comm`? (The example tests can be run with `catkin_make run_tests_rostest_rostest_test_faketopicpublisher.test` and `catkin_make run_tests_rostest_rostest_test_faketopicpublisher0.test` in the catkin workspace.)





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/22) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="fkromer, post:21, topic:2405"]
ROS is a multi process system on the robot/(multi-)processor level itself. I guess you mean multi device system instead of multiprocess system
[/quote]

Actually I do mean multiprocess. But I do not mean "We cannot make it work/do what we want" or even "We cannot make it do what we want all the time". I mean "We cannot be sure that it will never do [things](https://www.engadget.com/2017/07/17/dc-security-robot-throws-itself-into-pool/) that weren't intended".
And I know for a fact that there are more robots out in the world, doing unexpected dangerous thing, because they weren't programmed with total safety in mind, than most people know about. All it takes is an unchecked integer to wrap around, and disaster strikes in the real world. Make it distributed, and the disaster likeliness increases exponentially.

So sure we can build robots, and distributed systems, but, when it come to a robot that can poke the eye of a child because it looks like it's a button to press, it's different than a harmless backend database cluster in the basement, so you better be sure of what you're programming... and for distributed systems (multiprocess) the theory is quite new, so most language/frameworks won't help you there.

[quote="fkromer, post:21, topic:2405"]
Isnt it the time to ease the development of such systems (and single robots) by providing better framework capabilities?
[/quote]

Definitely yes, but instead of adding potentially heavy features, without being sure they will be used and maintained, I would first focus on doing like https://jepsen.io/, that is, provide tools that show people working in robotics, what and where the problems are in the system they build. Actually probably doing the same as what works for security hackers : tell people their system is broken/unsafe, nobody cares. Make anyone (including their customer) able to [break it](https://usbkill.com/), and then they react... and some might listen.

[quote="fkromer, post:22, topic:2405"]
Instead of waiting for more capabilities in ROS2 it is probably more valuable to add more capabilities to the current ROS1 framework.
[/quote]

I personally fully agree with this statement, but I _think_ most people are focusing on ROS2 these days, which means even less maintenance resource for ROS1, so we need to be careful that what we add is really worth it.

[quote="fkromer, post:22, topic:2405"]
Currently the set of reusable test nodes is limited. What about adding more generic test nodes
[/quote]

I would also agree there. You can always send a PR to add the tests node you miss to rostest, and discuss it with the maintainers :-)
And you can also write a package for the specific nodes you need. I started doing that for my own needs in https://github.com/pyros-dev/pyros-test.
But these days I am thinking we need something more like a ROS [Simian Army](https://github.com/Netflix/SimianArmy) :

- some package that randomly kill and restart nodes, probably based on launch files...
- some package that randomly sends messages around, like a ros-[hypothesis](http://hypothesis.readthedocs.io/) that would generate any valid message based on a ROS definition, to test your nodes against. I have already implemented most of this one, as part of other projects, but I still need to make it a package on its own, whenever I get the time and motivation...
- probably a few more...





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/23) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="asmodehn, post:23, topic:2405"]
Actually I do mean multiprocess. But I do not mean We cannot make it work/do what we want or even We cannot make it do what we want all the time. I mean We cannot be sure that it will never do things that werent intended.

And I know for a fact that there are more robots out in the world, doing unexpected dangerous thing, because they werent programmed with total safety in mind, than most people know about. All it takes is an unchecked integer to wrap around, and disaster strikes in the real world. Make it distributed, and the disaster likeliness increases exponentially.

So sure we can build robots, and distributed systems, but, when it come to a robot that can poke the eye of a child because it looks like its a button to press, its different than a harmless backend database cluster in the basement, so you better be sure of what youre programming and for distributed systems (multiprocess) the theory is quite new, so most language/frameworks wont help you there.
[/quote]

I cannot visualize a possible bad case situation better than you did (I have to remember that one for the future.) However even if you are working in an environment where your work could potentially lead to disastrous situations you should change your mind set from "prevent from/find every bug" (which could lead to something like an displaced child's eye for sure, but what cannot be prevented from with 100% probability for sure as well) to "become better in preventing from/finding the most important bugs"... better than suffering a depression.

[quote="asmodehn, post:23, topic:2405"]
Definitely yes, but instead of adding potentially heavy features, without being sure they will be used and maintained, I would first focus on doing like https://jepsen.io/, that is, provide tools that show people working in robotics, what and where the problems are in the system they build.
[/quote]

Thanks for that hint.

[quote="asmodehn, post:23, topic:2405"]
Make anyone (including their customer) able to break it, and then they react and some might listen.
[/quote]

Good to not provide an USB port...

[quote="asmodehn, post:23, topic:2405"]
I would also agree there. You can always send a PR to add the tests node you miss to rostest, and discuss it with the maintainers :slight_smile:
And you can also write a package for the specific nodes you need. I started doing that for my own needs in https://github.com/pyros-dev/pyros-test1.
[/quote]

I am not going to PR into ROS1 ;) Right now I am fine with a fork of ros_comm/rostest for dummy, fake, spy and mock nodes. (In case I consider your package as template.)

[quote="asmodehn, post:23, topic:2405"]
But these days I am thinking we need something more like a ROS Simian Army :

some package that randomly kill and restart nodes, probably based on launch files
some package that randomly sends messages around, like a ros-hypothesis that would generate any valid message based on a ROS definition, to test your nodes against. I have already implemented most of this one, as part of other projects, but I still need to make it a package on its own, whenever I get the time and motivation
probably a few more
[/quote]

Having something like that would be great.





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/24) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="asmodehn, post:23, topic:2405"]
But these days I am thinking we need something more like a ROS Simian Army :
[/quote]

I began a new ROS package [roschaos](https://github.com/fkromer/roschaos). The package is in an early stage but it can already be used to kill local ROS node processes randomly using a command line interface. To get feedback and proposals for improvement right from the beginning I decided to make the project public already in this early stage. However there are a lot of features missing (refer to issues). Feel free to contribute to get more features implemented.

BTW: Thanks @gavanderhoorn for your [answers on answers.ros.org like this one](https://answers.ros.org/question/271776/how-can-i-retrieve-a-list-of-process-ids-of-ros-nodes/?answer=271783#post-id-271783) which helped a lot to get started.





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/25) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="asmodehn, post:23, topic:2405"]
And you can also write a package for the specific nodes you need. I started doing that for my own needs in https://github.com/pyros-dev/pyros-test.
[/quote]

I plan to put generic test nodes which act as *dummy* or *fake* nodes (according to the [general terminology of test doubles in software engineering](https://en.wikipedia.org/wiki/Test_double#Types_of_test_doubles)) into a package **rosfake**.

(It makes sense to put *spy* and *mock* nodes into a package **rosmock**. However to get something like that generic is not easy because e.g. the package depends on the test framework used to assert.)

What license do you use for [pyros-test](https://github.com/pyros-dev/pyros-test)?





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/26) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


Quick replies :
- pyros-test is MIT or BSD, along these lines, I just haven't taken the time to put a file there... I ll try to do it soon.
- python makes things simpler than C++ as there are defacto standard test frameworks. So I am trying to support whatever basic python and ROS support ( unittest, doctest, and nose ) and pytest. unittest already includes a mock library by the way in python3, which is just the mock library in python2 .
- pyros-test is currently very very simple (probably too simple to need a separate package), and I'd like to eventually improve it when I get the chance...

But IMHO you re probably better of extracting what I already started in https://github.com/pyros-dev/pyros-msgs and https://github.com/pyros-dev/pyros-schemas : have a look at property based testing and [hypothesis](http://hypothesis.works/), it should be simple enough to automatically generate fake nodes based on an existing message definition and then send fake messages around :-) .

Some example of hypothesis use here : https://github.com/pyros-dev/pyros-schemas/blob/nested_merged/tests/test_pyros_schemas/hypothesis_example.py and an example of generating messages with it there https://github.com/pyros-dev/pyros-schemas/blob/nested_merged/tests/test_pyros_schemas/test_basic_fields.py

By the way, roschaos looks fun, and there is probably a clever way to integrate it with roslaunch or feed it launch/test files... I ll play with it when I get some time.





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/27) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="asmodehn, post:23, topic:2405"]
But these days I am thinking we need something more like a ROS Simian Army :

* some package that randomly kill and restart nodes, probably based on launch files
* some package that randomly sends messages around, like a ros-hypothesis that would generate any valid message based on a ROS definition, to test your nodes against. I have already implemented most of this one, as part of other projects, but I still need to make it a package on its own, whenever I get the time and motivation
* probably a few more
[/quote]

Wow, a ros-hypotesis could be very powerful and effective. I knew about property based testing (e.g. [RapidCheck](https://github.com/emil-e/rapidcheck) for C++) but did not think about to adapt it to ROS so far. Creating a framework for property based ROS node testing could be a hard task I guess. What use cases are you thinking about exactly? I would love to contribute to it :blush:





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/28) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


While I'm a big fan of automatic checking, I wonder which problem(s) this proposal tries to solve.

This is not to say there are no problems. I just think it would help the discussion a lot if we knew what people here are  interested in, in terms of outward behavior of system.

The proposed contracts relate to a) rates and b) response times. From my own work, I know that rate information is necessary but not sufficient for determining whether a system can have sampling effects. I am also strongly of the opinion that rates should be a property of a *system*, not a component.

I don't know of much utility, but a great deal of problems, in specifying response times for a distributed system, particularly one with one very little real-time support, such as ROS or ROS2. If you really care about that, put your stuff into one process and ensure response times in the usual means, which have little do to with ROS.





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/29) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="iluetkeb, post:29, topic:2405"]
While Im a big fan of automatic checking, I wonder which problem(s) this proposal tries to solve.

This is not to say there are no problems. I just think it would help the discussion a lot if we knew what people here are  interested in, in terms of outward behavior of system.
[/quote]

Initially my intention to start this thread was to discuss if and how formal specification and formal verification of ROS 2 node/nodelet interfaces by means of DbC could be applied to/implemented in ROS 2. I am interested in DbC because it has the big advantage that it could speed up the integration of ROS 2 nodes/nodelets within a system because it prevents from struggling with bugs which relation to the interface based interaction between nodes/nodelets. However I do not consider DbC as measure for formal verification in the testing context but in the debugging context instead because it can hardly be accurate enough w.r.t. timing, as you said, especially in real-time systems. (I do not know about any tracing tools for ROS which are usual tools for real-time related verification.) in the debugging context a comparably rough estimate is often sufficient. Assuming "system" means the overall sum of ROS nodes/nodelets this thread is not about "outward behavior of system" but it
 s "internal" integration only. If your system exposes interfaces in terms of ROS interfaces which interacts with another system DbC could address "outward behaviour" of the single sub-systems.

As DbC would be hard to implement in ROS 2 and it's benefits are not considered relevant enough in comparison to other quality improving measures like developing and using tools like a "ROS Simian Army" the direction of the thread turned into the direction of what tools are missing and could be helpful to verify a ROS based system. (Not in terms of verifying real-time behaviour.)

[quote="iluetkeb, post:29, topic:2405"]
The proposed contracts relate to a) rates and b) response times. From my own work, I know that rate information is necessary but not sufficient for determining whether a system can have sampling effects. I am also strongly of the opinion that rates should be a property of a system, not a component.
[/quote]

For me the proposed contracts are more about valid and invalid values/value ranges of the node/nodelet interfacess like topics.

However w.r.t. rates and response times I would consider different "classes":
1) the "incoming" rates one node/nodelet expect from other nodes to receive, the nodes/nodelets "outgoing" rates which are expected from other nodes, the response time of one node
2) the rates and response time of a component
2) the rates and response time of a system

If 1) the node/nodelets do not satisfy "rough" rate or response time requirements there is a pretty good chance that 2) the component or 3) the system will not behave like you would like it to behave as well.

[quote="iluetkeb, post:29, topic:2405"]
I dont know of much utility, but a great deal of problems, in specifying response times for a distributed system, particularly one with one very little real-time support, such as ROS or ROS2. If you really care about that, put your stuff into one process and ensure response times in the usual means, which have little do to with ROS.
[/quote]

If we care about rates and response times we are already putting nodelets into a single process if this is possible. (If nodes are distributed over different machines I do not know about any way to improve rates and response times anyway.)





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/30) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="fkromer, post:30, topic:2405, full:true"]
Initially my intention to start this thread was to discuss if and how formal specification and formal verification of ROS 2 node/nodelet interfaces by means of DbC could be applied to/implemented in ROS 2. I am interested in DbC because it has the big advantage that it could speed up the integration of ROS 2 nodes/nodelets within a system because it prevents from struggling with bugs which relation to the interface based interaction between nodes/nodelets.
[/quote]

Can you give one or more examples of the kind of interface bugs you mean?

In my experience, the most frequent issue -- at least initially -- with connecting nodes is that something is not connected, because of a wrong name. The next most frequent seems to be spelling and/or range issues in parameters.

I wouldn't call these interface issues as such, or at least we don't need *new* specs on that, it would be sufficient to actually check the current ones.

[quote="fkromer, post:30, topic:2405, full:true"]
(I do not know about any tracing tools for ROS which are usual tools for real-time related verification.)
[/quote]

Have you seen my talk at ROSCon? It's not specifically about that, but I mention how we used LTTng to trace messages and callback invocations. This is in the soon-to-be-released tracetools package.

[quote="fkromer, post:30, topic:2405, full:true"]
However w.r.t. rates and response times I would consider different "classes":
1) the "incoming" rates one node/nodelet expect from other nodes to receive, the nodes/nodelets "outgoing" rates which are expected from other nodes, the response time of one node
2) the rates and response time of a component
2) the rates and response time of a system

If 1) the node/nodelets do not satisfy "rough" rate or response time requirements there is a pretty good chance that 2) the component or 3) the system will not behave like you would like it to behave as well.
[/quote]

Aren't those specs strongly related to your requirements or, in other words, to the outward behavior your system is expected to have? For example, when I drive a 1m/s in a warehouse, I might have different requirements regarding pose update rate then when I drive 50km/h through city traffic.

[quote="fkromer, post:30, topic:2405, full:true"]
[quote="iluetkeb, post:29, topic:2405"]
"I am also strongly of the opinion that rates should be a property of a system, not a component."
[/quote]
People having a background in safety critical, real-time embedded system development could disagree here.
[/quote]

Hmm, most of what I know about embedded systems comes from the automotive people here at Bosch, and they are very concerned with safety. I might have got it wrong, of course, but given how often we've internally talked about this, I would be surprised.

What is true is that rates are often *specified* on a task level, with callbacks being assigned to a task according to their rate needs, but that is an implementational aspect, which most people actually don't like, but accept as the way things are, unfortunately, implemented.

Anyway, in most cases, they are not interested in rates at all, only in *response time*, and that's again a system-level aspect.





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/31) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


The response time of the system can be broken down into the response time of each stage of the processing pipeline's longest path. Thus you need to know what the response time patterns of the nodes in that path look like. If we are dealing with a hard real-time system, then it should be possible to define, for a given execution environment, what response time requirement the node is capable of satisfying, which would be useful information for a system integrator. But it seems to me that this is a very specific requirement, as it would be tied completely to your specific environment, including the other nodes and how *they* execute (e,g, does one of them tend to hog the CPU a little?). So after @iluetkeb's talk and speaking to him in person, I think that he's right about the need for tools that make it really easy to understand these sorts of properties of nodes when they are installed in your system, rather than a way to specify something like a maximum update rate capability
  as part of a node's interface. That seems like a figure that would change too much between execution environments based on things as obvious as CPU speed and as esoteric as the structure of the disc controller.





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/32) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="iluetkeb, post:31, topic:2405"]
Can you give one or more examples of the kind of interface bugs you mean?
[/quote]

One example: You integrate 2 nodes ("a" and "b") which have not been tested before (or if so not well enough). The first node "a" is publishing a topic and the second one "b" subscribes to it and publishes an own topic as well. You know that the subscribing nodes ("b") topic message values may never be outside a valid value range. During integration something goes wrong and you locate the root cause in wrong topic message values published of node "b". You do not know why exactly the topic message values are invalid. It could relate to invalid topic message values which "b" received from "a", or in a wrong implementation of "b" itself which does not prevent from publishing invalid values. Having a DbC mechanism violations of such exceptions would be notified about during node integration even before integration issues could be discovered at all. (It would be possible that integration issues occur just in rare cases and the issue could keep undetected during integration and pop
  up in the field the first time.)

[quote="iluetkeb, post:31, topic:2405"]
In my experience, the most frequent issue  at least initially  with connecting nodes is that something is not connected, because of a wrong name. The next most frequent seems to be spelling and/or range issues in parameters.
[/quote]

As far as I know the only way to detect unconnected nodes which should be connected is to use `rqt_graph`. (Mismatched topic types are even harder to detect and require to look for missing `Connections:` when using `rosnode list <node>`.) Does anyone know how to check issues like that in an automated fashion?

Aren't rostests with [paramtest](http://wiki.ros.org/rostest/Nodes#paramtest) test nodes the candidate to prevent from range issues in parameters?

[quote="iluetkeb, post:31, topic:2405"]
I wouldnt call these interface issues as such, or at least we dont need new specs on that, it would be sufficient to actually check the current ones.
[/quote]

Right. (It was never about interface issues in terms of the ROS implementation just in terms of its usage.)

[quote="iluetkeb, post:31, topic:2405"]
Have you seen my talk at ROSCon? Its not specifically about that, but I mention how we used LTTng to trace messages and callback invocations. This is in the soon-to-be-released tracetools package.
[/quote]

Not yet. I was looking for exactly something like LTTng. Will there be open sourced tools in addition to LTTng?

[quote="iluetkeb, post:31, topic:2405"]
Arent those specs strongly related to your requirements or, in other words, to the outward behavior your system is expected to have? For example, when I drive a 1m/s in a warehouse, I might have different requirements regarding pose update rate then when I drive 50km/h through city traffic.
[/quote]

If the same node is used in different applications and/or environments the requirements that's true abd could be classified via use cases (here: warehouse, car) and the checks parametrized according to that. (Application specific: A warehouse robot should never drive faster than 1m/s. Environment specific: Every of your self-driving cars should not drive faster than 50km/h in the city and faster than 200km/h on the autobahn.)

[quote="iluetkeb, post:31, topic:2405"]
Hmm, most of what I know about embedded systems comes from the automotive people here at Bosch, and they are very concerned with safety. I might have got it wrong, of course, but given how often weve internally talked about this, I would be surprised.

What is true is that rates are often specified on a task level, with callbacks being assigned to a task according to their rate needs, but that is an implementational aspect, which most people actually dont like, but accept as the way things are, unfortunately, implemented.

Anyway, in most cases, they are not interested in rates at all, only in response time, and thats again a system-level aspect.
[/quote]

I do not know about the automotive sector at all but the industrial automation sector (hard real-time safety critical, up to IEC61508 SIL3) only. However I know for sure that people use trace tools on the RTOS level like [Trace for FreeRTOS/SafeRTOS](https://www.highintegritysystems.com/tools/safertos-trace/) and on the C function level like [microtrace](http://www.lauterbach.com/frames.html?microtrace.html) for function runtime and response time analysis. However it's possible that this is just done for cases when the system's overall response time cannot be determined using measurements on the system level (e.g. error case handling which could be hard to force on the system level for some cases).





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/33) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="fkromer, post:33, topic:2405"]
You know that the subscribing nodes (b) topic message values may never be outside a valid value range
[/quote]

Okay, range specs would certainly be interesting.

I also once had a case with parameters, where the *combination* of two parameters resulted in an out-of-range condition.

[quote="fkromer, post:33, topic:2405"]
As far as I know the only way to detect unconnected nodes which should be connected is to use rqt_graph. (Mismatched topic types are even harder to detect and require to look for missing Connections: when using rosnode list &lt;node&gt;.) Does anyone know how to check issues like that in an automated fashion?
[/quote]

You can use "roswtf" during runtime, it will report subscriptions that have no publisher, and also type mismatches.

The thing is, you never know whether a subscription might be optional, and there are also cases where either one or the other subscription can be used, but not both.

Again, I agree this would be useful to check. We've so far called such things "graph consistency checks".

[quote="fkromer, post:33, topic:2405"]
Not yet. I was looking for exactly something like LTTng. Will there be open sourced tools in addition to LTTng?
[/quote]

The video-link is here: https://vimeo.com/236186712

I have since published the related code at https://github.com/bosch-robotics-cr/tracetools, but be aware that it's not very useful without our instrumentation of roscpp. I'm also in the process of publishing that, but not quite there, yet.

[quote="fkromer, post:33, topic:2405"]
If the same node is used in different applications and/or environments the requirements thats true abd could be classified via use cases (here: warehouse, car) and the checks parametrized according to that. (Application specific: A warehouse robot should never drive faster than 1m/s. Environment specific: Every of your self-driving cars should not drive faster than 50km/h in the city and faster than 200km/h on the autobahn.)
[/quote]

Well, yes, you could, but that's why I would prefer to put those specs (at least in part) onto the resulting system.





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/34) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="iluetkeb, post:34, topic:2405"]
I also once had a case with parameters, where the combination of two parameters resulted in an out-of-range condition.
[/quote]

Right, that are typical cases for combinatorial ROS node unit tests. However often it can be hard to guess the right samples of combinations for input data (parameters, received topic messages, service requests, action requests) for tests in advance. It can be easier to define property based tests which can be setup to execute the tests with all combinations of input data and decide later if the input data for failing tests are actually invalid or if they are ok (and exclude them from further test executions to prevent from false positives if applicable). However I do not know about the current state of property based unit testing of nodes in ROS. Some of @asmodehn projects use [hypothesis](http://hypothesis.works) which is a property based testing framework for Python already but I don't know how it is used in detail yet:

[quote="asmodehn, post:27, topic:2405"]
Quick replies :

* pyros-test is MIT or BSD, along these lines, I just havent taken the time to put a file there I ll try to do it soon.
* python makes things simpler than C++ as there are defacto standard test frameworks. So I am trying to support whatever basic python and ROS support ( unittest, doctest, and nose ) and pytest. unittest already includes a mock library by the way in python3, which is just the mock library in python2 .
* pyros-test is currently very very simple (probably too simple to need a separate package), and Id like to eventually improve it when I get the chance

But IMHO you re probably better of extracting what I already started in https://github.com/pyros-dev/pyros-msgs and https://github.com/pyros-dev/pyros-schemas : have a look at property based testing and hypothesis, it should be simple enough to automatically generate fake nodes based on an existing message definition and then send fake messages around :slight_smile: .

Some example of hypothesis use here : https://github.com/pyros-dev/pyros-schemas/blob/nested_merged/tests/test_pyros_schemas/hypothesis_example.py and an example of generating messages with it there https://github.com/pyros-dev/pyros-schemas/blob/nested_merged/tests/test_pyros_schemas/test_basic_fields.py
[/quote]

[quote="iluetkeb, post:34, topic:2405"]
You can use roswtf during runtime, it will report subscriptions that have no publisher, and also type mismatches.

The thing is, you never know whether a subscription might be optional, and there are also cases where either one or the other subscription can be used, but not both.

Again, I agree this would be useful to check. Weve so far called such things graph consistency checks.
[/quote]

Thanks for that hint. (I thought about to wrap some ROS command line tool functionalities to find graph inconsistencies into a ROS node, run the node during node integration which asserts with error log messages if the graph would be inconsistent.)

[quote="iluetkeb, post:34, topic:2405"]
The video-link is here: https://vimeo.com/236186712

I have since published the related code at https://github.com/bosch-robotics-cr/tracetools, but be aware that its not very useful without our instrumentation of roscpp. Im also in the process of publishing that, but not quite there, yet.
[/quote]

Thanks a lot. I am looking forward to it.

[quote="iluetkeb, post:34, topic:2405"]
Well, yes, you could, but thats why I would prefer to put those specs (at least in part) onto the resulting system.
[/quote]

Right. Makes sense.





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/35) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


[quote="iluetkeb, post:34, topic:2405"]
I have since published the related code at https://github.com/bosch-robotics-cr/tracetools, but be aware that its not very useful without our instrumentation of roscpp. Im also in the process of publishing that, but not quite there, yet.
[/quote]

Hi @iluetkeb, the [ros_comm branch kinetic-devel](https://github.com/bosch-robotics-cr/ros_comm/tree/kinetic-devel) is BOSCH CRs instrumented version of ros_comm I guess.





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/36) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


Yes, though the plan is to make a PR soon.





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/37) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
Reply | Threaded
Open this post in threaded view
|

[Discourse.ros.org] [Next Generation ROS] Design By Contract

Tully Foote via ros-users
In reply to this post by Tully Foote via ros-users


In case someone is still interested in *Design by Contract* in ROS1 feel free to check out my spare time project [rosdbc](https://github.com/fkromer/rosdbc). (Don't expect fast progress.)





---
[Visit Topic](https://discourse.ros.org/t/design-by-contract/2405/38) or reply to this email to respond.


If you do not want to receive messages from ros-users please use the unsubscribe link below. If you use the one above, you will stop all of ros-users from receiving updates.
______________________________________________________________________________
ros-users mailing list
[hidden email]
http://lists.ros.org/mailman/listinfo/ros-users
Unsubscribe: <http://lists.ros.org/mailman//options/ros-users>
12