Quantcast

[sonar-dev] IT code coverage should be available to all coverage sensors

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[sonar-dev] IT code coverage should be available to all coverage sensors

Stéphane Nicolas
Hello all,

this is my first email on this list and I hope I am at the right place for this issue.

I am a Java Android dev and I like to test code. Unfortunately, the topic of testing is not very mature on the android platform.
Actually, I use the maven android plugin to build apps. This plugin allows 2 different kinds of tests : 
* integration tests through classic emma instrumentation (built-into the plugin)
* some way to integrate robolectric for unit testing android components (not so hard to achieve).

I recently discovered that robolectric tests could be instrumented as well via undercover. Then I wrote a plugin to integrate undercover coverage into sonar. I still have some work in progress, but the plugin is submitted to both sonar and undercover.

My problem is thatI need to say to sonar that emma will be used for Integration tests while undercover will be used for unit tests.
Unfortunately, sonar has a strong limitation against that : it only allows jacoco as the code coverage for it tests.

Could it be possible : 
* to add some system property into sonar so that every coverage tool could be used both as the unit test coverage provider and integration test coverage provider ?
* in a next step, to generalize those notions and to be able to add any test suite of any kind and coverage for them. For instance google distinguished small, medium and large tests. It would be nice to be able to launch each test suite independently and to provide independent coverage for each of them and display the result in a widget on the dash board.
Another example would be to have both unit tests, integration tests, UI tests (via robotium for instance) and to get both independent coverage statistics for them, and a combined one with merged test coverage and metedata.

Thanks in advance,
 Stéhane
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] IT code coverage should be available to all coverage sensors

Simon Brandhof
Hi Stéphane,

 
My problem is thatI need to say to sonar that emma will be used for Integration tests while undercover will be used for unit tests.
Unfortunately, sonar has a strong limitation against that : it only allows jacoco as the code coverage for it tests.


No. Currently only Jacoco supports integration tests, but that doesn't mean that other plugins can't. They just have to deal with the metrics prefixed by "it_". See
Could it be possible : 
* to add some system property into sonar so that every coverage tool could be used both as the unit test coverage provider and integration test coverage provider ?

Indeed every coverage plugin should easily support both unit and integration tests. Not sure a system property is the good solution, we should find a better generic way.
 
* in a next step, to generalize those notions and to be able to add any test suite of any kind and coverage for them. For instance google distinguished small, medium and large tests. It would be nice to be able to launch each test suite independently and to provide independent coverage for each of them and display the result in a widget on the dash board.

The widget for coverage by integration tests already exist. Here is a screenshot : http://www.coding-stories.com/wordpress/wp-content/uploads/2012/01/sonar-coverage-it.png.
 
Another example would be to have both unit tests, integration tests, UI tests (via robotium for instance) and to get both independent coverage statistics for them, and a combined one with merged test coverage and metedata.

Support of other categories like UI/acceptance tests is not planned yet. I don't know if these new categories should be predefined or if we should find a generic way to define them.
Merging of coverages is one of the most voted improvement : http://jira.codehaus.org/browse/SONAR-2392 :D
 
Regards


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] IT code coverage should be available to all coverage sensors

Tamás Kende
Hi,

this will be a great feature (almost as cool as the multi-language support) I am facing with the same problem with Erlang support. We have unit test and integration tests in small and end-to-end tests... I would be great if we can see all these results... but other groupings are also common: smoke tests, full regression tests...

We should put this to the roadmap.. at least investigate how hard is to implement it.

Cheers

Tamas

On Tue, Jun 5, 2012 at 9:44 AM, Simon Brandhof <[hidden email]> wrote:
Hi Stéphane,

 
My problem is thatI need to say to sonar that emma will be used for Integration tests while undercover will be used for unit tests.
Unfortunately, sonar has a strong limitation against that : it only allows jacoco as the code coverage for it tests.


No. Currently only Jacoco supports integration tests, but that doesn't mean that other plugins can't. They just have to deal with the metrics prefixed by "it_". See
Could it be possible : 
* to add some system property into sonar so that every coverage tool could be used both as the unit test coverage provider and integration test coverage provider ?

Indeed every coverage plugin should easily support both unit and integration tests. Not sure a system property is the good solution, we should find a better generic way.
 
* in a next step, to generalize those notions and to be able to add any test suite of any kind and coverage for them. For instance google distinguished small, medium and large tests. It would be nice to be able to launch each test suite independently and to provide independent coverage for each of them and display the result in a widget on the dash board.

The widget for coverage by integration tests already exist. Here is a screenshot : http://www.coding-stories.com/wordpress/wp-content/uploads/2012/01/sonar-coverage-it.png.
 
Another example would be to have both unit tests, integration tests, UI tests (via robotium for instance) and to get both independent coverage statistics for them, and a combined one with merged test coverage and metedata.

Support of other categories like UI/acceptance tests is not planned yet. I don't know if these new categories should be predefined or if we should find a generic way to define them.
Merging of coverages is one of the most voted improvement : http://jira.codehaus.org/browse/SONAR-2392 :D
 
Regards



Loading...