Quantcast

[sonar-dev] New Sonar Plugin: Comparing plugin

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[sonar-dev] New Sonar Plugin: Comparing plugin

David François

Hi all,

 

I want to publish a new plugin : sonar-comparing-plugin

This plugin add two new global widget to show the number of lines of code and the number of projects by language in the SONAR instance.

It add a new project page as well to compare the different metrics for the current project selected with the average value of the metric (for the language). It possible to select another project in the same language to compare the different metrics.

 

I have already push the source on the svn (/sonar-plugins/comparing)

Is it possible to create the Bamboo Job, JIRA and make a review on it before start a vote?

 

During that, I will create a new confluence page to describe this plugin (I have initialized the Page : http://docs.codehaus.org/display/SONAR/Comparing+Plugin)

 

Thank you in advance,

 

Regards,

David FRANCOIS


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] New Sonar Plugin: Comparing plugin

Simon Brandhof
Hi David,

Thanks for the contribution, this plugin should be quite useful. How is it technically designed ? I suppose that language average measures are pre-computed and stored in db ?

Is it possible to create the Bamboo Job, JIRA and make a review on it before start a vote?


pom is updated with bamboo and JIRA urls. I started to review the plugin by fixing the compatibility with Java 5. I continue with some tests.
 

 

During that, I will create a new confluence page to describe this plugin (I have initialized the Page : http://docs.codehaus.org/display/SONAR/Comparing+Plugin)


The version 0.1 declared in the page does not exist. It should be replaced by the SNAPSHOT version.

Thanks

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [sonar-dev] New Sonar Plugin: Comparing plugin

David François

Hello,

 

How is it technically designed ? I suppose that language average measures are pre-computed and stored in db ?

Yes, exactly, the SNAPSHOT_ID column in the PROJECT_MEASURES table can be null, so I use this fact to compute the different average measure and store in DB and through this, the measure is common for all projects. This solution requires me to use ORM entity in the JAVA Code, and ActiveRecord objects in JRuby Code.

 

After, a possible problem with this solution that I haven’t managed is to have concurrent access.  At the beginning, I wanted to manage this problem with a Optimitic Locking solution but without version_number or modification_date in the PROJECT_MEASURES table, it is not obvious to check if my object have changed since last read. But one measure record is very fast, and I think we will have little probability to have record for the same measure at the same time during two parallel analysis.

 

Regards,

David FRANCOIS

 

De : Simon Brandhof [mailto:[hidden email]]
Envoyé : lundi 13 août 2012 11:04
À : [hidden email]
Objet : Re: [sonar-dev] New Sonar Plugin: Comparing plugin

 

Hi David,

Thanks for the contribution, this plugin should be quite useful. How is it technically designed ? I suppose that language average measures are pre-computed and stored in db ?

Is it possible to create the Bamboo Job, JIRA and make a review on it before start a vote?


pom is updated with bamboo and JIRA urls. I started to review the plugin by fixing the compatibility with Java 5. I continue with some tests.
 

 

During that, I will create a new confluence page to describe this plugin (I have initialized the Page : http://docs.codehaus.org/display/SONAR/Comparing+Plugin)


The version 0.1 declared in the page does not exist. It should be replaced by the SNAPSHOT version.

Thanks

 


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] New Sonar Plugin: Comparing plugin

Simon Brandhof
Using PROJECT_MEASURES with SNAPSHOT_ID is not substainable for the following reasons :
  • it's a hack. Hacks are not guaranted to be compatible with future versions. Hacks are crappy :D
  • concurrent analysis is a problem. Even if you estimate that probability is low, I can assure you that the pb will occur on "enterprise" environments that can have a dozen of parallel analysis.
  • it's not fault tolerant. If the job crashes during the insertion of measures, then data are corrupted.
Moreover I'm not convinced by the functional feature itself.
First of all, comparing quantitative values like number of duplicated lines or violations usually makes nonsense. If you have a look at this real use-case : https://skitch.com/sonarsource/ecqck/comparing-violations, you can see that my project has much more violations but its rules compliance is higher ! Except for some size measures like ncloc or complexity, comparison is usually interesting only on "qualitative" measures like density of dupl lines, rules compliance or complexity/method.
That means that comparison with quantitative measures and their related average can be removed.
Then remain qualitative measures. Their global average can be replaced by the more accurate values provided by the Views plugin. It's easy to define a view of all the Java projects. Just compare your project with this view, technically considered as another project.

All that to say that removing average measures and reducing the number of compared metrics would much improve the efficiency and the reliability of the plugin. That's my feeling. WDYT ?

Thanks

On 13 August 2012 11:47, Francois, David <[hidden email]> wrote:

Hello,

 

How is it technically designed ? I suppose that language average measures are pre-computed and stored in db ?

Yes, exactly, the SNAPSHOT_ID column in the PROJECT_MEASURES table can be null, so I use this fact to compute the different average measure and store in DB and through this, the measure is common for all projects. This solution requires me to use ORM entity in the JAVA Code, and ActiveRecord objects in JRuby Code.

 

After, a possible problem with this solution that I haven’t managed is to have concurrent access.  At the beginning, I wanted to manage this problem with a Optimitic Locking solution but without version_number or modification_date in the PROJECT_MEASURES table, it is not obvious to check if my object have changed since last read. But one measure record is very fast, and I think we will have little probability to have record for the same measure at the same time during two parallel analysis.

 

Regards,

David FRANCOIS

 

De : Simon Brandhof [mailto:[hidden email]]
Envoyé : lundi 13 août 2012 11:04
À : [hidden email]
Objet : Re: [sonar-dev] New Sonar Plugin: Comparing plugin

 

Hi David,

Thanks for the contribution, this plugin should be quite useful. How is it technically designed ? I suppose that language average measures are pre-computed and stored in db ?

Is it possible to create the Bamboo Job, JIRA and make a review on it before start a vote?


pom is updated with bamboo and JIRA urls. I started to review the plugin by fixing the compatibility with Java 5. I continue with some tests.
 

 

During that, I will create a new confluence page to describe this plugin (I have initialized the Page : http://docs.codehaus.org/display/SONAR/Comparing+Plugin)


The version 0.1 declared in the page does not exist. It should be replaced by the SNAPSHOT version.

Thanks

 


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.



--

Simon BRANDHOF | SonarSource
http://sonarsource.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [sonar-dev] New Sonar Plugin: Comparing plugin

David François

Hi Simon,

 

Thanks for your feedback.

I suspected that this solution would not please J

  • concurrent analysis is a problem. Even if you estimate that probability is low, I can assure you that the pb will occur on "enterprise" environments that can have a dozen of parallel analysis.

It’s why I estimate that probability is low, I have already deploy this plugin on “enterprise” enrironments which run parallel analysis in the night (not dozen but 2 or 3) and I had no problem.

  • it's not fault tolerant. If the job crashes during the insertion of measures, then data are corrupted.

I’m agree, even if the risk is low, you can have a corruption.

All this for my defense J But I’m agree with your assessment, I have a real risk of reliability, but without hack, I think it’s not possible to access on other resources than the project.

But I don’t want to remove the average value, actually, I have implemented this plugin for this fonctionnality in first and moreover I used the same solution to compute measure for the global widget. The comparison between project have only implemented at the end, the most important thing for me is to have a comparison point with a medium value for the language.

 

But another solution could be to compute all the average value in the JRuby code, without record in the DB. I know you are not fan of the usage of ActiveRecord for the reason of compatibility with future version of Sonar but I think it could be a good compromise. WDYT?

 

First of all, comparing quantitative values like number of duplicated lines or violations usually makes nonsense.

Yes, I’m absolutely agree with this. At the beginning, I had only the qualitative measure except for ncloc and complexity metrics, but I choosed to put quantitative measure too (but with a different management of color when the measure is display in the comparing page) because an improvement of the plugin for me will be to replace the page by one widget you can configure (title, which metric to display) and allow the admin to display what he wants.

 

Other important think to know to the computation of the average measure of qualitative metric. I take the duplication lines rate for example to explain what I want to say

I had 2 solutions to compute this metric:

·         compute the average value between the number of projects projects for the duplicate_lines_density metrics regardless of the number of lines of code by projects

·         compute the real rate of duplication adding the number of duplicate lines for all projects divided by the number of lines of code for all projects

I choosed the first solution because this simplified the implementation of the plugin and I really think it’s the good solution when you want to position your application in your application portfolio.

 

Their global average can be replaced by the more accurate values provided by the Views plugin. It's easy to define a view of all the Java projects.

This is a thinking I had but I didn’t want a dependency with other plugin for this first version. But I can imagine an improvement adding a widget property where you can choose to compare with all projects for the same language or with a specific view if the plugin is installed.

 

Hoping all is understandable, it very hard for me to explain exactly what I want to say in English J

 

Regards,

David FRANCOIS

 

De : Simon Brandhof [mailto:[hidden email]]
Envoyé : mardi 14 août 2012 10:53
À : [hidden email]
Objet : Re: [sonar-dev] New Sonar Plugin: Comparing plugin

 

Using PROJECT_MEASURES with SNAPSHOT_ID is not substainable for the following reasons :

  • it's a hack. Hacks are not guaranted to be compatible with future versions. Hacks are crappy :D
  • concurrent analysis is a problem. Even if you estimate that probability is low, I can assure you that the pb will occur on "enterprise" environments that can have a dozen of parallel analysis.
  • it's not fault tolerant. If the job crashes during the insertion of measures, then data are corrupted.

Moreover I'm not convinced by the functional feature itself.
First of all, comparing quantitative values like number of duplicated lines or violations usually makes nonsense. If you have a look at this real use-case : https://skitch.com/sonarsource/ecqck/comparing-violations, you can see that my project has much more violations but its rules compliance is higher ! Except for some size measures like ncloc or complexity, comparison is usually interesting only on "qualitative" measures like density of dupl lines, rules compliance or complexity/method.
That means that comparison with quantitative measures and their related average can be removed.
Then remain qualitative measures. Their global average can be replaced by the more accurate values provided by the Views plugin. It's easy to define a view of all the Java projects. Just compare your project with this view, technically considered as another project.

All that to say that removing average measures and reducing the number of compared metrics would much improve the efficiency and the reliability of the plugin. That's my feeling. WDYT ?

Thanks

On 13 August 2012 11:47, Francois, David <[hidden email]> wrote:

Hello,

 

How is it technically designed ? I suppose that language average measures are pre-computed and stored in db ?

Yes, exactly, the SNAPSHOT_ID column in the PROJECT_MEASURES table can be null, so I use this fact to compute the different average measure and store in DB and through this, the measure is common for all projects. This solution requires me to use ORM entity in the JAVA Code, and ActiveRecord objects in JRuby Code.

 

After, a possible problem with this solution that I haven’t managed is to have concurrent access.  At the beginning, I wanted to manage this problem with a Optimitic Locking solution but without version_number or modification_date in the PROJECT_MEASURES table, it is not obvious to check if my object have changed since last read. But one measure record is very fast, and I think we will have little probability to have record for the same measure at the same time during two parallel analysis.

 

Regards,

David FRANCOIS

 

De : Simon Brandhof [mailto:[hidden email]]
Envoyé : lundi 13 août 2012 11:04
À : [hidden email]
Objet : Re: [sonar-dev] New Sonar Plugin: Comparing plugin

 

Hi David,

Thanks for the contribution, this plugin should be quite useful. How is it technically designed ? I suppose that language average measures are pre-computed and stored in db ?

Is it possible to create the Bamboo Job, JIRA and make a review on it before start a vote?


pom is updated with bamboo and JIRA urls. I started to review the plugin by fixing the compatibility with Java 5. I continue with some tests.
 

 

During that, I will create a new confluence page to describe this plugin (I have initialized the Page : http://docs.codehaus.org/display/SONAR/Comparing+Plugin)


The version 0.1 declared in the page does not exist. It should be replaced by the SNAPSHOT version.

Thanks

 


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.




--

 

Simon BRANDHOF | SonarSource
http://sonarsource.com


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [sonar-dev] New Sonar Plugin: Comparing plugin

Jorge Costa
Hi,

Not a big problem but c++ project is incorrectly displayed in one of the widgets.



BR,
JC
Best Regards
Jorge Costa
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [sonar-dev] New Sonar Plugin: Comparing plugin

David François
In reply to this post by David François

Hi,

 

After some tests, it’s not very complicate to do all computation for average or other aggregate values in the JRuby code (it’s already test for the 2 widgets). Efficiency of this solution is not already tested but I have an instance with 150 projects, I think this will be a good start for my tests and if it’s successful, I will refactor all the code to remove all records in the database.

 

Regards,

David FRANCOIS


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [sonar-dev] New Sonar Plugin: Comparing plugin

David François

I have rewritten the plugin to compute all aggregate measures in the JRuby code without record in the database and commit this new implementation.

Can you check and say me if you are agree with this new solution?

 

Regards,

David FRANCOIS

 

De : Francois, David [mailto:[hidden email]]
Envoyé : jeudi 16 août 2012 18:51
À : [hidden email]
Objet : RE: [sonar-dev] New Sonar Plugin: Comparing plugin

 

Hi,

 

After some tests, it’s not very complicate to do all computation for average or other aggregate values in the JRuby code (it’s already test for the 2 widgets). Efficiency of this solution is not already tested but I have an instance with 150 projects, I think this will be a good start for my tests and if it’s successful, I will refactor all the code to remove all records in the database.

 

Regards,

David FRANCOIS


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [sonar-dev] New Sonar Plugin: Comparing plugin

David François

Hi,

 

Can i start the vote for this plugin ?

 

Regards,

David FRANCOIS

 

De : Francois, David [mailto:[hidden email]]
Envoyé : vendredi 17 août 2012 16:07
À : [hidden email]
Objet : RE: [sonar-dev] New Sonar Plugin: Comparing plugin

 

I have rewritten the plugin to compute all aggregate measures in the JRuby code without record in the database and commit this new implementation.

Can you check and say me if you are agree with this new solution?

 

Regards,

David FRANCOIS

 

De : Francois, David [[hidden email]]
Envoyé : jeudi 16 août 2012 18:51
À : [hidden email]
Objet : RE: [sonar-dev] New Sonar Plugin: Comparing plugin

 

Hi,

 

After some tests, it’s not very complicate to do all computation for average or other aggregate values in the JRuby code (it’s already test for the 2 widgets). Efficiency of this solution is not already tested but I have an instance with 150 projects, I think this will be a good start for my tests and if it’s successful, I will refactor all the code to remove all records in the database.

 

Regards,

David FRANCOIS


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] New Sonar Plugin: Comparing plugin

Freddy Mallet
You can go ahead David !

-----
Sonar for Continuous Inspection



On Wed, Aug 22, 2012 at 5:16 PM, Francois, David <[hidden email]> wrote:

Hi,

 

Can i start the vote for this plugin ?

 

Regards,

David FRANCOIS

 

De : Francois, David [mailto:[hidden email]]
Envoyé : vendredi 17 août 2012 16:07


À : [hidden email]
Objet : RE: [sonar-dev] New Sonar Plugin: Comparing plugin

 

I have rewritten the plugin to compute all aggregate measures in the JRuby code without record in the database and commit this new implementation.

Can you check and say me if you are agree with this new solution?

 

Regards,

David FRANCOIS

 

De : Francois, David [[hidden email]]
Envoyé : jeudi 16 août 2012 18:51
À : [hidden email]
Objet : RE: [sonar-dev] New Sonar Plugin: Comparing plugin

 

Hi,

 

After some tests, it’s not very complicate to do all computation for average or other aggregate values in the JRuby code (it’s already test for the 2 widgets). Efficiency of this solution is not already tested but I have an instance with 150 projects, I think this will be a good start for my tests and if it’s successful, I will refactor all the code to remove all records in the database.

 

Regards,

David FRANCOIS


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.


Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

Loading...