Quantcast

Batch performance suggestion

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Batch performance suggestion

Javier Delgado-3
When starting an analysis, every plugin at the Sonar server gets downloaded (and its dependant libs). With Sonar 3.1 and the default core plugins (checkstyle, pmd, jacoco, dbcleaner ... ) this sums roughly 20 MB. I have seen people in this list with a pretty big plugins list, so this numbers could be bigger.

Wouldnt it be nice to limit this downloads (as its elapsed time added for each analysis) ?

One solution would be to just download the required plugins. Instead of excluding plugins already downloaded, directly skip them (i.e. a xml analysis won't need checkstyle or cpd).
Another workaround could be caching those plugins. Instead of copying them to a temp folder, a permanent one could be used. With a CRC, there could be checks to force a "refresh".

What do you think about this?

Regards
--
It is better to die() than to return() in failure
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Batch performance suggestion

Patroklos Papapetrou
Hi Javier

I am really missing something in your message.
How do u understand that Sonar dowloads core plugins and extra plugins each time you perform analysis?
Core plugins are located in $SONAR_HOME$/lib/core-plugins and are stored locally with Sonar installation. Never downloaded.
Additional plugins are located under $SONAR_HOME$/extensions/plugins and are downloaded once when you ask for it within update center

Regards

2012/7/11 Javier Delgado <[hidden email]>
When starting an analysis, every plugin at the Sonar server gets downloaded (and its dependant libs). With Sonar 3.1 and the default core plugins (checkstyle, pmd, jacoco, dbcleaner ... ) this sums roughly 20 MB. I have seen people in this list with a pretty big plugins list, so this numbers could be bigger.

Wouldnt it be nice to limit this downloads (as its elapsed time added for each analysis) ?

One solution would be to just download the required plugins. Instead of excluding plugins already downloaded, directly skip them (i.e. a xml analysis won't need checkstyle or cpd).
Another workaround could be caching those plugins. Instead of copying them to a temp folder, a permanent one could be used. With a CRC, there could be checks to force a "refresh".

What do you think about this?

Regards
--
It is better to die() than to return() in failure

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Batch performance suggestion

Evgeny Mandrikov
Hi guys,

First of all: I do confirm that Sonar Batch downloads all plugins (core and others) during analysis from Sonar server to the machine, where analysis performed.

And indeed - there is plenty space for improvements. I suppose that current situation is following: we know that this is not optimal, but this is not blocker for most of the users. However we had an internal discussion about this some time ago, but I'm not sure that it was converted into Jira tickets, so can't provide a plan, when it will be improved.

On Wed, Jul 11, 2012 at 4:19 PM, Papapetrou P.Patroklos <[hidden email]> wrote:
Hi Javier

I am really missing something in your message.
How do u understand that Sonar dowloads core plugins and extra plugins each time you perform analysis?
Core plugins are located in $SONAR_HOME$/lib/core-plugins and are stored locally with Sonar installation. Never downloaded.
Additional plugins are located under $SONAR_HOME$/extensions/plugins and are downloaded once when you ask for it within update center

Regards

2012/7/11 Javier Delgado <[hidden email]>
When starting an analysis, every plugin at the Sonar server gets downloaded (and its dependant libs). With Sonar 3.1 and the default core plugins (checkstyle, pmd, jacoco, dbcleaner ... ) this sums roughly 20 MB. I have seen people in this list with a pretty big plugins list, so this numbers could be bigger.

Wouldnt it be nice to limit this downloads (as its elapsed time added for each analysis) ?

One solution would be to just download the required plugins. Instead of excluding plugins already downloaded, directly skip them (i.e. a xml analysis won't need checkstyle or cpd).
Another workaround could be caching those plugins. Instead of copying them to a temp folder, a permanent one could be used. With a CRC, there could be checks to force a "refresh".

What do you think about this?

Regards
--
It is better to die() than to return() in failure




--
Best regards,
Evgeny Mandrikov aka Godin <http://godin.net.ru>
http://twitter.com/_godin_
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Batch performance suggestion

Javier Delgado-3
@Patroklos:

Performing an analysis with sonar-runner and debug (-X) output gives the following:

12:41:34.882 DEBUG s.a.u.HttpDownloader - Download: (...)/sonar-jacoco-plugin-3.1.jar (no proxy)
12:41:34.898 DEBUG atchPluginRepository - Excluded plugin: dbcleaner

(note theres a missing character for BatchPluginRepository). The source (https://github.com/SonarSource/sonar/blob/master/sonar-batch/src/main/java/org/sonar/batch/bootstrap/BatchPluginRepository.java#L79) shows the download+(include|exclude) workflow

@Evgeny:

Its sure the overal improvement for this change would be small, and I really doubt it would be a blocker issue for noone. Just knowing its a understood fact (and eventually modifiable) is enough for me.

Just were my two cents.

Regards

On Wed, Jul 11, 2012 at 12:27 PM, Evgeny Mandrikov <[hidden email]> wrote:
Hi guys,

First of all: I do confirm that Sonar Batch downloads all plugins (core and others) during analysis from Sonar server to the machine, where analysis performed.

And indeed - there is plenty space for improvements. I suppose that current situation is following: we know that this is not optimal, but this is not blocker for most of the users. However we had an internal discussion about this some time ago, but I'm not sure that it was converted into Jira tickets, so can't provide a plan, when it will be improved.

On Wed, Jul 11, 2012 at 4:19 PM, Papapetrou P.Patroklos <[hidden email]> wrote:
Hi Javier

I am really missing something in your message.
How do u understand that Sonar dowloads core plugins and extra plugins each time you perform analysis?
Core plugins are located in $SONAR_HOME$/lib/core-plugins and are stored locally with Sonar installation. Never downloaded.
Additional plugins are located under $SONAR_HOME$/extensions/plugins and are downloaded once when you ask for it within update center

Regards

2012/7/11 Javier Delgado <[hidden email]>
When starting an analysis, every plugin at the Sonar server gets downloaded (and its dependant libs). With Sonar 3.1 and the default core plugins (checkstyle, pmd, jacoco, dbcleaner ... ) this sums roughly 20 MB. I have seen people in this list with a pretty big plugins list, so this numbers could be bigger.

Wouldnt it be nice to limit this downloads (as its elapsed time added for each analysis) ?

One solution would be to just download the required plugins. Instead of excluding plugins already downloaded, directly skip them (i.e. a xml analysis won't need checkstyle or cpd).
Another workaround could be caching those plugins. Instead of copying them to a temp folder, a permanent one could be used. With a CRC, there could be checks to force a "refresh".

What do you think about this?

Regards
--
It is better to die() than to return() in failure




--
Best regards,
Evgeny Mandrikov aka Godin <http://godin.net.ru>
http://twitter.com/_godin_



--
It is better to die() than to return() in failure
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Batch performance suggestion

Patroklos Papapetrou
In reply to this post by Evgeny Mandrikov
Hi Evgeny

I am really confused now :( How can I track that all plugins are downloaded? 
We run Sonar analysis either from our Jenkins installation ( which is in the same machine with Sonar ) or from our local workstations and I've never noticed that plugins are downloaded :(
Is this happening only the first time?
Regards
Patroklos

2012/7/11 Evgeny Mandrikov <[hidden email]>
Hi guys,

First of all: I do confirm that Sonar Batch downloads all plugins (core and others) during analysis from Sonar server to the machine, where analysis performed.

And indeed - there is plenty space for improvements. I suppose that current situation is following: we know that this is not optimal, but this is not blocker for most of the users. However we had an internal discussion about this some time ago, but I'm not sure that it was converted into Jira tickets, so can't provide a plan, when it will be improved.

On Wed, Jul 11, 2012 at 4:19 PM, Papapetrou P.Patroklos <[hidden email]> wrote:
Hi Javier

I am really missing something in your message.
How do u understand that Sonar dowloads core plugins and extra plugins each time you perform analysis?
Core plugins are located in $SONAR_HOME$/lib/core-plugins and are stored locally with Sonar installation. Never downloaded.
Additional plugins are located under $SONAR_HOME$/extensions/plugins and are downloaded once when you ask for it within update center

Regards

2012/7/11 Javier Delgado <[hidden email]>
When starting an analysis, every plugin at the Sonar server gets downloaded (and its dependant libs). With Sonar 3.1 and the default core plugins (checkstyle, pmd, jacoco, dbcleaner ... ) this sums roughly 20 MB. I have seen people in this list with a pretty big plugins list, so this numbers could be bigger.

Wouldnt it be nice to limit this downloads (as its elapsed time added for each analysis) ?

One solution would be to just download the required plugins. Instead of excluding plugins already downloaded, directly skip them (i.e. a xml analysis won't need checkstyle or cpd).
Another workaround could be caching those plugins. Instead of copying them to a temp folder, a permanent one could be used. With a CRC, there could be checks to force a "refresh".

What do you think about this?

Regards
--
It is better to die() than to return() in failure




--
Best regards,
Evgeny Mandrikov aka Godin <http://godin.net.ru>
http://twitter.com/_godin_

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Batch performance suggestion

Patroklos Papapetrou
In reply to this post by Javier Delgado-3
Thanks a lot Javier...
I tried it with Maven and you are right. I wasn't aware of that!! :)
Regards
Patroklos

2012/7/11 Javier Delgado <[hidden email]>
@Patroklos:

Performing an analysis with sonar-runner and debug (-X) output gives the following:

12:41:34.882 DEBUG s.a.u.HttpDownloader - Download: (...)/sonar-jacoco-plugin-3.1.jar (no proxy)
12:41:34.898 DEBUG atchPluginRepository - Excluded plugin: dbcleaner

(note theres a missing character for BatchPluginRepository). The source (https://github.com/SonarSource/sonar/blob/master/sonar-batch/src/main/java/org/sonar/batch/bootstrap/BatchPluginRepository.java#L79) shows the download+(include|exclude) workflow

@Evgeny:

Its sure the overal improvement for this change would be small, and I really doubt it would be a blocker issue for noone. Just knowing its a understood fact (and eventually modifiable) is enough for me.

Just were my two cents.

Regards


On Wed, Jul 11, 2012 at 12:27 PM, Evgeny Mandrikov <[hidden email]> wrote:
Hi guys,

First of all: I do confirm that Sonar Batch downloads all plugins (core and others) during analysis from Sonar server to the machine, where analysis performed.

And indeed - there is plenty space for improvements. I suppose that current situation is following: we know that this is not optimal, but this is not blocker for most of the users. However we had an internal discussion about this some time ago, but I'm not sure that it was converted into Jira tickets, so can't provide a plan, when it will be improved.

On Wed, Jul 11, 2012 at 4:19 PM, Papapetrou P.Patroklos <[hidden email]> wrote:
Hi Javier

I am really missing something in your message.
How do u understand that Sonar dowloads core plugins and extra plugins each time you perform analysis?
Core plugins are located in $SONAR_HOME$/lib/core-plugins and are stored locally with Sonar installation. Never downloaded.
Additional plugins are located under $SONAR_HOME$/extensions/plugins and are downloaded once when you ask for it within update center

Regards

2012/7/11 Javier Delgado <[hidden email]>
When starting an analysis, every plugin at the Sonar server gets downloaded (and its dependant libs). With Sonar 3.1 and the default core plugins (checkstyle, pmd, jacoco, dbcleaner ... ) this sums roughly 20 MB. I have seen people in this list with a pretty big plugins list, so this numbers could be bigger.

Wouldnt it be nice to limit this downloads (as its elapsed time added for each analysis) ?

One solution would be to just download the required plugins. Instead of excluding plugins already downloaded, directly skip them (i.e. a xml analysis won't need checkstyle or cpd).
Another workaround could be caching those plugins. Instead of copying them to a temp folder, a permanent one could be used. With a CRC, there could be checks to force a "refresh".

What do you think about this?

Regards
--
It is better to die() than to return() in failure




--
Best regards,
Evgeny Mandrikov aka Godin <http://godin.net.ru>
http://twitter.com/_godin_



--
It is better to die() than to return() in failure

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Batch performance suggestion

Evgeny Mandrikov
In reply to this post by Patroklos Papapetrou
This is what we call "works out of the box" ;)

On Wed, Jul 11, 2012 at 4:54 PM, Papapetrou P.Patroklos <[hidden email]> wrote:
Hi Evgeny

I am really confused now :( How can I track that all plugins are downloaded? 
We run Sonar analysis either from our Jenkins installation ( which is in the same machine with Sonar ) or from our local workstations and I've never noticed that plugins are downloaded :(
Is this happening only the first time?
Regards
Patroklos

2012/7/11 Evgeny Mandrikov <[hidden email]>
Hi guys,

First of all: I do confirm that Sonar Batch downloads all plugins (core and others) during analysis from Sonar server to the machine, where analysis performed.

And indeed - there is plenty space for improvements. I suppose that current situation is following: we know that this is not optimal, but this is not blocker for most of the users. However we had an internal discussion about this some time ago, but I'm not sure that it was converted into Jira tickets, so can't provide a plan, when it will be improved.

On Wed, Jul 11, 2012 at 4:19 PM, Papapetrou P.Patroklos <[hidden email]> wrote:
Hi Javier

I am really missing something in your message.
How do u understand that Sonar dowloads core plugins and extra plugins each time you perform analysis?
Core plugins are located in $SONAR_HOME$/lib/core-plugins and are stored locally with Sonar installation. Never downloaded.
Additional plugins are located under $SONAR_HOME$/extensions/plugins and are downloaded once when you ask for it within update center

Regards

2012/7/11 Javier Delgado <[hidden email]>
When starting an analysis, every plugin at the Sonar server gets downloaded (and its dependant libs). With Sonar 3.1 and the default core plugins (checkstyle, pmd, jacoco, dbcleaner ... ) this sums roughly 20 MB. I have seen people in this list with a pretty big plugins list, so this numbers could be bigger.

Wouldnt it be nice to limit this downloads (as its elapsed time added for each analysis) ?

One solution would be to just download the required plugins. Instead of excluding plugins already downloaded, directly skip them (i.e. a xml analysis won't need checkstyle or cpd).
Another workaround could be caching those plugins. Instead of copying them to a temp folder, a permanent one could be used. With a CRC, there could be checks to force a "refresh".

What do you think about this?

Regards
--
It is better to die() than to return() in failure




--
Best regards,
Evgeny Mandrikov aka Godin <http://godin.net.ru>
http://twitter.com/_godin_




--
Best regards,
Evgeny Mandrikov aka Godin <http://godin.net.ru>
http://twitter.com/_godin_
Loading...