Quantcast

TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

Buckel, Wolfgang
TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

I get this error under Windows with Sonar 1.4.2 when a java file with German umlauts is run through CPD (see attachment). The maven build stops with an error and unfortunately gathering of sonar measures fails completely (would be nice if it simply ignored failed sub tasks).

This seems to be related to http://jira.codehaus.org/browse/MPMD-58. However, Sonar already uses the fix version 2.3 of maven-pmd-plugin.

Also pmd:pmd reports a warning but at least doesn't fail.
([INFO] [pmd:pmd]
[WARNING] Error while processing N:\tmc\dvp\qle\src\com\sitraffic\tmc\qle\cfg\qle\GeraetefachverwaltungEnum.java: Lexical error at line 44, column 15.  Encountered: "\ufffd" (65533), after : "" )

I've also tried to run maven/sonar with -Dfile.encoding=UTF-8 but that doesn't change anything. Any ideas? Help appreciated.

With best regards,
Wolfgang Buckel

Siemens AG
Industry Sector
I MO TS R&D SE
Hofmannstr. 51
81379 Muenchen, Germany
Tel.: +49 (89) 722-22370
Fax: +49 (89) 722-44477
Mobile: +49 (172) 8421190
[hidden email]

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard Cromme; Managing Board: Peter Loescher, Chairman, President and Chief Executive Officer; Wolfgang Dehen, Heinrich Hiesinger, Joe Kaeser, Jim Reid-Anderson, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen; Registered offices: Berlin and Munich; Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322

Important notice: This e-mail and any attachment thereof contain corporate proprietary information. If you have received it by mistake, please notify us immediately by reply e-mail and delete this e-mail and its attachments from your system. Thank you.

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

Buckel, Wolfgang
TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""
Oops, forgot attachment.

From: Buckel, Wolfgang
Sent: Tuesday, October 14, 2008 1:30 PM
To: [hidden email]
Subject: [sonar-user] TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

I get this error under Windows with Sonar 1.4.2 when a java file with German umlauts is run through CPD (see attachment). The maven build stops with an error and unfortunately gathering of sonar measures fails completely (would be nice if it simply ignored failed sub tasks).

This seems to be related to http://jira.codehaus.org/browse/MPMD-58. However, Sonar already uses the fix version 2.3 of maven-pmd-plugin.

Also pmd:pmd reports a warning but at least doesn't fail.
([INFO] [pmd:pmd]
[WARNING] Error while processing N:\tmc\dvp\qle\src\com\sitraffic\tmc\qle\cfg\qle\GeraetefachverwaltungEnum.java: Lexical error at line 44, column 15.  Encountered: "\ufffd" (65533), after : "" )

I've also tried to run maven/sonar with -Dfile.encoding=UTF-8 but that doesn't change anything. Any ideas? Help appreciated.

With best regards,
Wolfgang Buckel

Siemens AG
Industry Sector
I MO TS R&D SE
Hofmannstr. 51
81379 Muenchen, Germany
Tel.: +49 (89) 722-22370
Fax: +49 (89) 722-44477
Mobile: +49 (172) 8421190
[hidden email]

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard Cromme; Managing Board: Peter Loescher, Chairman, President and Chief Executive Officer; Wolfgang Dehen, Heinrich Hiesinger, Joe Kaeser, Jim Reid-Anderson, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen; Registered offices: Berlin and Munich; Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322

Important notice: This e-mail and any attachment thereof contain corporate proprietary information. If you have received it by mistake, please notify us immediately by reply e-mail and delete this e-mail and its attachments from your system. Thank you.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

GeraetefachverwaltungEnum.java (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

Simon
Administrator
Hi Wolfgang,

Before trying to understand the PMD issue, could you help me to
compile your file with UTF-8 ? ;o)
My java plateform encoding is file.encoding=MacRoman. However pom.xml
explicitly defines UTF-8 encoding :

<properties>
    <project.build.sourceEncoding>utf-8</project.build.sourceEncoding>
  </properties>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
          <source>1.5</source>
          <target>1.5</target>
          <encoding>${project.build.sourceEncoding}</encoding>
        </configuration>
      </plugin>
    </plugins>
  </build>

But it fails :

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure

/Users/sbrandhof/projects/sonar/svn/trunk/samples/utf8-sample/src/main/java/org/sonar/samples/utf8/GeraetefachverwaltungEnum.java:[43,14]
illegal character: \65533

/Users/sbrandhof/projects/sonar/svn/trunk/samples/utf8-sample/src/main/java/org/sonar/samples/utf8/GeraetefachverwaltungEnum.java:[44,18]
'class' or 'interface' expected

/Users/sbrandhof/projects/sonar/svn/trunk/samples/utf8-sample/src/main/java/org/sonar/samples/utf8/GeraetefachverwaltungEnum.java:[46,4]
'class' or 'interface' expected

/Users/sbrandhof/projects/sonar/svn/trunk/samples/utf8-sample/src/main/java/org/sonar/samples/utf8/GeraetefachverwaltungEnum.java:[63,0]
'class' or 'interface' expected

no more tokens - could not parse error message:
/Users/sbrandhof/projects/sonar/svn/trunk/samples/utf8-sample/src/main/java/org/sonar/samples/utf8/GeraetefachverwaltungEnum.java:64:
'class' or 'interface' expected


It works when setting ISO-8859-1 encoding.

Thank you

On Tue, Oct 14, 2008 at 1:33 PM, Buckel, Wolfgang
<[hidden email]> wrote:

> Oops, forgot attachment.
> ________________________________
> From: Buckel, Wolfgang
> Sent: Tuesday, October 14, 2008 1:30 PM
> To: [hidden email]
> Subject: [sonar-user] TokenMgrError: Lexical error at line 50, column 35.
> Encountered: "\ufffd" (65533), after : ""
>
> I get this error under Windows with Sonar 1.4.2 when a java file with German
> umlauts is run through CPD (see attachment). The maven build stops with an
> error and unfortunately gathering of sonar measures fails completely (would
> be nice if it simply ignored failed sub tasks).
>
> This seems to be related to http://jira.codehaus.org/browse/MPMD-58.
> However, Sonar already uses the fix version 2.3 of maven-pmd-plugin.
>
> Also pmd:pmd reports a warning but at least doesn't fail.
> ([INFO] [pmd:pmd]
> [WARNING] Error while processing
> N:\tmc\dvp\qle\src\com\sitraffic\tmc\qle\cfg\qle\GeraetefachverwaltungEnum.java:
> Lexical error at line 44, column 15.  Encountered: "\ufffd" (65533), after :
> "" )
>
> I've also tried to run maven/sonar with -Dfile.encoding=UTF-8 but that
> doesn't change anything. Any ideas? Help appreciated.
>
> With best regards,
> Wolfgang Buckel
>
> Siemens AG
> Industry Sector
> I MO TS R&D SE
> Hofmannstr. 51
> 81379 Muenchen, Germany
> Tel.: +49 (89) 722-22370
> Fax: +49 (89) 722-44477
> Mobile: +49 (172) 8421190
> mailto:[hidden email]
>
> Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard
> Cromme; Managing Board: Peter Loescher, Chairman, President and Chief
> Executive Officer; Wolfgang Dehen, Heinrich Hiesinger, Joe Kaeser, Jim
> Reid-Anderson, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen;
> Registered offices: Berlin and Munich; Commercial registries: Berlin
> Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322
>
> Important notice: This e-mail and any attachment thereof contain corporate
> proprietary information. If you have received it by mistake, please notify
> us immediately by reply e-mail and delete this e-mail and its attachments
> from your system. Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>



--
Simon Brandhof, +41 78 653 09 49
Hortis, Consulting & Innovation team
Sonar , Open Source Quality Dashboard for Java projects
http://sonar.codehaus.org

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

Buckel, Wolfgang
Ok, the file is obviously ISO-8859-1 encoded. However, that doesn't
solve my problem. sonar or rather pmd/cpd still fails with the described
error. Can you reproduce that behavior?

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

Buckel, Wolfgang
One step further. If I create a pom.xml with the following:
<reporting>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-pmd-plugin</artifactId>
      <configuration>
        <rulesets>
          <ruleset>/rulesets/basic.xml</ruleset>
        </rulesets>
        <sourceEncoding>${project.build.sourceEncoding}</sourceEncoding>
        <minimumTokens>100</minimumTokens>
        <targetJdk>1.5</targetJdk>
      </configuration>
    </plugin>
  </plugins>
</reporting>

> mvn org.apache.maven.plugins:maven-pmd-plugin:2.3:cpd
works if project.build.sourceEncoding is set to ISO-8859-1. It fails
with the same error when set to UTF-8.

Does that mean that sonar sets UTF-8 somehow when it executes pmd/cpd ?
How do I change that?

Wolfgang

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

Simon
Administrator
Good news, we reproduce the issue. It's a sonar bug that forces UTF-8
encoding for PMD/CPD plugins, even if  project.build.sourceEncoding is
set. See http://jira.codehaus.org/browse/SONAR-409.

It's fixed on 1.4.3, which should be released before the end of the week.

Sorry for the inconvenience.
Simon

On Wed, Oct 15, 2008 at 12:38 PM, Buckel, Wolfgang
<[hidden email]> wrote:

>
> One step further. If I create a pom.xml with the following:
> <reporting>
>  <plugins>
>    <plugin>
>      <groupId>org.apache.maven.plugins</groupId>
>      <artifactId>maven-pmd-plugin</artifactId>
>      <configuration>
>        <rulesets>
>          <ruleset>/rulesets/basic.xml</ruleset>
>        </rulesets>
>        <sourceEncoding>${project.build.sourceEncoding}</sourceEncoding>
>        <minimumTokens>100</minimumTokens>
>        <targetJdk>1.5</targetJdk>
>      </configuration>
>    </plugin>
>  </plugins>
> </reporting>
>
> > mvn org.apache.maven.plugins:maven-pmd-plugin:2.3:cpd
> works if project.build.sourceEncoding is set to ISO-8859-1. It fails
> with the same error when set to UTF-8.
>
> Does that mean that sonar sets UTF-8 somehow when it executes pmd/cpd ?
> How do I change that?
>
> Wolfgang
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>



--
Simon Brandhof, +41 78 653 09 49
Hortis, Consulting & Innovation team
Sonar , Open Source Quality Dashboard for Java projects
http://sonar.codehaus.org

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

Buckel, Wolfgang
That's great news. Thanks for the work Simon.

One more question. Will the fix also work, if use the sonar hudson
plugin? Since we actually build our software with ant I use the option
"Check if this project is NOT built with maven2". As you know this will
create a sonar_pom.xml on the fly for my project. Will the fix also work
in that scenario? If it simply uses the default encoding for the
platform it's ok for me.

Regards,
Wolfgang

-----Original Message-----
From: Simon Brandhof [mailto:[hidden email]]
Sent: Thursday, October 16, 2008 2:45 PM
To: [hidden email]
Subject: Re: [sonar-user] TokenMgrError: Lexical error at line 50,
column 35. Encountered: "\ufffd" (65533), after : ""

Good news, we reproduce the issue. It's a sonar bug that forces UTF-8
encoding for PMD/CPD plugins, even if  project.build.sourceEncoding is
set. See http://jira.codehaus.org/browse/SONAR-409.

It's fixed on 1.4.3, which should be released before the end of the
week.

Sorry for the inconvenience.
Simon

On Wed, Oct 15, 2008 at 12:38 PM, Buckel, Wolfgang
<[hidden email]> wrote:

>
> One step further. If I create a pom.xml with the following:
> <reporting>
>  <plugins>
>    <plugin>
>      <groupId>org.apache.maven.plugins</groupId>
>      <artifactId>maven-pmd-plugin</artifactId>
>      <configuration>
>        <rulesets>
>          <ruleset>/rulesets/basic.xml</ruleset>
>        </rulesets>
>
<sourceEncoding>${project.build.sourceEncoding}</sourceEncoding>

>        <minimumTokens>100</minimumTokens>
>        <targetJdk>1.5</targetJdk>
>      </configuration>
>    </plugin>
>  </plugins>
> </reporting>
>
> > mvn org.apache.maven.plugins:maven-pmd-plugin:2.3:cpd
> works if project.build.sourceEncoding is set to ISO-8859-1. It fails
> with the same error when set to UTF-8.
>
> Does that mean that sonar sets UTF-8 somehow when it executes pmd/cpd
?

> How do I change that?
>
> Wolfgang
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>



--
Simon Brandhof, +41 78 653 09 49
Hortis, Consulting & Innovation team
Sonar , Open Source Quality Dashboard for Java projects
http://sonar.codehaus.org

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

Simon
Administrator
Don't worry, the hudson plugin must not be updated. To upgrade to 1.4.3 :

- unzip the sonar distribution[1] in a new directory
- copy your /conf, /data and /extensions directories
- restart the server
- update the sonar version from hudson management console. It will
apply to all the projects.

Regards
Simon

[1] http://sonar.codehaus.org/downloads/

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: TokenMgrError: Lexical error at line 50, column 35. Encountered: "\ufffd" (65533), after : ""

Buckel, Wolfgang
Thank you so much Simon. It works like a charm!
I can't wait to try 1.5 when it comes out. Findbugs integration is
exactly what I'm looking for!

Wolfgang

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Loading...