Fixing Strange NoClassDefFoundError on App Engine

On Friday morning (Mar 25, 2011) Google’s AppEngine had a “nice” surprise for me… Two of my applications running on HR datastore started throwing NoClassDefFoundError out of nowhere. Interestingly, the same application was running fine on the MS datastore. I filled a production issue report, posted to the google-appengine group, etc. but no one had a clue. It looked like my case was isolated and there were no similar complains. I was trying re-deploying the applications, switching to older app version but no luck. I was getting desperate. Then, on Saturday morning, as it was epidemic, exactly the same exception happened to the application running on MS datastore. All these applications were based on the same code base. It became obvious that my applications were affected by some Google’s internal update (possibly upgrading for announced SDK 1.4.3) but that it’s not a wide spread issue. So, it was something I have to deal with…

The exception thrown by GAE was:

Caused by: java.lang.ClassNotFoundException: 
org.apache.james.mime4j.message.Body 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:321) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:266) 
    ... 26 more

What’s interesting is that I didn’t reference this class directly and could not find a reference to it in any of included jar files. (Full text search in jar archives did not find anything related.) However, my applications are using Apache HttpClient 4.1 and I remember that while the library was in beta there was a dependency on James’ mime4j package, which was removed in later releases. The exception above actually was thrown in part of code that uses HttpClient library.

My assumption was that Google possibly added an older version of HttpClient to its core libraries, which was conflicting with class loading in my application. I had similar jar hell stories in the past that I solved using the JarJar. I was already out of ideas and decided to give it a try.

What I did is that I used JarJar to move HttpClient libraries to a different namespace - I renamed package org.apache.http to repackaged.org.apache.http:

1
2
3
4
5
6
7
8
9
10
<property name="httpclient.jars.dir" location="/path/to/httpcomponents-client-4.1.1/lib" />
<target name="repack-http-client">
  <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask" classpath="lib/jarjar-1.1.jar"/>
  <jarjar jarfile="httpcomponents-client-repackaged.jar">
    <zipfileset src="${httpclient.jars.dir}/httpclient-4.1.1.jar" />
    <zipfileset src="${httpclient.jars.dir}/httpcore-4.1.jar" />
    <zipfileset src="${httpclient.jars.dir}/httpmime-4.1.1.jar" />
    <rule pattern="org.apache.http.**" result="repackaged.org.apache.http.@1"/>
  </jarjar>
</target>

After this I removed the original HttpClient libraries from my project and put httpcomponents-client-repackaged.jar instead. Obviously, I had to change all imports in java files to reference repackaged.org.apache.http.

I crossed my fingers, deployed to GAE and… it worked! NoClassDefFoundError has disappeared and repackaged HttpClient library was working fine. (Thanks Google for one more excitement although I’d rather pass this one.)

Related GAE articles:

Comments