Fixing a memory leak

Symptom
A few weeks ago one of our production nodes started to behave erratically. The Java process would grab 100% of one core and the node wasn't able to process messages fast enough.

Identifying a memory leak
I started by sending some garbage collection flags statistics to the JVM:

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

These flags generate output like this:

164406.663: [GC [PSYoungGen: 13266K->129K(128064K)] 163867K->150939K(1504320K), 0.0355750 secs]
164406.699: [Full GC [PSYoungGen: 129K->0K(128064K)] [PSOldGen: 150809K->150909K(1376256K)] 150939K->150909K(1504320K) [PSPermGen: 53237K->53237K(55168K)], 1.6966950 secs]

With a small perl script to rip out the time, heap size after GC and plot it with gnuplot i eneded up with a graph like this:


This clearly shows a linear increase in retained memory and GC-time.

Getting more info
So now I know the application is leaking memory. To find the leak I grabbed a heapdump with:

jmap -heap:format=b 

I downloaded the heap to my workstation and loaded it up in the excellent free eclipse memory analyzer tool.

The largest objects in the heap was a bunch of UnifiedClassloader3 which retained almost half the heap. However there was no trace to any classes in our application.
The classloader objects all looked normal except for one collection, 'resourceBlackList'. These collections were huge and seemed to contain xsd-paths: jboss/server/.../tmp/JBossWS_service_123123.xsd

So even though the heap didn't reveal the exact location of the leak it gave a clue that the problem is webservice related.

Finding the leak
Our application uses several webservices. However, since the leak growth was linear I could conclude that the problem lied in a core polling service.
This function was called in 20 second intervals:

public class runtimeClassHeldByMainThread {
...
public Object service.poll() {
 ...
 try {
     OurWS service = new OurWS();
     OurWSPort port = service.getPort();
     ...
 }
 return obj;
}
}

Obviously, creating a new OurWS-object on each polling is wasteful and wrong but it shouldn't leak memory, right?
But remember that the leak was a classloader collection running amok and not a bunch of dangling instances getting retained. I don't know exactly how JBoss' UnifiedClassloader works, but it seemed to pull data from the OurWS object every time a new one was instanced (for as long as the parent was alive anyway) and shove it into it's resourceBlackList.

Plugging the "leak"
By creating the OurWS object only once, in the CTOR, the leak disappeared. I'm still not sure about how JBoss handles the blacklist, when it's cleared and when stuff goes into it. If anyone could educate me on this I would appreciate it!