Planet Geek

Zorching Memory Leaks. I Am Teh Stud

Business yammerings , Programming-fu

Tonight, I am Da Man.

It's rare I get to sit down and see, so directly, a problem in the code I'm working on. In this case, while running at Ubercon the other weekend, I noticed that CONGO was showing every sign of a memory leak. After a while I'd get sluggish performance, and eventually I'd get an OutOfMemory error from the JVM. Didn't look good.

This had me in somewhat of a panic, since Arisia is right around the corner, and I need the system stable and useable for the event. With the help of the wonderous tool that is JConsole, I was able to actually see the memory usage going up in the JVM as I worked with the server. There was definately a problem.

It turns out my hastily set up code to handle changes to my JDBC driver was not being handled correctly. Apparently with the old mm.mysql driver, SQL handles were automatically closed at the end of use. With the new Connector/J driver from MySQL, you had to explicitely issue a close() statement on any Statement, or the handle would never be freed. This was probably always the case, it was just my sloppy code and a twitch of the old driver that made it never come up.

Unfortunately, this necessitated changing just about every SQL call in the server, all 10,000 lines of it. It was long, painful, boring work. I had some help from blk, but it still took a good week of off-hour fiddling to finish the changes and do some basic regression tests.

Tonight, I made the last set of changes, did some basic runthroughs, and didn't have a single crash. With someone trembling fingers, I started up JConsole again, connected to the application server, and started working with CONGO. Registrant lookups, reports, modifications, creations, subscriptions, convention detail editing - throughout it all, JConsole happily continued showing the normal 'sawtooth' pattern of memory usage. Things would get allocated, used, and then the garbage collector would come along and reap the memory.

Success! I had successfully searched for, found, and fixed a major memory leak that was persistent throughout the code.

For those insisting on the details, here's a properly structured, Prepared, and appropriately exception-managed call to get a number from the database. (And, before I get abusive comments all over the place, the Exception handling here is VERY POOR, and will result in a clean result coming back from the method, even though the call may have crashed. I know, thank you, now go away).

private static int calcRegistered(int cid, String typeName) {
        PreparedStatement p = null;
        ResultSet rset = null;
        String sql = "SELECT count(*) AS total FROM reg_state " +
            "WHERE state_cid=? AND state_regtype=? " +
            "AND state_registered=1";
        int value = 0;
        try {
            p = cserver.Conn.prepareStatement(sql);
            p.setInt(1,cid);
            p.setString(2,typeName);
            rset = p.executeQuery();
            if (rset.next()) {
                value = rset.getInt("total");
                logger.debug("\tcalcRegistered: count for conference '" + cid + "', type '" +
                   typeName + "' is --: " + value);
            }
        }
        catch (Exception e) {
            dumpException(e);
        }
        finally {
            try { p.close(); }
            catch (Exception e) { dumpException(e); }
        }
        return value;
    }

I feel all geeklystudly.

Next is to start seriously hammering away at the buglist, and get a test environment set up so other CONGO users can test out the new Merge function against 'real' data. If all goes well, we'll roll this version into production in time to run Arisia.

Posted by dbs at November 12, 2007 9:46 PM