Perl Coding Standards – Wish there were some!

Some day some bright light will come forward and lay down the law about how Perl code should be formatted. And, just as likely, they will be roundly ignored, primarily out of sheer spite, but with a grain of “perl lets us do what we want. We don’t need no steenkin standards” tossed in just to muddy the argument.

Today’s annoyance is cleaning up some long forgotten programmer’s way of of formatting blocks of code. Only slightly edited, here’s a sub in one of the cgi scripts:

sub _getMediaList( $ $ $ ;$ )
{
my ($carrierId, $contentType, $contentIds, $formatted) = @_;
return undef if (!defined($contentType)); # Charts can have no contents and thus no content type
my $gpd = GPD::getDbh();
my @mediaList;
foreach my $contentId (@$contentIds)
{
my $resultList = mediaSearch(dbh => $gpd,
carrierId => $carrierId,
contentType => $contentType,
id => $contentId,
showdown => 1);
next if (!$resultList->[0]); # Shouldn't happen
push(@mediaList, $resultList->[0]);
}
if ($formatted)
{
foreach my $media (@mediaList)
{
$media->{title} = "$media->{artist} - $media->{name} - $media->{album}";
}
return formatMediaListColored('[%s] [%s] [%s] %s [%s] [%s] [%s] [%s]', [ 'id', 'isrc', 'upc', 'title', 'genre', 'label', 'relate
d_content_isrc', 'release_date'], @mediaList, 1);
}
return @mediaList;
}

Now, I’m not harping on the code itself – this is pretty innocuous, if slightly braindead. But what annoys me is the overuse of newlines to stand off functional blocks. They’re not necessary, the indenting is painful, and it makes the code hard to read. How hard is it to rework it like this:

sub _getMediaList( $ $ $ ;$ ) {
my ($carrierId, $contentType, $contentIds, $formatted) = @_;
my $gpd = GPD::getDbh();
my @mediaList;
return undef if (!defined($contentType)); # Charts can have no contents and thus no content type
foreach my $contentId (@$contentIds) {
my $resultList = mediaSearch(dbh => $gpd,
carrierId => $carrierId,
contentType => $contentType,
id => $contentId,
showdown => 1);
next if (!$resultList->[0]); # Shouldn't happen
push(@mediaList, $resultList->[0]);
}
if ($formatted) {
foreach my $media (@mediaList) {
$media->{title} = "$media->{artist} - $media->{name} - $media->{album}";
}
return formatMediaListColored('[%s] [%s] [%s] %s [%s] [%s] [%s] [%s]',
[ 'id', 'isrc', 'upc', 'title', 'genre', 'label', 'related_content_isrc', 'release_date'],
@mediaList, 1);
}
return @mediaList;
}

To me this is far easier to work with and understand where teh blocks are and how they work together. In my opinion, there’s a couple basic rules about code layout:

  • Consistent indenting – 4 spaces or 2, I don’t really care which, just use the same thing EVERYWHERE.
  • Curly braces to open a block go on the same line as the introductory keyword (if, foreach, sub, whatever).
  • Lines longer than 80 characters should be broken into sane sub-lines. Just hit enter and indent, folks. Not that hard.
  • Declarations at the top of the block when the variables are used across the entire block.
  • Separate declarations from functional code by a newline.

Now, this particular piece of code has no comments in it, so I’ll leave comments for another rant. But I think these basics sure would make perl code a lot more maintainable if everyone at least marginally followed them.

Gripe to Eclipse – Fix bugs, don’t put them off

Dear Eclipse Developers… having a bug open for 3+ years, through a myriad number of releases is inexcusable. Pointing fingers at different development groups saying “it’s their fault” and others saying “it’s those peoples fault” is not appropriate.
Just fix the damned bug already.
(NB – yes, that bug is only 6 months old. It is a merge from several other bugs reporting the exact same thing, see )

CONGO – And so we go to alpha testing

On Sunday night I released CONGO v2 into alpha testing. The first client to be using the new platform wants to be up and running on January 1st, and this is on-schedule. The last couple weeks haven’t quite been a death march, but there has been a helluva lot of code written, checked in, and tested. The log of messages posted to congo-dev tells the tale. While November wasn’t as intense as October, we still did over 105 commits against the server. The CONGO v2 codebase is rivalling the original system in size, though now it is entirely java based (the old system was part PHP, part Java).
I’m pleased with how the system is coming along. An entirely new payment interface, refactored contact information, and a new Events module for managing activities at conventions is worked into the new model (though not all of it is complete yet). I’m satisfied with the choice to move to using Struts2 as my web framework, but my dissatisfaction with OGNL grows daily. Why OGNL was necessary for the struts tag library, when the expression language (EL) was around is beyond me. For most of my code I’m replacing struts tags with JSTL and assorted libraries, and using EL for referencing action, stack, and session based content.
The next 2 weeks should see the last pieces of functionality falling into place, at which point we go into beta testing. All coding should be done, and we’ll be tuning the system for “go live” on January 1st.
After that? Well, due to the awesome new build structure, I’ll be able to release CONGO v2 for use by other folks. The system is now self-installing and configuring, so setting up a new host to run a convention is an order of magnitude easier than it was in the past. Stay tuned for details on how you can download CONGO and install it for your convention!

Randomness…

Misc ramblings, just have to get them out there.

  • KDE4 – Not Ready
    KDE4 is so not ready for prime time. Or maybe it’s whatever version is currently bundled with Hardy. Ick ick ick. Not stable, unuseable, bleah.

  • Struts2 – So far so good
    I’m getting into the proper mindset to go “Make that, wire it up, go.” in S2. I’m missing a good powerful JPA layer though – too late to roll Hibernate into v2.0, but I so see it in the future.

  • Laptop battery, not so good
    clipper has a dying battery. After 3.5 years, I’m not surprised, but it is sad. I’m only getting 15-20 minutes out of it now, the question is replace the battery or the laptop and the battery. Decisions decisions.

  • Scratch is still cool
    Zach is still totally emfatuated with Scratch. I watched him ‘program’ for a bit today, and he clicks and moves components around with the best of them. He even figured out how to get it to generate an error and pop up a diagnostic window (“Cool, huh? What is that?” “That’s smalltalk.” “Neat!”), etc. Ahh, my little hacker.

  • Work kicks my ass
    Tech Barbie says “Work is hard!”. It is – it’s taking up a huge portion of my brain and focus, which is something I’m not used to. I finish the day drained and wrung out, but still force myself to get code done for Congo before falling over for the night.

  • Hudson is cool
    We have a hudson install going on one our servers. It’s doing continuous integration builds and deployments. FAR better than the hacked up scripts of yor. Need a new build? *click* Build in progress!

  • Perl made tolerable
    I’m still stuck with working on perl sometimes. But if I have to go total immersion, I’ll likely use EPIC. It’s a Perl IDE plugin for Eclipse, and actually seems to work. Very tasty.

Nuff rambling. Back to the grindstone.

The weekend. Let me tell you about it.

So this weekend had me out to Ubercon down in NJ. All in all, things went pretty well. It was the second time I took Zach with me to an event, and he and blk’s son Justin had a riproaring time gaming, socializing, and geeking.

On a personal level, this wasn’t one of my banner events. It’s been a while since I ran at at-con registration of a reasonable size, and a lot of things conspired together to fail so that, by today (the last day of the con), I felt pretty down about my showing. Let me esplain. No, there is too much, let me sum up.

  • Mame – My MAME cabinet, which I’d been hauling down to Ubercon for now the third event, gave up the ghost last week. deathstar refused to boot, and I almost cancelled bringing the machine. The UC folks happily offered up some spare hardware, and I decided to bring the machine down. Early Friday morning I did an emergency load of Kubuntu 8.10 on a spare laptop, installed my MAME drive that has my roms in as an external drive, and configured up KXmame. The end result? Unstable, video modes not working right, and general bleah. I managed to keep it limping along through the weekend, but it was not the glorious, elegant machine of the last event. There needs be work here.
  • CONGO – Hardware – The server I use for events, ‘endor’, has been running faithfully for almost 20 events now. I’ve done one full OS reload, and for the most part it has been dependable as all git out. This weekend however some hardware twitches started to come up. First, the CMOS battery died, which causes the ‘things have reconfigured!’ message on boot. What I didn’t realize was it had also reset the dates, so that all the log entries for CONGO this weekend have a datestamp somewhere in 2006. This will require manual fixing. We also lost power twice due to a flaky outlet. The last bit was I attempted to cut back the amount of hardware I bring to events, and in doing so managed to arrive short 2 keyboards. Fortunately, Ubercon loaned me a pair so things were fine, but Grr.
  • CONGO – Software – For the most part, CONGO behaved appropriately and did all we asked of it. I’m itching to get v2 up and running, because of all the deficiencies I keep seeing in v1. But, the old tried-and-true still chugs along, and we cranked out hundreds of badges over the weekend.
  • Organization – I normally have a very competent reg manager running the event with me. This time, the normal Ubercon chap I work with was unavailable (due to health issues). While other folks helped man the desk (we were never short on people), not having an “In charge” Ubercon person with us really pointed out weaknesses in the process control in CONGO (things like cash drawer management).

Despite my grumblings, the convention was a success, and I think we did a bang-up job on keeping everything flowing nicely. I am utterly, 100% exhausted – the drives up and down take their toll, and caring for a 9yr old while running an event can be a bit taxing. Would I do it again? Absolutely, and will next year. For now, I’m going to go fall over.

CONGO Progress and Pretty Pictures

Another good night of coding!

After finishing a hefty project at work, and a busy week of Life Fun, I took 4 or 5 hours last night to hack into CONGO and try and get some rough edges worked out. I really feel like I’m approaching a useable system, with workflow doing what it’s supposed to, database tables updating the way they should, and crashes being few and far between.

I’m a visual person, so I wanted to come up with an up to date ERD for the v2 table structures. Ages ago I used DbDesigner4 to create an ERD for v1, but the tables have shifted a lot since then, and I wanted to see if I could auto-generate a diagram from table structures

Hunting around, it looks like DbDesigner has been absorbed into MySQL Workbench. I downloaded the latest version, and found myself bewildered by it’s interface until I realized what this thing was. It’s a tool for creating and designing databases, as opposed to a passive maintenance tool. Not… quite the approach I was shooting for – I was hoping for something like dbVisualizer that had a button that said “Make an ERD of this.”. While dbVis does have that to a certain degree, it’s pretty weak.

Eventually I figured out the incantations necessary to create an ERD with MySQL Workbench from existing table structures (it involves dumping the MySQL tables to a .sql file, and ‘reverse engineering’ the structures from that file). A few more mutterings and a dead chicken later, I had a visual representation of all my tables and their relationships to each other. Score!

Alas, the tool isn’t really a ‘reporting’ system, so the resulting graph needed some fiddling in Gimp to add titles and other text. Naturally I realized after the fact that I didn’t move a couple tables into the view before snapping the picture, but it’s still a pretty good representation of where I am.

Coding wise, v2 is missing a vital component – the Properties system is not yet working, and this is critical for basic operation. I started into it last night, building up new DAOs and domain objects representing a Property for a registrant and an associated PropertyDefinition. What’s left to do is write all the management tools around them to make them workable from Coconut, and start testing.

The last big project will be the public interface. In many ways, that’s going to be the simplest and most rewarding – it’ll be the outward facing side of CONGO. Essentially what registrants see when they want to attend a convention. Previous stabs at this worked reasonably well. Attendees at the various events were able to do the basics of what the event needed, but I’m really looking forward to going the ‘next step’ and offer a full service deployment.

My delivery date for CONGO v2 FC is December 1st. So far, I think I’m going to make it.

Alternating Row colors in tables in Struts

Have to share this one. I’ve been wracking my brain trying to figure out how to get table rows in a struts and JSTL supported page. I haven’t found an easy example anywhere, but I finally sussed it out. So if you do JSTL and struts based stuff, and want to do those nice ‘greenbar’ like lines on a table, here’s an example of how to do it:

<c:forEach var="rows" items="${registrantList}">
<c:set var="cellColor">
<c:choose>
<c:when test="${cellColor == '#f0f0f0'}">#ffffcc</c:when>
<c:otherwise>#f0f0f0</c:otherwise>
</c:choose>
</c:set>
<s:url id="url" action="coconut/gotoUser" >
<s:param name="rid">${rows.rid}</s:param>
</s:url>
<tr bgcolor="${cellColor}">
<td>
<s:a href="%{url}">${rows.rid}</s:a>
</td>
<td>${rows.lastName}</td>
<td>${rows.firstName}</td>
<td>${rows.company}</td>
<td>${rows.badgeName}</td>
</tr>
</c:forEach>

See? Simple!

Nobody Cares about your Build

My pal Owen (who, incidentally, is doing some wonderful work on CONGO) has just written an excellent high level of of build system philosophy entitled Nobody Cares about your Build. If you’re a developer or release engineer, or are just plain curious about build systems, check it out – here’s a sample:

A reliable, comfortable build system has measurable benefits for software development. Being able to build a testable, deployable system at any point during development lets the team test more frequently. Frequent testing isolates bugs and integration problems earlier, reducing their impact. Simple, working builds allow new team members to ramp up more quickly on a project: once they understand how one piece of the system is constructed, they can apply that knowledge to the entire system and move on to doing useful work. If releases, the points where code is made available outside the development team, are done using the same build system that developers use in daily life, there will be fewer surprises during releases as the “releaseâ€? build process will be well-understood from development.

Perl GRRRR.

Warning. I’m ranting. I’m annoyed. I’m frustrated. You might want to put on your rant-proof galoshes before proceeding.
Ready?
Perl should be considered a Write-Only Language.
Code that is written in it is generally incomprehensible by anyone but the person who wrote it. It’s great for one-offs, quick simple jobs, and low-scope tools. But anyone who says it’s appropriate for an enterprise level application (and I mean anything over about a thousand lines) should be strung up, draw, quartered, glued back together, and shot into space.
The main problem is Perl is so weakly typed, it should be considered not to have any. That means that parameters passed to methods are not type-checked, and therefore there’s no way to tell if you’re calling a method correctly, except to see if the app blows up when you run it. Editors and IDE’s cannot magically determine that “such and such a method requires an integer, a string, and a hashmap containing such and such values. You’re doing it wrong.”
Because of this weak typing and lack of structure, Perl libraries become worthless. Let’s leave aside the fact that Perl OOP implementations are complete and total trash (OOP is designed to organize your data into object form, and stabilize how components are used, defining a rigid structure so that when you use the component, you must use it correctly). In the case of libraries, it’s impossible to use a library unless you understand how it works. Libraries are not self documenting, they don’t even have a standard method of organization. They’re just loose collections of Perl code, again, with very little defined structure.
Therefore, people who come along to maintain code after the first person has suffered the above fate has no clue how a method is supposed to work, unless the original coder documented it, or built in type checking in their code. I have seen NO Perl code that actively makes sure the parameters being passed in are of the correct type. At best, they make sure the parameter are not null. And if they’re super-advanced, they even use prototypes.
And yes. I am in the unenviable position of maintaining thousands upon thousands of lines of undocumented, badly maintained, incomprehensible Perl code. Yes, the developers could have documented, formatted, and put the code into a form that’s easier to maintain. But they didn’t, and to me that’s the languages fault, not the developers. A language should have some element of maintainability, and not require the programmer to make up for it’s inherent ambiguity. If it does, it’s doubling the burden on the programmer. They not only have to write the code, they have to document it, and make it maintainable.
In my opinion… Perl by its very nature encourages sloppy organization and unmaintainable code.
And now, back to the trenches.

Vacation, Geeking, Code, and MySQL

I think I just made up for my lack of productivity toward the end of last week.
Database Fixes
I had been totally stymied by a problem in the CONGO rewrite – how to handle state of a registrant in a way that didn’t involve scanning history logs and essentially replaying an audit each time. The real stumbling block was defining exactly what states a registrant could be in.
Going by a recent event, it appears there really is a situation where someone may have signed up to work up to attending an event, but hadn’t actually completed the registration yet. In early version of CONGO, we called this ‘subscribed’. It caused all sorts of havoc in the larger events, because there should never have been a situation where someone was subscribed, but not registered.
Cept, it kept happening. The counts of people subscribed compared to the counts of people registered would drift, but in weird ways, and also in ways that were hard to source, due to the funky way CONGO v1 was calculating state of the registrant.
All of this is going away now, with a directly updated and maintained reg_state table that underwent some major surgery tonight. I introduced reg_state to help with this ‘what is the current situation of this person’ question that arises constantly. Apparently, though, I never quite completed tuning it. To wit:
o The table was a MyISAM table, rather than InnoDB
o It had no foreign key indexes at all
o It was not tracking subscription status at all (the logic was ‘if they were in the table, they were subscribed’
Etc etc. That’s all fixed now, and logic was introduced to automatically toss the registrant into the reg_state table whenever they were looked up in the current event. So they’re added, but NOT subscribed, that can be done later, but their state is no longer non-deterministic.
Layout Schmayout. Just show it
Well, not so much. The ShowRegistrants.jsp file may be my number-one edited file in the whole project. It’s the one that determines what the ‘registrant zoom’ screen looks like – probably the most viewed page in CONGO. And I keep fiddling with it. I’m currently trying to cut down the number of pushbuttons on it, and organize them in a useful way. It’s frustrating work, because I’m not really a UI designer. I’m sure I’ll good solid feedback from the CONGO power users out there. I hope ya’ll like it. *worry*.
The next step is attaching the reworked status display with functions (like ‘subscribe’ ‘register’ and ‘print badge’). These new hooks really change the workflow of the application, but I think it’ll be for the better.
Unfortunately, I’ve just gotten a poke that the next major event to use CONGO may be going live in about 10 weeks with pre-registration, so I have a hard deadline now for when v2 really needs to at least be in basic useful form. I really don’t want to deploy the old version again, so maybe this is a good kick in the pants to work toward a functional version in a reasonable timeframe.
Yeah, but… Vacation?
So, I did mention vacation up there. This weekend I’ve been up at the house in Maine with the family and the water and the woods. I suppose it helped to get away from the code for a few days, because sitting down tonight definitely got things rolling again. It’s the last ‘official’ day of the summer-vacation-house season, so we did things like pull the sailboat and power boat out of the water, put the docks away for the winter, and cleaned out the freezer. There’s always some meloncholy associated with labor day because of this. The weekends spent up here during the summer are great, and we had a wonderful time this summer. On the flipside, we’ll start having weekends back at home, which will primarily be taken up with packing for the move to Mosaic in a month or two, but I’ll still miss the mornings waking up to loons calling and the calm still lake out our windows.

Weekblog: Day 3 – Baby steps!

Another night of hax0ring, with not as much progress as I’d like, again. This time though it was mostly distraction as opposed to actual problems getting in the way. I did finish up the DAO for the Notes, and note creation and updating is working correctly. I’m having a little problem with auto-populating the TIMESTAMP columns, but I’ll figure that out. I’m just glad Struts is playing nice with the DAO and Doing The Right Thing [tm]

Oddly, I’m sort of daunted by the next steps, which are going to really start impacting the workflow in CONGO – making direct changes to how Notes are handled, how registrations are managed, and how history is kept. Up until now, things have been pretty much duplicating functionality (or at least adding to existing limited functionality). But I’m looking at rewriting the Registration page and how history and status are being kept. This is at the core of how CONGO works – the very reason that, with something like 50,000 registrations having been run through the software since it’s first rollout, we haven’t corrupted a single table, or ended up with a mysterious situation where a users status is wrong or lost in a way that was not easily determined.

I’ll get past this, I know, but it still makes me nervous.

On the “face of the future’ front, though, we have the emerging view of what the database will look like. This first image is the old view of CONGO’s tables. No foreign keys whatsoever, that means only programmatically enforced referential integrity.

Here’s what my working v2 table structure looks like. I’ve only been working with address contacts and notes so far, as well as conference master data, but it’s all starting to tie together as it should. By the way, these images are generated via dbVisualizer, an awesome generic SQL database tool that has a free mode that’s extremely useful (and includes ERD diagrams like this 🙂

Bit by bit, I’m moving forward.

Weekblog: Day 3 – The JDBC Whubbidah huh?

Today, durnit, I was going to get some work done. I specifically scheduled time after $dayjob to spend working on CONGO v2, and ya know what? I actually did it!

Tonight’s joy was parking at the Panera in Marlborough, plugging in the iPhone for music, and getting down to coding. With the network being twitchy in the restaurant, I actually had about 2 hours of uninterrupted focus time to code, and I was productive!

I finished up much of the underlying work on the Notes system (the DAO and Data object are done), and was working on wiring up the Notes interface into Coconut (the web front end). One of my goals with this whole project is to upgrade how I’m managing database resources, and doing things ‘properly’. For instance, CONGO v1 had absolutely no referential integrity mechanisms in place. No foreign keys of any sort. This meant I could easily have had orphaned data in many of the tables, there was just no way to tell.

The first step to setting up proper RI is to use foreign keys to link one table to another. The newly formed Notes table now has foreign keys linking to the reg_master table (which holds registrant master information). I did learn a couple basic concepts while doing this though. For instance, if the source of a key reference is null, then the foreign key constraint is not checked (handy!). If it has any other value, it must have a corresponding key in the target table. I found out quickly that specifying source columns as NOT NULL in the database scheme wasn’t going to work, particularly on columns that get populated later.

The other thing I wanted to fix was date handling. I had a mishmash of DATE column types and raw text fields storing dates in the old version. MySQL (and most databases) have a handy column type called TIMESTAMP that can not only store time down to fractions of a second, but also support basic math operations (“Show me all rows that have been modified in ‘now() – 3 HOURS’ is a valid construct).

Here’s the problem though. MySQL DATETIME columns cannot be null. So what if I don’t have a date to put into the column yet? Well, the docs say the schema should say ‘colname TIMESTAMP DEFAULT 0’ – I did this, and it seemed to take. But when I tried to retrieve the row in my DAO, I was getting this error:

java.sql.SQLException: Cannot convert value '0000-00-00 00:00:00' from column 6 to TIMESTAMP.

The offending line causing this was (‘where’ is a Note object, postDate is a java.sql.Timestamp, and ‘fromwhat’ is a ResultSet):

where.postDate = fromwhat.getTimestamp("note_postdate");

I completely and totally don’t understand this. I’m wondering if this is a bug in the JDBC driver I’m using, or what, but I should not be trying to load a date in the format 0000-00-00 into the java.sql.Timestamp object. It should be returning NULL, and therefore ‘postDate’ should be null, as it was when it was created.

I ended up having to code logic around it to say basically “Don’t try to load the note_postdate value unless circumstances can guarantee there was a value there.” I tried various approaches for ‘Is this column going to be null?’ – but it really appears as if the JDBC driver is delivering a Timestamp object, even though the source is null.

It gets a little better. In the MySQL docs, they say, as I mentioned above, that the declaration for a TIMESTAMP without a value should be ‘default 0’. But after I create my tables from my schema (using that syntax), and then do a ‘SHOW CREATE TABLE reg_notes’, lo and behold, I see:

CREATE TABLE `reg_notes` (
`note_id` int(8) NOT NULL auto_increment,
`note_rid` int(8) NOT NULL default '0',
`note_cid` int(8) default NULL,
`note_postrid` int(8) default NULL,
`note_ackrid` int(8) default NULL,
`note_postdate` timestamp NOT NULL default '0000-00-00 00:00:00',
`note_ackdate` timestamp NOT NULL default '0000-00-00 00:00:00',
`note_message` varchar(100) default NULL,
KEY `id` (`note_id`),
KEY `rid` (`note_rid`),
KEY `note_cid` (`note_cid`),
KEY `note_postrid` (`note_postrid`),
KEY `note_ackrid` (`note_ackrid`),
CONSTRAINT `reg_notes_ibfk_1` FOREIGN KEY (`note_rid`) REFERENCES `reg_master` (`master_rid`) ON DELETE CASCADE,
CONSTRAINT `reg_notes_ibfk_2` FOREIGN KEY (`note_cid`) REFERENCES `con_detail` (`con_cid`) ON DELETE CASCADE,
CONSTRAINT `reg_notes_ibfk_3` FOREIGN KEY (`note_postrid`) REFERENCES `reg_master` (`master_rid`) ON DELETE CASCADE,
CONSTRAINT `reg_notes_ibfk_4` FOREIGN KEY (`note_ackrid`) REFERENCES `reg_master` (`master_rid`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1

Observant folks may note that the note_postdate and note_ackdate columns now have default values of ‘0000-00-00 00:00:00’. Not what I specified in my creation SQL.

I’m past this now, and it’s working, but if anyone has ideas about why this is behaving this way, I’d love to hear it. It’s damned frustrating.

UPDATE 08/28/2008 – Through the magic of RTFM and some nudges from Imre, my table definition was wrong… Here’s what it should be:

note_postdate TIMESTAMP null DEFAULT null,
note_ackdate TIMESTAMP null DEFAULT null,

This little bit of MySQL fun is in fact documented, it’s just buried in the TIMESTAMP page on dev.mysql.com

Weekblog: Day 2 – or ‘That didn’t go as planned’

As the title says, yesterday hardly went as planned, as evidenced by me just getting around to posting at 9:30 this morning.

The Geekery. It is everywhere!
First, the day was full of IT-related geekiness, trying to get SASL authentication working on the the mail gateway for the Greater Homeport Server Cluster. What should have been an hour long setup and fiddling turned into a 6 hour “Learn what SASL is, learn what TLS is, learn how Postfix manages it, learn how Cyrus implements SASL, learn how to set up standard authentication, and learn how to do this all on an extremely busy mail gateway without killing all inbound mail” process.

In the end, we were only partially successful. The main impetus for this setup was making it so Cat and I could send mail from our iPhones without using Googles SMTP gateway (if you have a gmail account, you can use SASL authentication and send mail from anywhere via smtp.google.com. The drawback? Any mail going through that gateway gets it’s From: line rewritten to the gmail account ID.)

We got most bits working, except for finding out at the end that v2.0.2 firmware on the iPhone has a problem with self-signed certificates on TLS-enabled mail servers (but this is only for SMTP connections – Safari et al has no problem with self signed certs. Annoying). One assumes Apple will be fixing this soon, but it means we can’t have encrypted inbound mail connections from the iPhone. Adam has suggested there is a list of certs that free and Apple certified, I may end up going that route.

Oh yeah, and about the project I’m supposed to be working on…
But that’s not what I came here to talk about.

The point of this week’s blogging was to get focused on CONGO v2 and make some progress on getting toward an Alpha release. To accomplish this, I need to get 2 hours or so a night of fairly focused work time.

Last night I failed.

The main culprit? Lack of sleep. I went out to dinner with my mom last night (mm, steak!), and ended up getting home around 8:30. Not too bad, thought I, I still have a couple hours before bed. Flopped down on the couch, had the laptop out, tossed a movie in, and started work and… fell asleep right on the couch. An hour and a half later, I woke up (now about 10;30), was immensely groggy, and decided “Right, that’s it. Bed.” and flopped right into bed, sleeping like the dead til 8am this morning. Total sleep time? About 10.5 hrs. How I feel today? Stiff, but caught up.

Unfortunately, this meant I got only a minimal amount of work done on CONGO last night. I fixed some of the Notes DAO setups, and started work on modifying the ShowRegistrant functions to enable the notes browsing. I’m trying to decide if the Notes listing should be attached directly to an instance of a Registrant, which means it gets populated whenever a Registrant is instantiated. This really only happens on single views (ala, getting a list of Registrants from a search query doesn’t load the entire Registrant, just an index), so it shouldn’t make things too burdensome. It has the advantage of making everything easily accessible if you have the Registrant at hand. Right now a registrant’s attendance history is attached on instantiation (for Arisia, this is perhaps 20 rows, but that’s somewhat unusual). If I don’t embed the list in the Registrant object, I’ll need a second set of lookups and queries when viewing a registrant, which is a little cumbersome.

For now I think I’ll embed it, and hook up the DAO’s appropriately. It’ll make the display logic a lot easier (data is at hand), and if we end up beating the bejeezus out of the database for largish events, I’ll look at optimizing it.

Conclusion
Onward and forward. Today I need to focus on the paying job and finish up an annoying script, hoping to get it audited and approved by the other engineer. Have I mentioned that Perl is not my favorite language? Oh there is a rant pending…

Weekblog: Day 1

I didn’t get as far as I had hoped to tonight, due to a variety of distractions – it was an impressive list of “things that get in the way of previous plans” (Work actually had work for me to do, then a stop by the Mosaic work site to check on some supplies, then dinner, then home to find the freezer has stopped freezing (due to a broken icemaker), then helping out Cat with some finance stuff so we can get our Mortgage application in. Ung!

Finally I sat down, and with The Time Machine on as background entertainment (not so good a movie – great for background noise – and what the heck is with Jeremy Irons? Is there no movie so bad he won’t act in it? Sheesh) – I got down to work.

There were some great comments from yesterday on how to handle historical triggers on database updates. There’s some challenges to using triggers (such as “who made this update?” – though using a ‘last updated by’ column on every record could fix this, I’m still not sure if it’s a good idea). Jonah pointed out also that using database triggers limits the portability of the database layer. Given that MySQL isn’t going anywhere, and that it’s getting more and more attention as an enterprise-level database system, I’m not that worried about it, but I am taking that into consideration.

Tonight though I needed to find something short term to work through, so I focused on how ‘notes’ are applied to registrants. In the past, I would make a “NOTE” entry in the history table. These NOTEs could be anything from just an informational note by the operator (“This person was nice”) to something important (“Paid by check”). Having NOTEs in history was “okay” but without filtering, it was hard to see operator comments in a meaningful way.

I also had a concept of ‘NOTICE’ records. These are essentially NOTEs to be displayed to the operator whenever a registrant logged in. They would show up as red messages on the registrant detail screen, and the operator would have to acknowledge the NOTICE before proceeding (that’s the theory – in fact there was no logic to enforce this). These NOTICEs were very, er, noticeable to the operator, and were great for things like “DO NOT ADMIT THIS PERSON UNDER ANY CIRCUMSTANCE” (yes, we had a few of them, and I know of one instance where the CONGO ‘NOTICE’ record actually did prevent them from entering the event. Yay!). Generally though they were used for innocuous things like “STILL OWES $25” or the like.

For v2, I’m seperating out the NOTICE / NOTES into a separate table simple called ‘Notes’. The option to making any note entry a ‘notice’ will be on the note creation, and it’ll just be a flag on the record. Also the record will keep track of when the note was created, and by whom, as well as when it was acknowledged, and who acknowledged it.

The basic DAO structure is there, as is the data object, and I’ve created the schema file for it as well. Nothing is wired in yet, but I’m also still noodling around how I want the record to look, so I’m sure there’ll be more work moving forward.

The other bit I did tonight was trying to squash a problem I’m having with the JDBC connection pooling. I’m using the C3PO connection pooling tool, and if CONGO has been idle for a while, I’ll get a connection communications failure. Googling around hasn’t showed me anything immediately obvious, though others have said they’re having the same problem. What I’m trying now is adding some auto reconnection options to the JDBC connection string, so my c3po setup now looks like this:

private void configurePool() {
logger.debug("Empty connection pool, recreating...");
ComboPooledDataSource cpds = new ComboPooledDataSource();
try {
cpds.setDriverClass( "com.mysql.jdbc.Driver" );
} catch (PropertyVetoException e) {
e.printStackTrace();
}
cpds.setJdbcUrl("jdbc:mysql://localhost/congo?autoReconnect=true&autoReconnectForPools=true");
cpds.setUser("root");
logger.debug("Initted.");
applicationMap.put("cpds",cpds);
}

I won’t know if it’s working yet until I idle long enough to force a timeout. I’ll let ya’ll know.

Weekblog: Day 0

Just quick update on this. Last night I did get some coding time in after blog-posting. The fix this time was a problem with Create Registrant that was throwing SQL errors all over the place. Turns out I never actually completed writing the DAO for the Registrant data object, and the create() method was broken in a ton of interesting ways.
There’s still a twitch in restoring the state to the editor after creation – it’s not reloading the new registrant into the sessionspace after creation, so you end up with a sort of ‘blank’ registrant screen. Exiting and coming back shows the new entry, so it is creating it, just isn’t reloading.
I’m sort of worried about keeping history right now. The old code did everything in the history table programmatically – so whenever I made a change to a registrant’s information, I had to make a call to logEntry() to create a new history record. So, naturally, there was some inconsistencies in the logging. It also didn’t help that the history table was critical information. It was part informational, part auditing. In fact, some parts of CONGO used the history table to recalculate statistics and status on registrants. (Note – that is SO not going to be in the rewrite).
Anyway – regarding logging, I’m looking into learning MySQL triggers to have registrant history updated automatically whenever a change happens. That way I don’t have to always remember to put in a logging call. We’ll see if this is viable (for instance, how does the trigger know what the ID of the operator is? The one who is actually making the change?)
Stay tuned. 🙂