Things I Learned about CONGO at Arisia

This year’s Arisia, as previously noted, was special in that while I wasn’t actively running registration, I was functioning as technical support and consultant for the process. There were a number of organizational, technical, and logistical challenges that came up, and I learned some interesting things.

The Printers

Ahh, the printers. If there’s anything that better personifies the challenges of running registration, it’s dealing with the badge printers. I made a decision back when I first starting doing events in 2003, that badges for events should be printed on the fly, in realtime. When a person comes to the registration desk, they should be able to be checked in, and their badge should come right out of the printer. What this means is the system must be up, functional, and running at peak throughout the entire event.

I figured out at one point that I’ve run something like 50,000 attendees through CONGO and my printers. Some at large > 10,000 attendee events, and some events as small as 150. Every one of those events has it’s own challenges. But in there, I got a very good handle on queue operation and throughput management. Badge printers only run at such and such a speed. If you’re registering people faster than the badge printers are running, there’ll be a backlog, etc etc. I’ve only had a serious backlog at the badge printers three times in all my time working events, and each time I’ve increased capacity and stability each time. That doesn’t help the folks who had to wait in line (and in fact I’ve lost a major customer because of this), but it does make it so it won’t happen again.

Until this past year, I had been using the same badge printers and print queuing system. Fargo Pro thermal printers. I have 3 of them, and for the most part, they’re solid, hard working printers. At the last big event, I had all 3 running, but realized too late that the HP Jetdirect EX 3 Plus ethernet print server, which sounded like a dandy way to drive multiple printers… was in fact single threaded. It could handle only one print job at a time. So while I could have 3 printers hooked up, only one could be printing at any given moment. Needless to say, this was not an optimum solution

So over the last year, I’ve invested in replacing all my badge printers with Evolis thermal printers. I now own 2 Pebble3 printers, and a Dualys, all purchased off eBay. These printers are USB based, are much smaller and lighter than the Fargos, and are much much faster. Cycle time from power up to badge coming out is about 4 seconds (a Fargo, from a dead stop, would take about 12). We had used CONGO and Evolis printers at two other events already, but this was the first time they’d be under ‘heavy load’, and being pooled into a shared CUPS print destination

First, I have to say, it worked fine. We ran the event beautifully, but it turns out we were not running the printer pooling right. Due to the speed of the Evolis printers, it didn’t matter, but because of some oddities in how CUPS assigns USB ports when a printer goes on or offline, we rarely had both printers in ‘pool’ mode for more than about a half an hour. Hopefully, I’ll be able to figure this problem out going forward, but alas, it may be that a USB based printer pool, with identically named print devices, may not work well. There is no way to identify each printer – they’re assigned dynamic printer port names as each device powers up. So printer 1 will become printer 2, etc etc. Adding to the issue is that it appears the Evolis printers take themselves completely offline if there is a problem. CUPS then removes the printer from the pool, and that printer no longer receives jobs, even if it’s put back online. I need to go into the CUPS control panel, and re-add the printer. But which one is it? Printer 1 or Printer 2? Aaargh

Newer versions of the Evolis have ethernet ports built in, and I suspect the proper thing to do is start using them directly. But I already have a fairly heavy investment in these printers, so I’m not sure really what I’ll do.

The Software

I have been doing some fairly heavy hacking on CONGO over the last year. I completely flushed out the horrific mess that was the badge printing interface, and replaced it with a generic layout mechanism that any registration operator can use to lay out the badge imagery. I also replaced the “draw the badge into a big GIF, and then print the GIF” way of doing things with a PDF library – the fonts look much better, text automatically wraps in blocks on the badge – a Major improvement.

Also along the line here I modified and cajoled CONGO into a configuration that will work properly on Windows machines, and documented the whole process on the CONGO wiki. These are just the next steps necessary to release CONGO into the opensource world.

Yep, you read that right. I’m finally opensourcing CONGO, under a new license. It’s really time for other events, particularly fandom and gaming events, to have access to this system. Stay tuned for more info on that front. I still have to work out some of the license complexities, but it’s definately coming

The Terminals

I bit the bullet and ditched LTSP for simply installing a mini Linux installation on each of the SFF PC’s I had accumulated. I settled on Damn Small Linux for it’s very simple installation and “It just plain works” desktop. I had high hopes for PuppyLinux, because of it’s nice desktop and active community, but the project seems to be stalled, and I could not figure out how to do a hard disk install. After 2 hours of tinkering, I gave up. DamnSmallLinux installs from power on to “installed on the HD and ready” in about 11 minutes. Hard to argue with that.

There was some trepidation doing so much change just before the event, but I spent a fair amount of time in the basement, setting up each workstation, testing it’s booting and configuration, and documenting all the settings. I was pretty comfortable.

The Server

Well, if you’re going to revamp everything, why not do it whole hog? Endor, veteran of dozens of cons, and even being left in an airport, was due for a rebuild. Debian Etch came off, and Ubuntu Feisty went on. I kept all my data dumps, and spent a few days reconfiguring all the little things that had been bugging me through all these events. The primary goal was to make the server and workstations ‘setup-able’ by mere mortals. To that end, I…

• Made ‘endor’ boot up to a desktop, rather than a login screen
• Made the CONGO appserver start and stop automatically via a custom written /etc/init.d script
• Made sure the print queues came up by default
• Configured the workstations to boot cleanly and start up Firefox, with CONGO as their home page

With these changes, the folks running Registration had no problems setting up and tearing down the system. Even when someone mistakenly unplugged the server, they were able to bring things up without any intervention by me. This is in stark contrast to previous years, where manually starting and stopping the server was a complex process that I or one of my well trained cohorts had to do.

The Supplies

We purchased the badges from a print shop in Texas, and I have to say, I’m totally satisfied with the results. The quality is excellent, and they were communicative and helpful. For once, I didn’t have huge stress around the badges (previous events have seen us receiving badges with the wrong coating on them – when the badge is run through the printer, the thermal ‘ink’ doesn’t stick to the badge. I’ve had 3 events where this has happened, and let me tell you, it’s no fun.

We also used nylon ‘clips’ for each badge. These come in bags of 100 or 500 each, and unfortunately, some overly eager operator ‘cleaned up’ the area Friday night, which resulted, we suspect, in some clips being thrown away. Saturday late morning we realized we weren’t going to make it with the current stock, and I made a quick emergency drive to my home in Natick, and picked up another bag full of clips. Yay for being nearby!

Ribbons and cleaning supplies went over fine, and changing the printer ribbons in mid-con went great. The Evolis printers take very simple ribbon assemblies, and each ribbon runs for 1000 prints. We changed out the ribbons in both printers on Friday evening, and after figuring out how to run the cleaning badge through one machine, we were all set for the weekend.

One thing I’m still kicking myself about is I did not bring any lanyards. I had tried to get an order of lanyards from one of my suppliers, but a comedy ensued where the sales rep quoted me a price that was 3x as expensive as what was on the website. Alas, it got too late, and I missed the opportunity. What made this doubly frustrating was when I was home getting the badge clips, I discovered a box of 1000 lanyards up on a storage shelf. Sheesh.

So where to now?

Now that the con is over, I’m ready for the next phase of work. I don’t think there’s a lot of need for change in the system layout (though moving the monitors to and from events is still a pain), but CONGO needs ‘the next phase’ of work. I’m already in mid-coding, and our Bugreports page still has a lot of open issues. But even with these known things, I have, in my head, a good idea of what the next steps are:

• Make a ‘publically accessible’ registration page, where attendees can visit and check on their registration status or register for events.
• Finish cleaning the codebase
• Finish documentation so ‘those other than me’ can use it
• Release the source
• Work on some more detailed reports and information analysis
• And, most importantly, start expanding CONGO into more event management areas. Manage events and panels as well as just plain registration. There’s another application called Zambia, written by some Arisia folks, that interfaces loosely with CONGO that can manage panel subscriptions and brainstorming. We’re working together to see if we can make the connection stronger and more stable.

Conclusions

For me, this was the most stable and best run convention yet, particularly for it’s size. I feel like I’ve hit a stable plateau for the system and the software, and I’m ready to go to the next step.